Iterative Marketing Podcast Episode 40: Tools For Effective Experimentation

In this episode, we explore the specific tools that are necessary for effective marketing experimentation. Setting up the infrastructure, or template, for which the experiments will be conducted is the first step for successful experimentation. We will also identify what to test, how to measure the data, best practices for analysis and, most importantly, how to communicate the results to stakeholders.

In this episode, we discuss…

Tools for Pre-Planning (05:20 – 22:15)

• This must be in place before an experiment can start so we can track the objective of the experiment.
• The template can come in various forms, such as a Word Document, Excel Spreadsheet or a plug-in for existing software or project management system.  Be sure that the template that is chosen makes sense for the organization.
• The form, or template, is vitally important to the overall success of an experiment because it will help maintain the overarching objective and keep everyone on track.
• The following information should be included within the form:
  • Objective: what impact will the insights have?
  • Question: what questions are we trying to answer?
  • Persona: who is our target audience for the experimentation?
  • Measurement: what metric(s) will we use to determine our outcome?
  • Estimated sample size: how many people will comprise the sample size to produce a worthwhile result?

Tool: Documented Persona/Market Segment

• This is the “who” of our experiment and allows us to zone-in on an audience.
• Good experiments have a defined audience that are segmented by persona.
  • The persona should be front-and-center while designing an experiment.
• Ultimately, our goal is to answer the question: What don’t we know about the persona that would help us market to them more effectively in the future?

Tool:  Analytics (11:30 – 13:20)

• This is the “what” of the experiment.
• Google Analytics, or similar programs, will help to recognize patterns in website traffic.  This will allow us to identify opportunities for experimentation to gain a better understanding what we should be testing.
• Examine the flow of visitors through a website and look at which landing pages have the highest levels of engagement.  
• Engagement metrics include:
  • Pages per visit
  • Bounce rate
  • Dwell time
  • Scroll depth

Tool:  Screen Recording  (13:28 – 14:35)

• Screen recording is a tool that can approximate a visitor’s engagement with a particular site.
• While statistically significant data is hard to gather from these anecdotal recordings, watching people interact with a website in real- time provides invaluable insight into customer habits.
  • HotJar and CrazyEgg will record visitors’ interactions with a website.
• Take note of sessions that start on key landing pages.  This information will generate great questions that should be added to our experiment template.

Tool:  Heat Maps (14:44 – 16:12)

• Like screen recordings, these maps provide insight into how visitors are interacting with our site, and provides layers of data that will help us recognize patterns.
  • Hotjar and CrazyEgg will also generate heat maps. The visual will show:
    • Where visitors are hovering (desktop only)
    • Where they click or tap
    • How far they scroll
• Reviewing heat maps will generate questions that should be added to the experiment template.

Tool:  User Experience Testing Platform (16:14 – 19:33)

• This is a real-time recording of people interacting with your website.  It provides real-time feedback from targeted users, allowing us to get inside the head of our target audience while they interact with our site.
  • Usertesting.com is a great tool for B2C companies to gather data on their customer’s digital experience.
    • A good test would include: the flow from ad to purchase with a panel of users and watch them interact, and more importantly, hear them speak aloud about what they are doing and why.
  • This tool is too expensive to gather statistically significant data but an awesome way to identify things to test.

Tool:  Tickler File (19:35 – 21:08)

• This is a place to store experimental ideas.  Use whatever internal document that works best for the organization.  Also, keep track of what should be tested and maintain a tickler file of potential experiments that can be conducted in the future.
  • Be disciplined when it comes to documenting future experiment ideas so that information can be accessed quickly.

Tool: Sample Size Calculator (21:10 – 22:16)

• A calculator is helpful in the following ways:
  • Identify a minimum sample size to ensure our test produces statistically significant results.
  • Helps to avoid an experiment that may not be fruitful.
  • Examine the feasibility of a test before committing to it.
• A sample size calculator indicates how big our sample (how many website visitors, ad impressions, email sends, etc…) needs to be to have confidence in a winner.   Also, how wide of a margin is needed to make an experiment worthwhile.
• For example, if we are only expecting a 1-2% improvement (and that small of an improvement is worth your time) we might need a really big sample size.
• However, if we’re expecting a more dramatic upturn we can get away with a smaller sample.
• Optimizely has a great sample size calculator on their site.

Tools for Measurement and Gathering Data (23:35 – 27:40)

• Google Analytics is the backbone of our data selection.  We can aggregate website, or application, data and slice it up a million different ways.
  • Google Analytics helps us measure the data in ways that the other tools cannot.  If analytics are configured correctly:
    • It can be set it up to measure our KPI’s using quantitative data.
    • It handles large volumes of data with ease.
    • It let’s us dissect our data by time period, by traffic source, and by many other dimensions.
  • Three things must be set up properly before starting an experiment:
    • Good URL tagging
      • Information must be passed into Google Analytics about where traffic is coming from and what we know about the traffic.
      • Make sure that every single link within a website has proper tracking parameters, or tags on the end of the URL, so we can measure the media’s impact on our customer journey.
    • Conversion tracking
      • In order to effectively measure our results, conversions must be set up in Analytics.
      • These directly tie to our KPI’s and allow us to measure:
        • Leading indicators such as case study downloads and add-to-cart clicks.
        • Trailing indicators which include actual sales and revenue.
    • Data cleanliness filters
      • Unfortunately, our target audience isn’t the only group visiting our website.  Bots, human testers and our own team can adversely impact our data.
      • Basic filters should be set up in Google Analytics to ensure our data is clean and remove as much superfluous “noise” as possible.

Tool:  A/B Testing Web (27:45 – 30:16)

• Includes: Optimizely, Convert.com or Google Experiments
• A good A/B testing tool is imperative to optimize landing pages and decide what words and images move our audience.
• These tools are great for testing headlines, calls to action, visual cues and content order.
• The gold standard is Optimizely.  It lets us set a single piece of code on a website or web page and then work within the tool to set up our experiments.
  • A what-you-see-is-what-you-get editor creates an alternate, version B, of a landing page.
  • Right within the tool, we can configure the experiments so a percentage of visitors will view version A and a percentage will view version B.
    • If a visitor returns to the site, they will always get the same version of the page they got the first time.
    • The tool then reports back on the results, even reporting if the results are statistically valid.
• Optimizely is the easiest to use and most powerful of the tools, but it can be expensive.
• A lower-cost alternative to Optimizely is Convert.com. It is only slightly less powerful than Optimizely, but is just as good and easy to use. We have had great success with this tool.
• Of course, we don’t have to pay anything for A/B testing. Google has a free tool built into Analytics called Google Content Experiments.
  • The downside to Google Content Experiment is that it’s not quite as easy to use and a redirect is necessary to load version B (or a developer must write the code to support the experiment).

Tool: A/B Email Testing (30:17 – 31:15)

• This approach utilizes email marketing or a marketing automation platform to run A/B tests.
• Most email marketing tools have an A/B tester built-in, which helps us in two ways:
  • It guarantees a random sample
  • It will send both versions at the same time

Tool: A/B Testing Ads (31:16 – 32:55)

• This is tricky to do and typically requires a third-party ad server to execute correctly.  To gather the best results from an A/B ad test we must ensure that once someone sees version A they only ever see version A.
  • If access to a third-party ad server such as Atlas or DoubleClick Campaign Manager (formerly Doubleclick For Advertisers (DFA) isn’t feasible, reach out to an analytics firm such as Brilliant Metrics to assist with the set up.
• A/B testing ads work well for display and video ads
  • Unfortunately, with Facebook and Twitter, setting up an A/B test is complicated. But there are some ways to work around it.

Tool: Online Survey Platform (33:00 – 35:29)

• Sometimes the information we are seeking can not be determined by analyzing behavior, such as: how people feel or what they know about our brand, product or service.  This is where an online survey works well.
  • There are two paths to choose between:
    • Assemble a survey in-house and select respondents
      • The most popular platform is SurveyMonkey but an alternative called SurveyGizmo works well, too.
    • Pay for a panel and utilize a company like Survata. Or, take advantage of the panels offered in some other platforms like “SurveyMonkeyAudience.”
  • Another alternative is Google Surveys which is a hybrid between a survey tool and a pop-up polling tool.
    • A benefit of Google Surveys is that it provides a large panel in which to work; however, it has serious limitations on the number of questions that can be asked and also how narrow an audience can be targeted.

Tool: Website Poll Platform (35:30 – 36:40)

• An alternative to a survey can be a website poll. Platforms for consideration include: HotJar or Google Survey
• Two hurdles that will need to be addressed:
  • First, we need a lot of website visitors, and website visitors willing to answer our poll, to gain statistical significance.
  • Second, we’re limited to asking one question and it had better be easy to answer or we won’t collect enough data.
• For some brands with a large number of website visitors, a website poll can be an effective way to get questions answered that can’t be obtained from behavior.

Tools for Analysis (36:42 – 37:54)

• Tool:  Google Analytics
  • Analytics is an easy choice because it provides decent visualization tools and it is fast for ad-hoc reporting.
  • Analysis can be done without Google Analytics however another platform will be necessary to gather after-the-click data.
  • Pro Tip: learn custom reports on the customizations tab to really get the power from analytics.
• Tool:  Excel/Google Sheets (37:58 – 39:40)
  • Sometimes data will need to be combined from multiple sources and then dissected for proper analysis.
  • Be sure to have a good understanding of pivot tables.
• Tool: A/B Confidence  Calculator (39:44 – 40:15)
  • We must avoid the pitfall of assuming that because we have a winner that we have a result.  
  • As we discussed in episode 7, we need to use an A/B confidence calculator to determine if your results are statistically significant.
• Tool: Google Data Studio (40:16 – 41:10)
  • This is a new tool offered by Google which pulls data from multiple sources and produces gorgeous visualizations.

Tools for Communication (41:11 – 44:30)

• In order to realize the true benefit of experimentation, we must disseminate it to the rest of the organization.
• Tool:  Graphs
  • Graphs can be generated from Analytics, Excel or Google Sheets
  • A dashboarding platform can be a great asset, but only after important KPI’s are established and it’s important to continually update the platform.
  • Excel works just fine to produce graphs.
• Tool:  Experiment Log
  • A simple log is an invaluable tool to make sure that we are building off of past experiments and not repeating the same experiments too early.
    • The log should include a way to review experiments by audience or persona.
    • Google spreadsheets are a good tool or whatever works well for the organization.
• Tool:  Persona
  • We circle back to the persona again as a tool in our experimentation toolbox at the tail end of the experimentation process.
  • Updating the persona serves to close the loop and make sure to articulate what we’ve learned to marketers, sales professionals, vendor partners, customer service team, etc… this is vitally important to training and decision-making.

For more information on the charity in this episode, please visit Map International.

The Iterative Marketing Podcast,  a production of Brilliant Metrics, ran from February 2016 to September 2017. Music by SeaStock Audio.

Learn more about Iterative Marketing and listen to other episodes on Apple Podcasts, YouTube, Stitcher, SoundCloud and Google Play.