Iterative Marketing Podcast Episode 7: Designing an Effective Marketing Experiment

Marketing experiments provide valuable data and insights for organizations to improve customer experience and ROI. Learn how to effectively design your next marketing experiment to get the best results.

Episode Show Notes

Introduction to Marketing Experimentation

(0:00 – 1:45) Introduction To Iterative Marketing Podcast: The Iterative Marketing podcast provides actionable ideas and techniques to help marketers and entrepreneurs improve their marketing results. The hosts, Steve Robinson and Elizabeth Earin discuss various topics each week. In this episode, they talk about the process of designing an effective marketing experiment. The podcast also offers a blog and a LinkedIn group for its community of “iterative marketers” to share ideas and ask questions. Please visit to learn more.

(1:45 – 2:05) Introduction To  Effective Marketing Experiment: In marketing, there are two main ways to use experiments: to refine content and develop insights and to refine channels or delivery methods for that content. These experiments help marketers improve their results and better understand their target audience.

Creating Effective Experiments

(2:05 – 4:43) Refining Content for Better Results: By conducting experiments on content, marketers can gain insights into their target audience’s preferences and motivations. These experiments often focus on direct response content, such as landing pages, ads, or emails that ask users to take specific actions. By testing different variations of these direct-response ads, marketers can gather data on what works best for their target audience. This data can then be applied to other types of advertising, such as brand advertising, to better appeal to their audience’s emotions and needs. Examples of experiments on content could include testing landing pages for different types of properties in the real estate market or split testing new property listings in email marketing campaigns.

(4:43 – 7:33) Controls and Variables for Accurate Results: To set up an effective marketing experiment, marketers need to establish controls and variables. Control is the version of an ad that is already running and works consistently, while the variable is the one where a specific element is changed. When conducting experiments, it’s important to maintain the same objective for both versions of the ad to ensure that they can be compared accurately. Additionally, only one variable should be changed at a time to avoid invalidating the test results. The visual hierarchy of the ad or landing page must also be maintained to keep the same visual weight and ensure that any differences in the results are due to the one variable that was changed. By following these guidelines, marketers can gain insights into what works best for their target audience and apply these findings to their overall advertising strategy.

(7:33 – 9:22) Testing Message, Delivery, and Call to Action in Experiments: Marketing experiments can involve changing three main elements: the core message, the way the message is delivered, and the call to action. By testing variations of these elements, marketers can gather data on what resonates best with their target audience. For example, in the real estate industry, marketers can test different imagery, headlines, and calls to action to see which ones generate the most conversions. It’s important to change only one variable at a time to ensure accurate results. Although limiting changes can be difficult, it’s crucial to obtain the cleanest data possible from the experiment.

(9:22 – 11:46) The Role of Technology in Effective Marketing Experiments: To ensure accurate results in marketing experiments, it’s crucial to use technology that allows the same people to see the same version of an ad or landing page every time. This requires using tools such as, Optimizely, Google Experiments, or other enterprise-class tools for web pages. For banner ads, the platform used for distribution can usually be used for testing. Facebook is more challenging due to the platform’s tendency to favor the winning ad, but split tests can still be conducted. For email marketing, the email tool being used usually offers A/B testing. While more expensive options like Adobe also have tools for experiments, there are plenty of user-friendly and affordable tools available for marketers on any budget.

Measuring and Segmentation

(11:46 – 14:32) Tying Metrics to Revenue or Qualified Leads: The effectiveness of an experiment is determined by the goal, which is typically to make more money or spend less money. The ideal situation is tying the experiment to direct revenue, but that’s not always possible. Another way is to measure qualified leads that actively work and have a timeframe to convert. Sales teams have insights into what those triggers are in their industry. In eCommerce, revenue is a good metric to measure. However, if there’s a complex distribution channel, it gets more complicated, and measuring signals that come in, such as clicks out to a distributor’s website or the quality of traffic on the site, may be necessary.

(14:32 – 16:13) Segmenting by Persona in Advertising Experiments: When conducting experiments on creative and content, it is crucial to segment them according to distinct personas. Each persona has different wants, needs, and motivations and responds differently to ads. By setting up experiments that take this into account, better insights can be gained, and this influences future programs. It’s important to mechanically split up the data based on the persona that was targeted when analyzing results. Signals need to be sent into analytics software using a UTM parameter or something similar to differentiate between personas and gain insights.

Charity Outreach

(16:13 – 17:23) Charity Break: Blue Dog Rescue

Channel Experimentation

(17:23 -20:15) How to Target the Right Persona for Effective Marketing Experiments: In addition to experimenting with content, optimizing channels is also important. Channels should be targeted to a specific persona for maximum effect, and experiments should be limited to one persona per test. By doing so, insights can be gained into which channels work best for each persona, and this information can be used to update personas and improve traditional advertising. Direct response is the focus of experimentation for channels, and it’s essential to measure results to determine the most effective channels for a particular persona.

(20:15 – 21:18) Maximizing Marketing Experiments: The different tests that can be run include channel versus channel and targeting within one channel. Channel versus channel can determine which channel is more effective at achieving results. Targeting within one channel can be done by testing different types of targeting within the same channel, such as interests and behaviors on Facebook, to determine which works better for a given persona. Results can vary by persona.

(21:18 – 22:23) Understanding Cost Per Conversion: The KPIs for measuring channels are different from those for measuring creative. For channels, the main metric is the cost per conversion, which measures the cost for each desired action taken by the customer. For example, in real estate, it could be the cost per quality showing. For eCommerce, it could be the cost per dollar in revenue. Efficiency is the key focus, and it is measured by how much it costs to bring in each person.

The Important of Statistical Significance

(22:23 – 25:57) Understanding Statistical Significance in Marketing Experiments: The key point in this conversation is about analyzing the results of experiments. Statistical significance is the one key number that marketers need to understand. It is the probability that if the experiment were run over and over again, the same result would be achieved. A statistical significance calculator can be used to measure this number. A 95% or higher level of statistical significance is what marketers should aim for to have confidence in their results. However, if the level is between 80-95%, it can be a judgment call for the business on whether to accept the result or try again with a larger sample size.

Join Us Next Time

(25:57 – 27:17Conclusion: Next week, we’ll be discussing reporting and feedback and taking all this wonderful data and these insights that we’ve produced and getting them in the hands of the right person in a way that they understand. Until then, have a great week, and we’ll see you next time. This concludes this week’s episode. For notes and links to resources discussed on the show, sign up to get them emailed to you each week at

Iterative Marketing is a part of the Brilliant Metrics organization. If you would like more information on the marketing services provided by the expert team at Brilliant Metrics, reach out today for a free discovery call.

The Iterative Marketing Podcast,  a production of Brilliant Metrics, ran from February 2016 to September 2017. Music by SeaStock Audio.

Learn more about Iterative Marketing and listen to other episodes on Apple Podcasts, YouTube, Stitcher, SoundCloud and Google Play.

Get The Most From Us

Don’t miss a post! Sharing knowledge is part of what makes us special, and we take it seriously. Sign up below to continue to grow and walk up the marketing maturity curve!

Try Us On For Size

We know you’re not about to add or switch your agency on a whim. That’s why we offer a series of workshops to let you give us a spin and see what it’s like to work with us, while getting some serious value along the way.