Marketing experiments in the context of Iterative Marketing serve a bigger function than simply increasing conversion rates. They help us gain insight and knowledge into our target audience. This podcast explores the role of experiments in Iterative Marketing and shares how marketers can get the most out of their own experiments.
Episode Show Notes
Introduction to the Podcast: The Role of Experiments
(0:00 – 2:20) Introduction to the Iterative Marketing Podcast: Welcome to the Iterative Marketing Podcast, where, each week, hosts Steve Robinson and Elizabeth Earin provide marketers and entrepreneurs with actionable ideas, techniques, and examples to improve marketing results.
The topic of this episode is the concept of experiments, particularly within the context of Iterative Marketing. It’s important to recognize that the idea of experiments in this framework may differ from the conventional marketing approach to experiments, A/B tests, or split tests.
The resources discussed on the show can be found at brilliantmetrics.com, which includes a blog and a LinkedIn group for community interaction.
Introduction To Experiments
(2:20 – 5:57) Understanding Experimental Marketing: Experiments in marketing involve comparing an existing method (version A or control) against a new method (version B or variable) in an A/B test or split test. Many marketers use A/B testing to measure the success of their campaigns.
More Than A/B Testing: Iterative marketing introduces additional parameters to an experiment beyond a simple A/B test. In iterative marketing, experiments must be scientific, using controls, variables, and statistical analysis to validate results. Experiments should generate insights beyond just increases in conversion rates or revenue. A hypothesis should guide these tests.
Scientific Approach: While the scientific aspect of marketing experiments may seem intimidating, it’s crucial for generating reliable results. Applying statistical principles ensures that the results accurately reflect the intended outcomes. This point is elaborated in Episode 22: Let’s Talk Statistics, which demystifies the statistics involved in marketing experiments.
Gaining Valuable Insights: Iterative marketing emphasizes the importance of generating insights through experiments, ensuring that findings are true and reliable. It goes beyond surface-level changes like button color, testing specific hypotheses that have broader implications for the entire organization. This topic is further discussed in Episode 7: Designing an Effective Marketing Experiment, which delves into the art of crafting impactful experiments.
Ensuring Maximum Value: The value of marketing across an organization can be reinforced through these insights. The experiments conducted should deliver maximum value to the organization, an important aspect of effective experimental marketing.
Starting Small
(5:57 – 6:29) Two Ways Experiments Compliment Iterative Marketing:
- Testing Tactics on a Small Scale: The first step of an experiment is to test a tactic on a small scale before implementing it on a larger scale. This approach aligns with the fundamental principle of starting small in iterative marketing.
- Facilitating Continuous Improvement: Experiments help to facilitate continuous improvement in running marketing programs. Regular testing and analysis can help identify areas for enhancement and drive incremental progress.
(6:29 – 10:52) The Concept of Starting Small: Before launching a new program, strategy, or tactic, it’s beneficial to start small. This approach allows marketers to execute a controlled experiment comparing a new idea against the existing approach without allocating substantial resources.
For example, consider a company selling poker sets. Currently, they offer a 15% discount via a coupon code. However, they want to test if providing a free deck of premium cards with each order, thus adding value, would be more effective. A simple experiment could be set up, targeting a limited audience on Facebook, to compare the response and conversion rates.
This experiment isn’t about immediately launching the new strategy across all platforms; rather, it’s about starting with a small audience and budget. The insights gained from this experiment can be applied across the program going forward.
Benefits of Starting Small: Starting with a small audience and budget allows the collection of valuable insights that can be applied across the program in the future. These insights can also be applied to other various aspects of the business, such as display advertising, print advertising, and customer service strategies. For example, a technology company could use this approach to determine the most viable market for expansion by setting up minimal-spend Facebook programs for each potential target audience.
Importance of Statistical Significance: The experiment must be statistically significant; confidence calculators and sample size calculators can help ensure the test isn’t too small. If the experiment yields better results than the current marketing activities, it may be beneficial to implement the new strategy on a larger scale.
Exploring Multiple Opportunities: If the audience size is large enough, starting small allows for the testing of multiple ideas simultaneously, potentially speeding up the decision-making process.
Facilitating Continuous Improvement
(10:52 – 11:42) Concept of Continuous Improvement: Continuous improvement lies at the heart of iterative marketing. It involves consistently optimizing existing programs to achieve enhanced performance. Experimentation plays a crucial role in this process. Consider a successful program running on Facebook. To improve its performance, marketers might examine audience targeting mechanisms. By splitting off a separate segment and testing different targeting approaches (e.g., behavioral vs. interest-based targeting), they can identify which method yields better results.
The key is to keep all other variables the same—using the same creative content and targeting the same audience with the same objective. This method allows for a clear comparison of the different targeting mechanisms.
The Goal of Continuous Improvement: Always seek the potential for improvement, even when a program is performing well. The setup of the experiment can enable concurrent testing, providing continuous insights for further application and enhancement of the marketing strategy.
Why Do We Run These Experiments?
(11:42 – 19:03) Importance of Experiments: Experimentation is central to iterative marketing; without it, you risk making uninformed changes and losing the opportunity for improvement. Constant changes in market dynamics, consumer preferences, and competition necessitate continuous testing and experimentation. Without experimentation, you run the risk of your strategies becoming outdated, leading to decreased effectiveness over time.
Loss and Opportunity Cost: Not conducting experiments leads to missed opportunities for optimization, resulting in wasted resources and potential gains. For example, if you could produce leads at $80 per lead but are currently doing so at $100, that’s a $20 per lead loss due to lack of experimentation. When launching new initiatives, you risk more if you don’t test first. Effective experiments can minimize this risk and maximize your returns.
Taking Emotion Out of Decisions: Experiments provide data-driven evidence, taking personal opinions and emotions out of the decision-making processes. For example, if someone claims that a certain creative won’t work due to a specific color, you can test that assumption and provide objective evidence to support or refute the claim.
Embracing Failure: Experimentation makes failure acceptable, as the goal is to gain insights and knowledge, not just immediate success. By setting the expectation that you’re running an experiment and the outcome is uncertain, you shift the measure of success from immediate results to learning and improving.
Empowerment for Marketers: Experiments can empower marketers by taking the pressure off predicting the success of a creative idea. It allows marketers to propose ideas based on data and reasoning, test them, and adjust strategies based on the results. This approach reinforces the value of marketers in an organization, as it demonstrates a thoughtful and data-driven approach to decision-making.
Charity Outreach
(19:03 – 19:57) Charity Break: We encourage listeners to donate to their local schools.
How Many Experiments To Run?
(19:57 – 24:34) Executing Experiments: The number of experiments that can be run concurrently depends on resources, audience size, and the number of opportunities to experiment:
- Resources: Setting up, administering, and measuring experiments requires resources, both internal and external. You may need to create multiple versions of creatives or digital experiences, which incurs additional costs. Sometimes, extra funding might be required to reach a sufficient audience size. A limited budget can constrain the number of experiments an organization can run.
- Audience Size: The audience size limits the number of experiments that can be run due to the need for statistical significance. For statistically significant results, you need a large enough audience. Tools like a sample size calculator and a statistical significance calculator can help with determining this. Not every audience is big enough for an experiment. For instance, if a key persona for a manufacturer only includes about 100 people, that’s not large enough to run a statistically significant experiment.
- Experiment Slots: The number of available “experiment slots” also limits how many experiments can be run concurrently. A slot refers to an intersection of an audience and a creative or targeting mechanism. You can only run one experiment at a time per slot. For example, if you’re testing two creatives for a particular audience, that slot is filled, and you can’t test any other creatives for that audience until the experiment is complete. The same applies to targeting mechanisms. If you’re running a Twitter ad targeting experiment, you can’t test other targeting methods on Twitter until the current experiment is complete. The goal is to have experiments running wherever there is a large enough audience, maximizing insights over time.
Extracting Value from Experiments
(24:34 – 27:57) Run as Many Experiments as Possible: Maximize the number of experiments within the limits of your resources, audience size, and available programs. Each experiment builds on the knowledge and insights of the previous ones, allowing for a continuous learning and improvement process.
Keep a Ledger of Insights: Record the results of each experiment, especially in the context of your audience segments or personas. This can help identify knowledge gaps, inspire new experiments, and generate ideas for future content, creative designs, or programs.
Apply the Insights: Use the insights from your experiments to improve your marketing strategy. This isn’t only about discontinuing non-performing strategies but also about identifying where these insights can be applied. Insights can contribute to updating personas, ideating future experiments, and expanding programs.
Report what you Learn: Share your findings not only with your superiors but also laterally across the organization. Each bit of knowledge about the consumer of your product or service can be beneficial to other departments in ways you might not have thought of. Sharing these insights can increase the value of marketing within the organization, as it provides useful data that can impact operations across the board.
Join Us Next Time
(27:57 – 29:10) Conclusion: In this episode, we discussed the role of experiments in the iterative marketing ideology. Join us next week as we deep dive into the Think state of the buyer’s journey.
Have a great week and we’ll see you next time. This concludes this week’s episode. For notes and links to resources discussed on the show, sign up to the Brilliant Metrics newsletter.
Iterative Marketing is a part of the Brilliant Metrics organization. If you would like more information on the marketing services provided by the expert team at Brilliant Metrics, reach out today for a free discovery call.
The Iterative Marketing Podcast, a production of Brilliant Metrics, ran from February 2016 to September 2017. Music by SeaStock Audio.
Learn more about Iterative Marketing and listen to other episodes on Apple Podcasts, YouTube, Stitcher, and SoundCloud.