Episode 7 – Control Groups and Testing

Rokture Ramblings – Episode 7 – Control Groups and Testing

Hello everyone, and welcome to Rokture Ramblings, where I discuss topics around digital 

marketing, marketing technology, and marketing operations for the financial services industry. 

My name is Fernando Pena, and I’m the founder of Rokture, where I can help you master the digital 

channel. And today’s topic doesn’t really have to do with digital, it has to do more so with marketing 

fundamentals, but I do believe it’s something that I’ve received a lot of pushback on and required 

convincing, especially to executives, that this is something that should be done when you 

are embarking upon your digital journey. And that topic, of course, are control groups and 

testing. The question that I get is, is it really necessary? And so what I’d like to do is just 

take a few minutes to discuss why I feel it’s important, and how you may be able to navigate a lot 

of internal resistance and challenges that you’ll encounter in attempting to use these tactics 

in your digital marketing journey. So we’ll  

00:00:55 Speaker 2

undergo a basic review, can you do without, resolving conflicts, and then also what the final 

benefits end up being.  

00:01:04 Speaker 2

So for basic review, what I’m talking about here when I discuss control groups are random samples 

of target audiences that are intended to be used as baselines. And so what that means is that 

if you have a group of like, let’s say 100 potential individuals that you’d like to market an 

offer to, you may want to take 10 % of that audience of 10 individuals, and just put them aside 

and do not employ any sort of marketing treatments on them. And there’s several reasons for 

that. And I’ll get into a little bit more of it as we go further into this presentation. But it 

is a sound strategy when it comes to making sure that you can track what is taking place, and if 

your actions are actually positive or not. So testing, on the other hand, it includes different 

permutations and variations of messaging, calls to action, imagery, timing, and many others. 

And I’ve discussed this in some detail in the personalization videos, as well as the video about 

AI, where there’s just a number of different combinations  

00:02:05 Speaker 2

that can be made use of. And the only way to really tell which one’s going to work is by testing. 

Now the problem with all this is that there is likely going to be concerns about limiting the 

responding audience. And so that is if you have a finite audience, which most of you do, you may 

not want to necessarily limit some of your best customers by putting them into a control group. 

And that is a valid concern that we can address here. There’s also the effort required in order 

to create these different permutations. So that’s probably some resistance that you’ll receive 

as well, as it takes long enough just to get one piece of creative approved from legal, risk compliance, 

etc. And now if you’re going to ask to have these large number of variations approved, it’s just 

going to set back the timeline a little bit. So that is again, a valid concern. And then when you’re 

talking about salespeople who are trying to, and their livelihood depends upon sales, there 

could be concerns around declining  

00:03:04 Speaker 2

revenue and also impact to their compensation. So those are our concerns that will have to be 

addressed and we’ll discuss here in today’s presentation.  

00:03:16 Speaker 2

So can you do without these control groups and testing? And the answer here is that yes, certainly 

you can. But in today’s highly competitive space, if you’re trying to maximize your investment 

in digital and you’re trying to make sure that the performance of that channel is as best as it 

can be, then you really do need to create a control group and also do a number of different testing 

scenarios. If you do without this, you’ll potentially lose the opportunity to improve results 

and performance. And the reason for that is because you really don’t know where you started 

from. I mean, you can see where you started from, from an overall perspective, but you won’t 

understand if that is because of what you did or if it’s because of market conditions or just 

the competitive environment. You also have the inability to figure out exactly which of your 

tactics or strategies really did have an impact on your customer base. And this is because again, 

you have not established a baseline and because  

00:04:17 Speaker 2

of that, you’re unable to really do any sort of real tracking and analytics. So as a result of 

that, you’re going to have flawed or skewed models and analysis. So your metrics really are 

going to be just sort of a point in time, but not necessarily in comparison to what the rest of 

the audience may have done without the actions that you took. So it’s really important given 

the fact that in order to be able to really succeed in digital, you need to track and report and 

analyze virtually everything you’re doing. And the only way to do that is to again, have a baseline 

to be able to establish that this is where we need to go. And when it comes to testing and variations, 

again, that’s critical because everything evolves very quickly in digital. So by making sure 

that you are ahead of the curve and you’re testing what resonates with your audience, you’ll 

be able to make sure that you’re getting the most response for the different tactics that you’re 

employing. Now, inevitably, as I mentioned  

00:05:16 Speaker 2

earlier, there are going to be conflicts in this because of the fact that you have, again, a finite 

audience and there’s likely going to be some individuals that fall into the control group that 

are considered some of your more desirable customers. And with that, your sales teams and those 

that are again, judged on performance figures are going to resist wanting to place these individuals 

into a no treatment category. So what needs to happen is that some sort of a playbook has to be 

established with clear rules of engagement. So this might mean that there’s instances where 

if a marketing blitz is going to take place, the sales people need to observe that some of their 

key customers are going to end up in a control group. Now, it’s not to say that if like, let’s say 

you have personalization triggers or event based triggers that are taking place with AI or 

just using conditional logic take place and an action needs to be performed to keep a great customer 

from leaving the organization that  

00:06:15 Speaker 2

you don’t do it. But more often than not, try to avoid any sort of additional treatment so that 

your metrics are not going to be impacted by those actions. But by all means, the priority here 

is to make sure that you do maintain your existing customer base and that you keep them happy. 

But again, let’s try to make sure that we minimize any sort of involvement if they happen to be 

part of a control group. You’ll need to also likely adjust compensation models and performance 

metrics. So if the performance metric says you have to reach out to 100 % of your book every quarter 

and you’re doing, let’s say quarterly outreaches and this individual ends up in a control group, 

then you may have to adjust expectations just to make sure that it is fair and that you’re not 

asking your teams to try to meet some sort of expectations that cannot be met based on the fact 

that some of these individuals really shouldn’t be reached upon. You’ll need to emphasize 

data capture as a necessary part of marketing  

00:07:13 Speaker 2

and sales. And what I mean by that is that when you’re doing testing and control groups, you’re 

trying to capture data and you’re using this data to help refine your approach on a go forward 

basis. So that has to be sort of the mindset and emphasize throughout the entire organization 

that this is something that’s critical to not only marketing, but to the success of the entire 

structure of the organization and make sure that it’s really a, everyone buys into this because 

ultimately you’ll be able to improve when you have this data at your disposal and show and demonstrate 

that you have had successes based on the data that you’ve captured. But probably the most important 

is to practice transparency of the marketing calendar, the marketing strategy, and what’s 

really taking place so that the sales team doesn’t necessarily feel alienated and that you’re 

trying to hide this segment of the customer base to them. And instead you’re showing them that 

this is what we’re trying to accomplish. We’re  

00:08:08 Speaker 2

trying to get you more data. We’re trying to get you more targeted leads and really get you in 

a place that is much more beneficial and conducive to a positive selling environment because 

it works great for the clients and it also works when prospects and it works well for sales as 

well. There’s less resistance to products and services being sold.  

00:08:30 Speaker 2

Now when you get all this going, then what are the benefits? And so there’s going to be unquestionable 

impact results. And what I mean by that is not necessarily that everything’s going to magically 

work on your behalf, but there will be data demonstrating and showing and illustrating what 

has taken place so you can show that, okay, we improved performance based on our observations 

of what this control group did compared to what our targeted audience did. And if you don’t have 

successes, then again, you can see that, oops, we made a mistake. We did something that unfortunately 

made our target audience not respond as favorably as the control group. It also helps you continually 

refine your targeted message, your offers, your creative, everything that has to do with marketing. 

And you’ll get lessons learned on what are the successful tactics, what works for you. Keep 

in mind that, especially in financial services, marketing is a very personal type of process 

where your message has to resonate  

00:09:32 Speaker 2

with those buyers and by testing and learning over the different tactics that you’ve employed, 

you’ll be able to figure out exactly what resonates with your buyer and make sure that your offers 

and your messaging reflects that. It also helps tremendously when it comes to the development 

of predictive models and reporting that can be trusted and relied upon. If you have questionable 

sources for analytics, then of course your judgment and maybe even your reputation is potentially 

going to be damaged by the fact that you don’t have reliable data to base your observations on. 

And now, especially with AI coming up, you need great data to make sure that the foundation of 

these systems is well populated so that it can make decisions on your behalf using the best data 

available. Also one side benefit here is the potential avoidance of audience fatigue. So let’s 

say that you’re running offers on a continual basis and you’re always hitting the same individuals. 

Eventually they just become blind to  

00:10:36 Speaker 2

whatever it is that you’re communicating to them. So by making use of control groups and tests, 

you’re going to number one, create different variations of that message that they’ll see. 

Number two, they may not necessarily get every single offer because they may fall into the control 

group occasionally. So that just helps to make sure that everything remains fresh and relevant 

and that they’re not being just hit over the head with your offers on a continual basis. That 

may work in some cases, but I do think that, again, with financial services, you’re looking 

at making sure that it’s the right messaging and not just a message because there’s way too much 

noise out there and you need to make sure that your message is spot on. So that was it for today. 

Again, this was more of a fundamental approach to digital marketing in terms of foundational 

skills and tactics that need to be employed before you embark upon this journey for digital. 

But if you do like some of this content and you’d like  

00:11:36 Speaker 2

to discuss it a little bit more in detail, please feel free to reach me. You can see my contact 

information on this slide, and we will see exactly how we can work within your environment and 

try to maximize your use of digital. So until the next time, thank you so much for listening, 

and we’ll talk to you in the next video. Bye bye.