Mike: How’s it going, guys.
Happy Wednesday! It’s Mike Norris. I’m here to talk to you today about A/B testing and Personalization. And for this conversation, I brought in Charles Brunn from Dynamic Yield, who is an expert in this field. So, Charles, I’ll let you introduce yourself a little bit here.
Charles: Thanks, Mike! Dialing in from New York. It’s also kind of a gray day this Wednesday. But, it’s good to be here! I’ve essentially been at Dynamic Yield for three and a half years. I head up partnerships for all the Americas. In my time here, I have definitely learned quite a few tricks about Personalization. I’ll be happy to share a little bit more about that with you.
Landing Page Best Practices
What’s the Difference Between A/B Testing and Personalization?
Mike: I’d imagine! Alright, let’s jump right into it. What’s the difference between A/B Testing and Personalization? I think people use the terms kind of interchangeably sometimes and there is a pretty stark difference.
Charles: That’s a good question! When you think about A/B testing, A/B testing is the concept of basically having a scientific methodology for experimenting — whether it’s on your site, mobile app, email, whatever it may be — for your different marketing campaigns. This started many years ago. There were a lot of great books (I’m butchering the name now). Can it Be Tested?… and if you think about what it was, it’s really about letting statistics determine what exchanged should happen, what banner should be changed, or what type of content should be changed.
Now, it’s obviously something that’s critical, but it’s also been commoditized over the years. When you think about Personalization, it’s kind of that next evolution of the A/B testing methodology, which is part of A/B testing. But it’s obviously not exactly the same thing. Basically, when we think about it, here at Dynamic Yield, we kind of blend the two together.
What it means is that when you’re thinking about launching a Personalization campaign, what essentially you’re going to be doing is create a hypothesis for a segment or a type of visitor — basically a unique set of customers or prospects that you want to target with a specific message based on the way they were behaving, based on what you know about them. So when you launch these campaigns, it’s essentially A/B testing that’s more fine-tuned and deep in terms of how you’re setting up that campaign.
Instead of saying, “Hey we’re A/B testing everybody…” We’re actually testing different segments and visitors differently with different types of messaging and letting the algorithms kind of take over and deliver a personalized experience.
So again, at Dynamic Yield, when we think of the two together. We think of Personalization as the next evolution of testing and obviously testing is critical to everything you do. Obviously, a lot of us come in with specific use cases or specific ideas based on best practices. But, what I’ve seen in three and a half years is no two use cases are going to generate the same results with different customers, depending on their size, the type of traffic they acquire, the vertical they’re in, and all things like that.
Marketing For Beginners 101: How to Learn More
Which Is More Important?
Mike: Got it! That makes a lot of sense. Between Personalization and A/B Testing, Is One More Important Than the Other One? Or is it kind of case-by-case basis?
Charles: I think it is a case-by-case basis, you’re right about that! First of all, it’ll depend on what team you have, what you’ve been doing. Typically, I think of the more generic testing as things that are more related to UX UI. If you’re redesigning a site because you’ve seen that the checkout process is super problematic and there’s a lot of issues. Maybe even in terms of performance, you have challenges. You could assume that new check out experience is going to be improved throughout all the different segments. So in a scenario like that, you could certainly personalize.
You could say, “hey we want to test it for mobile visitors, versus VIP visitors, returning visitors, and all these different segments.” But the truth is when you do these kinds of deeper UX redesigns, typically you apply it across the board. So, for a use case like that, maybe start with testing. That doesn’t mean that you can’t change some other things. You might change the specific layout of the banners. Instead of having the main hero banner, you’d have multiple hero banners or a slider — it doesn’t mean that after the fact, you can do some Personalization. But, I do think it depends.
And then when I said it depends on the team. The reality is the team you have determines how much you can do. Even though a lot of technology stacks, such as Dynamic Yield, have AI (and everybody talks about AI).
It is not some kind of Frankenstein machine that takes over your site, mobile app, or your emails, and literally does everything for you. There still needs to be quite a large amount of work that comes into strategizing around the goals, around the current challenges you see with your customers and your flows and thinking about how you want to deliver that experience — that personalized experience for all your different customers.
So, in many ways personalization — even though it’s the next evolution of A/B testing —, it also requires more resources when it comes to deploying these campaigns effectively. If you’re a small start-up, not doing much, the Personalization you might do is deliver personalized recommendation… then that’s super low touch. You don’t need a ton of resources. But, if you want to launch a full-blown Personalization program, you do need either a team internally to handle that or work with a partner such as Youtech to be able to help drive that personalization.
Mike: Yeah! I think that’s really well said. Oftentimes, clients and potential clients talk to us and they’ll say… “Well, why do we have to continue doing it? We’d like it to be a little more set it and forget it.” And our answer is: We would too. We’d love for it to be more set it and forget it, but that’s just not natural. You’ve got to keep an eye on it. You have to keep working through it.
Charles: That’s right, and you need to keep refining. There’s no such thing as a setting and forgetting when it comes to everything. Everything is in constant motion. From our standpoint, when we launched some of these campaigns, we see that, first of all, you might segment in two different segments. But then you see after running that experience for two to three weeks, you start seeing things happening within those segments.
You start sub segmenting and you create additional segments. Then you can keep on doing this and find amazing nuggets of uplift or amazing increases in average order value or revenue procession based on this additional, continuous work that you do. So we see basically Personalization campaigns are just getting more refined, better, more sophisticated and to your point, it’s constant work.
Mike: Sure, yeah.
So you just mentioned something that I think is pretty interesting and I think it’s a question a lot of people have.
Search Engine Optimization (SEO) The art of making the search engine’s job easier.
At What Point Do You Begin to See A/B Testing or Personalization Results?
Mike: Let’s say we’re running some A/B or Personalization campaigns. At what point would we begin to see results?
Charles: I hate to say this again, but it depends. First of all, when you think about the campaign you’re launching, it depends on the baseline. Let’s say you have this amazing website. You have this amazing partner like Youtech that did a really good job of fetching the right type of customers through the
acquisition campaigns. The site is beautifully designed, optimized, and everything.
If everything’s great, Personalization is going to take longer to show results. Or maybe moving that needle is going to be more difficult. But let’s assume, all things considered, you have a site that’s benchmarked average with the overall industry, you can see results pretty quickly in a matter of weeks. Obviously, it also depends on traffic. It depends on how you’re segmenting your traffic.
If for example, you launched a campaign on your cart page, you might have only five percent of the traffic that’s going to that cart page. Having said that, that five percent might be the place where you should be focusing right. But because you have lower traffic, it can take two weeks to actually get to a result which moves the needle. So if you’re a massive company with ten million monthly unique visitors, it’s not going to be the same as if you’re a startup with a hundred thousand monthly unique visitors.
Again, it doesn’t mean that you can’t see results in a matter of weeks, but it means that your options in terms of slicing and dicing all the traffic that you’re getting might be a little bit more limited because you don’t want to have segments that are so small that essentially you know you’re targeting — I don’t know a thousand people a month — and you’d need maybe three months to get that statistical significance.
So it does depend on simple things like average order value. If you’re selling things that are super, high-end — I don’t know, ten thousand dollar sofas. Maybe, launching a Personalization campaign increases conversion by ten percent and you’re going to see massive results in terms of your performance. But if you’re selling potato chips online, that’s going to take quite a bit longer to actually see the type of result that that moves the needle in terms of ROI and in terms of performance.
What Kind of Results Are Expected When Running A/B or Personalization Campaigns?
Mike: I can go for some potato chips right now. You touched on this a little bit and I’m sure the answer kind of depends, but what kind of results can people expect to see when they’re running campaigns like this?
Charles: So, I guess maybe I’ll take a step back. We think about the results that we deliver and when I say “we deliver,” it’s not really us, it’s people like you. It’s people that are actually using the tool. Our tool is just an enabler. But when we look at the results, it’ll vary based on the KPI. We’ll typically, at least in the world of eCommerce, you are going to want to optimize for maybe revenue like top-line revenue.
I mean it could be increasing the average order value. It could be increasing your revenue procession, increasing conversions. For anyone of these different campaigns, essentially, what you’re doing is selecting a KPI that you want to optimize and increase, and then Dynamic Yield will be focusing on that and using that as the measuring stick to increase performance. That’s the context that I wanted to share.
Mike: Sure.
Charles: In terms of what those numbers can look like. We have some customers that see a five percent lift in conversion. We have customers that see a five 10x increase in conversion, right or we have customers even in terms of dollar value that they’ve seen close to 40x return on investment based on what they’re spending. Based on what they’re spending, numbers can vary quite a bit. Look in the industry. If you can increase by 10 percent of these different KPIs, it’s a big number. Typically, we’re comfortable thinking of a 10 percent increase, whether it’s revenue per session, top on revenue, et., — that’s typically an easy kind of metric to accomplish.
And if you want to double your revenues using its solution like Dynamic Yield, you’re going to have to put some folks to work and you’re going to have to really define a strategy and a plan of action. Also, a plan of execution because I think what we see is again the clients that are the most successful are not just the clients that you know have these big hairy goals, but the clients that are willing to invest with their own teams or using an agency like Youtech, in order to deliver those because at the end of the day.
You have typically the folks that use Dynamic Yield. There’s going to be somebody that’s like a front-end developer. There’s going to be a UX designer. There’s going to be typically ahead of optimization or testing manager. There’s also going to be somebody who’s in the world of analytics. And all these different roles are critical, because analytics will be reviewing the reports, understanding what works, and then the testing manager understands the different tests and how they run, and the statistical significance that are applied.
Then the first one on the UX side will be the one submitting the different templates or submitting the different visual layouts or changes you’re making to the website or the app or whatever the case. It really does depend on who is going to be doing this and how much a client is willing to
invest to hit those goals.
McDonald’s Acquires Dynamic Yield
Mike: Well, thank you, Charles. This was exceptionally enlightening. Any final words?
Charles: I guess the only thing I want to reiterate is the fact — I hope I’m not like scaring the folks that are watching this video — personalization is critical. I’ll step back. Four or five months ago, we got acquired by McDonald’s and you could think like why the hell is a tech company like Dynamic Yields is getting acquired by McDonald’s? Especially McDonald’s out of all folks, right? I think that speaks in fact that really McDonald’s is really smart about being very customer-centric and really wanting to deliver an experience that’s amazing with every bite with every order, etc, and sometimes they fail, but most of the time they’re really good at what they do.
I think Personalization is core to customer centricity and truly being able to craft an experience that’s meaningful, that you love, that makes you want to come back to experience again. Personalization is critical, and again to be successful at that is important to get the right people on the bus in order to deliver those experiences. So working with folks like you guys are critical to be able to deliver a customer experience that people have come to expect these days.
Mike: Sure, yeah. You bring up a good point in that there’s so much you can do that having a solid team in there to really help implement that and guide the process over time is instrumental. Congrats on the acquisition with McDonald’s this big news.
Charles: Thank you. Thank you! Yeah, it’s been big news and it’s been great because they’ve been very much looking to us for what to do with technology, but leaving us alone at the same time. They’re looking to us as a kind of thought leader in how you can leverage technology in a smart way to deliver improved customer experiences. So, it’s been amazing.
Mike: So can we expect more personalized experiences in McDonald’s coming soon?
Charles: Definitely yes there’s going to be more and more. And again, it’s crazy to get acquired by a company with the reach of McDonald’s, the impact you have is amazing. The daily lives of so many millions of people, so you can definitely expect much more personalized experiences in the near future, whether that’s using apps, whether that’s your store, or whether that’s online — you name it.
Mike: Excellent well thank you so much for your time today, Charles! I really appreciate it. This was very beneficial. I’m sure our viewers found it the same, so thank you!
Comments are closed.