The problem with A/B testing
Almost everyone has heard of A/B testing. Even if they don’t fully understand conversion rate optimisation (CRO), they probably have heard about the basic principle of testing version A of a webpage or website versus version B. It’s pretty straightforward and easy to understand, even by someone outside of the technology bubble.
Because of this, most clients come to us wanting A/B testing. In some cases, we must explain that they simply don’t have the traffic volumes for traditional A/B testing, which can be around 10,000 sessions per variation (although it is case-by-case dependent). This can come as a shock as it is seen as the standard for CRO.
A CRO stopgap
So, what can we do for people with very low traffic sites? Traditionally we have offered one of our CRO Audits. These highlight issues with the user experience that are likely harming conversions and offer potential solutions which (in an ideal world) should be tested before being rolled out.
We also offer expert led moderated user research, but as a lot of these companies are just starting out, sometimes this simply isn’t budget friendly enough for them.
In reality, most of these kinds of clients take the findings away from their audit and choose whether or not to implement the changes themselves without testing, which is perfectly understandable, but not ideal.
This always felt like we weren’t delivering a full CRO service for these clients. Although we could help them implement the changes if they wanted us to, for those with very low traffic volumes it just didn’t seem right to set them up on a course of A/B testing after implementation.
Technology to the rescue?
We’ve recently started using an AI based A/B testing tool that calibrates synthetic shoppers based on a site’s analytics data and other information known about the client’s user demographics. These AI users navigate through the site and attempt to complete specific tasks, much like users in user research study or indeed, real users on a site.
This technology is in my opinion nothing short of revolutionary for low traffic sites. Whilst it could have taken months or even years to gather enough data for a conclusive A/B test on one of these sites, it can now be done in around an hour for much lower overall cost.
Introducing Squoosh
Squoosh is the company behind this groundbreaking technology. Founded by former Google employees, this software is cutting edge in the world of CRO.
Squoosh tests run much like ordinary A/B tests. You give the software the URL of your current site, and the URL of your variant, and let the synthetic shoppers do their thing. Once they have finished navigating through your site you are given a result. This may be statistically significant as in our test on adding social proof to collection pages on an e-commerce site shown below.
Or the results may be inconclusive when the change is small. Such as when we tested adding tooltips to product pages.
The downsides of Squoosh
Okay I hear you, this article has been very one sided so far, and you want to hear the not-so-great things about AI A/B testing too.
Well, let’s address the elephant in the room. These are synthetic shoppers, not real users. Squoosh doesn’t attempt to hide this fact. It clearly states that the software accurately determines a winner with 81% accuracy on their blog. It should be noted that this is up from 71% not too long ago, showing how the guys at Squoosh are constantly fine tuning the software to further increase its accuracy.
Another limitation those who are used to live A/B testing may find is that synthetic shoppers can’t actually buy anything on your store, so you don’t get a true conversion rate for e-commerce. However, the team is currently working on a solution for the AI users to enter a credit card and complete the full checkout flow.
The positives of Squoosh
1. It is very easy to use.
Unlike some A/B testing tools, Squoosh is incredibly intuitive. Running a test is as easy as copying and pasting in some URLs (or if you are on Shopify, selecting the themes you want to test). The results are easy to interpret too, clearly telling you which variation won.
2. It only takes an hour on average to run a test.
Imagine if your A/B tests could be run in an hour. No more waiting anxiously for a month to see if you can get a statistically significant result. Not only is this great for smaller sites, but we are also seeing its application to larger sites who want to pre-validate their hypotheses before pushing it out to a live A/B test.
3. You can see why users decided to or did not decide to add to cart/make an enquiry.
Unlike with live A/B tests, you get a short summary of why each user ultimately decided to add to cart/make an enquiry, or to leave without doing so. This is information you would typically struggle to obtain without running user testing.
Plus, a new summary feature just became available. So, as well as reading about individual user experiences, you can now see why a variant won or lost overall according to the synthetic shopper experience.
Not only is this helpful in determining why a test won or lost, but it also highlights other issues with your user journey that you may not have been aware of.
4. The team behind it is dedicated to making it the very best it can be.
Being Squoosh partners means we have a direct line to the key decision makers. They are extremely responsive to feedback and are dedicated to improving Squoosh every single day.
Their most recent upgrades based on client feedback have been to add ‘Product Views’ and progress to checkout as metrics so you can measure not only add to carts on an e-commerce site, but also how many of the AI users saw PDPs and how many progressed to checkout. Other metrics have been requested and may be active in the near future.
As well as other metrics, something we’ve requested (testing on mobile) is slated to become available next week. This is really exciting as most of our client’s users are mobile device heavy.
5. It drives better value for our clients.
Perhaps one of the most important positives for us is the ability to add value to our services. With the ability to run A/B tests faster, we can test more each month. This means we are delivering more value to our clients month in month out.
The inevitable backlash
As with any new technology there are bound to be some who would rather see the drawbacks rather than focus on the opportunities it brings. For those who will inevitably cry “That’s not real A/B testing!” I say “Yes, I agree”, and that’s why it’s so exciting. Conversion rate optimisation A/B testing has for too long been only accessible to larger traffic sites.
To those who argue that AI A/B testing isn’t accurate enough or in-depth enough I would ask them what they are doing to help those who are just starting out. We’ve always tried to help these kinds of people and now we can do more than audit their sites and run user testing.
Squoosh is also becoming more powerful week by week. We’ve already seen huge strides being taken to improve the software. Who knows what the future will bring.
Wrapping things up
So, there you have it, my early evaluation of AI A/B testing software. I truly believe it will revolutionise the world of CRO. I also believe it will continue to improve and develop over time, going from strength to strength. At the moment, I’m only aware of Squoosh offering this exciting new technology and my experience with the founders has been overwhelmingly positive.
If you’d like to experience Squoosh for yourself, we are happy to chat to you about your conversion rate optimisation needs. We can even get you a cheeky 20% off! Just reach out to us using the button below. We can’t wait to explore this fantastic new software with you.