What happens when conversion rate increases due to a ‘successful’ A/B test but the quality of those conversions tanks? Or the opposite, a test that seemed to have failed on the face of it, actually increases conversion quality? Do we accept quantity over quality, or quality over quantity? How do we spot the right trends before we make a big mistake?
When quality may suffer for quantity
Most SaaS sites will claim that their middle of the range offering is the most popular, but what would happen if actually their cheapest offering was the most popular? Would seeing reassurance that the cheapest offering is the right choice persuade more users to buy – maybe. But would it also reduce the percentage of users buying the higher priced offering – quite probably.
What about discounts? If you apply a discount to a product on an e-commerce store, chances are orders of that product will increase, boosting your conversion rate. But offering this discount could be hurting your profit margins, making them less quality conversions.
When quantity may suffer for quality
So, when does the reverse happens? This can happen if you add a qualifying question to an agency contact form. Users are more likely to be able to identify themselves as being a potential customer or not being a potential customer. As a result, the percentage of high-quality leads may increase, even if the total number of leads decreases.
For example, asking users how many employees they have and including a ‘Fewer than XX’ could suggest that really, the service is only suitable for companies with more employees than XX.
It could also happen if you own an e-commerce store and decide to switch focus from selling at cheap rates to pushing the quality and value of your products. Because your products are now more expensive, you could see your conversion rate drop, but your average order value may increase.
Strategies to boost quantity vs quality
Quantity
- Sales
- Promoting a lower value most popular SaaS offering
- Offer the first month of a SaaS product free
- Removing fields from contact forms
- Using urgency
Quality
- Re-writing copy to appear premium
- Increasing the cost of one-time purchases compared to a subscription
- Adding a qualifying field to a contact form
- Adding higher-resolution imagery and/or video
- Encourage users to have a demo of a higher-value SaaS product
Which is better quantity or quality conversions?
Well, it depends. The first thing that comes to mind is are you focusing on making a profit, or on increasing your brand awareness. Because being known for cheap offerings can get you that awareness if it’s what you are chasing. Several brands will deliberately operate at a loss in order to increase brand awareness at first.
It also depends how much each sale is really worth to you. If you are an e-commerce store selling mid to low priced items, then quantity may be the way to go. But if you are an agency where the right client may stick with you for years to come, then quality will likely outweigh quantity.
Of course, whether you prioritise quantity or quality may change over time. You may start off valuing quantity and then switch to quality as you become more established. Or, you may be struggling with only a small number of customers and need to increase quantity to stay afloat.
It’s important to identify what your goals are at the point at which you run an A/B test because typically the metrics tend to be geared towards quantity, not quality.
Drawing conclusions from A/B tests when the results are mixed
It’s important not to get too hung up on any one metric when looking at A/B tests, especially if that metric is conversion rate and you’re testing something that could be several micro-conversions (steps in the flow) away from your end goal.
Not all metrics will be tracked in A/B tests. Lead quality for one is not something that can be attributed by A/B testing software (at least not as far as I’m aware). So, this will take some manual working out.
The long and short of it is to look at all the data you have available to you before jumping to conclusions. Try to see the big picture. Sometimes that failing test can actually be a winner, and that winning test could be a huge mistake.
Hi, I’m Charlotte. I love writing about user psychology and tips I’ve picked up from leading user research and CRO projects. Get in touch if you’d like to work with me to improve the user experience on your site or click here to learn more about your future CRO team.