Knowing what works and what doesn’t can make or break a business, so at By Association Only we encourage merchants to put the time, effort, and where necessary, money, into testing theories and pushing boundaries.
A/B testing tools are a great way to accurately and effectively do this, and, with Google Optimize being put to pasture in September 2023, we’ve had an increasing number of clients ask for recommendations on which paid testing platforms are best.
So let’s look at the advantages and disadvantages of A/B testing, what can be tested, and which tools are most useful.
We asked By Association Only’s Head of Merchant Strategy Natalia Howard and Ecommerce Account Strategist Rory O'Hara what the advantages of A/B testing are. Here’s what they had to say…
We also asked what the disadvantages of A/B testing can be. Let’s take a look…
If you do decide that A/B testing is something that could benefit your ecommerce business, there are a wide range of aspects you can test including the following which are popular with our merchants:
A/B testing tools usually have two ways to run tests – a visual editor and a code editor.
Visual editors are often used to make small changes, such as changing the style or colour of a call to action (CTA).
Slightly more complex tasks require a code editor, which allows developers to directly implement new code.
There is also a third option for even more complicated tasks, which is to make changes to a theme – the test version that a user will be served can be determined using a CSS class or data attribute.
This is then deployed and the testing platform is used to assign users to either the control or test variations when they land on a page included in that test.
Here are Natalia and Rory’s top tips for successful A/B testing:
There are various factors to consider when choosing which A/B testing platform is best for your business – including cost, functionality and ease of use.
These points and more are considered below in our breakdown of some of the most prominent A/B testing platforms in the ecosystem.
All can be integrated with Shopify and have been tested by various By Association Only clients with the guidance of our Strategy team.
AB Tasty was founded by Alix de Sagazan and Rémi Rémi Aubert in 2009. The company believes, “experimentation and personalisation should be fast – and simple.”
“Success isn’t spontaneous, it’s built on continuous optimisation: the process of trying, failing, refining, trying again and flying,” says the company.
AB Tasty works with over 1,000 brands including Sephora, L’Oreal and Eurosport.
AB Tasty offers A/B testing, as the name suggests, split testing, multivariate testing and funnel testing.
Its visual editor can be used to put together variants and tests including those based on URL, geolocation, demographic and more.
Users can also enrich the platform’s data by integrating existing data from CRMs and DMPS.
From Rory’s experience using AB Tasty the tool is well suited for teams looking to make an impact without the need for a high level of technical support.
Setting up the tests is pretty straightforward, partly thanks to a seven-step guide taking users from ideas to publication, as well as the visual editor being really easy to use.
The interface is simple to understand with a useful preview mode, but, if you do face any problems the AB Tasty team is on hand to resolve any issues.
AB Tasty is reasonably priced and makes a great starting point for companies just beginning to experiment with conversion optimisation.
While the post-test reporting features are good and certainly sufficient for informing decision-making, the data gathered is not as in-depth as in other platforms.
The statistical significance calculator is fairly basic, for example, and there can be a delay in receiving test results.
Californian company Convert was founded in 2008. It claims to have optimised over 40,000 sites since then – including for Sony, Audi and Virgin Mobile.
Convert boasts over 40 filters for accurate targeting as well as a powerful API that allows you to run a headless version of Convert in your ecosystem.
Most plans come with a 10-step onboarding that includes proper snippet installation, setting up your first project, drafting and monitoring an A/A test, goals configuration and enabling integrations.
Convert’s drag-and-drop system is super easy to use, as is its visual editor which you can use to create A/B multivariate, multipage, and split tests.
Utilise over 35 different elements to build customer profiles.
Convert claims to be “the most affordable full-stack tool in the world” and its plans start at $499 per month. However it’s worth bearing in mind that most A/B testing tools’ plans are traffic-based, so this monthly fee would apply to a store with very low traffic.
Convert says its support is 10 times faster than other platforms and “gets you in touch with folks who know the app like the back of their hands.”
They’re not wrong – from our experience, the live chat support feature is very responsive and can help work through issues with the visual editor or configuration for tests.
… and user data in the listing page – very helpful for keeping track of expansive testing programs at a glance.
For example, confidence intervals to help users feel as confident as possible in their data before making decisions.
Convert has plenty of targeting options to display tests to specific user groups based on country, browsing behaviour, traffic sources and more.
The company says it has structured its app, policies, and operational procedures “to offer enterprise-grade security and total alignment with the privacy and compliance shifts happening in the market.”
As you can see Convert has a lot going for it, but it has one fairly major downside – it’s not easy to use.
For most changes in the visual editor, and for setting up custom tracking metrics, it’s helpful for users to have at least a basic understanding of HTML structure – you also need to be comfortable reading HTML while inspecting your website in the browser.
Even for experts such as ourselves the visual editor can be somewhat cumbersome – as can the revenue tracking setup, setting up goals and reporting metrics.
.png)
Launched by Paras Chopra in 2o09, VWO works with over 2,500 customers including Disney Holidays, Suzuki and Equinox.
The company is a key player in the A/B testing tools market and is popular with brands and marketers with more modest budgets.
VWO splits its tools into five categories: Testing, Insights, FullStack, Engage, and Plan. It’s access to this variety of tools that makes it so popular.
BAO Ecommerce Account Strategist Holly Robinson says she found VWO easy to implement thanks to its simple step-by-step process.
And its reports are visually appealing, making them easier to analyse – perfect for anyone new to A/B testing.
Speaking of reporting, the tool provides statistical analysis of your results including a built-in heatmap tool that tracks visitor click behaviour and tracking habits.
VWO says it’s “fanatical about making sure you succeed with experimentation.” The company therefore provides 24-hour customer support, a dedicated customer success manager and on-demand team augmentation.
Smart notifications give you a quick overview of your test. You can see if a variation resulted in a smart decision, recommended action for your test, and how long you need to run this test.
Basic plans don’t include click, form submission, and engagement tracking for custom triggers. And all tests require basic development knowledge.
.png)
Since its inception in 201o, Optimizely has worked with over 10,000 businesses including H&M, PayPal, Toyota and Vodafone.
The company was founded by former Google employees Dan Siroker and Pete Koomen and, with its 21 global offices, prides itself on being “large enough to serve” yet “small enough to care.”
Optimizely is a one-stop shop with a fully integrated stack and the ability to integrate content, commerce, personalisation, search and analytics. On top of that, there are features such as headless and a cloud offering.
Optimizely is a powerful tool that caters to websites with high traffic. So if you’re very serious about testing and personalisation, it might be just the ticket.
Optimizely allows you to optimise your entire customer experience by expanding it to your mobile apps, messaging platforms and more.
As you can probably tell from the list of brands that work with the tool, Optimizely is aimed at enterprise customers exclusively and is, therefore, one of the most expensive on the market.
With a solid strategy and appropriate support, A/B testing tools can be a brilliant way to utilise learnings from site visitors and push your site to achieve its full potential.
Following our top tips and using the right tool for your business could lead to improved customer experiences, increased priority metrics, and reduced risks.
By Association Only is the Shopify Plus agency for the world’s most design-conscious luxury brands. Get in touch to discover how our team of experts can support your brand’s Shopify ambitions.
* Hero image by W. Carter.
%20(1).jpg)
Last week By Association Only hosted its annual summer soiree at Coal Office in London. Read all about it here.
We break down the most valuable updates from Shopify Editions, Unified.
Last week London-based ecommerce consultancy and paid media agency Vervaunt hosted the third Pulse Ecommerce Summit at The Brewery in London. Here are the highlights.
