A/B Testing for ecommerce

A/B Testing for ecommerce

Tips, tricks and the best tools for effective ecommerce testing 

Knowing what works and what doesn’t can make or break a business, so at By Association Only we encourage merchants to put the time, effort, and where necessary, money, into testing theories and pushing boundaries.

A/B testing tools are a great way to accurately and effectively do this, and, with Google Optimize being put to pasture in September 2023, we’ve had an increasing number of clients ask for recommendations on which paid testing platforms are best.

So let’s look at the advantages and disadvantages of A/B testing, what can be tested, and which tools are most useful.

What are the advantages of A/B testing?

We asked By Association Only’s Head of Merchant Strategy Natalia Howard and Ecommerce Account Strategist Rory O'Hara what the advantages of A/B testing are. Here’s what they had to say…

  • Improves the customer experience
  • Increases priority metrics such as conversion rate, add-to-cart rate and more
  • Helps merchants make informed decisions and replaces emotional responses with data-led recommendations
  • Helps reduce risks 
  • Helps merchants make data-led design choices
  • Gathers specific user data reasonably quickly
  • Maximises the value you get from your existing traffic – it’s a lot more expensive to get a new customer than retaining your existing customers

What are the disadvantages of A/B testing?

We also asked what the disadvantages of A/B testing can be. Let’s take a look…

  • Can be time-intensive – setting up your tool, running tests and analysing the  results will inevitably take time, so this is worth considering when deciding if a theory is worth testing 
  • Can halt progress – especially if a merchant wishes to test every change on site
  • Site speed and performance can be affected – due to the nature of the tools, which are quite heavy JavaScript
  • Can create false confidence – for example when testers just wish to see something reaffirmed
  • Skewed results – there are so many external variables that affect results, such as big product launches, key consumer periods, changes to the site, or changing which channels are driving traffic

What can retailers A/B test?

If you do decide that A/B testing is something that could benefit your ecommerce business, there are a wide range of aspects you can test including the following which are popular with our merchants:

  • Styling or placement of CTAs
  • Hero banner images
  • PDP galleries
  • Exposed search fields
  • Product card design
  • Text links versus image icons
  • Typography
  • Pagination versus infinite scroll
  • Menu structures and layout
  • PDP details UI

Implementing tests

A/B testing tools usually have two ways to run tests – a visual editor and a code editor.

Visual editors are often used to make small changes, such as changing the style or colour of a call to action (CTA).

Slightly more complex tasks require a code editor, which allows developers to directly implement new code. 

There is also a third option for even more complicated tasks, which is to make changes to a theme – the test version that a user will be served can be determined using a CSS class or data attribute. 

This is then deployed and the testing platform is used to assign users to either the control or test variations when they land on a page included in that test.

Top tips for successful A/B testing

Here are Natalia and Rory’s top tips for successful A/B testing:

  • Only test one thing at a time – so you know there are fewer factors influencing the data being returned
  • Don’t test for testing's sake – test for something you’d like to clarify based on a reasoned hypothesis or a problem you’re trying to solve
  • Have a structured approach – turn the data and insights you’ve collected into a clear hypothesis with a defined goal. Decide what your KPIs are and what successful results look like for your business
  • Prioritise – it’s easy to come up with lots of great ideas but where do you start? Consider effort level versus impact 
  • Run tests over an extended period – to account for variations in day-to-day traffic and performance – two weeks is an ideal amount of time although there are exceptions to this “rule”
  • Re-testing is important – something that was true six months ago might not be true now and it’s a good way to move the needle on priority metrics. Small adjustments on site can have a big impact

Which A/B testing platform is best for your business?

There are various factors to consider when choosing which A/B testing platform is best for your business – including cost, functionality and ease of use.

These points and more are considered below in our breakdown of some of the most prominent A/B testing platforms in the ecosystem.

All can be integrated with Shopify and have been tested by various By Association Only clients with the guidance of our Strategy team.

AB Tasty dashboard
AB Tasty dashboard

AB Tasty

AB Tasty was founded by Alix de Sagazan and Rémi Rémi Aubert in 2009. The company believes, “experimentation and personalisation should be fast – and simple.”

“Success isn’t spontaneous, it’s built on continuous optimisation: the process of trying, failing, refining, trying again and flying,” says the company.

AB Tasty works with over 1,000 brands including Sephora, L’Oreal and Eurosport.


AB Tasty offers A/B testing, as the name suggests, split testing, multivariate testing and funnel testing.

Its visual editor can be used to put together variants and tests including those based on URL, geolocation, demographic and more.

Users can also enrich the platform’s data by integrating existing data from CRMs and DMPS.


Ease of use 

From Rory’s experience using AB Tasty the tool is well suited for teams looking to make an impact without the need for a high level of technical support.

Setting up the tests is pretty straightforward, partly thanks to a seven-step guide taking users from ideas to publication, as well as the visual editor being really easy to use.

The interface is simple to understand with a useful preview mode, but, if you do face any problems the AB Tasty team is on hand to resolve any issues.


AB Tasty is reasonably priced and makes a great starting point for companies just beginning to experiment with conversion optimisation.


Less sophisticated data than other platforms

While the post-test reporting features are good and certainly sufficient for informing decision-making, the data gathered is not as in-depth as in other platforms.

The statistical significance calculator is fairly basic, for example, and there can be a delay in receiving test results.

Convert dashboard
Convert dashboard


Californian company Convert was founded in 2008. It claims to have optimised over 40,000 sites since then – including for Sony, Audi and Virgin Mobile.


Convert boasts over 40 filters for accurate targeting as well as a powerful API that allows you to run a headless version of Convert in your ecosystem.


Smooth onboarding process

Most plans come with a 10-step onboarding that includes proper snippet installation, setting up your first project, drafting and monitoring an A/A test, goals configuration and enabling integrations.

Ease of use

Convert’s drag-and-drop system is super easy to use, as is its visual editor which you can use to create A/B multivariate, multipage, and split tests.


Utilise over 35 different elements to build customer profiles.


Convert claims to be “the most affordable full-stack tool in the world” and its plans start at $499 per month. However it’s worth bearing in mind that most A/B testing tools’ plans are traffic-based, so this monthly fee would apply to a store with very low traffic. 

Live chat

Convert says its support is 10 times faster than other platforms and “gets you in touch with folks who know the app like the back of their hands.”

They’re not wrong – from our experience, the live chat support feature is very responsive and can help work through issues with the visual editor or configuration for tests.

Quick view of goal progress

… and user data in the listing page – very helpful for keeping track of expansive testing programs at a glance.

In-depth reporting features

For example, confidence intervals to help users feel as confident as possible in their data before making decisions.


Convert has plenty of targeting options to display tests to specific user groups based on country, browsing behaviour, traffic sources and more.


The company says it has structured its app, policies, and operational procedures “to offer enterprise-grade security and total alignment with the privacy and compliance shifts happening in the market.”


Tricky to use

As you can see Convert has a lot going for it, but it has one fairly major downside – it’s not easy to use.

For most changes in the visual editor, and for setting up custom tracking metrics, it’s helpful for users to have at least a basic understanding of HTML structure – you also need to be comfortable reading HTML while inspecting your website in the browser.

Even for experts such as ourselves the visual editor can be somewhat cumbersome – as can the revenue tracking setup, setting up goals and reporting metrics.

VWO dashboard
VWO dashboard

Visual Website Optimizer (VWO)

Launched by Paras Chopra in 2o09, VWO works with over 2,500 customers including Disney Holidays, Suzuki and Equinox.

The company is a key player in the A/B testing tools market and is popular with brands and marketers with more modest budgets.


VWO splits its tools into five categories: Testing, Insights, FullStack, Engage, and Plan. It’s access to this variety of tools that makes it so popular.


User friendly

BAO Ecommerce Account Strategist Holly Robinson says she found VWO easy to implement thanks to its simple step-by-step process.

And its reports are visually appealing, making them easier to analyse – perfect for anyone new to A/B testing.

Informative reporting

Speaking of reporting, the tool provides statistical analysis of your results including a built-in heatmap tool that tracks visitor click behaviour and tracking habits.

Responsive customer service team

VWO says it’s “fanatical about making sure you succeed with experimentation.” The company therefore provides 24-hour customer support, a dedicated customer success manager and on-demand team augmentation.

Alerts of winning/losing tests

Smart notifications give you a quick overview of your test. You can see if a variation resulted in a smart decision, recommended action for your test, and how long you need to run this test.



Basic plans don’t include click, form submission, and engagement tracking for custom triggers. And all tests require basic development knowledge.

Optimizely dashboard
Optimizely dashboard


Since its inception in 201o, Optimizely has worked with over 10,000 businesses including H&M, PayPal, Toyota and Vodafone

The company was founded by former Google employees Dan Siroker and Pete Koomen and, with its 21 global offices, prides itself on being “large enough to serve” yet “small enough to care.”


Optimizely is a one-stop shop with a fully integrated stack and the ability to integrate content, commerce, personalisation, search and analytics. On top of that, there are features such as headless and a cloud offering.



Optimizely is a powerful tool that caters to websites with high traffic. So if you’re very serious about testing and personalisation, it might be just the ticket.

Optimise your entire customer experience

Optimizely allows you to optimise your entire customer experience by expanding it to your mobile apps, messaging platforms and more.



As you can probably tell from the list of brands that work with the tool, Optimizely is aimed at enterprise customers exclusively and is, therefore, one of the most expensive on the market.


With a solid strategy and appropriate support, A/B testing tools can be a brilliant way to utilise learnings from site visitors and push your site to achieve its full potential.

Following our top tips and using the right tool for your business could lead to improved customer experiences, increased priority metrics, and reduced risks.

About By Association Only

By Association Only is the Shopify Plus agency for the world’s most design-conscious luxury brands. Get in touch to discover how our team of experts can support your brand’s Shopify ambitions.

* Hero image by W. Carter.

Recent articles you may also like

“The most unified Shopify yet” 

We break down the most valuable updates from Shopify Editions, Unified.

By Association Only sponsors Vervaunt’s Pulse Ecommerce Summit for the third year

Last week London-based ecommerce consultancy and paid media agency Vervaunt hosted the third Pulse Ecommerce Summit at The Brewery in London. Here are the highlights.

‍By Association Only creates bespoke Story Chain visualiser for jewellery brand Astrid & Miyu