How to asses conversion uplifts when you cannot run A/B tests

If your website gets less than 750 conversions a month, based on my experience, it’s quite difficult to run AB tests.

A/B tests that run across the entire website (like top ribbons, menu), or check-out A/B tests make an exception. However, running experiments on a homepage that gets 100 conversions a month is a no-go.

As a result, you may sometimes have to consider deploying a variation without running the split test. 

Crazy as it may sound and blasphemy as it may be, there are ways to minimize the risks of such an endeavor. This is precisely what we’ll be talking about in this article, throughout topics such as: 

• The 3 stages of assessing a landing page’s performance when you implement changes without A/B tests
• How to harness the power of surveys and heatmaps to make sure you’re moving in the right direction 

The 3 Stages of Assessing Webpage Performance When Changes Are Deployed Without A/B Testing

Here’s the approach I recommend taking whenever you find yourself in this type of situation – where, because of the low traffic amounts, conversions are limited, and running a split test is not realistic. 

Stage 1: What You Should Do BEFORE Going Live With the Change

Don’t change the traffic strategy

Before going live with the change, we recommend keeping the traffic acquisition strategy the same 15 days before implementing the change and 15 days after. 

For instance, if the page is getting Google Ads & Facebook traffic, you don’t want to throw Tik Tok ads in the mix once you go live with your design/copy change. 

Implement a page heatmap (and run it for 15 days before going live with the change)

The second thing you want to do is implement a heatmap on the “original” page – remember, you’re yet to have implemented the “variation”. 

You do this to establish a user-behavior benchmark so you can later compare it against the variation. If you’re not sure about how to implement heatmaps, check out our guide on the subject here. 

Implement an open-questions survey (and run it for 15 days before going live with the change)

Moving on to the next item on the list, you’re also going to run a brief survey on the original page.

What’s the purpose of this, you rightfully wonder? 

Well, you want to get a general feel for the first impression the control website version makes on visitors. 

Remember, it takes less than a few seconds for people to judge your website and decide if they’re going to leave or stay. 

Because time’s not on your side, the survey is going to be brief, but eye-opening. Here are the 3 questions we recommend asking your visitors:

• What is the purpose of your visit to our website today? 
• Were you able to find what you were looking for? 
• If not, why? 

Take a full-page screenshot of the current page version

Good. Now that you’re all done with the survey, the last thing on your list is to take a full-page screenshot of the current website version. There are free browser extensions that do this for you. 

For instance, you could use Full Page Screen Capture. Once you have your screenshot, create a separate folder where you’ll store all the information related to this process and copy it there. 

Stage 2: What to Do On the Day the Variation Goes Live

Fast forward to the day you plan on bringing the changes live. 

First things first. 

Turn off your heatmap recordings as well as your surveys on the “control”. 

However, you’re not quite done with these two tools as you’re about to set them up on the “variation” as well. 

Same heatmap, same survey. But starting from scratch so additional insights attributed to the new variation become crystal clear.

Finally, go ahead and take another screenshot of the entire page and place it in the folder where you added the control version. You will need both of them for your final analysis and result interpretation. 

Stage 3: What to do 15 days later after deploying the changes

Here we are now, fifteen full days after you’ve implemented the changes on the website without actually running an A/B test.  It’s finally time to assess the results. 

It’s high time to:

• Compare the conversion rate before and after the change (hopefully no big seasonality or traffic acquisition shifts happen within this 30-day period). I recommend comparing the before and after conversion rates by channel as well.

• Download the heatmaps and add them to your designated folder (alongside your existing control and variation screenshots) and start comparing user behavior before changes went live and after. 

• Then download the survey before and after the change survey answers and see what changed. Try to answer the answer to questions such as:
  • Do you see a difference in the questions users ask?
  • Is the control or the variation that confuses more/less the users?
  • Which variation brings fewer buying objections?

Once you’ve piled those stats up, it’s time to build your qualitative and quantitative insights report. You can either use our template you can download down below or you can set up your own. 

Final Thoughts

Implementing changes without running A/B tests is somehow an extreme measure but a good solution until you get more traffic and conversion on the website.

Taking into account the three stages of assessing a web page’s performance will not only minimize risk but also set you on the right path towards stacking up those wins and being the data-driven one in the team.