Here’s How and Why You Need to Run Hotjar Surveys During Your A/B Tests
It’s no secret that split-testing is the bread and butter of any conversion optimization process. However, few people are aware that analyzing an A/B test’s performance means more than just looking at the overall conversion rate.
Don’t get this wrong; conversion rate – or any other meaningful micrometric – uplift remains a key indicator. However, getting and inside perspective of your users’ key objections and friction points can lead to a windfall of A/B testing hypotheses – all with a huge chance of emerging as winners.
This is where survey tools, such as Hotjar, genuinely show their worth. Usually, setting up a survey on an URL is relatively straightforward. The challenge comes from setting up surveys for an A/B test when the variation and the control use the same URL.
However, you’ll find this easy-peasy, too, once you go through our detailed how-to demo from inside this post.
We’re also going to talk about:
- The Importance of Post-Analyzing A/B Tests
- Why Hotjar & Google Tag Manager Are a Match Made in Heaven
- How to Set up Surveys in the Blink of an Eye
The Crucial Importance of Post-Analyzing A/B Tests
We’ll start by saying A/B testing is a BIG topic. One that we thoroughly analyze in numerous other articles you can find on the blog section of our website.
Like the one on how to implement Personalized Surveys for New and Returning Visitors. Or this one where we show you how to always win at A/B testing. And maybe you’d also want to take a look at the one on how to prove your A/B testing’s worth. Aha-aha, lots of interesting stuff awaits you on our blog.
Where were we? Ah, yes… Now, if there’s one thing we’ve learned from our A/B testing experience, it’s this:
If you’re only looking at the general conversion rate, you’re completely missing the point of A/B testing.
From our point of view, you can achieve bigger wins by analyzing the difference in the way people interact differently with your website once the variation has been deployed. Yet most marketers completely ignore this opportunity to gain insights into their customers’ behavior.
There are two ways to analyze user interaction during a split-test.
- By segmenting the A/B test audience in tools like Google Analytics, Adobe, etc., and looking at conversion rate and other micro-metrics per segment (devices, age, prospect type, etc.). For more details about this, make sure to check out our post on why marketers mistakenly identify A/B testing winners for losers.
- By setting up qualitative tactics like surveys or heatmaps to capture users’ opinions and behavior.
For today’s article, we’ll focus only on surveys. Even more specific, we’ll show you how to implement them and capture your users’ opinions when running an A/B test that uses the same URL for both the Control and Variations pages.
Our go-to tool for this is Hotjar. However, keep in mind that the technical approach is similar regardless of which survey tool you use.
Setting Up The A/B Test
If you run A/B tests where the control and variation sit on different URLs, check out our post on generating new A/B testing ideas based on insights gathered from previous tests.
Now, let’s see the exact steps you need to take to set up the surveys when your split test uses the same URL.
If you’ve never set up an AB Test in Google Optimize, simply follow the steps below and you’ll be fine.
Start by clicking the ‘Create Your Experience’ button. Next, give your test a Name, and type in the URL of the Control page.
Pay attention to the type of experience you choose. For this type of experiment, you’ll need to pick A/B Test.
Google Optimize will take the wheel from here on. This means you’ll be able to code your Variation changes based on the Control page – all of which will have the same URL.
Now back to the steps…
Imagine the first image below represents your Control page. The one after it is Variation 1. Both share the same URL, because the Variation is directly coded or edited in your testing tool (Google Optimize, in our case). The only difference between the two is an extra menu button called ‘Our Clients’.
Control Page:
Variation 1:
Now focus! Here comes the not-so-tricky-but-essential part.
Once the split test goes live, you can’t waste any time before setting up Hotjar via Google Tag Manager. You’ll find out how to do exactly that in the paragraphs below.
Why Hotjar & Google Tag Manager Are a Match Made in Heaven
The first step of this process is understanding how Google Tag Manager and Hotjar intertwine.
To add your survey tracking, you’ll first need to implement a tag through Google Tag Manager.
Going back to our example, you’ve just created your A/B experiment that shares the same URL. The next step is to set up a survey for each variation. The purpose of this is getting as much information as possible about what keeps your users from converting.
Now, Hotjar needs a little bit of help for identifying each of the variations so it can fire the correct survey for each variation. The aid comes in the form of an ‘event’ (formerly called Javascript trigger).
(!) Note: this is not a Google Tag Manager event, so let’s call it ‘Hotjar event‘ further on.
Let’s walk you through the entire process and see how it’s done.
Step 1. Define Naming Conventions
Here at Growth Savvy, we’re not ashamed to admit we’re process freaks. Keeping things as clean and simple as possible is what makes us happy.
Naturally, this translates into a very well-structured A/B test naming convention. The convention must be as obvious as possible and must be reflected in the Hotjar Event naming.
For example, our test name is [Test 30] Menu Test.
The Hotjar events are named accordingly:
- menu_test_v0 for Control Page
- menu_test_v1 for Variant Page
As you advance through the next steps, you’ll need these event names, so we recommend sticking to the chosen naming conventions.
And since we’re such suckers for control, we find keeping a summary of our experiment status helps tremendously. (See image below).
Step 2. Setting Up the Triggers and Tags in Google Tag Manager
The basics: to fire Hotjar events for specific Optimize experiments/variants you need the Optimize JavaScript API.
Pretty straightforward so far, right? Now that we’ve got the groundwork set up for us, we can move on to the nitty gritty aspects.
2.1. Create a Custom HTML tag
You can either do this the hard way – by creating your custom template yourself – or our way – by using our very own custom-built template. If you’d like to use our template, drop us your email in the sticky box on the right side of your screen or inside the box at the end of this article. We’ll send you the step-by-step process on how to add it to your Google Tag Manager Container ASAP.
Now, let’s assume you’ll be using our template. Once the template is ready, go ahead and create a new tag.
Scroll down from the Tag Configuration menu and select the new custom template you’ve just added to your workspace.
The next step is to tell GTM which A/B test to look at. Do so by filling in the A/B test ID found in your experiment tool (for us, that’s Google Optimize.)
Not sure how to find the Experiment ID? Follow this 20 seconds tutorial.
Once you’ve identified it, copy-paste the Experiment ID in the appropiate field of the Tag Configuration.
It’s now time to instruct GTM about which Hotjar Event it must use for each variant. Here’s how you do that.
- Add menu_test_v0 in the field named Hotjar event name for original/control group:
- Add menu_test_v1 in the Hotjar event name for the variant field. (Add new rows for each Hotjar event you want to fire.)
Now your tag should look like this:
2.2. Add a trigger to this tag
The next thing on the list is to make sure this tag only fires once per page and only after the Optimize and Hotjar libraries have been loaded. This is because Google Optimize will use the callback when an experiment is active. This means using a window-loaded trigger on all pages would be totally safe.
Let’s do a quick recap of what you’ve accomplished so far:
- You’ve successfully drafted your experiment
& - You’ve added GTM setups to tell Hotjar where to register your users’ feedback
Little do you know, but you’ve successfully navigated through the most challenging part. It’s a downhill ride from here on out.
Let’s Go Live with the Hotjar Surveys
Time to add the finishing touches.
Head to your Hotjar dashboard. Locate the Surveys menu – it’s in the left panel. Now click the New Survey button in the top right corner.
You’re now ready to set up your survey.
Start by giving it a name in the Details sections (we usually use the Experiment name followed by the variation number). For example, the survey for our Control page would be “[Test 30] Menu Test-V0“. Now, click Next.
Select the Location of the survey you would like to have. We usually recommend the Popover type since it offers you the chance to connect the responses you get with the recordings from the same user.
Next you’ll have to set up the Questions and Appearance of your survey.
Our suggestion is not to go for more than three questions in your survey. Remember, your goal here is to build a better understanding of what keeps them from hitting that action button – not to create further frustration.
The Targeting section is next – and you need to keep your eyes peeled here, since this is where the magic happens.
Start off by picking the list of devices you want to target. Once you’ve done that, head to the Pages subsection -> Specific Pages. From the drop-down menu, select Event and copy and paste the Hotjar event name for the Control page (menu_test_v0, in our case).
You’re now ready to wrap up the survey setup.Let’s move on to the Behaviour tab. Our advice here is to show the survey on exit intent for desktop users. Meanwhile, mobile visitors will see it as a pop-up 45-seconds after landing on your page.
Note: Make sure to set up two different surveys if you plan on making this type of differentiation.
Last but not least, make sure you tick the box that instructs Hotjar to only show the survey once. The last thing you want to do is irritate your prospects.
The last two boxes you need to tick are found in the Forward Response section – where you’ll toggle the setups you want – and the Review & Activate section – where you’ll set your survey to Active and click Create Survey.
That’s it! Your first survey is ready and waiting to feed you insights directly from the mouths of your prospects.
You’ll now have to follow the same steps to add the second survey to Variation 1. We’d name it [Test 30] Menu Test-V1, then add the menu_test_v1 Hotjar event name into the Page Targeting section. Set your Survey to Active then click Create Survey, and you’re all set!
Now both surveys are live!
Don’t forget to update your beautiful summary table. At this point, ours would look something like this.
Final Thoughts
Next time you’re A/B testing a hypothesis, try adding a survey to both the control and your variation. There’s hardly a better way of seeing how and if your changes altered the prospects perception of your product. Plus, it gives you a great opportunity to see how your changes create different friction points and objections.
For us, surveys have proven invaluable assets. Being able to identify the prospect’s grievances gives us a nearly fail-proof method of addressing them and boosting our overall conversion rate.
Don’t forget to make life easier for yourself by getting our custom-built Google Tag Manager template via email. It’s 100% free, as are all our other resources. Being the data-geeks that we are, mining insights from the deepest data-pits is what we do best. We’d love to help your business organize your app or web analytics, improve the performance of your advertising efforts, or generate new conversion rate optimization opportunities. If that’s something you’d be interested in, make sure to book a free strategy call or drop us an email.
