Kissmetrics Academy

Learn how Kissmetrics can optimize your marketing

Built to optimize growth. Track, analyze and optimize your growth cycle.

How to Know if an A/B Test Impacts The Bottom Line

Before you launch your next A/B test, there’s an important thing you need to know.

Variants that appear to be winners may not actually be winners when it comes to making you more money. In fact, sometimes they may be losers that make you less money. And, if you aren’t using the right tool, you may unwittingly launch those losing variants.

Here’s a graph to visualize the problem:

ab tests funnel kissmetrics

If you were using a tool that tests only to the next conversion step, you would pick the variant as the winner and show 100% of all visitors the variant headline. But as we can see, that would actually be detrimental to your business because it results in fewer customer signups.

Luckily, you don’t have to be left in the dark with this data. The Kissmetrics A/B Test Report is here to save the day (and your tests).

If you want to test for outcomes further in your funnel, you’ll have to use a tool like the A/B Test Report. (This tool does not help you create A/B tests. Keep using your favorite tool for that.)

The A/B Test Report connects all the data from a test into your Kissmetrics account, and it allows you to track any part of a funnel. You aren’t limited to testing to the next conversion step.

Let’s create a test and track the results in the A/B Test Report.

1. Create an A/B Test

Chances are Kissmetrics integrates with the A/B testing platform you’re using. And connecting Kissmetrics to your platform is super easy. It’s just like flipping a switch.

Once we’re connected, we’ll can create a new A/B test. We’ll name it Homepage Headline Test, and launch the test.

2. Track the Data with the A/B Test Report

We head over to Kissmetrics and load the A/B Test Report:

ab test report kissmetrics setup

We have two things to select – the conversion event and the experiment. The conversion event is where we select the outcome. For our test, we’ll choose “Signed up”, which triggers when a visitor signs up for a free trial. Again, we can select any part of our funnel. We aren’t limited to testing which headline brought more visitors to the next conversion step.

The experiment is the test name. What we name the test in our A/B testing tool is what it’s saved under in Kissmetrics.

A Couple of Guidelines

Before we go on, I want to share a couple guidelines for running A/B tests:

  • A/B tests need to run long enough to reach statistical significance. If you cut a test short and declare a winner, you are increasing your odds of having false positives, which are variants that look like winners but actually aren’t. To be safe, you should wait until you have at least 95% certainty. Ideally, wait until you reach 99% or 100% certainty. The A/B Test Report will tell you the percentage certainty of each variant.
  • The more tests you run, the higher your conversions will be. As you continually run tests and get data, you’ll begin to understand the variables that can move the needle. You’ll reach better conversions because you’ll be running more tests; and, over time, you’ll accumulate more wins. It’s important to balance this with the fact that you need to allow your tests to run long enough.

If you allow your tests to run long enough and stick with the mindset that more tests means higher conversions, you’ll be setting yourself up for success. Now let’s get back to our test and see our data.

3. Get the Results

We’ve let the test run, now it’s time to view our results. We’ll start by viewing which variation brought more free trial signups. Here’s our data:

ab test report example variant

This is the summary of the test. Up top we see the Report has determined the Variant to be the winner. Under that we see how long we ran the test, how many people were in the test, the number of total conversions (for both the original and the variant), the percentage improvement the variant had over the original, and the certainty of improvement (which is the statistical significance referred to earlier).

The center of the Report is the visualization of the test data. The variant has been the clear winner for as long as the test has been running.

The bottom rows contain metrics for both the original and the variant.

So this appears to be great news. Our test variant beat the original on free trial signups. But we can’t pop the cork just yet. We need to go further into the funnel and test for the outcome where it really matters, which is the conversions of free trial users to paying customers. To change our Report for that outcome, we’ll simply change the conversion event from “Signed up” to “Subscription Billed – New Customer.” It’s pretty easy:

changing outcomes ab test report kissmetrics

We’ll hit Run Report and get our data:

funnel testing ab test kissmetrics ab test report

This shows us that our original actually got more free trial users to convert to paying customers. While the variant beat the original on free trial signups, it actually lost when it came to converting free trials to customers. This may be because our variant headline made a promise it couldn’t keep.

If we had not used the A/B Test Report and looked at only free trial signups, we would go with the variant that brings fewer customers, and we’d be shooting ourselves in the foot.

This exemplifies why the A/B Test Report is so important for marketers: it allows you to test any part of the funnel, which can help you avoid launching variables that bring you less money. In fact, you can even go out into the wild and test for outcomes outside your funnel.

Want to know if changing your features page leads more people to use it? No problem.

Want to test a new onboarding process to see if it increases product utilization? Simple, just set your desired outcome and get your results.

Go beyond simple “which button color led to more click-throughs” and test for real outcomes that matter to your business with the A/B Test Report.

Watch a Two-Minute Demo

Want to watch a quick demo of the A/B Test Report? Just click Play below.

Signup for a Kissmetrics trial to get your hands on the A/B Test Report.

Want to tour Kissmetrics without signing up? No problem. We have a demo site where you can see how Kissmetrics benefits both SaaS and ecommerce companies.

About the Author: Zach Bulygo (Twitter) is a Content Writer for Kissmetrics.

  1. Pawel Ogonowski May 08, 2015 at 5:27 am

    Zach, you have pointed out extremely important thing — many people who dive into A/B testing focus way to much on a single metric like conversion rate from one step to another. This way they forget that business is not about conversion rate optimization, but rather making money. As you have pointed out there is a long way between users who register for free trial and users who start generating cash for the business. Focusing on the former group will get business nowhere if they forget about the latter.

    If you were interested, a few days ago I have written an article that covered a similar topic here: https://goo.gl/GH7Jyo — I would appreciate your comment there.

    Pawel Ogonowski, @Pajex

    • Pawel,
      Solid stuff – it’s a big mistake to not see the big picture. If you’re increasing conversions to the next step but decreasing it at the end of the funnel, you’re shooting yourself in the foot. What’s worse is that a lot of marketers don’t have this data available.

      Thanks for your comment and sharing your article!

3 comments

Please use your real name and a corresponding social media profile when commenting. Otherwise, your comment may be deleted.

← Previous ArticleNext Article →