We have a very cool feature coming out today in Kissmetrics: our new A/B Test Report! In the past if you wanted to run an A/B test using your Kissmetrics data you would need to first run a funnel report and then manually enter your data into whatever external tools you use for your analysis. Now you can do all of your A/B test reporting from within Kissmetrics!
Turning your Kissmetrics data into insights
Let me give you a quick walkthrough of what you’ll find.
To start out we have to choose a target event which we’ll consider our “conversion event”, and the Kissmetrics property that will serve as our “experiment”.
For this test we’re going to use the ‘Signed up’ event as our conversion event. The really cool thing to note here is that any event being tracked in Kissmetrics can be used as conversion event!
Next we have to select which experiment we’re going to look at. In this case we’re going to use the property associated with an Optimizely test we’re running. This is a great example of the fact that you don’t have to give up on using your favorite A/B testing tools in order to get all your data and reports in one place.
A/B Testing is all about making a comparison; so all of your results are given in relation to a baseline. Normally this would be your control or original variant, but our reporting tool lets you choose any of the values of your property to serve as a baseline. It’s also important to point out that while this example only has 2 values for the property this report will work with any number of values that a property might have. Testing more than one variant at a time is no problem.
When you run the report it gives you a nice summary of the results. Notice at the top we provide you with a simple explanation of your results. In this case there is enough data and strong enough significance to make the call.
We also give you all the key summary data you’ll need to understand your results:
- How long the test ran
- The number of people in the test
- Total conversions
- The improvement you’re likely to see
- How sure you can be of seeing an improvement
Exploring in Your Data
For those looking for information beyond a simple summary we have more for you!
First the report provides a visual history of the estimated improvement as more data is collected. This is important because just looking at significance alone can be deceptive. We mark when the results have achieved what is commonly considered to be statistical significance with a trophy icon and shading in the timeline after this has occurred. However in the early stages of a test it is not unlikely that the inferior variant will temporarily look like the winning one.
Looking at this history you can clearly see how much your improvement is jumping around and likewise use that to get a better intuition of how trustworthy your results are. The more stable your estimate of improvement the more likely it is to be accurate.
Finally, for anyone that is looking to do more analysis on their own, our report provides all the data you’ll need. This gives you much of the information found in our typical funnel report plus you get the likely improvement of each variant as well as the certainty that there is an improvement.
Kissmetrics is about People!
The focus of Kissmetrics is people, and so we have also built in the ability to explore the individual people that are going through your test.
Click on any of the points in the improvement timeline and you’ll be presented with links to run a People Report so you can see who in each variant converted, or simply everyone who passed through that variant.
Changing Your Conversion Event
Now let’s go back to the feature that you can leverage all of your Kissmetrics data in this report. We have our result for conversion to signups, and things look great, but thinking about it maybe we should also see how the result look if we swap out ‘Signed up’ for a more important conversion event ‘Received data’.
All we have to do is go back to the top of the report, change our conversion event and rerun our report!
Now we can see that if ‘Received data’ is the conversion event we really care about, we don’t have enough data to call our test. In this case the results are looking pretty close even after over 40,000 observations. The report is letting us know that maybe in this case it is best just to stop the test and stick to our original. Of course we’re free to continue running our test, and eventually we should reach a conversion, however it is unlikely we’ll see the real gains were looking for.
Comparing Multiple Variants
To highlight a few more features let’s go back and look at results from a test that we ran a few months back, long before work was even started on this report! Because of the way we can make use of any of our Kissmetrics data – the event/property combination we’re investigating can be a test we ran long ago, or even a combination that was never even thought of as being an A/B test in the first place!
In this report we’re comparing 3 variations against the original page. Here we can see how the report handles this. We get 3 lines, one for the comparison of each variant against the baseline. The report tells you which of the 3 variants is the superior.
This is great, but it looks like the variants are all doing well against the original, let’s see how they do against each other.
This time you can see we’ve switched our baseline from the original to variant 2, the clearly superior variant. We’ve also deselected the original so that it no longer appears in our visualization. In our data table all of our improvements and certainties are expressed in terms of our new baseline. Right away we can see that variant 2 and variant 3 are actually very close, certainly too close to call. This is extremely useful to know, maybe your design team prefers variant 3, now you know you’re free to make that choice with likely little or no loss in conversion.
The new A/B Test report will open up many new ways to explore and gain insights from your existing Kissmetrics data. I hope you find it as exciting and useful as I have!
Google Analytics tells you what’s happening. Kissmetrics tells you who’s doing it.
If you’re interested in trying Kissmetrics to gain actionable insights about the people who pay you, then sign up today: