Kissmetrics Blog

A blog about analytics, marketing and testing

Built to optimize growth. Track, analyze and engage to get more customers.

A/B Testing Plot Twists We Didn’t See Coming

The last turn of events you want in your A/B testing program is a M. Night Shyamalan-eque plot twist where a key measure plummets after an initial success.

This is a common risk for most testing programs if you judge your test’s success by what you immediately want a visitor to do next. Do more of them do that next step? Excellent! Winner! Celebrations all around! But honestly, did you look and see if the right people took all the right subsequent steps? For example, a variant may lead more people to the next step in your funnel, but does that carry throughout the entire funnel? Often times, a variant wins in one step of the funnel but actually decreases signups. It’s happened to us plenty of times.

Here at Kissmetrics, we recently ran a test on the homepage, with the not-too-uncommon problem that we don’t love our main call-to-action. It asks for a lot of trust right away, it’s confusing about what you get from a Google-based login, and so on. But it keeps winning, each and every time, against all kinds of other formulations. (If you’ve got ideas about how it makes visitors feel, or what might be better, please suggest away in the comments.) Here it is:

kissmetrics-homepage-tagline

This “Log In With Google” has beat every CTA we’ve tried it against.

But then we tested it on the blog, and got a much different result.

We tested Try Kissmetrics as the CTA instead. Much more indicative of the offering, we told ourselves with anticipation of great results. And it had a near 19% increase over the previous CTA:

signed-up-kissmetrics-blog-ab-test

Completed trial signups were up 42%, 74% increase in active trial use. Time to move it across the site, right?

We’ll call this plot twist the Usual Suspects: If the test won in one area of the site, apply it everywhere and it should still work. But that’s not what happened.

When we tested again on the Kissmetrics homepage, it won – but only on the surface, like pulling the name of Kobayashi from a coffee mug.

In terms of getting to the next action, the test seemed to do very well. More than 85% improvement on clicks, actually. With 99% certainty.

optimizely-kissmetrics-test-data

Sounds like you’re ready to leave the planet of the apes, right? Only, no. You’re still on Earth and the Statue of Liberty is right there in the sand.

In this case, we ended up with fewer completed trial signups (down 2%), fewer active trial users (down 25%), and less revenue:

kissmetrics-received-data-down-2-percent

25.24% fewer active users:

kissmetrics-fewer-active-users-ab-test-report

Not a winner. So we’ll keep trying to find the right match, or accept that we might have found what works for visitors, even if it’s not an internal favorite.

And regardless for your love of the test element, it’s just always better when you know what’s in the box. And that talking with dead people is okay. Plot twists are much better in movies than in your sales.

Essentially, our conclusion is that the customer is always right in A/B testing, and so we need to live with the CTA on our homepage. It’s withstood and won so much testing. At least we solved a problem for blog readers, who are visiting us to learn about marketing techniques, and are better understood as a separate audience.

Update: We kept this test running and saw that, over time, more total subscriptions were coming in via the Try Kissmetrics CTA button. Not only more accounts, but accounts that were worth more revenue overall. Another reason it’s good to keep watching tests over time. As of December, Try Kissmetrics is our new homepage CTA.

About the Author: Maura Ginty is the VP of Marketing at Kissmetrics.

  1. To tell you the truth, I’ve taken a look at the homepage, but at first glance I didn’t know what to do with it.

    Here’s an idea you can check there if you like, It will at least help with the clarity issue.

    Add a small line of text just above the buttons in the center .

    ——
    To try it out
    [Enter your website URL] [and login with Google]

    ——

    I would love to know if you tried it and how it fared throughout the flow

    ** yes, an “and” in the button, the sky won’t suddenly collapse :)

    Love your work, good luck.

    • Following on from Yaron’s suggestion, you might want to connect the additional text/CTA to the main text in order to propose value/state a reward for the action.

      i.e. Instead of “To try it out”, something like “Start optimizing”.

      • We’re planning some tests along these lines and we’ll definitely let you know the results back on this post. Please keep the suggestions rolling and thanks for sharing your great ideas with us!

  2. This is why I use Kissmetrics A/B test for “completed purchase” , instead of Optimizely, which does not let me choose conversion as a goal.

    My question is, how can I use the Kissmetrics A/B test to optimize for the higher amount of revenue? I have a feeling I can’t, but that’s what I’m really after. Not necessarily more orders; higher value orders.

    Thx

    • Hey T,
      There is no easy way to do this in the A/B Test Report. However, if you have all the tracking, you should be able to hack something together in the Power Report.

      You may want to contact Kissmetrics Support (support@kissmetrics.com) and they can provide further assistance.

5 comments

Please use your real name and a corresponding social media profile when commenting. Otherwise, your comment may be deleted.

← Previous ArticleNext Article →