Kissmetrics Blog

A blog about analytics, marketing and testing

Built to optimize growth. Track, analyze and engage to get more customers.

A Beginner’s Guide To A/B Testing: An Introduction

A/B testing is a fantastic method for figuring out the best online promotional and marketing strategies for your business. It can be used to test everything from website copy to sales emails to search ads. And the advantages A/B testing provide are enough to offset the additional time it takes.

Well-planned A/B testing can make a huge difference in the effectiveness of your marketing efforts. Narrowing down the most effective elements of a promotion, and then combining them, can obviously make your marketing efforts much more profitable and successful. Here is the first in a series of posts covering A/B testing that we’ll be publishing in the coming weeks.

How Do You Plan an A/B Test?

plan your ab test

The first thing to do when planning an A/B test is to figure out what you want to test. Are you running an on-site test, or an off-site test? If you’re running an on-site test, you’ll want to think of all the sales-related pieces of your website, and then figure out which elements you want to test.

With off-site tests, you’re probably testing either an ad, or a sales email. Testing ad copy to see which ad brings in more converting visitors can help you focus your advertising efforts. Once you know your ad is converting as well as possible, it’s easier to justify spending more money on it. The same goes for emails. If you send out two versions to your list (randomly selecting which half gets which email), and then track which one converts better, you can send only that version the next time.

Once you know what you’ll test, make a list of all the variables you’ll test. For example, if you’ve decided to test your call to action, you might test:

  • the location of the call to action
  • the exact text used
  • the button color or surrounding space

It’s a process, and it’s common for multiple A/B tests to be carried out prior to making a final decision or final change.

Make sure that before you start testing you have a clear idea of the results you’re looking for. You should already know your baseline result, which is the results you’re currently getting. You want to test option A and B against each other, but you also want to know that whichever one does better in the test is also doing better than your current results. Alternatively, you can use A as your control (leaving it whatever you’re currently using) and then use something new for B.

Tests need to be run simultaneously to account for any variations in timing. You can’t test one variation today and the other one tomorrow, because you can’t factor in any variables that might have changed between today and tomorrow. Instead, you need to split the traffic seeing your variations at the same time.

A/B Testing Can Improve Your Bottom Line

increase sales bottom line through ab testing

Accurate A/B tests can make a huge difference to your bottom line. By using controlled tests and gathering empirical data, you can figure out exactly which marketing strategies work best for your company and your product. When you figure that one variation might work two, three, or even four times better than another, the idea that you would conduct promotions without testing starts to seem a bit ludicrous.

A/B testing, done consistently, can improve your bottom line substantially. If you know what works and what doesn’t, and have evidence to back it up, it’s easier to make decisions, and you can often craft more effective marketing materials from the outset. Just remember to keep testing regularly, since the effectiveness of anything can change over time.

Things to Test

You can test virtually anything in your marketing materials: headlines, calls to action, body copy, images, etc. If you can change it, you can test it. But that doesn’t mean you should necessarily spend months testing every little thing. Instead, focus on the things that are most likely to have a big impact.

On your website, this likely includes:

  • the headline
  • your call to action
  • any graphic you use in direct correlation to your sales efforts
  • the sales copy or product descriptions

In an email, it probably includes the same. In an ad, especially a text ad (like a search ad), you have fewer things to change, and so likely you’ll test either the main headline or the offer itself.

Testing different offers is also important. Just make sure that you have methods in place to ensure that each person is always offered the same promotion. For example, if a free gift is offered to group A, and a discount is offered to group B, then you want to make sure that group A always contains the same visitors, as does group B.

You can also test things in conjunction with each other. For example, you might want to test newsletter A with landing page A, and newsletter B with landing page B. And then, later, you may want to test newsletter A with landing page B, and vice versa. This can give you a more concrete result if you’re getting mixed results, or if your results are very close.

Devoting Time to Testing

A/B testing is not an overnight project. Depending on the amount of traffic you get, you might want to run tests for anywhere from a few days to a couple of weeks. And you’ll only want to run one test at a time for the most accurate results.

Giving a test insufficient time can mean skewed results, as you don’t get a large enough group of visitors to be statistically accurate. Running a test for too long can also give skewed results, though, since there are more variables you can’t control over a longer period. Make sure that you stay abreast of anything that might effect your test results, so that you can account for any statistic anomalies when reviewing your results. If you’re in doubt, it’s perfectly reasonable to retest.

Considering the impact A/B testing can have on your bottom line, though, it’s worth taking a few weeks to properly conduct tests. Test one variable at a time, and give each test sufficient time to run.

Can I Test More Than One Thing At a Time?

There are two approaches to this question. First of all, let’s say you just want to test your headline, but you have three possible variations. In that case, running a single test and splitting your visitors (or recipients in the case of an email) into three groups instead of two is reasonable, and would likely still be considered an A/B test. This is more efficient than running three separate tests (A vs. B, B vs. C, and A vs. C). You may want to give your test an extra couple of days to run, so that you still have enough results to base any conclusions on.

Testing more than one thing at a time, such as headline and call to action, is a multi-variate test, and is more complicated to run. There are plenty of resources out there for multi-variate testing, but we won’t be covering that when talking about A/B testing.

Be sure to check out our previous article on website testing: A Beginner’s Guide to Google Website Optimizer. Also, we’ll be covering a number of other A/B testing topics as part of this series, including website copy, email campaigns, pay-per-click advertising, and SEO landing pages.

About the Author: Cameron Chapman is a freelance designer, blogger, and the author of Internet Famous: A Practical Guide to Becoming an Online Celebrity.

  1. Love it great write up is also a great place to check out testing

  2. Andreas Nurbo Feb 01, 2011 at 10:43 am

    Good intro to splittesting. When it comes to testing I really liked the book Always Be Testing: The Complete Guide to Google Website Optimizer. Covers stuff more thoroughly.

    On a similar note if you are using WordPress and want to use Google Websiteoptimizer I’ve made simple plugin that supports both A/B splittest and multivariate. Also implemented linkclickconversion support.
    Find download link here:
    Currently it only covers pages and posts. Got some ideas for making it possible to test all areas of your WP site.

  3. Lauralee Flores Feb 01, 2011 at 7:42 pm

    Great reminder of the need to continually be testing! Thanks for this post.

  4. One of the best A/B testing articles I have read. Very important to do this to see which is a better ROI. Many times you will be surprised as to what works best.

  5. Thanks @ Andreas for the book recommendation “always be testing..”. I have just begun to focus on conversion optimization . It has been a shift from traditional SEO disciplines and I think that conversion optimization is simialr to SEO in that its an art and a science.

    • SEO has become so popular that too many people know about it. The difference between people are successful and those who aren’t, are the ones who do testing and comparisons.

  6. Awesome write up. Im in the middle of setting up some split testing with a wordpress install. You’ve given me some very helpful tips here…thnx! I love that we can take the guessing out of what works and doesn’t. Design and layout usually end up being driven by the decision makers perception about what will work and not. This way it will be straight up statistics that can decide.

  7. Shilpi Banerjee Mar 20, 2015 at 3:28 am

    Awesome article for beginners

  8. This is a nice article (nice link bait too!) for those who are starting off in A/B testing. I guess the proof is in the URL! (“introduction”).

    In terms of the Things to Test section – you mention, correctly, that you don’t want to spend months and months testing the wrong thing. We boil this down quite simply to “if you’re not testing perception or behaviour, you’re not testing”. Usability changes are, simply put, usability changes. They are somewhat subjective in nature too.

    Here are 10 questions you can ask yourself to see if you’re A/B testing correctly:

  9. Alex Palmer (LNG-CAM) Feb 10, 2018 at 1:13 pm

    Sorry for the stupid question – how does one publish 2 different versions of a web page at the same time? Take this example:
    User clicks a button on page 1
    They are taken to page 2 where there is a button ‘buy this item’;
    I want two versions of page 2, one with a big ‘buy’ button and one with a small one, to see which button people press more.

    How do I send the user from page 1 to one of my 2 variant page 2s so I can test the results? I am not a coder so if there is anything that will help a layperson understand that would be great!


Please use your real name and a corresponding social media profile when commenting. Otherwise, your comment may be deleted.

← Previous ArticleNext Article →