Kissmetrics Blog

A blog about analytics, marketing and testing

Built to optimize growth. Track, analyze and engage to get more customers.

The Strategic Approach to Split Testing

Contrary to popular belief, every split test is not a great split test.

The first thing many people do when they plan to run a split test is decide what they’re going to test. This is a common “tactical” approach that starts with specifics and can, no doubt, produce wins. But, it also can cause people to accidentally split test useless factors that have no effect on overall conversion rates.

In fact, previous experiments have taught us that more than 71% of self-created “assumption” designed split tests will not increase conversions, and may even reduce conversions.

With such a low margin for randomized success, the overall result of assumption split tests are often small minor improvements or flaky results that leave people scratching their heads.

Huge variations are possible in the quality of split test results.

The SplitGen Process – Introduction

Below is a “time tested” split testing system that we coined the “SplitGen” process. It is a strategic method for developing and generating a list of potential split tests that can produce big wins instead of incremental improvements.

You can adopt this process if you want to turn assumption style results into more solid, reliable wins. For best results, compare (or split test!) the following SplitGen process against any tactical assumption style of split testing.

The result will be that the SplitGen process produces more big wins consistently, time and time again.

the split gen process

The Tactical “Assumption” Approach

From experience, we know that most split testing is done with a very tactical approach. A tactic is something like changing the color of a “buy now” button or adding a “satisfaction guaranteed” logo. Those are tactics.

A tactical approach means the team/person in charge of running the split test starts out with a very rigid idea of what will be split tested and which exact tactics will be used.

They start out with a team brainstorming session or, even worse, sit alone and predict which factors will affect conversion rates. Then, they use these predictions and self-deductions to build a list of things to test. Or, they approach the split test with their own bag of tricks, and try to paint every split test with the same brush.

This is the equivalent of taking a questionable recipe and running a taste test with a single person. Then, on that single person’s approval, taking the recipe and producing it on a mass scale and expecting everyone to like it. Even if the single taste tester is a chef, this method of food production is rarely done in the commercial world. This method is all based on one person’s opinion, and it is bad business sense.

Yet, when it comes to split testing, the common practice is to start out with a previously created list of tactical split tests, and then apply those split tests to a website, etc.

This happens because of the Endowment Effect, which tells us that we value our own ideas more than the ideas of others. In addition, we tend to seek only validation (not criticism) of our ideas. So, a fair analysis of the list of split tests is never truly realized; and, then, when the tests are run, we find only minor improvements.

But, even worse, the minor improvements are seen as an overall success, so the weakest method of split testing continues on and on. This leads to achieving small wins when major wins could be produced.

So, don’t start a split test with a preconceived idea of what needs split testing. Instead, start with a strategic overview of how to conduct a “big win” split test.

The Strategic Approach

In order to find enduring “consistent” big wins, split testing needs to be approached strategically. Strategy is a long-term overall picture. It does not contain any rigid details like button color, etc.

Skillfully planning your split test formula, developing a methodological approach, researching your factors, and then executing efficiently is all part of the SplitGen process. It’s a strategy for producing big wins.

The reason people focus more on tactics is because they’re much more evident. Compare that to strategy, which takes time and effort.

But, strategy always works to produce larger wins. If you put in the work to be as strategic as possible, you’ll produce much bigger wins on a much more consistent basis.

The best way to be strategic is to use a split testing “process” – an actual system that acts as an overall strategy. When you run that system, its job is to deliver the tactical factors you need to split test in order to produce big wins that will make a real difference.

The SplitGen Process – Implementation

The SplitGen process is actually refined down from the system used by top level companies that split test in the real world to improve manufacturing and sales processes across multiple industries.

There, it’s possible to be working within an industry that you have no experience in at all, and where a website is not being used to sell anything. Often, there are no button colors, headlines, or CTA’s to test against each other.

In such an open and seemingly “harsh” environment, there’s no way you can approach the split test with a preconceived bag of tricks or a previously created list of factors and constraints to test.

If you’re a tactical split tester, you’ll have no clue. But, if you use the strategic method of split testing, you’ll understand how to do this right away.

Legendary statisticians, like R. L. Plackett, J. P. Burman, Walter Shewart, W. Edwards Deming, and Charles Holland, have been refining this statistical process control into an advanced method of split testing for many years.

We can take this advanced method of performing split tests and modify it for online use. When applied to a sales website, this strategic approach will consistently produce much bigger wins than a tactical approach alone.

The strategic approach explained below has been adjusted so that it can be applied to sales websites. It produces consistent big wins.

Here’s how to use the SplitGen process for a strategic approach to split testing:

Phase 1: Diagnose

You don’t go to your doctor and say, “Hey doc, I need a heart transplant, let’s get this done!” Neither does your doctor look at you as soon as you walk in and tell you what you need. The first thing the doctor does is ask questions and diagnose your condition.

Approach split tests in the same way. You shouldn’t be brainstorming any split testing ideas at this time.

web diagnosis record

Ironically, in the manufacturing process, most split tests start with asking the highest paid person’s opinion (HiPPO).

However, it was actually found that HiPPOs and top level executives are just as bad as other employees at predicting what factors increase conversions. Successful suggestions aren’t limited to any specific level of employee. History has proved that anyone can suggest a winning factor.

It’s the same with websites.

You need to watch how people use your website. What slows them down? What gets in their way? What stops them from making a purchase? Are there any technical issues? Does your site look the same way across multiple browsers? Is there a coding error on your website? Is something confusing your users?

You need to know exactly where the hold-ups are.

If you’re not able to watch a group of people using your website, a service like can be helpful. Another great tool for observing how visitors interact with your website is Crazy Egg.

Phase 2: Quantify

You need to make sure you understand exactly how you define success. Are you trying to increase the number of email opt-ins? Are you trying to increase the conversion rate of people who move from page four to page five? Are you trying to lower the cost of each new customer? Are you trying to maximize your earnings per visitor?

Whatever you’re trying to achieve, you need to know what specifically it is and how you will measure it. You also need to make sure that your method of measuring these objectives is extremely accurate.

Phase 3: Build a Representative Sales Funnel

Create a visual representation of your entire sales funnel, and seek to understand these three important things:

  1. What happens to a visitor before they arrive on your website?
  2. What process does the visitor go through while buying on your website?
  3. What happens to the visitor after they have bought on your website?

Don’t create this as a text file. It’s very easy to skip this part, but this is not the time to be lazy.

You need to have a visual representation of every single step named above. Think about the individual actions that your visitor takes, and draw them out as a path.

This will give you a great overview of the business.

You should understand how a visitor first learned of your website. Did they see an ad or were they a referral? What did the ad promise them? Was that promise kept when they arrived on your website? Where was the ad displayed? What is a visitor from that traffic source like? For example, Google users tend to be more technically minded compared with Bing users.

Also, think about the keywords that bring people to your site. What sort of visitors are they attracting? What do visitors want to do on your site? Is the majority of traffic on your website made up of first time or repeat visitors? Is there a specific traffic source that converts very well? If so, you can build a custom landing page for visitors from that source.

This information will come in handy later on as well.

So, try to get as much information about your exact sales funnel as possible, and build a visual representation of it.

The aim here is to look for potential areas of your business where you can make simple changes that will potentially produce big wins.

kissmetrics funnel screeny

The KISSmetrics funnel and KISSmetrics people reports are extremely useful for this.

You also should be looking for areas where you can increase revenue. For example, if your website sells shoes, you may be able to replace your current “thank you” page with an additional purchase offer for a matching belt. Or, if you sell luxury car wax, you can ask buyers if they want you to send them a reminder to buy more wax in 60 days (when their bottle starts to run low).

Phase 4: Gather Competitive Intelligence

Take a look at your competitors and their businesses. What do they do successfully? Have they been running a specific ad for a long period of time? If so, try and go through their sales process via that ad. If the ad has been appearing for a long time, you can bet it is producing good sales for your competitor. Try and figure out how and why.

You also need to understand what’s being said about you and your competitors online. What do people like? Why do they do business with one company instead of another? You need to have this information and use it in your business.

It’s just as important to look at the complaints online, too. What don’t people like about your competitors? What don’t they like about you? You should eliminate all the reasons for these complaints from your company.

This also is a great place to find ideas for improvement and new features to add to your service. For example, if your business is a luxury hair salon and consumers are complaining that luxury hair salons do not offer head massages, you can include head massages with your service for an additional price. This will help you drive up overall profitability while serving a market need that you didn’t even know existed.

Phase 5: Discover Buyer Friction and Skepticism

A good copywriter always does their research. One of the key pieces of information to understand is what friction a buyer faces and what stops them from buying? What is the buyer skeptical about?

These issues are then acknowledged in the sales copy to remove buyer friction and overcome any skepticism.

discover website buyer friction

For example, there is a very famous Ogilvy stock and bond ad. David Ogilvy understood that common buyer friction was caused by the belief that stocks and bonds is a complicated business.

some plain talk

He overcame the skepticism at the very beginning of the ad with this text:

“Some plain talk about a simple business that often sounds complicated.” This ad went on to build Merrill Lynch.

Although it sounds simple, understanding buyer friction online can be difficult. The key process is that you must obtain significant feedback from your visitors. Then, you have to use the information you collect to build your website and sales copy so that it naturally overcomes visitor skepticism.

You can use Qualaroo, SurveyMonkey, and Kampyle to get visitor feedback on your website, without intruding too much on your visitor experience.

Phase 6: Shine as Brightly as You Can

There are always hidden “proof” assets within a company that they never show to their visitors. You need to find these and make sure your visitors know about them. You’re basically showing proof that you are:

  • An authority
  • Credible
  • Expert in your industry
  • Trusted and Respected

You can do this by showing publications you’ve appeared in and awards you’ve won, naming previous well known clients (with their permission of course), showing case studies of your results, using testimonials and reviews, etc.

Many people think this is about additional credibility. However, it’s more about the extra believability you can provide.

As an example, a chiropractor client had a basic, standard website. He did not mention that he was one of just nine chiropractic craniopaths in the country. He also did not mention that it takes one year longer to become a chiropractic craniopath in his country than it does to become a physician.

These facts instantly make the chiropractor look like more of an expert, yet he was failing to pass along the information.

These are just a few ways you can prove you are a good company to do business with.

Another common mistake is not showcasing other products you sell. For example, one client sold a solution to prevent industrial heating systems from becoming clogged. They also sold a product to remove any current buildup inside the heating system, but hardly any of their clients knew about it.

When they added the “buildup remover” to their sales system and told buyers about it, the additional sales almost doubled their profits on that specific product line.

If you’re going to shine brightly and sell everything you can, you need to make sure your customers are fully educated about who you are and what you sell.

Phase 7: Focus on Roars

At this stage in the system, you should have a whole lot of ideas that were generated from the previous research steps.

But, before we design our new webpages, we need to be careful not to make the mistake of focusing on “squeaks” in order to try to produce the fastest wins. We should focus on “roars.”

The Squeaks

A squeak is an element on a webpage that can boost conversion rates by only small amounts. For example, changing the background color of your website will produce only small incremental improvements.

We don’t want minor improvements. We want big wins.

The great Gary Halbert once noted that the addition of extra order forms in his direct mail packages boosted response rate. Then, adding another order form also slightly lifted responses again, and so on. But, these are minor wins. They are squeaks.

If you have a 1% response rate and are looking to break even on your front end with a 2% response rate, adding in additional order forms won’t produce the big win you need (besides, this has been tested by us and has never seemed to increase results).

So, don’t aim for minor wins. Stay away from the squeaks.

The Roars

A roar is the opposite of a squeak. A roar is a specific factor on a webpage that has the potential to boost conversion rates by 100% or more.

For example, switching to a higher quality traffic source, drastically changing the headline of a page, adding upsells/continuity offers, giving premium bonus gifts, and switching the offer to a free trial are roars that can make big differences in your response rates.

split testing a roar

In one case, an educational item was being sold for $139. We introduced a split test where the item was sold at $139 as the control, and at $159 and at $169.

The response on the $139 and $159 prices was exactly the same. So, the new $159 sales price allowed us to make an extra $20 (14%) on every sale.

(The $169 price cut response in half. So, adding an extra $10 to the $159 sales price meant that we lost 50% of all orders, which, in turn, meant a lot less profit overall.)

It’s important to make sure that any changes you make contain as many roars and “big picture” changes as possible.

This way, you’ll be running real split tests that mean something. You will see faster improvements much more quickly, and most of your tests will reach statistical significance, too, which is much more rewarding for you.

Another thing to remember is to focus on simple, fast, and cost free changes. Don’t switch to an entirely new CMS that will take a long time to implement and cost a lot of money. Start with something simpler.

Look to your previous research as well. For example, your research may uncover that a certain need was not being met by your current products. You can introduce an existing product that meets that need into the sales funnel and probably see drastic increases in conversion.

Phase 8: Think Outside the Box

When running your split tests, try hard to focus outside the box of ideas that first come to your mind. Definitely look at non-page factors you can play with. (Phase 3 will help you with this, too.)

As an example, in a mail campaign, sales response was boosted by 50%+ without making any changes to the actual sales letter. This is a difference of between 100 sales and more than 150 sales. That is huge, considering the exact same sales copy was used in both letters. (The envelope was changed.)

Thinking like this will put you light years ahead of your competitors. It’s something lots of split testers don’t do, so it will give you the competitive edge you need when trying to stay ahead of the pack.

In business, all these advantages add up.

Phase 9: Design Your New Webpage

At this stage, we’ve done our research and eliminated all the assumptions. Now, it’s finally time to design the new webpage we’ll be testing.

If you’ve stuck to being methodological so far, the chances of success are high. You also shouldn’t have any problems designing the new page, because the previous research steps will tell you exactly what is needed on the new page.


As you can see, this system is a well-refined process. When setting up any split test, don’t be afraid to aim high. Try to avoid starting out with a list of split tests. Don’t jump in and start tweaking button colors when they may not mean a thing.

Instead, use the process above, and it will help you steamroll the competition if you’re ever going head to head with a tactical split tester (which will be more often than not).

Good luck!

About The Author: Michael Maven is an author, speaker, coach, and business growth expert. In addition to creating systems and founding businesses, he has grown sales for Amazon, IBM, eBay, and See detailed case studies at the Carter & Kingsley website.

  1. Hey Michael, great article! Thanks for writing it.

  2. Michael Maven Apr 08, 2015 at 5:21 pm

    Hey Neil, thanks very much! I’m glad I could contribute to the KISSmetrics blog and have had such a good response to the material.

    If anyone has further questions I’d be happy to answer them here.


Please use your real name and a corresponding social media profile when commenting. Otherwise, your comment may be deleted.

← Previous ArticleNext Article →