And optimizing your website means your online marketing efforts, like pay per click (PPC) and social media marketing, will be more profitable because your clicks will convert much more frequently.

But, unfortunately, you may have found that testing and optimizing your website aren’t as easy as you thought, and you likely aren’t getting the big conversion rate lifts expected, either. Often, one of the key reasons for this is a lack of website testing buy-in and budget from senior executives.

These influential people sometimes are referred to as HiPPOs (highest paid person’s opinion); and, in many instances, they believe they know what is best for their website. Therefore, they don’t feel the need for (or understand the benefits of) running website tests.

Further, these HiPPOs also usually hold the keys to the ample budget you need for two testing essentials. First, you need some funds to use a good website testing tool (Visual Website Optimizer, at the very least). Second, you need to hire dedicated testing resources because using a web analyst or online marketer is inefficient and causes bottlenecks. We all know what happens when you make an employee wear too many hats! Without a good budget for either of these, it severely impacts the efficiency and potential results of your website testing efforts.

So, if you manage to tame your HiPPOs and grow your testing buy-in and budget, this may result in much higher conversion rates and greater revenue from your website tests. To help you do this, here are 7 strategies and tips for you to consider. These are broken down into two key areas: education and peer pressure. Let’s get started!

Educate Them

1: Prove your competitors are testing their websites.

One of the simplest ways to educate your HiPPOs is to show them that your competitors are getting great results from testing their websites. Senior executives certainly won’t like knowing that a competitor is doing something new and cool that they aren’t (much like keeping up with the Joneses). There are two ways to do this:

• Find out which competitors are running website testing using a free debugger tool like WASP and check to see if they are using a website testing tool.
• Look on testing tool vendor websites to find case studies about your competitors’ testing efforts. (Maxymiser and Visual Website Optimizer have great sections for this in particular.) These highlight the return on investment (ROI) and great results from testing (more on this later) and act as great education ammo!

2: Create a presentation to demonstrate the impact of testing on revenue.

You wouldn’t invest in something if you didn’t think it was going to get great results, would you? No. And neither would your senior executives. Accordingly, you should create a presentation outlining why understanding and investing in testing is a great idea for ROI. Here are some tips for doing this:

• Research the potential by using your web analytics data to find problematic pages with high bounce and exit rates.
• Show them mockups of elements that could be improved on problematic pages to increase conversion rates, along with projected conversion lifts.
• Translate conversion rates into revenue! Senior executives probably won’t care much about conversion rates in isolation. Turn this into something they really care about – the impact on revenue! A 2% increase in conversion rates sounds pretty boring and low; but if you translate it into revenue, it often turns into huge increases in online revenue (music to their ears).
• Calculate and show them the projected overall ROI from running website testing for just 6 months, and include how long it would take to break even from the increased budget you are seeking.
• If they don’t listen to you, you could try using a third-party testing expert to do this for you. With the added credibility of an external consultant’s opinion, your HiPPOs may be more likely to agree than if you present it as a solo and internal undertaking.

3: Learn better ways to communicate with them and deal with office politics.

Bosses can be a real pain to work with sometimes, particularly if they are opinionated HiPPOs. To help you get them on your testing wavelength, it’s important to learn how to deal with office politics and to influence them better. There are a few great books that I recommend for this specifically:

• Lead Your Boss by John Baldoni
• Secrets to Winning at Office Politics by Marie McIntyre
• How to Win Friends and Influence People by Dale Carnegie
• Influence: The Psychology of Persuasion by Robert Cialdini

4: Ask to borrow from your search engine marketing (SEM) budget to prove results.

Often your search engine optimization (SEO) or PPC team will have a very large budget to spend. So a great way to suggest an increase in the testing budget is to ask to “borrow” just 10% of their budget for a short term. To help you explain this disparity in budget, you should tell them that a recent survey showed that, for every $92 spent driving traffic to websites, only $1 is spent converting visitors once they are on the sites.

Before you do this, get the head of SEM on your side by explaining what you are trying to do. Be sure they understand that the purpose is to make the company more profitable, which is better for everyone. Then get them to help you convince your HiPPOs of this plan.

Obtaining just 10% of their budget for 6 months gives you enough time to get a few big testing wins under your belt and report back with clear metrics on the gains and ROI realized from the investment, thereby proving the need for an increase in the testing-related budget.

Peer Pressure Them

5: Find an executive sponsor to help.

One of the most influential forms of peer pressure comes from someone at a similar executive level. Your HiPPOs are going to be much more willing to listen to and value input from their level. If you can find an executive sponsor to help champion the benefits and get testing pushed through, it will be much easier to gain buy-in from your HiPPOs. Figure out who in your organization might make an ideal candidate for this sponsor responsibility and discuss your plans with them over lunch. Get them excited by pointing out that, ultimately, it will make them look great, too!

6: Gain buy-in from key department heads to help.

In addition to an executive sponsor, it’s important to gain additional help and pressure from key department heads. Present to them the benefits of testing and optimization and the impact it can have on your business. (Higher revenue often means more bonuses for them, too!)

It’s particularly important to get this buy-in from department heads that play a key role in optimization; for example, web design and IT. This will make it even easier for you to test effectively and efficiently.

7: Find other testing evangelists internally to help.

Finally, to help with your HiPPO peer pressure, you need to find testing evangelists at lower levels of your organization, too. You often will find sympathetic web designers or web developers who understand the real benefits of testing. So keep your ear to the ground to find out about potential evangelists. Then offer to take them out to lunch and explain what you are trying to do and that you need to get everyone talking about the benefits of testing.

If you are successful with this last step of peer pressure and evangelism, pretty soon a testing culture will grow rapidly in your organization, and your HiPPO will be more than willing to relent and offer their buy-in!

If you start using these education and peer pressure methods with great success, it won’t be long before your HiPPOs turn into mice that are much more willing to increase their budget and buy-in for your testing efforts. This means you will be able to be much more influential and successful with your website testing efforts (and, hopefully, gain a bigger bonus for you, too!)

About the Author: Rich Page is a passionate website testing and conversion rate expert and the author of Website Optimization: An Hour a Day and co-author of the 2nd edition of Landing Page Optimization with Tim Ash. He is a consultant available for hire to help improve conversion rates for all types of online businesses.

And it sounds like the salvation you’ve been dreaming of. What’s not to like? Being able to test two different pages on your site to see which one gives you the most customers sounds amazing. After running these things a few times, those extra customers sure will add up!

You get everything set up and…

…realize you don’t know what to test first. And the A/B testing euphoria starts to wear off.

Right before you’re about to give up, you remember an argument you had with your designer (or was it the boss? Or the developer?) about the button color on one of your pages. They decided to go with green but you KNOW that orange is the better option. You’re so confident in your choice that you feel it in your BONES.

You decide to settle the matter once and for all.

The current button on your site looks like this:

And you test it against this stud of a button:

Everything’s set up, the test is running, it’s only a matter of time.

A week or two later, you log back into the account for your A/B testing tool, go to the test, scroll straight to the results…

The green button out-performed the orange by 0.1%. What a letdown. You don’t know what’s worse:

• The orange button didn’t perform better than the green like you predicted
• The difference was so minor that it wasn’t worth worrying about in the first place.

This happens ALL the time. Especially for people just getting started with A/B testing. You see, randomly picking elements to test produces lackluster results. The needle doesn’t move at all.

Right now, there are several big-win tests you could run that would make a HUGE difference to the growth of your company. Like 5-20% in customer growth kind of huge. These optimizations are just sitting there, waiting for you to start running A/B tests.

Finding big-wins doesn’t happen by accident.

Here’s the thing: if you A/B test all sorts of random stuff, you’ll never find these big-wins.

To find them, we’ll need a completely different process for deciding which tests to run. There’s a time and a place for testing everything we can think of (I’ll get to this in a second) but the big-wins require a completely different approach to testing.

Stage 1: Finding the Big Wins

If you’ve never tested before, you’ll find several 5-20% increases to your bottom line. I’m not talking about increases to some random conversion rate. This is an increase to your revenue and customer base. Finding a few 10% improvements to your revenue will take your business to a completely new level.

The best part? These are usually permanent increases to your customer growth. Make a single change to your business and reap the rewards for years to come.

But randomly testing all sorts of stuff on your site won’t find these big wins for you.

We need some guidance on where to start looking.

If you’re not an optimization pro that throws out A/B tests like candy, this is the process you want to use in order to get moving quickly.

To find the big wins with A/B testing, follow these steps:

1. Get qualitative insights (customer feedback)
2. Predict how to improve
3. Confirm the prediction with an A/B test

Let’s work through each of them:

1. Get Qualitative Insights

Qualitative data does a great job with alerting us to problems. More importantly, it helps us learn the WHY behind the WHAT. Using analytics, you’ll see where you customers bail, which features they use, and who your most profitable customers are. But to understand why your customers are doing what they’re doing, you need to go talk to them. At KISSmetrics, here are our 5 favorite ways to get customer feedback:

1. Surveys
2. Feedback Boxes
3. Reach Out Directly
4. User Activity From Your Analytics
5. Usability Tests

To find the big-win optimizations, we want to continue to look for trends in the feedback we’re receiving while diving deeper into issues we think are stirring up trouble.

Let’s say you’ve been looking at your funnel and you notice that only a few people upgrade to a paid plan or purchase your product. You have a TON of people clicking on your offers but as soon as they see the price, they bail.

This is where we want to get surgical with our qualitative data.

Throw up a one-question survey on the purchase page asking people if they have any questions about the product. You could also include a support button to collect feedback. And reach out to customers that HAVE purchased and ask them why they chose to become a customer. Once you’ve gotten feedback from 15-20 people, I bet you’ll be able to find a trend in the responses. Maybe you’ve oversold your offer in your marketing. Or maybe you haven’t addressed a critical objection in your copy.

Here’s the main take-away: qualitative data helps us understand which elements will have the biggest impact when running A/B tests.

Set up you customer feedback systems so you can easily spot emerging trends. And when one pops up, dive deeper so you know exactly what’s going on.

2. Predict How to Improve

This step is pretty straight-forward. You’ve already collected a bunch of qualitative data on how your customers are behaving. And you know WHY they’re behaving that way.

So it’s time to brainstorm some solutions to your problem.

Remember, this is a “prediction.” It’s just a hypothesis. It might work, it might not. And we won’t know until we get data to back it up.

3. Confirm Your Prediction With an A/B Test

Notice how the actual test comes at the END of this process, not at the beginning? By using qualitative data to help us understand what changes could be the most important, we’re setting ourselves up to find big wins with these tests.

Now it’s just a matter of testing to see if you’re right. You need to confirm that people will BEHAVE the same way they SAY they will (usually, they don’t). So get your hands on some data and run that A/B test.

Focus on finding your big wins and solving the major problems that you find from customer feedback. You can then make a huge impact on the growth of your business with a small amount of work.

But these big wins will run out. Before long, You’ll find the ideal funnel to acquire customers, the best pricing structure, and the most persuasive messaging. And if want to keep going, you’ll have to jump into Stage 2.

Stage 2: Get Methodical and Chase the Small Wins

Most improvements from A/B testing are small wins. Each one doesn’t amount to much. But if you can find dozens or hundreds of these things, you can double your growth several times.

While it might be easy to find a small win here and there, it’s not easy to crank these out week after week. You’re going to have to commit a lot of resources to this process.

Ideally, you’ll have a team of several people that can handle the marketing, development, and design of everything. If your conversion optimization team has to constantly fight for help from the engineering or design teams, they’ll never move fast enough to make a significant impact.

Give them the resources they need to test rapidly. Speed is the name of the game.

Also, your company needs to have enough data to work with. Typically, this means you need to be acquiring hundreds of new customers every month (thousands is even better). If you’re a smaller company, you could assign a single person to this task. Just remember that it’ll take a lot longer before you find enough small wins to make a difference.

Even during Stage 2, we’re not testing stuff randomly. Instead, we’re testing EVERYTHING. Brainstorm a list of changes and start running them back-to-back. Your testing pace should be relentless.

Some Joe Schmoe blogger swears that a photo of a person looking at your headline gives great results? TEST IT. Your best friend at the hottest startup in town says a video on the homepage makes money rain from the sky? TEST IT. You just found a list of 50 elements everyone should test from an A/B testing company? TEST IT ALL.

To decide which tests to run first, throw them into a list. It really doesn’t matter which order they go in, start at the top and work your way down as fast as you can. Some will make a difference, most won’t. And there’s really no way to know beforehand. So pick fast and get moving.

Here’s a list of tests to get you started:

• Headlines
• Remove or add steps to your funnel
• Social proof (testimonials, customer logos, etc.)
• Calls to action
• Copy
• Long-form vs. short-form
• Layout
• Add and remove elements from a page
• Pricing (changing your pricing structure usually gives you a big win but smaller tests like $37 vs. $39 can also help you grow)
• Purchase/signup bonuses
• Up-sells and cross-sells

The Common Pitfalls

Many people also run into a number of pitfalls when they start A/B testing. Here’s how to dodge them:

Get Statistical Significance

When you start getting results for your test, the data is completely random. It might LOOK like version A does better but in the long run, version B is the real winner.

Flipping a coin works the same way. It’s entirely possible to get four heads in a row. But that doesn’t mean you’ll always get heads. The 50/50 split only shows up if you do hundreds and thousands of flips. Even then, it can get slanted one way or the other.

We minimize (but we can never eliminate) the odds of getting a bad result by collecting lots data.

So how do we know that we have enough data?

Statistical significance measures how confident we are that we’ll get the same results if we repeat the test. And we measure that confidence with a percentage.

If we say that we have a 90% confidence level, that means we’ll get the same results 9 out of 10 times. At 99% confidence, 99 out of 100 tests will give us the same results. Getting more data will steadily increase your confidence level and help you hit statistical significance.

Most people say they have statistical significance when they hit the 95% confidence level. That’s when they pick the winner and move to the next test. Don’t worry about trying to get to 99% confidence. It usually takes too long to get enough data. You’ll grow your business faster by picking the winner and focusing on the next A/B test as soon as you hit the 95% mark.

But remember, the 95% mark is arbitrary. So if you’ve got a test that’s sitting at 87% or 93% confidence and you have other tests in the pipeline, it’s okay to pick a winner and move on. Balance speed with data and don’t sacrifice one for the other.

Visual Website Optimizer has built an Excel sheet that does all the fancy math for you,you can download it here. Just plug in the results from your split test and it’ll tell you if you’ve hit statistical significance. If you haven’t, keep your test running until you do.

The “New” Effect

Introducing something new to your site can impact your conversion rate just because it’s new. This is the “new” effect. What this means is that the new version could out-perform your old version initially. But over time, the performance difference can shrink or even flip. The new version might perform better this week but over the long term, your old version might be the best choice.

And in some cases, “new” will negatively impact conversion rates. This happens when you introduce changes that interrupt the habits of your users.

Let’s say you’ve used the same navigation for a while. Even if you test a version that’s truly better, conversion rates will likely drop in the short term. Once people become familiar with your site, they don’t actively look through the navigation each time. Whenever they need something, they know right where to click. And anything that interrupts that habit will slow them down (at least in the short term). Tests that interrupt established habits will perform worse in the short term.

So how do we deal with these pitfalls? Even if you have a massive amount of data to work with and can establish statistical significance fairly quickly, give yourself more time if you suspect “newness” might be impacting the results. A couple of weeks will do it. And if time is of the essence, look at the trend lines of your conversion rates. Are they getting closer to each other? If they are, you might be looking at the “new” effect. If they look stable and you have a solid week’s worth of data, you’re good to go.

Track Your Entire Funnel

When testing improvements on how you acquire customers with your marketing funnel (making a change to your homepage falls into this group), be careful about only tracking the conversion rate for the next step. On a regular basis, you’ll find something that increases the next-step conversion but LOWERS the conversion rate for the entire funnel. So if you’re not tracking your A/B tests through the entire funnel, you might slow your customer growth by accident.

You’ll need customer analytics to do this. Regular A/B testing tools like Optimizely and Visual Website Optimizer only track the next step.

KISSmetrics has integrations with Optimizely and Visual Website Optimizer to help you avoid this trap.

But there is ONE exception to this. If you’re just getting started and don’t have much data to work with, you might only be able to test changes at the top of your funnel. You won’t have enough data to go any further.

For example, you might have enough traffic to measure free-trial sign ups from your home page but not enough data to track which version gives you more paid subscribers. If you spot a potential roadblock at the top of your funnel, don’t let a lack of data at the bottom get in your way of trying to fix it.

Data is Always Changing

What’s true today may not be true tomorrow. Market’s shift, customer needs change, your business model evolves.

So if you run a test today, you might get completely different results 6 months from now. When you find the “perfect” version, it won’t stay perfect. Everything has a half-life and the only way to stay on top is to periodically refresh by running another batch of tests.

What about seasonality?

Every business experiences fluctuations throughout the year, some more than others. In some cases, seasonality is blatantly obvious. Apple’s best quarter is the holiday season. Toy companies also bring home the majority of their revenue during November and December.

But seasonality can impact our results in more subtle ways.

Take the B2B market for example. Doesn’t seem like a candidate for seasonality right? Well, I reached out to several of our KISSmetrics customers to get feedback on a new feature we were building. Usually, I get a 75-90% response rate. But last August, only 1 out 10 replied. I was shocked. Was it the copy in my emails? What did I say to get such bad results?

It had nothing to do with my emails, everyone was on summer vacation. Over the next 2 weeks, just about all of them got back to me once they returned to the office.

The same thing can happen to your tests too, regardless of your industry.

But don’t use seasonality as an excuse. We all love taking credit when things go well but avoid it like a deathly plague as soon as things aren’t so rosy. Don’t blame seasonality unless you have strong evidence to back it up. Just keep an eye out for it.

Multivariate Tests and Other Hoopla

If you spend much time in the conversion optimization space, you’ll hear about these fancy schmancy things called multivariate tests. Basically, they let you test dozens of variables all at the same time to find the ideal landing page, home page, checkout page, etc.

Sounds great right? Here’s the rub: you need a MASSIVE amount of data before these become a viable option.

They also take a ton of time to set up and manage. Until you become an A/B testing pro and have an entire team that can hunt for optimizations around the clock, multivariate tests just aren’t worth the effort.

There are also other testing algorithms that get WAY more complicated than what most people need. Things like this will just get in the way because you’ll spend too much time trying to get started. Don’t sacrifice action for complexity.

Bottom Line

If you’ve never run an A/B test before, you’re in Stage 1 and there are several optimizations that will grow your company by 5-20%. But to find them, you can’t just test random stuff.

Instead, use this 3-step A/B testing process:

1. Use customer feedback to get your hands on qualitative data
2. Predict which optimizations will make the biggest difference
3. Run an A/B test to see if you’re right

Once you’ve found your big-wins, it’s time to start Stage 2 of your optimization plan. At this point, dig in for the long-haul, build a growth team, and start hunting for every little conversion increase you can find.

To make a dent in your growth, you’ll need to find hundreds of these little guys over the course of a year. To support tests at such a high frequency, you’ll need to be acquiring hundreds of customers every month, preferably thousands.

When you get into the meat of your tests, don’t forget these common pitfalls:

1. Get statistical significance
2. Be careful of the “new” effect
3. Track your entire funnel
4. Your data will always be changing
5. Stay simple and don’t use things like multivariate tests unless you have a good reason

Happy testing!

About the Author: Lars Lofgren is the KISSmetrics Marketing Analyst and has his Google Analytics Individual Qualification (he’s certified). Learn how to grow your business at his marketing blog or follow him on Twitter @larslofgren.

This often leads to queries like: “How can I create an A/B test where one variation is a different button color on the old website and the other variation is our new website?”

This post is a step by step guide to solving that conundrum. I’ll show you how to ensure that your new design is a high converter even before it is launched, and how to use basic, cost effective research to understand what your current website is doing right.

Start with understanding your old website

The testing process should begin the day you decide to create a new website, so that you can use the insights generated from studying your current website while you’re making the new one. And one of the best ways to gather the required insight is by usability testing.

Usability testing (user testing) is a process to evaluate a product by testing it on real users. What you do is gather about 10 users for each of your customer segments and ask them to achieve certain goals on your website. While they are working through the tasks, you record their session and ask them to vocalize their thoughts.

Here’s how to set up a usability test:

1. Know your user and cut cleanly

Always start by thoroughly understanding your different users and segmenting them so that there is no overlap. Do remember that if the test participants are not completely representative of your visitor segments, you’re essentially collecting invalid data. In a worst case scenario, you’ll think you’re collecting good data and insights when it’s actually downright misleading.

2. Cost, Time, and Significant Sample Sizes

Collect about 10 to 15 participants that are representative of each segment. Many practitioners say that 5 participants from each segment are all you need, while some data analysts debunk that as bad testing methodology (not enough, they claim). However, the cost of significant sample size and the time required may be prohibitive for the average small business. Therefore, I’ve used the research presented here to arrive at “10 to 15 participants.”

3. An environment without any impressing

Set up the test environment so that the participant is not influenced in any way. Especially, please do not make the rookie mistake of standing over their heads. Studies show that in such cases, test participants attempt to “impress” by trying to complete tasks either too quickly or in ways they normally wouldn’t.

It is best if the participant is in their usual environment (home/office). This increases the probability that they won’t modify their behavior and will use your website as usual.

4. The wise old man said “Ask the right questions”

The key to asking the right questions is to not ask participants to complete specific tasks, but to solve problems they’re facing. For example, when testing Visual Website Optimizer, we wouldn’t say “Sign up for a trial.” Rather, the instruction would be “Start a test to optimize the headline on a client’s homepage” (prior permission obtained from the client, of course).

As you’ll see, the second instruction will involve a number of steps that will mean heavy usage of the tool. For us, this means that there’s much more to observe and learn from. The killer will be when, instead of diving in, she pauses and asks “But won’t that have negative SEO implications?” It’s a single question that smashes many of our assumptions and gives us far more insight into what goes on in the customer’s mind.

What to test?

Start out by clearly identifying the goals you want your website visitors to accomplish. These could be:

• Signing up for a free trial
• Signing up for a paid account
• Creating an account
• Making a purchase
• Signing up for a newsletter
• Completing a lead generation form
• Downloading a product

Next, think of customer problems that are solved by achieving one or more of these goals. Finally, ask test participants to solve the problems. Once you’ve got your problem sets down, you should look to generate clear quantitative data and qualitative insights from your test. Some of the usual ones are:

Quantitative

1. Number of clicks to solve the problem
2. Time taken to solve the problem
3. Number of page loads to solve the problem
4. Errors while attempting to solve the problem
5. Number of goals completed while solving the problem
6. Distractions that make participants leave the conversion funnel

Qualitative

1. What catches the user’s attention in the first 5/10/15 seconds after the landing page has loaded
2. How quickly the headline is able to communicate the business’s primary offering
3. How the user searches for information
4. How convinced the user is with and without trust signs/badges
5. Questions that the user generates while attempting to solve the problem
6. How pleasant the user finds the website

Based on these and the fact that the test participants are vocalizing their thoughts, you’ll understand a lot about what on your current website is working and what isn’t.

Important:

1. Pay special attention to the words that test participants use to describe their experiences. These are the keywords you might want to try when A/B testing website copy.
2. Keep a close look out for moments when the participant is confused, irritated, or spending too much time looking for a specific bit of information.

Create your new website

The insights generated from your first usability test will give you an excellent guide on which issues to address and which ones to replicate in your new website. The pain points can be marked for improvement or as inputs for an A/B test. The parts that are working for the user should be replicated or benchmarked against (for example, number of steps it takes to complete a goal).

Usability test the new website design

Get some proxy customers on your new design and understand exactly what’s happening. What are the roadblocks, what are the irritations, and where are they getting confused? After this, you’ll be left with a good list of elements for your A/B tests. That happens because some things just can’t be usability tested. No visitor will be able to tell you that she’s more likely to submit a form if the button is colored green instead of blue. Such elements are prime candidates for an A/B test.

Launch

By now, you should have no problem with launching your new design. You’ve ironed out whatever kinks were reported by all the testing and will almost certainly have a list of elements (button colors/placement, headline copy, trust badges, placement of links, etc.) that you want to A/B test. This is the time to go ahead and launch your new website.

Initially, regular visitors might react negatively or positively (negatively because you’ve changed their usual interaction flow or positively because they might find the change refreshing). Don’t panic or celebrate just yet. Wait for the initial reactions to settle down. Then, once visitors have learned their way across the changed UI (user interface), measure the difference in key conversion rates.

Start A/B testing

This is when you get out that list of elements you want to A/B test and go at them one by one. Depending on your traffic, it might take you a few weeks or months to go through the entire list; at the end of which, you should have a very high converting website.

The tools you’ll need

The market for usability testing tools abounds with both free and paid options. A really good comparison is the Smashing Magazine article entitled “Comprehensive Review Of Usability And User Experience Testing Tools.”

Important do’s and don’ts

Some basic principles that need to be reiterated:

• Don’t ask the test participants to directly complete a goal. Instead, ask them to solve a problem your real users might face that involves completing the goal(s). For example, instead of the goal being “Sign up for the newsletter,” the goal should be “See how you can regularly keep up with all the interesting tips and tricks this website releases.” It’s quite likely that the user may start looking for your RSS feed button or head directly to your Twitter/Facebook profile.
• This one is fairly simple, but it still needs to be emphasized: let the test participant do all the talking.
• Ensure that participants understand and feel that it is the website being tested and not them. They should feel no pressure to display their prowess and/or their ability to get things done.
• Make sure the results and insights generated are clearly communicated to the Marketing, Market Research – Insights, and IT departments.

At the end of this comprehensive process, you should be left with a new website that’s a high converter, achieves your business goals, and can be proudly shown off to your bosses. It’s a newer, better design, and it converts well, too. All you need to do now is further tweak it to make it sing!

Useful links

• Usability 101: Introduction to Usability – useit.com
• Seven Common Usability Testing Mistakes – uie.com
• Usability Testing Demystified – alistapart.com
• Usability testing hints, tips and guidelines – uxforthemasses.com
• The Ultimate Guide to A/B Testing – smashingmagazine.com
• 234 Tips and Tricks for Recruiting Users as Participants in Usability Studies – nngroup.com

Image Credit:

• Bargain Hunting Mamas: Get Satisfaction
• Dilbert Comic Strip: Dilbert.com
• Website Success Benchmark: AFX Design
• Rocket Launch: puliarfanita@Flickr

About the Author: Siddharth Deswal works at Visual Website Optimizer, the world’s easiest A/B testing software. He’s been involved with web development for about eight years and actively looks to help online businesses discover the value of Conversion Rate Optimization. He tweets about A/B testing, landing pages, and effective marketing tips on @wingify

With over 600,000 apps competing for sales in each app store, an app’s icon acts like a mini banner ad, so choosing the right one can mean the difference between success and failure.

But when you’re presented with 3 icon options, how can you be certain which one will perform the best?

If you drop your price from $1.99 to $0.99, will your sales double? Or triple? Or more?

Should your description start with your ranking “#3 Best Selling Zombie App!” or with a list of features? Which will convert better?

Many app marketers will do their research, compare competitors and best sellers and then answer these questions based on gut feeling and intuition.

The iTunes app store and Google Play don’t allow for proper, scientific A/B testing, so marketers are forced to make their choices this way.

How to “Micro Test” Your App

There is a better way to make these decisions. Let me introduce you to “Micro Testing,” a method that lets you test all of these marketing elements — your icon, price, description and more — and then apply your findings to the app store.

Although some of the bigger app development companies are already using similar methods, the practice is still relatively unknown in the app marketing community.

I’m going to explain here, step by step, how you can use free and inexpensive tools to create your own micro tests and optimize your app store marketing today.

Micro testing is based on this simple premise: We can’t perform A/B testing within the app store, so we’ll mimic the app store experience and run our experiments there.

How to Find the Perfect Icon

TapTapTap, a well known app development company, shared the story about how they picked the wrong icon for one of their apps, which lead to a disastrous launch.

During development they had 3 design concepts to explore. Each person on the team had a favorite. Everybody had their own beliefs about which icon would perform best.

Unfortunately, they had no clinical, objective facts on which to base a decision.

Everyone had their own opinions on what concepts they liked the best and what should be changed or tweaked. This lead to some pretty heated arguments.

The story goes on to explain how, in the absence of objective data, design-by-committee took hold and produced a generic, middle of the road icon that performed very poorly. This is where micro testing should come into play. To help you avoid making the same mistakes, we’ll show you how to create your first micro test in 3 steps:

1. Create an A/B Landing Page
2. Send Traffic to Test Your Pages
3. Analyze Your Results

This same process can be used for all the app’s marketing elements. We’ll use icons to illustrate the steps in detail, and then touch on pricing, description and screenshots.

Step 1: Create a Mobile Landing Page

A landing page is a single web page that will showcase your app with the same elements that a customer might see in the app store “an icon, description, screen shots and buy now button. Even if this is a new app, you should create your landing page as if the app is available to purchase, with a main Call-to-Action (i.e., a big button) saying Available on iTunes “Install Now”.



The goal here is to replicate the experience users will have when they view your app in the app store in order to test how many actually will try to buy it when presented with the different icons.

Landing Page Resources

You have 3 main options for building your mobile landing page:

1. Custom code it or hire a developer on elance or oDesk. This is the most time/cost intensive, but it gives you good flexibility.
2. Use a template from a site like Themeforest. They cost about $10, but this requires some basic coding knowledge.
3. Use a mobile landing page builder like Pijnz, Landr, Atmio, Convrrt or SparkPage.

In this guide, we won’t go into detail on how to modify html templates or upload them to your server; you should pick those options only if you know the basics of coding and have the time to invest.

Building a landing page

Set Up an A/B Test

Now that you have a landing page built, you need to create several variations of that page. Each variation will display a different icon, but the rest of the page content will remain exactly the same. In the screenshot below, you can see an example of 4 different landing pages used to test 4 slightly different logo styles.

A/B tests will randomly split traffic among these test pages to show what percentage of visitors click on “Get it on iTunes.”

If you’ve custom built your landing page using a template or html, you can use a service like Optimizely or VWO to create your variations.

Right now, Optimizely has mobile A/B testing in only their higher price plans; but if you do choose to use them, you can integrate directly with KISSmetrics to track your results.

Step 2: Find Testers for Your Test

Once your landing page is ready to test, you need to start sending people to it. You can do free promotion, such as sharing the link on Twitter or Facebook, but we recommend a small ad campaign for a reliable, non-biased flow of users to test your icons.

By running a small ad campaign, even as low as $50, you can send hundreds of visitors to your landing pages.

This is another reason it’s important to keep your variations to a minimum. Testing 2-4 icons is optimal; but for any more, you need to send larger and larger amounts of traffic to get significant results (which is fine, of course, if you have the budget).

You can use AdMob, Google Adwords or any other mobile ad network to run this small campaign.

You can use a network like AdMob for a quick and easy set up, but a more targeted network like Google Adwords can have its advantages, too. For example, if you’re testing the icon for your latest Zombie Game, you can target ads at people searching “Download Zombie Games” on Google, iPhone and Android. This will ensure that you test your icon with the right kinds of users.

Step 3: Get Results

As your ad campaign sends traffic to your page, you should begin to get a clear picture of which icon is resulting in the most clicks on the “download” button.

If you build the landing page with custom code or a template, you also will need to use a service like KISSmetrics or Optimizely to track the number of clicks on your “download” button. To set up tracking with KISSmetrics, you just need to create a new event, as described in this help page.

A sample micro test showing “Icon 1″ as the winner

You should run the experiment over a couple of days until you get at least 100 clicks on the download button to give you an accurate picture.

Although the test can’t make you 100% certain, you’ll quickly get a sense of which icons are resonating with users and which are turning them off.

Even small increases from one icon over another can have a huge effect. The extra sales they encourage can mean the difference between making it into the Top 10 chart or being noticed and featured by Apple.

Both of these would skyrocket your app’s sales. This is a much more scientific approach than relying on your gut or intuition to pick the best icon. This is how to set up your first micro test to help you pick the perfect icon for your app. Using the same method, we’ll now run through the other main elements you can test — pricing, description and screenshots.

What Is Your Most Profitable Price Point?

Here’s an example of a common conversation I have with app developers before they launch.

One person was torn between a $0.99 and a $1.99 price tag for their next app.

“I know,” the developer said, “that I’m more likely to buy an app if it’s $0.99 rather than $1.99.”

“But how much more likely?” I asked. “Twice as likely? Three times as likely? Or 50% more likely?”

“Hmmm… I don’t know!” was the response.

This is the key question you need to answer before setting a price. If you double your price from $0.99 to $1.99, will your sales fall by more or less than half?

I’ll illustrate this with an example:

If you sell 100 apps in a day at $0.99, you’ll earn $99. So what happens when you increase your price to $1.99?

• If sales drop to 70 per day, you will make $139
• If sales drop to 55 per day, you will make $109
• If sales drop below 50 (half) to 45, you’ll make only $89

If you double the price, how far will sales fall? Economists call the answer to this question “elasticity”, and knowing your app’s elasticity is vital for picking your most profitable price point.

As with icons, micro testing can be used to find the most profitable app price.

For testing your price points, you can follow the same steps as before, but make sure that you include the price as a prominent element on each page. All other elements should remain constant, so that you’re seeing only the effect of pricing.

Making Sense of Your Pricing Results

As your test is running, you should begin to get a sense of the conversion rate for each price point.

Although KISSmetrics, SparkPage and Optimizely will calculate this for you, here’s the simple formula if you need it:

(Number of Clicks / Number of Visitors) x 100 = Conversion Rate.

I’ll run through some example figures to demonstrate.

Let’s imagine you tested two price points, $0.99 and $1.99. Each landing page had 1,000 visitors.

Page A ($0.99) had 250 clicks on the download button. This gives us a conversion rate of 25%.

Page B ($1.99) had 150 clicks on the download button. This gives us a conversion rate of 15%.

But what does this mean for your overall revenue?

The simplest way to compare the two is to multiply the price by the conversion rate and see which is bigger.

• Page A: $0.99 x 25% = 24.7c
• Page B: $1.99 x 15% = 29.8c

This tells us that every visitor to Page A earns us 24.7c, but every visitor to Page B earns us 29.8c.

If an app marketer uses micro testing to pick Price B over Price A, as in this example, the marketer would stand to increase total revenue by 20.6%

(Note: This doesn’t show you what your conversion rate will be in reality; it just shows you the difference between price points. Use micro testing to pick the optimal price, but not to forecast sales.)

Discover the App Description that Sells

Of all the elements in an app store listing, the description still seems the least understood, even though it occupies most of the page’s real estate.

If you open an app store now and look through the top selling apps, you’ll see a wide variety of description styles and tactics. Some start with a marketing message and some with user reviews or worldwide rankings. Others list app features, and some start with the newest features in the most recent update.

There’s no clear “best practice” when it comes to descriptions, which can leave app marketers stumped.

Should it be a really long description to convince anybody who wants to read about the app?

Should it be short and sweet? Just enough to get them interested? What about bullet point feature lists — they seem popular?

Fear not; once more, data driven decision making will come to the rescue!

Running Your Description Micro Test

For this micro test, you can build your landing page as before, but make this one very light on imagery. We want to isolate and test the effect of descriptions on conversion rates, so excess imagery can distract from this goal.

Unlike the other experiments, you should place your Call-To-Action above the description text here. This may seem paradoxical at first — after all, you want to test clicks after people read — right?

Not quite! You want to assess what description style works best in the app store, so you want to mimic the iTunes store as closely as possible. In the iTunes store, the download CTA is above the description.

As with the other tests, you want to keep the number of variations relatively low so that you don’t have to spend too much sending traffic to each page.

Because of this, you might want to test your descriptions in two phases:

First, start with broad description styles. For example, start with rankings on one page, features on the next and a catchy tagline on the third.

Then drill down into the best style, and test different variations of content. For example, if a features list works best for your app, run a second round of testing to experiment with different features in the list, different list lengths, etc.

Find the Screenshots Users Love

At this stage in the guide, hopefully, you see a pattern forming!

Screenshots are often underrated when it comes to app marketing. They aren’t as prominent as the icon or the price, and they appear “below the fold” in both app stores.

But they can be a vital part of encouraging a user to purchase your app. They are the closest a user can get to trying out the app before downloading it. They are your window display.

A screenshots micro test operates much like the others, but with one key difference. As we want to replicate the app store experience as closely as possible, screenshots are best presented in a side scrolling list, where the user “swipes” through them one by one.

Other Uses and Caveats

Ideally, this guide has given you a good introduction to the concept of micro testing to optimize your app store presence.

If you try these tests yourself, there might be other elements you can experiment with or different ways to test the elements we described. If you do have any success or ideas for other tests, please leave a comment and let us know.

Finally, it is important to consider the results from these tests in the wider context of your app strategy.

For instance, testing could show that a higher price might result in higher profit, but you could still choose to keep the price low in order to build up your initial user base, if social sharing is a big part of your strategy.

Micro testing can be an incredibly powerful optimization tool. We trust that this guide will help you get more scientific with your decision making and that you will be able to promote your apps even more effectively.

About the Author: Peter Tanham is the co-founder of SparkPage, a platform for creating mobile-optimized landing pages. At the time of publishing SparkPage is in private beta, you can request an invite here.

What you need to understand before reading this article

The active word in this article is “Test.” The point of optimization is to find out what works best in your particular case and on your specific target audience. Since all products, offers, and companies are different – just like the motivations of potential customers are going to be different – there really is no “one size fits all” solution that works every time.

So use the tips in this article as inspiration – but make sure to test what works best on your landing page.

1. The placement of your call-to-action

The placement of your call-to-action has a major impact on conversions. And, “No,” above the fold is not automatically the best solution – despite what many best practice believers out there want you to think.

I’ve seen many cases where the CTA has performed best above the fold, just like I’ve seen cases where a CTA below the fold has gotten the best results. My general observation, from a wide range of landing page tests, is that there is a correlation between the complexity of the product/offer and the optimal placement of the CTA.

If the product/offer is complex, and the prospect has to digest a lot of information in order to make an informed decision, positioning the CTA lower on the page generally works best.

And vice versa, if the product/offer is very simple, and the prospect doesn’t have to do hardly any thinking in order to make an informed decision, positioning the CTA above the fold generally works best.

Here’s an example from a case study where a landing page treatment, with the CTA placed way under the fold, outperformed the control version (CTA above the fold) by 304%.

We are looking at a PPC (Pay-Per-Click) landing page that pitches a subscription service where busy families can get recipes and ingredients delivered to their doorstep 3-5 times a week. This is a relatively complex offer. It turned out that the version that waited until the very bottom of the page to ask for the conversion got 3 times as many signups as the one with the CTA above the fold.

(Read the full case study here: http://contentverve.com/how-moving-the-call-to-action-below-the-fold-generated-a-304-lift/)

What you should do now:

Experiment with different placements of your call-to-action and see which one performs best. Try different versions with the CTA either at the top, the middle, or the bottom of the page.

You also could try a combination where you place the CTA both at the top and the bottom. I’ve seen this combination work well on very long landing pages where potential customers end up scrolling all the way to the bottom of the page.

2. Your button copy

Button copy is just as important as the button itself, and even minor changes can significantly impact conversions.

Here’s an example from a case study where changing one word in the button copy increased conversions on a B2B landing page by 38.26%.

(Read the full case study here: http://contentverve.com/how-changing-1-word-in-the-call-to-action-generated-a-38-26-lift-in-conversions/)

Even though changing that one word represents a minor change on the page, it markedly changes the messaging. “Order” emphasizes what you have to do. Whereas “Get” emphasizes what you’re going to get – rather than what you have to do to get it. In other words, the treatment copy conveys value.

However, conveying value is not always enough. To get the best results, your button copy must also be relevant to the conversion scenario at hand.

Here’s an example from a case study where a button that conveyed only value significantly underperformed a button that did the job of conveying both value and relevance.

We’re looking at a landing page on a popular essay site. The page features a preview of an essay, and the goal is to get potential customers to click through to the signup page.

Although “Get Instant Access Now” conveys value (It could have said “Buy access”), it’s very generic compared to “Read Full Essay Now,” which is the real motivation for clicking the button and moving further down the conversion path.

What you should do now:

In my experience, the more value and relevance you can convey via your button copy, the more conversions you are likely to get. So review the button copy you are currently using on your landing page. If it’s something generic like “BUY NOW,” “CLICK HERE” or “SUBMIT,” work on a treatment that conveys more relevance and value to the potential customer in the specific conversion scenario at hand. Experiment with different variations until you find the one that works best on your landing page.

For more on writing calls-to-action that convert, check out this how-to video on high-impact button copy: http://contentverve.com/high-impact-button-copy-how-to-write-calls-to-action-that-convert/

3. The amount of information on your landing page

Marketers seem to be divided into two groups here: those who swear to short-form landing pages and those who swear to long-form landing pages.

But again, there is no “one size fits all” solution. Your landing page should contain as much information as your potential customers need in order to make an informed decision.

In my experience, long-form landing pages perform well in connection with complex offers that have a high level of related anxiety. On the other hand, short-form landing pages work well with simple offers that have a low level of related anxiety.

Here’s an example from a case study where a short landing page outperformed a longer version. In this case, we’re looking at a PPC landing page where the goal is to get potential customers to buy a gym membership.

That’s a pretty simple offer with very little anxiety involved in the decision-making process, and the version with the smallest amount of information had the highest conversion rate.

In this next example, a long landing page outperformed a shorter version significantly. Here we’re looking at a PPC landing page where the goal is to get potential clients to sign up for a home energy audit.

A home energy audit is a rather complex offer. Moreover, there’s quite a bit of anxiety related to having someone come to your house and do an audit of your home that could lead to a large investment in insulation.

What you should do now:

Review your landing page and ask yourself these questions:

“What do my prospects need to know in order to accept my offer?”

“Is there any important information that’s critical to the offer – special circumstances they need to be aware of or maybe an incentive that will make them act now?”

“Are there any specific points of anxiety or friction that we need to mitigate?”

If your landing page is very short, consider whether there’s any important information you’ve left out. But, if your landing page is very long, consider whether there’s any information that is not necessary.

Try experimenting with different amounts of information until you find the fit for your offer.

Time to start testing!

All right – now that you know what to experiment with, it’s time to start testing, so you can find out what works best on your particular landing page and on your specific target audience. If you have any questions, simply post them in the comments section, and I’ll be happy to answer.

About the Author: Michael Lykke Aagaard, is a self-employed, self-confessed split test junkie and landing page fanatic who’s obsessed with finding out what really works in online marketing. He’s Danish and hails from the fair city of Copenhagen. You can follow him on Twitter, or check out more of his work on his newly launched international blog ContentVerve.com.

You see, there’s a huge benefit to understanding how most people use A/B testing. It helps us benchmark how well we’re doing with our own business. It also shows us what to expect from our competitors. And even if one competitor is ahead of the curve, the rest of your industry probably isn’t.

For our next webinar, we’ll be showing you current data on questions like these:

• How much extra revenue do companies get from their A/B testing?
• How often do they test?
• Are the conversion rate increases usually worth it?

But where are we getting our data?

I just finished building a survey that will give you all sorts of great insights into these questions. We’re spreading this survey far and wide in our quest to find out how people REALLY use A/B testing (instead of what they SAY they do in order to sound super smart on their blogs).

To get high-quality data, we’re putting the survey right before the webinar registration. So those of you that complete the survey will get access to the webinar and be able to reserve your spot.

It gets even better…

During the webinar, we’re going to go through the results. Once we know where you are, I’m going to customize the entire webinar to help you get to the next level. By the end of the webinar, you’ll know how to get the best return from your A/B testing efforts with the least amount of work. I’m going to put together every tactic, process, and insight that matches your needs. You’ll get my whole bag of tricks.

By filling out the survey, you’ll tell me what content will help you the most. I’m going to build the entire webinar to solve the problems you’re going through right now.

And this isn’t some 50-question soul-sucking survey of doom. It’s 11 questions. And 8 of those questions are multiple choice. It will take you 4 minutes to complete. Seriously. 4 minutes.

Here’s how this is going to work:

1. Complete this survey
2. At the end of the survey click the “Done” button ←–this is super important
3. Register for the webinar

I’m going to close the survey on Tuesday, August 21 at Noon PDT. So fill it out while there’s still time!

Once you’ve registered for the webinar, you’ll get an email with all the juicy info on how to join next week.

In case you’re wondering, you HAVE to take the survey to get access to the webinar.

When was the webinar?

Slides: http://www.slideshare.net/kissmetrics/kissmetrics-summer-2012-state-of-ab-testing
Presenter: Lars Lofgren
Format: An amazing journey into the world of A/B Testing with Q&A at the end.
Day: Friday, August 24th 2012
Time: 10 AM PDT – GMT-08:00 Pacific Time (US and Canada);Tijuana

How do I sign up?

To take the survey, click here on the button below:

Or just click on the link below:

https://www.surveymonkey.com/s/2S8QG3J

If you have any questions please contact Sean Work or Lars Lofgren for information.

Here are the 5 places to get qualitative data that we’ll cover:

1. Surveys
2. Feedback
3. Reaching Out Directly
4. User Activity from Your Analytics
5. User Testing

Then we will show you how to integrate your A/B testing tool so that you can track how your tests perform over the long-term (instead of just the next click).

Finally, our compatriots over at Optimizely are going to give you their best tips for how to apply all of this to your business.

This is what they will cover:

1. How most people A/B test and why it’s ineffective
2. The 5 places to get insights to direct your A/B testing
3. How to integrate KISSmetrics into Optimizely to get this all to work flawlessly

When is it?

Presenter: Lars Lofgren and Jodie Ellis (Director of Marketing at Optimizely)
Format: A wonderful journey into the world of A/B Testing with Q&A at the end.
Day: Wednesday, August 8th 2012
Time: 10 AM PDT – GMT-08:00 Pacific Time (US and Canada);Tijuana

How do I sign up?

To register, click here on the button below:

Important!

Be sure to update Java on your computers to ensure that the webinar software runs well for you.

Sneak Peeks

First, we’ll cover the most tested item of all time: button colors!

But you’re wasting your time. Instead of testing random elements on your site like button colors, you really need to start qualitative testing.

Then we’ll give you a step-by-step process for how to move the needle with A/B tests instead of merely making that needle twitch.

So please join us next Friday for this truly insightful webinar – you can’t afford to miss it!

Click on the infographic below to view a larger image:

Want to display this infographic on your site?

Sources

• Vertster – 3 Tips For Writing Better Headlines
• Unbounce – The Anatomy of a Landing Page
• Emagine – 6 Ways to Make Your Landing Pages Convert
• Copyblogger – 5 Landing Page Mistakes that Crush Conversion Rates
• Search Engine People – Marketing 101: Sell Benefits and Not Features
• Think Vitamin – Human Photos Double your Conversion Rate
• ConversionXL – How Images Can Boost Your Conversion Rate

Jonathan Kay from Grasshopper Virtual Phone Systems, proposed the original concept of this challenge. He believed that:

“People feel more comfortable with brands that they can put a face behind. Even though you might purchase a product exclusively online, having a phone number on your site and the ability to talk to a real person (who cares) in turn makes you feel more comfortable taking out your wallet (or recommending someone else to) for this brand.”

TheFlowr.com Home Page Variants

Before we get into the results of this simple A/B test, let’s quickly look at the differences between the two home pages.

The image below is the original Flowr home page. If you look closely, you will see that there is no phone number on the page.

In the next image, you will see a screenshot of the home page variant with a phone number and the call to action “Want to have a chat? Call us at..” (look for the red asterisk).

The Results

Flowr ran a simple A/B test with one home page variation using KISSmetrics. Again, the only difference in the variation home page was the addition of a small phone number and some supporting call-to-action text. The results were as follows:

Test Conditions

• Test Duration: From September 9, 2011 to October 24, 2011. (approx. 6 weeks)
• Test Item: Website home page of Flowr vs. home page variation.
• Test Type: A/B Test (only difference between variation was a phone number and associated call-to-action).
• Test Goal: Increase software sign ups from home page.

Results

• 53.96% of sign-ups originated from the home page variation with the phone number.
• 46.04% of sign-ups originated from the original home page without a phone number.
• Conversion Increase: +.5% (half of a percent increase)

Statistical Significance

We didn’t hit a statistically significant threshold during the allotted time for the test. However, we like the trend that we saw and we’re going to try another test with a bigger phone number next.

Conclusions

The first thing we would like to mention is:

The Flowr didn’t follow instructions and they still got some sign up lift!

The rules explicitly said to have a highly visible phone number on their home page variation. As you can see the phone number is tiny (at least it was above the fold). But even with this tiny phone number, Flowr was able to increase their sign ups.

Davorin Gabrovec from Flowr concluded:

“Even though we didn’t receive a lot of calls I believe that having a phone number visible on the website gives more credibility to our product and trust to our visitors. When we re-design our website we will definitely include appropriate space for a bigger phone number.”

The bottom line is that having a phone number does bring peace of mind to consumers and people you do business with. If, at the very least, it instills trust in your visitors and removes any “fly-by-night-operation” fear they may have. If you run a Software as a Service (SaaS) business, we encourage you to try testing a phone number on your site (and let us know what happens!).

About the Author: Sean Work is the Marketing Director at KISSmetrics.

Editor’s Note: Everyone wants to improve their conversion rates right? But are “conversion rates” the right indicators for steering you towards your most important goal: increasing revenues? The case study presented here is about how FoxTranslate.com (a document translation service), found that focusing on conversion rates alone can be misleading. Read this article to discover what they learned. You’ll be surprised and it may change the way you run and think about your website tests.

As most of you know, a popular platform for A/B testing new website features and landing pages is Google Website Optimizer. Google Website Optimizer uses conversion rate (goal reached / number of visitors starting the goal funnel) to determine the winner of A/B tests.

Over the past few months FoxTranslate has been running A/B tests to improve our website conversions. As we got involved with setting up and running A/B tests, we began to bury our heads in the sand by being enveloped with the busy work of testing.

It wasn’t until we decided to make a redesigned home page variation that we started to “see the light”. After a week of tweaking and improving what we thought a better home page would be, we came up with these initial assumptions as to why our new home page variation was more effective:

We felt the new home page:

1. Highlighted our core service features.
2. Segmented the data into digestible sections.
3. Offered a slicker and professional feel.

We expected our redesigned homepage to be the clear champion.

Below is a screenshot of our original home page:

And here is a screenshot of our home page variation:

Our new homepage showed a 6% improvement in conversion rate vs. the old homepage. While the 6% was a nice win, the improvement was not capturing the full story.

We have a diverse customer base of businesses, law firms and consumers with very different spending habits. Even within these segments, transaction sizes vary greatly. The problem with focusing on conversion rate alone is that the conversion rate metric suffers from not telling you who and how much each visitor is spending.

Hoping that the new site was attracting higher spenders, a measure not tracked by conversion rate, we started tracking each page based on the revenue generated. We accomplished this by coding custom variables into each page and importing the by-page information into Google Analytics. For more information on custom variables, check out this Google Analytics overview.

Revenue Improvement of Current vs. Redesigned Homepage

With our new revenue measure, our new homepage really shined, generating a 17% improvement vs. the current homepage on similar impressions. We previously noted the 6% improvement in order volume (conversion rate), meaning that 11% of the improvement was due to higher transaction sizes.

Our use of custom variable tagging also allowed us to compare metrics on customer engagement available in Google Analytics.

On average, customers stayed longer on our site and were less inclined to immediately exit the site. While potentially a stretch, it appears that customers who spend more tend to do more research. Based on average time on site, our new homepage appeared to better engage higher spending customers.

What We Learned

Conversion rate is golden, but revenue generated is platinum. While the improvement in conversion rate was nice, it was really revenue generated that showed the vast improvement from our new homepage design.

While conversion rate is a good metric, the metric assumes that every transaction is of the same value. Therefore, an improved conversion rate may just be masking lower revenue due to a higher concentration of lower value customers, a negative to your company.

About the Author: Jason Thai is a marketing manager for FoxTranslate, a provider of certified translation services, specializing in translation of business, legal, immigration and academic documents in over 30 different languages.