Back in September Flowr set off on the Grasshopper / KISSmetrics Phone Number Challenge. The idea was that they were going test placing a phone number on their home page to see if they could increase sign ups. The hypothesis was that by having a visible phone number on their home page, the trust factor would increase and therefore sign ups would too.

Jonathan Kay from Grasshopper Virtual Phone Systems, proposed the original concept of this challenge. He believed that:

“People feel more comfortable with brands that they can put a face behind. Even though you might purchase a product exclusively online, having a phone number on your site and the ability to talk to a real person (who cares) in turn makes you feel more comfortable taking out your wallet (or recommending someone else to) for this brand.”

TheFlowr.com Home Page Variants

Before we get into the results of this simple A/B test, let’s quickly look at the differences between the two home pages.

The image below is the original Flowr home page. If you look closely, you will see that there is no phone number on the page.

In the next image, you will see a screenshot of the home page variant with a phone number and the call to action “Want to have a chat? Call us at..” (look for the red asterisk).

The Results

Flowr ran a simple A/B test with one home page variation using KISSmetrics. Again, the only difference in the variation home page was the addition of a small phone number and some supporting call-to-action text. The results were as follows:

Test Conditions

• Test Duration: From September 9, 2011 to October 24, 2011. (approx. 6 weeks)
• Test Item: Website home page of Flowr vs. home page variation.
• Test Type: A/B Test (only difference between variation was a phone number and associated call-to-action).
• Test Goal: Increase software sign ups from home page.

Results

• 53.96% of sign-ups originated from the home page variation with the phone number.
• 46.04% of sign-ups originated from the original home page without a phone number.
• Conversion Increase: +.5% (half of a percent increase)

Statistical Significance

We didn’t hit a statistically significant threshold during the allotted time for the test. However, we like the trend that we saw and we’re going to try another test with a bigger phone number next.

Conclusions

The first thing we would like to mention is:

The Flowr didn’t follow instructions and they still got some sign up lift!

The rules explicitly said to have a highly visible phone number on their home page variation. As you can see the phone number is tiny (at least it was above the fold). But even with this tiny phone number, Flowr was able to increase their sign ups.

Davorin Gabrovec from Flowr concluded:

“Even though we didn’t receive a lot of calls I believe that having a phone number visible on the website gives more credibility to our product and trust to our visitors. When we re-design our website we will definitely include appropriate space for a bigger phone number.”

The bottom line is that having a phone number does bring peace of mind to consumers and people you do business with. If, at the very least, it instills trust in your visitors and removes any “fly-by-night-operation” fear they may have. If you run a Software as a Service (SaaS) business, we encourage you to try testing a phone number on your site (and let us know what happens!).

About the Author: Sean Work is the Marketing Director at KISSmetrics.

Editor’s Note: Everyone wants to improve their conversion rates right? But are “conversion rates” the right indicators for steering you towards your most important goal: increasing revenues? The case study presented here is about how FoxTranslate.com (a document translation service), found that focusing on conversion rates alone can be misleading. Read this article to discover what they learned. You’ll be surprised and it may change the way you run and think about your website tests.

As most of you know, a popular platform for A/B testing new website features and landing pages is Google Website Optimizer. Google Website Optimizer uses conversion rate (goal reached / number of visitors starting the goal funnel) to determine the winner of A/B tests.

Over the past few months FoxTranslate has been running A/B tests to improve our website conversions. As we got involved with setting up and running A/B tests, we began to bury our heads in the sand by being enveloped with the busy work of testing.

It wasn’t until we decided to make a redesigned home page variation that we started to “see the light”. After a week of tweaking and improving what we thought a better home page would be, we came up with these initial assumptions as to why our new home page variation was more effective:

We felt the new home page:

1. Highlighted our core service features.
2. Segmented the data into digestible sections.
3. Offered a slicker and professional feel.

We expected our redesigned homepage to be the clear champion.

Below is a screenshot of our original home page:

And here is a screenshot of our home page variation:

Our new homepage showed a 6% improvement in conversion rate vs. the old homepage. While the 6% was a nice win, the improvement was not capturing the full story.

We have a diverse customer base of businesses, law firms and consumers with very different spending habits. Even within these segments, transaction sizes vary greatly. The problem with focusing on conversion rate alone is that the conversion rate metric suffers from not telling you who and how much each visitor is spending.

Hoping that the new site was attracting higher spenders, a measure not tracked by conversion rate, we started tracking each page based on the revenue generated. We accomplished this by coding custom variables into each page and importing the by-page information into Google Analytics. For more information on custom variables, check out this Google Analytics overview.

Revenue Improvement of Current vs. Redesigned Homepage

With our new revenue measure, our new homepage really shined, generating a 17% improvement vs. the current homepage on similar impressions. We previously noted the 6% improvement in order volume (conversion rate), meaning that 11% of the improvement was due to higher transaction sizes.

Our use of custom variable tagging also allowed us to compare metrics on customer engagement available in Google Analytics.

On average, customers stayed longer on our site and were less inclined to immediately exit the site. While potentially a stretch, it appears that customers who spend more tend to do more research. Based on average time on site, our new homepage appeared to better engage higher spending customers.

What We Learned

Conversion rate is golden, but revenue generated is platinum. While the improvement in conversion rate was nice, it was really revenue generated that showed the vast improvement from our new homepage design.

While conversion rate is a good metric, the metric assumes that every transaction is of the same value. Therefore, an improved conversion rate may just be masking lower revenue due to a higher concentration of lower value customers, a negative to your company.

About the Author: Jason Thai is a marketing manager for FoxTranslate, a provider of certified translation services, specializing in translation of business, legal, immigration and academic documents in over 30 different languages.

Last month we announced the winners of the Grasshopper / KISSmetrics Phone Number Challenge. Out of about 30 entries, three businesses were chosen: Flowr, The Site Slinger and DODOcase.

Each one of these outfits were given a free lifetime account of both KISSmetrics and Grasshopper phone service. In return, we get to conduct a case study of each website to see if sign-ups will increase due to the addition of a visible phone number on their home page. Click here to read Jonathan Kay’s hypothesis about adding a phone number to a company home page.

Initially we were going to track each of these websites simultaneously. However, DODOcase got so excited with KISSmetrics that they jumped into several conversion optimization trials. This means that we’re going to have to conduct their test later when they have finished their initial rounds of experiments.

For now we are going to focus on Flowr and The Site Slinger. Read on to see what home page variants were created for this case study.

TheFlowr.com Home Page Variants

The image below is the original Flowr home page. If you look closely, you will see that there is no phone number on the page.

In the next image, you will see a screenshot of the home page variant with a phone number and the call to action “Want to have a chat? Call us at..” (look for the red asterisk).

The Site Slinger Home Page Variants

Below is a screenshot of the original Site Slinger home page. Again, there is no phone number visible on the page.

Next is a screenshot of the home page variant of The Site Slinger. Notice that the phone number is in two locations. Once at the top right hand corner of the page and once above their video.

Ready, Set, Go!

Over the next six weeks, we are going to A/B test these home page variations. The hypothesis is that having a phone number will present visitors with an increased feeling of trust which could lead to more sign-ups. In addition, a phone number may provide an easier way to order, which may lead to increased sign-ups as well.

Check back with us in six weeks to find out the results of this test!

If you do enough A/B testing, I promise that you will eventually have some variation of this problem:

You run a test. You see a 10% increase in conversion. You run a different, unrelated test. You see a 20% increase in conversion. You roll both winning branches out to 100% of your customers. You donʼt see a 30% increase in conversion.

Why? In every world Iʼve ever inhabited, 10 plus 20 equals 30, right? Youʼve proven that both changes youʼve made are improvements. Why arenʼt you seeing the expected overall increase in conversions when you roll them both out?

There are lots of different reasons this can happen. Here are a few:

The Changes Affected the Same Group of People

You may be causing some problems for some of your users. I know, itʼs a chilling thought. But the truth is, something youʼre doing is probably keeping some of your users from doing something you want them to do.

The interesting thing is that there may be a lot of different ways to fix this problem youʼre causing. For example, if itʼs impossible for some people to purchase products on your site because you only accept Visa, you might experiment with also offering American Express or MasterCard.

So, letʼs say that you run two separate tests, one that adds AmEx, and one that adds MasterCard. You run them as separate tests because you want to gauge which is most attractive to your users. Or because you just love running tests.

Now letʼs say that each new payment method gives you a positive increase in revenue of 20%. Youʼd think that, if you combined them, youʼd get a revenue increase of 40%, right? Nope.

Imagine somebody who doesnʼt have a Visa, but does have both a MasterCard and an AmEx. Their problem (that they canʼt purchase because they donʼt have Visa) goes away regardless of which new payment method you end up implementing. Implementing both doesnʼt mean theyʼll spend twice as much.

Want to Avoid This?

If youʼre running several different tests that all act on the same metric, youʼre almost certainly going to hit this problem. Itʼs enough to be aware that itʼs happening and not to plan on test results adding up exactly.

Other Changes Are Hurting You

It seems like it would go without saying, but A/B tests only test A against B. They donʼt take into account other changes you may be making at the same time.

For example, letʼs say you run several tests to improve your registration flow. In each test, youʼre seeing statistically significant improvements in registrations. However, when you eventually release all the changes out to 100% of your new users, youʼre just not seeing as much improvement in the actual number of people who are registering as you expected.

The first question you should ask yourself is, “What else did I change?” So many things can affect registrations. For example, maybe youʼve changed the type of user youʼre acquiring, and the new users are less likely to register. Maybe something you did is slowing down your page load, and thatʼs causing fewer people to make it through registration. Maybe youʼre running another test that is having a seriously negative impact.

Want to Avoid This?

Make sure you know everything that might have an impact on the key metrics youʼre measuring, and test all of it.

If youʼre only testing your major features, youʼll never know if some ʻtrivial changeʼ you pushed out without thinking about it is actually counteracting the benefits you expect to receive from an experiment.

The Changes Interact

A/B tests are perfect for what I call ʻone variable changesʼ. For example, theyʼre great if youʼre testing a landing page, and you want to see if you get more conversions with a blue button or a red button. Things like this can have a surprising impact.

The problem comes when you try to merge the outcomes of two completely separate tests. Imagine a stupidly simple scenario in which you had two tests running:

• The first tests a blue button vs. a red button on a white background.
• The second tests a white background vs. a blue background with a red button.

Now, imagine that, in the first test, the blue button won, while in the second test, the blue background won. If you tried to merge the results of both tests, youʼd end up with a blue button on a blue background, which Iʼm going to guess isnʼt going to convert terribly well, since the button will be invisible.

Obviously, those are terribly designed tests, and you wouldnʼt just merge them, but the point is that all sorts of different experiments can combine in surprising ways. Certain messaging might be effective when presented with a particular image, while it might be awful with another image. Regardless of how each test performs independently, they might combine very poorly.

Want to Avoid This?

Be aware of potentially conflicting changes, and make sure that youʼre always testing the final version of designs against the original, even if youʼve tested each element on its own.

Your Test Wasnʼt Statistically Significant

If youʼve done any A/B testing, youʼve probably gotten excited or sad about early results only to see them change wildly over time. The problem here is frequently statistical significance.

When youʼre working with very small numbers, the behavior of even one user can drastically throw off your metrics.

Consider if you only have two users. If one person buys and the other doesnʼt, youʼve got a conversion rate of 50%. Not bad. But if both people buy, youʼve got a conversion rate of 100%. Thatʼs remarkable! And totally unsustainable!

Want to Avoid This?

You need to use a big enough sample size to make sure that your results are significant. And, this is important, you need to determine the sample size ahead of time in order to avoid something called repeated significance testing errors. I wonʼt try to describe it here, but feel free to read more about it if you donʼt believe me. Just be prepared to learn a little math.

Another big problem is that it can be really tough to find the right sample size because of the natural variance of whatever youʼre trying to measure. One way to get an idea of your variance is to run an A/A test. This means that you split your users into two different groups and show them both the same thing.

Youʼd imagine that youʼd get exactly the same results from each group, but youʼll notice that there will be natural differences between the two groups. If this difference is very large, that means that youʼll most likely need a larger sample size to get statistically significant results.

As a note to all the math majors out there, yes, I know that this is not a very good explanation of statistical variance, and you probably have something written in Greek that will explain it all much better. But the truth is, just this is little bit of information can help you plan a much more predictive test.

Your Metrics Are Wrong

Here is the dirty little secret of gathering metrics: doing it is really kind of hard to get right. If anything about your metrics is confusing, itʼs very possible that itʼs because the numbers youʼre collecting are just wrong.

I canʼt tell you the number of times a new feature that should have performed well has failed, and we have traced the failure to a small bug that only affected the way the events were being recorded.

Want to Avoid This?

Use your common sense. If youʼve done your homework, and you really believe something should be doing better or worse than it is, do a little digging.

Look for things like race conditions in the code that could be breaking the information gathering system. Also, make sure that youʼre recording everywhere an event occurs. If someone can purchase from multiple places in your product, make sure youʼre recording a ʻpurchaseʼ event everywhere it can happen.

Still Not Adding Up?

There will be times when you do all of these things right, and you still get surprising results from your experiments. That means itʼs time to observe some users and find out whatʼs really going on.

Qualitative research is wonderful for understanding why your users are doing what theyʼre doing, especially when theyʼre doing something surprising! But thatʼs a different blog post.

About the Author: Laura Klein is a user experience and research expert in Silicon Valley, where she teaches startups how to make their products more appealing and easier to use. She blogs about UX, metrics, customer development, and startups at Users Know. You can also follow Laura on Twitter.

A well-built sales funnel is never complete until every part of it has been tested and optimized. For maximum success, marketers should dig deep and experiment with every customer interaction point. What follows is a brief guide that outlines what things are good to regularly test and optimize—including PPC, media buys, landing pages, and email campaigns. You don’t have to test everything all at once. Start with the marketing activity the produces the highest return and then work your way down.

Click on the image below to view an enlarged version of this infographic:

Want to display this infographic on your site?

Simply copy and paste the code below into the html of your website to display the infographic presented above:

Tips and Tricks to Tweet:

• Too many unrelated links within eyesight of your call to action can distract visitors and lead them away from your sales funnel. »tweet«
• Some fonts are easier on the eyes than others, and experimenting with various sizes and styles can have a noticeable impact on the success of your sales funnel. »tweet«
• Use KISSmetrics to A/B test your landing pages and easily view their performance with the best conversion funnels in the business. »tweet«
• Fewer form fields don’t necessarily mean more sign ups. Sometimes, as in the case of lead generation, it’s beneficial to add extra form fields to extract more information. »tweet«
• Use KISSinsights to find out what copy and keywords are attracting customers, and use them in future ads. »tweet«
• Try displaying your ads on a wide variety of websites and test the responsiveness of each site’s audience to build an optimized display network. »tweet«
• KISSmetrics can track which combination of displays ads and landing pages are producing the best ROI. »tweet«

A/B testing SEO landing pages can be more tricky than testing an email or pay-per-click campaign, especially when you are using a tool like Google Website Optimizer. There are a few precautions you should take to make sure you obtain accurate test results and to not damage your previous SEO efforts. Read below to find out why.

Let’s Clear A Few Things Up Before We Begin

This post is not about testing different variations of a SEO landing page to see which version attracts more traffic from the search engines. This post is about how to test SEO landing page variations to see which version converts higher for a desired goal. An easy example that would help illustrate this point would be the desire to enhance a product page that will ultimately send more visitors to a checkout page. You obviously want the product page to rank high in the search engines, but the point of testing the product page is to achieve the highest possible conversion between the product page and the checkout page.

Also, I will be talking a lot about landing page variation “A” and “B”. Variation “A” will be the first SEO landing page you build and the first that will be live on your site. Landing page “B” will be the first test variation of “A”. Finally, a “landing page” refers to a webpage specifically designed to attract search engine traffic for a particular keyword.

Since You Will Be Testing SEO Landing Pages, Make Sure They Are Optimized!

Here is a checklist of important on-page SEO items you will want to focus on when you create your first SEO landing page:

• SEO friendly URLs:

If your website is able to have SEO friendly URLs, then do so. Simply make sure you have keywords in your URLs and separate each word by dashes. For example: http://example.com/seo-friendly-urls

• Title Tag:

The title tag is the biggest influencer in terms of “SEO weight” for any webpage. Include the keyword near the beginning of the title tag.

• H1 Heading:

The H1 should match the title tag if possible. Again, be sure to include the keyword in the H1 heading.

• Bold the keyword once in the text:

You may want to bold the keyword once and make sure it appears in the body text.

• H2 Headings:

It’s a good idea to have a couple H2 heading tags to act as sub headings. These sub headings should include keywords that are similar or related to the main keyword.

• Images With Alt Text:

Images provide a better user experience, so try to include a couple images on your landing page that contain descriptive alt text.

• External Links:

In order to show the search engines how your landing pages is related to other pages on the web, place a few links for “further reading” or to include “references”.

Finally create separate landing pages for all the keywords you want to rank for. I generally recommend for service based businesses to build niche landing pages for each service they provide. For example, if you’re dealing with a landscaping business, you may want to create landing pages for “walkway lighting”, “irrigation”, and “putting green installation”, because these would all be pages of different services the landscaping business provides.

What To Test

We have covered what to test extensively in this series. However, here is a quick list of webpage elements that can be tested:

• The headline.
• Your call to action.
• Any graphic you use in direct correlation to your sales efforts.
• The sales copy or product descriptions.

For more information please read this blog post: A Beginner’s Guide To A/B Testing: An Introduction

Launch Version “A” And Wait

A critical part of launching your first SEO landing page is that you have to wait for the page to get indexed by the search engines. The reason for this is to make sure you have enough traffic to run an A/B test. You need to hit a certain threshold of traffic in order to generate results worth analyzing.

If you are building links toward your landing pages, you should wait until those links are acquired and wait about a month for the “link juice” to be applied to those pages. Watch your analytics and wait until traffic to that page ramps up and levels out. Then you will be ready to run your tests.

Make Sure Variation “B” Does Not Get Indexed

A/B testing tools like Google Website Optimizer require you to build landing page variations on separate URLs. Anyone familiar with SEO will automatically see this as a “duplicate content” issue. Some people don’t think this is a big issue when it comes to A/B testing, but I think it can be (especially if you’re a small site without much search engine trust) and I think it’s better to be safe than sorry.

Since you will be testing two different web pages with very similar content and are about the exact same subject matter, you will want to make sure that you keep variation B out of the search engines. Here’s why:

• Keyword Cannibalization: By having two live URLs on the same topic, the search engines will get confused on which landing page variation will be the more important version to index. What will happen is the search engines will choose the more important page arbitrarily and therefore only send traffic to one page (or a majority of the traffic). This can cause damage your previous SEO efforts. Read more about keyword cannibalization here.
• Duplicate Content Issues: In regards to search engine optimization you almost certainly want to minimize the amount of repetitive content on your site. Your goal should be to have most of your site consist of unique content if possible. By have two URLs with very similar content, chances are you will be given a duplicate content penalty. A similar result would happen in the keyword cannibalization issue mentioned above, where the search engine would “pick” one URL over the other.

How To Make Sure Variation B Doesn’t Get Indexed

• Apply a meta “No Index” tag to the landing page B head. This will instruct the search engines to not index landing page B.
• Apply the canonical reference to landing page A in the landing page B head. This will tell the search engines that landing page A is the original version among to the two landing pages.
• Don’t mention variation B in your robots.txt file. There is no point in alerting others that this page exists. Sometimes this is a way to actually get a page indexed that you don’t want to get indexed (because other “bad” robots can follow those URLs and link to them elsewhere on the web).

Run, Rinse, and Repeat

Once your A/B tests are ready to begin, let ‘em rip and monitor your website testing software to see how each variation is performing. Most website testing tools will give you real time feedback as to how each page is fairing. They will also let you know when you have hit a statistically significant threshold where you can make a decision on which landing page variation was the winner.

If your variation B landing page turns out to be a winner by a long shot, then congratulations! Now you can make variation B the new A and prepare for another series of tests.

What if the difference in results is minimal?

This is indeed a problem with A/B testing webpages. If the results between your tests are not significant enough you will have to keep testing drastic variations until you find a winning variation that produces significant results. Your website testing tools will alert you if your test did not hit a statistically significant result.

Will The Winning Page Hurts My SEO?

It’s possible that your winning page may alter your H1 headings, body text, and other on-page factors that affect the SEO of your landing page. This could change the search engine result ranking of your landing page. However, it’s far more important to increase the conversion power of a landing to page than to worry about the somewhat negligible effects on-page modifications will have.

Besides you should be continually link building towards your landing pages and increasing the overall traffic to your site externally. You don’t want to drive a ton of traffic to a page that converts poorly do you?

Conclusion

With tools like Google Website Optimizer and KISSmetrics, you can easily test different variations of your SEO landing pages. You just have to use a little caution when preparing your tests and be sure to give any new SEO landing pages enough time to build up enough traffic to make testing fruitful.

With that said, please leave some comments as to your thoughts on this post and what other things first time testers should be aware of.

About The Author: Sean Work is the marketing coordinator at KISSmetrics. Follow him on twitter (@seanvwork) and ask him for a free cup of coffee :)

This is a guest post written by Anand Rajaram. Anand likes to create products that people love (and use). He is the “product guy” at OfficeDrop. This post is a case study that illustrates the powerful synergistic effect a company can get when using both KISSmetrics and KISSinsights to solve conversion funnel problems.

The Problem

We expanded our product offerings

But growth wasn’t expanding along with it

We knew we had a problem – but what?

And how should we fix it?

The Context

OfficeDrop had originally set out as a startup focused on offering a document scanning service to consumers and small businesses using Netflix-style prepaid envelopes. In the middle of 2010, using the principles of customer development, we had:

1. Launched a new cloud-connected scanning software.
2. Narrowed down our target market to 1-25 person small businesses (by offering accounts with multiple users and auditing).
3. Expanded our product offering to plans that offered only a self-scan option (No mail-in scanning).

Our new focus seemed to be paying off, as visits to our website doubled very quickly in the summer of 2010. New customer sign ups also increased, but at a lower rate. Our conversion funnel was clearly not keeping up with our growth, but where exactly was the problem?

So, is our funnel leaking or are we not filling it right?

Either our new positioning was not bringing the right target to the top of the funnel or potential customers were entering the sign up process and then leaving. We didn’t know which.

We needed better data. With our free analytics provider at the time, the conversion data almost never matched with our own database. We also needed to know where in the funnel potential customers were falling off of the conversion cycle. Enter KISSmetrics.

Funnel v 1.0

Once we installed KISSmetrics we had a much better visualization of our conversion funnel. And it was leaking:

Users came to the pricing page from multiple landing pages and then to a registration form and a confirmation for signing up.

This visually rich funnel makes it obvious that we were losing people from both our pricing page (Step 1) and signup form (Step 2). We needed to improve both.

So our next goal was to identify what is wrong with our pricing page (including the prices), and how we can improve the entire sign up process.

Right question at the right time = Right answer every time

We had already done some qualitative testing on the pricing (via services like usertesting.com), so we wanted to try something different. More importantly, we wanted insights into users’ thought process when they had the most context  – when they were on the pricing page trying to decide if they wanted to purchase our product or not.

We tried initiating Live Chat with users in the pricing page, but it wasn’t particularly productive or effective. Enter KISSinsights.

We knew that visitors spent an average of one and half minutes on the pricing page, so we configured a KISSinsights survey that asked “Is our pricing clear”?, once a user has spent 40 seconds. This delay ensures that we are get answers from highly qualified (that is, highly interested) visitors.

What we learned

In short order, we got 20 insightful responses saying, “Not so much, here is why” with several recurring themes.

The most important source of confusion was that the pricing page did not make the obvious difference between our new product, self-scanning plans and the mail-in scanning service.

What we did

Like almost everything else that we do here, we adopted an iterative and incremental approach to fix our funnel. Phase 1 was very quick copy edits to the pricing page. Simple steps increased the differentiation between mail in scanning plans and self-scanning digital filing solutions.

Funnel v 1.1 – 15% increase in conversion from incremental changes

We then A/B tested the two pricing pages and found a 15% increase in conversion and a corresponding a 15% increase from the pricing page to the sign up page.

During this period, there was also a 10.5% increase in visitors saying the pricing was clear. We were also pleasantly surprised when some of our users reported that they really liked the “pop-up thingy”. It was an easy and effective way to reinforce our commitment to listen to them. For us, this was also a very effective way to augment our input from “getting out of the building”.

Funnel v 2.0 – 40% increase in conversion from larger workflow changes

Our original sign up form was single long sign up form that asked for all details in one-shot. KISSmetrics allowed us to clearly see that we were losing a lot of interested parties after they had clicked through to the form. There were two important changes that we made to the form. First, we reduced the number of fields in the form to enable an easier sign up process. We also significantly improved the look and feel of the sign up form, such that it blended with the main site’s look and feel.

The form started converting much more efficiently, and we saw a 40% improvement in a matter of days.

Summary of results

So, what are the lessons learned?

In hindsight, a lot of changes, tweaks and adjustments may seem obvious. But, to quote the American statistician, Williams Edwards Deming, “In God we trust, everyone else must bring data”.

For startups seeking to iterate through the Build->Measure->Learn Loop, having this data and insight makes it easy to commit resources to where it is needed the most and where it will have the maximum impact.

• Data is gold, only when it is actionable.
• Asking users the right question at the right time, gives you a real answer.
• Always be testing and measuring.

How do you gather input from your customers and visitors? And how do you use it to improve your conversion?

About OfficeDrop

OfficeDrop is a cloud filing system, scanner software provider and document scanning service that helps small businesses manage paper and digital documents.

This is a guest post written by Ryan Fujiu. Ryan is based out of Venice, CA and does product and business development for about.me.

Like many other startups, we take pride in being data driven. We watch our key conversion metrics, measure CTR, ROI, we even use data to drive what features we launch on the site. But when we started making our product video, we *assumed* we knew our top value props and use cases. So confident in our assumptions, we wrote a script and started shopping it around to production companies. Then we took a step back, did some research, and the results changed everything.

Almost every new product or service has a product video or screencast. Furthermore, almost every new product *launches* with one. The goals of the video are simple; to get the viewer’s attention, to explain the benefits of the product, and to explain why they should use it. In some cases, it should also devote a small amount of time explaining how to use the product.

As a founder, product manager or marketer, it’s easy for you to come up your key benefits and use cases for your product, right?  “For sure” you say?  But really, the only way to know for sure is to *test your assumptions*. As Patrick Vlaskovits said in his book, The Entrepreneurs Guide to Customer Development, “Dont mistake guesses for facts.”

At about.me, we practice customer development. So, we decided to run a couple surveys. The first one, we ran through KISSinsights, which asked a very simple question “What do you use your about.me profile for?”

This was only served after the user logged in twice, and had been on the ‘edit profile’ page for 20 seconds. We put these relatively strict requirements on the survey to get responses from our engaged users. These requirements are key to filter out the noise.

The results were staggering.  The survey had a 10% conversion rate, generating 2,500 responses over 2 days. As we compiled the results, the key benefits and use cases we needed to highlight in the screencast became abundantly clear.  Here are the top results from the survey:

• To promote your business / self online
• As a hub for your social activity
• To let people know who you are and what you do
• As an online business card
• To put in email signature / twitter bio
• As a place to send anyone who wants to know more about me
• Stats on visitors

To validate these results, we ran the product/market fit survey (aka the Customer Development survey). It asks a couple very important questions, one being, “What is the primary benefit that you received from about.me?” The results from this survey mirrored the results from the KISSinsights survey, so we knew, at that point what to base our screencast on.

Another benefit of asking open-ended questions is the responder can express what the product is to them in their own words. You can use this method to discover a tagline or value prop statement instead of paying an expensive branding firm. Here were some of the gems that came straight from our users’ mouths:

• “As a one stop page that lets the user find out who I am what I do and how to reach me.” Ardail Smith
• “As a quick and easy place to provide links to my online footprint.” Lisa Proctor
• “Simplistic way to tell people who I am and what I stand for… :)” Deanna Obenauf
• “As a starting place to find more about me on the web.” Jay Bernard
• “Personal online brand management” Jonathan Baritugo

After we analyzed all of the data and agreed on the key benefits and use cases, the script for the screencast fell into place.  We went through the rest of the process with the confidence that the benefits we were highlighting were both real, and important to our users. If you watch our video closely, you can see where we mentioned each of the benefits and use cases:

It turned out, that if we would have built the video on our assumptions rather than data, we would have totally missed 2 important use cases (including one that most users said was the reason they use our product). This came as a shock to us, and opened our eyes to the importance of customer research and qualitative data.

If you build your screencast off your assumptions rather than data, you run the risk of missing some of your key benefits or use cases. Or even worse, you may introduce an idea that is flat out not a benefit, thoroughly confusing your potential users. The information you gather here is critical for any business to know. So, the moral of this story is: test your business assumptions.

If you want to read more about this and other tenets of customer development, get “The Entrepreneurs Guide to Customer Development” by Patrick Vlaskovitis.

Special Offer: Use discount code KISS and get 33% off the Customer development book! Buy it now »

Pay-per-click advertising is a key component of many online marketing campaigns. It can also be one of the most expensive ongoing costs in a campaign. Therefore, it’s key that you test your ads regularly, to make sure you aren’t letting conversions slip through the cracks.

To an extent, PPC testing is simpler than many other kinds of A/B tests, partly because there are fewer things to test. But that doesn’t mean any less care and planning should go into preparing and executing these tests.

This is the fourth installment in our A Beginner’s Guide to A/B Testing series. Be sure to check out our previous posts: A Beginner’s Guide To A/B Testing: An Introduction, A Beginner’s Guide To A/B Testing: Exceptional Web Copy and A Beginner’s Guide to A/B Testing: Successful Email Campaigns, and stay tuned for our final installment on testing SEO landing pages.

Deciding What to Test

Pay-per-click ad testing is a bit more streamlined than the other topics we’ve covered in this series. For the most part, you’re going to test one or more of the following four things:

• The headline
• The body text
• The link
• The keywords the ad displays for

The headline is the part that’s going to show up as a link (in blue) in search results. This is usually created to reflect whatever is being offered. It should be short (4-5 words seems ideal, though less is good if it still gets your point across), to-the-point, and include the keyword being searched for if at all possible (on Google, these keywords will appear in bold).

The body text is the equivalent to your page’s description meta tag in organic search results. It should give searchers a bit more of an idea of what you’re offering, and what they’ll get if they click on your link.

Where your ad links can have just as much impact on conversion as the ad itself. In most cases, you’ll want your ad to link to a landing page or product page, not to your home page (unless you only sell one product, and your home page doubles as your landing page). You might also consider having it link to a mini-site designed specifically for use with your PPC campaigns.

Some marketers feel like their ad should show up for any relevant keyword search. But some keywords and phrases will convert better than others. The only way to figure out which ones work best for your products is to test different ones and track the results. When you find certain keyword combinations that convert best for you, focus your marketing dollars there.

Brainstorm a list of possible keywords your target customer might use when looking for your product or company. You’ll want to test not only which keywords seem to work best, but also which keyword/ad combinations get you the best results, as some ads may perform better under specific searches than others.

Once you know which variables you want to test, and have values for each one, you’ll want to make a list of the different tests to run. If you have two possibilities for your headline, and two for your description, then you’ll run four tests (AB, BA, BB, AA). The more variables you come up with, the more tests you’ll need to run. One note: make sure that all of your variables make sense when combined; you may need to remove some possible combinations if they’re repetitive or conflicting.

What Are You Testing For?

A/B testing for PPC ads can be done for a variety of reasons. Sometimes, using PPC to get people to sign up for your newsletter can lead to more conversions down the road, especially for higher-end products. Other times, getting people to buy right away is a better strategy. In either case, you need to have a goal from the start, so that you know how to analyze your results.

Decide, too, whether you’re only going to focus on click-throughs or if you want to focus on the entire conversion process. If your main goal for your PPC campaign is to raise awareness, then click-throughs might be enough. But in most cases, you want to find the highest-converting ads. You may find that some keywords have high click-through rates, but low conversion rates, generally because people searching using those specific terms aren’t yet ready to buy and are only collecting information.

Track and Analyze Your Results

You’ll want to track two primary things: how many click-throughs you get, and how many conversions those lead to. The goal here is to maximize not only the click-throughs, but also the conversions. High numbers in each are obviously going to give you the best ROI. KISSmetrics offers tools to track these conversion funnels, so you can get an accurate picture of what’s working and what’s not, and insight into why that might be.

Like testing web copy, you’ll want to let your test run for at least a few days, and upward of a couple of weeks, depending on how much traffic you get. Make sure you monitor where you ad appears during that time, too. If you’ve bid too low, you may find that your ad is showing up too low in results to be effective. It can also skew results if your ad position shifts too much during the test.

If your results aren’t clear-cut, or are conflicting (such as one ad with much higher click-throughs, but the other with a much higher conversion rate), then you’ll want to run more tests, changing variable slightly until you find the best combination. Remember, too, that consistency between your ad and the page searchers land on is key, and could account for poor conversions on and ad with a higher click-through rate.

Best Practices

For conducting A/B tests on your pay-per-click ad campaigns, you’ll have better results if you follow these simple guidelines:

• Test your ad variations simultaneously to minimize time-based factors that might skew your results.
• Only test one thing at a time so you can pinpoint what is affecting the success of your ads.
• Test early and test often.
• Pay attention to the hard data you collect, rather than your “gut instinct”.
• Make sure you let your tests run for long enough to collect enough results to get accurate results. You need at least a few thousand impressions in order to get any kind of accurate data. Remember, with PPC, you’re only paying when people click, not for impressions.

Be sure to check out the other posts in this series, too: A Beginner’s Guide to A/B Testing: An Introduction, A Beginner’s Guide to A/B Testing: Exceptional Web Copy, and A Beginner’s Guide to A/B Testing: Successful Email Campaigns. And we have still have one more post in this series coming up, covering SEO landing page testing.

About the Author: Cameron Chapman is a freelance designer, blogger, and the author of Internet Famous: A Practical Guide to Becoming an Online Celebrity.