Four More Ways to Crack the Keyword (not provided) Code

Secure search now obscures up to 60% of search visits, making it virtually impossible to determine which keywords are driving your site traffic. Appearances of the dreaded (not provided) have been steadily increasing. Now the default Google Analytics reporting is nearly worthless for understanding the impact and opportunities for your SEO program.

This has been driven primarily by the inclusion of secure search in Google Chrome and Firefox browsers for all logged-in Google account holders. If one of your website visitors comes from search while logged into Gmail, that referring keyword is hidden from your view. The same goes for a user logged into Google+, YouTube, Google Drive… You get the picture.

And while the percentage of hidden keywords varies from industry to industry, it has reached a point where large chunks of search traffic are distilled down into nothing more than a single line item in Google Analytics – an enigma wrapped in a riddle, as it were.

traffic from google not provided

(image via Search Engine Land)

1. Making the Most of Your Marketing Efforts

If you’re investing time and money in search engine optimization, blogging, or other content marketing, secure search presents a big problem. How do you know what’s working and what’s not? You’re left back in the dark ages of marketing pioneer John Wanamaker, who famously said, “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.”

In a previous KISSmetrics blog post, Claire Broadley looked at three methods for gleaning insight into (not provided) traffic. In this post, we’ll look at four more ways you can crack the (not provided) code: dissecting Google search strings, surveying users, instrumenting your site search, and using query classification and advanced segmentation in Google Analytics. These techniques will give you additional tools to help you peer into the secure search black box and find the keywords that really matter to your business.

2. Dissecting Google Search Strings

One of the most challenging (but exciting!) ways to gain insight into (not provided) keyword traffic is to analyze the search strings of traffic coming from Google. If you look at the URL from a Google search in your address bar, you’ll see that it is not the actual URL of the page. Rather it is a redirect URL with a string of parameters and codes attached to the end. This provides Google a ton of information to help them refine and tweak their algorithms.

For example, when you search Qualaroo in Google, the click-through link in the result isn’t qualaroo.com; it’s actually:

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&sqi=2&ved=0CC8QFjAA&url=https%3A%2F%2Fqualaroo.com%2F&ei=AKmzUfCdLdLkqAHrxYGQCg&usg=AFQjCNH5974-G-W96Df9ZGAG6TZUZmovoQ&sig2=hFgiEZk95TlCEa9bsvItIA

This string provides a wealth of information about the keyword and relative link position of the result that was clicked. The two most interesting parts of the string are the “ved” and “cd” sections, where the codes correlate to critical keyword information.

Tim Resnick of Moz gave an incredibly detailed explanation of how to parse this information by configuring Google Analytics advanced segments. You can get step-by-step instructions on how to leverage this data in his post and webinar.

The “ved” portion of the URL provides information on the search vertical and the absolute and relative position of the keyword. You can use advanced segments in Google Analytics to parse out traffic based on these variables to get insights into the keywords and search results that are driving traffic to the site.

ved

(image via Moz)

Resnick and the SEO community have divined some of the more popular combinations of “ved” variables in the table below. If you follow along in the original post’s comment thread, you’ll find plenty of links to additional research and discovery.

vertical universal search code

(image via Moz)

It’s easy to see the implications of parsing this referral data to help optimize your search efforts and content marketing. Say you’re driving a lot of traffic from images but not videos. This information will help you make decisions about how to enhance your multimedia content marketing strategy.

You can choose to combine this approach with the landing page strategy outlined in the previous post on (not provided) to determine which types of results are driving traffic to your top landing pages. Then you can set up goal funnels based on these parameters to see which optimization efforts are driving your real business goals: purchases, signups, and more.

While advanced, it does provide an inside look from Google itself on how and where your (not provided) traffic is coming from. But if that’s making your head hurt, there are easier ways to crack the (not provided) code.

3. Surveying Users

Using on-site surveys is a quick and easy way to ask search visitors how they found you. There are plenty of software options that allow you to quiz your site visitors. We humbly suggest Qualaroo, but the basic theory remains the same for all of them.

Configure your survey tool to display an onsite survey for visitors coming from only search engines. Some products, like Qualaroo, let you configure your settings to display the survey for users of only secure search. But even if it’s just targeted for search in general, you’ll get a more detailed picture of your most popular search terms as reported by your search-sourced visitors.

Here’s how this works. If you search for “product market fit” in Google, you’ll get my blog as one of the top results:

If you click through and your search is a secure one, you’ll get the following when you land on my site:

product market fit

the startup pyramid

(Qualaroo for SEO showing up on Startup-Marketing.com for (not provided) visitors.)

Now I can see which keywords are recorded for secure search. I can use the relative weights of the responses to the number of nudge views to make some informed decisions about the keywords that make up the (not provided) bucket. For example, if 20% of survey respondents answer “product market fit” as their referring keyword string, it’s a pretty good bet that 20% of the (not provided) traffic is from searches for “product market fit,” and so on.

Best of all, you also can see which keywords and phrases drive conversions and purchases by configuring custom events and goals in Google Analytics to tie survey responses to conversions.

4. Instrumenting Site Search

Another easy-ish way to triangulate (not provided) keywords is to instrument your on-site search with Google Analytics. With this approach, you’re capturing the keywords used by visitors to search for information and products within your site to determine what keywords are important to the people that are using your site.

The theory is that people who use your site search are searching for things similar to those searched for by people using Google. While not perfect, it does provide a very clear picture of the types of information visitors think your site should have available. It’s another smart way to get inside the user’s head and find the keywords that matter to your visitors.

You can use this information to create content and optimize your site for the types of things people are looking for on your site. Using a similar methodology in analyzing survey results, you can determine the relative importance and prominence of keywords that your audience is searching for.

If one of your desired keyword phrases isn’t showing up in your site search, it probably means that either your audience doesn’t know the phrase is relevant to your business or the phrase isn’t very important to your audience. Both are actionable pieces of information.

You can instrument your site search in the Admin section of Google Analytics. The Google Knowledge base provides detailed instructions on configuring it properly.

site search setting

(image via Crazy Egg)

Using Query Classification and Advanced Segments

This technique is another one for the more technically minded. It involves using Regex (regular expressions) and advanced segments to parse inbound (not provided) traffic into buckets that provide another level of detail about the traffic.

David Harry built on insights from Google’s resident analytics expert Avinash Kaushik to create this analysis. We refer you to the full post for the brain-spray awesomeness that is his analytics wizardry. The basic idea is to use broader query classification and advanced segments in Google Analytics to bucket users with similar intent, and then use the relative proportion of those visitors to determine the keywords that make up your (not provided) set.

Query classification is the practice of bucketing different types of keywords into similar types of visitor intent. For example, some users might be seeking specific types of information and use searches that help them discover that information; some users could be looking for specific brands and URLs; and others may be searching for specific items to purchase.

Query classifications account for the difference between searching for “eggs benedict recipes,” “epicurious.com,” and “where to find eggs benedict in Orange County.” David Harry’s table below gives you an idea of how this works:

examples of queries

(image via Search News Central)

By using query classifications, you can set up new advanced segments in Google Analytics which allow you to bucket users by search type. For example, you can create segments for your brand searches, head term searches, and long tail searches. This gives you the relative proportion of the types of search traffic you’re receiving.

Then you can look at the segmented search traffic to inbound landing pages to determine the proportion of search types hitting particular pages. Taking this relative weighting by page gives you an idea of the breakdown of (not provided) keyword traffic coming to those pages.

nofollow traffic segments

(image via Search News Central)

The power of setting up this kind of breakdown should be apparent. Now you can parse the (not provided) data set to understand, at the site and page level, which types of keywords are driving traffic and in which proportions.

Putting It All Together

The (not provided) keyword set will continue to grow in size as more and more users use Google products and more products integrate to secure search. But as we’ve just shown, there are creative ways, of varying difficulty, to crack the (not provided) keyword code and reclaim some of your data.

Using surveys, referral string analysis, and some smart segmentation, you can start to divine which keywords are driving your (not provided) traffic, allowing you to make smarter decisions about your search engine optimization and content marketing efforts.

In addition to the four ideas presented here, you can refer to the methods by Claire Broadley, and you can use the research by David Harry, Tim Resnick, and Avinash Kaushik as great jumping off points for further analysis opportunities.

Don’t be a modern-day John Wanamaker. It’s too expensive to not know where 50% or more of your search traffic is coming from. Get insights so you can get more out of your online marketing efforts. You’ll get better results, find more opportunities for improvement, and pass your competition, which is busy bemoaning the lack of insight into their search efforts.

Want some further reading on the subject? These posts are all super helpful in a continuing quest to crack the (not provided) code.

Have another resource or idea? Share with us in the comments. It’s an ongoing process, and Google is constantly evolving. So stay on your toes and look for new opportunities to shed light on the (not provided) black box.

About the Author: Sean Ellis is currently the CEO of Qualaroo, a marketing software company that empowers marketers to better engage, understand and convert their website visitors. Prior to founding Qualaroo, he was the first marketer at Dropbox, Lookout, Xobni, LogMeIn (IPO), and Uproar (IPO) and also held interim marketing executive roles at Eventbrite, Socialcast, and Webs. Follow him on Twitter.

  1. Deep coverage right there. Great resource(s). Qualaroo looks useful, surveying visitors is smart practice. It gave me a fancy idea to test a similar, ‘what did you search’ audio survey using SpeakPipe. You never know. Any feedback on the reliability of WordPress plugins that track and post incoming search queries?

  2. Is this possible to track keyword in (not provided)? I don’t understand what’s defined above, is it use of UTM parameters to find (not provided) keywords?

  3. Detailed ( Understandable by Mid level and Pro Level )but hard to understand for newbies i think video blog will help them to learn.. Well this posts will definitely helps you to score you marketing efforts.

  4. Hey Sean,

    Question for you. Assuming it’s distributed evenly, why wouldn’t you just apply the keyword distribution for your known traffic to the unknown traffic to get a rough approximation?

    Is the thought that those using private search are searching for an entirely different set of keywords?

    I mean, I understand you could potentially be losing insight into your long-tail keywords but wouldn’t still be representative?

    -Matt

  5. @Matt I agree that the tracked keywords can be extrapolated to give you a pretty good idea of search distribution. Ultimately you are trying to compile the best picture possible of what is going on so every piece of relevant additional data helps. One of the things I’ve started doing in the last couple of days with Qualaroo is to tie the search terms to a longer form explanation of exactly what the user was trying to find. That helps me going beyond what I was even able to do before secure search. Hope this helps.

  6. As more and more users use secure search (and with recent NSA revelations they will) its pretty apparent that SEO tracking will quickly move out of the capability of the average analytics user, and be for specialists only.

    Of course, the other possibility is that KISSmetrics and other providers will create automated tools to do this for average users.

  7. Today itself, i read your article on how to unlock ‘not provided’ keyword. I was able to crack it to some extent. I will follow these four more techniques and mention the results.

  8. Great post really useful information on the ‘not provided’ keywords. Thanks

  9. Cool post. It’s a very essential post for every networking and internet marketing experts. I think you can more clarify the point of “Surveying Users”. Anyways thanks A lot :)

  10. Unfortunately Google will be making a change so to completely not provided for everything. This is going to be a huge blow to the industry of SEO when you won’t be able to see what people visited your site for, or how they found you.

  11. Josh, this is a very bad news. How do we able to extract and know the behaviour of our visitors? Is there any alternative to this?

  12. Deeply coverage in front of them. Great resource(s). Qualaroo appears useful, surveying visitors is smart practice. It afforded me any fancy idea to evaluate a comparable, ‘what does you search’ sound survey employing SpeakPipe. You never know. Any feedback about the reliability of WordPress extensions that trail and publish incoming look for queries?

  13. Comprehensive ( Simple to comprehend by Core level and also Pro Degree )but hard to recognise for newbies i think video blog will help them to know.. Well this kind of posts will certainly helps you to score an individual marketing endeavours.

  14. Hi, thanks for the post.

    If we use the search term exclusion list with Universal Analytics, do you know if the keyword is excluded even though it is not provided – or not ?

    For example let’s say I want to exclude my own brand : will the search terms report include in the not provided section people who came from organic trafic, but using my brand as a keyword ?

  15. Targeting the wrong keywords is like burning your $ and effort in ranking those terms. However, just like John said “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.”, we always learn from our mistakes and improve our keyword research next time. What has been working for me is to think from customers’ perspective – if I were the customer, what would I search to find for “target keyword” I want….
    Tony Nguyen

  16. Hi, thanks for the post.

    If we use the search term exclusion list with Universal Analytics, do you know if the keyword is excluded even though it is not provided – or not ?

    For example let’s say I want to exclude my own brand : will the search terms report include in the not provided section people who came from organic trafic, but using my brand as a keyword ?

  17. Targeting the wrong keywords is like burning your $ and effort in ranking those terms. However, just like John said “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.”, we always learn from our mistakes and improve our keyword research next time. What has been working for me is to think from customers’ perspective – if I were the customer, what would I search to find for “target keyword” I want….
    Tony Nguyen

    • Tony, thanks for the great points. It’s all about getting the right keywords right off the bat :)

Comments are closed.

← Previous ArticleNext Article →