Complete guide to Direct Traffic Analysis in Google Analytics

 

Direct traffic is increasing day by day and if you don’t do anything about it now, then one day all you will see in your analytics reports will be direct traffic: little to no referral data, just direct traffic and your favourite ‘not provided’ keywords.

I am not kidding. Every major website/browser/search engine is moving to secure connection (https://) while your website is still using http. So what will happen is not so distant future is that you will stop getting referral data. This is because by default HTTPS websites do not send referrer data to non-HTTPS (or HTTP) websites.

You need to capture as much referral data as possible if you wish to continue to do meaningful analysis in the distant future too. This article is all about how to do that.

But first let us redefine what direct traffic really is:

Direct traffic is a web session (or visit) which starts without a referrer being passed by the user browser.

This is the most accurate definition of direct traffic.

All other definitions of direct traffic are lame like: direct traffic is a traffic which comes from bookmarks or traffic from typed URLs etc. as they don’t accurately describe what direct traffic really is.

In order to understand how direct traffic works, you first need to understand the concept of ‘referrer’. In order to understand ‘referrer’ you first need to understand HTTP. So let us first start with HTTP and then we will gradually move to ‘referrer’.

 

HTTP and Referrer

HTTP or Hypertext Transfer Protocol is a protocol (set of rules) used by web browsers and web servers to communicate with each other.

A web browser communicates with a web server by sending HTTP requests for each requested resource (html document, image file, css file, JavaScript file etc). A web server communicates back by sending HTTP responses for each HTTP request.

Each HTTP request is made up of request line and request headers (also called HTTP request headers).

Similarly, each HTTP response is made up of status line and response headers (also called HTTP response headers).

Here is how HTTP request looks like: 

HTTP Request Headers

This request was made to my web server (where my website is hosted) by a user’s browser when he clicked on a link to one of my blog post (You are doing Google Analytics all wrong. Here is why) from my Google+ page.

As so you can see from the screenshot above, referrer is just one of the HTTP request header and it is ‘Google Plus’ in this case:

https://plus.url.google.com/url?sa=z&n=1400603604168&url=http%3A%2F%2Fwww.seotakeaways.com%2Fgoogle-analytics-wrong-why&usg=ZSnkIqTswUCpggL84jvJcY3KXfk

Now since the request was made to my web server, my server has to communicate back (it has to obey HTTP protocols) to the user’s browser. It does so by sending HTTP response.

Here is how HTTP response looks like:

HTTP Response Headers

This response was sent by my web server to the user’s browser.

Now if you want to see these HTTP request and response headers, you just can’t see them by viewing the source code of a web page. You won’t find them there.

I use a tool called ‘Chrome Developer Tools’ to see HTTP requests and responses for a particular web page.

This tool is already built into Google Chrome browser. So as long as you are using Google chrome, all you have to do is right click on a web document and then select ‘inspect element’:

inspect element

This will open developers tool window at the bottom of your browser.

Now click on the ‘Network’ tab and then double click on the URL of your webpage:

networks tab 1

 

networks tab

On the right hand side of the developers window, you can see all the HTTP request and response headers. This is the window from which you can find out whether a referrer was passed or not.

Note: cookie is one of the HTTP request header. So cookie data is sent via HTTP request to a web server.

If your website is on https:// and you are linking out or redirecting to another https:// website, your browser will send referrer data.

Similarly, if your website is on https:// and you are linking out or redirecting to a https:// website, your browser will send referrer data.

 

HTTPs and Referrer

HTTPS or Hypertext Transfer Protocol Secure is simply a secure version of the HTTP.

If your website is on https:// and you are linking out or redirecting to a http:// website, your browser will not send referrer data because of security reasons. That means all of the websites you are sending the traffic to will see your traffic as direct traffic.

However if your website is on https:// and you are linking out or redirecting to another https:// website, your browser will send referrer data because both websites are using secure connection.

when referrer data is passedSo if one day all search engines, browsers and social networks move to secure connection (https://) and they will while your website won’t then you will lose majority of your referrer data for good as HTTPS websites don’t pass referrer data to HTTP websites by default.

I used the word ‘default’ because it is possible to send referrer data from a HTTPS website to an HTTP website.

For example Google+, Facebook and twitter are already sending referrer data to HTTP websites and they all use https://

How they are able to do that?

They all use an internal redirect script that first redirects a visitor to a non-HTTPS page (which creates its own referrer data) before sending the visitor to the actual URL on a HTTP website.

For example,

If you click on this link: http://www.optimizesmart.com/excel-seo-powerful-cheat-sheet-boost-productivity/
from my Google+ page: https://plus.google.com/109426632283601974817/posts

you will first be temporarily redirected to the following URL:

http://plus.url.google.com/url?sa=z&n=1400603253616&url=http%3A%2F%2Fwww.optimizesmart.com%2Fexcel-seo-powerful-cheat-sheet-boost-productivity%2F&usg=Yr4dTmDnq34M5t5_Y93W7CqqLJk

Then you will be redirected to http://www.optimizesmart.com/excel-seo-powerful-cheat-sheet-boost-productivity/

All of this will happen so fast, that you can’t easily identify that you were first redirected to an intermediate non-HTTPS page.

Here Google+ is not sending the original referral data from a HTTPs website to a HTTP website thus honouring the Secure Protocol which states that:

If a website is accessed from a HTTP Secure (HTTPS) connection and a link points to anywhere except another secure location, then the referer field is not sent.

Source: https://en.wikipedia.org/wiki/HTTP_referer

So what Google+ is doing here is creating its own referrer data by using an intermediate page (http://plus.url.google.com/url?) something like this:

<a href=”http://plus.url.google.com/url?sa=z&n=1400603604168&url=http%3A%2F%2Fwww.seotakeaways.com%2Fgoogle-analytics-wrong-why&usg=ZSnkIqTswUCpggL84jvJcY3KXfk”>https://www.optimizesmart.com/excel-seo-powerful-cheat-sheet-boost-productivity/</a>

Facebook and twitter follow the same tactic. They create and send their own referrer data instead of the original referrer data.

Facebook uses facebook.com/l.php as an intermediate page whereas twitter uses t.co/ as an intermediate page.

Now since these social networks don’t send the original referrer data they can rewrite the referrer data whatever way they want and can hide any information they like. Another advantage of using this tactic is that these social networks can easily hide personally identifiable information and can thus protect users’ privacy.

Google Search Engine also use the same tactic as it redirects you through another URL. That is why even when being on a secure connection, Google is able to send you referrer data. Since Google doesn’t send you the original referrer data, it can also rewrite the referrer data whatever way it wants and can hide any information it likes.

So Google choose to hide the ‘keyword’ data from its referrer.

You can use the same tactic to send referrer data from your HTTPS website to HTTP websites.

If you want to know what referrer data search engines like Google, Bing and Yahoo are sending to a HTTP website, then simply search for ‘what is my referrer’ on a search engine and then click on the website: http://www.whatismyreferer.com/

Similarly, If you want to know what referrer data search engines are sending to a HTTPS website, then simply search for ‘what is my secure referrer’ on a search engine and then click on the website: https://referer.rustybrick.com/

For example:

If I search for the keyword ‘what is my referrer’ on the HTTP version of Bing http://www.bing.com/ and click on the website: https://www.whatismyreferer.com/ , I get following referrer data:

what is my referer 1But if I search for the keyword ‘what is my referrer’ on the HTTPS version of Bing https://www.bing.com/ and then click on the website: http://www.whatismyreferer.com/ , I get no referrer data:

what is my referer 2This is because as mentioned earlier, a HTTPS website doesn’t pass referrer to a HTTP website.

But if I search for the keyword ‘what is my secure referrer’ on the HTTPS version of Bing https://www.bing.com/ and then click on the website: https://referer.rustybrick.com/ , I get following referrer data:

what is my referer 3

This is because a HTTPS website pass referrer data to another HTTPs website.

 

Move your entire website to secure connection (HTTPS)

You should move your website to secure connection ASAP even if your website is not e-commerce and you are not collecting any sensitive data. In this way you will be able to track referrals from HTTPS websites which you won’t be able to otherwise. For example in case of Bing secure search, you will be able to get the referral data which is not available otherwise.

Because of this reason, I moved my entire website to secure connection, even when it is a not an e-commerce website.

In order to move your website to HTTPS you will need to purchase an SSL certificate for your domain.

Make sure that the certificate you purchase support mobile, your CDN supports SSL, your SSL supports SPDY and all of your images, CSS files, JavaScript files etc use HTTPS too. For more details about the certificate, contact your web host.

 

Make special efforts to send the referrer data to non HTTPS websites

One thing worth pointing out is that, sites like Facebook, Google and Twitter will never hide all of the referrer data from you because of vested interest. They don’t want people esp. advertisers to start believing that because they no longer send any traffic (as without referrer, traffic will be treated as direct traffic by GA), they are losing popularity and hence are not good advertising platform.

Similarly, if you run an affiliate website and you are using https, you would not want your advertisers to believe that your site doesn’t send any traffic. Hence it is very important that you make special efforts to send referrer data to non HTTPS websites once you have moved to secure connection.

 

Situations when the referrer is not passed

In order to understand how direct traffic is generated you must be familiar with the various situations when the referrer is not passed.

Whenever a referrer is not passed, the traffic is treated as ‘direct traffic’ by Google Analytics.

So if you can make sure that the referrer is being passed wherever you can, you will be in a better position to purge your direct traffic.

In following situations, a referrer is not passed: 

1. Bookmarks – if you visit a web page via bookmark, it means you have been on the web page before (unless you have imported the bookmarks from another browser). That means a campaign cookie has already been set on your hard disk. Campaign cookie stores campaign information like source and medium of the visit.

So for example if a user lands on your web page via Google organic search, bookmarked the page and then later returned to the page (within 6 months) via bookmark, then his visit will still be treated as visit from organic search and not direct.

This is because campaign cookie expires after 6 months. If however the user returned to your web page via bookmark after 6 months, then his visit will be treated as direct visit as the campaign cookie would have expired by then.

2.Apps – majority of mobile, desktop and social media applications don’t send a referrer. So traffic from such apps is treated as direct traffic by Google Analytics.

3. Non web documents (Microsoft Word, Excel, PowerPoint, PDF etc) don’t send a referrer. So traffic from such documents is treated as direct traffic by Google Analytics.

4. Browsers issues – sending of the referrer data depends entirely on the web browser and redirect method being used. If the user browser doesn’t pass the referrer data then there is no way to get it.

For example Internet Explorer 8 loses referrer data when following methods are used a redirect methods: Javascript:location.href and Meta refresh – 0.

Similarly Internet Explorer loose referrer data when you click on a link that uses ‘windows.open’ JavaScript method or when you click on a link embedded in a flash application.

5. Redirect methods – sending of the referrer data depends a lot on the redirect method being used. For example if you use following redirect methods then there is no guarantee that all web browsers will pass the referrer data: Javascript:location.href, Javascript:location.replace, windows.open and meta refresh.

Use server side redirects (301 and 302) instead of Meta and JavaScript redirects to redirect visitors and search engines as they let all major web browsers pass the referrer data.

Give preference to 301 over 302 wherever you can, as 301 redirects have better cross browser compatibility when it comes to passing the referrer data. 302 redirect often cause the referrer to be dropped.

6. Browser privacy settings and add-ons – Sometimes a user’s browser privacy settings and/or add-ons used (like Firefox No Referrer Add on, ScriptSafe for Chrome) can cause the referrer header to drop and not passed. Private browsing, incognito mode, in private browsing and other similar settings don’t let the browser to pass the referrer data.

Here one thing worth noting is that Privacy settings and add-ons can’t stop search engines like Google to send referrer data because they don’t send original referrer header in the first place as mentioned earlier in this article.

That is why they are also able to bypass the HTTP secure connection protocol and can pass referrer from HTTPS to non-HTTPS website. Wicked smart.

7. IOS open in … – When you use an option like ‘open in safari’ or ‘open in chrome’ in IOS, the referrer data is not passed.

8. HTTPS to HTTP – During HTTPs to HTTP redirect and vice versa, referrer is not passed by default.

9. NoReferrer Attribute – Referrer is not passed in case of visits that came from a link which uses the ‘rel=noreferrer’ attribute.

For example: <a href=”https://www.example.com/about/” rel=”noreferrer”>No referrer for you</a>

10. Desktop Email Clients – visits that came from a link within a desktop email clients (like Microsoft Outlook) are all treated as direct visits.

11.Improperly tagged marketing campaigns can cause the referrer data to drop.Google analytics ignore the referrer when it sees a campaign tracking code.

For example if you share a tagged link on twitter with ‘utm_source=facebook’, then all the twitter visits will be treated as ‘facebook visits’ by Google Analytics.

But if you went one step further and used something like ‘utmSource=facebook’ (which is not a correct campaign tracking variable) while tagging the URL then Google Analytics will completely ignore the referrer and treat twitter visits as direct visits.

12. IM – visits that came from a link within an instant messenger (like Skype, Google Hangout) and online chat rooms are all treated as direct traffic.

13. Code errors – sometimes errors in your script can cause the Google Analytics cookies to be reset, thus resulting in a direct visit.

14. Firewall – sometimes your company’s firewall settings can cause the referrer to drop.

15. Type in traffic – sometimes users try to visit your website by typing your brand name followed by .com in the address bar under the assumption that this will be your website address. If they succeed in visiting your website, all such traffic is treated as direct traffic by Google Analytics.

 

Methods to purge your Direct Traffic

1. Tag the URLs of all marketing campaigns – one of the most popular and widely used method to clean up direct traffic is to tag each and every URL of your marketing campaigns with campaign tracking parameters: utm_source, utm_medium, utm_term, utm_content and utm_campaign.

Following is an example of tagged URL: https://www.optimizesmart.com/google-analytics-cookies-ultimate-guide/?utm_source=facebook&utm_medium=social&utm_campaign=article-promotion

Same URL when untagged will look like this: https://www.optimizesmart.com/google-analytics-cookies-ultimate-guide/

2. Tag each marketing campaign correctly – as mentioned earlier if you share a tagged link on twitter with ‘utm_source=facebook’, then all the twitter visits will be treated as ‘Facebook visits’ by Google Analytics. Similarly if you used incorrect campaign parameters (like UTMSource=facebook) then Google analytics will completely ignore the referrer and treat referral visits as direct visits. Use Google URL builder in order to tag campaign URLs correctly.

3. Use shortened tagged URLs if you are deploying contents via non-html documents (word, excel, powerpoint, PDF etc) and/or desktop email clients. Campaign tagging parameters make a URL looks ugly to share.

So some people either don’t share such URLs or remove the campaign tagging parameters before they share. Moreover if you are deploying contents via non-html documents, it won’t look good to embed URLs with tracking parameters: https://www.optimizesmart.com/google-analytics-cookies-ultimate-guide/?utm_source=microsoft-word&utm_medium=non-html-document&utm_campaign=article-promotion.

word referrer

The best way to embed such URLs is through a URL shortening service like bit.ly. For example: https://bit.ly/1ibcdZl

4. Keep different browser referral issues, privacy settings and add-ons in mind while redirecting visitors and search engines and always use server side redirects (301 and 302) instead of Meta and JavaScript redirects.

5. Move your website to HTTPS as explained above.

6. Make sure there is no rel=”noreferrer” on your website links especially if you are an affiliate.

7. Make sure that your code doesn’t cause Google Analytics cookies to be reset, thus resulting in a direct visit. This is one of the most overlooked issue and often hard to diagnose.

8. Check your company’s firewall settings to make sure that the referrer is not dropped.

9. Block your own company’s traffic from being tracked by Google Analytics by installing Google Analytics Opt-out Add-on. Blocking traffic by IP doesn’t work well unless you are on a static IP. Often internal traffic end up as direct traffic.

10. Segment your direct traffic into two categories. The first category includes desktop and tablet direct visits. The second category includes mobile direct visits. You can create these categories by using advanced segments in Google Analytics. This will give you a good idea of what your direct traffic is made up of and what you can do about it.

11. Look for correlation between your direct visits and marketing campaigns. Often when we launch a new marketing campaign (online or offline) there is a considerable increase in direct traffic. You need to note down all such changes in direct traffic through Google Analytics Annotations. So that later you can attribute traffic and conversions through direct traffic in a particular time period to your marketing campaign.

12. If your website has been set up mainly to generate leads through phone calls (quite common in case of websites which sell high priced items like properties, cars, yacht, consultation services etc.) then you will miss all the referral data if you don’t implement a phone call tracking solution.

13. Don’t just rely on Google Analytics to capture referral data esp. if you are tracking mobile apps. There are lot of third party tools (like Mobile App Tracking) out there which can track referral data much more accurately than Google Analytics.

14. Be innovative and device new ways to capture referral data. For example you can capture referral data through lead generation forum, on-page surveys, email surveys, contests etc.

Related Posts:

How to determine the impact of Organic, PPC, Social Media and Email on Direct Traffic

How Google attributes conversions to direct traffic


Quick announcement about my new book on Web Analytics and Conversion Optimization

The role of maths and statistics in the world of web analytics is not clear to many marketers and web analysts. Not many talk or write about the usage of statistics and data science in conversion optimization. That is why I have written this book Maths and Stats for Web Analytics and Conversion Optimization to fill this knowledge gap.

This expert guide will teach you, how to leverage the knowledge of maths and statistics in order to accurately interpret the web analytics data and take business and marketing decisions which can quickly improve the bottom-line of your online business. Read first few chapters of this book for free here.


 

Himanshu Sharma

Certified web analyst and founder of OptimizeSmart.com

My name is Himanshu Sharma and I help businesses in finding and fixing their Google Analytics and conversion issues.
  • More than ten years' experience in SEO, PPC and web analytics
  • Certified web analyst (master level) from MarketMotive.com
  • Google Analytics certified
  • Google AdWords certified
  • Nominated for Digital Analytics Association Award for Excellence
  • Bachelors degree in Internet Science
  • Founder of OptimizeSmart.com and EventEducation.com
I am also the author of the book Maths and Stats for Web Analytics and Conversion Optimization If you have any questions or comments please contact me