Google Analytics Measurement Protocol & Hit Builder

 

What is Google Analytics measurement protocol?

Measurement protocol is a set of rules which your application must follow in order to send raw hit data directly to Google Analytics server.

These are the rules regarding:

  1. formatting the protocol requests
  2. length and format of the payload data
  3. length and format of each parameter
  4. which parameters can be sent together and which can’t be sent together
  5. which parameters are required for particular hit type.
  6. which parameters can be sent for a particular hit type.
  7. allowed key and value pairs
  8. valid keys
  9. valid hit types
  10. length and format of each value
  11. supported data types
  12. format of transport

 

What is the advantage of using the measurement protocol?

First thing first. Every time you send a hit to Google Analytics, you use the measurement protocol. You can’t send hit data to GA without using measurement protocol. So you are already using the protocol.

When someone refer to ‘using’ the measurement protocol explicitly, they are actually talking about constructing a measurement protocol hit.

In that context, measurement protocol is used and beneficial for implementing cross device tracking.

Through this protocol you can collect and send raw user interaction data (hits) directly to GA server from any device/system which can be connected to the internet. This device/system can be a smart phone, tablet PC, digital appliances (washing machine, coffee machine, slot machine), point of purchase systems etc.

However, you need to format your data according to the measurement protocol before you can send it to the Google Analytics server. So for example, if you want to send data from your washing machine to the Google Analytics server, then you first need to format the data according to the measurement protocol.

Similarly, if you want to send data from a cash counter machine, coffee machine or slot machine then you first need to format the data according to the measurement protocol. Without such formatting, you can’t send data to GA.

Through measurement protocol you can measure offline conversions in GA. Thus it helps in understanding the offline behaviour of the customers and help in fixing attribution modelling issues.

Note: You don’t need measurement protocol, if you just want to send hit data from a website or mobile app. Use the Google Analytics tracking code for website/mobile app for that purpose.

 

What type of data you can/can’t send via measurement protocol?

You can collect and send only hit data (pageviews, screenviews, events, transactions etc) via the measurement protocol. You can’t collect and send data in aggregated form (like tables) via measurement protocol.

 

How to use measurement protocol?

measurement protocol

#1 Ask your developer to write an application that can pull data from an external data source (like washing machine, point of purchase system etc).

#2 Your application must convert the retrieved data into the payload data according to the measurement protocol.

#3 Your application must make a HTTP POST/GET request to the end point (GA server) in order to send the payload data to the GA server. The HTTP request is made up of user agent, transport and payload data.

#4 When GA receives the HTTP request from your application, it try to process the payload data in the HTTP request.

#5 If GA successfully processes the payload data then its server will send 2XX response code back to your application. If GA is not successful, in processing the payload data, then its server does not send any error code back to your application.

 

Measurement Protocol Usage Policy

According to measurement protocol usage policy, you should use measurement protocol only when:

#1 You have the permission to use it from the rightful owners.

#2 You are not going to collect any personally identifiable information (names, email addresses) or any data that can help in identifying a particular user by his/her device.

#3 You have given your end users, proper notice about the data you are collecting via the measurement protocol and how you will use it, as well as the option to opt out from being tracked.

 

Measurement Protocol Requests

The measurement protocol request is an HTTP request which is made up of user agent, transport and payload data.

Some background about HTTP requests & response

All web browsers and web servers communicate with each other by sending HTTP requests and HTTP responses. A web browser communicate with a web server by sending HTTP request for each request resource (like html document, image, css etc).

The web server communicated back to the web browser by sending HTTP response for each HTTP request.

Here is how HTTP request looks like:

http request

Here is how HTTP response looks like:

http response

To see HTTP requests and responses, open the Chrome Developers tool (right click on a page and select inspect element) and then click on the ‘Network’ tab:

network tab

On the right hand side of the developers window, you can see all the HTTP request and response headers. This is the window from which you can see the payload data sent via measurement protocol.

payload data network tab

In order to make an HTTP request to the GA server for sending the raw hit data directly, you need to format your HTTP request according to the measurement protocol.

Here is how the formatted HTTP request aka measurement protocol request looks like:

protocol request

 

Measurement Protocol Payload data

payload data

Payload data is the data you send to Google Analytics server using the measurement protocol.

In other words, the payload data is the data which has been formatted according to Measurement Protocol.

Following is an example of payload data:

v=1&tid=UA-123456-2&cid=36009a59-2a05-49e7-b826-2b884d0f935b&t=pageview&dp=%2Fhome

The payload data resembles a URL query string (query string is the part of the URL which comes after the ‘?’ Character) and is made up of several parameters.

 

Types of Payloads

There are two types of payloads:

#1 Single hit payload – the payload data in which only one hit is sent to the GA server.

#2 Multiple hits payload – the payload data in which multiple hits are sent to the GA server.

Note: The size of single hit payload can’t be greater than 8k bytes.

 

Measurement Protocol Parameters

Payload data is made up of multiple parameters. Each parameter is separated from another parameter via & character.

Each parameter is made up of ‘key = value’ pairs. So each ‘key=value’ pair is separated by ‘&’ character.

Syntax of payload data:

parameter1&parameter2&parameter3 

or

key1=value1&key2=value2&key3=value3

Example of payload data:

v=1&tid=UA-123456-2&cid=36009a59-2a05-49e7-b826-2b884d0f935b&t=pageview&dp=%2Fhome

Here v, tid, cid, t and dp are the keys.

1, UA-123456-2, 36009a59-2a05-49e7-b826-2b884d0f935b, pageview and %2Fhome are the values.

All values you sent via measurement protocol to GA must be both UTF-8 encoded and URL encoded.

Note: You can learn more about URL encoding from here: http://www.w3schools.com/tags/ref_urlencode.asp

For example, / is encoded to %2F

If the values are not encoded correctly then they will be replaced with unicode replacement character xFFFD

Note:  All protocol parameters must be set on the same line.They should not have any spaces between them. Comments are not allowed in protocol parameters. 

 

Each payload data must contain following parameters

1. v – It is used to denote the measurement protocol version. The value of this parameter should be 1. For example: v=1

2. tid – It is used to denote the tracking ID. The value of this parameter should be the property ID of the Google Analytics property where you want to send the payload data. For example: tid=UA-123456-2

3. cid –  It is used to denote the client ID (ID which anonymously identifies a particular device, or browser and is unique to a particular visitor/user). For example: cid = 36009a59-2a05-49e7-b826-2b884d0f935b

4. t – It is used to denote the hit type. The value of this parameter should be anyone of the following: ‘pageview’, ‘event’, ‘social’, ‘transaction’, ‘item’, ‘exception’,’appview’ or ‘timing’. For example: t= pageview or t= event or t= social or t= transaction or t= item

 

Rules for using measurement protocol parameters

#1 There are rules regarding which parameters can be sent together and which parameters can’t be sent together.

#2 There are rules regarding the length and format of parameters

#3 Each parameter has specific type that requires a particular format.

#4 Each payload data must contain a valid hit type

#5 Each hit type must have it own set of required parameters set.

For more details regarding these rules, refer to:

Measurement Protocol Parameter Reference: https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters?hl=en 

 

List of Parameters which are commonly used in payload data

  Parameter Description Value Example
1 dr Document Referrer. It is the referral source which sent traffic to your website. The value of this parameter should be a URL (URL encoded) dr=http%3A%2F%2Fabc.com

All values of the parameters must be URL encoded before they are sent to the Google Analytics server. For this reason the URL https://abc.com was first encoded to http%3A%2F%2Fabc.com before it is passed as a value to the ‘dr’ parameter.

2 cn campaign name The value of this parameter should be a URL encoded text string. cn=attribution

 

3 cs campaign source URL encoded text string. cs=%28direct%29

Here the campaign source is ‘(direct)’. But when the text string is URL encoded the opening parenthesis ‘(‘ is replaced by ‘%28’ and the closing parenthesis is replaced by ‘%29’

4 cm campaign medium URL encoded text string. cm=organic
5 ck campaign keyword URL encoded text string ck=event%20education
6 cc campaign content URL encoded text string cc=top%20left
7 dl document location,

the full URL of the page which contains the contents.

URL (URL encoded) dl=http%3A%2F%2Fexample.com%2Fhangers%3Fprod%3D231

Here the original value (before URL encoding) was: https://example.com/hangers?prod=231

8 dh document hostname,

the host where the contents are hosted.

Hostname dh=example.com
9 dp document path The value of this parameter should begin with /

 

dp=%2Fhangers%3Fprod%3D231

Here the original value (before URL encoding) was: /hangers?prod=231

10 dt document title URL encoded text string dt=event%20education%20courses

Here the original value (before URL encoding) was: event education courses

11 ec Event category URL encoded text string ec=videos
12 ea Event Action URL encoded text string ea=play
13 el Event Label URL encoded text string el=spider%20man
14 ev Event Value Integer value ev=100
15 ti e-commerce transaction ID URL encoded text string ti=4567DT
16 ta Transaction affiliation,

denotes the store name

URL encoded text string ta=tesco
17 tr Transaction revenue Currency tr=45.56
18 ts Transaction shipping Currency ts=2.32
19 tt Transaction tax Currency tt=10.11
20 in Item name, the name of the product item URL encoded text string in=hangers
21 ip Item price, the price of a single product item Currency ip=34.21
22 iq Item quantity, the number of product items purchased. Integer iq=45
23 ic Item code, this is the product SKU URL encoded text string ic=23ds53
24 iv Item category, the category to which the item belongs to URL encoded text string iv=retail%20display
25 cu Currency code, the currency used for all the transactions URL encoded text string cu=EUR

Note (1): All text values have size limit which is expressed in bytes.

Note (2): You can get complete list of all of the measurement protocols form here: https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters

 

Measurement Protocol supported data types

The measurement protocol supports following data types:

  1. Integer – used to represent a number
  2. Text – used to represent a string
  3. Boolean – used to represent a boolean value (true or false)
  4. Currency – used to represent the total value of a currency in up to 6 decimal points.

These data types are for the values in the ‘key = value’ pairs of the payload data.

 

When payload data is not processed by Google Analytics?

A payload data is not processed by GA, when it is:

  1. malformed (not correctly formatted)
  2. contains incorrect keys or values

If a parameter is malformed and/or is not required for a particular hit, then that parameter will be ignored and not processed by GA. All other parameters will be processed as normal.

However if a required parameter (the parameter which is required for a particular hit) is malformed, then the entire payload data will not be processed by GA.

Note: When your application makes HTTP request to the GA server and the payload data is successfully processed by GA, then the GA server return a 2XX server response code. However, the GA server does not return any error code, if the payload data is not successfully processed by GA.

 

Measurement Protocol Transport

transport

Transport defines where and how to send the payload data.

A transport is made up of:

#1 GET/POST method – which defines how to send the payload data

#2 Location of the GA server – which defines where to send the payload data

#3 URL endpoint – this is used to define whether a single or multiple hits should be sent to the GA server or whether to send the payload data to measurement protocol validation server.

 

GET/POST method

You can send payload data by using either GET or POST method. Google recommends using POST method, as then you can send larger payload data to GA server.

// Size of Payload data (if you are making HTTP POST request to the GA server): 8191 Bytes.

// Size of Payload data (if you are making HTTP GET request to the GA server): 2000 Bytes.

// Format of Payload data (if you are making HTTP POST request to the GA server): URL encoded.

// Format of Payload data (if you are making HTTP GET request to the GA server): URI escaped query parameters.

 

Sending Payload data via POST method

Write your application to make the following HTTP POST request to the GA server:

User-Agent: user_agent_string
POST http://www.google-analytics.com/collect?payload_data

If you want to send the payload data via a secure connection (over SSL), the make the following HTTP POST request to the GA server:

User-Agent: user_agent_string
POST https://ssl.google-analytics.com/collect?payload_data

 

Sending Payload data via GET method

If you can’t send payload data via HTTP POST then write your application to make the following HTTP GET requests to the GA server:

GET /collect?payload_data HTTP/1.1
Host: http://www.google-analytics.com
User-Agent: user_agent_string

If you want to send the payload data via a secure connection (over SSL), the make the following HTTP GET request to the GA server:

GET /collect?payload_data HTTP/1.1
Host: https://ssl.google-analytics.com
User-Agent: user_agent_string

One downside of making an HTTP GET request to the GA server is that, GET request may get cached and when it get cached, the request is no longer unique and hence subsequent requests are retrieved from the cache and not sent to GA.

To prevent HTTP GET requests from being cached, add the parameter called ‘Z’ at the end of the payload data. Assign a random number to this parameter. For example:

GET /collect?[payload_data&z=12345] HTTP/1.1
Host: https://ssl.google-analytics.com
User-Agent: user_agent_string

For example:

https://www.google-analytics.com/collect?v=1&t=pageview&tid=UA-1029844-34&cid=c461d1ba-d341-499f-bc9e-673b67f2aa18&dh=mydemo.com&dp=%2Fhome&dt=testing%20page1111111%20tracking&z=12345

 

Location of the GA server

Generally the payload data is sent to:

https://www.google-analytics.com/collect

However if you want to send the payload data via a secure connection (over SSL), then you need to send the data to:

https://ssl.google-analytics.com/collect

 

URL endpoint

There are 3 types of URL endpoints:

#1 /collect

#2 /batch

#3 /debug/collect

/collect

use this endpoint to send a single hit in your HTTP request. For example:

https://www.google-analytics.com/collect?v=1&t=pageview&tid=UA-1029844-34&cid=c461d1ba-d341-499f-bc9e-673b67f2aa18&dh=mydemo.com&dp=%2Fhome&dt=testing%20page%20tracking

 

/batch

use this endpoint to send multiple hits in a single HTTP request. For example:

https://www.google-analytics.com/batch?

v=1&t=pageview&tid=UA-1029844-34&cid=c461d1ba-d341-499f-bc9e-673b67f2aa18&dh=mydemo.com&dp=%2Fhome12&dt=testing%20page12%20tracking

v=1&t=pageview&tid=UA-1029844-34&cid=c461d1ba-d341-499f-bc9e-673b67f2aa18&dh=mydemo.com&dp=%2Fhome13&dt=testing%20page13%20tracking

v=1&t=pageview&tid=UA-1029844-34&cid=c461d1ba-d341-499f-bc9e-673b67f2aa18&dh=mydemo.com&dp=%2Fhom14&dt=testing%20page14%20tracking

Note: when you are sending multiple hits in a single HTTP request then specify each payload on its own line

 

/debug/collect

use this endpoint to send hit to the measurement protocol validation server instead of the GA server. For example:

https://www.google-analytics.com/debug/collect?v=1&t=pageview&tid=UA-1029844-34&cid=c461d1ba-d341-499f-bc9e-673b67f2aa18&dh=mydemo.com&dp=%2Fhome&dt=testing%20page%20tracking

Note: If you are sending hits to validation server then they won’t show up in your GA reports.

 

Batch Requests

The HTTP request which contains /batch end point is called the Batch request.

You can send maximum of 20 hits at a time per batch request. The total size of the payload data in a batch request can not be greater than 16k bytes.

 

Validating measurement protocol hits

Following are the various methods you can use to validate measurement protocol hits:

#1 Google Analytics Real time reports

#2 Chrome developer tool

#3 Measurement protocol hit builder

#4 Postman chrome extension

#5 Directly using the measurement protocol validation server.

 

#1 Using GA Real time reports for validating protocol hits

This is the easiest method to use. Here is how the following page load data would appear in GA real time reports:

https://www.google-analytics.com/collect?v=1&t=pageview&tid=UA-1029844-34&cid=c461d1ba-d341-499f-bc9e-673b67f2aa18&dh=mydemo.com&dp=%2Fhome&dt=testing%20page%20tracking

testing page tracking

Similarly, here is how the following page load data (sent over secure connection) would appear in GA real time reports:

https://ssl.google-analytics.com/collect?v=1&tid=UA-1029844-34&cid=36009a59-2a05-49e7-b826-2b884d0f935b&t=pageview&dp=%2Fhome1234

home1234

 

#2 Using Chrome developer tool to validate protocol Hits

Chrome developer tool provides much more details about the measurement protocol hits than GA real time reports.

You can use the ‘console’ or ‘network’ tab to see the details of the hits, as explained earlier.

 

#3 Using Measurement Protocol Hit Builder to validate protocol hits

Through measurement protocol hit builder you can create and validate protocol hits. Follow the steps below to use this tool:

Step-1: Go to measurement protocol hit builder: https://ga-dev-tools.appspot.com/hit-builder/

Step-2: Click on the ‘click here to authorize’:

click here to authorize

Step-3: Sign in with your Google account. You can now use the hit builder

Step-4: Just to give you an example, I will create and validate following event hit via the hit builder:

v=1 // Version.
&tid=UA-12345-1 // Tracking ID / Property ID.
&cid=de0b6f13-03f9-467f-a3fe-78021e52952e // Anonymous Client ID.

&t=event // Event hit type
&ec=video // Event Category. Required.
&ea=play // Event Action. Required.
&el=predator // Event label.
&ev=10 // Event value.

hit parameter details

Step-5: Click on the ‘validate hit’ button to validate your hit:

validate hit

Once you clicked on the ‘validate hit’ button, you must see the ‘hit is valid’ message:

hit is valid

Step-6: You can now click on the ‘send hit to Google Analytics’ button to send the event hit to GA or copy the hit payload data or click on the ‘copy shakeable link to hit’ if you want to share the hit details with others.

 

#4 Using postman chrome extension for validating protocol hits

Through postman chrome extension you can send payload data via GET or POST method. The GA hit builder allows you to send payload data only via POST method:

postman extension

Through postman extension you can also:

# test sending payload data over secure connection.

# send payload data by using a different URL end point (/debug/collect or /batch)

Note: GA hit builder use only one URL end point, /collect

For more details about postman, read this article: Creating and validating measurement protocol requests through postman

 

#5 Directly using the measurement protocol validation server

You can validate protocol hits by sending them to the measurement protocol validation server.

You can access the validation server directly or via the measurement protocol hit builder tool.

To access the validation server directly use the /debug/collect endpoint as explained earlier:

https://www.google-analytics.com/debug/collect?v=1&t=pageview&tid=UA-1029844-34&cid=c461d1ba-d341-499f-bc9e-673b67f2aa18&dh=mydemo.com&dp=%2Fhome&dt=testing%20page%20tracking

Copy-paste this URL in your browser window and then press enter:

validation measurement protocol

Note: If you are sending hits to validation server then they won’t show up in your GA reports.

 

Interpreting responses from the Measurement Protocol Validation Server

Responses from the Measurement Protocol Validation Server will be in JSON format. Following is an example of such response:

validation response

Here hitParsingResult is an array and ‘valid‘, ‘hit‘ and ‘parserMessage‘ are the objects of this array. If the value of ‘valid’ key is ‘false’ then it means the hit is invalid. On the other hand, if the value of ‘valid’ key is ‘true’ then it means the hit is valid.

If the parserMessage array is not empty then it means the hit is invalid.

If the hit is invalid, you can then get details about the error through following parserMessage keys:

  1. messageType
  2. description
  3. parameter

 

Sending Event Tracking data to Google Analytics server using measurement protocol

In order to send event tracking data to GA server, you need to send following similar HTTP POST request for each unique event:

User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36

POST https://www.google-analytics.com/collect?v=1&tid=UA-123456-2&cid=36009a59-2a05-49e7-b826-2b884d0f935b &t=event&ec=video&ea=play&el=spider%20man&ev=100

 

Sending Ecommerce data to Google Analytics server using measurement protocol

To send ecommerce tracking data, you need to send one HTTP POST request for an entire transaction and one HTTP POST request for each item in the transaction. For example:

HTTP POST request for an entire transaction

User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36

POST https://www.google-analytics.com/collect?v=1&tid=UA-123456-2&cid=36009a59-2a05-49e7-b826-2b884d0f935b&t=transaction&ti=4567DT&ta=tesco&tr=45.56&ts=2.32&tt=10.11&cu=EUR

 

HTTP POST request for one item in the transaction

User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36

POST https://www.google-analytics.com/collect?v=1&tid=UA-123456-2&cid=36009a59-2a05-49e7-b826-2b884d0f935b&t=item&ti=4567DT&in=hangers&ip=34.21&iq=45&ic=23ds53&iv=retail%20display&cu=EUR

 

Note: You need to use the same transaction ID to associate a product item with a transaction.

 


Quick announcement about my new book on Web Analytics and Conversion Optimization

The role of maths and statistics in the world of web analytics is not clear to many marketers and web analysts. Not many talk or write about the usage of statistics and data science in conversion optimization. That is why I have written this book Maths and Stats for Web Analytics and Conversion Optimization to fill this knowledge gap.

This expert guide will teach you, how to leverage the knowledge of maths and statistics in order to accurately interpret the web analytics data and take business and marketing decisions which can quickly improve the bottom-line of your online business. Read first few chapters of this book for free here.


 

Himanshu Sharma

Certified web analyst and founder of OptimizeSmart.com

My name is Himanshu Sharma and I help businesses in finding and fixing their Google Analytics and conversion issues.
  • More than ten years' experience in SEO, PPC and web analytics
  • Certified web analyst (master level) from MarketMotive.com
  • Google Analytics certified
  • Google AdWords certified
  • Nominated for Digital Analytics Association Award for Excellence
  • Bachelors degree in Internet Science
  • Founder of OptimizeSmart.com and EventEducation.com
I am also the author of the book Maths and Stats for Web Analytics and Conversion Optimization If you have any questions or comments please contact me