Is your conversion Rate Statistically Significant?

 

Majority of us take marketing decisions on the basis of conversion rate. There is an industry trend to invest more in the marketing channel which has higher conversion rate. But sometimes such thinking can backfire and it can backfire really bad.

What if the marketing channel with higher conversion rate is actually performing poorly and results in monetary loss if you invest in it. 

Learn to fix this problem beforehand by determining whether your conversion Rate is Statistically Significant.

For e.g. consider the performance of three campaigns A, B and C in the last one month.

One look at this table and majority of marketers will blindly assume that campaign ‘B’ is performing much better than campaign ‘A’ and Campaign ‘C’ because it has the highest conversion rate. So we should invest more in campaign ‘B’. But wait a minute. Let us dig out how these conversion rates are calculated.

We can see from the chart above that the sample size (4 transactions out of 20 visits) in case of campaign B is too small to be statistically significant. Had campaign B got 1 transaction out of 1 visit, it conversion rate would be 100%. Will that make its performance even better? No.

So we can filter out campaign B performance while determining the best performing campaigns.  I will explain it later in great detail why campaign B performance is statistically insignificant.

 

Now we are left with two campaigns: A and C. Clearly now Campaign ‘A’ is the winner because it has the higher conversion rate.  But wait a minute. We are not done yet.  We are still not sure whether the difference between the conversion rates of campaign ‘A’ and Campaign ‘C’ are statistically significant.

Let us assume that after conducting a statistical test we came to the conclusion that the difference in the conversion rates of the two campaigns can’t be proved to be statistically significant. Under these circumstances we cannot draw the conclusion that campaign ‘C’ is not performing better.

So what we can do then. Well we need to collect more data to compute statistical significance of the difference in the conversion rates of the two campaigns. At this stage investing more money in campaign ‘A’ may not produce optimal results as you may think it will.

Before I explain you how I did all this analysis in great detail, we need to refresh our statistic skills. Love it or hate it but statistics is the way to go if you wish to dive deep into advanced web analytics.

This post requires the basic knowledge of statistics as prerequisite. But I will try my best to explain every stat terminology I have used in this post in an easy to understand words. In case you catch any error, let me know in the comments below. I am just a student of statistics and no expert.

 

Statistical Significance

I have talked about statistical significance throughout this post. But what it is? Statistically significant result is the result which is unlikely to have occurred by chance. Statistically insignificant result is likely to have occurred by chance.

 

Inference

It is the process of drawing conclusions from premises which are known or assumed to be true. The conclusion drawn is known as idiomatic. You have read my idiomatics in the post above.  Statistical inference is the process of drawing conclusions from data which is subject to random variation.

One example of statistical inference is observational errors. You assumed that conversion rate of campaign ‘B’ is highest only on the basis of your observation. This is your statistical inference which is wrong.

 

Statistical population

Statistical population is the set of entities (values, potential measurements) from which statistical inferences are to be drawn. These inferences are often drawn from the random sample taken from the population.

The set of campaigns above is an examples of statistical population from which statistical inferences (like which is the highest performing campaign) are drawn. The subset of statistical population is called sub population.

For e.g. if you consider a PPC campaign as statistical population then its ad groups can be considered as sub populations.  To understand the properties of statistical population, statisticians first separate the population into distinct sub populations (provided they have distinct properties) and then try to understand the properties of individual sub-populations.

For the same reason, analytics experts recommends to segment analytics data before you draw statistical inferences from it. So if you want to understand the performance of a PPC campaign, then you should first try to understand the performance of its individual ad groups.

Similarly if you want to understand the performance of an ad group you should first try to understand the performance of the keywords and ad copies in that ad group. I hope it is clear now, why data segmentation is so important in web analytics.

Sample

It is the subset of statistical population that represents the entire population. Since a sample represents the entire population, its analysis will produce similar results to analyzing all of the population.

 

Sampling

It is the selection of a sample to understand the properties of entire statistical population.  Sampling is carried out to analyze large data sets in a cost efficient manner and in a reasonable amount of time. For e.g. Google analytics select and analyze only a subset of data from your website traffic to produce reports in a reasonable amount of time.

 

Sample Size

As the name suggest, it is the size of a sample. In data sampling, the larger the sample size, the more reliable are the estimates and vice versa.  Since the sample size of campaign B is very small, its conversion rate is not statistically significant.

 

Effect Size (or Signal)

In statistics ‘effect’ is the result of something. Effect size is the magnitude of the result. For e.g. if increasing the daily ad spend of a PPC campaign improves its conversion rate by 2% then ‘improvement in conversion rate’ is the ‘effect’ and ‘improvement of 2%’ is the effect size.

 

Noise

According to Wikipedia, noise is the recognized amount of unexplained variation/randomness in a sample.

 

Null Hypothesis

According to null hypothesis, any kind of difference or significance you see in a set of data is due to chance and not due to a particular relationship.

For example, According to Null hypothesis, any difference in the conversion rates of the two campaigns ‘A’ and ‘C’ is due to chance. To prove that the difference is not due to chance, I need to conduct a statistical test which refutes the null hypothesis.

When Null hypothesis is rejected the result is said to be statistically significant.

 

Important points about Null Hypothesis

1. Null hypothesis corresponds to a general/default position.

2. Null hypothesis can never be proven.

For e.g. a statistical test can only reject a null hypothesis or fail to reject a null hypothesis. It cannot prove a null hypothesis. So if the difference in the conversion rates of the two campaigns ‘A’ and ‘C’ is not statistically significant, it doesn’t mean that there is no difference in reality. It only means that there is not enough evidence to reject the null hypothesis that the difference in conversion rates is by chance.

 

Alternative Hypothesis

It is the negation or opposite of the null hypothesis. So if null hypothesis is that the difference in conversion rates of the two campaigns ‘A’ and ‘C’ is by chance then alternative hypothesis would be that the difference in conversion rates is due to a particular relationship and not by chance.

In statistics the only way to prove your hypothesis is to reject the null hypothesis. You don’t prove the alternative hypothesis to support your hypothesis.

 

Confidence

It is the confidence that the result has not occurred by a random chance. Statistical significance can be considered to be the confidence one has in a give result.  Confidence depends upon the signal to noise ratio and the sample size and is calculated by following formula:

So confidence that the result has not occurred by a random chance is high if signal is large and/or sample size is large and/or noise is low.

Now back to our campaigns ‘A’ and ‘C’. In order to find out whether or not the difference in the conversion rates of the two campaigns is statistically significant we need to calculate the confidence i.e. how confident you are statistically that the difference has not occurred by chance.

If confidence is less than 95% than the difference is not statistically significant and we need to collect more data before drawing any conclusions.

 

Enough theory now, let us see how we can use confidence in real life to take better marketing decisions.  Consider the following scenario:

From the table above we can see that the ecommerce conversion rate of Google CPC is higher than that of Google Organic. Does that mean Google CPC campaigns are performing better than organic?

Before we jump into any conclusion and invest more in PPC, let us calculate the statistical significance of the difference in conversion rates of Google organic and PPC campaigns.

 

So according to my statistical test (Z-test), I have only 65% confidence that the difference in the conversion rates of Google organic and Google PPC is not by chance.  As confidence is less than 95% the difference is not statistically significant and we need to collect more data before drawing any conclusions.

To calculate the confidence in my Google analytics report I used the ‘z-test bookmarklet’ developed by Michael Whitaker. You can find details regarding installing and using this bookmarket from here.

There is one more very important thing that you need to remember.

It is possible and quite common for a result to be statistically significant and trivial or statistically insignificant but still important. 

For example even if the difference in the conversion rates of Google organic and Google PPC turned out to be statistically significant we should still be investing more in Google organic (in this particular case) as the effect size (here revenue) of Google organic is much larger than that of Google PPC.

Just because a result is statistically significant, it doesn’t always mean that it is practically meaningful.

That is why we should interpret both the statistical significance and effect size of our results.

Other Posts you may find useful:

Subscribe to my blog
Join my free newsletter and learn to avoid the analytics mistakes everyone seems to be making over and over again.

 

About the Author:

My business thrives on referrals, so I really appreciate recommendations to people who would benefit from my help.Please feel free to endorse/forward my LinkedIn Profile to your clients, colleagues, friends and others you feel would benefit from SEO, PPC or Web Analytics.