The following article is an excerpt from my best selling book: Maths and Stats for Web Analytics and Conversion Optimization which I am sharing here to benefit the wider audience:

Is your Conversion Rate statistically significant?

The majority of us take marketing decisions on the basis of conversion rate. 

There is an industry trend to invest more in the marketing channel which has a higher conversion rate.

But sometimes such thinking can backfire and it can backfire really badly.

What if the marketing channel with a higher conversion rate is actually performing poorly and results in a monetary loss if you invest in it?

Learn to fix this problem beforehand by determining whether your conversion rate is statistically significant. 

For e.g. consider the performance of three campaigns A, B and C in the last one month.

One look at this table and the majority of marketers will blindly assume that campaign ‘B’ is performing much better than campaign ‘A’ and Campaign ‘C’ because it has the highest conversion rate. 

So we should invest more in campaign ‘B’.

But wait a minute. 

Let us dig out how these conversion rates are calculated.

We can see from the chart above that the sample size (4 transactions out of 20 visits) in the case of campaign B is too small to be statistically significant. 

Had campaign B got 1 transaction out of 1 visit, its conversion rate would be 100%.

Will that make its performance even better? No.

So we can filter out campaign B performance while determining the best performing campaigns. I will explain later in great detail why campaign B’s performance is statistically insignificant.

Now we are left with two campaigns: A and C. 

Clearly now campaign ‘A’ is the winner because it has a higher conversion rate.

But wait a minute. We are not done yet.

We are still not sure whether the difference between the conversion rates of campaign ‘A’ and campaign ‘C’ are statistically significant.

Let us assume that after conducting a statistical test, we came to the conclusion that the difference in the conversion rates of the two campaigns can not be proved to be statistically significant.

Under these circumstances, we cannot draw the conclusion that the campaign ‘C’ is not performing better.

So what we can do then? Well, we need to collect more data to compute the statistical significance of the difference in the conversion rates of the two campaigns.

At this stage investing more money in campaign ‘A’ may not produce optimal results, as you may think it will.

Before I explain to you, how I did all this analysis in great detail, we need to refresh a few statistical skills.

Love it or hate it but statistics is the way to go if you wish to dive deep into advanced web analytics.

Introduction to Statistical Significance

I have talked about statistical significance throughout this article. But what it is?

The statistically significant result is the result that is unlikely to have occurred by chance.

The statistically insignificant result is likely to have occurred by chance.

Introduction to Inference

The inference is the process of drawing conclusions from premises that are known or assumed to be true.

The conclusion drawn is known as idiomatic. You have read my idiomatic in the article above.

Statistical inference is the process of drawing conclusions from data that is subject to random variation. One example of statistical inference is observational errors.

You assumed that the conversion rate of campaign ‘B’ is the highest only on the basis of your observation. This is your statistical inference which could be wrong.

Introduction to Statistical population

The statistical population is the set of entities (values, potential measurements) from which statistical inferences are to be drawn. These inferences are often drawn from the random sample taken from the population.

The set of campaigns above is an example of a statistical population from which statistical inferences (like which is the highest performing campaign) are drawn.

The subset of the statistical population is called subpopulation. For e.g. if you consider a PPC campaign as a statistical population then its ad groups can be considered as subpopulations.

In order to understand the properties of a statistical population, statisticians first separate the population into distinct subpopulations (provided they have distinct properties) and then try to understand the properties of individual subpopulations.

For the same reason, analytics experts recommend to segment analytics data before you draw statistical inferences from it. So if you want to understand the performance of a PPC campaign, then you should first try to understand the performance of its individual ad groups.

Similarly, if you want to understand the performance of an ad group you should first try to understand the performance of the keywords and ad copies in that ad group.

I hope it is clear now, why data segmentation is so important in web analytics.

Introduction to Sample

A sample is a subset of a statistical population that is supposed to represent the entire population.

If a sample represents the entire population then its analysis should produce similar results to analyzing all of the population.

WHAT’S INSIDE: My Step-By-Step Blueprint For Generating Record Sales And ROI By Leveraging Analytics Data.

Learn To Generate Record Sales And ROI By Leveraging Analytics Data. (150 pages)

Yes I want the ebook

Introduction to Sampling

Sampling is the selection of a sample to understand the properties of the entire statistical population.

Sampling is carried out to analyze large data sets in a cost-efficient manner and in a reasonable amount of time. For example, Google Analytics select and analyze only a subset of data from your website traffic to produce reports in a reasonable amount of time.

Related Article: Understanding Data Sampling in Google Analytics

Introduction to Sample Size

As the name suggests, it is the size of a sample.

In data sampling, the larger the sample size, the more reliable are the estimates and vice versa.

Since the sample size of campaign B is very small, its conversion rate is not statistically significant.

Effect Size (or Signal)

In statistics ‘effect’ is the result of something.

Effect size is the magnitude of the result. For e.g. if increasing the daily ad spend of a PPC campaign improves its conversion rate by 2% then ‘improvement in conversion rate’ is the ‘effect’ and ‘improvement of 2%’ is the effect size.

Noise

According to Wikipedia, noise is the recognized amount of unexplained variation/randomness in a sample.

Null Hypothesis

According to a null hypothesis, any kind of difference or significance you see in a set of data is due to chance and not due to a particular relationship.

For example, according to a null hypothesis, any difference in the conversion rates of the two campaigns ‘A’ and ‘C’ is due to chance.

To prove that the difference is not due to chance, you need to conduct a statistical test that refutes a null hypothesis.

When a null hypothesis is rejected the result is said to be statistically significant.

Important points about a null hypothesis

1. Null hypothesis corresponds to a general/default position.
2. Null hypothesis can never be proven.

For e.g. a statistical test can only reject a null hypothesis or fail to reject a null hypothesis. It cannot prove a null hypothesis.

If the difference in the conversion rates of the two campaigns ‘A’ and ‘C’ is not statistically significant, it doesn’t mean that there is no difference in reality. It only means that there is not enough evidence to reject the null hypothesis that the difference in conversion rates is by chance.

Alternative Hypothesis

It is the negation or opposite of the null hypothesis.

So if a null hypothesis is that the difference in conversion rates of the two campaigns ‘A’ and ‘C’ is by chance then the alternative hypothesis would be that the difference in conversion rates is due to a particular relationship and not by chance.

In statistics, the only way to prove your hypothesis is to reject the null hypothesis. You do not prove an alternative hypothesis to support your hypothesis.

Realated article: The little known details about Hypothesis in Conversion Optimization

Confidence

It is the confidence that the result has not occurred by a random chance.

Statistical significance can be considered to be the confidence one has in a given result.

Confidence depends upon the signal to noise ratio and the sample size and is calculated by the following formula:

Confidence that the result has not occurred by a random chance is high if the signal is large and/or the sample size is large and/or noise is low.

Now back to our campaigns ‘A’ and ‘C’.

In order to find out whether or not the difference in the conversion rates of the two campaigns is statistically significant, we need to calculate the confidence i.e. how confident you are statistically that the difference has not occurred by chance.

If confidence is less than 95% then the difference is not statistically significant and we need to collect more data before drawing any conclusions.

Enough theory now, let us see how we can use confidence in real life to take better marketing decisions.

Consider the following scenario:

From the table above we can see that the ecommerce conversion rate of Google CPC is higher than that of Google Organic.

Does that mean Google CPC campaigns are performing better than organic?

Before we jump to any conclusion and invest more in PPC, let us calculate the statistical significance of the difference in conversion rates of Google organic and PPC campaigns.

So according to my statistical test (Z-test), I have only 65% confidence that the difference in the conversion rates of Google organic and Google PPC is not by chance.

As confidence is less than 95% the difference is not statistically significant and we need to collect more data before drawing any conclusions.

There is one more very important point that you need to remember here.

It is possible and quite common for a result to be statistically significant and trivial or statistically insignificant but still important. 

For example, even if the difference in the conversion rates of Google organic and Google PPC turned out to be statistically significant we should still be investing more in Google organic (in this particular case) as the effect size (here revenue) of Google organic is much larger than that of Google PPC.

Just because a result is statistically significant, it doesn’t always mean that it is practically meaningful.

That is why we should interpret both the statistical significance and effect size of our results.

Other articles on Maths and Stats in Web Analytics

• Beginners Guide to Maths and Stats behind Web Analytics
• How to Analyze and Report above AVERAGE
• What Matters More: Conversion Volume or Conversion Rate – Case Study
• The little known details about hypothesis in conversion optimization
• Calculated Metrics in Google Analytics – Complete Guide
• Here is Why Conversion Volume Optimization is better than CRO
• Bare Minimum Statistics for Web Analytics
• Understanding A/B Testing Statistics to get REAL Lift in Conversions
• 10 Techniques to Migrate from Data Driven to Data Smart Marketing
• Data Driven or Data blind and why I prefer being Data Smart
• The Guaranteed way to Sell Conversion Optimization to your Client
• SEO ROI Analysis – How to do ROI calculations for SEO