# Is your conversion Rate Statistically Significant?

Following article is an excerpt from my best selling book: **Maths and Stats for Web Analytics and Conversion Optimization** which I am sharing it here for the first time, to benefit the wider audience:

Majority of us take marketing decisions on the basis of conversion rate.

There is an industry trend to invest more in the marketing channel which has higher conversion rate.

But sometimes such thinking can backfire and it can backfire really badly.

**What if the marketing channel with higher conversion rate is actually performing poorly and results in monetary loss if you invest in it?**

Learn to fix this problem beforehand by determining whether your conversion Rate is Statistically Significant.

For e.g. consider the performance of three campaigns A, B and C in the last one month.

One look at this table and majority of marketers will blindly assume that campaign ‘B’ is performing much better than campaign ‘A’ and Campaign ‘C’ because it has the highest conversion rate.

So we should invest more in campaign ‘B’.

But wait a minute.

Let us dig out how these conversion rates are calculated.

We can see from the chart above that the sample size (4 transactions out of 20 visits) in case of campaign B is too small to be statistically significant.

Had campaign B got 1 transaction out of 1 visit, it conversion rate would be 100%.

Will that make its performance even better?

No.

So we can filter out campaign B performance while determining the best performing campaigns.

I will explain it later in great detail why campaign B performance is statistically insignificant.

Now we are left with two campaigns: A and C.

Clearly now Campaign ‘A’ is the winner because it has the higher conversion rate.

But wait a minute.

We are not done yet.

We are still not sure whether the difference between the conversion rates of campaign ‘A’ and Campaign ‘C’ are statistically significant.

Let us assume that after conducting a statistical test, we came to the conclusion that the difference in the conversion rates of the two campaigns can not be proved to be statistically significant.

Under these circumstances, we cannot draw the conclusion that campaign ‘C’ is not performing better.

So what we can do then?

Well, we need to collect more data to compute statistical significance of the difference in the conversion rates of the two campaigns.

At this stage investing more money in campaign ‘A’ may not produce optimal results, as you may think it will.

Before I explain to you, how I did all this analysis in great detail, we need to refresh few statistic skills.

Love it or hate it but statistics is the way to go, if you wish to dive deep into advanced web analytics.

## Introduction to Statistical Significance

I have talked about statistical significance throughout this article.

But what it is?

Statistically significant result is the result which is unlikely to have occurred by chance.

Statistically insignificant result is likely to have occurred by chance.

## Introduction to Inference

Inference is the process of drawing conclusions from premises which are known or assumed to be true.

The conclusion drawn is known as **idiomatic**.

You have read my idiomatics in the article above.

**Statistical inference **is the process of drawing conclusions from data which is subject to random variation.

One example of statistical inference is **observational errors**.

You assumed that conversion rate of campaign ‘B’ is highest only on the basis of your observation.

This is your statistical inference which could be wrong.

## Introduction to Statistical population

Statistical population is the set of entities (values, potential measurements) from which statistical inferences are to be drawn.

These inferences are often drawn from the random sample taken from the population.

The set of campaigns above is an examples of statistical population from which statistical inferences (like which is the highest performing campaign) are drawn.

The subset of statistical population is called **sub population**.

For e.g. if you consider a PPC campaign as statistical population then its ad groups can be considered as sub populations.

In order to understand the properties of statistical population, statisticians first separate the population into distinct sub populations (provided they have distinct properties) and then try to understand the properties of individual sub-populations.

For the same reason, analytics experts recommends to segment analytics data before you draw statistical inferences from it.

So if you want to understand the performance of a PPC campaign, then you should first try to understand the performance of its individual ad groups.

Similarly if you want to understand the performance of an ad group you should first try to understand the performance of the keywords and ad copies in that ad group.

I hope it is clear now, why data segmentation is so important in web analytics.

## Introduction to Sample

Sample is the subset of statistical population that is supposed to represent the entire population.

If a sample represents the entire population then its analysis should produce similar results to analyzing all of the population.

## Introduction to Sampling

Sampling is the selection of a sample to understand the properties of entire statistical population.

Sampling is carried out to analyze large data sets in a cost efficient manner and in a reasonable amount of time.

For e.g. Google analytics select and analyze only a subset of data from your website traffic to produce reports in a reasonable amount of time.

Related Article: Understanding Data Sampling in Google Analytics

## Introduction to Sample Size

As the name suggest, it is the size of a sample.

In data sampling, the larger the sample size, the more reliable are the estimates and vice versa.

Since the sample size of campaign B is very small, its conversion rate is not statistically significant.

## Effect Size (or Signal)

In statistics ‘effect’ is the result of something.

Effect size is the magnitude of the result.

For e.g. if increasing the daily ad spend of a PPC campaign improves its conversion rate by 2% then ‘improvement in conversion rate’ is the ‘effect’ and ‘improvement of 2%’ is the effect size.

## Noise

According to Wikipedia, noise is the recognized amount of unexplained variation/randomness in a sample.

## Null Hypothesis

According to null hypothesis, any kind of difference or significance you see in a set of data is due to chance and not due to a particular relationship.

For example, According to Null hypothesis, any difference in the conversion rates of the two campaigns ‘A’ and ‘C’ is due to chance.

To prove that the difference is not due to chance, you need to conduct a statistical test which refutes the null hypothesis.

When Null hypothesis is rejected the result is said to be statistically significant.

**Important points about Null Hypothesis**

**Null hypothesis corresponds to a general/default position.****Null hypothesis can never be proven.**

For e.g. a statistical test can only reject a null hypothesis or fail to reject a null hypothesis.

It cannot prove a null hypothesis.

So if the difference in the conversion rates of the two campaigns ‘A’ and ‘C’ is not statistically significant, it doesn’t mean that there is no difference in reality.

It only means that there is not enough evidence to reject the null hypothesis that the difference in conversion rates is by chance.

## Alternative Hypothesis

It is the negation or opposite of the null hypothesis.

So if null hypothesis is that the difference in conversion rates of the two campaigns ‘A’ and ‘C’ is by chance then alternative hypothesis would be that the difference in conversion rates is due to a particular relationship and not by chance.

In statistics the only way to prove your hypothesis is to reject the null hypothesis.

You do not prove the alternative hypothesis to support your hypothesis.

Realated article: The little known details about Hypothesis in Conversion Optimization

## Confidence

It is the confidence that the result has not occurred by a random chance.

Statistical significance can be considered to be the confidence one has in a give result.

Confidence depends upon the signal to noise ratio and the sample size and is calculated by following formula:

So confidence that the result has not occurred by a random chance is high, if signal is large and/or sample size is large and/or noise is low.

Now back to our campaigns ‘A’ and ‘C’.

In order to find out whether or not the difference in the conversion rates of the two campaigns is statistically significant we need to calculate the confidence i.e. how confident you are statistically that the difference has not occurred by chance.

If confidence is less than 95% then the difference is not statistically significant and we need to collect more data before drawing any conclusions.

Enough theory now, let us see how we can use confidence in real life to take better marketing decisions.

Consider the following scenario:

From the table above we can see that the ecommerce conversion rate of Google CPC is higher than that of Google Organic.

Does that mean Google CPC campaigns are performing better than organic?

Before we jump into any conclusion and invest more in PPC, let us calculate the statistical significance of the difference in conversion rates of Google organic and PPC campaigns.

So according to my statistical test (Z-test), I have only 65% confidence that the difference in the conversion rates of Google organic and Google PPC is not by chance.

As confidence is less than 95% the difference is not statistically significant and we need to collect more data before drawing any conclusions.

There is one more very important point that you need to remember here.

**It is possible and quite common for a result to be statistically significant and trivial or statistically insignificant but still important. **

For example even if the difference in the conversion rates of Google organic and Google PPC turned out to be statistically significant we should still be investing more in Google organic (in this particular case) as the effect size (here revenue) of Google organic is much larger than that of Google PPC.

**Just because a result is statistically significant, it doesn’t always mean that it is practically meaningful**.

That is why we should interpret both the statistical significance and effect size of our results.

**Related Articles:**

## Learn about the Google Analytics Usage Trends Tool

The Google Analytics usage trend is a new tool which is used to visualise trends in your Google Analytics data and to perform trend analysis.

Take your knowledge of Web Analytics to the next level. Checkout my **web analytics training course**.

## Take your Analytics knowledge to the next level. Checkout my Best Selling Books on Amazon

**Maths and Stats for Web Analytics and Conversion Optimization**

This expert guide will teach you how to leverage the knowledge of maths and statistics in order to accurately interpret data and take actions, which can quickly improve the bottom-line of your online business.

**Master the Essentials of Email Marketing Analytics**

This book focuses solely on the ‘analytics’ that power your email marketing optimization program and will help you dramatically reduce your cost per acquisition and increase marketing ROI by tracking the performance of the various KPIs and metrics used for email marketing.

**Attribution Modelling in Google Analytics and Beyond**

Attribution modelling is the process of determining the most effective marketing channels for investment. This book has been written to help you implement attribution modelling. It will teach you how to leverage the knowledge of attribution modelling in order to allocate marketing budget and understand buying behaviour.