4

Background

I am currently testing conversions on a web page and would like to test if the changes I made yesterday produced a significant change in conversions.

The Data Set

For each day I test, I have the total number of visitors to the site and the number of visitors who click through a certain link. Then I divide the number of visitors who click through the link by the total number of visitors. This gives me my click through ratio.

The Problem

I would like to test if a certain day has a significantly different click through ratio to the preceding days.

Hzmy
  • 141
  • 1
  • 2
    To get significance you should keep the numbers of visitors and the number who click through, rather than just the ratio. – Henry Dec 20 '12 at 16:33
  • I have done this and have both numbers available. – Hzmy Dec 20 '12 at 16:40
  • 1
    If you have many days data available, you should look at time series things, I think, and add that tag to your question. – Peter Flom Dec 20 '12 at 16:46

2 Answers2

2

If you are willing to assume that the ratio was constant up until you made the change then you can do this as a simple test of 2 proportions or a $\chi^2$ test on a $2 \times 2$ table. You can expand the later to compare more than 2 groups. If you want to consider the ratio changing smoothly over time (with possibilities of sudden jumps at know changes) then logistic regression is an option. If you need to take into account possible serial correlation then you are getting into generalized estimating equations and should probably consult with a local statistician.

Greg Snow
  • 46,563
  • 2
  • 90
  • 159
2

If you want to test whether a change to a website has led to a difference in the conversion-rate, you have to perform an A/B-Test with

  • group A: original site, aka control
  • group B: site including the change

After the test has run several days you can compare the conversion-rates of both groups using e.g. the $\chi^2$-test, as suggested by Greg.

Why ? What is wrong with comparing the results before and after the change, i.e. performing a sequential test ?

The reason are confounders. In order to make the results comparable you have to

  • either guarantee, that the only thing which has changed is the change to the site
  • or control everything else by which may affect the conversion-rate by randomly assigning the participants / visitors to one or another group (which is what an A/B-Test does).

Here are some examples what can go wrong when a "sequential test" is performed:

  • You change the site friday evening, so you basically compare the working days with the weekend
  • You change the site right before a sale starts (assuming an e-commerce-site)
  • You change the site color from blue to green right before or during the "National Blue Celebretation Day".
  • Some days before the change a special advertising has lead to an increased traffic with increased conversion-rate
  • Around the change the wheather in the geographic region with the most clickers also changes, influencing the mood and hence the results (e.g. if the site is about sunblockers)

etc.etc.

In fact, in the fascinating, (messy) data-generating environment of the internet, it is nearly impossible to control all confounders by thinking of them and excluding them beforehand. Hence, use A/B-Tests.

You might find more about this interesting subject by browsing the tag here on crossvalidated.

Community
  • 1
mlwida
  • 9,922
  • 2
  • 45
  • 74
  • Thank you for a very detailed response. I understand the need for control in experiments and testing. I was under the assumption that a chi-squared test, implies two matching data sets, whereas I wanted to test one particular data point to see if it was a statistical outlier in a current set of data. Presumably I would test its variance and conclude a hypothesis based upon that. – Hzmy Feb 07 '13 at 00:29
  • 1. I do not understand what you mean by "matching". It is sufficient that both sets have been generated by a binomial distribution with maybe differing parameters p1,p2. 2. In case of a binomial distribution (conversion-rate), what is "one particular data point" ? – mlwida Feb 07 '13 at 09:34