1

I am new to statistics, but I need to see if results of my findings are significant, but I’m not sure, which t test to use in my case.

I have a product page, that may have from 1 to ~10 videos. So, I need to compare CTR(Click Through Rate - number of users who clicks on my video to number of users who saw this page with video).

I need to calculate, if there is significant difference in CTR depending on video is alone on he page or video is located together with other videos.

Let’s say I have 7 videos and I want to compare where those videos performed better (where CTR is better) : when they are alone on page or when they are together with other videos.

So my input for t test is looks like this:

enter image description here

where column TRUE - means that this videos is alone on page vs FALSE - video is not alone. CTR is in %

From this table, I can see that all videos are performing better (meaning that has better CTR) when they are alone on page. But I need to confirm that this results are statistically significant.

So my question is which t-test I need to use to confirm this? As I understand, I cannot use independent sample T-test, since I am comparing same videos, but in different condition?

Bilberryfm
  • 113
  • 2
  • Can you post a histogram of your data? It looks like they might be skewed (see here: https://stats.stackexchange.com/questions/111320/should-i-use-t-test-on-highly-skewed-and-discrete-data?rq=1) so I'm not sure any sort of t-test is really appropriate. You might want to considered the Wilcoxon signed rank test as an alternative. – TPM May 14 '18 at 16:20

1 Answers1

0

you should use a paired sample t-test.

one way to implement it is:

  1. for each video you compute the difference TRUE-FALSE
  2. you apply a 1-sample t-test to the so obtained array: if it is significantly bigger than zero, then CTR is significantly better when the video is alone
fabiob
  • 622
  • 4
  • 13
  • I am trying to use online tool to get my p value (this one: http://www.socscistatistics.com/tests/ttestdependent/Default2.aspx) There is a checkbox for One-tailed or two-tailed hypothesis? Would my hypothesis considered to be one-tailed? – Bilberryfm May 14 '18 at 17:05
  • So I tried 2 t tests: - One called "T-Test Calculator for 2 Dependent Means " (http://www.socscistatistics.com/tests/ttestdependent/Default2.aspx) - Second called: "Single Sample T-Test Calculator" (only for TRUE-FALSE diff, as you suggested) http://www.socscistatistics.com/tests/tsinglesample/Default2.aspx. And I got completely diff results: for first t test results are statistically significant (p is 0.030506). For second one - it's not (P-Value is 0.5) – Bilberryfm May 14 '18 at 17:11
  • whether you use a one tail or two tail is not a statistics question, it depends on your expectations as a researcher. for example if you have a theory saying that many videos work as a distractor and as a consequence you expect they reduce the CTR, then fine to use a 1 side. if you are just planning a new website and you have no idea which is better, or if one of the two possibilities is better at all, then you should use 2 sided. – fabiob May 15 '18 at 19:42
  • on the other hand for me the two tests you used give the same result (both for 1-tailed and 2-tailed). I first computed "T-Test Calculator for 2 Dependent Means", which then also provides a column with the difference. this column (Diff(T2-T1)) I used as an input for "Single Sample T-Test Calculator", where I also had to set population mean =0. – fabiob May 15 '18 at 19:57