1

In the past month(s) I had to get familiar with statistics in order to validate my results although my background is computer science/programming. I currently examine whether I can enhance user's skills with a tutorial. So I prepared a pre and a post test and measured various skills to compare them. The Wilcoxon Signed Rank test helped me to determine significance since my data was not normally distributed.

Now here is my current problem: I examine whether my subjects used a tactic within a (fact-finding search) task and whether the post results differ significantly from the pre test results. For each user, my data keeps track whether he applied (used) the tactic or not (not used).

UPDATE I want to conclude whether the intervention had a significant impact on my subjects' search skills, i.e., whether more users applied the (search) tactic in the post test.

My data looks as follows:

N = 58 users
0 = no
1 = yes

user pre  post
1    1    1
2    1    1
3    0    1
4    0    1
5    0    1
6    0    1
7    0    1
8    0    1
9    1    1
10   0    1
11   0    1
12   1    1
13   1    1
14   0    1
15   0    1
16   0    0
17   1    1
18   1    0
19   1    1
20   0    1
21   1    1
22   0    1
23   1    0
24   0    1
25   0    0
26   1    0
27   1    1
28   0    0
29   1    1
30   1    1
31   0    0
32   0    1
33   1    0
34   0    0
35   1    1
36   0    1
37   0    1
38   0    1
39   0    1
40   1    1
41   1    1
42   1    1
43   0    1
44   1    0
45   0    1
46   1    1
47   1    1
48   1    1
49   0    1
50   1    1
51   0    1
52   1    1
53   0    1
54   0    1
55   0    1
56   1    1
57   0    0
58   1    1

Summary:

                         Post 
                 |Not used | Used|
Pre   |Not used  |6        |25   |
      |Used      |5        |22   |

*UPDATE END *

I read a lot about significance tests, Chi-square, McNemar, Fisher’s exact test, ... but I am still not sure which test is appropriate or whether I can determine significance at all for that kind of data.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Ioannis K.
  • 123
  • 7
  • 1
    Have a look at: https://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs – kjetil b halvorsen Dec 30 '18 at 14:27
  • Thanks for the hint! My problem is I have only 1 group: the same users are measured twice. And I want to check whether the score differ. – Ioannis K. Dec 30 '18 at 15:26
  • What are you trying to conclude from that? Whether some intervention had a effect? That usage varies over time? That people may vary their behaviour? – Björn Dec 30 '18 at 18:25
  • I'm trying to conclude whether my subjects performed significantly better in search tasks after the tutorial (intervention) in terms of tactics. (I examine 1 tactic at a time). Thats why I check how many users used that particular tactic before and after training. But I'm not sure whether this works out. – Ioannis K. Dec 30 '18 at 19:04
  • 2
    The table you have shown can be looked upon otherwise: Each one of the 58 users can have one of four transitions: not use -> not use; not use -> use; ... and so on. Can you show us that table also? (as an edit to the post) – kjetil b halvorsen Dec 31 '18 at 20:28
  • 1
    Also, you could get some ideas from [this post](https://stats.stackexchange.com/questions/76875/what-is-the-difference-between-mcnemars-test-and-the-chi-squared-test-and-how) – kjetil b halvorsen Dec 31 '18 at 20:39
  • Sure! I edited my post. It includes my current data. Appreciate the link! So the appropriate test would be McNemar? – Ioannis K. Jan 02 '19 at 21:54
  • 2
    Yes, I would go for the McNemar test. But read well the linked posts here about that test! – kjetil b halvorsen Jan 02 '19 at 23:24
  • I am really sorry for the delayed response! Apologies! I read the linked post(s) and performed McNemar test as described by gung (with the table in my original post): > mcnemar.test(mat, correct=FALSE) McNemar's Chi-squared test data: mat McNemar's chi-squared = 13.333, df = 1, p-value = 0.0002607 My result is significant. So, I may conclude there was an enhancement? I.e., the intervention had a significant positive impact on skills and more subjects used the tactic after the training than before? – Ioannis K. Jan 06 '19 at 02:00
  • Wrong conclusion. McNemar is a two-sided test. The p-value (0.0002607) tells me there IS a difference. Finally I have compare bottom left (5) and top right (25) values. Since 25 > 5, I may conclude that the intervention helped people to actually enhance their skills. This should be the right answer? – Ioannis K. Jan 06 '19 at 12:07

1 Answers1

1

Go for a McNemar test it may give you the accuare results in your case. also you can compare the results from wilcoxon signed rank test i.e dependent test with McNemar test then you can conclude your result. https://rcompanion.org/handbook/H_05.html above link will tell you a path.

Sandip Khot
  • 36
  • 4