0

I performed 3 experiments, each experiment tested a different system, and each experiment involved a distinct group of subjects. In total there where 25 participants (7 for the first system, 7 for the second, 11 for the third). Subjects were asked to identify a stimulus (i.e. a vibration provided by a haptic device).

Recognition was measured as 1 = correct identification, 0 = incorrect identification. Stimuli where repeated, each stimulus receive 0 or 1 as measurement of participants' responses to it.

The stimuli where different for each system. Still I want to test whether participants using one system performed better than the participants using other systems.

My goal is only to assess the statistical differences between the 3 groups. Which non-parametric analysis I have to perform for this case of between-subjects experimental design?

I was thinking to the Mann-Whitney-Wilcoxon test, but I am unsure if I am correct. Is there anyone who could suggest which is the right analysis and provide a R example?

L_T
  • 1,463
  • 7
  • 20
  • 25
  • 1. Can you clarify what "various stimuli in a repeated fashion" involves? It sounds like you have treatment ("system"), subject group and stimulus as variables, in which case I don't know if Friedman will be suitable. 2. Can you talk about your response variable? How is it measured? 3. What exactly did you test with the Shapiro-Wilk? I can't see you being able to perform a test that relates to any assumptions in the absence of a suitable model (and if you don't know which test to apply I don't see how you can have one). ... ctd – Glen_b Jul 07 '18 at 23:12
  • ctd... 4. Formally testing assumptions (even when done correctly) is often [not helpful](https://stats.stackexchange.com/questions/2492/is-normality-testing-essentially-useless/2501#2501) ... and using it to choose between tests impacts the properties of your subsequent tests, intervals etc - so your p-values in your analysis will be wrong, for example. 5. There are numerous questions that ask about nonparametric tests for repeated measures of various designs, some of which have answers that may have relevant information to you, so it would be important to try some searches. – Glen_b Jul 07 '18 at 23:15
  • 1. There were various stimuli to identify which were provided by the 3 systems (the stimuli were different for each system). 2. The response variable was measured after an identification task. It had two values, 1 = participants correctly identified the stimuli, 0 if they did not. 3. I tested the normality of my data. In any case I want to use a non-parametric analysis as in total I only have 25 participants, so ANOVA is not really well suited as far as I know. – L_T Jul 08 '18 at 17:49
  • 4. and 5. Thanks, what is your suggestion then? I guess that for my case the correct technique is to use three different comparisons between couple of groups using the mann whitney wilcoxon test. What do you think? – L_T Jul 08 '18 at 17:49
  • I don't follow your answer in 1, and so still don't understand the experiment properly (... there's danger in [deciding on analysis after you have data](http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf), by the way). In 2. where you say "the stimuli" -- in particular the use of the plural here -- participants are identifying more than one stimulus at once? Or are your responses a single 0 or 1 each time? On 3. Why would you test normality of scores of 0 or 1? That can't be normal. On 4. can you explain what makes a nonparametric analysis better at small sample sizes?... – Glen_b Jul 09 '18 at 00:53
  • ... It may be there's a good reason to choose a nonparametric technique, but I am concerned you're being led to it for an unclear reason. Note also that there are *parametric* analyses that may suit better than ones that assume normality. Even if a nonparametric analysis is a good choice, I don't think a Mann-Whitney-Wilcoxon will be the best idea here but if I understand the situation more clearly\* there may be better advice. $\quad$ $\:$ \*(assume I know nothing about what you're doing. Explain it - in your question - as if you were talking to an intelligent 12 year old) – Glen_b Jul 09 '18 at 00:59
  • There where 3 experiments. Each experiment tested one system, and each experiment involved a distinct group of subjects. In total there where 25 participants (7 for the first system, 7 for the second, 11 for the third). Subjects were asked to identify a stimulus (i.e. a vibration provided by a haptic device). Recognition was measured as 1 = correct identification, 0 = incorrect identification. The stimuli where different for each system. Still I want to test whether participants using one system performed better than the participants using other systems. Hope this is clear now – L_T Jul 11 '18 at 10:02
  • 2. Stimuli where repeated, each stimulus receive 0 or 1 as measurement of participants' responses to it. 3. Huge mistake from my side here. 4. As far as I know nonparametric statistics are better suited for small samples so I avoided the use of ANOVA. Could you please suggest the correct analysis? Why Mann-Whitney-Wilcoxon does not work well? Thanks – L_T Jul 11 '18 at 10:05
  • Thanks for clarifying. Can you please edit this kind of information into your question? You should probably be choosing an analysis specifically designed for binary (0-1) responses, and specifically related to your null and alternative hypothesis. – Glen_b Jul 13 '18 at 02:01
  • Thanks, which analysis are you referring to? Any suggestion? – L_T Jul 13 '18 at 13:44
  • Your question doesn't specify a null and alternative, but it's likely to involve say a binomial GLM (logistic regression, perhaps) or perhaps a GLMM. Mann-Whitney-Wilcoxon is really for continuous data rather than highly discrete data (binary data is about as discrete as it comes, taking only two distinct values). – Glen_b Jul 13 '18 at 13:53
  • Dear @Glen_b I used the binomial GLM, but I don't understand how to interpret the results in relation to my goal of assessing whether participants performed significantly better with one system compared to the other two. This is the formula I used fit – L_T Nov 11 '18 at 22:50
  • Given the usual assumptions, the significance of the coefficient for System implies that either the expected values for the System groups differ or you made a type 1 error. You can compute the corresponding estimate of the difference in means if you need it (either by using the `predict` function or by hand directly from the `summary` output). – Glen_b Nov 11 '18 at 23:20
  • Thanks. Sorry but I am a beginner so I would need extra help to properly understand your answer. From the output of summary(fit) I get that the intercept and the second system have significant p-value, while the third system not. I deduce that the second system lead to significantly different responses to the second system. However, it is not clear to me which conclusions I can draw on the performance differences resulting between the second and third system. Also, would it be possible for you to show the exact code for the predict function? How should I interpret its results? – L_T Nov 12 '18 at 00:19
  • "I deduce that the second system lead to significantly different responses to the second system" --- no, it's different from the first, not from itself. " it is not clear to me which conclusions I can draw on the performance differences resulting between the second and third system." -- correct, you can't tell directly. If you wanted to make this specific comparison (rather than just conclude there were differences) you could have looked at setting one of those two groups to be the baseline. ... ctd – Glen_b Nov 12 '18 at 01:01
  • ctd... Alternatively, if you have particular orthogonal contrasts you want, a simple recoding of dummies might be sufficient. On the other hand if you wanted to do all possible pairwise comparisons, again that should have been made clear; this might be done in several ways but it would be best as a new question. – Glen_b Nov 12 '18 at 01:03
  • Thanks Glen_b. My understanding is that I have to use the wald test. I have created a new question here: https://stats.stackexchange.com/questions/376511/reporting-binomial-logistic-regression Do you think that the procedure I made is correct? Is the wald test the way to go? – L_T Nov 12 '18 at 01:20
  • Dear Geln_b, would it be possible for you to provide an answer? – L_T Nov 13 '18 at 15:13

0 Answers0