3

Update Added more details about the Experimental setup.

My experiment comprised two groups, control (N=25) and experimental (N=26). Each participant belonged to one group. Their performance has been tested three times through knowledge tests once before a training course, and twice after. My spreadsheet contains the following variables:

UserID, Group, T1, T2, T3

T1, T2, and T3 represent scores of the particular tests on a ratio scale. Participants could score from 0 up to 62.

First of all, I wanted to check whether there are any significant improvements between T1, T2, and T3 inside a group, and therefore analyzed paired differences. Thus, for each of the two groups, I had to run 3 Tests : A: T2-T1, B: T3-T1, C: T3-T2. I planned to use non-parametric tests beforehand as I was not sure how the sets would look like. Density graphs showed that almost all of the sets do not approximate a normal distribution, so I stick to my decision. As my sets (T2-T1, T3-T1, T3-T2) were paired, I used the Friedman test to determine whether there is any significant difference at all followed by the Dunn-Bonferroni test to determine where the differences lie (as offered by SPSS). This worked fine. Results for both the groups are the same : Participants where better at T2 and T3 compared to T1. The differences between T1 and T2, and T1 and T3 are significant which tells me that my training cours(es) had a measurable and substantial effect. Performances dropped after T2 but not significantly (T3-T2).

Question: Now I want to carry out a between-group comparison. More precisely, I want to check whether one group performed better than the other given T2-T1, T3-T1, and T3-T2. This leads to these three tests:

Control T2-T1 vs Experimental T2-T1
Control T3-T1 vs Experimental T3-T1
Control T3-T2 vs Experimental T3-T1

This time, I have to run 3 tests with 2 unpaired sets each to compare the groups. (The sets themselves, e. g., Control T2-T1 still consist of paired differences like before.) I know that I can run three Mann-Whitney-U tests and have to adjust the p-values at the end (see Update). This procedure tells me where the differences actually lie. But, is there a test I could use in my setup to determine whether there are any differences? If so, how can I perform it under SPSS or R?

I know this question has been asked quite a few times (e.g., here Is there an equivalent to Kruskal Wallis one-way test for a two-way model? and here https://www.researchgate.net/post/Is_there_a_nonparametric_test_equivalent_to_a_2x3_ANOVA), but I am not exactly sure whether 1) such a "complex" operation is doable using one hypothesis test only and 2) which of the proposed methods are suitable in my case.

  • Ioannis K

UPDATE

I provide my data for better clarification:

Test A:

Control T2-T1       = [14, 14, 14, 10, 22, 20, 31, 21, 28, 10, 12, 22, 33, 28, 15, 8, 7, 16, 18, 22, 25, 33, 2, 24, 18, 26, 27, 29, 27, 9] (Mean = 19.5)
Experimental T2-T1  = [25, 28, 15, 20, 34, 20, 27, 17, 22, 16, 7, 16, 9, 16, 23, 8, 30, 20, 17, 14, 15, 22, 21, 23, 20, 18, 12, 17]        (Mean = 19)

Test B:

Control T3-T1       = [14, 10, 14, 21, 27, 22, 16, 2, 7, 11, 25, 20, 20, 9, -1, 17, 20, 18, 19, 10, 21, 17, 19, 20, 26] (Mean = 16.16)
Experimental T3-T1  = [22, 25, 12, 20, 30, 17, 24, 17, 16, 17, 7, 5, 4, 14, 21, 14, 23, 17, 8, 15, 22, 22, 24, 13, 12]  (Mean = 16.84)

Test C:

Control T3-T2       = [0, -4, 0, 1, -4, 1, -12, -8, -5, -11, -8, -8, 5, 1, -8, 1, -2, -7, -14, 8, -3, -9, -8, -9, -1] (Mean = -4.16)
Experimental T3-T2  = [-3, -3, -3, 0, -4, -3, -3, 0, -6, 1, 0, -11, -5, -2, -2, 6, -7, 0, -6, 0, 0, 1, 1, 1, -5]      (Mean = -2.12)

T1 has been conducted before the training course. T2 one week after and T3 after a three-month hiatus. Therefore I expected a little drop in performance between T2 and T3.

In order to determine whether control or experimental group show better improvements, I conducted three Mann-Whitney-U tests (two-tailed, 5%):

Test                        U           z           p (raw)     p adj (holm)
A: ControlT2-T1 vs. ExperimentalT2-T1   U=441       0.327220    0.743502    1
B: ControlT3-T1 vs. ExperimentalT3-T1   U=301       -0.223595   0.823073    1
C: ControlT3-T2 vs. ExperimentalT3-T2   U=234.50    -1.522182   0.127963    0.3912776

The differences are minimal and not statistically significant. Both groups performed the same at any given T.

Now the question is, whether theres an ANOVA-ish test for my non-parametric 2x3 matrix of sets. I know there is no significant difference given any T..If I would have known this in advance, then I wouldnt run three post-hoc tests to find where differences exactly lie that are actually not there.

Ioannis K.
  • 123
  • 7
  • 1
    *post hoc* means "after this". Your Friedman test is not "before this", it *is* the "this" being referred to that a *post hoc* test would be after. – Glen_b Jul 29 '19 at 01:25
  • As far as I have found out, I have to use Friedman test to check whether there is any significant difference at all between different samples. If so, I use post-hoc test(s) to compare all the pairs to actually determine where the differences lie. Or, did I get s.th. wrong? Now the question is, whether I can apply this procedure (pre-hoc, post-hoc) to my between-groups test, to compare tests group wise. – Ioannis K. Jul 29 '19 at 08:34
  • 2
    I was objecting to the term "pre-hoc". There's the Friedman test (as the primary test), and *post hoc* refers to the tests done after it you described. The Friedman is not before some main event, it is the main event. *pre hoc* makes no sense to me in this context (it implies some more important test is taking place *between* the Friedman and the post hoc tests). – Glen_b Jul 29 '19 at 09:29
  • Do you think about a randomization test of no difference in population medians (null hypothesis) against a two tailed alternative, where the difference in sample medians is the test statistic? – Nick Jul 29 '19 at 16:05
  • @Glen_b Got you now. The term pre-hoc was indeed wrongly chosen. – Ioannis K. Jul 29 '19 at 16:36
  • @Nick I am not really sure how the test I want looks like or even if it exists for non-parametric data. It doesnt need to be a randomized test. Basically, I have a 2 by 3 matrix and instead of running 3 unpaired tests (two tailed) and adjust the p values, I want to end up with only one test statistic that tells me whether there is any difference between T1, T2, T3. It might be that control group is better at T1, but experimental is better at T2 and T3. In this case, the test should tell me only that there is a difference. – Ioannis K. Jul 29 '19 at 16:36
  • 1
    Do we expect that Control T1 and Treatment T1 are different? From what I gather they are the baselines tight? Also is it certain that the treatment does not have a time-dependency? i.e. that there is the same probability of observing some differences in T2 and T3? (Disclaimer: I see different subjects and a time element, I first consider an LME.) – usεr11852 Jul 29 '19 at 23:34
  • I updated my post. I expect that both groups perform the same at any given T. – Ioannis K. Jul 30 '19 at 11:26
  • Thank you for this clarification, I will try to look at this tonight. – usεr11852 Jul 30 '19 at 12:59
  • 2
    Regular reminder that your raw data does not need to be normally distributed - it's the distribution of the *residuals* that you need to be concerned about. However, you have additional reasons in your case that may make it advisable to avoid methods that assume normality of residuals (bounded and discrete data). – mkt Aug 02 '19 at 09:00
  • 2
    Speaking of which, you state in your question that the data ranges from 0 to 60, but your example data has negative values. Could you clarify this? (And while you're at it, it may be best if you removed the confusing 'pre-hoc' coinage that's already been commented about.) – mkt Aug 02 '19 at 09:02
  • I did a lot more tests on that data.. so I just wanted to be sure and sticked to non-parametric tests. (My background is not statistics. I just try to avoid basic pitfalls.. :) Oh snap. Removed the range. Actually I am looking into the differences in improvements and compare them group-wise. T1 is T2-T1, T2 is T3-T1, and T3 is T3-T2. The difference tells me by how much people inside a group improved. Now I want to find out for any given T, whether Control or Experimental group have had better improvements. Skipped this only to not confuse people with my setup... – Ioannis K. Aug 02 '19 at 15:49
  • 1
    @IoannisK. I would add that detail to the question - it is inaccurate and more confusing without that information. – mkt Aug 05 '19 at 08:55
  • @mkt You're right. Now my question is complete. – Ioannis K. Aug 05 '19 at 12:35
  • 1
    I would suggest using a multilevel model that can incorporate all three time points at once. – Peter Flom Aug 10 '19 at 11:16
  • Can you please give me more details? As far as I know, multilevel models serve for regression..? – Ioannis K. Aug 12 '19 at 05:23

0 Answers0