2

So a little background. 36 men were surveyed and 40 women on 15 questions. Each question is in the form of Strongly Disagree, Disagree; Neutral; Agree; Strongly Agree.

Now the teacher is saying using a t test find out if there is a difference in the way male and females respond (as in does gender influence responses). But I just can’t wrap my head around how this can even be possible at all using a t test. I’m completely lost am I missing something here? Can someone give me some direction or explain how you would go about it.

EDIT: Added after some comments. If I assign a value 1-5 to their answers. Then for each question I do a t test. Then check if there is any real significance between all their answers. And if all tests there is no big significance on how they answer, I can say that the way males and females respond it is likely that the gender doesn’t play a role?

Or I am beginning to think the teacher gave us a trick problem. Expecting us to answer with this isnt the correct test to use and use another test to come to the conclusion? Do you guys think this might be what is happening.

Please excuse the poor English. This is not my first language and I am using a translator mostly. I have also missed a few classes due to health reasons and haven’t heard back from teacher if I can get extension. Assignment is due tomorrow. This is why I was thinking maybe I’m not understanding something.

Mike Victor
  • 21
  • 2
  • One cannot say there is no difference, one can at most say that if a difference exists that it is at most probably less than a certain amount, provided that the appropriate statistic is calculated to make that statement. – Carl Sep 01 '19 at 08:06
  • Yes that’s what I meant. Sorry for the poor English. Would you happen to know how it can be possible to do so with a t-test. I cant for the love of me figure out how in the world it can be possible using a t-test given the survey. – Mike Victor Sep 01 '19 at 08:50
  • You can't with ttest unless you have been given scoring eg strongly agree=2 etc. You can do chisquared. Or you could do? Ordinal regression and then t test coefficient? – seanv507 Sep 01 '19 at 09:55
  • @seanv507 If i do ordinal regression. I would have gender as the categorical variable. And the scoring 1-5 as the ordinal dependent variable? And then given there are 15 questions. Would I have to do this 15 times? – Mike Victor Sep 01 '19 at 17:07
  • I assume @bruceET's answer is probably all that is required, it is hard to say with the information given and you are better off asking your classmates – seanv507 Sep 01 '19 at 18:57

3 Answers3

4

In order to use a two-sample t test, you would have to assume that the Likert scores are numerical in the sense that averages are meaningful. The validity of that assumption has been debated by social scientists for some time.

Here are some fake data in which 200 men are consistently neutral, while 200 women tend to be decisively pro or con. Thus their average Likert scores are about the same. The result is that a two-sample Welch t test finds no difference between the responses of men and women. (P-value 0.53 > 0.05.) There might be a slight difference in average Likert scores, but not large enough to be statistically significant. (There is no way to prove men and women have the same population average Likert scores.)

set.seed(901)
m = sample(1:5, 200, rep=T, p=c(1,2,3,2,1))
f = sample(1:5, 200, rep=T, p=c(2,2,1,2,2))
t.test(m,f)

        Welch Two Sample t-test

data:  m and f
t = 0.62763, df = 380.6, p-value = 0.5306
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.1812832  0.3512832
sample estimates:
mean of x mean of y 
    3.015     2.930

However, as mentioned above, the patterns of Likert responses are substantially different between men and women. Below we tabulate how many Likert responses of each type we got from the men and (separately) form the women. The table MF shows the counts. A chi-squared test for homogeneity of responses decisively rejects the null hypothesis that men and women have the same probabilities of responses across Likert categories (P-value near 0.)

M = tabulate(m);  F = tabulate(f)
MF = rbind(M, F);  MF
  [,1] [,2] [,3] [,4] [,5]
M   23   47   60   44   26
F   44   53   20   39   44

chisq.test(MF)

        Pearson's Chi-squared test

data:  MF
X-squared = 31.872, df = 4, p-value = 2.032e-06

For these data, the Pearson residuals highlight that men and women have very different proportions of neutral (Likert-3) responses. (if there were no differences between men and women in this regard, we would expect about 40 Likert-3 responses among men and among women---as you could see from chisq.test(MF)$exp, not shown)

chisq.test(MF)$resi
       [,1]       [,2]      [,3]       [,4]      [,5]
M -1.814124 -0.4242641  3.162278  0.3880753 -1.521278
F  1.814124  0.4242641 -3.162278 -0.3880753  1.521278

Of course, I don't have your real data, so I can't say whether a two-sample t test would give a meaningful result for them. I'm just saying that a t test might not be at all what you need to use. What does it mean quantitatively to ask does "gender influence responses"? The two tests answer that question in completely different ways.

BruceET
  • 47,896
  • 2
  • 28
  • 76
  • 1
    The comparison in a contingency table with an associated chi-squared test is problematic. The OP has a situation where the men and women are answering 15 questions. When you would tabulate this in a contingency table then this should not be considered as multinomial distributed data. This is because the answers on the questions might be over-dispersed due to correlation in answers within the same person. The result is that you will be more likely to observe a larger chi-squared value than expected. – Sextus Empiricus Sep 02 '19 at 15:58
  • "In order to use a two-sample t test, you would have to assume that the Likert scores are numerical in the sense that averages are meaningful." Not only that, but that the scores are drawn from a normal distribution (the CLT helps somewhat with justifying that assumption, but with an $n$ of 36, that gets rather dodgy). In the toy example you gave, the distribution is decidedly non-normal, especially for the women). – Acccumulation Sep 03 '19 at 22:20
  • @Accumulation. I used samples of size 200 for my t test discussion, specifically to avoid doubts about non-normality. Simulating 5000 averages of 200 according to the hypothetical dist'n of Likert scores for women, gave a vector of averages that passes the Shapiro-Wilk test for normality. As I have discussed, there are difficulties using a 2-sample t test, but in my discussion nonnormality is not among them. [5000 is max capacity of S-W test in R.] – BruceET Sep 03 '19 at 23:11
0

You could take the sum on all 36/40 participants answers (with weights 1 to 5), giving scores respectively between 36 and 180 and 40 and 200.

Those summed statistics are variables that are approximately normal distributed (since it derives from a multinomial distributed variable* - probabilities $p_i$ to choose level $i$ - which is approximately multivariate normal distributed and thus a linear combination/sum of its components are normal distributed).

You can use variance in the samples (of size 36 and 40 for men and women) to estimate the variance of the statistic and use a t-test to test the difference for men and women.

*Technically this distribution might not be a multinomial distribution, but instead more like an overdispersed multinomial. For instance the parameters $p_i$ may be different for different women/men.

Note that this weighted sum statistic is a bit simplistic and arbitrary and there may be more powerfull or general methods to test the difference.

The chi-squared test described by BruceET does not take a linear combination of the numbers in the 5 categories but instead compares them all separately. What still remains is that the test approximates the distribution of the observed numbers as normal distributed variables.

I imagine that instead of doing the chi-squared test (which has an issue if you wish to aggregate all numbers in the categories for both questions and participants, since then it is not anymore a binomial/multinomial distributed variable as there different multinomial distributed variables are correlated within questions/participants) it might be possible to devise some linear model with random effects.

Sextus Empiricus
  • 43,080
  • 1
  • 72
  • 161
  • Treating the sum of variables as normal, when $n$ is only $15$ and there are likely significant correlations, isn't very sound. There is also the question of how meaningful it is. What does "How much you like X" plus "How much you like Y" even mean? Especially if X and Y are opposites in some way. – Acccumulation Sep 03 '19 at 22:26
  • @Acccumulation you are right there may not be consistency which needs to be first researched. But, I would say that a score created out of 15 categorical variables can be approached as normal. Of course there might be correlations causing under/over-dispersion but that is why you need to use the variance in the sample as a measure of this instead of computing it based on a iid multinomial distributed variables. – Sextus Empiricus Sep 04 '19 at 07:06
  • I have edited it now since, indeed, summing 15 variables is not automatically a good solution. – Sextus Empiricus Sep 04 '19 at 07:12
0

Because my answer below has two downvotes???, let me (partially) derive the exact 2-sample tests (two of them) for this particular problem to show that they have absolutely nothing to do with any t-test.

For each male/female group, let pi, i=1,5 be the probability to get an answer of type i (Strongly Disagree = 1, Disagree = 2, ...).

p1 + p2 + p3 + p4 + p5 = 1

Therefore, there are 4 free parameters of interest pi, i=1,4.

Let Ni, i=1,5 be the number of answers of type i in the sample.

If the answers are conditionally i.i.d. upon parameters pi, i=1,4, the likelihood is

p(answers|p1,p2,p3,p4) = p1^N1 p2^N2 p3^N3 p4^N4 (1-p1-p2-p3-p4)^N5

Assign a joint prior over [0,1]^4 for the parameters

p(p1,p2,p3,p4), e.g. p(p1)p(p2)p(p3)p(p4)

(this looks interesting: how to assign a prior such as p(p1,p2,p3,p4,p5) is exchangeable/invariant under permutations?)

Now let theta1 = (p1,p2,p3,p4) be the vector of the parameters for the male group and theta2 be the vector of the parameters for the female group.

The problem boils down to testing the null hypothesis H0

H0: theta1 = theta2 (no difference between male and female answers)

against the omnibus alternative H1

H1: theta1 <> theta2

This is done by computing the generalized Bayes factor

p(H0|male answers, female answers) / p(H0)

as explained here

Bayes-Poincaré solution to k-sample tests for comparison and the Behrens-Fisher problem?

and here

Bayes-Poincaré solution to the Behrens-Fisher problem 2: calculations for Jeffreys’ priors

If you do the calculations (good exercise), you will find something that has absolutely nothing to do with Student t distribution or any t test. Good luck!

PS: Oops, I forgot that there are 15 questions for each subject, whose answers are not necessarily identically distributed in each group! My answer assumes that there is only one question for each subject or that the answers are identically distributed over the questions. But this is not a big deal: just introduce parameters (p1j,p2j,p3j,p4j) for each question j=1,15 and the numbers Nij of answers of each type i=1,5 for each question. The likelihood is the product of the likelihoods for each question. Same for the joint prior. Theta is the vector of all 4x15=60 parameters.

Whether you should use the 2x4=8 parameters test or the 2x4x15=120 parameters test depends on the data at hand: if the counts Nij are available, then you should use the 120 parameters test unless you have good reasons to believe that the answers are identically distributed over the questions. If only the counts Ni are available, then use the 8 parameters test.

  • Wow. Thanks Fabrice Pautot! And I will definately look into the Bayes-Poincare and others you mention. I always do love doing my own calculation because I’m being taught this for reason that can be handy later in life. I think I understand you what you mean. To be honest I learned more from this assignment through research to figure out than I did in classes. Because before posting here I tried to figure out for 3 days researching to figure it out. My teacher is known to be very harsh but everyone that takes class with teacher is considered very smart. – Mike Victor Sep 03 '19 at 21:52
  • @MikeVictor I'm very happy to help Mike! It warms my heart. A little warning:the Bayes-Poincaré stuff is non-academic, unofficial, unpublished research material. I've analyzed the official, academic solutions (including the t test(s)) and found them to be incorrect. Then, I've proposed new and completely different tests. I've asked whether my solution is right or wrong. No objection against my solution as of today but no confirmation too. It will take some time. My solution definitely looks better than the offical one but make your own opinion. –  Sep 03 '19 at 22:04
  • @MikeVictor I'd like to know what your teacher said: is it possible or not to use a t-test for this problem? –  Sep 03 '19 at 22:06
  • @MikeVictor And don't hesitate to validate my answer please, I don't have many points! –  Sep 03 '19 at 22:07