3

Let's say you have the following basic one-sample t-test, taken from Khan Academy:

A neurologist is testing the effect of a drug on response time by injecting 100 rats with a unit dose of the drug, subjecting each to neurological stimulus, and recording its response time. The neurologist knows that the mean response time for rats not injected with the drug is 1.2 seconds. The mean of the 100 injected rats' response times is 1.05 seconds with a sample standard deviation of 0.5 seconds. Do you think that the drug has an effect on response time?

When reviewing inferential stats, I was rethinking an issue I didn't understand with t-testing. What if the sample standard deviation for rats injected with a drug is completely different than the standard deviation of the population of rats who are not injected with the drug?

It would make sense that this is a real possibility. Maybe the drug clusters everyone around the sample mean (i.e. every rat who takes the drug gets really close to 1.05 seconds), and the sample standard deviation is half that of the standard deviation of the population of non-injected rats. I could see this happening if you were measuring how an athletic drink affects sprint time, and the treatment sample (people who had this drink) has a very small standard deviation compared to the population (people who did not have the drink) because everyone who drinks it gets around the same very fast sprint time.

Why can we then use the treatment-group sample standard deviation to estimate the control-group population standard deviation if they have a chance of being very different from each other? Or do we just assume they aren't?

amoeba
  • 93,463
  • 28
  • 275
  • 317
rb612
  • 644
  • 5
  • 16
  • It seems to me that none of the three answers addresses your main question, and that's maybe because you formulated it wrongly: in your example it's not that the sample SD is different from the population SD! The problem here is that the SD (both sample and population) in the treatment group is not equal to the SD in the control group. Or in other words, SD under null hypothesis (control group) is different from the SD under treatment. However, I don't think it matters for one-sample t-test (as opposed to the two-sample t-test): here you are told that the control mean is *exactly* $1.2$. – amoeba Oct 29 '15 at 10:24
  • @amoeba you're right, and my new question describes this much better. http://stats.stackexchange.com/q/179145/83967 – rb612 Oct 29 '15 at 12:41
  • @amoeba so it seems like you're the one who completely understands my question. Any way you can explain why the different SD doesn't matter? – rb612 Oct 30 '15 at 04:43
  • 1
    This whole thread is very confusing. First of all, you shouldn't accept an answer if you think it did not answer your question (note that you can always de-accept an answer, or move the acceptance tick any way you want). Currently this Q has an "accepted" answer so it *looks* like the issue is solved for you (whereas it isn't). Second, your new question looks to me like an **exact duplicate** of this one. It's not good, and I am considering to vote to close it as such. In general, if a Q is misunderstood, try editing it instead of posting another one. – amoeba Oct 30 '15 at 09:49
  • In addition to the above, I have now edited your question to make it more clear. In particular, I edited the title which was outright misleading. Can you please take a careful look at my edits and see if they indeed represent what you want to ask? – amoeba Oct 30 '15 at 10:00
  • 1
    @amoeba thank you for helping clear all this confusion up. I'll try to delete the other question - I posted it because of what John said about it being a very different question, but you're right, it was a duplicate. Your edits indeed to represent what I'm trying to ask. You understand what I'm trying to say, and I'd appreciate it if you could post an answer explaining why we don't need to estimate the population variance. I believe it's because the t-test only represents if the null hypothesis is true, but I'm not sure if there's more to it than that. – rb612 Oct 30 '15 at 17:59
  • This post [Unequal variance in randomized experiments to compare treatment with control?](https://stats.stackexchange.com/questions/434928/unequal-variance-in-randomized-experiments-to-compare-treatment-with-control) is related! – kjetil b halvorsen Nov 03 '21 at 00:21

3 Answers3

2

The answer by @Glen_b and @John are very good general answer, but they don't adress a misconception of yours.

You write

What if the sample standard deviation is completely different than the population standard deviation? It would make sense that this is a real possibility. Maybe the drug clusters everyone around the sample mean (i.e. every rat who takes the drug gets really close to 1.05 seconds), and the sample standard deviation is half that of the population standard deviation. I could see this happening if you were measuring how an athletic drink affects sprint time, and the sample has a very small standard deviation compared to the population because everyone who drinks it gets around the same very fast sprint time.

The sample needs to be representative of the population you are interested in. In the case of the rats who take the drugs, your sample represents the population of all rats if the entire population also takes the drug. Any affect the drug has on the sample standard deviation, it would also have on the entire population. The same for the runners: the sample represents all runners when drinking the special drink.

The population in statistical inference does not always exists physically, often it does so just conceptually. For example, if you look at throwing a coin, there is no real physical population of all throws that could ever be made with the coin.

Erik
  • 6,909
  • 20
  • 48
1

You're absolutely correct that they're very different. The particular possibility you mentioned, that the SD is much smaller, is the reason that n-1 is the denominator in the sample standard deviation (SD) but, of course, the SD could be very different for other reasons. That's true of every property of the sample.

This is why the assumption of representativeness is critical in statistics. You won't find it listed as an assumption of any test but if you don't assume that the sample is in some way representative then using samples is pointless.

Keep in mind that one doesn't always use the sample SD, or only the sample SD. In Bayesian statistics you look at prior research and weight or adjust the sample SD given the relatively believability of the prior information and the sample. Or, you might have some reasonable estimate of population SD that you substitute knowing that SD estimates can be highly variable. And consider that in the t-test itself you typically pool the the SD's from the samples to generate a more reliable estimate.

If you're thinking along these lines you might enjoy working with simulations. You can get a better feel for exactly what happens when sampling occurs moreso than calculations of standard error or Type X error rates provide.

These are easy to do in R:

sd( rnorm(100, 1.2, 1.05) )

That gives you a SD from a random sample. You can use a function like replicate to make lots of them and look at the distributions of them with hist.

John
  • 21,167
  • 9
  • 48
  • 84
  • Great answer! Thank you! Is there a simulation you would recommend me checking out that goes along with this idea? – rb612 Oct 29 '15 at 01:44
  • And also, just to make sure I understand, when you say that we're assuming representativeness, we have to assume that the sample standard deviation is a reflection of the population standard deviation even though they could be very different? If so, why can we do this at all? – rb612 Oct 29 '15 at 01:49
  • 1
    I've noted some exceptions now in the answer but the main reason is that you have to when you have no other options. – John Oct 29 '15 at 01:50
  • so I think I'm starting to understand it better. I think now my question can be rephrased: it seems like with t-testing, we have to assume the sample comes from the population distribution if the null hypothesis is true. However, the population distribution if the null hypothesis is false might be much different, with a completely different standard deviation. But we're assuming the sample comes from a population distribution when the null hypothesis is true. Why is this a valid assumption? – rb612 Oct 29 '15 at 02:05
  • This is a new question. – John Oct 29 '15 at 03:20
1

It can certainly happen that the sample sd is substantially larger or smaller than the population sd, $\sigma$.

Indeed, if the data are drawn independently from a normal distribution, the distribution of sd is right skew, and the smaller the sample size the more skew it is. Here's two example distributions for $s/\sigma$ for sample sizes 7 and 11:

enter image description here

Note that both peaks are to the left of 1, and indeed, you can see that $s$ is more often below $\sigma$ ($s/\sigma<1$) than above it (the median of the distribution of $s$ is also below $\sigma$).

But this is exactly what the $t$-distribution accounts for. The $t$ distribution has heavier tails than the normal distribution precisely because $s$ can be smaller than $\sigma$. It is also "peakier" than the normal, because $s$ can be larger than $\sigma$.

So in the end, you can use the sample standard deviation because the t-distribution exactly accounts for the fact that $s$ may be substantially larger or smaller than $\sigma$ (at least it accounts for it when the data are iid normal; fortunately if the sample sizes aren't small and the distribution isn't very far from normal it's not especially sensitive to the distribution).

See the discussion here for further discussion about the connection between the distribution of $s$ and the distribution of the $t$-statistic.

Community
  • 1
Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • +1, but I think this answer (together with both other answers) misunderstood the real question OP is asking. The title used to say "why use sample variance to estimate population variance", but that seems to have been misleading. My interpretation of the Q is: why can we use sample variance *of the treatment group* to estimate population variance *of the control group*? The answer to this is that we usually can't but also don't need to; in particular, the quoted Q is about one-sample t-test, where control variance simply does not matter at all. Or am I misreading the Q? I have now edited it. – amoeba Oct 30 '15 at 10:04