0

This is an elementary question, but I'm a bit confused on the concept of standard error.

On some sources, I've seen that standard error is defined as the standard deviation or the square root of the variance.

On other sources, I've seen standard error defined as the standard deviation divided by the square root of the sample size.

When is each one used? From my understanding, the former is used when computing the standard error for a random variable, and the latter is used when computing the standard error of some number of samples. In addition, the former involves probability of each event a random variable can take on whereas the latter does not involve probability. Is this correct?

roulette01
  • 271
  • 1
  • 7
  • See https://stats.stackexchange.com/a/18609/919. – whuber Apr 01 '20 at 20:10
  • the latter is the standard error of the mean – gunes Apr 01 '20 at 20:10
  • @gunes What is the former called? Standard error of a random variable? – roulette01 Apr 01 '20 at 20:17
  • @whuber In your answer, it appears you used the latter version, with the division over the square root of the sample size. In which case is the former used? – roulette01 Apr 01 '20 at 20:44
  • 1
    I did not define the standard error in terms of a formula: I provided the concept. The formulas all depend on the probability model and the statistic. – whuber Apr 01 '20 at 21:44
  • 1
    It is difficult to know how to answer your question with no clues about the level of your course. You give no context in your question. And your profile for the site says nothing about your current interests or level of study. – BruceET Apr 01 '20 at 22:27

1 Answers1

1

Here simple examples, in which I focus on the terminology that seems to be confusing you.

Suppose you have a sample of size 100 from the distribution $\mathsf{Norm}(\mu=200, \sigma=25).$

Then the standard deviation of a single observation is $SD(X_i) = \sigma = 25.$ The standard deviation of $\bar X,$ also called the 'standard error of the (sample) mean', is $SD(\bar X) = \sigma/\sqrt{n} = 25/10 = 2.5.$

The usual estimated standard deviation of single observation $X_i$ from the population is $S = \sqrt{\frac{\sum_{i=1}^n (X_i - \bar X)^2}{n-1}}.$

[Notice that $E(S^2) = \sigma^2,$ but $E(S)$ is not exactly equal to $\sigma.$ It's very close for large $n$ and in practice the small discrepancy is usually ignored. If $n = 5,$ for example, then $E(S) \approx 0.940\sigma.$ But if $n = 50,$ we have $E(S) \approx 0.995\sigma.)$

Accordingly, the usual '(estimated) standard error' of the sample mean is $S/\sqrt{n},$ where the word estimated is often omitted for brevity (because as soon as you realize you're using the estimated SD $S$ you have to know the standard error is also an estimate).

If $\mu$ is unknown and $\sigma$ is known, then a 95% confidence interval for $\mu$ is of the form $$\bar X \pm 1.96\frac{\sigma}{\sqrt{n}},$$ where the last factor $\frac{\sigma}{\sqrt{n}}$ in the margin of error is the (exact) standard error of the sample mean $\bar X.$

[The amount $1.96\frac{\sigma}{\sqrt{n}}$ added to and subtracted from $\bar X$ is the 'margin of error' of the confidence interval. Notice that it becomes smaller as $n$ increases. A CI based on $n = 400$ observations is half as long and a CI based on $n = 100$ observations from the same distribution.]

By contrast, if both $\mu$ and $\sigma$ are unknown, then a 95% CI for $\mu$ is of the form

$$\bar X \pm t^*\frac{S}{\sqrt{n}},$$ where $t^*$ cuts probability 0.025 from the upper tail of Student's t distribution with $n-1$ degrees of freedom and the (estimated) standard error of $\bar X$ is $\frac{S}{\sqrt{n}}.$

[In this case, it is also possible to get a confidence interval for $\sigma$ by using a chi-squared distribution, but this confidence interval is of an entirely different form: It is not symmetrical about $S$ and it does not use a 'standard error' of $S.]$

BruceET
  • 47,896
  • 2
  • 28
  • 76
  • Is the terminology "standard deviation of a single observation" valid? I'm tempted to say the standard deviation of a single observation as zero (because I'd just consider a single sample). Would it be more correct to say "standard deviation of a random variable?" – roulette01 Apr 01 '20 at 21:32
  • Also, if $E[s^2] = \sigma^2$, why is $E[s] \neq \sigma$? Shouldn't both be $\neq$? – roulette01 Apr 01 '20 at 21:36
  • OK then 'standard deviation of a randomly sampled observation'. // $E(aX + b) = aE(X) + b,$ for random variable $X$ and constants $a$ and $b.$ However, expectation is a 'linear operator' because of the way it is defined. So equality does not survive nonlinear operations like taking the square root. – BruceET Apr 01 '20 at 21:43
  • I'm confused why $E[S^2] = \sigma^2$ exactly. Could you explain? – roulette01 Apr 01 '20 at 21:47
  • Maybe you know the general formula $Var(X) = E(X^2) - [E(X)]^2.$ Consider $X \sim \mathsf{Norm}(\mu = 100, \sigma=15).$ Then $E(X^2) \ne [E(X)]^2.$ Instead $E(X^2) = \sigma^2 + \mu^2 = 15^2 + 100^2 = 10225.$ Maybe you haven't covered such relationships yet. If not, then I suspect soon. – BruceET Apr 01 '20 at 22:02
  • Proving $E(S^2) = \sigma^2$ is a bit much to show in a Comment. Maybe a simulation in R will help convince you. Generate a random sample of 10 million from NORM(100,15), using code $\mathtt{x = rnorm(10^7, 100, 15)}.$ Then take the variance of the huge sample with $\mathtt{var(x)}$ to get 225.0505 which is very nearly $\sigma^2 = 225.$ Maybe google 'expectation of sample variance'. – BruceET Apr 01 '20 at 22:15