0

Most examples in my Time Series Analysis slides and Multivariate Analysis textbook (Applied Multivariate Statistical Analysis, 6th Edition) conduct a hypothesis testing on data's normality, or zero correlation, or independence, or homogeneity of variance, or whatever, and readily assume these properties to be true on failure to reject the hypothesis. Actually, I am explicitly taught that a property holds if it cannot be rejected.

For example, in Example 6.14 (A profile analysis of love and marriage data) from the textbook AMSA, authors "conclude that the hypothesis of parallel profiles for men and women is tenable" when $H_0$ cannot be rejected. In addition, they also said "... we cannot reject the hypothesis that the profiles are coincident. That is, the responses of men and women to the four questions posed appear to be the same." Here is the whole example: P.325, P.326, P.327, P.328, where the quotes are from the bottom of P.327 and the top of P.328 respectively.

This seems ridiculous, as absence of evidence is not evidence of an absence. Even if $p = 1$, it's still circular reasoning, let along cases where $p$ isn't even remotely close to 1. Back when I was learning Mathematical Statistics, the professor would kick their ass if someone accept $H_0$ on failure to reject it, but the standard is clearly loosened a lot in upper division courses.

I just want to know is this practice a de facto procedure in intermediate Statistic courses like Time Series Analysis and Multivariate Analysis? If so, what the rationale?

nalzok
  • 1,385
  • 12
  • 24
  • Can you provide more context? Your examples (normality, etc.) suggest these claims have to do with assumptions of inferential tests/procedures. If so, while never strictly true (as you say), a lack of evidence for non-normality of residuals (for example) is taken as evidence that the assumption of normality is *approximately* true; i.e. the deviation is not so strong that you need to alter your procedures & conclusions. – mkt May 08 '19 at 08:23
  • @mkt Hi, I'm not sure what do you mean by "context", but on failure to reject normality, the authors usually go on to apply statistical methods only applicable to data drawn from a (multivariate) normal distribution. Another example is *Example 6.14 (A profile analysis of love and marriage data)* from the textbook AMSA, where authors "conclude that the hypothesis of parallel profiles for men and women is tenable" when $H_0$ cannot be rejected. – nalzok May 08 '19 at 08:36
  • Most of us do not have access to the book you are referring to, so relevant quotations that expand on your claims about what the authors say is what I meant by 'context'. In the one quote you provide, concluding that two groups being similar 'is tenable' based on a failure to reject H0 hardly seems like a radical claim. – mkt May 08 '19 at 08:50
  • 1
    @mkt The whole example is way too long to be typed by hand, so I have included it as screenshots in the original post. Hope you don't mind. – nalzok May 08 '19 at 09:06
  • "the standard is clearly loosened a lot in upper division courses": you are making a statement based on a sample of 1 textbook. Otherwise, I agree. I don't like the wording you cite from that text. Failure to reject the null is not establishing it as true. – Nick Cox May 08 '19 at 09:21
  • @NickCox Yeah I did jump to a conclusion, but my professor told us this textbook is really well-established and that "everyone uses it", so I was under the impression that it can represent the whole field? – nalzok May 08 '19 at 09:26
  • "Everyone uses it". Well, this is the same kind of problem. I have heard of it but never used it, even opened it. Is that enough refutation? More seriously, even good textbooks can be wrong, so never believe something puzzling just because you find it in one place. As a serious example, the multivolume text by M.G. Kendall, later Kendall and A. Stuart, later Stuart and J.K. Ord still contains some amazing errors. A standard blood sport for some decades was to write a review of each new edition pointing out what was wrong. (Some "errors" were lack of rigour by the reviewer's standards.) – Nick Cox May 08 '19 at 09:48
  • @NickCox Thanks for the insight! Actually, can you refer me to a better textbook? From my current understanding, hypotheses are chained together in profile analysis, so you are supposed to make some assumption about the first claim to proceed. Also, many other Statistical methods require normality, but hypothesis testing can only reject rather than prove it. How can this be resolved? – nalzok May 08 '19 at 10:05
  • All I could do is point to multivariate texts I like. I don't think that any necessarily says much about time series analysis. I have no idea what your background and tastes are. But -- as said, without reading it -- the Johnson and Wichern text I am guessing to be rather old-fashioned. Most recent texts show a lot of awareness of machine learning. I think you'll find that multivariate normality, although convenient when you have it, isn't nearly as central to multivariate analysis as some older texts imply. Any text with first edition before say 2000 is unlikely to be updated enough. – Nick Cox May 08 '19 at 10:13
  • @NickCox Thanks sir, I'll have a look. Regarding the (relative) insignificance of normality, let's consider homogeneity of variance or independence instead, which are presumably much more important. Again, hypothesis testing can only reject them, and failure to reject is sometimes incorrectly interpreted as acceptance. I'm curious to learn methods which can prove these properties. – nalzok May 08 '19 at 10:28
  • You can't prove that properties are satisfied absolutely. This is where you started! Even independence isn't crucial. What's crucial is whether procedures you use assume it. – Nick Cox May 08 '19 at 10:34
  • @NickCox Oh, you mean I should make assumptions about the data (sample), and then pick inference precedures accordingly? – nalzok May 08 '19 at 10:41
  • Say rather: Whatever assumptions you make should be checked, although that's not always easy. My final comment is that the term _assumptions_ isn't always helpful. It's true that there is much theory that says if this is so, then that is so. But in practical data analysis the assumptions serve more as _ideal conditions_, and not everything falls over if they aren't satisfied. In applying PCA, for example, I might find it better to work on a transformed scale, so that distributions are roughly symmetrical and outliers pulled in, but exact normality is never needed for any use I make of PCs. – Nick Cox May 08 '19 at 11:01
  • 4
    @NickCox I see, so hypothesis testing is more of a means to check/verify the data at hand don't drift too much from the ideal conditions of inference procedures to be employed, but still, nothing can be *proved* from this check. I will know if my assumptions are absurdly wrong, but there is a chance that the data will pass the check while the assumptions aren't meet, but that's OK since tiny drifts don't invalidate the applicability of inference procedures. – nalzok May 08 '19 at 11:12

1 Answers1

2

Here is a demonstration using four samples of size $n = 15$ from uniform, gamma, beta, and normal populations.

Tests of normality. In all four cases the data 'pass' a Shapiro-Wilk test for normality. That is, we cannot reject the null hypothesis that data are normal. Approximate P-values for the normality test are 0.40, 0.48, 0.09, and 0.36, respectively--all clearly above 0.05.

The PDFs of the first three distributions are all substantially different from normal, but $n = 15$ observations are not enough to detect the departure from normality.

Robustness of the t test: Also, in all four cases we use a one-sample t test to test a null hypothesis about the population mean that is false. For all four samples, the null hypothesis is (correctly) rejected. Even for a sample size as small as $n = 15,$ the test detects that the various hypothetical values of the population mean are not correct.

The t test is said to be 'robust' against departures from normality---as long as there are no extreme outliers. Although the first three samples are non-normal, the t test still gives useful results.

Here are boxplots of the four samples;

enter image description here

set.seed(1234);  n = 15
u = runif(n);  shapiro.test(u)$p.val;  t.test(u, mu=.7)$p.val
[1] 0.3963761   # P-value for normality test
 [1] 0.0145073  # P-value for t test

v = rgamma(n, 5, 1); shapiro.test(v)$p.val; 
t.test(v, mu=2)$p.val
[1] 0.4827358
 [1] 5.112448e-05

w = rbeta(n,3,4); shapiro.test(w)$p.val; 
t.test(w,mu=.7)$p.val
[1] 0.08774671
 [1] 9.815778e-05

z = rnorm(n); shapiro.test(z)$p.val; 
t.test(z,mu=.7)$p.val
[1] 0.3592276
 [1] 0.003356458

Note: The seed for the simulation is given so that you can reproduce exactly the same simulated data used here. Different seeds might give substantially different results.

BruceET
  • 47,896
  • 2
  • 28
  • 76