7

When performing GAM and GLM fits to the same data set, I get an almost identical fit in terms of fitting metrics. However, the variables that are identified as significant differ between the two approaches. How do decide which method is most appropriate?

Lyngbakr
  • 741
  • 1
  • 7
  • 16

1 Answers1

6

The reason that the significance of the coefficients differs is because, under the modeling assumptions, the calculation of standard errors differs. The correct answer is to choose the model that is most appropriate for--not the data--the scientific question at hand.

For instance, suppose I am interested in measuring a difference in rates of some disease in communities before and after applying some intervention. I could model the additive rates using a Poisson GLM with the identity link. This presumes that the distribution of rates follows a Poisson distribution. On the other hand, I could model these rates using a simple t-test. This makes no assumption about the distribution of rates and accounts for the possible underdispersion due to infections within a community not necessarily having a constant inter-arrival time. These two models will estimate the same rate difference, but give different standard errors.

Which, then, is correct? Well, the adequacy of the underpinning assumptions is what is called into question. Therefore, you should endeavor to describe and rationalize the model that is appropriate for the data--whether or not that model happens to be significant.

AdamO
  • 52,330
  • 5
  • 104
  • 209
  • 1
    I'm not sure it's quite correct to say that a t-test "makes no assumption about the distribution"; how then does one derive the t-distribution that is used to calculate the p-values? – Glen_b Mar 05 '15 at 00:51
  • Thanks for the reply, AdamO. What I am aiming to achieve is to identify the most significant variables, but I don't know what the distribution is or should be - Poisson, Gaussian, or other. When fitting a GAM or GLM there are points of agreement, however (e.g., one variable _always_ is most significant). In the absence of knowledge of the distributions, I defaulted to Gaussian & identity and assumed that if GAM and GLM give the same results, it means I should go with GLM. I have no reason to think it's _not_ Gaussian, but that isn't great rationale. Do I try many different distributions? – Lyngbakr Mar 05 '15 at 03:08
  • @Glen_b In two steps: by way of the Central Limit Theorem and also by way of using the T-approximation to the normal distribution when the standard deviation is estimated using the sample mean. The independence of events is an important assumption as well as having a sufficient sample size so that the T approximation to the sampling distribution of the test statistic is "good enough". This is usually a modest number. – AdamO Mar 05 '15 at 19:07
  • 1
    @Lyngbakr you say that you aim to "...identify the most significant variables". This is not an aim, it is a statistical fishing expedition. In both exploratory and confirmatory analyses, it is very useful for the statistician to work directly with the investigators to prespecify a set of hypotheses. – AdamO Mar 05 '15 at 19:09
  • AdamO The t-statistic consists of a numerator and a denominator, so an argument about the numerator alone isn't enough to establish normality of the ratio (though Slutsky's theorem can be applied), and then the result is only in the limit as $n\to\infty$; to say anything about finite sample sizes you'd need to use something like Berry-Esseen. ...(ctd) – Glen_b Mar 05 '15 at 21:11
  • (ctd) ... But even then, saying "In large samples, under certain conditions and given the null, the distribution of the t-statistic gets within a certain distance of a normal distribution, and the t-distribution also gets close to that same distribution" is *not at all* the same thing as "the t-test makes no assumption about the distribution". For that to be true, you'd have to be able to show that at a given $n$, without making any distributional assumption at all, you can derive the t-distribution (at least approximately) from the t-statistic. – Glen_b Mar 05 '15 at 21:12
  • @Glen_b We can both agree that it is easy to contrive a variety of distributions for which the power of the t-test for the sample mean will be vastly different... in some cases *better* than with normally distributed samples--in most cases *worse*. However, for a test to be *valid*: it must be consistent, i.e. power goes to 1 as n goes to infty. Also it must achieve nominal 0.05 level for all n. I encourage you to attack this with simulation. You see the t-test does this for all distributions. That is what I mean by application of t-test in this context. – AdamO Mar 06 '15 at 19:04
  • @Glen_b in my particular example, though, I want to underscore that we are considering differences in a model based approach versus using the t-test for Poisson distributed data. The GLM with binary exposure (treated vs untreated) assumes the counts for each group to follow a Poisson distribution. The model estimates a relative rate and conducts inference based on the assumed mean-variance relationship: variance=mean. However, the t-test estimates as an effect a rate difference (diff of means of counts), and uses the empirical variance of those counts rather than a mean-variance relationship. – AdamO Mar 06 '15 at 19:06
  • The issue I raised was your statement about the *assumptions* of the t-test, not its behaviour when the assumptions don't hold. However, it's rather easy to see by simulation that the t-test doesn't have its nominal significance level for all distributions, even in very large samples. – Glen_b Mar 07 '15 at 18:04
  • There's an example [here](http://stats.stackexchange.com/questions/111320/should-i-use-t-test-on-highly-skewed-and-discrete-data). Distinctly non-t-distributed results (with clear effects on type I error rate) can also occur with continuous data. I wonder if perhaps I've misnterpreted your intentions with the statement about the assumptions (such as its generality). – Glen_b Mar 07 '15 at 18:16
  • @Glen_b I contend that the t-test is consistent (power goes to 1 as n goes to infinity) and is of the correct size for any sample size and for any underlying distribution of finite variance. Therefore the only assumption necessary for the t-test is independent and finite variance distributions. – AdamO Mar 09 '15 at 21:45
  • @AdamO I don't think the contention is true, since, for example, in my previous comment I linked to an example with finite variance where it definitely isn't of the correct size. I can show many more (indeed its size under various conditions has been the subject of numerous papers). – Glen_b Mar 10 '15 at 00:50
  • @Glen_b yes the consequence of the spike is that the test winds up being conservative (does not achieve the actual level of test) and may be of low power. Despite this, the test is still consistent and achieves the right level in large sample sizes regardless. It's not perfect, but there are actually few tests (aside from convenient mathematical corrections) that do not have this properties with highly erratic data. [Here is my take on the issue, same findings but different conclusions](http://pastebin.com/AAjpVdsg). – AdamO Mar 10 '15 at 18:29