11

I am running glms in R (generalised linear models). I thought I knew pvalues - until I saw that calling up a summary for a glm does not give you an overriding pvalue representative of the model as a whole - at least not in the place where linear models do.

I am wondering if this is given as the pvalue for the Intercept, at the top of the table of coefficients. So in the following example, while Wind.speed..knots and canopy_density may be significant to the model, how do we know whether the model itself is significant? How do I know whether to trust these values? Am I right to wonder that the Pr(>|z|) for (Intercept) represents the significance of the model? Is this model significant folks??? Thanks!

I should note running an F-test will not give a pvalue as I get an error message saying that running F-tests on binomial family is inappropriate.

Call:
glm(formula = Empetrum_bin ~ Wind.speed..knots. + canopy_density, 
    family = binomial, data = CAIRNGORM)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.2327  -0.7167  -0.4302  -0.1855   2.3194  

Coefficients:
                   Estimate Std. Error z value Pr(>|z|)  
(Intercept)          1.8226     1.2030   1.515   0.1298  
Wind.speed..knots.  -0.5791     0.2628  -2.203   0.0276 *
canopy_density      -2.5733     1.1346  -2.268   0.0233 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 59.598  on 58  degrees of freedom
Residual deviance: 50.611  on 56  degrees of freedom
  (1 observation deleted due to missingness)
AIC: 56.611
Martin
  • 111
  • 1
  • 1
  • 3
  • 1
    If I recall correctly in a binomial regression you should be using a test of deviance, aka a Log Likelihood test - which is analogous to the F-test used in a linear regression. This is the -2 * (Null LL - Saturated LL) Which fits a chi-squared distribution. Although I cannot see a model Log likelihood in your summary output. Is it in the model object but not in the summary output? – SamPassmore Dec 22 '14 at 00:49

3 Answers3

10

You can either do an asymptotic chi-square test of (59.598-50.611) vs a chi-square with (58-56) df, or use anova() on your glm object (that doesn't do the test directly, but at least calculates (59.598-50.611) and (58-56) for you).

This is effectively analysis of deviance.

Here's the sort of calculations you could do (on a different data set, which comes with R):

spray1=glm(count~spray,family=poisson,data=InsectSprays)  # full model
spray0=glm(count~1,family=poisson,data=InsectSprays)      # null model
with(anova(spray0,spray1),pchisq(Deviance,Df,lower.tail=FALSE)[2])

Which gives the p-value for an asymptotic chi square statstic based on the deviance.

Or you can use the deviance and df.residual functions to do this:

 pchisq(deviance(spray0)-deviance(spray1),
     df.residual(spray0)-df.residual(spray1),
    lower.tail=FALSE)

--

Many people would use the comparison between full and null-model AIC (or in some cases, perhaps a comparison between a model of interest and the saturated model) to work out whether the model was better than the null in that sense.

--

Am I right to wonder that the Pr(>|z|) for (Intercept) represents the significance of the model?

It doesn't. Indeed, the intercept p-value is usually not of direct interest.

If you're considering a model with a dispersion parameter, I have seen some people argue for doing an F-test instead of an asymptotic chi-square; it corresponds to people using a t-test instead of a z on the individual coefficients. It's not likely to be a reasonable approximation in small samples. I haven't seen a derivation or simulation that would suggest the F is necessarily a suitable approximation (i.e. better than the asymptotic result) in the case of GLMs in general. One might well exist, but I haven't seen it.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
3

Assuming that you model is in the object 'fit' you could use this code to perform a log-liklihood test on your binomial model As you have noted a F-test is not appropriate, but this test will test if your model is predicted better than random. 

LLR = -2 * (fit$null.deviance - fit$deviance)

This is the formula for the Log-likelihood ratio test.

pchisq(LLR, 2, lower.tail = FALSE)

And this will give you the p-value. Althought I'm not 100% confident that is the correct df. I am pretty sure it is the difference in the number of parameters, of which you have 2 in your saturated model and none in the Null model, ergo df = 3 - 1 = 2. But that might be something to follow up on.

SamPassmore
  • 597
  • 3
  • 18
2

As @SamPassmore mentioned, you can use Analysis of Deviance (see for example car::Anova() for something similar-ish) to get something roughly equivalent to the $F$-test, but with a $\chi^2$ distribution. Related to this is the likelihood ratio test (comparison of your model to the null model), but these tests only perform well asymptotically.

Alternatively, you can look at AIC, or related measures like BIC.

Please note though: For this type of model, it's hard to get something like p-value for the same reasons it's hard to define a meaningful $R^2$-value, see for example this "sermon" by Doug Bates.

Livius
  • 2,066
  • 14
  • 17