23

I'm looking for an advanced linear regression case study illustrating the steps required to model complex, multiple non-linear relationships using GLM or OLS. It is surprisingly difficult to find resources going beyond basic school examples: most of the books I've read won't go further than a log transformation of the response coupled with a BoxCox of one predictor, or a natural spline in the best case. Also all examples I've seen so far approach each data transformation problem in a separate model, often in a single predictor model.

I know what a BoxCox or YeoJohnson transformation is. What I'm looking for is a detailed, real-life case study where the response/relationship are not clear cut. For example, the response is not strictly positive (so you can't use log or BoxCox), the predictors have non-linear relationships between themselves and against the response, and the maximum likelihood data transformations don't seem to imply a standard 0.33 or 0.5 exponent. Also the residual variance is found to be non-constant (it never is), so the response has to be transformed as well and choices will have to be made between a non-standard GLM family regression or a response transformation. The researcher will likely make choices to avoid overfitting the data.

EDIT

So far I gathered the following resources:

  • Regression Modeling Strategies, F. Harrell
  • Applied Econometric Time Series, W. Enders
  • Dynamic linear models with R, G. Petris
  • Applied Regression Analysis, D. Kleinbaum
  • An Introduction To Statistical Learning, G. James/D. Witten

I only read the last (ISLR) and it is a very good text (a 5 five stars on my watch), although more oriented towards ML than advanced regression modeling.

There is also this good post on CV that presents a challenging regression case.

Community
  • 1
Robert Kubrick
  • 4,078
  • 8
  • 38
  • 55
  • 8
    I believe Frank Harrells book (http://www.amazon.com/Regression-Modeling-Strategies-Applications-Statistics/dp/0387952322) might be helpful. – Adam Robinsson Nov 02 '14 at 23:26
  • @AdamRobinsson I see the TOC is touching several relevant subjects (multivariate models, splines, multicollinearity), but are those methodologies illustrated *together* in a real-life example or each topic is explained separately? Because usually in real-life examples all the problems come at you together and it's never obvious how to manage them well. – Robert Kubrick Nov 02 '14 at 23:36
  • 1
    I haven't read the whole book yet, but the first 150 pages have been absolutely great (I'm not a statician, just an enthusiast). Example are extensive and elaborated upon. The book is accompanied by the RMS (regression modeling strategies) package to R. I've also looked at David Kleinbaums competing book (forgotten the title unfortunately) but it contained much less about strategies and examples (and was twice as expensive). – Adam Robinsson Nov 03 '14 at 15:29
  • Yes, the case studies are realistically complex & followed through from start to finish. – Scortchi - Reinstate Monica Nov 08 '14 at 21:11
  • @Scortchi Good to know. Sounds like that's the only option! – Robert Kubrick Nov 08 '14 at 21:12
  • @RobertKubrick: I don't recall any multivariate regression examples in *RMS*, but is that (i.e. multiple *responses*) what you mean anyway? – Scortchi - Reinstate Monica Nov 08 '14 at 21:15
  • @Scortchi No, multivariate regression, single response. – Robert Kubrick Nov 08 '14 at 21:19
  • 3
    @RobertKubrick: "Multivariate regression" means with more than one response (see the wiki for the tag you added, or [here](http://stats.stackexchange.com/questions/2358/explain-the-difference-between-multiple-regression-and-multivariate-regression)). "Multiple regression" means with more than one predictor. – Scortchi - Reinstate Monica Nov 09 '14 at 10:48
  • 3
    You might want to check out Applied Econometric Time Series by Enders. The new version covers non-linear models towards the end of the book. Nearly all the data is publicly available on the St. Louis Fed website (accessible via quantmod in R) so you can follow along real life examples. Dynamic Linear Models with R also has a few examples with real data that are pretty decent. – Eric Brady Nov 10 '14 at 23:09
  • Try Applied Regression Analysis by David Kleinbaum, which I think is what Adam Robinsson was alluding to. – Jayce13 Jul 15 '15 at 02:23

6 Answers6

10

Regression Modeling Strategies and ISLR, which have already been mentioned by others, are two very good suggestions. I have a few others that you might want to consider.

Applied Predictive Modeling by Kuhn and Johnson contains a number of good case studies and is pretty hands-on.

Practical Data Science with R treats practical (regression) modeling in the context of its applications $-$ mostly as predictive models in a business situation.

Generalized Additive Models: An Introduction with R by Simon Wood is a good treatment of generalized additive models and how you fit them using his mgcv package for R. It does contain some nontrivial practical examples. The use of GAM models is an alternative to figuring out the "correct" transformation as this is done in a data adaptive way via a spline expansion and penalized maximum-likelihood estimation. However, there are still other choices that need to be made, e.g. the choice of link function.

The mboost package for R also fits GAM models but using a different approach via boosting. I recommend the tutorial for the package (one of the Vignettes).

I will also mention Empirical Model Discovery and Theory Evaluation by Hendry and Doornik, though I have not yet read this book myself. It had been recommended to me.

StasK
  • 29,235
  • 2
  • 80
  • 165
NRH
  • 16,580
  • 56
  • 68
5

One of the best course material that you can find on advanced, multiple, complex (including nonlinear) regression is based on the book Regression Modeling Strategies by Frank E. Harrell Jr.

The book is being discussed in the comments but not this material, which itself is a great resource.

Sean Easter
  • 8,359
  • 2
  • 29
  • 58
rnso
  • 8,893
  • 14
  • 50
  • 94
2

I would recommend the book Mostly Harmless Econometrics by by Joshua D. Angrist and Jörn-Steffen Pischke

This is the most real-world, salt to the earth, text I own and it is super cheap, around $26.00 new. The book is written for the graduate statistician/economist so it is plenty advanced.

Now this book is not exactly what your asking for in the sense that it doesn't focus on "complex, multiple non-linear relationships" as much as core fundamentals like endoegeneity, interpretation, and clever regression design.

But I am offering this book to try to make a point. Which is, when it comes to real world application of regression analysis, the most challenging issues generally do not have to do with the fact that our models aren't complex enough...believe me we are plenty good at drumming-up very complex models! Rather the biggest issues are things like

  1. Endogeneity
  2. not having all the data we need
  3. Having to much data...and it's all a mess!
  4. To many people cannot interpret their own models correctly (a problem that becomes more prevalent as we make models more complex)

A firm understanding of GMM, non-linear filters and non-parametric regression pretty much covers all the topics you have listed and can be learned as you go along. However, with real world data, these frameworks have the potential to be needlessly complex, often harmfully so.

All to often it's the ability to be cleverly simple rather than completely generalized and highly sophisticated, that benefits you most with real-world analysis. This book will help you with the former.

Zachary Blumenfeld
  • 3,826
  • 1
  • 14
  • 21
1

You can refer Introduction to Statistical Learning with R (ISLR), the book talks about splines and polynomial regression in detail with cases.

Vikram Venkat
  • 286
  • 1
  • 7
1

I'm not sure what is the objective of your question. I can recommend Greene's Econometric Analysis text. It has a ton of references to papers inside. Pretty much each example in the book references a published paper.

To give you a flavor, look at Example 7.6 "Interaction Effects in a Loglinear Model for Income" on p.195. It refers to a paper and the data set: Regina T. Riphahn, Achim Wambach, and Andreas Million, "Incentive Effects in the Demand for Health Care: A Bivariate Panel Count Data Estimation", Journal of Applied Econometrics, Vol. 18, No. 4, 2003, pp. 387-405.

The example is about usage of the loglinear models and the interaction effects. You can read the whole paper, or this textbooks description of it. This is not a made up use case. It's a real published research. This is how people actually use the statistical methods in economics research.

As I wrote the book is pestered with use cases like this on the usage of advanced statistical methods.

Aksakal
  • 55,939
  • 5
  • 90
  • 176
0

Have you looked into some of the Financial Time Series Analysis courses/books that Ruey Tsay (UChicago) writes?

http://faculty.chicagobooth.edu/ruey.tsay/teaching/

Ruey Tsays classes and the textbook provide multiple real world examples in Finance of complex regressions of the type that are created for use in financial markets. Chapter 1 begins with multifactor regression models and expands to Seasonal Autoregressive Time series models by chapter 5 or 6.

zhqiat
  • 103
  • 4
  • 2
    Yes I did and don't like it at all. It's very wide in breadth (anything from volatility models to high frequency to ARIMA...), touch each subject lightly (how couldn't with so many topics at hand) and the R studies and challenges are reduced to a minimum. It's a rehash of academic papers and already stated theory/models you can find somewhere else. This is precisely what I mean by school cases that never deal with the complexity of multiple challenges in a real-world, advanced problem. – Robert Kubrick Jul 17 '15 at 15:33