2

I am involved in a project that is attempting to measure the impacts of person-level, county-level and medical service area-level predictors on individual probability of returning to a hospital within a year window. I see this as a 3 level random effect logistic regression model. There are lots of predictors, probably 10 at the person level, 10 at the county and probably 5 at the hospital level. There are nearly 300,000 individuals nested withing 3000 counties and 154 hospitals. I would think that I should have good statistical power to detect differences in my outcome. My question, is: since the sample is already determined and I'm not collecting data (this is all secondary data), is a power analysis even appropriate? If it is appropriate, but my PI can't specify the effect size they are looking for, then what?

I would appreciate any input the list may have on this problem. Best, Corey

Corey Sparks
  • 824
  • 6
  • 8
  • You're right on all three points: 1) it doesn't seem like there's much of a point to the power analysis if there's nothing you can do about it. What is the logic behind this? What motivated this inquiry? 2) This sample size seems plenty large. There would only be power issues if your effect sizes were _very_ small. 3) You won't be able to do any calculations without an effect size. – Macro May 26 '12 at 00:25
  • I disagree with @Macro, there is certainly utility in doing an apriori power analysis in this circumstance. Remember, the standard error of any estimate is not only dependent on the effect size, but it will also be dependent on the extent any independent variable is correlated with the other independent variables (a problem in observational studies, and especially with ecological variables at the county level). In addition, logistic regression models tend to take higher sample sizes than OLS models, suggesting cross-level interactions could certainly be under-powered. – Andy W May 26 '12 at 00:40
  • @AndyW, I wasn't saying there was no point, only that the point wasn't apparent to me (which is why I asked what motivated this inquiry). You've made some useful comments, but I'm still not quite sure what the utility would be. I've seen people try to use power analysis after the fact to rationalize why their results weren't statistically significant (I believe this is called post-hoc power analysis) but I've also read a paper (I can't place it right now) whose message what basically that post-hoc power analysis has some very bad properties and essentially should never be done. – Macro May 26 '12 at 00:42
  • In general I think it would be good if people conducted power analysis in observational studies. It makes people focus on effect sizes as opposed to solely statistical significance anyway. Although I can't say I'm familiar with power analysis in multi-level models, I would suspect @Macro is a much better person to ask! – Andy W May 26 '12 at 00:43
  • I'm in favor of doing a power analysis, but since the data are SOOOO unbalanced with some clusters having lots of observations, and some having none, the standard equations for power fall apart from what I've seen. Still having this many level 2 units and individuals in the anlaysis should make testing for even a small effect size (say a 5% difference) non-problematic, no? – Corey Sparks May 26 '12 at 00:44
  • How do you do power analysis with observational studies when samples are not drawn at random? – Michael R. Chernick Jun 29 '12 at 02:24
  • I really appreciate everyone's thoughts on this, I think that a simulation approach may be the only way forward, given the complexity of the data. On the non-random sampling issue, these data are completely non-random, they are everyone in the population (uses of a large federal service), i.e. we have everyone who ever went into a hospital and their records on re-admission, although I don't know the %'s for readmission now. The "sample" is completely representative of the users of this federal service. – Corey Sparks May 27 '12 at 18:56

2 Answers2

2

Power analysis depends on the distribution of the test statistic under alternatives when the critical value has been specified to control the type one error. All this assumes some form of random sampling. In an observational study where is the random sampling for a target population? I first question whether hypothesis testing makes sense in this situation and it surprises me that no one has mentioned this. If we were in a situation where the assumptions about sampling at random are reasonable you need some idea of variability. Sample size problems almost always involve guess work on variability and so several scenarios should be looked at. Even if the investigator has trouble determining effect size or clinically important difference the statistician canlook at how the results change over a resonable range of values. But in my experience a clinically important difference that is worth detecting may be the easiest thing to determine.

Michael R. Chernick
  • 39,640
  • 28
  • 74
  • 143
  • Nothing about the OP's question made me think that this was non-random sampling. The term 'observational study' usually does describe a study of treatment effect when there is non-random treatment assignment but in this case there's no "treatment" variable that I can see, so maybe this was a slight misuse of terminology in the thread title. These data could conceivably arise from a random sample from the joint distribution of the outcome (hospital re-admittance) and the covariates. The large sample size indicates it may also be a census (possibly of all people who have been to the hospital), – Macro May 26 '12 at 02:11
  • (cont) in which case it's certainly not a random sample from the population at large but it unbiasedly characterizes the population of "people who have been to the hospital". – Macro May 26 '12 at 02:11
  • @macro Okay if you agree with me about the lack of randomness I don't think it matters whether or not the sample is representative the sample/power question is no longer relevant. The distributional properties needed for the calculations don't hold. – Michael R. Chernick May 26 '12 at 03:41
  • yes I agree, it just wasn't clear to me that non-random sampling was an issue here. – Macro May 26 '12 at 03:49
  • @macro I don't think the OP recognized that it was an issue but that was the point of my answer to explain the issue. I also discussed the sample size issue had there been a valid sample size question. – Michael R. Chernick May 26 '12 at 04:03
2

Power analysis is typically undertaken a priori to conducting an experiment, in which the researcher, given a pre-specified effect size, identifies a sample size that sufficiently controls type II error. So two things are immediately different in observational designs, 1) you can't change the sample size , 2) the effects of interest are likely to have non-zero correlations with one another (whereas in experiments they should be zero by design.)

So even with your very large individual sample, I would not take for granted the power of tests for the direct effects of the higher levels, especially given ecological correlations between aggregate units tend to be very high. Also I don't know the distribution of the outcome, and if re-admittances are rare one needs (much) larger sample sizes than in typical experiments with continuous outcomes. Further, cross-level interactions may potentially have little power as well.

As far as motivation for conducting power analysis in observational settings (to be clear power analysis before the study is undertaken), one may wish before hand to check whether the study is sufficiently powered, or whether more data should be collected before the study is undertaken. Also one may wish to determine if certain exploratory analysis (such as identifying interaction effects) are worth undertaking.

I admit, this conflicts with the notion that one should have an identified model before hand (e.g. if you need to include an interaction effect for other effects to be properly identified it needs to be included regardless of its power). But, my response would be such interaction effects are very rarely specified at the onset as necessary components in the model, and are typically added in a model building strategy. Such as that advocated by Raudenbush and Bryk starting with empty models and iteratively adding higher level variance components and interactions (as esimation problems frequently occur with complicated models, especially if a higher level variance component is zero).

For estimating power, it is unlikely any standard equation will capture the complexity of the situation. And so one frequently turns to simulation. One then generates a fake set of data that approximates what the true data will look like, then generates an outcome distributed according to pre-specified effect sizes, then estimates the model and calculates the number of true and false negatives. Even if your client can't put a number on a specific effect size, you can graph the expected power of a test given a range of effect sizes.

As Michael already stated, it will likely be a chore to specify the variability in the data (i.e. the variances and the co-variances of the independent variables). It will likely take custom coding to simulate the nested structure of the data while at least approximately constraining the variances and covariances of the independent variables. Quickly reviewing the multi-level modelling books on my shelf suggests most power analysis were considered in the context of multi-level experiments, so you may be foraying into new territory.

I will refer to some other literature in hopes others have provided more directly actionable advice.

  • Andrew Gelman and Jennfier Hill in their book, Data Analysis Using Regression and Multilevel/Hierarchical Models has a chapter on power analysis, and gives a brief example in R code. They also give a few references at the end of the chapter on the topic.
  • Tom Snijders webpage has a windows program that calculates power for fixed effects in two-level models, PINT. He also appears to have publications of potential interest. I found this via the chapter in Joop Hox's multi-level book on power analysis.
  • The G*Power 3 software tool doesn't appear to have any modules for multi-level analysis, but they do have a wealth of references to other literature on power analysis.

As a note on post hoc power analysis, I agree it is typically considered inappropriate (see this response by Freya Harrison on our site). Although I admit I recently conducted one in a publication at the request of a reviewer. Although to be fair, in retrospect I wish I had conducted a-priori power analysis, as the question of power in and of itself is interesting, even though it was an observational, quasi-experimental study.

Community
  • 1
Andy W
  • 15,245
  • 8
  • 69
  • 191