Power analysis for ordinal logistic regression

Question

I am looking for a program (in R or SAS or standalone, if free or low cost) that will do power analysis for ordinal logistic regression.

Answer 1

I prefer to do power analyses beyond the basics by simulation. With precanned packages, I am never quite sure what assumptions are being made.

Simulating for power is quite straight forward (and affordable) using R.

1. decide what you think your data should look like and how you will analyze it
2. write a function or set of expressions that will simulate the data for a given relationship and sample size and do the analysis (a function is preferable in that you can make the sample size and parameters into arguments to make it easier to try different values). The function or code should return the p-value or other test statistic.
3. use the replicate function to run the code from above a bunch of times (I usually start at about 100 times to get a feel for how long it takes and to get the right general area, then up it to 1,000 and sometimes 10,000 or 100,000 for the final values that I will use). The proportion of times that you rejected the null hypothesis is the power.
4. redo the above for another set of conditions.

Here is a simple example with ordinal regression:

library(rms)

tmpfun <- function(n, beta0, beta1, beta2) {
    x <- runif(n, 0, 10)
    eta1 <- beta0 + beta1*x
    eta2 <- eta1 + beta2
    p1 <- exp(eta1)/(1+exp(eta1))
    p2 <- exp(eta2)/(1+exp(eta2))
    tmp <- runif(n)
    y <- (tmp < p1) + (tmp < p2)
    fit <- lrm(y~x)
    fit$stats[5]
}

out <- replicate(1000, tmpfun(100, -1/2, 1/4, 1/4))
mean( out < 0.05 )

Answer 2

I would add one other thing to Snow's answer (and this applies to any power analysis via simulation) - pay attention to whether you are looking for a 1 or 2 tailed test. Popular programs like G*Power default to 1-tailed test, and if you are trying to see if your simulations match them (always a good idea when you are learning how to do this), you will want to check that first.

To make Snow's run a 1-tailed test, I would add a parameter called "tail" to the function inputs, and put something like this in the function itself:

 #two-tail test
  if (tail==2) fit$stats[5]

  #one-tail test
  if (tail==1){
    if (fit$coefficients[5]>0) {
          fit$stats[5]/2
    } else 1

The 1-tailed version basically checks to see that the coefficient is positive, and then cuts the p-value in half.

Answer 3

Besides Snow's excellent example, I believe you can also do a power simulation by resampling from an existing dataset which has your effect. Not quite a bootstrap, since you're not sampling-with-replacement the same n, but the same idea.

So here's an example: I ran a little self-experiment which turned in a positive point-estimate but because it was little, was not nearly statistically-significant in the ordinal logistic regression. With that point-estimate, how big a n would I need? For various possible n, I many times generated a dataset & ran the ordinal logistic regression & saw how small the p-value was:

library(boot)
library(rms)
npt <- read.csv("http://www.gwern.net/docs/nootropics/2013-gwern-noopept.csv")
newNoopeptPower <- function(dt, indices) {
    d <- dt[sample(nrow(dt), n, replace=TRUE), ] # new dataset, possibly larger than the original
    lmodel <- lrm(MP ~ Noopept + Magtein, data = d)
    return(anova(lmodel)[7])
}
alpha <- 0.05
for (n in seq(from = 300, to = 600, by = 30)) {
   bs <- boot(data=npt, statistic=newNoopeptPower, R=10000, parallel="multicore", ncpus=4)
   print(c(n, sum(bs$t<=alpha)/length(bs$t)))
}

With the output (for me):

[1] 300.0000   0.1823
[1] 330.0000   0.1925
[1] 360.0000   0.2083
[1] 390.0000   0.2143
[1] 420.0000   0.2318
[1] 450.0000   0.2462
[1] 480.000   0.258
[1] 510.0000   0.2825
[1] 540.0000   0.2855
[1] 570.0000   0.3184
[1] 600.0000   0.3175

In this case, at n=600 the power was 32%. Not very encouraging.

(If my simulation approach is wrong, please someone tell me. I'm going off a few medical papers discussing power simulation for planning clinical trials, but I'm not at all certain about my precise implementation.)