Is an f-test the correct method for power analysis of my multivariate regression data?

Question

I have been working on solving a problem that requires power analysis to be run in order to determine if the subset of data that we're running analysis on is sufficient before we do the analysis. I think an f-test is the correct test to use for power analysis, but wanted to ask you guys here since programming is my specialty not math, so a lot of this is new to me.

The data that I have is basically a response value that responds to about 15 different practices. Every observation does have a value recorded for the response variable as well as each of the 15 practices. Each practice has a simple "Yes" or "No" recorded for it. I'm trying to determine whether the practices do affect the response variable. The response variable works like a normal number (where going up is good and going down is bad). The response variable cannot be negative (not sure this matters).

I've been trying for a few weeks to determine the best method of running power analysis for this dataset. I already have the number of observations (data is already collected). What I'm trying to do is calculate the power to ensure power is > 0.80 for my data.

My question for you is would an f-test be appropriate for calculating the power in this situation, or would you recommend something else? The specific implementation that I'm using is the f2.test from this R package https://cran.r-project.org/web/packages/pwr/pwr.pdf .

I would like to add that I have another response variable recorded with everything that can be negative. The response variables are never analyzed together. In other words I want to run power analysis and subsequent analysis (assuming power was high enough) for the first response variable + all independant variables and then run a separate analysis using the same data with a different response variable. Would the type of power analysis need to be different for the second response variable that can be negative?

Thank you for your help. Let me know if a data sample would be helpful.

Edit: I should also add that when we're running power analysis we're running it on the subset of data that we currently have for a given user in our software system. More data is entered daily, so the power value will change as more data comes in. What we're trying to do is determine if the subset of data that we're looking at for a given user has adequate power at that point in time to run the report we're running.

Answer 1

For a linear regression of either of your outcomes against your predictors, the F-test evaluates whether any one of your predictors ("practices" in your case) has a regression coefficient significantly different from zero. So in principle you are on the right track.

Without knowing more about the data, however, it's hard to know whether linear regression will be an appropriate model. It doesn't matter whether the outcome is necessarily non-negative. What matters most is that each of your predictors has an additive effect upon the outcome; it's better if the magnitudes of the deviations from the model don't depend on the predicted values; it's best if those deviations have something close to a normal distribution as assumed by the F-test. A search for "linear regression diagnostics" will show ways to evaluate these properties of your models.

Finding the number of cases needed to provide a desired power requires knowing the significance level of the test, the difference that you would like to detect, and an estimate of the variability in what you wish to detect. The ratio of the last 2 factors (desired detectable difference to the variability) is related to the effect size that is expected as an input to functions like pwr.f2.test().

What you seem to want to do is to make some power estimate without looking at the data. That leads to two problems. First, you don't know whether the linear regression will be appropriate for your data. Second, just what will you specify as an effect size, if you don't know anything about the variability of the data or of the results from your modeling of the data?

As you are already collecting data and plan to collect more, take advantage instead of the data that you have (or some appropriately sampled subset) and treat it like a pilot study, which is generally the best way to proceed with power analysis for study design. You will see how well the linear regression model works and whether it might need to be modified in some respects. You will get values of regression and error mean-square values to inform your power analysis, rather than depending on the arbitrary "small," "medium" and "large" effect sizes that the pwr package will give you in the absence of data. You will find whether certain predictors have relationships to outcome that are of particular interest; in that case, your power analysis might be better done with respect to the values of those individual regression coefficients rather than the omnibus F-test.

Also, you probably should evaluate how your two outcome measures are related to each other. Sometimes analyzing multiple outcomes together works better than analyzing them separately. Again, you need to see a sample of your data to evaluate that.