How to determine power using standard error and effect size?

Question

I was reading this article published from airbnb around gaurdrail metrics. In it, they discuss this power guardrail they derived but it's very unclear how they went about it.

How are they determining Power given the effect (0.8)*t using no other information?

Answer 1

Taking your question in the body of the post very literally, it is indeed impossible to determine the power given the effect $0.8 * t$ without using other information.

I believe the blog post has made the following assumptions as well:

• The sample size in Airbnb is sufficiently large for experimenters to rely on CLT and run Student's/Welch's t-tests with a very large degrees of freedom, i.e. having the test statistic effectively normally distributed (common in digital experiments);
• Given they are discussing guardrail metrics, they are likely to be looking for potential harm only and hence using a one-tailed test, i.e. $H_1: \mu_B - \mu_A < 0$ (common in digital experiments);
• The null hypothesis is $H_0: \mu_B - \mu_A = 0$ (common in digital experiments - one can also use the composite variant, see e.g. this question);
• The significance level (or size), given we are using t-tests, is $\alpha=10\%$. (stated in blog post); and
• The standard error "if an experiment just meets the power guardrail" is $0.8 * t$. (stated in blog post).

Under these assumptions, The power of a one-tailed (less than) t-test with a very large degrees of freedom for a specific effect size $\theta$ is:

$$B(\theta) \approx \Phi\left(z_{\alpha} - \frac{\theta}{\textrm{std. error}}\right), $$ where $\Phi(\cdot)$ is the standard normal CDF, and $z_{1-\alpha}$ is the $\alpha$ quantile of a standard normal. Note the linked wiki page gives the power calculations for a "greater than" one-tailed test, and we have to flip the signs here as we have a "less than" one-tailed test.

Given we know $z_{\alpha} \approx -1.28 $ for $\alpha=0.1$, this means:

• If $\theta = -t$, $$\begin{align} B(\theta) & \approx \Phi\left( -1.28 - \frac{-t} {0.8*t}\right) \\ & = \Phi(-1.28+1.25) \\ & \approx \Phi(0) = 0.5 \end{align}$$

• If $\theta = -2t$, $$\begin{align} B(\theta) & \approx \Phi\left( -1.28 - \frac{-2t} {0.8*t}\right) \\ & = \Phi(-1.28+2.5) \\ & \approx \Phi(1.28) \approx 0.9 \end{align}$$

I can only assume the multiplier $0.8$ is chosen as $1/z_{1-\alpha} = 0.78030...$ may be harder to remember by heart. Having that said, all of the above are meant as rules of thumb and thus we should expect approximations here and there.