How many user interactions do I need to perform search relevance experiment?

Question

I know similar questions have been asked many times here. But I still don't get simple formula or idea how estimate this.

I have site with traffic and I want to split portion of it to test new search relevance algorithms, i.e., result sorting. I will measure abandonment rate (searches without click/total amount of searches). I want to get this statistics with $\pm2\%$ accuracy with $95\%$ confidence.

Is there simple heuristic (something practical) I can apply to get required number of experiments ($10^3,~10^6,~\ldots$)?

Answer 1

Your question relates to the idea of statistical power. There are a lot of threads on CV that provide information to help you think about this, search under: power and power-analysis. In particular, you may want to read through this answer of mine: How to report general precision in estimating correlations within a context of justifying sample size, which discusses the idea of Accuracy in Parameter Estimation--a framework similar to power, but specific to the question of how much data you need to get a confidence interval down to a certain size.