What technique would you use to understand the statical significance of a test where the target outcome is a continuous measure?
Say there are three tests
Control, A and B each with 50k samples and they generate the following revenues of 1054, 750 and 450.
I've done this with discrete measures like 20 people out of 1000 converted but never continuous measures.
I don't really have enough information (objective, etc.?), but I think the following table covers most possibilities of what you want to achieve (as far as I can derive from your question).
If it helps: the following PowerPoint presentation from Stanford Universit covers this (mainly repeated-measures ANOVA): Analysing Continuous Outcomes
What is normally advised is the mean revenue per user, $\mu$, and the error on that mean, $\sigma_\mu$. Since your samples are large you can likely use the central limit theorem to get the error on the mean as the standard deviation of your samples divided by the square root of the number of users. If you do not trust this you can use bootstapping to get the error on the mean. $\mu$ and $\sigma_\mu$ for each sample gives you enough to build three Gaussian distributions which you can compare using standard methods (eg t-test).
I was unconvinced of this method so I posed this question here Drawing numbered balls from an urn. I phrased it in terms of a classical statistical problem in the hopes of getting a good answer from statisticians. To convert back to business intelligence: ball numbers are price points for products, balls are sales and rounds are users. I am very likely being too rigorous and you are safe to use the method above.
CAUTION: It would be a mistake to calculate $\mu$ or $\sigma_\mu$ using the number of purchasers instead of the number of users who could have purchased. You need to account for the fact that some people did not contribute to the revenue and that this can be different between your splits.
