0

I have an ab test with two factors (color of image and size of image).

Is the correct methodology to do these steps (assuming I want to do a full factorial):

  1. Calculate the f-stat for the control (nothing is done) and all the possible iterations. A positive result here will show that there is some effect.

  2. Follow this up by doing a t-test or z-test using the bonferroni correction to see which of the iterations is signficant.

  3. Of those that are significant, look for the one with the highest effect size and that will be the test "winner".

Does this seem reasonable or is there something I'm missing (I'm not sure I can run t/z test after running the f-test)

The example above was made up, but in actuality it might be something like the page of an ecommerce site that shows all the different products. For example: lee.com/shop/men-clothing-shirts-tees The images sizes could have three values (smaller, control, bigger). Let's say the background of each image could be white, gray or red.

The idea would be to find what maximizes the number of times people click to a pdp page. The intent is to find a) which size and which color are best overall. b) if there is an interaction between size and color that leads to a better result than just the best size and best color.

ramen_noodles
  • 71
  • 4
  • Can you tell us more about the data and the hypothesis you want to investigate? What is the outcome you want to study? How is size measured? Is your intent to find the combo of size and image which maximizes some quantity (e.g. click through?) – Demetri Pananos Jun 23 '21 at 00:48
  • Sure! The example above was made up, but in actuality it might be something like the page of an ecommerce site that shows all the different products. For example: https://www.lee.com/shop/men-clothing-shirts-tees The images sizes could have three values (smaller, control, bigger). Let's say the background of each image could be white, gray or red. – ramen_noodles Jun 23 '21 at 03:23
  • The idea would be to find what maximizes the number of times people click to a pdp page. The intent is to find a) which size and which color are best overall. b) if there is an interaction between size and color that leads to a better result than just the best size and best color. – ramen_noodles Jun 23 '21 at 03:27

1 Answers1

1

I wrote a fairly thorough example on how to Bayesian decision making in AB tests here. A similar approach would work for you.

The loss function would be similar to the example I linked to, except you would have more groups defined by the Cartesian product of size and color. You could very easily estimate a logistic regression for the probability of clicking on the image given the exposures.

Demetri Pananos
  • 24,380
  • 1
  • 36
  • 94