0

I have goal of measuring CTRs of several titles of an article on a website using Bayesian approach.

In a simple setup, what one will do is to select Beta Prior (for example Uniform distribution) and run series of experiments until he gets credible intervals narrow enough to select proper title.

Let's assume now that I run the experiment on several websites simultaneously and want to understand statistics regarding each title-website pair - what should I do in this case?

Again, the simplest way is just to treat all pairs as independent and reuse approach from first phase.

What is however the best way to use the shared information, like number of trials and successes of particular title on all websites, while doing the experiment? It seems that this can lead to the credible results much faster (i.e. maybe one site is just not converting at all as there only bots there, or one title is performing much better on all websites)

What is the best way to update posterior distributions (or to reselect priors) using shared information between the entities?

Data Example

site_id title_id impressions clicks
123     t1       10051       150
123     t2       560         1
.
.
.
789     t1       101         15
789     t2       1050        2
.
Tim
  • 108,699
  • 20
  • 212
  • 390
Sergio Kozlov
  • 103
  • 5
  • Sounds like you are after a hierarchical Bayesian model. Could you tell us more about your data (e.g. give a small example), so that we can understand it better? – Tim Nov 07 '17 at 21:52
  • I've added data example by @tim 's request – Sergio Kozlov Nov 08 '17 at 09:08

1 Answers1

2

As described in Bayesian Data Analysis book by Gelman et al. and related blog entry on the eight schools example, treating everything as independent may not be the best idea. More reasonable, and accurate, apprach would consider the fact that you are comparing the same titles on different sites, so they should have something in common. The usual Bayesian way to go would be to define some kind of hierarchical model, for example something like

$$ \begin{align} \alpha \sim \mathcal{N}(0, \sigma^2_\alpha) \qquad \beta_i &\sim \mathcal{N}(\mu_\beta, \sigma^2_\beta) \qquad \gamma_j \sim \mathcal{N}(\mu_\gamma, \sigma^2_\gamma) \\ p_{ij} &= \mathrm{logit}^{-1}(\alpha + \beta_i + \gamma_j) \\ \mathsf{clicks}_{~ij} &\sim \mathcal{Bin}(p_{ij},\; n_{ij}) \end{align} $$

where $n_{ij}$ are the number of impressions (if I understand your data correctly), with some additional hyperpriors on $\sigma^2_\alpha, \sigma^2_\beta, \sigma^2_\gamma, \mu_\alpha, \mu_\beta$. As you can notice, this is a variation of Bayesian logistic regression model with "random effects" for each site $\beta_i$ and for each title $\gamma_j$. This lets you model the impact of individual sites and titles on views. Moreover, you can make use of all the data you have. Additionally, you can include some other explanatory variables in your model as well (in the regression part), what makes it very flexible.

Unfortunatelly, to estimate such model you won't have a simple closed-form solution like in the case of beta-binomial model and you would need to use MCMC, but this can be easily and quite efficiently done in software like Stan (it is not that hard to learn and has great documentation).

Tim
  • 108,699
  • 20
  • 212
  • 390