Finding weight/value of each person on a team

Question

If I have a team, with between $n_1$ and $n_2$ people per team, with results of team's head to head matchups, how would I be able to estimate each person's value?

Example data (I drew this up quickly, the actual one is many lines longer, with more players per team):

| Home   | Away   | Home Score | Away Score | Home Members  | Away Members     |
| ------ | ------ | ---------- | ---------- | ------------- | ---------------- |
| Team 1 | Team 2 | 4          | 2          | John, Joe     | Mary, James      |
| Team 1 | Team 3 | 3          | 2          | Joe, Patricia | Mark, Paul, Ross |
| Team 2 | Team 5 | 1          | 2          | Mary, Robert  | Steve, Lisa      |
| Team 4 | Team 3 | 4          | 3          | Ashley        | Mark, Matt       |
| Team 4 | Team 5 | 2          | 3          | Donna, Emily  | Anne, Lisa       |

For the example data, I want to be able to say something like John is the best with a individual score of $s_1$ and Mary is right behind him with a individual score of $s_2$ and James is the worst with a score of $s_n$. Thus, any team replacing any person with John will become better and and team replacing any person with James will become worse.

I am assuming that the score is a combination of everyone's individual scores and teamwork will not come into play (or more realistically, teamwork is built into each player's individual score). I do not, however, want to say that because James scores all the points on every team he is on, that he is worth the most. It may be that James is simply the person that scores the points by setup of internal team communication, but if he is not good at this, his score should be lower (ex. If I run this against a football league, the strikers shouldn't always be at the top, with the defenders, presumably not having scored many goals, at the bottom).

I would also want to be able to weight matches (ie. match 1 is a regular match that doesn't really matter, but match 5 is an elimination match that really matters, so if a player consistently under-preforms the elimination matches, they have a lower score and visa-versa).

I care about both predicting the outcome of any match I want and placing a score on an individual player, but if I had to choose, I would place getting a more accurate individual player score at a higher importance.

The weights should also be tied to winning. However, it should be fine to tie it to the score (any team that scores more is generally better and any team that scores less is generally worse). However, the one problem with this is given a game where both teams score a million points and tie, both values would go up like crazy. However, ultimately they tied. So while you could argue that it shows potential to win later, I'm not altogether convinced that the scores should go up that much for a tie, because it also should go down for having a million points scored against them (by the football example, even the striker should have his scored, as they are not contributing to the defense or are not good at it - we also make no difference between strikers and defenders in the data, so it would be impossible to differentiate them).

I was thinking of something like an ELO system, where each player has their own ELO, but I'm not sure if this is the best way to do it. The problem with ELO is even if I give each player their own ELO ($e_1$), then calculate the team elo ($\sum e_n$) in order to determine the prediction and ELO multiplier, the player's will never reach their true level. Let's say a new superstar (so he has low/base ELO) joins a good team (so the average ELO is much higher than him, but his true ELO should be much higher than everyone else's — $\hat e_s < e_\mu$ but $e_s > e_\mu$). If the team starts winning and the ELO gains get evenly divided, the superstar never gains ELO compared to his teammates, thus being consistently undervalued by this system. The Bradley-Terry model or TrueSkill may be better solutions, although I'm unfamiliar with the shortcomings of both of these; Bradley-Terry has 2 players per team and TrueSkill relies on constant team switches, so I'm unclear how well they extend to this.

This seems fairly trivial, but I can't seem to wrap my head around how to do it.

Answer 1

I don't think this question is at all trivial. To answer, I'll first discuss a couple things to consider and then offer a partial solution based on your post that may help. Also, do check out @Ryan Volpi's linked posts, they are probably quite relevant to your question.

Couple considerations

1. What does 'best' mean in your context, and more generally how do you evaluate performance? Is it simply based on highest contribution to your team's score? Or is it based on whether your team wins, and the score does not matter except as it factors into winning? Or is your performance a combination of your team's score relative to the other team's score? From your writing, it seems like you care about score for your own team, which I'll revisit soon.

2. How are you going to think about modeling this as a statistical process? First, do you care about estimating performance in the sample of games you observe, or are you interested more generally in determining the true value of everyone. This difference matters: in the simplest case, if I just observe you once, and you do really well, it means you did really well in that one game, but maybe you just got really lucky that one game and you're actually not a good performer. Second, how do you think about teamwork? Can some people work better together? Maybe John and Joe are a power team, but John hates playing with Robert. And how do the other team's player affect your team's score? How are you going to model that, and how will it further affect the error terms.. maybe they are now correlated depending on the players.

A partial (potential) solution

From your writing, it seems like you care about best in terms of the highest score, and seem to not care about relative score or winning, and it also seems like you don't want to factor the effect someone has on maybe lowering the other team's score. In that case, you don't really care at all about the games, and you can actually just consider a vector of scores and team members (so each game is really two observations: home score and the 2 home players, and away score and the 2 away players... since we assume they dont affect each others' scores, we can just consider them independently, somewhat like every team is just doing their own thing. This kind of approach to modeling individual performances is a simple version of modeling that is relatively common in some empirical research, and can be extended to model some of these factors I assumed away, but I will present this basic model to introduce concepts. As mentioned in the comments, it may hold in games like darts or bowling where teammates and opponents have little to no effect on each other.

Since we care about individual score, let's assume we are interested in their mean performance. Every player $i$ has some mean score performance $s_i$ that we want to know. Let's suppose that for every team/game $j$ (recall that games are really two team/game observations from our assumptions), their performance may be different from this performance due to random noise $\epsilon_{ij}$ where $\epsilon_{ij} \sim N(0,\sigma_i^2)$ (in your example, scores are discrete, so you may want to model it somewhat differently, but the same ideas will hold). Then, since we assumed away teamwork and other team's member's effects, $i$'s contribution to game $j$ is $s_i + \epsilon_{ij}$, and every $j$ team/game's observed score $s_j$ is $$s_j = \sum_{i\in K_j} (s_i + \epsilon_{ij}) = \sum_{i\in K_j} s_i + \sum_{i\in K_j} \epsilon_{ij}$$

where $K_j$ is the set of $i$ individuals in team/game $j$. Since each $\epsilon_{ij}$ is independent of each other and normal, we have that $$\sum_{i\in K_j} \epsilon_{ij} \equiv \epsilon_j \sim N\bigg(0,\sum_{i\in K_j} \sigma_{i}^2\bigg)$$

Additionally, instead of summing only over those in $K_j$ for team/game $j$j, we can define an indicator variable $W_{ij} = \mathbb{1}[i \in K_j]$ that denotes if the player $i$ was in that team/game $j$. Then we can re-write the previous equation as

$$s_j = \sum_{i\in K_j} (s_i + \epsilon_{ij}) = \sum_{i\in K_j} s_i + \sum_{i\in K_j} \epsilon_{ij} = \sum_i s_iW_{ij} + \epsilon_j$$

and from here, it's clear that $s_i$ for each individual corresponds to the beta coefficients of a linear regression of $s_j$ on the full vector of indicators ${W_{ij}}_{i}, so running the regression

$$S_j = \sum_i \beta_i W_{ij}$$

on all your data will identify $\beta_i = s_i$ for each $i$. The estimated performance rankings can then be determined by ordering each $\hat{\beta}_i$, and you can also test for significant differences between them and other tests.

Here is a simple example using R

n = 100000

x1 = 5
x2 = 2
x3 = 8

x1_err = .1
x2_err = .5
x3_err = 1

dt = data.table("x1" = rep(x1,n), "x2" = rep(x2,n), "x3" = rep(x3,n),
                "x1_err" = rep(x1,n), "x2_err" = rep(x2,n), "x3_err" = rep(x3,n))
dt[,n := 1:.N]

dt[, ':=' (x1_ind = ifelse(n < 60000,1,0),
           x2_ind = ifelse(n< 30000 | n > 60000,1,0),
           x3_ind = ifelse(n>= 30000,1,0))]

dt[, score := x1*x1_ind + x2*x2_ind + x3*x3_ind + 
     rnorm(n,mean = 0, sd = x1_err*x1_ind + x2_err*x2_ind + x3_err*x3_ind)]

#Note that I am omitted the y intercept since I know I fix it at 0
lm(score ~ -1 + x1_ind + x2_ind + x3_ind,data = dt)

Call:
lm(formula = score ~ -1 + x1_ind + x2_ind + x3_ind, data = dt)

Coefficients:
x1_ind  x2_ind  x3_ind  
 5.054   1.929   7.964 

As you can see, in each game, I had two individuals perform together, and the coefficient estimates for each indicator is very close to their true mean performance!

You can extend this toy model in many ways, but it's up to you how you want to explore your data. The assumption that individual performances follow normal means is relatively common in a lot of research (for example, value added models in the context of teachers' effects on student test scores, use this same idea), and you can introduce correlation between players and other factors such as the effect of the other team. One way could be to think that individuals are defined by their own team performance and also have a negative effect on the other team's performance, and so you want to jointly estimate those. And what does performance mean here? Well again, up to you, but maybe some weighted combination of offense and defense. But that will introduce other complications, since not everyone may benefits from the 'best' player: if a player is really good at offense but awful at defense, maybe you don't want him with someone who is similarly skilled.