Average value paradox - What is this called?

Question

I have a dataset. Say $10$ observations and $3$ variables:

obs  A   B   C
1    0   0   1
2    0   1   0
3    1   0   1
4    1   1   0
5    1   0   1
6    1   0   0
7    1   1   0
8    0   0   1
9    0   1   1
10   0   1   1

Say that is $10$ customers who have bought (1) or not (0) in each category A, B, C. There are $16$ ones there so these $10$ customers buy into $1.6$ product categories on average.

Note customers can buy into more than one of A, B and C.

If I look at only those who buy A, there are $5$ customers who have bought into $9$ product categories, so that's $1.8$ on average.

B is $9/5$ again, or $1.8$.

C is $10/6 = 1.67.$

All of them above $1.6.$

which seems strange. I understand it but need to explain this to marketing next week and so need help!

What is this thing called?

I know it's not Simpson's paradox. To me it feels similar in logic to the Monty Hall problem and conditional probability.

Answer 1

The average of every subcategory can be above the overall average if the subcategories overlap on the larger customers.

Simple example to gain intuition:

• Let $A$ be an indicator whether an individual purchased an item in category A.
• Let $B$ be an indicator whether an individual purchased an item in category B.
• Let $X = A + B$ be the number of items purchased.

\begin{array}{ccc} \text{Person} & A & B \\ i & 1 & 0 \\ ii & 0 & 1 \\ iii & 1 & 1 \end{array}

The set of individuals where $A$ is true overlaps the set of individuals where $B$ is true. They are NOT disjoint sets.

Then $\operatorname{E}[X] \approx 1.33$ while $\operatorname{E}[X \mid A] = 1.5$ and $\operatorname{E}[X \mid B] = 1.5$

The statement that would be true is:

$$ P(A)\operatorname{E}[X\mid A] + P(B)\operatorname{E}[X\mid B] - P(AB)\operatorname{E}[X\mid AB] = \operatorname{E}[X]$$

$$ \frac{2}{3}1.5 + \frac{2}{3}1.5 - \frac{1}{3}2 = 1.3333$$

You can't simply compute $P(A)\operatorname{E}[X\mid A] + P(B)\operatorname{E}[X\mid B] $ because sets $A$ and $B$ overlap, the expression double counts the person who purchases both item $A$ and $B$!

Name for illusion/paradox?

I'd argue it's related to the majority illusion paradox in social networks.

You may have a single dude who networks/friends everyone. That person may be one out of a million overall, but he'll be one of each persons's $k$ friends.

Similarly, you have 1 out of 3 here purchasing both categories A and B. But within either category A or B, 1 out of the 2 purchasers is the super purchaser.

Extreme case:

Let's create $n$ sets of lotto tickets. Every set $S_i$ includes two tickets: a losing ticket $i$ and the jackpot winning ticket.

The average winnings in every set $S_i$ is then $\frac{J}{2}$ where $J$ is the jackpot. The average of each category is WAY above the average winnings per ticket overall $\frac{J}{n+1}$.

It's the same conceptual dynamic as the sales case. Every set $S_i$ includes the jackpot ticket in the same way that every category A, B, or C includes the heavy purchasers.

My bottom line point would be that intuition based upon disjoint sets, a full partition of the sample space does not carry over to a series of overlapping sets. If you condition on overlapping categories, every category can be above average.

If you partition the sample space and condition on disjoint sets, then categories have to average out to the overall mean, but that's not true for overlapping sets.

Answer 2

I would call this the family size paradox or something similar

Suppose, for a simple example, everybody had one partner and a Poisson-distributed number of children with parameter $2$:

• The average number of children per person would be $2$
• The average number of children per person with children would be $\frac{2}{1-e^{-2}} \approx 2.313$
• The average sibling group size for each individual (counting their brothers and sisters and themselves) would be $3$

Real demographic and survey numbers produce different numbers but similar patterns

The apparent paradox is that the average size of individuals' sibling groups is larger than the average number of children per family; with stable population dynamics, people tend to have fewer children on average than their parents did

The explanation is whether the average is being taken over parents and families or over siblings: there are different weightings being applied to large families. In your example there is a difference between weighting by individuals or by purchases; your conditional averages are pushed up by fact you condition on a particular purchase being made.

Answer 3

The other answers are overthinking what's going on. Suppose there is one product and two customers. One bought the product (once) and one didn't. The average number of products bought is 0.5, but if you look only at the customer who bought the product, the average rises to 1.

This doesn't seem like a paradox or counterintuitive to me; conditioning on buying a product will generally raise the average number of products bought.

Answer 4

Is this not merely the "average of averages" confusion (e.g. previous stackexchange question) in disguise? Your temptation appears to be that the subsample averages should end up averaging to the population average, but this will rarely happen.

In the classical "average of averages", someone finds the average of N mutually exclusive subsets, and then is flabbergasted that these values do not average to the population average. The only way this average of averages works out is if your non-overlapping subsets have the same size. Otherwise, you need to take a weighted average.

Your problem is made more complex than this traditional average of averages confusion by having overlapping subsets, but it appears to me to just be this classic mistake with a twist. With overlapping subsets, it is even harder to end up with subsample averages that average to the population average.

In your example, since users who appear in multiple subsamples (and therefore have bought many things) will increase these averages. Basically you're counting each big-spender multiple times, while the frugal people that only buy one item are only encountered once, so you're biased to larger values. This is why your particular subsets have above average values, but I think this is still just the "average of averages" problem.

You can also construct all kinds of other subsets from your data where the subsample averages take on different values. For example, let's take subsets somewhat similar to your subsets. If you take the subset of people who did not buy A, you get 7/5=1.4 items on average. With the subset that did not buy B, you also get 1.4 items on average. Those who did not buy C, bought 1.5 items on average. These are all below the population average of 1.6 items/customer. Given the right dataset and the right collection of subsets, you could end up with overlapping subsets whose averages average to the population average; however, this would be uncommon in normal applications.

Is it just me, or does the word average now seem weird after so many repetitions... Hope my answer was helpful, and sorry if I ruined the word average for you!

Answer 5

Since the issue is "I understand it but need to explain this to marketing", OP seems concerned with how a layman will interpret these facts - (not whether the facts are true, or how to show that they are). The question references 10 product categories, (A-J), so how about this example:

[in meeting with marketing group]
OP: So, as you can see here, customers who buy A, B, and C, are all more valuable than average.
Layman: Wait?! How can everyone be higher than average?
OP: Good question. This slide focuses on customers of A, B, and C, but there are other, low performing, groups not shown. For example, customers of categories D and G are each worth about half of average.

This should quell everyone's internal bs-alarm about 'everything is above average'.

Answer 6

Ignore the other answers here. This actually is not a paradox at all. The actual issue at hand here which everyone seems to be ignoring is that you are mistaking which probability you are actually looking at. There are in fact two completely different averages and statistics at play here which both have there own uses and interpretations in your proposed example (marketing)!

First off there is the average number of products bought per customer. So on average, one customer buys 1.6 items. Of course, a customer cannot but 0.6 of the product (assuming it isn't something like rice or grain that has a continuous measurement associated with it).

Secondly, there is the average number of customers that buy a particular product. Sounds weird right? So on average a product has 5.33333333... customers buying it. This is different however. What we're describing here is not the number of products bought (there's only three of them!) but rather the number of people actually purchasing said product.

Think of the two values this way: What would these two values represent if there was only one customer or only one product? After all, the average of a single data point is just that given data point.

Or better yet, think of the chart as if it were giving you dollar amounts spent to buy the product. Obviously the average amount spent by an individual customer will be far less than the amount of money made on average by a product supplied by a major corporation (or even just a small business). I'm sure you can think of good ways to use both values when discussing the well-being of the company.

When you go to explain this to the marketing staff, explain it to them just like I have said. It isn't a paradox. It's just a completely different statistic. The only issue here was noticing that there was in fact, two different ways to read the chart (i.e. number of people buying per product vs. number of products bought per person).

tl;dr the first thing you described is the average amount an individual customer is willing to spend buying your products. The second is the average demand for a given product by the public. I'm sure you can see now why both are most certainly not the same thing. Comparing them as such will just give you garbage information.

---

EDIT

It would appear the question is actually asking about the average money spent by customers who buy some product a,b, or c. Alright. This is actually just an error in calculations. I wouldn't call this a paradox. It's really just a subtle flub.

Look at your columns. There are people that are shared between columns. Let's assume you did a proper weighted average. You are still adding up people twice. This means that the average will contain extra people with a value greater than or equal to 2. Now what was your average? It was 1.6! In essence your average looks like this:

$\frac {\sum_{i = 0}^{n} valueOfPerson_i*valueOfPerson_i} {n}$

That is definitely not the right formula. It is a weighted average though assuming mutual exclusiveness that is how you would adjust to get a true average in your situation.

$\frac {\sum_{i = 0}^{n} numberOfPeopleBuying_i*averageSpentByPersonBuying_i} {n}$

Either way you'll get a messed up average. One mistake was ignoring the need for a weighted average as one category has a greater "weight" in terms of the average. It's like density. One value is denser in people represents. The other issue is duplicate adding which will distort the average. I don't call either of these "paradoxes" though. Once I saw what you were doing it seemed obvious to me why that wouldn't work. The weighted average is somewhat self-explanatory for its need and I think now that you see that you added values multiple times... that cannot work. You basically took the average of the squares of their values.