0

I am trying to measure how well scores from members of a group are dispersed over possible values. Think of it as a measure of diversity function. Thus,

f(1.0, 1.0, 1.0) = 0
f(1.0, 0.5, 0.0) = 1

In other words, if values are equally dispersed, I want a high value. If any values are clumped near the top, bottom, or middle, the value should be low. 

f(1.0, 0.66, 0.3, 0) > f(1.0, 1.0, 0, 0)

This property is not true of standard deviation.

Presumedly, since it is measuring the scatter, the actual values shouldn't factor into the result. That is,

f(1.0, 0.9, 0.5) = f(0.5, 0.1, 0)

Taking the minimum difference between sorted data and comparing this with the average difference would be close. However, this would not provide a quantitative differentiation between a few clumped data and a lot of clumped data. That is, in the ideal function:

f(1.0, 0.5, 0.5, 0) > f(1.0, 0.5, 0.5, 0.5, 0)

Inspired by a goodness to fit test, I tried using the sum of the squares of the distance from an ideal distribution, but my implementation was not scaled in relation to the inputs.

Is there a statistical function that has these properties?

Bonus points for clues how to implement it programmatically.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Wes Modes
  • 101
  • 2
  • 3
    What do you mean by *my implementation was not scaled in relation to the inputs*? Aren't you just trying to perform a goodness of fit test to a uniform distribution? – Frans Rodenburg Feb 26 '18 at 00:08
  • Maybe there is a proper way to do a goodness to fit, but I don't know how to do it. I just divided the space into the number of elements and measured the distance of the sorted data from this ideal. I squared this distance and summed them. – Wes Modes Feb 26 '18 at 00:39
  • Maybe what you're looking for is related to information entropy : https://en.wikipedia.org/wiki/Entropy_(information_theory). – baruuum Feb 26 '18 at 01:21
  • 3
    There are a great many functions with these properties including K-L divergence and the KS statistic (both relative to uniform distributions on the possible values). If you could tell us more about the intended use of the measurement we might be able to suggest some appropriate candidates. – whuber Feb 26 '18 at 17:51
  • @whuber, I am assembling project groups of students. I had students self-report their expertise and group style. I want each group to have a diverse range of expertise and group styles. Therefore, when evaluating a potential group, among other things like matching schedules, I create a score that rates how diverse each group is. for example, a group of 4 with values f(0.0, 0.33, 0.66, 1.0) should get a score of 1.0. while a group that is all clustered at any value should be closer to 0.0. – Wes Modes Mar 05 '18 at 02:52

1 Answers1

1

It is not totally clear what your arguments to $f$ represents (they do not sum to 1 or any other constant, for instance). But what you describe, "to measure the diversity of some group of students" (in expertise and working style), you need some measure of diversity. These are known in ecology as measures of biodiversity, in economics for instance as measures of (income) inequality, and in other fields.

You could get some ideas from Wikipedia: measuring biodiversity, Wikipedia: diversity index and Wikipedia: Income inequality metrics.

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467