Plot a highly skewed dataset

Question

I have a dataset that contains music genres, songs and a "speechiness" rate. An example looks like this:

Song, Genre,  Speechiness
Dance with wolves, Trance, 0.05
My heart will go on, Pop, 0.09

I have used this data to create charts of the with diffent parameters per genre (including speechiness). Thing is that the data is highly skewed (the danceability ranges from 0-1 and 80% of the data is <0.1.

If you plot the data this is what I get:

https://www.flickr.com/photos/112983354@N05/23025503853/in/dateposted-public/

If I plot the average "speechiness" per genre the visables do not really cut it. If I plot it in a radar chart, this is what I get (the S represents the speechiness):

https://www.flickr.com/photos/112983354@N05/23356885200/in/dateposted-public/

All are between 0.05 and 0.1. I now would like to change the values so the graphs better show the difference between the genres. Of course I could limit the axis but something tells me there is a better option.

Any thoughts on something like a scaling method so I get a better and more informative graph?

Could you please provide a graph? I'm having a hard time understanding what you are exactly asking, so all your distributions are overlapping and you cannot discern their difference? — maininformer, Dec 10 '15 at 11:38
Also providing sample data would help people to produce meaningful examples. — Tim, Dec 10 '15 at 11:52

score 2 · Answer 1 · answered Dec 10 '15 at 11:49

For heavily skewed data I would normally get the log() of the value. The only issue here is that your scale is so small it would become hard to interpret. You could scale up your data and then the get the log e.g. log(x*1000) or alternatively you could normalise it, center the scale at zero and adjust the standard deviation to one.

set.seed(9467)

d <- abs(rnorm(10000, 0.075, 0.005))    #generate most freq values
d.larg <-rnorm(10, 0.5, 0.3)            #generate some high skewing values

ds <- rbind(d, d.larg)                  #combine data
skewness(ds,na.rm=TRUE)                 #highly skewed 


# Using the log to scale back larger values
x11()
par(mfrow = c(3,1))
hist(d)
hist(ds)
hist(log(ds + 0.000001))                #adding the 0.000001 is so you don't get NA's for zero values


#centering the dataset by normailising it 
normalised.ds <- (ds - mean(ds))/sd(ds)
x11()
par(mfrow = c(3,1))
hist(d)
hist(ds)
hist(normalised.ds) 
hist(normalised.ds, breaks = 20)        #could add more breaks to see smaller distributions

The major issue is likely to be with any reported zeros. Adding a very small number such as 0.000001 (as commented in your code) is **not** at all a conservative solution. Consider log base 10 for convenience; then zeros are mapped to -6; 0.01 to just above -2 and 1 to just above 0. So the fudge will create a massive outlier spike. The issue arises with any other base, naturally. A better although still imperfect solution is to add a number such as half the smallest reportable positive value: thus if results are reported to 2 d.p., adding 0.005 will reduce (but not remove) this problem. — Nick Cox, Dec 10 '15 at 13:09

score 1 · Answer 2 · answered Dec 10 '15 at 13:49

The problems are definitely

Extreme skewness

and very likely

Reported zeros
Granularity of results, e.g. reporting to $2$ decimal places, i.e., $0, 0.01,$ etc.

Absent data to play with, I invented some as, given a sample $x$ of size $1000$ from a uniform on $(0,1)$, "scores" $\lfloor 100 x^5 \rfloor/100$. Here is what they look like fairly conventionally in a particular sample which may understate the problem as $662$ are less than $0.01$.

The first idea is clearly a transformation of the scale. Logits are ruled out by the occurrence of exact zeros. Rather than fudge or generalise the definition of logit, I suggest considering folded powers as discussed by J.W. Tukey, most accessibly in 1977. Exploratory data analysis. Reading, MA: Addison-Wesley. By analogy with $\text{logit}\ x = \ln[x/(1-x)] = \ln x - \ln (1 - x)$ consider any positive power $p$ and $x^p - (1 - x)^{1-p}$. Square roots $(p = 1/2)$ and cube roots $(p = 1/3)$ spring to mind. Even the cube root leaves that distribution skewed and the granularity is a little disconcerting.

The problem is naturally that we have not transformed frequencies either. Here a root scale is also a possibility (which Tukey elsewhere called a rootogram).

The trade-off here is delicate. To see any but the grossest skewness of the data, we need to think on what are likely to seem unconventional or unfamiliar scales.

Plot a highly skewed dataset

2 Answers2

Linked