1

I am pretty new to statistics and I'd like to get pointers on the correct way to do this.

I have a dataset in which I'm interested in the 50th and 90th percentile. I'd like to take a sample of that set but still preserve (with some margin of error) the 50th and 90th percentiles.

I am unsure how to approach this -- I know the t-test compares means, but it doesn't seem to me it can be used for my purposes. My data also has a strong skew (long right tail). If someone can suggest an algorithm or approach to this I would much appreciate it.

I am not writing a paper, just need to drop a large portion of data in a reasonable manner.

Yana K.
  • 111
  • 1
  • 1
    Could you elaborate on what you mean by "preserve"? Several interpretations come to mind (but I suppose there may be others): (1) You want to take a *random* sample whose 50th and 90th percentiles are "close" to those of the dataset in some sense and you are asking how large a sample to take. (2) You want to take a *nonrandom* sample whose 50th and 90th percentiles are "close" to those of the dataset but you otherwise do not care (much) about how well this sample represents other features of the dataset. (3) You want to compress the data in a way that recovers these percentiles correctly. – whuber Oct 02 '14 at 15:31
  • 1) would be ideal. I could live with 2). I am not interested in compressing the data. My application will process a stream, so knowing that I can keep K random items is ideal. Keeping non-random or compressing would be harder, I think – Yana K. Oct 02 '14 at 16:28
  • Processing a stream is a crucial aspect. Are you perhaps asking the question at http://stats.stackexchange.com/questions/7959/algorithm-to-dynamically-monitor-quantiles? – whuber Oct 02 '14 at 17:00
  • We started with the thought of estimating the percentiles over a stream but decided it would be better to retain a subsample of the stream so we can answer ad-hoc questions over it later. So this is why I'm hoping to be able to keep a random sample of sufficient size to preserve most aspects of the stream. But the percentiles of one metric are the most important aspect to preserve, if I had to give up something. Just not sure if this is doable – Yana K. Oct 03 '14 at 15:43

0 Answers0