2

I am trying to make an hypothesis test using bootstrapping.

I compute a quantity Q from a sample set (the exact calculation should not be relevant, but let's say that Q is the average time between two particular events). I have N different sample sets, so that I have Q1...QN.

I am interested on the following two quantities:

  • Q_max: this is the maximum of (Q1, ... QN)
  • Q_05: this is the median of (Q1, QN)

Q_max and Q_05 are therefore two different statistics which depend on N different sample sets.

I want to test the hypothesis that Q_max is larger than 0, and also that Q_05 is larger than 0, i.e., the maximum average and the median average are larger than 0. Note: the same procedure should work for every percentile.

I start with the null hypothesis which assumes that all the Qi are 0 (which would imply that the median and the maximum and all other percentiles are also 0), i.e., I subtract the mean mu1 from the first sample s1 to obtain sample p1, so that Q1 is zero, etc.

Then I apply bootstrap as follows. I re-sample (with replacement) from p1 and compute the average E1, I do the same for all samples so that I get all averages E1..EN. Then I compute E_max=max(E1..EN) and E_05=median(E1..EN). I do the same process 10000 times, now I have an empirical distribution of E_max and E_05.

Now I can compute the p-value of (the initial) Q_max and Q_05 to reject the null hypothesis. Assume I decide that the max p-value is 0.1.

I have two questions:

  • I obtain that the p-value of Q_max is 0.15 (so I cannot reject the hypothesis that the maximum of the means is 0) but I obtain a p-value of 0.06 for Q_05. This implies that the median is larger than 0 but the maximum is not significantly larger. How is this possible? By logical arguments the maximum should also be larger than 0 if the median is larger than 0?
  • Is the method described above OK?
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
user60117
  • 21
  • 2
  • 1
    You say "the exact calculation should not be relevant...", but you actually must verify that the distribution of this test statistical is *pivotal* under the null hypothesis in order to apply the bootstrap for inference. – AdamO Nov 06 '14 at 20:12
  • Be *very* wary of trying to bootstrap a maximum. It typically doesn't work. [Even the median can run into difficulties, especially with small samples -- sometimes people modify their bootstrap procedure (such as moving to a smoothed bootstrap) when dealing with the median.] – Glen_b Nov 07 '14 at 02:00
  • @AdamO: yes, we can assume that the distribution is pivotal, I just tried to avoid details which I thought are not relevant – user60117 Nov 08 '14 at 17:28
  • @Glen_b: what's about testing that the median is >0 on a very large sample and then infer that all quantiles >0.5 (including the max therefore) are also positive? – user60117 Nov 08 '14 at 17:35
  • I'm not sure what you're getting at - how would that relate to discreteness problems in small samples? – Glen_b Nov 08 '14 at 22:37

1 Answers1

3

Bootstrapping primarily works in the center of your distribution. It works progressively less well as you try to bootstrap quantiles towards the tails of your distribution. It necessarily fails when trying to bootstrap the minimum value or the maximum value. It may help you to read @Cardinal's excellent answer here: What are examples where a "naive bootstrap" fails?, which explicitly discusses bootstrapping the max.

Moreover, trying to test the max of a distribution doesn't make much logical sense (as you note). If you believe the median is >0, there is no need to try to test the max also. In addition, most distributions that people work with are treated as unbounded, meaning that the true population max is $+\infty$. It may help you to read some of CV's treads that pertain to the max. Clicking on the tag () and sorting by votes will let you start with the highest rated threads.

Community
  • 1
gung - Reinstate Monica
  • 132,789
  • 81
  • 357
  • 650
  • thanks, I am in the meantime aware of the problems in estimating the max. However, I'd like to stress that I am not really interested in the max itself, I am only interested to know if the max is (significantly) larger than 0 using a hypothesis test. If this is still an issue, do you suggest to just test that the median (or let's say the 0.75 quantile) is larger than 0 and then infer that all larger quantiles (including the max) are also positive? Thx – user60117 Nov 08 '14 at 17:34
  • What kind of data do you have? The max is presumably $+\infty$ (ie >0). – gung - Reinstate Monica Nov 08 '14 at 17:36
  • No, the max is bounded, it is the time between two events which I know is between 0 and 10'000 seconds. – user60117 Nov 08 '14 at 17:38
  • If your data are bounded [0, 10,000], then the max is >0. – gung - Reinstate Monica Nov 08 '14 at 17:39
  • well, I actually subtract 200 as in reality I want to test that the max is larger than 200...so interval is [-200, 9800], sorry for the confusion – user60117 Nov 08 '14 at 17:41
  • You still don't need a hypothesis test. If you have *any* data >200, then the max is >200. If you really need a p-value, p = 0. – gung - Reinstate Monica Nov 08 '14 at 17:44
  • each data is a statistic (sample average) which in turn depends on another sample, i.e. I want to know if the maximum average is >0. So if I observe one sampled average >200 can I conclude that the sample with the maximum **true** (unobserved) mean has a mean which is >0? – user60117 Nov 08 '14 at 17:49
  • It's hard to say. I gather there is a lot more going on in your data than I'm aware of. Most likely you should be fitting a multilevel model, and the random effects terms would be assumed to go to infinity. If you wanted them bounded, you'd probably have to do some heavy coding to fit the model & get the test. I strongly suspect that a test of the max isn't really coherent, though, & comes from people believing that everything needs a p-value. – gung - Reinstate Monica Nov 08 '14 at 18:02
  • yeah it is a tricky problem.. I have N timeseries, each has an average value, and I want to test if the max (or the median) of these N averages is positive. There is no theoretical underlying model. That's why I thought the bootstrapping is a good method to reject the null hypothesis that all the means are 0 (and hence the max is 0). In other words, I want to make sure that if I would have N random series with 0 mean each I would not obtain a similar max average value than the one which I compute from my data, does this make sense to you? – Mannaggia Nov 08 '14 at 19:57