16

I was reading a paper recently that incorporated randomness in its confidence and credible intervals, and I was wondering if this is standard (and, if so, why it is a reasonable thing to do). To set notation, assume that our data is $x \in X$ and we are interested in creating intervals for a parameter $\theta \in \Theta$. I am used to confidence/credibility intervals being constructed by building a function:

$f_{x} : \Theta \rightarrow \{0,1\}$

and letting our interval be $I = \{ \theta \in \Theta \, : \, f_{x}(\theta) = 1\}$.

This is random in the sense that it depends on the data, but conditional on the data it is just an interval. This paper instead defines

$g_{x} : \Theta \rightarrow [0,1]$

and also a collection of iid uniform random variables $\{U_{\theta} \}_{\theta \in \Theta}$ on $[0,1]$. It defines the associated interval to be $I = \{ \theta \in \Theta \, : \, f_{x}(\theta) \geq U_{\theta} \}$. Note that this depends a great deal on auxillary randomness, beyond whatever comes from the data.

I am very curious as to why one would do this. I think that `relaxing' the notion of an interval from functions like $f_{x}$ to functions like $g_{x}$ makes some sense; it is some sort of weighted confidence interval. I don't know of any references for it (and would appreciate any pointers), but it seems quite natural. However, I can't think of any reason to add auxillary randomness.

Any pointers to the literature/reasons to do this would be appreciated!

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
QQQ
  • 161
  • 2
  • 5
    (+1) This is called a *randomized procedure.* They are a standard part of the statistical estimation and testing framework, so you can rely on just about any rigorous textbook to provide explanations. Additional motivation for their use can be found in the game theory literature. – whuber Dec 28 '14 at 17:01
  • Thanks for the response. I realized after reading this comment that e.g. bootstrapping fits into this framework, but in that situation the reason for randomization is clear (you don't have access to f, just g). In my case, the authors explicitly calculate $f_{x}$, and THEN look at $g_{x}$. Although I have many stats textbooks, I don't see this anywhere... do you have a suggested text? – QQQ Jan 01 '15 at 03:33
  • 3
    Actually, bootstrapping is *not* a randomized procedure. It is a *determinate* procedure whose *approximate calculation* is performed by means of random sampling. – whuber Jan 01 '15 at 05:31

2 Answers2

4

Randomized procedures is used sometimes in theory because it simplifies the theory. In typical statistical problems, it does not make sense in practice, while in game-theory settings it can make sense.

The only reason I can see to use it in practice, is if it somehow simplifies calculations.

Theoretically, one can argue it should not be used, from the sufficiency principle: statistical conclusions should only be based on sufficient summaries of the data, and randomization introduces dependence of an extraneous random $ U $ which is not part of a sufficient summary of the data.

UPDATE

To answer whuber's comments below, quoted here: "Why do randomized procedures "not make sense in practice"? As others have noted, experimenters are perfectly willing to use randomization in the construction of their experimental data, such as randomized assignment of treatment and control, so what is so different (and impractical or objectionable) about using randomization in the ensuing analysis of the data? "

Well, randomization of the experiment to get the data is done for a purpose, mainly to break causality chains. If and when that is effective is another discussion. What could be the purpose for using randomization as part of the analysis? The only reason I have ever seen is that it makes the mathematical theory more complete! That's OK as long as it goes. In game-theory contexts, when there is an actual adversary, randomization my help to confuse him. In real decision contexts (sell, or not sell?) a decision must be taken, and if there is not evidence in the data, maybe one could just throw a coin. But in a scientific context, where the question is what we can learn from the data, randomization seems out of place. I cannot see any real advantage from it! If you disagree, do you have an argument which could convince a biologist or a chemist? (And here I do not think about simulation as part of bootstrap or MCMC.)

kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
  • 1
    Why do randomized procedures "not make sense in practice"? As others have noted, experimenters are perfectly willing to use randomization in the *construction* of their experimental data, such as randomized assignment of treatment and control, so what is so different (and impractical or objectionable) about using randomization in the ensuing *analysis* of the data? – whuber Jan 01 '15 at 19:58
  • whuber: As I said, it goes aganst he sufficiency principle. – kjetil b halvorsen Jan 01 '15 at 20:13
  • 1
    @kjetil I think you might not have completed your statement about the sufficiency principle, it seems to have been cut off mid-sentence ("statistical conclusions should..."). – Silverfish Jan 01 '15 at 22:33
  • Thank you for completing that sentence. However, it does not really seem to address the point I was quoting. After all, the *selection* of the data themselves was based on "extraneous random" variables, so what is the matter with using one more such variable as part of the overall experiment? – whuber Jan 02 '15 at 00:15
  • 1
    (+1) But I think it's begging the question to invoke the Sufficiency Principle, the argument for which is that once you know the observed value of the sufficient statistic, taking any other aspect of the data into account is *equivalent to introducing an extraneous random $U$*. So someone proposing to do just that wouldn't give a fig for the Sufficency Principle. Also, see Basu (1978), "Randomization in Statistical Experiments", *FSU Statistics Report M466* for a couple of randomized procedures proposed in earnest. – Scortchi - Reinstate Monica Feb 18 '15 at 09:56
  • 1
    @whuber: It is a clear, principled argument that randomization in obtaining the data may be advantageous. (It breaks causal chains). What is that principled argument for using randomization as part of the analysis? – kjetil b halvorsen Feb 18 '15 at 10:24
  • 1
    Kjetil: It enables you to achieve the intended risk function, rather than accepting a risk function (often in the form of nominal size and power) that is not what you wanted. Moreover, if a procedure is "theoretically" useful then there certainly can be no objection to its use in practice, other than impracticability (which is usually not the case with randomized procedures). Thus your question should be turned on its head: the burden is on *you* to demonstrate there is something wrong with using randomized procedures. How do you accomplish that without contradicting yourself? – whuber Feb 18 '15 at 16:14
  • 1
    @whuber: I have problems accepting this. In a real decision problem, maybe. But the use I am accustomed to is really in evaluating scientific evidence. There can be no role for extraneous randomization in defining the evidence about some hypothesis from a scientific experiment! – kjetil b halvorsen Feb 18 '15 at 16:18
  • 1
    That's an insightful distinction: we cannot demand that all statistical procedures be optimal for all applications. We could argue over whether randomized procedures have a role in doing science--which is rather an epistemological debate rather than a statistical one--but I'll let that drop and just point out that statistical methods are applied in many other circumstances, such as helping people make decisions and take actions. Note, too, I am not maintaining that people *should* use randomized procedures, but only suggesting that their use should not be ruled out. – whuber Feb 18 '15 at 16:32
  • @whuber: Yes, thanks. I can even imagine situation where using a randomized decision procedure could give you problems in a court case ... needing to defend the use of randomization in front of a jury ... – kjetil b halvorsen Feb 18 '15 at 16:41
  • 1
    That's right: the suspicion with randomization is that it's too easy to cheat--just keep drawing random numbers until you get the decision you want! But in some circumstances you can get around that problem by announcing your procedure in advance and even drawing the random numbers in advance. It is only through such careful controls that randomization actually works for selecting experimental subjects in heavily regulated arenas such as drug trials. – whuber Feb 18 '15 at 16:46
  • 1
    @whuber I think your comments here and under the original post, about the decision-theoretic advantages, possibly deserve breaking out into a separate answer. – Silverfish Feb 18 '15 at 17:07
3

The idea refers to testing, but in view of the duality of testing and confidence intervals, the same logic applies to CIs.

Basically, randomized tests ensure that a given size of a test can be obtained for discrete-valued experiments, too.

Suppose you want to test, at level $\alpha=0.05$, the fairness of a coin (insert any example of your choice here that can be modelled with a Binomial experiment) using the probability $p$ of heads. That is, you test $H_0:p=0.5$ against (say) $H_1:p<0.5$. Suppose you have tossed the coin $n=10$ times.

Obviously, few heads are evidence agaist $H_0$. For $k=2$ successes, we may compute the $p$-value of the test by pbinom(2,10,.5) in R, yielding 0.054. For $k=1$, we get 0.0107. Hence, there is no way to reject a true $H_0$ with probability 5% without randomization.

If we randomize over rejection and acceptance when observing $k=2$, we may still achieve this goal.

Christoph Hanck
  • 25,948
  • 3
  • 57
  • 106
  • This is a nice explanation of the use of randomization, but it would be nice if it explained why we might be interested in attaining arbitrary $\alpha$ in the first place. Why is it a desirable goal? – Silverfish Feb 18 '15 at 10:24
  • Well, that I guess brings us back to the history of statistics, when R.A. Fisher somewhat arbitrarily decided to work with a significance level of 5% to decide whether some initial evidence warrants further study. As we know, 5% has since morphed into a sort of gold standard in many fields, despite lacking good decision-theoretic foundation. – Christoph Hanck Feb 18 '15 at 15:49