Normal Distribution: Is it made with replacement or without replacement?

Question

I am confused about how a normal distribution is drawn. Are the elements drawn with or without replacement?

Answer 1

While the question is more or less wrong because the type of sampling depends on your need, it has however an interesting point. We extend the discussion on discrete/continuous random variables, since it is more general topic.

In general you want a sample with replacement when you need draws which are independent and you want a sample without replacement when you need dependence between draws for some purpose.

We will continue our discussion on a technical side. When you have the whole population at hand, specified by enumerating all population values, on sampling multiple draws you have the choice to replace the previously drawn value or not. What about the case when you do not have the whole population available, only a functional form of it the question is how do you replace a value? The only way to replace it is to remember it somehow, but even if you remember, how do you proceed?

Let's take a continuous distribution. You draw a sample, you want to remember it. If if you would remember it, you have a finite float representation. On the other hand you have an infinity of values for each limited floating point value. If you remember what you have drawn you simply interdict drawing values from the infinity of values with the same limited representation and your sample is simply not from the same distribution. So for continuous variables is close to impossible to implement 'with replacement' and even if you do, is practically useless since for most applications there will be no notable differences.

Let's take a discrete distribution, Binomial(10,0.5). How many observations are in your population? 10 at each draw. You draw a value from that distribution, let's say is 5. The question is: which would be the meaning of remembering 5 and not drawing after the first draw? It certainly does not mean 'without replacement' because you do not obtain a value from the same population described by one draw of Binomial(10,0.5), it' something else. So you can say that drawing samples from Binomial(10,0.5) can be done only with replacement.

Hypergeometric distribution assumes again a finite population and a finite composition, like 10 balls, 3 black and 7 red and count reds in 3 draws. In those cases the drawing is without replacement by construction.

As a conclusion: if the population is finite and enumerated or in an equivalent form, that you can do either with or without replacement. Some discrete variables on finite populations assumes by construction only one way of sampling. For continuous distributions or infinite populations is simply impossible and/or impractical to implement it, and whenever is possible to implement it produces very similar results. In those cases the sampling can be considered with replacement.