3

I'm a physicist who has some issues in data treatment, so sorry if I get the maths wrong.

I have some diffractograms, which are displayed as counts vs. angle. The points have standard deviations estimated from a poisson statistic on the bins (an external program does that).

Getting the average of different spectra is straightforward by averaging corresponding counts. But how do I get the correct standard deviation for the averaged counts?

When I apply this method

Is it possible to find the combined standard deviation?

the errors actually get larger compared to single data points. This appears counterintuitive, because I would expect the errors to get smaller when I take into account more datapoints?

Where is my fault?

Community
  • 1
user31240
  • 31
  • 1
  • 1
    You'd expect standard error (possibly) to get smaller, sd should get larger, because the spectra have different means. sd(1,2) and sd(10, 11) are both smaller than sd(1, 2, 10, 11). (If I've understood correctly). – Jeremy Miles Oct 08 '13 at 18:11
  • That example works because the means of 1,2 and 10,11 are far apart. The data I'm dealing with is in principle very well described by a small deviation of a true value, which is caused by the limited counting time. The longer you would count, the smaller the standard deviation should get (if i got the concept right). Conceptually, combining, lets say, 2 datafiles by averaging should be the same as counting 2 times the counts of a single file. However the standard deviation grows, this is not understandable for me. – user31240 Oct 08 '13 at 18:21
  • Can you show (or construct) a small example of original values and the effect you're asking about? Note that if what you're doing is right, the answers done two different ways should be the same. – Glen_b Oct 08 '13 at 19:29
  • The original value is a continuous flux of photons, with a continuous dis – user31240 Oct 09 '13 at 05:48
  • Sorry, still not comfortable with the way commenting works. So the two values counted for 5 s at a certain angle are: y1=12.32 \pm 0.03 cps, y2=12.33 \pm 0.03 cps The value counted for 10s is: Y=12.31 \pm 0.02 cps If i now combine both values counted for 5s in the way it is done in the link provided I end up with: Y'=12.33 \pm 0.03 cps What bothers me is that the error here is larger than for the 10s counting case. Of course this is a small deviation but the behaviour is the same for combining more than 2 different diffraction patterns, this amplifies the effect. – user31240 Oct 09 '13 at 09:19

1 Answers1

5

If I understand the question correctly, you have two values ("counts") $x$ and $y$ with accompanying estimates of their standard deviations, $\sigma_x$ and $\sigma_y,$ respectively. Let's suppose $x$ and $y$ are independent realizations of random variables $X$ and $Y,$ both of which have the same mean $\mu$. Let us further suppose that $\sigma_x$ and $\sigma_y$ are highly accurate estimates of the true standard deviations of $X$ and $Y$. You wish to find a linear combination

$$\xi x + \eta y$$

that estimates $\mu.$ Among the many ways to do this, one that is commonplace in physics (going back about 210 years) is to find a linear combination that minimizes the mean squared deviation

$$\mathbb{E}_{X,Y}[\left(\xi X + \eta Y - \mu\right)^2] = \mathbb{E}_{X,Y}[\left(\xi(X-\mu) + \eta(Y-\mu) + \mu(\xi+\eta-1)\right)^2] .$$

Upon expanding the argument algebraically and using the definitions $\mathbb{E}_X[X]=\mathbb{E}_Y[Y]=\mu,$ $\mathbb{E}_X[(X-\mu)^2]=\sigma_X^2,$ and $\mathbb{E}_Y[(Y-\mu)^2]=\sigma_Y^2,$ we obtain

$$\xi^2 \sigma_X^2 + \eta^2 \sigma_Y^2 + (\xi+\eta-1)^2\mu^2.$$

Because we do not know $\mu$, the only way to proceed is to arrange for the estimator to be independent of $\mu,$ whence

$$\xi + \eta = 1.$$

Subject to that constraint, the unique solution $(\xi,\eta)$ minimizing the objective makes $\xi$ proportional to $1/\sigma_x^2$ and $\eta$ proportional to $1/\sigma_y^2$, yielding

$$\frac{\sigma_Y^2 X + \sigma_X^2 Y}{\sigma_X^2 + \sigma_Y^2}$$

for the estimator. Its variance is

$$\frac{\sigma_X^2 \sigma_Y^2}{\sigma_X^2 + \sigma_Y^2}.$$

This never exceeds one-half the larger of $\sigma_X^2$ and $\sigma_Y^2$.

whuber
  • 281,159
  • 54
  • 637
  • 1,101
  • Ok,thank you for the help, this looks reasonable. Is this easily extended for more than 2 input values? My idea would be to do that stepwise, but then it becomes asymmetric. I could not follow the derivation completely (don't even know some symbols) but I guess that's not necessary anyway. Do you know if there is something citeable that contains the formula? Like a textbook for example? – user31240 Oct 09 '13 at 11:20
  • The strange symbols are from the [Greek alphabet](http://en.wikipedia.org/wiki/Greek_alphabet). The analysis easily (and almost automatically) generalizes to any number of variables. The most direct way to obtain the general solution is with [Lagrange multipliers](http://en.wikipedia.org/wiki/Lagrange_multiplier), with which you--as a physicist--are surely intimately familiar. The result is extremely well known, having been rediscovered in many fields. It plays an important role in [finance](http://www.columbia.edu/~ks20/FE-Notes/4700-07-Notes-portfolio-I.pdf), for instance. – whuber Oct 09 '13 at 15:56
  • Well thank you for the insightful comment, I'm sure there's no \mathbb{E} in the greek alphabet. Took me a while to figure out we're talking about simple linear regression. Could have thought about that myself. Once you know the name, it's easy to follow. – user31240 Oct 10 '13 at 07:33
  • I am sorry that I did not correctly guess that your reference to "symbols" meant "$\mathbb{E}$;" I had to guess which ones needed explanation and apparently I guessed wrong. "Simple linear regression" is not a term conventionally applied to this situation. The solution derived here is indeed an example of a *generalized least squares* estimator, as well as an application of *unbiased linear estimation.* In the same vein, perhaps it would be helpful to know that this is an extremely simple example of a Kriging estimator. – whuber Oct 10 '13 at 15:31