1

I am running some simulations and need a hand with the stats.

Suppose I have $N$ systems. I simulate all of these systems and an event occurs in $x$ of the $N$ systems. I then make a small change to the code such that when I re-run the simulations, the event occurs in $y$ of these systems, where $x <y$.

I want to determine to what confidence I can say that changing the code caused the number of events observed across all systems to increase. The standard in science I've learnt is the 3$\sigma$ confidence - a 99.73% confidence.

So my questions are:

  1. To what percentage confidence (what sigma level) can I say changing the code increased the number of events?
  2. How many events $y$ would I need in order for a 3$\sigma$ confidence level?
kjetil b halvorsen
  • 63,378
  • 26
  • 142
  • 467
Tom
  • 11
  • 1
  • 1
    See [significance of difference between two counts](http://stats.stackexchange.com/q/155307/17230). – Scortchi - Reinstate Monica Aug 25 '15 at 14:07
  • Thanks for the link but I'm afraid I really don't understand that very well at all – Tom Aug 25 '15 at 16:07
  • 1
    Conditioning on the total no. counts reduces the problem to inference about binomial proportions: see [Confidence interval for Bernoulli sampling](http://stats.stackexchange.com/q/4756/17230). But on re-reading your question I wonder whether it really does concern Poisson counts. If $x$ is the total count of an event that can either occur (once) or not in each of the $N$ systems then you're comparing two binomial distributions: see https://onlinecourses.science.psu.edu/stat414/node/268. – Scortchi - Reinstate Monica Aug 26 '15 at 09:55

0 Answers0