8

I am analyzing a dataset containing observations from n number of attempts by players in a game. If I am building a regression model to predict the outcome of each attempt given 1 or more descriptors regarding each players attempt, how do I measure how many attempts a player must have before any particular descriptor becomes "meaningful" or "significant" given our dataset?

A good example would be a group of people playing darts. An absolute novice might close his eyes, throw the dart and hit a bull's eye. Obviously, we know that was pure chance, he has never played before, and heck, he might even be intoxicated. But let's say he throws 5 more darts and hits a bull's eye 3 more times out of those 5. He has now thrown 6 darts and hit a bull's eye 50% of the time. Now things are getting suspicious...

At what point does this true novice throw enough bull's eye darts, and how many attempts does he need before we can say he has serious talent above other more experienced players? What is this term in statistics and where can I learn more about it?

Disclaimer: I tried to avoid using loaded statistical terms like "significance" and "observations" as much as possible, though I felt they were appropriate in certain places here.

If anyone can point me in the right direction, that would be fantastic. Thanks!!

KidMcC
  • 203
  • 1
  • 4
  • 2
    You are just a few steps away from becoming a Bayesian... – Marcelo Ventura Jul 04 '16 at 21:33
  • So it's a question of confidence intervals as more darts are thrown, to some degree? – KidMcC Jul 04 '16 at 21:38
  • What you are looking at might be sequential testing. Or, you might want to calculate the statistical power, that is how many samples do you need to draw to reduce the beta-error; notice that when you are looking at one certain player the samples are not independent. All of this could also naturally be embedded into a bayesian framework, as the first commenter said. – Regenschein Jul 05 '16 at 20:14
  • *Adaptive* and *sequential* are two useful keywords. – whuber Jul 05 '16 at 20:35

1 Answers1

1

Darts is the simplest of games. Each player starts with a score of 501 and takes turns to throw 3 darts. The score for each turn is calculated and deducted from the players total. Bullseye scores 50, the outer ring scores 25 and a dart in the double or treble ring counts double or treble the segment score.

Image from http://www.shotdarts.com/darts-info/play-darts

Now the probabilities have been examined elsewhere . On that site, we are told "A medium skilled darts player will have a larger standard deviation; even though the shots may, on average, be centered around the same target, they will be distributed over a broader region. A poor darts player will have a high standard deviation and their shots will be, probabilistically, scattered over a much wider region." That is, darts scores.

Thus, to answer the question, we do what we always do. We build up a histogram of scores, and for the game itself we might use 501-score, and then we fit a density function, and then we test that density function against other players' density functions. So, we need enough data so that our location and its deviation have enough predictive power to discriminate properly between players. The less data, the more fuzzy the answers, and there is no magic number for it, the more the merrier.

Carl
  • 11,532
  • 7
  • 45
  • 102