Statistical Analysis Stack Exchange
Statistical Analysis Stack Exchange

Questions tagged [statistical-significance]

Statistical significance is a characteristic of a statistic viewed in light of a null hypothesis and a given significance level. It reflects whether the statistic belongs to the rejection region (is statistically significant) or the acceptance region (is not statistically significant).

Statistical significance is a characteristic of a statistic viewed in light of an (implicit or explicit) null hypothesis and a given significance level. It reflects whether the statistic belongs to the rejection region or the acceptance region defined by the null hypothesis and the significance level. The statistic is then statistically significant or not statistically significant, respectively.

5649 questions
169
votes
16 answers

Are large data sets inappropriate for hypothesis testing?

In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of no effect is true — will declare a statistically…
Carlos Accioly
  • 4,715
  • 4
  • 25
  • 28
102
votes
9 answers

Is this really how p-values work? Can a million research papers per year be based on pure randomness?

I'm very new to statistics, and I'm just learning to understand the basics, including $p$-values. But there is a huge question mark in my mind right now, and I kind of hope my understanding is wrong. Here's my thought process: Aren't all researches…
n_mu_sigma
  • 1,071
  • 2
  • 8
  • 6
95
votes
2 answers

How much do we know about p-hacking "in the wild"?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ways to procure a "more significant" result,…
Silverfish
  • 20,678
  • 23
  • 92
  • 180
81
votes
9 answers

Regarding p-values, why 1% and 5%? Why not 6% or 10%?

Regarding p-values, I am wondering why $1$% and $5$% seem to be the gold standard for "statistical significance". Why not other values, like $6$% or $10$%? Is there a fundamental mathematical reason for this, or is this just a widely held…
Contango
  • 1,387
  • 1
  • 16
  • 15
71
votes
3 answers

Is this the solution to the p-value problem?

In February 2016, the American Statistical Association released a formal statement on statistical significance and p-values. Our thread about it discusses these issues extensively. However, no authority has come forth to offer a universally…
whuber
  • 281,159
  • 54
  • 637
  • 1,101
67
votes
3 answers

References containing arguments against null hypothesis significance testing?

In the last few years I've read a number of papers arguing against the use of null hypothesis significance testing in science, but didn't think to keep a persistent list. A colleague recently asked me for such a list, so I thought I'd ask everyone…
Mike Lawrence
  • 12,691
  • 8
  • 40
  • 65
64
votes
2 answers

Do we need a global test before post hoc tests?

I often hear that post hoc tests after an ANOVA can only be used if the ANOVA itself was significant. However, post hoc tests adjust $p$-values to keep the global type I error rate at 5%, don't they? So why do we need the global test first? If…
even
  • 2,147
  • 6
  • 18
  • 13
63
votes
3 answers

Explain the xkcd jelly bean comic: What makes it funny?

I see that one time out of the twenty total tests they run, $p < 0.05$, so they wrongly assume that during one of the twenty tests, the result is significant ($0.05 = 1/20$). xkcd jelly bean comic - "Significant" Title: Significant Hover text:…
DJG
  • 693
  • 1
  • 7
  • 6
63
votes
6 answers

Test if two binomial distributions are statistically different from each other

I have three groups of data, each with a binomial distribution (i.e. each group has elements that are either success or failure). I do not have a predicted probability of success, but instead can only rely on the success rate of each as an…
Scott
  • 900
  • 1
  • 8
  • 12
62
votes
13 answers

Two-tailed tests... I'm just not convinced. What's the point?

The following excerpt is from the entry, What are the differences between one-tailed and two-tailed tests?, on UCLA's statistics help site. ... consider the consequences of missing an effect in the other direction. Imagine you have developed a new…
FromTheAshes
  • 743
  • 5
  • 6
61
votes
10 answers

What does "Scientists rise up against statistical significance" mean? (Comment in Nature)

The title of the Comment in Nature Scientists rise up against statistical significance begins with: Valentin Amrhein, Sander Greenland, Blake McShane and more than 800 signatories call for an end to hyped claims and the dismissal of possibly…
uhoh
  • 685
  • 1
  • 6
  • 10
61
votes
6 answers

Is the "hybrid" between Fisher and Neyman-Pearson approaches to statistical testing really an "incoherent mishmash"?

There exists a certain school of thought according to which the most widespread approach to statistical testing is a "hybrid" between two approaches: that of Fisher and that of Neyman-Pearson; these two approaches, the claim goes, are "incompatible"…
amoeba
  • 93,463
  • 28
  • 275
  • 317
57
votes
4 answers

Manually Calculating P value from t-value in t-test

I have a sample dataset with 31 values. I ran a two-tailed t-test using R to test if the true mean is equal to 10: t.test(x=data, mu=10, conf.level=0.95) Output: t = 11.244, df = 30, p-value = 2.786e-12 alternative hypothesis: true mean is not…
herbps10
  • 673
  • 1
  • 5
  • 6
56
votes
2 answers

A/B tests: z-test vs t-test vs chi square vs fisher exact test

I'm trying to understand the reasoning by choosing a specific test approach when dealing with a simple A/B test - (i.e. two variations/groups with a binary respone (converted or not). As an example I will be using the data below Version Visits …
L Xandor
  • 1,119
  • 2
  • 9
  • 13
55
votes
4 answers

How do we decide when a small sample is statistically significant or not?

Sorry if the title isn't clear, I'm not a statistician, and am not sure how to phrase this. I was looking at the global coronavirus statistics on worldometers, and sorted the table by cases per million population to get an idea of how different…
Avrohom Yisroel
  • 673
  • 5
  • 7
1
2 3
…
99 100