The extent to which subsequent studies of the same phenomenon reproduce the results obtained in the original study.
Questions tagged [replicability]
30 questions
51
votes
4 answers
Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?
I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- and am confused by one of its central claims.…
amoeba
- 93,463
- 28
- 275
- 317
12
votes
2 answers
What fraction of repeat experiments will have an effect size within the 95% confidence interval of the first experiment?
Let's stick to an ideal situation with random sampling, Gaussian populations, equal variances, no P-hacking, etc.
Step 1. You run an experiment say comparing two sample means, and compute a 95% confidence interval for the difference between the two…
Harvey Motulsky
- 14,903
- 11
- 51
- 98
9
votes
2 answers
Automated ML vs the entire replicability/reproducibility crisis
There is a trend in machine learning implementations to make things easier and easier for implementers, a very natural engineering concern. Easy APIs to create any kind of model you want, easy infrastructure to manage versions of data and models,…
Mitch
- 1,691
- 2
- 18
- 33
8
votes
1 answer
Meaning of low power in neuroscience after combining results of many meta-analyses (Button et al 2013)
In a 2013 review article in Nature Neuroscience, Button et al. Power failure: why small sample size undermines the reliability of neuroscience, it was stated that:
the average statistical power of studies in the neurosciences is
very low
They…
arkiaamu
- 735
- 1
- 6
- 13
7
votes
3 answers
Study replication from a Bayesian point of view
Consider following situation: Study A compares two groups and finds a mean difference with an effect size of d = .8 (p<.05). Study B is a direct replication, and finds an effect in the same direction of d = .3 (p < .05).
From an NHST perspective,…
Felix S
- 4,432
- 4
- 26
- 34
7
votes
0 answers
Has the reproducibility crisis affected confidence intervals as well?
The reproducibility crisis has given many pause over the value (?) of $p$-values to measure the relevance of statistical findings. Given the interpretation of a $p$-value and some knowledge of probability, it's not surprising to see how many…
AdamO
- 52,330
- 5
- 104
- 209
5
votes
1 answer
Independent replication experiments yielding contrasting results; how to combine them?
Imagine a simple experiment, trying to answer a simple question. For example, is body temperature the same in men and in women ?
To answer this question, let's say you sample 10 men, and 10 women, randomly from a given city, and measure their…
Rodolphe
- 861
- 5
- 16
4
votes
1 answer
Why replication studies use two-tailed tests?
I have been searching replication studies and I found that all of them use two-tailed tests. For example:
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
I am wondering why…
danilinares
- 381
- 2
- 8
4
votes
1 answer
If original data are available, should replication data be added and everything re-analysed (IPD) or is a meta-analysis better?
One common practice which increases the odds of obtaining spurious
results is to keep collecting observations after a preliminary
analyses are performed[1]. This occurs when the cutoff point for
collecting cases is set as the time when significance…
post-hoc
- 677
- 1
- 6
- 14
3
votes
2 answers
Could we estimate replicability of empirical research with conformal predictions?
A review article Threats of a Replication Crisis in Empirical Computer Science reviews reproducibility issues. The authors present distinctions among repeatability, replicability, and reproducibility. They were a bit pessimistic. Now, the question…
msuzen
- 1,709
- 6
- 27
3
votes
0 answers
Repeating experiment with biological replicates
My experiment gives data of 12 biological replicates. This would be sufficient to calculate the mean and give a valid standard deviation. However, there might have been some pipetting errors that affect all replicates. So I repeat the experiment…
SecondLemon
- 151
- 4
2
votes
1 answer
Can confidence intervals or uncertainty intervals provide information on replication?
Let's say I'm looking at a forest plot from a meta-analysis. I notice that most of the width of the confidence intervals are fairly consistent and the point estimates are all on one side (showing a benefit). Is it possible to use these trends to…
Dylan A
- 31
- 5
2
votes
2 answers
Biological and technical replicates for statistical analysis in cellular biology
These are questions regarding basic statistics/reporting in biology. I have already read a couple of articles on this subject, but couldn't find a clear answer applying to my research.
I have the following scenario:
I have 3 independent cell…
Dunn Wallis
- 23
- 1
- 4
2
votes
2 answers
Why is the power of studies that only report significant effects not always 100%?
As I was reading the following passage of this blog, which defines R-(replicability-)indices:
To correct for the inflation in power, the R-Index uses the inflation
rate. For example, if all studies are significant and average power is
75%, the…
z8080
- 1,598
- 1
- 19
- 38
2
votes
1 answer
How to identify studies that should be replicated?
In psychology voting on which studies should be replicated is established on a website. The ReplicationWiki (that I founded) offers a voting option for studies in economics and related fields, but it is not yet used much. I already saw a couple of…
Jan Höffler
- 171
- 1
- 6