56

Bottom line, the more I learn about statistics, the less I trust published papers in my field; I simply believe that researchers are not doing their statistics well enough.

I'm a layman, so to speak. I'm trained in biology but I have no formal education in statistics or mathematics. I enjoy R and often make an effort to read (and understand...) some of the theoretical foundations of the methods that I apply when doing research. It wouldn't surprise me if the majority of people doing analyses today are actually not formally trained. I've published around 20 original papers, some of which have been accepted by recognized journals and statisticians have frequently been involved in the review-process. My analyses commonly include survival analysis, linear regression, logistic regression, mixed models. Never ever has a reviewer asked about model assumptions, fit or evaluation.

Thus, I never really bothered too much about model assumptions, fit and evaluation. I start with a hypothesis, execute the regression and then present the results. In some instances I made an effort to evaluate these things, but I always ended up with "well it didn't fulfill all assumptions, but I trust the results ("subject matter knowledge") and they are plausible, so it's fine" and when consulting a statistician they always seemed to agree.

Now, I've spoken to other statisticians and non-statisticians (chemists, physicians and biologists) who perform analyses themselves; it seems that people don't really bother too much about all these assumptions and formal evaluations. But here on CV, there is an abundance of people asking about residuals, model fit, ways to evaluate it, eigenvalues, vectors and the list goes on. Let me put it this way, when lme4 warns about large eigenvalues, I really doubt that many of its users care to address that...

Is it worth the extra effort? Is it not likely that the majority of all published results do not respect these assumptions and perhaps have not even assessed them? This is probably a growing issue since databases grow larger every day and there is a notion that the bigger the data, the less important is the assumptions and evaluations.

I could be absolutely wrong, but this is how I have perceived this.

Update: Citation borrowed from StasK (below): http://www.nature.com/news/science-joins-push-to-screen-statistics-in-papers-1.15509

Adam Robinsson
  • 2,083
  • 3
  • 19
  • 39
  • 35
    I would like to warn that closing this question would shut off an important opportunity to discuss how statistics is being used in the "real world" outside the walls of statistics departments. CV is one of few venues were the real world people and statisticians mingle, and keeping the lines of communication open is an important, albeit implicit, mission of CV. I entirely agree with those who clicked "close" as this is very broad and opinion-based, but I still hope we can keep this discussion. – StasK May 07 '15 at 15:23
  • 5
    Wouldn't the fact that many publications, researchers or even entire fields treat assumptions casually suggest we don't care *enough*? It's also possible that statisticians and textbooks exaggerate their importance but surely their popularity with practitioners and lay people can't be the yardstick for that. Also conventions and standards vary quite a bit. Some disciplines probably care more about this than the ones you are familiar with, whereas others probably do not care much about things that are very important to get publish in your field. – Gala May 07 '15 at 15:48
  • 6
    Having written a comment paper criticizing a previous study where model assumptions were knowingly violated and incorrect conclusions drawn, my advice would be not to trust the results and be a self-critical as you can. – Dikran Marsupial May 07 '15 at 16:39
  • 1
    Thank you – StasK, Gala, Dikran – for the comments. I asked this because it is an important issue. I'm not reducing the importance of statistical expertise – on the contrary: I've learnt that any statistical calculation done by a non-statistician is to a varying degree uncertain and shaky. However, one must realize that the availability of software (such as R) and tutorials along with the shortage of statisticians is propelling non-statisticians to do the math... There is a obvious risk that the lack of fundamental understanding leads to fundamentally wrong results. I've made a few myself. – Adam Robinsson May 07 '15 at 21:08
  • 7
    "The bigger the data, the less important the assumptions" does not hold with the actual growth of data: What grows is (mainly) the number of dependent observations / features due to advancements in measurement techniques. In contrast, the number of independent observations, i.e. experimental units or probands, remains rather bounded (as populations do not increase by technical advancements...). Unfortunately, only big data w.r.t. independence make the assumptions (often, not always) less important due to the central limit theorem. – Horst Grünbusch May 08 '15 at 00:29
  • @Gala: I'd say that if statisticians care, but practicioners don't it may also be that the practicioners need a "2nd line of defense". An assumption guaranteeing optimality that is never met in practice may be much less worth from a practical point of view than a lesser but practically sensible/achievable assumption that allows to judge that the model will perform sufficiently well. – cbeleites unhappy with SX May 10 '15 at 00:40
  • 3
    @AdamRobinsson: I like the conclusion you posted at the top of your question. I would be even more wary of papers published by Big Names or backed by Big Institutions. When I see papers with 15 senior authors from respected institutions in top-tier journals, my first instinct is actually to scrutinize the paper *even more*, because there is a possibility that the paper was published because of the authors'/institutions' clout. These papers also tend to have far-reaching policy implications. One example from my field (public health) were the male circumcision for HIV prevention studies (cont.) – Marquis de Carabas May 10 '15 at 03:10
  • One frequently cited study, published in Lancet (http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2807%2960313-4/abstract) found that HIV incidence was nearly 2x higher among uncircumcised men than circumcised men. It turns out, the number of new cases among the treated was 21, compared to 45 among the control, out of 4,996 men in the sample. It's a "large" sample, so of course we will find statistical significance. But is it a clinical significance? What about behavioral consequences of pushing this policy forward (which we are, by the way)? Thus my point that we should be – Marquis de Carabas May 10 '15 at 03:22
  • wary of what we read--even in refereed journals--because science is never entirely neutral, and people find a way to throw their weight around. Even if the review was double blind, sometimes the field is so narrow or the study in question so widely known that the referee could tell who the authors of the manuscript are! – Marquis de Carabas May 10 '15 at 03:24

9 Answers9

30

Well, yes, assumptions matter -- if they didn't matter at all, we wouldn't need to make them, would we?

The question is how much they matter -- this varies across procedures and assumptions and what you want to claim about your results (and also how tolerant your audience is of approximation -- even inaccuracy -- in such claims).

So for an example of a situation where an assumption is critical, consider the normality assumption in an F-test of variances; even fairly modest changes in distribution may have fairly dramatic effects on the properties (actual significance level and power) of the procedure. If you claim you're carrying out a test at the 5% level when it's really at the 28% level, you're in some sense doing the same kind of thing as lying about how you conducted your experiments. If you don't think such statistical issues are important, make arguments that don't rely on them. On the other hand, if you want to use the statistical information as support, you can't go about misrepresenting that support.

In other cases, particular assumptions may be much less critical. If you're estimating the coefficient in a linear regression and you don't care if it's statistically significant and you don't care about efficiency, well, it doesn't necessarily matter if the homoskedasticity assumption holds. But if you want to say it's statistically significant, or show a confidence interval, yes, it certainly can matter.

Glen_b
  • 257,508
  • 32
  • 553
  • 939
  • 2
    Glen_b's comment is well said from the statistical approach to the importance of assumptions. I think it also should be noted though, that for the purposes of publication the checking of assumptions is a slightly different matter, in that the violation of the assumptions only matter as much as the reviewers or editors care for them to matter. As an example the theory behind the result may provide sufficient merit for publication, where the hope is that issues with the analysis may be resolved by future publications. – Jonathan Lisic May 07 '15 at 13:06
  • Well, even if reviewers check the paper, the responsibility with errors stay with the authors. So in own interest, you should check ... – kjetil b halvorsen May 07 '15 at 13:09
  • Indeed, the responsibility will always stay with the authors. But authors these days are pushed by inappropriate drivers that forces them to publish, not seldom in a quick and dirty fashion. I would like to see a review-process that mandates declaration of the most critical statistical assumptions. Today it is taken for granted that the statistical analysis has been done according to the book, but that I believe is more rare than common. – Adam Robinsson May 07 '15 at 21:13
  • 3
    +1 "The question is how much they matter"--that essentially puts the entire issue in a nutshell. I should also point out that in the application of statistical inference, it is not possible to know the extent to which certain statistical assumptions are violated. We can only postulate or consider the robustness of the analysis if they are, and this is an essential but often-overlooked aspect of statistical practice. – heropup May 10 '15 at 01:11
27

I am trained as a statistician not as a biologist or medical doctor. But I do quite a bit of medical research (working with biologists and medical doctors), as part of my research I have learned quite a bit about treatment of several different diseases. Does this mean that if a friend asks me about a disease that I have researched that I can just write them a prescription for a medication that I know is commonly used for that particular disease? If I were to do this (I don't), then in many cases it would probably work out OK (since a medical doctor would just have prescribed the same medication), but there is always a possibility that they have an allergy/drug interaction/other that a doctor would know to ask about, that I do not and end up causing much more harm than good.

If you are doing statistics without understanding what you are assuming and what could go wrong (or consulting with a statistician along the way that will look for these things) then you are practicing statistical malpractice. Most of the time it will probably be OK, but what about the occasion where an important assumption does not hold, but you just ignore it?

I work with some doctors who are reasonably statistically competent and can do much of their own analysis, but they will still run it past me. Often I confirm that they did the correct thing and that they can do the analysis themselves (and they are generally grateful for the confirmation) but occasionally they will be doing something more complex and when I mention a better approach they will usually turn the analysis over to me or my team, or at least bring me in for a more active role.

So my answer to your title question is "No" we are not exaggerating, rather we should be stressing some things more so that laymen will be more likely to at least double check their procedures/results with a statistician.

Edit

This is an addition based on Adam's comment below (will be a bit long for another comment).

Adam, Thanks for your comment. The short answer is "I don't know". I think that progress is being made in improving the statistical quality of articles, but things have moved so quickly in many different ways that it will take a while to catch up and guarentee the quality. Part of the solution is focusing on the assumptions and the consequences of the violations in intro stats courses. This is more likely to happen when the classes are taught by statisticians, but needs to happen in all classes.

Some journals are doing better, but I would like to see a specific statistician reviewer become the standard. There was an article a few years back (sorry don't have the reference handy, but it was in either JAMA or the New England Journal of Medicine) that showed a higher probability of being published (though not as big a difference as it should be) in JAMA or NEJM if a biostatistican or epidemiologist was one of the co-authors.

An interesting article that came out recently is: http://www.nature.com/news/statistics-p-values-are-just-the-tip-of-the-iceberg-1.17412 which discusses some of the same issues.

Greg Snow
  • 46,563
  • 2
  • 90
  • 159
  • 1
    I share your point of view, Greg. I think your reply explains it clearly. But I'd like to quote you: "[...] then you are practicing statistical malpractice. Most of the time it will probably be OK". There is a risk that this notion is spreading and that people perceive it as: anyone can do statistics (which is fundamentally wrong if you ask me, who scratched a bit on the surface of statistics). The question is, how do we assure that published work is correct in terms of the statistical analyses? Because I'm starting to wonder how many articles out there that did not pass a statistician... – Adam Robinsson May 09 '15 at 21:20
  • @AdamRobinsson, see my addition above. – Greg Snow May 15 '15 at 15:29
  • This is rather elitist. I agree that professional statisticians might have the competence and expertise to perform statistical analyses, but so too might scientists. Indeed, some of the groundbreaking advances in statistics came from scientists (e.g Fisher & Jeffrey) practicing statistics in the real world. – innisfree Apr 20 '19 at 09:18
  • It’s also weird to say that what constitutes statistical malpractice depends not only the the quality or results of the statistical analysis but on subjective judgements about the strength of the analysts understanding of statistics – innisfree Apr 20 '19 at 09:22
  • @innisfree, I don't understand your comments, or maybe you don't understand my position. I am not saying that only statisticians can look at the assumptions/conditions, just that they are important and either a statistician should be consulted, or the non-statistician scientist(s) should learn enough statistics to understand the issues beyond just plugging the numbers into a formula/computer. Personally I would like to see a lot more statistics minors with expertise in medicine/engineering/etc., but also with a better understanding of the ideas behind statistics. – Greg Snow Apr 23 '19 at 15:42
18

While Glen_b gave a great answer, I would like to add a couple of cents to that.

One consideration is whether you really want to get the scientific truth, which would require polishing your results and figuring out all the details of whether your approach is defensible, vs. publishing in the "ah well, nobody checks these eigenvalues in my discipline anyway" mode. In other words, you'd have to ask your inner professional conscience whether you are doing the best job you could. Referring to the low statistical literacy and lax statistical practices in your discipline does not make a convincing argument. Reviewers are often at best half-helpful if they come from the same discipline with these lax standards, although some top outlets have explicit initiatives to bring statistical expertise into the review process.

But even if you are a cynical "publish-or-perish" salami slicer, the other consideration is basically the safety of your research reputation. If your model fails, and you don't know it, you are exposing yourself to the risk of rebuttal by those who can come and drive the ax into the cracks of the model checks with more refined instruments. Granted, the possibility of that appears to be low, as the science community, despite the nominal philosophical requirements of reputability and reproducibility, rarely engages in attempts to reproduce somebody else's research. (I was involved in writing a couple of papers that basically started with, "oh my God, did they really write that?", and offered a critique and a refinement of a peer-reviewed published semi-statistical approach.) However, the failures of statistical analyses, when exposed, often make big and unpleasant splashes.

Community
  • 1
StasK
  • 29,235
  • 2
  • 80
  • 165
  • I really like the approach: professional conscience, and I believe many do have conscience but still lack the knowledge; but it won't stop them from presenting data as if it was perfectly done. Interestingly, you are citing a Science article that spells it out clearly: "[...] widespread concern that basic mistakes in data analysis are contributing to the irreproducibility of many published research findings." I really think that us laymen, we do not really respect the difficulty of statistical methods and it should be a growing problem as explained above. – Adam Robinsson May 07 '15 at 21:26
  • 1
    In my current research group, we are 15 researchers (biologists, physicians) and some of them are really productive but none is a statistician. All junior researchers have installed R or SAS and carries out calculations, typically after reading just few tutorials on the Internet. This is a big issue. – Adam Robinsson May 07 '15 at 21:32
  • 4
    @AdamRobinsson, the mere fact that you asked about this on CV does speak about your conscience. One more Nature link for you to peruse, on a somewhat related issue of scientific computing (and lack of implementation of the best practices in software development by scientists writing their own C/Fortran/R code) -- http://www.nature.com/news/2010/101013/full/467775a.html. – StasK May 08 '15 at 20:34
  • 3
    _as the science community, despite the nominal philosophical requirements of reputability and reproducibility, rarely engages in attempts to reproduce somebody else's research_ I completely agree with that statement. – Robert Smith May 09 '15 at 16:52
  • http://software-carpentry.org, "a volunteer non-profit organization dedicated to teaching basic computing skills to researchers", have pretty good tutorials and seminars. – denis Mar 01 '16 at 10:58
9

The nature of violations of assumptions can be an important clue for future research. For example, a violation of the proportional-hazards assumption in Cox survival analysis might be due to a variable with a large effect on short-term survival but little effect in the longer term. That's the type of unexpected but potentially important information you can get by examining the validity of your assumptions in a statistical test.

So you do yourself, not just the literature, a potential disservice if you don't test the underlying assumptions. As high-quality journals start requiring more sophisticated statistical review you will find yourself called on more frequently to do so. You don't want to be in a position where a test required by a statistical reviewer undermines what you thought had been a key point of your paper.

EdM
  • 57,766
  • 7
  • 66
  • 187
9

I'll answer from an intermediate perspective. I'm not a statistician, I'm chemist. However, I've spent the last 10 years specializing in chemometrics = statistical data analysis for chemistry-related data.

I simply believe that researchers are not doing their statistics well enough.

That is probably the case.

Short version:

Now about the assumptions. IMHO the situation here is far too heterogeneous to deal with it in one statement. Understanding of both what exactly the assumption is needed for and in which way it is likely to be violated by the application is necessary in order to judge whether the violation is harmless or critical. And this needs both the statistics as well as the application knowledge.
As a practitioner facing unachievable assumptions, however, I need something else as well: I'd like to have a "2nd line of defense" that e.g. allows me to judge whether the violation is actually causing trouble or whether it is harmless.

Long version:

  • From a practical point of view, some typical assumptions are almost never met. Sometimes I can formulate sensible assumptions about the data, but often then the problems become so complicated from a statistical point of view that solutions are not yet known. By now I believe that doing science means that you'll hit the borders of what is known likely not only in your particular discipline but maybe also in other disciplines (here: applied statistics).

  • There are other situations where certain violations are known to be usually harmless - e.g. the multivariate normality with equal covariance for LDA is needed to show that LDA is optimal, but it is well known that the projection follows a heuristic that often performs well also if the assumption is not met. And which violations are likely to cause trouble: It is also known that heavy tails in the distribution lead to problems with LDA in practice.
    Unfortunately, such knowledge rarely makes it into the condensed writing of a paper, so the reader has no clue whether the authors did decide for their model after well considering the properties of the application as well as of the model or whether they just picked whatever model they came across.

  • Sometimes practical approaches (heuristics) evolve that turn out to be very useful from a practical point of view, even if it takes decades until their statistical properties are understood (I'm thinking of PLS).

  • The other thing that happens (and should happen more) is that the possible consequences of the violation can be monitored (measured), which allows to decide whether there is a problem or not. For the application, maybe I don't care whether my model optimal as long as it is sufficiently good.
    In chemometrics, we have a rather strong focus on prediction. And this offers a very nice escape in case the modeling assumptions are not met: regardless of those assumptions, we can measure whether the model does work well. From a practicioner's point of view, I'd say that you are allowed to do whatever you like during your modeling if you do and report an honest state-of-the-art validation.
    For chemometric analysis of spectroscopic data, we're at a point where we don't look at residuals because we know that the models are easily overfit. Instead we look at test data performance (and possibly the difference to training data predicitve performance).

  • There are other situations where while we're not able to predict precisely how much violation of which assumption leads to a breakdown of the model, but we're able to measure consequences of serious violations of the assumption rather directly.
    Next example: the study data I typically deal with is orders of magnitude below the sample sizes that the statistical rules-of-thumb recommend for cases per variate (in order to guarantee stable estimates). But the statistics books typically don't care much about what to do in practice if this assumption cannot be met. Nor how to measure whether you actually are in trouble in this respect. But: such questions are treated in the more applied disciplines. Turns out, it is often quite easy to directly measure model stability or at least whether your predictions are unstable (read here on CV on resampling validation and model stability). And there are ways to stabilize unstable models (e.g. bagging).

  • As an example of the "2nd line of defense" consider resampling validation. The usual and strongest assumption is that all surrogate models are equivalent to a model trained on the whole data set. If this assumption is violated, we get the well-known pessimistic bias. The 2nd line is that at least the surrogate models are equivalent to each other, so we can pool the test results.

Last but not least, I'd like to encourage the "customer scientists" and the statisticians to speak more with each other. The statistical data analysis IMHO is not something that can be done in a one-way fashion. At some point, each side will need to acquire some knowledge of the other side. I sometimes help "translating" between statisticians and chemists and biologists. A statistician can know that the model needs regularization. But to choose, say, between LASSO and a ridge, they need to know properties of the data that only the chemist, physicist or biologist can know.

cbeleites unhappy with SX
  • 34,156
  • 3
  • 67
  • 133
6

Given that CV is populated by statisticians and people who are curious, if not competent, about statistics, I am not surprised about all the answers emphasizing the need to understand the assumptions. I also agree with these answers in principle.

However, when taking account the pressure to publish and the low standard for statistical integrity currently, I have to say that these answers are quite naive. We can tell people what they should do all day long (i.e. check your assumption), but what they will do depends solely on the institutional incentives. The OP himself states that he manages to publish 20 articles without understanding the model's assumption. Given my own experience, I don't find this hard to believe.

Thus I want to play the devil's advocate, directly answering OP's question. This is by no means an answer that promotes "good practice," but it is one that reflects how things are practised with a hint of satire.

Is it worth the extra effort?

No, if the goal is to publish, it's not worth it to spend all the time understanding the model. Just follow the prevalent model in the literature. That way, 1) your paper will pass reviews more easily, and 2) the risk of being exposed for "statistical incompetence" is small, because exposing you means exposing the entire field, including many senior people.

Is it not likely that the majority of all published results do not respect these assumptions and perhaps have not even assessed them? This is probably a growing issue since databases grow larger every day and there is a notion that the bigger the data, the less important is the assumptions and evaluations.

Yes, it's likely that most published results are not true. The more involved I am in actual research, the more I think it is likely.

Heisenberg
  • 4,239
  • 3
  • 23
  • 54
  • 1
    I think your having some very good points that are not mentioned above; namely the pressure to publish and the journals pressure to find articles to fill the journal. It is a nightmare for editors to not have articles to publish, and researchers must publish for their mere existence. However, from a methodological point of view, the more I get involved in research, the more I believe that most published findings are wrong/less correct/imperfect to a varying degree (from a statistical point of view). – Adam Robinsson May 09 '15 at 21:28
4

The short answer is "no." Statistical methods were developed under sets of assumptions that should be met for the results to be valid. It stands to reason, then, that if the assumptions were not met, the results may not be valid. Of course, some estimates may still be robust despite violations of model assumptions. For example, multinomial logit appears to perform well despite violations of the IIA assumption (see Kropko's [2011] dissertation in the reference below).

As scientists, we have an obligation to ensure that the results we put out there are valid, even if the people in the field don't care whether assumptions have been met. This is because science is built on the assumption that scientists will do things the right way in their pursuit of the facts. We trust our colleagues to check their work before sending it out to the journals. We trust the referees to competently review a manuscript before it gets published. We assume that both the researchers and the referees know what they are doing, so that the results in papers that are published in peer-reviewed journals can be trusted. We know this is not always true in the real world based on the sheer amount of articles in the literature where you end up shaking your head and rolling your eyes at the obviously cherry-picked results in respectable journals ("Jama published this paper?!").

So no, the importance cannot be overstated, especially since people trust you--the expert--to have done your due diligence. The least you can do is talk about these violations in the "limitations" section of your paper to help people interpret the validity of your results.

Reference

Kropko, J. 2011. New Approaches to Discrete Choice and Time-Series Cross-Section Methodology for Political Research (dissertation). UNC-Chapel Hill, Chapel Hill, NC.

Marquis de Carabas
  • 1,667
  • 19
  • 31
  • I agree with these valuable comments. But don't You think that the "may" in "may not be invalid" is the reason that people might not bother too much about them. I believe that the mere existence of a hypothesis may induce analysis bias that is relevant to this issue. – Adam Robinsson May 08 '15 at 14:30
  • TIL Kropko is a CV user. – Sycorax May 08 '15 at 14:48
  • @AdamRobinsson, I think people don't bother with them because they don't fully understand how or why the results may be invalid. A huge part of the problem lies in statistics education in an "applied" field. My own training is in an applied field. My econometrics-lite classes made no pretenses about being full econometrics classes and were called "advanced research methods" or something like that. Because there were no statistics prerequisites, profs would gloss over model assumptions in favor of spending more time on Stata commands and interpretation of results. – Marquis de Carabas May 08 '15 at 16:46
  • @AdamRobinsson, The first time I was in a graduate program, it was also for an applied field, but the classes were taught by biostatisticians. The biostatisticians taught in-depth about model assumptions and the various checks we are supposed to do, because some of the students in the class were biostatistics students. However, it was clear that the rest of the students who were in these classes didn't understand why we were checking those assumptions, because the profs didn't communicate the importance in a language that the students understood. – Marquis de Carabas May 08 '15 at 16:51
  • 1
    @marquisdecarabas, agreed. Not understanding them is a fundamental issue, obviously. However, I sometimes wonder if it wasn't due to the immense amount of time spent on data manipulation, that drains ones lust to take on the assumptions. – Adam Robinsson May 09 '15 at 21:24
  • Taking togeter "may be invalid" with the"obligation to ensure that the results we put out there are valid", I'd say that methods to measure whether the results/model are valid despite the assumptions being violated are needed. – cbeleites unhappy with SX May 10 '15 at 01:10
  • @user777: I took a dose of my own medicine and Googled "TIL" (I am not up-to-date on internet slang). I think you are implying or assuming I am Kropko with your comment. I am neither Kropko, nor am I one of his associates. I am merely a discrete choice methods enthusiast who came across his dissertation (then still a working paper) whilst working on my own dissertation. – Marquis de Carabas May 10 '15 at 03:27
  • @AdamRobinsson--Agree that data manipulation is exhausting. It must take about 80% of the "data" part of my studies. I also think we overlook it because it is invisible in the papers that we read, so by the time we're done fitting the models, we often don't think about doing the diagnostics or testing the assumptions. – Marquis de Carabas May 10 '15 at 03:31
2

If you need very advanced statistics it's most likely because your data is a mess, which is the case with most social sciences, not to mention psychology. In those fields where you have good data you need very little statistics. Physics is a very good example.

Consider this quote from Galileo on his famous gravitational acceleration experiment:

A piece of wooden moulding or scantling, about 12 cubits long, half a cubit wide, and three finger-breadths thick, was taken; on its edge was cut a channel a little more than one finger in breadth; having made this groove very straight, smooth, and polished, and having lined it with parchment, also as smooth and polished as possible, we rolled along it a hard, smooth, and very round bronze ball. Having placed this board in a sloping position, by raising one end some one or two cubits above the other, we rolled the ball, as I was just saying, along the channel, noting, in a manner presently to be described, the time required to make the descent. We repeated this experiment more than once in order to measure the time with an accuracy such that the deviation between two observations never exceeded one-tenth of a pulse-beat. Having performed this operation and having assured ourselves of its reliability, we now rolled the ball only one-quarter the length of the channel; and having measured the time of its descent, we found it precisely one-half of the former. Next we tried other distances, compared the time for the whole length with that for the half, or with that for two-thirds, or three-fourths, or indeed for any fraction; in such experiments, repeated a full hundred times, we always found that the spaces traversed were to each other as the squares of the times, and this was true for all inclinations of the plane, i.e., of the channel, along which we rolled the ball. We also observed that the times of descent, for various inclinations of the plane, bore to one another precisely that ratio which, as we shall see later, the Author had predicted and demonstrated for them.

For the measurement of time, we employed a large vessel of water placed in an elevated position; to the bottom of this vessel was soldered a pipe of small diameter giving a thin jet of water which we collected in a small glass during the time of each descent, whether for the whole length of the channel or for part of its length; the water thus collected was weighed, after each descent, on a very accurate balance; the differences and ratios of these weights gave us the differences and ratios of the times, and this with such accuracy that although the operation was repeated many, many times, there was no appreciable discrepancy in the results.

Note the highlighted by me text. This is what good data is. It comes from a well planned experiment based on a good theory. You don't need statistics to extract what you're interested in. There was no statistics at that time, neither there were computers. The outcome? A pretty fundamental relationship, which still holds, and can be tested at home by a 6th grader.

I stole the quote from this awesome page.

UPDATE: To @Silverfish comment, here's an example of statistics in experimental particle physics. Pretty basic, huh? Barely over MBA level. Note, how they love $\chi^2$ :) Take that, statisticians!

Aksakal
  • 55,939
  • 5
  • 90
  • 176
  • 2
    "In those fields where you have good data you need very little statistics. Physics is a very good example." I can see the point, but it's worth pointing out that particle physicists have both an abundance of data and have developed a whole range of advanced statistical techniques to analyse them. – Silverfish May 10 '15 at 03:19
  • @Silverfish, can you give me example of advanced statistics there? What I have seen didn't impress me much. An average econ PhD student will know more stats than world class particle physicist, in my opinion. What physicists are really good at is things like statistical mechanics, but that's very different from what we'd call "advanced stats". I totally agree with you that their datasets are mind boggling, they started big data before "big data" became an annoying buzz word. – Aksakal May 10 '15 at 03:34
  • 1
    They have quite an interesting take on "confidence interval" particularly for parameters which can't be negative, see e.g. http://en.wikipedia.org/wiki/CLs_upper_limits_%28particle_physics%29 – Silverfish May 10 '15 at 07:06
  • (I think it might be unfair to take Day 1 of an introductory course as representative of all the stats you need to be a particle physicist!) – Silverfish May 10 '15 at 07:09
  • @Aksakal, my personal understanding is that Bayesian methods do wonders when there is little model uncertainty, i.e., hard sciences where models are whatever the Schrodinger equation gives you, after some 5,000-dimensional integration. In economics, Bayesian approach will hardly survive given the focus of the empirical researchers on inference that is robust to potential major model misspecifications. (I am a physicist by an undergraduate degree, although have not done any physics in 20+ years, and a quantitative social scientist by the line of work now.) – StasK May 10 '15 at 07:12
  • @StasK, my take on this is that when you have good data, such as in physics, you can use anything, and it will work :) In the end Bayesian approach when applied to an infinite set of data will render the same result as frequentist. The information in the data will overwhelm your prior. It's in social sciences when you get very different results with different statistics, and job of a statistician becomes "interesting". Bayesian methods are increasingly popular in econometrics, e.g. Bayesian VAR. – Aksakal May 10 '15 at 22:23
  • It’s true that particle physics doesn’t typically use a rich set of tests or theory, but do remember that it features arguably the most careful frequentist analyses in history and a slew of (sometimes arcane but novel) methods e.g. CLs, Feldman-Cousins, Gross-Vitells – innisfree Apr 21 '19 at 07:34
  • @Aksakal how about Jeffreys-Lindley paradox? Your statement about agreement with infinite data isn’t true, unfortunately – innisfree Apr 21 '19 at 07:35
2

This question seems to be a case of professional integrity.

The problem seems to be that either: (a) there isn't enough critical assessment of statistical analysis by lay persons or (b) a case of common knowledge is insufficient to identify statistical error (like a Type 2 error)?

I know enough about my area of expertise to request an experts input when I am near the boundary of that expertise. I have seen people use things like the F-test (and R-squared in Excel) without sufficient knowledge.

In my experience, the education systems, in our eagerness to promote statistics, have over-simplified the tools and understated the risks / limits. Is this a common theme that others have experienced and would explain the situation?

MarkR
  • 146
  • 7