You so often come across in the press various studies that conclude directionally opposite results. Those can be related to the testing of a new prescription drug or the merit of a specific nutrient or anything else for that matter.
When two such studies arrive at conflicting results how can you tell which one of the two is closest to the truth?
The meta analysis literature is relevant to your question. Using meta-analytic techniques you could generate an estimate of the effect of interest pooled across studies. Such techniques often weight studies in terms of their sample size.
Within the meta analysis context researchers talk about fixed effect and random effect models (see Hunter and Schmidt, 2002). A fixed effect model assumes that all studies are estimating the same population effect. A random-effects model assumes that studies differ in the population effect that is being estimated. A random-effects model is typically more appropriate.
As more studies accumulate looking at a particular relationship, more sophisticated approaches become possible. For example, you can code studies in terms of various properties, such as perceived quality, and then examine empirically whether the effect size varies with these study characteristics. Beyond quality there may be some theoretically relevant differences between the studies which would moderate the relationship (e.g., characteristic of the sample, dosage levels, etc.).
In general, I tend to trust studies with:
But that said you need to keep random sampling and theoretically meaningful differences between studies as a plausible explanation of conflicting study findings.
I would hold off on considering meta-analysis until you've scrutinized sources if potential bias or variation in the target populations. If these are studies of treatment effects, was treatment randomly assigned? Were there deviations from the protocol? Was there noncompliance? Is there missing outcome data? Were the samples drawn from the same frame? Was there refusal to participate? Implementation errors? Were standard errors computed correctly, accounting for clustering and robust to various parametric assumptions? Only after you have answered these questions do I think meta-analysis issues start to enter the picture. It must be rare that for any two studies meta-analysis is appropriate, unless you are willing to make sone heroic assumptions.
I think Jeromy's answer is sufficient if you are examining two experimental studies or an actual meta-analysis. But often times we are faced with examining two non-experimental studies, and are tasked with assessing the validity of those two disparate findings.
As Cyrus's grocery list of questions suggests, the topic itself is not amenable to short response, and whole books are in essence aimed to address such a question. For anyone interested in conducting research on non-experimental data, I would highly suggest you read
Experimental and quasi-experimental designs for generalized causal inference by William R. Shadish, Thomas D. Cook, Donald Thomas Campbell (Also I have heard that the older versions of this text are just as good).
Several items Jeromy referred to (bigger sample sizes, and greater methodological rigour), and everything that Cyrus mentions would be considered what Campbell and Cook refer to as "Internal Validity". These include aspects of the research design and the statistical methods used to assess the relationship between X and Y. In particular as critics we are concerned about aspects of either that could bias the results, and diminish the reliability of the findings. As this is a forum devoted to statistical analysis, much of the answers are centered around statistical methods to ensure unbiased estimates of whatever relationship you are assessing. But their are other aspects of the research design unrelated to statistical analysis that diminish the validity of the findings no matter what rigourous lengths one goes to in their statistical analysis (such as Cyrus's mention of several aspects of experiment fidelity can be addressed but not solved with statistical methods, and if they occur will always diminish the validity of the studies results). There are many other aspects of internal validity that become crucial to assess in comparing results of non-experimental studies that are not mentioned here, and aspects of research designs that can distinguish reliability of findings. I don't think it is quite appropriate to go into too much detail here, but I would often take the results of a quasi-experimental study (such as an interrupted time series or a matched case-control) more seriously than I would a study that is not quasi experimental, regardless of the other aspects Jeromy or Cyrus mentioned (of course within some reason).
Campbell and Cook also refer to the "external validity" of studies. This aspect of research design is often much smaller in scope, and does not deserve as much attention as internal validity. External validity essentially deals with the generalizability of the findings, and I would say laymen can often assess external validity reasonably well as long as they are familiar with the subject. Long story short read Shadish's, Cook's and Campbell's book.
