Is there any procedure to test the significance of meta-analytic estimate of correlation based on a set of sample-correlations?

Question

Meta-analysis generates an estimate of an overall correlation. How could we examine that it is a significant estimate.

Answer 1

In terms of statistical significance, you examine the confidence interval of the pooled estimate. But validity is a much wider concept here, and there are a lot of issues that need to be addressed in order for a meta-analytic result to be credible, valid and unbiased. There are tons of discussing these issues, here is a good one for example, and a good meta-analysis paper should itself point out most potential limitations its results.

In short (very!), some of the most important issues to examine are (in no particular order):

• Heterogeneity: if results from individual studies are very disparate (this can be seen in the forest plot already, and formally assessed with the I^2 statistic) the pooled estimate loses in importance because individual studies may not be measuring the same thing. As a rule of thumb, heterogeneity should be explored (e.g using meta-regression or subgroup analyses) but that's not always possible.
• Publication bias: this can be explored with a funnel plot or the Begg or Egger statistical tests. If significant, then the individual studies may not reflect the whole truth and the pooled estimate may be biased.
• Literature search and inclusion/exclusion criteria: if there is any evidence that important studies may have been missed, then there is potential for bias in the pooled estimate. Searching multiple literature databases is important, and having clear and relevant inclusion/exclusion criteria is crucial. The latter is also related to having a clear, specific and answerable question that the meta-analysis attempts to answer; it is no good trying to analyze apples and oranges together.
• Quality of individual studies, or, as goes the motto, "garbage in, garbage out". Of course that's a whole other issue in itself. For many study types (e.g. Randomized Controlled Trials) there are also tools to formally assess their quality. Some people also suggest using study quality to adjust the weights of individual studies, in deriving the pooled estimate (i.e. quality-adjusted meta-analysis) but that's controversial; sensitivity analyses are very helpful in this regard, and help illustrate the stability (or fragility) of the result.
• Appropriate statistic analyses: this is mostly about fixed-effects and random-effects estimates. Both should be reported, especially when heterogeneity is present. Also (this is fairly basic but there are many such examples), different study types and especially studies reporting different effect measures should not be analyzed together but separately.

As should be obvious, assessing the validity of a meta-analysis requires some experience, both in terms of meta-analytic methods and in terms of the particular field of science.

Hope this is helpful!