and Interpretation of Clinical Trials
LIMITATIONS OF META-ANALYSES
The assumption that a meta-analysis routinely represents the final and accurate viewpoint in an area of research is not warranted.
A meta-analysis combines similar trials in order to obtain a larger
number of patients to improve the evaluation of whether statistically reliable
differences exist between comparison groups.
Meta-analyses are by no means perfect. On some occasions, a large
clinical trial has subsequently been performed evaluating the same clinical
question with an outcome quite different from the initial meta-analysis.
Discrepancies between meta-analyses and
subsequent large randomized clinical trials are documented in literature1.
The conclusions made by the authors of a
meta-analysis are subject to the same potential for bias as the smallest of
clinical studies. The authors of the meta-analysis must assess the
limitations of their analysis and decide what conclusions to state. In addition,
they need to determine how broadly their conclusions can be applied and to what
patient groups. Conservative conclusions derived
directly from the data with a realistic assessment of the limitations of the
study are optimal, but by no means universal. A meta-analysis is
particularly subject to biased conclusions when it is created by advocates of a
controversial opinion regarding the same topic the meta-analysis is addressing.
(See critique of oat bran meta-analysis, critique
of ALLHAT meta-analysis, and Beware
of Meta-Analyses Bearing False Gifts for
Similarly, a meta-analysis written by employees or representatives of a pharmaceutical company will have an inherent and expected favorable bias towards the product of that company. This type of meta-analysis will always be an advocacy meta-analysis.
A very large randomized clinical trial is the most reliable way of obtaining reproducible results. This means that if the same trial protocol was repeated with a similar patient population using a sufficient number of patients, the same trial results would be expected to occur. However, even a very large trial does not guarantee that the specific treatment protocol being studied has been constructed optimally or appropriate conclusions have been formulated. (See Tale of Two Large Trials.)
The more similar the trials are that are being added together, the more likely the meta-analysis will result in valid conclusions. The addition of study protocols that are significantly different from one another makes a meta-analysis less reliable.
Since a meta-analysis is a summation of trials, it is only as good as the trials that are combined in the meta-analysis. If a very large trial is poorly done and is part of a meta-analysis, the results of the meta-analysis can be adversely impacted by that trial.
meta-analysis has a number of areas with the potential for bias (which is
usually unintentional). The potential areas of bias in a meta-analysis
1. Inclusion/exclusion criteria used to select the studies for the meta-analysis.
2. Methods used to perform the meta-analysis.
3. The conclusions which are reached.
4. Statements by the authors regarding the reliability of the results of their meta-analysis.
5. Declarations of broad applicability for the conclusions of a particular meta-analysis.
Meta-analyses can be quite useful and beneficial for the analysis of similar trials. However, the assumption that every meta-analysis represents the final and accurate viewpoint on an area of research is unwarranted.
There is a type of meta-analysis called network meta-analysis that is more subject to erroneous conclusions than a routine meta-analysis. A network meta-analysis adds an additional variable to a meta-analysis. Rather than simply summing up trials that have evaluated the same treatment compared to placebo (or compared to an identical medication), different treatments are compared by inference. ( If A is better than B, and B is equal to C, then A is better than C.)
problem with network analysis in regards to a meta-analysis is that a network
meta-analysis tends only to be valid for very similar studies. Since
network meta-analysis combines studies with a higher degree of variability,
there is even more potential for combining studies that are not adequately
similar. The quality of some recent network analyses in the hypertension
literature highlights the problems of this type of analysis.
1. Discrepancies Between Meta-Analyses and Subsequent Large Randomized, Controlled Trials. LeLorier J, Gregoire G, et al. NEJM, 1997; 337:536-42.
copyright 2005 www.improvingmedicalstatistics.com