Subgroup Analysis  Potential
Limitations
Particularly
vulnerable to error, is the post hoc analysis of trial data
when a number is derived retrospectively from trial data and
is then said to separate the responders from the
nonresponders.
An analysis of the CARE study data^{3} by prominent
investigators was said to indicate that treatment of LDL cholesterol to a level
below 125 mg/dL was not helpful in patients with coronary disease (blocked
arteries of the heart). This was later shown to be erroneous. Similarly,
a post hoc subgroup data analysis concerning which
patients with a cardiomyopathy (weak heart muscle)
benefit from an implantable defibrillator led to erroneous
conclusions by the Medicare* administration. 

Inappropriate
subgroup analysis can lead to ludicrous results
Dr.
Peter Sleight
and colleagues have written insightful
comments in regards to the limitations of subgroup analysis.^{1,2}
"print
format"
Dr. Sleight and the ISIS2 trial investigators
performed a subgroup analysis of patients in the ISIS2 trial by astrological
sign to show the potential limitations in reliability of subgroup
analysis. details
This subgroup analysis suggested that the treatment was quite effective and statistically significant for all patients except those born under the sign of Gemini or
Libra.
The difference in outcome with respect to astrologic sign was naturally an artifact and would not be
reproducible in subsequent studies which was the point of their
analysis.
Validity
of subgroup analysis:
The view of this website is that subgroup analysis can be
useful, but the validity tends to be inversely proportional to the number of
subgroups which are analyzed. A study is not immune to an incorrect subgroup analysis outcome simply because the subgroups
have been prespecified. This is particularly the case if there are a large number of prespecified subgroup analyses.
(If 20 subgroup analyses are prespecified, then it is expected that one of these
subgroup analyses may show a false result for a P=.05 probability relationship.) Part
of the benefit of a prespecified subgroup analysis is that there are necessarily
fewer such analyses than the almost unlimited number of ways to subdivide the
data in a post hoc analysis after the trial results have been obtained.
What is the reliability of a finding for a small subgroup of
a trial who unexpectedly have a different outcome from the rest of the group?
In
particular, if a given therapy has a highly significant and strongly
beneficial effect for the group as a whole, a subgroup analysis that
results in an unexpected finding that certain subgroups do not have
benefit is frequently incorrect.
In
fact, it is more likely that the unexpected subgroup finding which runs
counter to the group finding, is simply not valid.
As pointed out by Dr Sleight, it is more reliable to assume that the subgroup actually
had the same outcome as the overall group.
Unexpected
results from a
subgroup analysis tend to be more useful as a potential starting place for a subsequent
clinical trial, rather than being viewed as a definitive result.
For an excellent look at potential limitations
of subgroup analysis the following two articles are recommended:
1. Debate: Subgroup analyses in clinical trials:
fun to look at but don’t believe them! Peter
Sleight. Current Control Trial
Cardiovasc Med. 2000 1(1): 2527.
2. ISIS2 (Second International Study of Infarct Survival). Lancet 1988: ii: 349360
(pages of interest 356357)


See
right column of this page for additional examples of subgroup analysis
leading to incorrect conclusions.
Reference
cited in right column of this page:
3. Sacks FM, Moyé LA, Davis BR, Cole TG, Rouleau JL, Nash DT, Pfeffer MA,
Braunwald E. Relationship between plasma LDL concentrations during treatment
with pravastatin and recurrent coronary events in the Cholesterol and Recurrent
Events trial.
Circulation. 1998 97:144652.
back
