|
Subgroup Analysis - Potential
Limitations
|
Particularly
vulnerable to error, is the post hoc analysis of trial data
when a number is derived retrospectively from trial data and
is then said to separate the responders from the
nonresponders.
Years ago, this type of analysis of the CARE study data by prominent
investigators was said to indicate that treatment of initial LDL cholesterol
levels below 124 was not helpful in patients with coronary disease (blocked
arteries). Clearly, this was later shown to be erroneous.
Similarly,
a different post hoc subgroup data analysis concerning which
patients with a cardiomyopathy (weak heart muscle)
benefit from an implantable defibrillator led to erroneous
conclusions by the Medicare* administration.
* Centers
for Medicare and Medicaid Services (CMS) |
|
|
Inappropriate
subgroup analysis can lead to ludicrous results
Dr.
Peter Sleight
and colleagues have written insightful
comments in regards to the limitations of subgroup analysis.1,2
"print
format"
Dr. Sleight and the ISIS-2 trial investigators
performed a subgroup analysis of patients in the ISIS-2 trial by astrological
sign to show the potential limitations in reliability of subgroup
analysis. details
This subgroup analysis suggested that the treatment was quite effective and statistically significant for all patients except those born under the sign of Gemini or
Libra.
The difference in outcome with respect to astrologic sign was naturally an artifact and would not be
reproducible in subsequent studies which was the point of their
analysis.
Validity
of subgroup analysis:
The view of this website is that subgroup analysis can be
useful, but the validity tends to be inversely proportional to the number of
subgroups which are analyzed. A study is not immune to an incorrect subgroup analysis outcome simply because the subgroups
have been prespecified. This is particularly the case if there are a large number of prespecified subgroup analyses.
(If 20 subgroup analyses are prespecified, then it is expected that one of these
subgroup analyses may show a false result for a P=.05 probability relationship.) Part
of the benefit of a prespecified subgroup analysis is that there are necessarily
fewer such analyses than the almost unlimited number of ways to subdivide the
data in a post hoc analysis after the trial results have been obtained.
What is the reliability of a finding for a small subgroup of
a trial who unexpectedly have a different outcome from the rest of the group?
In
particular, if a given therapy has a highly significant and strongly
beneficial effect for the group as a whole, a subgroup analysis that
results in an unexpected finding that certain subgroups do not have
benefit is frequently incorrect.
In
fact, it is more likely that the unexpected subgroup finding which runs
counter to the group finding, is simply not valid.
As pointed out by Dr Sleight, it is more reliable to assume that the subgroup actually
had the same outcome as the overall group.
Unexpected
results from a
subgroup analysis tend to be more useful as a potential starting place for a subsequent
clinical trial, rather than being viewed as a definitive result.
For an excellent look at potential limitations
of subgroup analysis, the following articles are recommended:
1. Debate: Subgroup analyses in clinical trials:
fun to look at- but don’t believe them! Peter
Sleight. Current Control Trial Cardiovasc Med. 2000 1(1): 25-27.
2. ISIS-2 (Second International Study of Infarct Survival). Lancet 1988: ii: 349-360
(pages of interest 356-357)
back
|
