Improving Medical Statistics    www.improvingmedicalstatistics.com
and Interpretation of Clinical Trials
_____________________________________________________

A study in the New England Journal of Medicine where the conclusions run exactly the opposite of what the data suggests.

Some studies have multiple errors.   This study concerned cardiac patients undergoing outpatient elective diagnostic cardiac catheterization.1

In this one study, the authors miscounted their trial's primary endpoints, included patients that did not belong in the trial, and failed to appreciate the limitations of the small sample size in regards to reliably proving a null hypothesis (that no difference exists).  Finally, the authors failed to be alarmed at the implications of their miscounted data. The authors incorrectly said their study demonstrated that outpatient heart catheterization was safe despite a heart attack occurring in this study in 1 out of 64 patients in this study, which was considerably higher than the expected rate of 1 out of 1000 patients2. Though outpatient cardiac catheterization is safe, the miscounted data in this poorly executed study was not reassuring.

Analogy of incorrectly declaring a procedure safe:

An example that would parallel this study would be a trial to evaluate whether drawing blood with needles that have never been sterilized is safe. Assume that 2 out of 100 patients develop an infection and die with the use of the needles that have not been sterilized. Assume 0 out of 100 patients die using the fully sterilized needles. This difference would not be statistically significant. However, it would be a Type II statistical error to state that this study demonstrates that unsterilized needles are safe because statistical significance has not been reached in this small sample size.

A Type II statistical error can easily occur when the numbers being studied are too small to reliably determine that no significant difference exists between the groups being studied.  A significant difference, at times may exist, but not reach statistical significance. Larger numbers of patients are often required to prove that no difference exists, compared to the smaller numbers that at times demonstrate that a difference is present. A Type II statistical error occurs if there is truly a difference present, but the study is incorrectly said to demonstrate that no difference exists between the groups.

The details of this flawed study:

This 1988 study1 was an attempt to assess whether outpatient cardiac catheterization was a safe approach. Elective diagnostic heart catheterization in stable patients is usually associated with a major complication (stroke, MI. or death) that is quite low, 1 out of 1000 patients. (The rate of complication is higher for unstable patients threatening a heart attack and for patients undergoing angioplasty or stent placement of the arteries of the heart.)  

This study reported that 3 of 192 outpatients (1 out of 64) in the study experienced a heart attack with elective outpatient cardiac catheterization. (This is in contrast to the usual rate of 1 out of 1000.)  One patient out of 189 patients with inpatient cardiac catheterization had a heart attack.

Implausibly, the authors concluded that these patient results demonstrated that outpatient heart catheterization was safe because this difference was not statistically significant.

If a cardiac catheterization lab actually experienced a rate of 3 heart attacks out of 192 previously in stable patients undergoing diagnostic heart catheterization, an investigation of that laboratory is in order rather than a commendation for procedural technique. Though outpatient catheterization is safe, the data in this study trends in the opposite direction.

 As it later turned out in response to a letter to the editor3, the authors answered4 that the seemingly high complication rate with outpatient catheterization was because of errors that they had made in the study. They stated they had mistakenly counted each patient having a heart attack twice rather than once. They also accidentally broke trial protocol by including a patient undergoing an elective angioplasty rather than a diagnostic heart catheterization. (This patient experienced a myocardial infarction in the cath lab and was assigned to the outpatient elective diagnostic heart catheterization group.)

 Hence, in this one study the authors not only misinterpreted their data, they miscounted their data endpoints and included patients that did not belong in this trial, and then incorrectly analyzed the data.

 

1.  Block P, Ockene I, Goldberg R, et al. A Prospective Randomized Trial of Outpatientversus Inpatient Cardiac Catheterization; N Engl J Med 1988: 219: 1251-55.

2. Davis K, Kennedy J,Kemp J, et al. Complications of Coronary Arteriography from the Collaborative Study of Coronary Artery Surgery (CASS). Circulation 1979: 59: 1105-12.

3. Letter. Outpatient versus inpatient catheterization. Roehm E. NEJM 1989; 320:938.

4.  Letter. Outpatient versus inpatient catheterization; Block P. NEJM 1989; 320:938-939  

 

copyright 2005  www.improvingmedicalstatistics.com

 

 

 

Hit Counter