Chapter 11.2: Surprising Results of Randomized Trials

Most Major Basic Science and Preclinical Promises for Effective Interventions Disappoint in Clinical Trials

Ideally, evidence for the effectiveness of diagnostic, preventive, or therapeutic interventions will come from rigorous randomized clinical trials (RCTs) measuring effects on patient-important outcomes, such as stroke, myocardial infarction, and death. Whenever an intervention is tested to see whether it is effective for patient-important outcomes, typically some other evidence of variable quantity and quality already exists. This evidence includes combinations of basic science findings, preclinical results, observational studies, and phase 1 or 2 clinical trials.

Sometimes, clinicians adopt interventions even though randomized trials have never been performed to test their effect on patient-important outcomes. This is very common for acute surgical interventions, common for elective surgical interventions and mental health interventions, and somewhat less common for medical interventions.¹ Nevertheless, even for medical interventions, randomized trials are usually unavailable for interventions that need to be applied for specialized decisions after some major first decision has been made. For example, in the hematology-oncology field there is little evidence from randomized trials for the treatment of recurrent diseases.² For interventions such as these, their adoption and continued use in clinical practice have been based on various combinations of basic science, preclinical, and observational evidence.

Moreover, there is a strong undercurrent in many scientific circles that supports the use of surrogate end points for adopting interventions for common diseases. Trials using surrogate end points require smaller sample sizes and shorter follow-up periods than trials of patient-important end points. Thus, drugs and other interventions can be rapidly tested and approved for clinical use.³

Given this patchy and uneven availability of evidence, surprises often occur when interventions that seem promising—or that have even been established according to relatively high-quality evidence—prove disappointing in large randomized trials. Typically, fewer and fewer promising interventions retain their postulated claims to effectiveness as we move from basic science experimentation to RCTs with patient-important outcomes. An empirical evaluation⁴ examined 101 major findings published in the top basic science journals between 1979 and 1983 in which the investigators confidently declared that their work would be translated to a major therapeutic or preventive intervention. Of those, only 27 eventually had a randomized trial, and by 2002 only 19 had positive results in at least 1 randomized trial with any kind of end point. At that time, only 5 interventions were approved for clinical use, with only 1 of them having a major effect in therapeutics and the other 4 having uncommon or questionable clinical indications. The credibility of basic science and preclinical claims or observational discoveries, fascinating as they may be, is often low.⁵

Types of Low-Quality Evidence

There are a number of reasons evidence may warrant low confidence⁶; here, we will highlight 3 categories. First, although the methods of a study may be pristine, the ...