This JAMA Guide to Statistics and Methods reviews the use of prerandomization run-in periods to improve treatment adherence and reduce loss to follow-up, and explains how they should be interpreted.
A prerandomization run-in is a period between screening a potential trial participant and their being randomized. In a 2019 article in JAMA Network Open, Fukuoka et al1 evaluated whether a mobile phone education application and in-person counseling could increase physical activity in 210 study participants. A prerandomization run-in was used to improve adherence and determine baseline physical activity levels of the participants. In this chapter, the advantages and limitations of run-in periods are reviewed.
Description of the Method
A potential participant is screened and asked for consent to take part in the trial but is not immediately randomized. Instead, for a period of weeks or months, the participant may try the intervention (or placebo) to determine if they are likely to be adherent to the study protocol if they were to be randomized. Participants are typically assessed at least twice before randomization to allow consent to be reconfirmed, adherence to treatment or with data collection determined, and baseline and end of run-in measures obtained. Only at the later assessment would the participant be randomized if they had been adherent with the study protocol and met all the inclusion criteria.
Why Are Run-in Periods Used in Trials?
For trialists, a key aim of the prerandomization run-in period is to improve adherence to trial treatments or procedures and to reduce loss to follow-up with consequent improvement in statistical power. A run-in also may allow assessment of treatment tolerability, response to treatment, or both. From the perspective of trial participants, a run-in allows time to reflect on whether they remain willing to be randomized, including giving more time to understand trial information, experience study procedures, and discuss participation with their managing physicians or family before committing to a long-term trial.
Nonadherence in clinical trials leads to systematic underestimation of the expected treatment effect that would result from actually taking the treatment (ie, 100% adherence and no drop-ins in the control group).2 A basic principle of randomized trials is that each participant is analyzed in the group to which they were assigned, irrespective of their adherence (ie, an intention-to-treat analysis).3 Undertaking “per-protocol” analyses (ie, analyzing only those who were adherent) introduces bias because of inherent differences between those who do or do not adhere to the trial protocol. By excluding participants likely to be poorly adherent or being unwilling to collect adequate data (such as occurred in the study by Fukuoka et al) before randomization, a run-in period can reduce the risk of bias due to differential dropout or data collection and improve statistical power.
Including a prerandomization run-in is particularly valuable in long-term trials, in which adherence to an intervention is required over a prolonged period. However, there is trade-off here between loss of generalizability of the results to a wider population (by excluding potentially nonadherent participants) and an improved chance of obtaining a reliable result by improved adherence. Whether a trial should include a run-in needs to balance these conflicting issues.
A randomized trial with poor adherence leading to an inconclusive result (a type II error) wastes time, money, and participant goodwill. If a statistically reliable result can be achieved by including a prerandomization run-in (while accepting some loss of generalizability), this is preferable. Typically, the randomized population after including a run-in will have a lower absolute event rate than those who were excluded. Those people randomized will tend to be healthier (a “healthy volunteer” effect), since older persons, individuals with more severe illness, and less adherent groups are excluded. However, when substantial randomized data from different types of people are available (as for blood pressure or low-density lipoprotein cholesterol [LDL-C] lowering), it indicates that the proportional effects are usually generalizable to populations at different levels of absolute risk.4 Run-ins are useful when treatments have known adverse effects that are likely to affect adherence. For example, a trial that investigated the effects of niacin on lipids and cardiovascular events in high-risk patients included an active run-in to minimize postrandomization dropouts due to known intolerance to niacin for some patients.5 One-third (33.1%) of participants did not tolerate niacin during the active run-in and were not randomized; if they had been included and stopped treatment during the trial, there would have been a serious reduction in study power to determine the effect of niacin.
When subgroup analyses of biomarker responses to treatments are anticipated, an active run-in is useful to characterize an individual patient’s biomarker response to treatment.6 This type of biomarker analysis is not possible without a run-in, as it is not possible to establish an appropriate comparator group in the placebo-allocated group because it is not known what biomarker response to the treatment is in the placebo-allocated group. In addition, the biomarker response to the intervention may also facilitate recalculation of sample size5 or facilitate selection of an appropriate dose of the study medication for patients to take. A run-in period can also be used to further check eligibility (eg, of blood markers that might render the intervention inappropriate) and allow time for other clinicians to be informed of patients’ possible recruitment into the clinical trial.6
A run-in period may facilitate the standardization of a patient’s usual treatment regimen or allow washout of a potentially interacting drug. For example, when assessing a new cholesterol-modifying drug, it may be desirable to standardize a patient’s current LDL-C–lowering therapy to ensure all the enrolled patients are adequately treated when the study is initiated. This approach can minimize the risk of the addition of postrandomization nonstudy lipid-lowering therapy, which would create difficulties in estimating the effect of the study drug.5 Run-in periods can also facilitate optimization of a patient’s disease treatment (eg, for renin-angiotensin system blockade treatment in a trial for kidney disease) or ensure a stable clinical condition (eg, for asthma) prior to randomization to the intervention.
WHAT ARE THE LIMITATIONS OF INCLUDING A RUN-IN?
Because a run-in excludes individuals who are poorly adherent or experience adverse effects from the intervention, the representativeness of the trial population may be compromised and results less widely generalizable.7 The hazards of treatments may be underestimated in the randomized study because vulnerable groups (such as the elderly) who may be both less adherent and at greater risk of adverse effects are excluded as a result of the run-in.
Including a run-in has cost implications for the study, and money spent could be used to increase the sample size. The counter argument is that a run-in improves cost-effectiveness, as better adherence allows a smaller sample size and fewer resources in follow-up. In the Physicians Health Study, including the run-in was estimated to increase pill-taking adherence by 20% to 40% and allowed for an estimated 30% smaller sample size.8
Participants may notice the change in treatment from run-in to the postrandomization period (eg, from placebo to active treatment, or vice versa), thereby compromising the blinding of the study. If the trial is collecting “hard” clinical outcomes, such as heart attacks or death, this is unlikely to be a problem, but if outcomes are subjective (such as symptomatic pain), there is a risk for introducing bias in the outcome assessment. This unblinding risk can be minimized by good matching of the treatment and placebo, as symptom reporting should not then be affected by knowledge of treatment assignment. Other limitations include the possibility of “carry-over” effects of the run-in treatment into the postrandomization phase. For example, it may be best to avoid an active run-in when an intervention has a long effective half-life.9
The appropriateness of run-in periods depends on the aims of the trial, the nature and duration of the intervention, the population, the condition, and length of follow-up. Run-ins are most relevant in long-term trials in which loss of adherence may adversely affect the ability to obtain a clear result or in which complex procedures or follow-up leads to poor adherence with the intervention or follow-up. Run-ins are not appropriate or desirable in trials in acute disease, surgical trials, or studies with short observation periods.
WHY DID THE mPED STUDY USE A RUN-IN?
In the study by Fukuoka et al, a 3-week run-in allowed determination of participants’ baseline average daily steps and adherence to the study intervention, defined by at least 80% response to daily messaging, 80% use of a daily activity diary, and wearing an accelerometer for at least 8 hours per day. Only those who used the accelerometer regularly and met the adherence requirements during run-in (n = 210) were eligible for randomization. The run-in provided the opportunity to obtain an average baseline for an outcome measure that had substantial variability (ie, physical activity) and allowed for adherence to be assessed.
HOW SHOULD THE RESULTS OF A RUN-IN BE INTERPRETED IN THE mPED STUDY?
Because of the run-in, 34% of potential study participants who initially provided consent (n = 108) were excluded before randomization, with poor adherence using the study equipment the most common reason (≈20%). Those withdrawals, discontinuations, and lack of end point collection occurring after randomization would have substantially reduced study power. The overall retention rate of 97.6% at 9 months allowed a statistically significant effect of the app-based intervention to be detected in this adherent population. The degree to which these results can be extrapolated to a less adherent group is a matter of clinical judgment.
The following disclosures were reported at the time this original article was first published in JAMA.
Conflict of Interest Disclosures: None reported.
Additional Contributions: We are grateful to Richard Haynes, DM, Will Herrington, MD, David Preiss, PhD, Marion Mafham, MD, Richard Bulbulia, MD, and Louise Bowman, MD (MRC Population Health Research Unit, Clinical Trial Service Unit and Epidemiological Studies Unit, Nuffield Department of Population Health, University of Oxford), and Dylan Morris, DPhil (Queensland Research Centre for Peripheral Vascular Disease, College of Medicine and Dentistry, James Cook University, Townsville, Queensland, Australia), for their constructive advice on the manuscript. None of these individuals received any compensation for their contributions.
E. Short-and long-term effects of a mobile phone app in conjunction with brief in-person counseling on physical activity among physically inactive women: the mPED randomized clinical trial. JAMA Netw Open. 2019;2(5):e194281. doi: 10.1001/jamanetworkopen.2019.4281 Medline:31125101
J. Sample size calculations for randomized controlled trials. Epidemiol Rev. 2002;24(1):39–53. doi: 10.1093/epirev/24.1.39 Medline:12119854
; Blood Pressure Lowering Treatment Trialists’ Collaboration. Blood pressure-lowering treatment based on cardiovascular risk: a meta-analysis
of individual patient data. Lancet.
2014;384(9943):591–598. doi: 10.1016/S0140-6736(14)61212-5 Medline:25131978
; HPS2-THRIVE Collaborative Group. Effects of extended-release niacin with laropiprant in high-risk patients. N Engl J Med.
Protection Study Collaborative Group. MRC/BHF Heart
Protection Study of cholesterol lowering with simvastatin in 20,536 high-risk individuals: a randomised placebo-controlled trial. Lancet.
2002;360(9326):7–22. doi: 10.1016/S0140-6736(02)09327-3 Medline:12114036
PM. External validity
of randomised controlled trials: “to whom do the results of this trial apply?” Lancet.
2005;365(9453):82–93. doi: 10.1016/S0140-6736(04)17670-8 Medline:15639683
CH. Estimating the effect of the run-in on the power
of the Physicians’ Health Study. Stat Med.
1991;10(10):1585–1593. doi: 10.1002/sim.4780101010 Medline:1947514
et al; Study of the Effectiveness of Additional Reductions in Cholesterol and Homocysteine (SEARCH) Collaborative Group. Effects of homocysteine-lowering with folic acid plus vitamin B12 vs placebo
on mortality and major morbidity in myocardial infarction survivors: a randomized trial. JAMA.[JAMA and JAMA Network Journals Full Text]
2010;303(24):2486–2494. doi: 10.1001/jama.2010.840 Medline:20571015