Using Effect Scores to Characterize Heterogeneity of Treatment Effects

INTRODUCTION

This JAMA Guide to Statistics and Methods article explains effect score analyses, an approach for evaluating the heterogeneity of treatment effects, and examines its use in a study of oxygen-saturation targets in critically ill patients.

It is common for treatments to yield different outcomes in different patients. If patient characteristics that predict treatment response can be identified, understanding this heterogeneity of treatment effect (HTE) should inform individual treatment choices.^1,2 Effect score analyses are an approach for evaluating HTE, using a model developed to predict treatment effect given a patient’s characteristics, and then examining differences in treatment effects across groups of patients with different effect scores.^3-5 In an issue of JAMA, Buell et al⁶ reported the results of an effect score analysis using data from 2 randomized clinical trials to evaluate HTE for treatment strategies targeting different levels of oxygen saturation in critically ill patients receiving mechanical ventilation. The authors found that patients in different strata defined by the effect score had different treatment effects. They also noted that the stratum showing benefit from a lower target had a higher proportion of patients with acute brain injury, and the stratum showing benefit from a higher target had a higher proportion of patients with sepsis and abnormally elevated vital signs.³

WHAT IS EFFECT SCORE ANALYSIS?

The effect score is a prediction of each patient’s treatment effect, given their individual characteristics. The analysis then stratifies or groups patients in a clinical trial based on their effect scores.^3,4 The effect score model can be developed internally, based on the same dataset in which HTE will be evaluated, or externally, using data from another study. When the effect score model is developed internally, data from a single trial are usually randomly split into a model development dataset and a treatment effect estimation and HTE evaluation dataset; when the model is developed externally, one dataset is used for model development and another for evaluation.

Effect score analysis proceeds in 4 steps (Figure 1). First, the development dataset is used to create an effect score model, a statistical model for the treatment effect based on patient characteristics. Second, the resulting model is used to obtain an effect score value for each patient in the evaluation dataset. Third, the patients in the evaluation dataset are sorted by their effect scores and grouped into a few strata, typically 3 to 5. Fourth, in each stratum of the evaluation dataset, the treatment effect and corresponding confidence interval are obtained and heterogeneity across strata is assessed, for example, using a test for interaction.

The success of effect score analyses depends on how closely the effect score model captures the relationship between ...