This JAMA Guide to Statistics and Methods discusses adjusting for covariates, a common cause of false findings in published research, by allowing variables to vary as they will but then using a mathematical model to assess their influence on the outcome.
Concern about erroneous conclusions of many published research findings has led to the conclusion that most published research findings are wrong.1,2 What can be done about that? In what follows, I will focus on one common source of false findings: adjusting for covariates.
Here, adjusting means allowing variables to vary as they will but then using a mathematical model to assess their influence on the outcome. In contrast, to control means manipulation of variables by the researcher for a particular purpose (eg, in experimental design). Unfortunately, the terms adjust and control are often used as if they were synonymous. Adjusting often leads to false conclusions because the models used may not correspond to reality.
To illustrate this point, consider a randomized clinical trial (RCT) in which those sampled from the population of interest are randomly assigned to 2 treatment groups, T1 and T2. A valid test simply comparing the outcomes in the 2 groups tests the overall effect size (overall ES) that a randomly sampled patient from T1 has an outcome clinically preferable to that of a randomly sampled patient from T2.3
Often, the first table of an RCT report compares the baseline characteristics of the T1 vs T2 samples to assess the success of randomization, ignoring the fact that randomization (1) is a process, not an outcome, and (2) is meant to generate 2 random samples from the same population, not 2 matched samples. When a few baseline variables significantly differentiate the 2 groups at the 5% level, researchers often propose to adjust for those covariates in testing the treatment effect. This is post hoc testing (like offering to bet at prerace odds on a horse as it approaches the finish line), which frequently leads to false-positive results.
Any covariates to be used in adjusting should be specified a priori, listed in the RCT registration, and taken into consideration in the power analysis. Such adjustment changes the hypothesis to be tested from comparing all T1 patients vs all T2 patients (overall ES) to comparing T1 patients only with T2 patients matched in one way or another on the particular covariates proposed. Let’s say that covariate ES is the ES for patients with one particular configuration of the covariates and typical ES is the ES specifically for patients who are at the mean of each such covariate (ie, for the typical patient). Overall ES, typical ES, and all possible covariate ES are the same only if the covariates are irrelevant to the treatment outcome. If the covariates are irrelevant, adjusting for those simply leads to a loss of power. If the covariates are not ...