Methods for Evaluating Changes in Health Care Policy—The Difference-in-Differences Approach

INTRODUCTION

This JAMA Guide to Statistics and Methods describes when to use a difference-in-differences analysis to evaluate changes in health care before and after changes in health care policy.

Observational studies are commonly used to evaluate the changes in outcomes associated with health care policy implementation. An important limitation in using observational studies in this context is the need to control for background changes in outcomes that occur with time (eg, secular trends affecting outcomes). The difference-in-differences approach is increasingly applied to address this problem.¹

Two studies by Rajaram and colleagues² and Patel and colleagues³ used the difference-in-differences approach to evaluate the changes that occurred following the 2011 Accreditation Council for Graduate Medical Education (ACGME) duty hour reforms. These 2 studies were conducted with different data sources and study populations but used similar methods.

USE OF THE METHOD

Why Was the Difference-in-Differences Method Used?

The association between policy changes and subsequent outcomes is often evaluated by pre-post assessments. Outcomes after implementation are compared with those before. This design is valid only if there are no underlying time-dependent trends in outcomes unrelated to the policy change. If clinical outcomes were already improving before the policy, then using a pre-post study would lead to the erroneous conclusion that the policy was associated with better outcomes.

The difference-in-differences study design addresses this problem by using a comparison group that is experiencing the same trends but is not exposed to the policy change.⁴ Outcomes after and before the policy are compared between the study group and the comparison group without the exposure (group A) and the study group with the exposure (group B), which allows the investigator to subtract out the background changes in outcomes. Two differences in outcomes are important: the difference after vs before the policy change in the group exposed to the policy (B2−B1, Figure 7) and the difference after vs before the date of the policy change in the unexposed group (A2−A1). The change in outcomes that are related to implementation of the policy beyond background trends can then be estimated from the difference-in-differences analysis as follows: (B2−B1) − (A2−A1). If there is no relationship between policy implementation and subsequent outcomes, then the difference-in-differences estimate is equal to 0 (Figure 7A,). In contrast, if the policy is associated with beneficial changes, then the outcomes following implementation will improve to a greater extent in the exposed group. This will be shown by the difference-in-differences estimate (Figure 7B,).

These estimates are derived from regression models rather than simple subtraction. Using regression modeling allows the estimates to be adjusted for other factors (eg, patient or hospital characteristics) ...