Meta-analyses Can Be Credible and Useful: A New Standard

INTRODUCTION

This JAMA Guide to Statistics and Methods contends that meta-analyses can become the new prototype for original research.

OVERVIEW

Carefully done meta-analyses constitute a major advance compared with expert opinion and nonsystematic attempts at summarizing, synthesizing, and integrating information. Meta-analyses serve many fields in summarizing an increasing stream of data and, for clinical purposes, streamlining information for decision making. However, there are flaws and caveats that threaten the validity and utility of meta-analyses.¹

Most of these efforts are retrospective exercises that try to piece together fragments of information from multiple completed studies. They depend on information that is already published (or at least retrievable) with all the accompanying errors and biases and rarely correct these problems. For example, 8 meta-analyses of imaging in unipolar depression have reached inconsistent conclusions because they have used different studies with diverse protocols and methods that are difficult to standardize post hoc and different errors that other meta-analyses may or may not correct for. A meta-analysis should systematically probe, detect, dissect, and highlight major errors and biases (instead of sanctifying flawed studies by their inclusion). Careful bias scrutiny alone can be a major service to a field. However, it is always tempting to take a shortcut to talk about summary effects and forget about the deficiencies of the evidence.

THE EXISTING EVIDENCE

Bias is inflated when meta-analyses are done by authors and/or sponsors with financial or other conflicts of interest. Authorship by company employees and/or sponsorship by companies was the strongest risk factor for reporting no caveats for antidepressants among a body of 185 meta-analyses on antidepressants for depression published between 2007 and 2014.² Allegiance bias may be an equivalent problem for evidence on psychotherapies.³ Conflicted meta-analyses compound the distortion that exists in the publication process of primary studies, including a publication bias against negative results, the selective reporting of negative trials as positive,⁴ and other spins (eg, changing the analysis plan or the focus of interpretation) that lead to more favorable results and interpretations.

Of approximately 20000 meta-analyses performed annually,¹ well over 1000 have relevance for mental health. In an empirical survey using stringent criteria for labeling meta-analyses,⁵ 7% pertained to mental and behavior disorders. Most of these meta-analyses look only at published data and circumscribed, small fractions of the evidence that might be relevant for the question of interest. For example, in therapeutics research, of 822 network meta-analyses of clinical trials published until May 2015, only 39 pertained to mental health. Moreover, to my knowledge, there are few meta-analyses in mental health that have been able to use individual-level data. In a database of 829 meta-analyses with individual-level data published until 2012,⁶ only 52 (6.3%) pertained to mental and behavioral disorders. Most of them either pertained to nontherapeutic questions (eg, prognostic, biomarker, imaging, and association ...