A Primer on the Precision and Accuracy of the Clinical Examination

A Primer on the Precision and Accuracy of the Clinical Examination: Introduction

This background article will introduce and explain the terms and concepts that are being used in the series of overviews on the rational clinical examination that begins in this issue of The Journal. It includes definitions and explanations of certain key concepts, clinical examples, guides for reading clinical journals about a diagnostic test, and a blank “working table” that you can use to apply the concepts on your own.

Background articles in this series will discuss selected issues in the precision and accuracy of the clinical examination in greater detail or extend them to more complex diagnostic situations. Some of these issues are also discussed in clinical epidemiology textbooks.¹

Of course, the precision and accuracy of the clinical examination are not the only concerns in the clinical encounter, and their proper application provides only the starting point for decisions about how certain we need to be about a diagnosis before we act on it (the decision threshold) and how we ought to incorporate the concerns of both patients and society in deciding whether and how to act. Later background articles will discuss these additional considerations; this one will be confined to precision and accuracy.

Like others in the series, this background article will be introduced with a patient.

The Patient

One of your patients, whom you have not seen for several years, is admitted to the orthopedic service after a packing crate has tipped over onto his leg, producing an unstable fracture of his distal tibia and fibula. You stop by to see him as he is being prepared for surgery. He is alert and hemodynamically stable but smells of alcohol (at 10 am) and has 3 spider nevi on his upper chest (but no gynecomastia or asterixis). He is obese, and his belly is prominent. Among the questions that are raised in your mind, xsthe following are of special significance:

1. Is this man an alcoholic? You would place the odds for this disorder at 50-50 (and the science of the art of how clinicians generate these odds will be the subject of a later background article). The answer to this diagnostic question is important in the long run and in protecting him from the complications of acute withdrawal during and after his operation.

2. Does he have ascites? You are much less sure here, but if he is alcohol dependent you would place the odds that the prominence of his belly represents ascites also at 50-50. Again, it would be important to know whether he has this manifestation of advanced alcoholic liver damage.

Your options for answering these questions are several. To explore his possible alcohol abuse or dependency, (1) you could take the time required for a thorough confrontation and interrogation about the amount of alcohol he consumes (and, in the process, risk alienating him, estranging the nursing staff, and exasperating yourself); (2) you could order 1 or more liver function tests; (3) you could even request one of the new, “hot” tests for platelet enzyme activity, reported to be elevated in persons with alcoholism²; or (4) you could ask him the 4 quick “CAGE” questions: Have you ever felt you should cut down on your drinking? Have people annoyed you by criticizing your drinking? Have you ever felt bad or guilty about your drinking? Have you ever had a drink first thing in the morning to steady your nerves or to get rid of a hangover (eye-opener)? This opening example in the series is all the more appropriate when we observe that the first report on the CAGE questionnaire in a general medical journal was by John Ewing³ and that it was accompanied by an editorial from a major supporter of this series, George Lundberg.⁴ To explore his possible ascites, (1) you could check him for shifting dullness, fluid wave, or even the puddle sign; (2) you could order an abdominal ultrasonographic examination; or (3) you could simply ask him whether he has ever had swollen ankles.

Stop for a moment and consider the implications, in terms of your time and somebody's money, of the alternative ways of answering these 2 questions. Would it not be better if you could answer them both with just 5 quick questions (4 for CAGE and 1 about ankle swelling)?

As it happens, you might be able to do just that. If he answers yes to 3 or 4 of the CAGE questions, he is an alcohol-abusing or alcohol-dependent man (and this medical history is far more powerful than any laboratory tests you can order). If he answers no to ankle swelling, you have pretty well ruled out clinically important ascites (you could double check the latter by testing for shifting dullness; like most such patients, he did not have a fluid wave, and as you will learn in a forthcoming overview on ascites, the puddle sign is not useful in him or anybody else). Thus, for both questions, a quick bedside examination has provided definitive diagnostic information, without the need for laboratory testing or diagnostic imaging.

How can we make such a bold statement about the power of these simple elements of the clinical history and physical examination? The answer lies in the science of the art of clinical diagnosis that underpins this series of overviews on the rational clinical examination. This first background article will introduce and illustrate the key elements of this science (and readers who want a more detailed discussion of what follows can consult a step-by-step discussion published elsewhere¹). The background articles also are intended to convey the fun and gratification physicians derive from making correct diagnoses with crispness and dispatch.

Taking an Alternative History for Alcoholism

Examine Figure 1-1. In it are shown the number of positive answers to the CAGE questions from 2 groups of patients admitted to the orthopedic or medical services of a community-based teaching hospital in Boston, Massachusetts.⁵ In the left-hand column are the responses from patients whose extensive evaluations (including, where indicated, detailed social histories, follow-ups, and liver biopsies) provided acceptable “proof” that they were alcohol abusers or alcohol dependent. In the right-hand column are patients whose evaluations showed that they were not alcohol abusers or dependent. These extensive confirmatory investigations often are referred to as criterion standards of diagnosis and typically consist of definitive findings at angiography, operation, autopsy, and the like.

This study is useful to clinicians because the CAGE history and the extensive (reference or criterion standard) investigations were carried out independently among a wide spectrum of well-described patients in whom it was clinically reasonable to inquire about alcohol abuse. It thus satisfies the first criterion of a valid, clinically useful article on diagnostic strategies that appears in Table 1-1 (has there been an independent, “blind” comparison with a criterion standard of diagnosis?). The readers’ guides in Table 1-1 have been used by the authors of this series on the rational clinical examination to “screen” articles for inclusion in their overviews of diagnostic approaches to specific clinical problems. Table 1-1 can be clipped and carried for easy reference when reading clinical articles that make claims about the usefulness of (especially new) diagnostic tests, and the reasoning behind its elements are described in detail elsewhere.¹

The study that generated Figure 1-1 also satisfied the second, commonsense guide, for it was carried out in a patient sample that included an appropriate spectrum of mild and severe, treated and untreated alcoholism, plus individuals with different but commonly confused disorders. The setting for the study (a large, urban, general hospital) was described, satisfying the third readers’ guide and permitting us to determine the applicability of the results to our own setting, and the term normal (the fifth guide) was clearly and sensibly defined as the absence of alcohol abuse or dependency (we shall return to the fourth guide of reproducibility later).

The authors of the CAGE study were not proposing that their questions be used as part of an extensive series (“cluster”) of diagnostic tests (so the sixth guide does not apply), and the questions were presented with their exact wording in the article, satisfying the seventh guide and permitting their exact application in the reader's own practice. The final readers’ guide (has the utility of the test been determined?) is satisfied to the extent that the CAGE questions recognized far more persons with alcoholism, especially alcohol abusers, than routine clinical diagnosis and made them candidates for treatment and counseling.

In summary, the CAGE study observed the methodologic standards required for a valid and clinically useful description of the clinical applicability of any diagnostic information, whether it comes from the clinical history, the physical examination, or the diagnostic laboratory.

The Precision of the Clinical Examination

For an item of the clinical history or physical examination to be accurate, it first must be precise. That is, we need to have some confidence that 2 clinicians examining the same, unchanged patient would agree with each other on the presence or absence of the symptom (such as our patient's answer to one of the CAGE questions) or sign (such as the presence of spider nevi on our patient's chest). The precision (often appearing under the name of “observer variation” in the clinical literature) of such clinical findings can be quantitated.⁶

Suppose 2 clinicians recorded whether they found spider nevi when they independently examined the same 100 patients suspected of having liver disease and generated the data shown in Figure 1-2. The 2 clinicians agreed that 23 of the patients (cell a) had spider nevi and that 66 patients (cell d) did not; thus, they agreed on (23 + 66)/100 = 89% of the patients they examined. However, 6 patients (cell c) judged to have spider nevi by the first clinician were judged not to have nevi by the second, and 5 patients (cell b) judged to have spider nevi by the second clinician were judged not to have nevi by the first. How should we interpret this precision? Is this degree of clinical agreement good, or should we expect better?

We might begin by recognizing that some clinical agreement would occur by chance alone. For example, if the second clinician merely tossed a coin for each patient instead of carrying out an examination, reporting nevi if the coin came up “heads” and no nevi if it came up “tails,” agreement would be 50%. We should begin, then, by determining how much of the observed agreement of 89% was because of chance, so that we can find out how much real clinical skill (agreement beyond chance) was being displayed by these clinicians. Chance agreement can be calculated by the formal process of “marginal cross-products” shown in Figure 1-2, but it also can be thought of as a coin toss in which, for example, the first clinician's coin came up heads 29% of the time (based on [a + c]/[a + b + c + d]). Thus, 29% of the 28 patients judged to have spider nevi by the second clinician (a + b) would also be judged to have them by the first clinician, and 29% of 28 is 8 (the number of patients we would expect to find in cell a by chance alone). Similarly, the first clinician's coin came up tails 71% of the time ([b + d]/[a + b + c + d]), and 71% of the 72 patients judged to be free of spider nevi by the second clinician (c + d) is 51 (the expected value for cell d). As a result, we would expect the 2 clinicians to agree (8 + 51)/100, or 59% of the time, on the basis of chance alone, and the remaining potential agreement beyond chance is therefore 100% – 59%, or 41%.

How much of this 41% potential agreement beyond chance was achieved? This is determined by comparing it with the actual agreement beyond chance of 89% – 59%, or 30%, and 30%/41% comes to 0.73, which means that about three-fourths of the potential agreement beyond chance was achieved by our 2 clinicians. This measure of agreement goes by the name κ and is rather like a correlation coefficient.¹ It ranges from –1.0 (where 2 clinicians would be in perfect disagreement), through 0.0 (where only chance agreement was accomplished), to +1.0 (where 2 clinicians would be in perfect agreement). As you can see in the listing of “conventional levels of κ” that appears in the legend for Figure 1-2, the agreement between our 2 clinicians is considered “substantial,” and this is the case for many “present/absent” aspects of the physical examination. As you might imagine, agreement is greater still when the 2 examinations are carried out by the same clinician.

Other items on the clinical examination do not fare as well. For example, in one study of the chest examination, the κ for cyanosis, tachypnea, and whispered pectoriloquy was 0.36, 0.25, and 0.11, respectively.⁷

No measure of clinical agreement is ideal, and κ is no exception. Its size is slightly affected by the frequency of the abnormal finding in the group of patients being examined (it is highest when half of the patients have the finding and tails off a bit when the finding is extremely common or uncommon). If your and our interests warrant, we shall come back to this in a subsequent background article.

But, of course, high precision is not enough, for examiners may be consistent but wrong in their assessments. All 5 members of my clinical team occasionally fail to detect a big liver or hear an important diastolic murmur. In other cases, clinicians may be neither precise nor accurate. For example, a group of iridologists was asked to examine the irises of a series of patients and distinguish those with gallstones from those who had sonographically empty gallbladders.⁸ Their clinical agreement was only “slight,” with an average κ of 0.18 (about like whispered pectoriloquy). More important, however, their diagnostic accuracy was no better than chance: they missed about half the patients with gallstones (sensitivity, 54%) and diagnosed gallstones in about half the patients with negative sonogram results (specificity, 52%). To understand sensitivity and specificity, we must now shift from determining the precision of the clinical examination to defining the characteristics of its accuracy.

The Characteristics of the Accuracy of Diagnostic Tests

Returning our attention to Figure 1-1, we can examine the accuracy characteristics of the CAGE questions. The 60 patients in cell a of Figure 1-1 answered yes to 3 or 4 of the CAGE questions and constitute 51%, or 0.51, of all the 117 patients (a + c) with a positive diagnosis of alcohol dependency or abuse. The shorthand term for this proportion of 0.51, or a/(a + c), is, and it is a useful measure of how well a diagnostic test (whether a symptom, sign, or laboratory test) detects a target disorder when it is present. The closer the sensitivity to 100%, the more “sensitive” the clinical or laboratory finding.

In the right-hand column are the responses from patients for whom the criterion standard ruled out the diagnosis of problem drinking. The 400 patients in cell d answered yes to 2, only 1, or none of the CAGE questions and constitute 99.8%, or 0.998, of all the 401 patients (b + d) who did not have alcohol dependency or abuse. The shorthand term for this proportion of 0.998, or d/(b + d), is, and it is a useful measure of how often a symptom, sign, or other diagnostic test is absent when the target disorder is not present. The closer the specificity to 100%, the more “specific” the clinical or laboratory finding. (Of course, clinicians are not interested in sensitivity and specificity as such but in their effect on the interpretation of positive and negative findings, and we shall get to that shortly. Sensitivity and specificity are properties that must be established beforehand, and that is why they are presented here.)

You will observe that the sensitivity of the CAGE questions is not impressive. The number of “true positives” in cell a is almost equaled by the number of “false negatives” in cell c, and the sensitivity of only 51% confirms that it “misses” about half the problem drinkers. On the other hand, the specificity of the CAGE questions is outstanding. The number of “true negatives” in cell d vastly outnumbers the number of “false positives” in cell b, and the specificity of 99.8% confirms that it almost never labels a patient as a problem drinker when this disorder is absent.

Now we can consider the “predictions” we make about our patient according to the foregoing characteristics. Because of the high specificity, virtually every patient in cell a who answered yes to 3 or 4 of the CAGE questions (a + b) has the target disorder, alcohol abuse or dependency, and the shorthand term for this proportion a/(a + b), which is 60/61, or 98%, is the positive predictive value or posttest probability of having the target disorder (among patients with 3 or more positive answers). Moreover, despite the rather unimpressive sensitivity, most of the patients in cells c and d who answered yes to none, just 1, or 2 of the CAGE questions were in cell d and did not have the target disorder. The shorthand term for this proportion d/(c + d), which is 400/457, or 88%, is the negative predictive value or posttest probability of not having the target disorder among those patients with 2 or fewer positive answers. The complement of this negative predictive value, or c/(c + d), describes the posttest probability of having the disorder among those patients with 2 or fewer positive answers, and this other way of saying the same thing is found useful by some clinicians.

The reason that the negative predictive value looks relatively high, despite the low sensitivity, lies in the fact that the proportion of all patients in this study who had alcohol dependency or abuse, (a + c)/(a + b + c + d), or 117/518, was only 23% to begin with. That is, 100% – 23%, or 77%, of the patients were not alcohol dependent before they were asked any questions. The shorthand term for the previous knowledge contained in this (a + c)/(a + b + c + d) is or, more usefully, the pretest probability of the target disorder (because this pretest probability is the starting point for making clinical use of the test characteristics, we will place it above the “predictions” entries in subsequent figures).

In contrast to this pretest probability of 23% in the clinical article describing the CAGE questions, in our patient, we judged that the pretest probability of alcohol abuse or dependency was 50%. How would the CAGE questions perform in patients like ours? If the patients in the study summarized in Figure 1-1 were like our own patient, we would expect the result shown in Figure 1-3.

As long as the patient “mix” and severity of disease in the CAGE study summarized in Figure 1-1 are similar to the patient mix and severity of disease in our practice, we would expect sensitivity and specificity to remain constant, despite changes from the study's to our patient's pretest probability of the target disorder. Thus, the sensitivity (51%) and specificity (99.8%) in Figure 1-3 are the same as those in Figure 1-1.

Notice, however, that the negative predictive value has decreased from 88% to 67% because predictive values must change with changes in the prevalence of the target disorder. One useful way to think about this is to carry through this concept of prevalence. After all, the predictive value of a positive test result is simply the prevalence of the target disorder among those patients with positive test results. Similarly, the negative predictive value is the prevalence of not having the target disorder among patients with a negative test result. No wonder, then, that predictive values must change with a change in the overall prevalence of the target disorder.

Back to the Patient

Your patient readily admitted that he had cut down on his drinking, that his spouse and workmates had annoyed him by complaining about his drinking, and that he often needed an “eye opener” to get going in the morning. According to this quick medical history, and given your previous judgment (before you had any knowledge of his responses to any of these questions) that his chances of being alcohol dependent were 50-50 (ie, a pretest probability of 50%), you can follow his response through Figure 1-3 and conclude that his posttest probability of alcohol dependency is 99.6%, or about as certain as you ever can be about any diagnosis.

Your patient helps us make another general point: because he gave a positive response to a diagnostic history whose specificity was extremely high (99.8%), you “ruled in” the target disorder. A simple way of remembering this property of a powerful diagnostic test is the acronym SpPin: when specificity is extremely high, a positive test result rules in the target disorder.

Would the laboratory tests you were considering ordering have saved you some time and done a better job of determining this diagnosis? In fact, and in addition to delaying the diagnosis, their accuracy is much worse. In the same investigation that studied the CAGE questions, the specificities for γ-glutamyl transpeptidase, mean corpuscular volume, and an entire liver function battery were only 76%, 64%, and 81%, respectively.³ Moreover, the hot new test of platelet enzyme activity has a specificity of only 73%.² Thus, in your patient, a simple medical history was not only quicker and easier but also far more specific.

What about his possible ascites? Given that you have established the diagnosis of alcohol dependency, you already can plan his perioperative and postoperative management to prevent, detect, and treat alcohol withdrawal syndromes. Nonetheless, you would like to know whether he has sufficient liver damage to affect his handling of the sorts of drugs he is likely to receive. Given his fractured ankle, the kneeling position required for eliciting the puddle sign is out of the question, and even a test for shifting dullness will cause him considerable pain. He has already been to radiology, and you do not want him to make the trip again for an abdominal ultrasonographic examination if you can avoid it. His uninvolved ankle is not swollen now, and he tells you he has never had ankle swelling in the past. Would this simple medical history for previous ankle swelling be of any use?

Figure 1-4 summarizes a study of 63 patients admitted to a general medical service in Durham, North Carolina.⁹ Of 15 patients with ascites on abdominal ultrasonographic examination (the criterion standard), 14 had a history of ankle swelling, for an impressive sensitivity of 93%. If we applied this sensitivity (93%) and specificity (66%) to our pretest probability for ascites of 50%, the result (shown in Figure 1-5) suggests that the posttest probability of not having ascites is 90% when the patient denies ankle swelling. Again, this simple element of the clinical history provides powerful diagnostic information: when the sensitivity of a symptom or sign is high, a negative response rules out the target disorder, and the acronym for this property is SnNout.

However, you may have observed that this study included only 15 patients with ascites, and you may well inquire how confident we should feel about this sensitivity of 0.93. As it happens, the degree of confidence we ought to place in this (or any other) estimate of sensitivity (or specificity) can be calculated and expressed as a confidence interval, within which you can be confident that the true sensitivity resides, say, 95% of the time.¹ In this case, the 95% confidence interval on this sensitivity of 0.93 based on 15 patients runs all the way from 0.81 (not terribly sensitive) to 1.00 (or perfect sensitivity). If, on the other hand, this sensitivity of 0.93 were based on 100 patients with ascites, the 95% confidence interval would run from 0.88 to 0.98, and you would be justified in being more confident that a negative medical history rules out ascites. Thus, you should look for information on the 95% confidence interval for measures of accuracy such as sensitivity and specificity when you read about them.

A Faster and More Powerful Approach: The Likelihood Ratio

Many of the overviews in this series will describe not only the sensitivity and specificity of specific symptoms and signs but also their likelihood ratios (LRs). This method of describing the accuracy of diagnostic information, once mastered, is much faster and more powerful than the sensitivity and specificity approach.¹ It is shown in Figure 1-6 for ankle swelling and ascites. In brief, an LR expresses the odds that a given finding on the medical history or physical examination would occur in a patient with, as opposed to a patient without, the target disorder. When a finding's LR is above 1.0, the probability of disease increases (because the finding is more likely among patients with than without the disorder); when the LR is below 1.0, the probability of disease decreases (because the finding is less likely among patients with than without the disorder); finally, when the LR is close to 1.0, the probability of disease is unchanged (because the finding is equally likely in patients with and without the disorder).

Figure 1-6, the LR for a positive history of ankle swelling is equal to sensitivity/(1 – specificity) or 0.93/0.33, or 2.8, indicating that a positive history is almost 3 times as likely to be obtained from a patient with, as opposed to a patient without, ascites. The LR for a negative history of ankle swelling is equal to (1 – sensitivity)/specificity or 0.07/0.67, or 0.10, indicating that a negative history is only as likely to be obtained from a patient with, as opposed to a patient without, ascites (and confirming our earlier conclusion that this negative history permitted us to SnNout this diagnosis).

The first advantage of LRs is that the LR for a given finding, when applied to the pretest odds of the target disorder, generates the posttest odds for that disorder. Because the LR is expressed as an odds, this may at first appear cumbersome, for it means that the pretest probability must also be expressed as an odds (although this is tedious to do by hand, later, we will show you how to avoid the calculations by using the nomogram shown in Figure 1-7). When done by hand, the pretest probability of the target disorder is converted into pretest odds by the formula:

In Figure 1-6, the pretest probability of ascites is 0.24, and the pretest probability of not having ascites is 1.00 – 0.24, or 0.76. Therefore, the pretest odds of ascites are 0.24/0.76, or 0.32, and this can be multiplied by 2.8 (generating a posttest odds of ascites of 0.90) when the history is positive for ankle swelling and by 0.10 (generating a posttest odds of 0.03) when this history is negative.

These posttest odds can then be converted back to probabilities by the formula:

Thus, the posttest odds of 0.90 following from a positive history of ankle swelling converts (by 0.90/1.90) to 47%, and the posttest odds of ascites of 0.03 following from a negative history converts (by 0.03/1.03) to 3%, and you will observe that these are the same values for the posttest probability of having ascites that we generated in Figure 1-4.

The necessity for converting probability to odds and back again can be obviated by using the nomogram shown in Figure 1-7, which has already carried out the conversions for us.¹ You can prove this to yourself as follows: anchor a straightedge at the left margin of the nomogram, at the pretest probability of 24%, and rotate the straightedge until it intersects the middle line of the nomogram at an LR of 2.8, corresponding to a positive history of ankle swelling. It will intersect the right margin of the nomogram at just below 50%. Similarly, rotate the straightedge until it intersects an LR of 0.10 for the negative history and observe that the posttest probability of ascites decreases to 3%.

The second advantage of LRs becomes apparent when we see that the nomogram permits us to determine the probability of ascites when the pretest probability changes from 24% in Figure 1-4 to 50% in Figure 1-5 without having to construct the latter. We can simply reanchor the straightedge at 50% and run it across the LRs of 2.8 and 0.10 as before, intersecting the posttest probability line at about 73% and 10%.

The third advantage of LRs is that, unlike sensitivity and specificity (which limit the number of test results to just 2 levels, “positive” and “negative”), they can be generated for multiple levels of the diagnostic test result. At each level, the proportion of patients with the target disorder at this level is divided by the proportion of patients who do not have the target disorder at this same level; the result is the LR for this level. This is shown in Table 1-2, in which LRs for 4, 3, 2, and 1 and no positive responses to the CAGE questionnaire are shown (the awkward, infinitely high LR for 4 positive answers can be avoided if 3 and 4 positive answers are combined, generating an LR of 206 for the combination).

The fourth advantage of the LR strategy is that the posttest probability of the target disorder obtained from the first item of diagnostic information (say, a history of ankle swelling) is the pretest probability of that diagnosis for the next item of diagnostic information (say, the physical examination for ankle edema). This example also identifies the problem we always face when we combine diagnostic information from the medical history and physical examination (and chemistry laboratory, and radiology suite!): the results of the medical history and physical examination are not independent from each other. Thus, a patient with a positive history of swollen ankles is far more likely to have pedal edema than a patient with a negative history, and we must either use an LR that considers both of the 2 items as a pair or modify the LR for the second, according to the results of the first. This issue of independence, along with the consideration of the site (primary care or a tertiary hospital) where the examination is carried out, will be taken up in a subsequent background article in this series.

Conclusion

This first background article has described readers’ guides for articles about diagnostic information and has shown how diagnostic data derived from the medical history and physical examination can be assessed for their precision and accuracy. It concludes with a working table (Figure 1-8) and glossary that can be photocopied or clipped. Kept handy, they can help readers study and understand the overviews published in this and subsequent issues of the series on the rational clinical examination.

Author Affiliations at the Time of the Original Publication

Departments of Medicine and Clinical Epidemiology and Biostatistics, Faculty of Medicine, McMaster University, and the Hamilton Civic Hospitals, Hamilton, Ontario, Canada.

Acknowledgments

I wish to thank Andreas Laupacis, MD, David Simel, MD, Richard Deyo, MD, Drummond Rennie, MD, James Nishikawa, MD, Charles Goldsmith, PhD, and the JAMA reviewers for their kind and helpful responses to earlier versions of this background article.

References

Updated Summary on Precision and Accuracy of the Clinical Examination

Original Review

What Is There to Update?

Each of the updates in The Rational Clinical Examination systematically evaluates the newly published literature on the topic, except this one. Updating the Primer requires a different approach to fulfill the original promise that the series would address methodologic concerns beyond precision and accuracy. What we will do is take a very utilitarian approach, driven by the topic updates themselves. The updates and our own lectures on the rational clinical examination unearthed topics that we need to address. Rather than conducting a systematic review of quality measures, sensitivity, specificity, likelihood ratios (LRs), and a plethora of related topics, we instead provide background information and answers to questions that our own authors required when preparing their reviews and updates.

Of course, the basic premise for diagnosis has not changed since the Primer (or since Thomas Bayes figured it out more than 3 centuries ago):

For the clinical examination, this means we (1) use information about the probability of a target disorder (frequently taken as the prevalence, which is then converted to the prior odds) and then (2) apply the results of symptoms or signs (in the form of an LR). After applying the LR associated with various symptoms and signs, we get the posterior odds of disease. The probability of disease increases when a clinical finding is more likely in a patient with the target disorder (reflected by an LR > 1). The probability of disease decreases when a clinical finding is more likely to occur in a patient without the target disorder (reflected by an LR < 1). The resultant probability becomes the “posterior” probability because the prior probability is established first and then modified with information from the medical history and physical examination quantitatively expressed in the form of the LR.^∗ Keeping the simple equation in mind focuses the goal of The Rational Clinical Examination series articles on providing all the data needed to solve the posterior odds equation.

Why LRs?

In the Primer, we emphasized the role of the univariate LR for clinicians. The term univariate means the results for 1 finding, without regard to the findings of other historical or clinical features. We chose this route for a variety of reasons, most important being its fundamental property that allows clinicians to apply the values to individual patients in a consistent pattern. LRs always convey the same information—they quantify the change in odds of disease for a particular test result. By tradition for dichotomous test results, we call the LR associated with a positive test the LR+ (positive LR), whereas the LR associated with a negative test is the LR– (negative LR). In either case, the actual LR value is related to the change in likelihood that the patient has the disease of interest. Thus, there can be no confusion, as is sometimes the case when physicians become overwhelmed with how to translate positive predictive value, true-positive rate, false-positive rate, negative predictive value, true-negative rate, or false-negative rate into a change in the likelihood of disease for an individual patient.

Many clinicians feel more comfortable with the terms and. However, these values in and of themselves have little application to the clinical setting. Sensitivity and specificity are values that apply to a screening test result before we know whether the patient has the target disorder. So which result do we use at the bedside? Sensitivity applies only to patients with disease, whereas specificity applies only to patients without disease. Because we use screening tests precisely because we do not know about the presence or absence of disease, how do we decide whether the value of sensitivity or the value of specificity applies to our patient? The simple answer is that we do not know. If we do know which result applies to our patient, then, by definition, we know the disease status, and the results of screening tests lose relevance. The true value of an LR comes from its mathematical definition that combines the values of sensitivity and specificity, making it applicable to each patient before we know whether disease is present or absent.

When evaluated in combination, the sensitivity and specificity are the building blocks of the LR for tests that are dichotomous (eg, “positive” or “negative,” “present” or “absent”). The LR for a positive result is sensitivity/(1 – specificity), whereas the LR for a negative result is (1 – sensitivity)/specificity. But what happens when a screening test has more than 2 outcomes (Table 1-3)?

Traditional laboratory tests are measured on continuous scales, where the result intervals have a mathematical meaning, but the clinician could not possibly know the LR for every outcome. A clinical laboratory reports the raw result, along with a designator for whether the result is “high,” “normal,” or “low.” The report takes the raw value and transforms it to an ordinal scale, making it easier for clinicians to review a large amount of data. When there are more than 2 outcomes of a screening test, sensitivity and specificity cannot be directly calculated, so the clinician must rely on LRs that are usually given for ordinal results.

A simple quantitative explanation helps explain why the sensitivity and specificity lose meaning when there are more than 2 screening test results. The presence of a third heart sound (S3) suggests left ventricular (LV) systolic dysfunction. Sometimes, the clinician is uncertain whether the sound is present. To illustrate this point, we can make up some data that might apply to the clinician's interpretation of the S3 compared with a reference standard echocardiogram that quantified the LV function (Table 1-4).

We can describe the sensitivity of the S3 as 30/(30 + 5 + 10) = 0.68 and the specificity as 50/(5 + 10 + 50) = 0.77. Although this may seem straightforward, closer inspection reveals some problems with that interpretation. First, the treatment of the “uncertain” results lacks consistency. For calculating the sensitivity, we “count” an uncertain S3 as if it were actually absent. But the clinical reality was that the physician could not state with certainty whether it was present or absent. When we calculate the specificity, we do the exact opposite and count the “uncertain” outcomes as if they were “positive.” How can one “uncertain” finding be considered “positive” for sensitivity but “negative” as specificity? This dual treatment creates problems that become even more pronounced as the number of results increases beyond 3 outcomes.

Second, even if we believed that the sensitivity and specificity captured the meaning of an S3 that is either present or absent, how do we describe the results for “uncertain?” Sensitivity provides an inadequate definition because sensitivity is the value that describes the percentage of patients with an abnormal result among all those with disease and “uncertain” is neither abnormal nor normal. A similar argument applies to the specificity, so that neither sensitivity nor specificity offers a reasonable description of the value of an uncertain result. The constructs just do not apply to a test result that is neither completely normal nor completely abnormal. The LR provides a way to describe not only the positive and negative results but also those that are uncertain.

At a fundamental level, the LR takes a given screening test result and for that outcome tells us the ratio of those with disease to those without disease. So once we know which row of the table a patient belongs in according to their test result (S3 present, S3 uncertain, or S3 absent), the LR tells us the likelihood that the patient will come from the first column vs the second column. We can calculate an LR for every row of an r × 2 table (where r represents the number of rows) (Table 1-5).

Thus, when we hear an S3 in the patient, we apply the value 8.7, which makes LV systolic dysfunction much more likely. When we feel confident that an S3 is absent, the likelihood of LV systolic dysfunction decreases. However, when we are “uncertain,” the LR we apply is 0.72, a value that approaches 1 and suggests that the “uncertain” result should not have a large effect on our estimate of the likelihood of disease. Oftentimes, it is useful to know that “uncertain” really means “not much information” with an LR approaching 1.

Isn’t All the Information in the Patient's Medical History?

We now need to address a common belief that the physical examination is not particularly helpful and, at best, only confirms the historical findings and symptoms. Oftentimes, a clinician takes a patient's medical history and makes a diagnosis before performing a physical examination. This process, although sometimes successful, leads to the inference that the physical examination was unnecessary. For a simple reason, the inference is not true: the physical examination begins from the moment the clinician meets a patient and before the patient utters a word! We observe body language, the patient's gait, vital signs (eg, tachypnea), and physical deformities, and we judge the acuity of illness. These findings derived from visual observations may be hard to quantify (eg, a sense that the quiet, sullen patient might be depressed), although most clinicians recognize the huge amount of information they collect in the first few moments of a patient interaction. Because describing and measuring the influence of our overall observations is difficult, researchers often overlook the clinical gestalt.

One way of isolating the clinical gestalt is to evaluate whether we can make a diagnosis in the absence of directly observing a patient. A symptom checklist (but not the patient's medical history) can be obtained through a completed patient self-administered questionnaire. Sometimes, we can infer a diagnosis from such questionnaires with our impression uncontaminated by physical findings, but the diagnosis typically requires confirmation obtained through a patient interview or physical examination. The ability to disentangle the history from the physical examination findings is often an illusion, leading to the inference that the patient's medical history (symptoms) dominates the clinical diagnostic process over the physical examination (signs).

The Pretest Probability

The most important part of the clinical examination and the resulting diagnosis is typically not the symptoms or signs—it is the pretest probability, transformed to the prior odds, that dominates the equation. Simply put, if a condition is highly unlikely (or vice versa), then the presence or absence of any addition findings will typically not change things. As a corollary, when the probability of a target condition is not so certain, the effect of the signs and symptoms on the prior probability creates a potentially bigger effect.

So, where does the pretest probability come from? We establish the pretest probability in the course of our clinical examination, and that creates a bit of a problem (for both researchers and clinicians). In other words, as we learn more about the patient's medical history, symptoms, and signs, we orient our approach to a narrower spectrum of disease possibilities. This approach requires that we “waste” a few findings to establish the pretest probability. For example, most patients we examine do not have sinusitis, and we do not ask questions about symptoms related to sinusitis, nor do we transilluminate the sinuses during the course of a clinical examination unless we have a suspicion of the disease. We might constrain our evaluation for sinusitis to patients who claim nasal stuffiness, nasal discharge, or maxillary facial discomfort or who come right out and state, “I think I have a sinus infection.” Each of these findings would prompt an appropriate evaluation for sinusitis and in a research study create the “entrance criteria.” Thus, when we refer to the pretest probability of sinusitis, we most likely are referring to the prevalence of sinusitis among patients with any of those findings rather than to the prevalence of sinusitis among all patients in general. This pretest probability becomes the value we use in the equation and the anchor for applying other symptoms and signs we uncover during our clinical examination.

The establishment of the pretest probability is the problem most learners fear, representing their main “excuse” for not using the concepts in The Rational Clinical Examination. Frequently, learners claim “lack of experience.” When existing studies adequately describe their study population, the pretest probability is not difficult to understand. Experience becomes more valuable when the literature is less clear, and perhaps this is part of the “art” of the clinical examination. Trainees may be quite good at estimating the pretest probability of common conditions. However, both trainees and experienced clinicians tend to overestimate the prior probabilities of less common diseases. Trainees express discomfort when estimating the prior probability because (1) they do not practice quantifying and then validating their clinical impression and (2) they may recall their own cases in which they pursued an unlikely diagnosis for a seemingly “classic” presentation, only to find that the disease was not present. Although the second reason emanates from overlooking the importance of prior probability, it requires a reassessment of the role of symptoms and signs.

What Is a “Good” Symptom or Sign?

The presence of a “good” symptom or sign creates a large effect on the probability, convincing the clinician that the target condition is much more likely to be present than the prior probability suggests. The suggestion that some prespecified LR threshold defines a good clinical finding for all disease is a myth so persistent that it represents a medical urban legend. Some researchers and clinicians define a “good” test result as that associated with an LR greater than 10 or an LR less than 0.1, but these results do not have intrinsic properties that are the sine qua non of high value. For example, a pretest probability of 10% and positive test with an LR = 10 generates a posttest probability of 53%; this is a big increase in the probability of disease but hardly an increase that clinches the diagnosis. Furthermore, this is a similar posttest probability that follows from a disease with a pretest probability of 20% and a positive test with an LR = 5. Thus, although positive test results are increasingly powerful as the LR increases and negative results are increasingly valuable as the LR decreases, the efficiency of the finding in making a diagnosis depends on the pretest probability.

When considering that multiple symptoms and signs are interpreted together, individual findings with much less impressive LRs alone (eg, LR+, 2-5; or LR–, 0.25-0.50) could prove useful when used in combination. If no LR threshold automatically qualifies a result as good, is there a way to compare the efficiency of different clinical findings?

A positive clinical finding with the highest LR+ or a negative finding with the lowest LR– will always have the greatest effect on posttest probability. Unfortunately, clinicians discover that a list of symptoms and signs for an individual patient sometimes simultaneously yields outcomes both suggesting (positive results) and pointing away from (negative findings) a target disorder. There is a way, though, to make sense of this. Rank ordering the LR+ associated with each result, along with the reciprocal of the LR– (1/LR–), reveals the single “best” clinical finding for a target condition. The value with the highest LR+ or 1/LR– is the single best symptom or sign result. A single symptom or sign may be useful when present (high LR+) or absent (small LR–). Unfortunately, most symptoms and signs will not produce both the best findings when positive and also the best when it is negative. For example, a clinical sign may have a low LR– when negative, whereas a positive result may have an LR+ that approaches 1. Creating a mental list of LR and 1/LR– for a variety of symptoms and signs is not easy. Some clinicians want to identify the single finding that overall is the most likely to give them the right answer (ie, positive when the patient has disease and negative when the patient is not affected).

The diagnostic odds ratio (DOR) creates a single measure of accuracy that tells us which symptom or sign is most likely to correctly classify a patient as having the target disorder or not.¹ The DOR is not difficult to calculate, as the DOR = LR+/LR–. The more accurate the symptom or sign, the higher the DOR. So when faced with a table of data on many clinical findings in which none distinguishes itself as the overwhelming favorite, the clinician should choose the finding with the highest DOR. Unfortunately, the DOR cannot be used like the LR for estimating the probability of a diagnosis, but it can help us choose the symptoms and signs of higher utility so that we can ignore those of lesser value. At this point, the skeptical reader might accept that there is a method for identifying better symptoms and signs in terms of their overall measurement properties (through the DOR) and better results applicable to individual patients (through the LR). However, a remaining question might be, How confident can I be that the symptoms and signs I think are the best really are the best?

The Confidence Interval

When The Rational Clinical Examination series began, we presented likelihood results as single point values as if they completely described a clinical finding—they do not. Like all statistical parameters, an LR has an associated confidence interval (CI) that helps us decide whether the data are sufficient for us to infer usefulness. These CIs are important because they provide transparency. An optimistic LR suggests a promising clinical finding, but a broad CI dampens the enthusiasm by implying that a small sample size accounts for some certainty. We are particularly cautious when the 95% CI includes 1 because LR values of 1 add no information to the pretest probability. Broad CIs around LR–, even when they do not include 1, are a particular problem. Because the LR– values are constrained between 0 and 1, a broad CI seems less of a problem than the broad CI around a high LR+. To compare the relative findings, the clinical reader can use the technique we described above (ie, taking the value 1/LR–) for comparing the breadth of the CIs of negative to positive LRs.

Some readers will be surprised that there are different methods that yield slight (but clinically unimportant) differences in CIs. We prefer the easiest computational method that also works well in spreadsheets.² One situation presents problems for researchers and clinical readers alike: what do we do when one cell of the 2 × 2 table is 0? When any single cell has a 0 value (typically, the cells for false positive or false negatives), adding 0.5 to each cell of the 2 × 2 table allows calculation of useful CIs.³ A sensitivity of 100% yields an LR– of 0, with the LR upper 95% CI obtained after adding 0.5 to each cell. A specificity of 100% yields an LR+ that is not calculable (∞), so we report both the LR+ and CI obtained after adding 0.5 to each cell. Although high-quality studies report both the sensitivity and specificity of clinical findings, not all of them calculate the LRs for us. When researchers provide the actual numbers of affected and unaffected patients, together with the sensitivity and specificity, we can generate the LRs and 95% CIs. Although it is sometimes easy to calculate CIs from individual research reports, meta-analysis offers us an even better way of describing the LRs of findings evaluated across several studies.

Meta-analysis

Meta-analysis of symptoms and signs combines the results described across several studies and summarizes them to get a single estimate and CI. Although some statisticians have a high degree of skepticism about the appropriateness of combining LRs, we take the position that summarizing results provides clarity for clinicians that at the very least allows them to assimilate data and decide whether a symptom or sign is useful, useless, or uncertain.

An important part of meta-analysis requires the investigator to make decisions about the appropriateness of combining data. Although statisticians often suggest a purely statistical approach (ie, studies that have statistically heterogeneous results should not be combined), we take a more pragmatic approach similar to that espoused by other clinical diagnosticians.⁴ First, we evaluate whether the universe of published studies represents the universe of patients for whom the target condition might be considered. When the studies reflect the population of patients for whom the symptoms and signs apply, we prefer to try combining the LRs. On the other hand, when studies use various definitions of disease or different thresholds for the symptoms and signs, we cannot combine the results in a meaningful way. When we cannot combine the results, we present ranges for the LRs. Second, we consider our target audience to be clinical readers. For a condition that might have a very different LR among different populations of patients (eg, findings for appendicitis among children vs geriatrics patients), we avoid combining results or we at least show how they vary. Part of this approach requires common sense, and part of this is statistical, in which we examine the outlier results to deduce whether there is anything recognizable that accounts for the variant LR findings. Third, we examine the actual results with their CIs after we combine the data. We always use random-effects measures for generating the LR and CIs, rather than the fixed-effects approach. Random-effects measures generate broader CIs than the fixed effects, providing at least some assurance that we are not overstating the importance and confidence in our findings. If a study is a statistical LR outlier, we still include it in the combined data if it does not make a large clinical difference in the LRs. We suggest that the clinician use clinical judgment when deciding whether 2 LRs yield clinically important differences in the posttest probability. For example, for a pretest probability of 30%, an LR of 5.4 produces a posttest probability of 70%, whereas an LR of 3.5 produces a posttest probability of 60%. These LRs “look” different, but a clinician might take a similar action for a posttest probability of 70% vs 60%. Thus, the 2 LRs could be statistically different but provide clinically similar results. We always provide the results from each study, and astute readers can decide from the point estimates and CIs whether they believe a finding is useful or useless.

More statistically experienced readers may recognize that meta-analysis of LRs differs from what they expect. Statisticians, when they accept meta-analysis of diagnostic tests at all, prefer summarizing the DOR as a global measure of test performance. We take a different approach because summarizing the DOR gives clinicians a value that they cannot use for individual patients. Although we do sometimes provide summary measures of the DOR, the summary measures of the prevalence of disease (pretest probability) and the LR are the values needed for solving the equation for posttest probability. Sometimes, we encounter studies that only provide sensitivity data. What do we do with studies that are case series of patients with disease and that do not have specificity values?

“Sensitivity-Only” Studies

When conditions are less common, investigators recognize that enrolling consecutive patients at risk for the target disorder creates a study population overwhelmed by those without disease. This approach is costly and takes time, and the small number of patients with disease leads to broad CIs around the sensitivity and LR–. The alternate approach of studying only patients with disease so that sensitivity can be defined is pragmatic, and it may be the best the investigator can do. These studies typically come from a narrow spectrum of diseased patients, and often, the clinical finding is recorded among patients when the clinician knows that disease is present. In addition to understanding the potential biases in the data, we must understand the inferences made from the sensitivity of symptoms and signs without specificity values. The goal of sensitivity studies is to identify a group of symptoms and signs that would unlikely all be negative in a patient with the target condition.

Symptoms and signs with high sensitivity are less likely to be negative in patients with disease. When presented with sensitivity data by itself, clinicians will count the number of absent findings in their patients and deduce that those with normal findings on multiple high-sensitivity symptoms and signs will be unlikely to have disease. For example, suppose we identify 2 symptoms and 1 sign, each of which has a sensitivity of 85% for the target condition. That means that each finding would be absent in 15% of patients with disease; all 3 would be absent in fewer than 1% of patients (0.15 × 0.15 × 0.15).

How Do We Use All the Symptoms and Signs?

Among several reasons for preferring LRs as our common statistical parameter, rather than the individual sensitivity and specificity values, the ability to multiply likelihood results from several findings is the most alluring. Unfortunately, a crucial assumption is not often fully addressed—sequentially multiplying LRs requires that the symptoms and signs be independent of one another.

Let us explain the independence concept with a simple example. Suppose you conduct a study of chest pain symptoms as a predictor of acute ischemia and you categorize words as having “physical” or “emotional” connotations. Words that describe location and radiation would be physical (eg, “center of the chest,” “in the neck”), whereas words that describe the interpretation of pain would be emotional (eg, “suffocating,” “crushing”). You decide to record whenever a patient refers to an “elephant” in describing their discomfort as emotional as in, “It felt like an elephant stepped on my chest.” We suspect it is obvious that a patient who is “elephant-positive” is experiencing crushing pain, but if they report they are having “crushing pain that feels like an elephant on my chest,” should we report the findings separately for “crushing positive” and “elephant positive?” Multiplying the LRs together for “crushing,” “elephant-like” discomfort probably overstates the importance, producing a posttest odds ratio that is too high because elephant-like pain is not independent of crushing pain. Although common sense might work as an initial judge of independence, common sense should not be the only arbiter of independence. What should you do when presented with an array of findings for many symptoms and sign without any assessment of independence?

To make teaching and performing the medical history and physical examination more efficient and accurate, we want parsimony. By “parsimony,” we mean the fewest number of symptoms and signs that yield the most accurate information. Parsimonious examinations force teachers to teach only the most relevant parts of the examination, allowing students to spend more time learning what is important while eliminating wasteful maneuvers. Of course, some of this waste is in eliminating maneuvers that do not work well. For example, a Rinne test is interesting to teach, but it does not add useful diagnostic information to the symptom of “decreased hearing” reported by the patient.⁵ We eliminate additional wasted effort when we discard nonindependent findings.

A parsimonious examination should mathematically make us more accurate because a “complete” medical history and physical examination almost certainly produces nonindependent findings. “Positive” nonindependent findings confuse us and distort our probability estimates, typically making us infer a higher probability of disease than is justified. Most authors of The Rational Clinical Examination articles emphasize no more than 3 to 4 findings, even when additional symptoms and signs have useful LRs. Narrowing down the number of recommended findings requires “face validity,” by which we mean using common sense to recommend the items with the best, seemingly independent LRs. When we take this approach, experienced clinicians then use semiquantitative reasoning and deduce that the more findings present, the more likely the patient has disease (or vice versa).

When clinicians want to incorporate the results of diagnostic studies into their decision making, they can take 3 approaches to prevent errors created by lack of independence.⁶ Performing the clinical examination and then using only one single history or physical examination finding to adjust the prior odds will guarantee there is no problem with independence. (Of course, it also guarantees that the clinician might be ignoring a lot of useful clinical information!) Typically, the clinician will want to use the single finding that has the greatest effect on the prior odds, or the “best” finding that we described earlier. The approach is not difficult since simple math allows you to rank the findings in order from most useful to least useful. Suppose you have 3 findings (A, B, and C) that can each be positive or negative, with the LRs associated with each result shown in Table 1-6. Is the finding that “A” is present more diagnostically useful than “C's” absence? To determine this, you can rank order these by comparing the LR for the positive results to 1/LR for the negative results. Table 1-6 shows the relative value each of the findings. If your patient had “A” absent, “C” present, and “B” present, then you would multiply the prior odds by the LR associated with the outcome for test “B” (LR = 5.0) because it had had the most useful outcome for that individual.

Although the above result removes any concerns with independence, the clinician must collect many data that ultimately are discarded. At the very least, it is not efficient, and at the worst, important information could be ignored. Not surprisingly, this approach lacks appeal because it ignores the way most clinicians incorporate many bits of information into their decision making.

Clinical researchers must analyze their data in a multivariate way to help clinicians. By “multivariate,” we mean that they must analyze combinations of findings so that there is less concern about independence. This can involve one of 2 general approaches. The easiest approach is to take the medical history and physical examination findings and perform logistic regression. Logistic regression takes a number of individual variables and determines their importance in predicting whether disease is present or absent. In the first strategy for assessing independence, logistic regression identifies variables that lack independence and that can be eliminated as redundant. In our example above, if all patients with wheezing were also dyspneic, then the finding on the “variable” dyspnea might be unimportant once we know the wheezing status. The logistic regression approach would identify this as being nonsignificant, and the investigator would suggest we concentrate our efforts at assessing for wheezing. Used as a “data-reduction” step to achieve parsimony, the clinician would use the simple, univariate LRs for any finding identified as being independently useful in the logistic model. This approach has a lot of appeal because it identifies the important and useful variables for the clinician, and it does not require that they understand the logistic model itself, because the univariate LRs are used. However, in using the simple, unadjusted LRs, we ignore the relationship between the various clinical findings in favor of simplicity.

The β parameters of a multivariate logistic analysis describe the relative importance of symptoms and signs. From algebra, you might remember the equation for a straight line is y = mx + b. The m in the equation is the slope, and it quantifies how a change in x affects y.^∗ A logistic model works similarly, except that now, rather than having 1 x, we have several symptoms and signs that we evaluate all at once. The equivalent of m in the logistic model now represents the β parameter, which is the odds ratio associated with each symptom or sign; the higher the β parameter, the more important the finding. When investigators provide us the actual multivariate models, we can put the results of our own patient's clinical examination into the model, and the outcome is the individual patient's actual probability of disease.

The Fuss About Precision

The Primer states, “for an item of the clinical history or physical examination to be accurate, it first must be precise.” By precision, we imply that 2 or more observers agree on the presence or absence of a finding in a patient who experienced no clinical changes.^∗

When we measure precision, describing the percentage of time that 2 observers agree on a symptom or sign fails to consider simple luck. Instead of reporting simple agreement, investigators report precision as the agreement beyond that attributable to chance. For dichotomous findings (“yes” vs “no” or “present” vs “absent”) compared between 2 observers, we quantify this agreement beyond chance with the κ statistic.^† The κ statistic varies from –1 (perfect disagree-ment) to 0 (chance agreement) to +1 (perfect agreement).

Suppose we are interested in whether a third heart sound identifies patients with LV systolic dysfunction. It is easy to imagine that a cardiologist might be better at identifying this correctly than a generalist internist, suggesting that a κ statistic might show lower agreement beyond chance than if we were comparing 2 general physicians. Should we conclude that a third heart sound is not a good test from the precision between a cardiologist and a general internist? The answer, of course, is no because test accuracy depends on the quality of the observation—the cardiologist might be a better observer than a less experienced clinician. These seemingly imprecise symptoms and signs are potentially useful when certain providers get consistently good results because they represent opportunities for improved performance and accuracy.

A second type of precision is more important for identifying inaccurate findings. Although a low κ between observers points to opportunities for improving, poor intraobserver agreement precludes high accuracy unless the problem can be eliminated. Intraobserver agreement describes whether a clinician gets the same result when assessing a symptom or sign on a patient who is clinically unchanged. For example, when a clinician inquires about unilateral headaches as a symptom for migraines but the patient changes his or her answer, the finding can never be accurate or precise. Although the natural assumption might be to blame the patient for inconsistency, part of poor intraobserver agreement may be attributable to poor technique that can be improved. This is true even when applied to symptoms as reported by the patient because different answers follow when the information is solicited differently (eg, asking the patient a leading question about unilateral headaches vs an open-ended question). But if clinicians cannot assure reliability on their own findings, they will never use the symptoms and signs accurately. If you cannot agree with yourself, the LR results will be random.

A Brief Word About Quality

Every article in The Rational Clinical Examination series and the updates in this book use a standard process for assessing the quality of data. Although the Primer focuses mostly on the sensitivity, specificity, and LR results, it should be clear that narrow CIs around the results do not assure methodologic rigor of the studies that generated the results. At the inception of The Rational Clinical Examination series, the evidence-based medicine movement was in its infancy. An early article in the series heralded its entry into the mainstream thoughts of clinical educators and investigators.⁷ Because standardized approaches had not been developed for assessing the quality of the medical history and physical examination, David L. Sackett, MD, and Charles H. Goldsmith, PhD, agreed on certain characteristics that they asked their reviewers to use when judging quality. The criteria were simplified and summarized in an early article of the series.⁸ Subsequently, several groups have published their criteria for the review of diagnostic accuracy studies, although none address the particular nuances of symptoms and signs.^{9, 10, and 11} Perhaps it is not surprising that many clinical investigators and epidemiologists have reported on a large number of quality measures that describe what seem like innumerable potential biases in diagnostic test studies. Despite the increasing complexity of rating systems and quality measures, the original criteria for reviewing articles have stood the test of time and pragmatism. If anything, we made the process easier and reduced the number of quality levels a reviewer might assign an article. We reviewed the recommendations for diagnostic test studies^{9, 10} and adapted them specifically for studies of the clinical examination.¹² In the early articles appearing in The Rational Clinical Examination series, we assigned Grades for levels of evidence. However, this blurred the distinction between Levels 3, 4, and 5. Because no study accepts Level 5 evidence in making recommendations, we dropped the Grade designation and now report only the Levels as shown in Table 1-7.¹³

Most of the important biases that compromise a study's results follow from the study population not being consecutive, prospective, or independently assessed with an appropriate blindly applied reference standard. By consecutive, we mean that the authors enrolled all patients for whom the target disorder was a reasonable consideration. Independent means that the symptom or sign under study was not used to select patients for the study. means that the symptoms and signs were applied without knowledge of the presence of disease determined by the reference standard, but also that the reference standard was interpreted without knowledge of the study questions. The size of a study (level 1 vs level 2) for quality assessment depends on the disease under consideration. The authors of The Rational Clinical Examination evaluate sample sizes according to their review of the literature because there is no uniform number that determines quality; for example, a large study of thoracic aortic aneurysms might likely not have as many patients as a large study of urinary tract infection in women.

One particular bias, verification bias, deserves special consideration because it can be insidious and have a big effect on the LR. Verification bias occurs when all the potentially eligible patients fail to undergo confirmation of their disease status. Often, this is done for pragmatic reasons. An example might be a study of headache patients that seeks to describe whether asymmetric neurologic findings (eg, weakness) indicating serious intracranial abnormalities were discovered through neuroimaging. Because it would be expensive and impractical to have every patient with headaches undergo imaging, an investigator typically chooses to maximize the chance of finding something by including all patients with asymmetric muscle strength but only a sample of those who are normal. We can highlight the effect of verification bias on the sensitivity, specificity, and LRs, through examining tables of example data. Suppose an investigator reports the findings displayed in Table 1-8.

In the example, the finding looks excellent, with a sensitivity and specificity of 90%. However, because the investigator could not justify the reference standard (eg, neuroimaging on every patient with a headache), the investigative team referred only a sample of those with positive clinical findings (for illustrative purposes, 10%). Had the investigator been evaluating every patient, the findings might have been as shown in Table 1-9.

The data demonstrate that verification bias tends to overestimate sensitivity while underestimating specificity.^∗ When the bias is left unadjusted, the investigator will not recognize that the presence of the finding is actually better than suggested (the adjusted LR+ should be higher), whereas the absence of the finding is not as good as suggested (the adjusted LR– should be closer to 1). Astute investigators will recognize that if they collect complete data on all the potentially eligible patients, the bias is one of the few in diagnostic test research that can be mathematically corrected.

References for the Update