Skip to Main Content

Using Free-Response Receiver Operating Characteristic Curves to Assess the Accuracy of Machine Diagnosis of Cancer

Chaya S. Moskowitz

INTRODUCTION

This JAMA Guide to Statistics and Methods discusses the use of free-response receiver operating characteristic curves to test the accuracy of computer algorithms to detect the localization of disease on pathology slide images.

In a machine learning study, Ehteshami Bejnordi et al1 evaluated and compared the ability of 32 computer algorithms to identify the presence and location of metastatic lesions on pathology slide images of sentinel axillary lymph nodes from women with breast cancer. The authors used free-response receiver operating characteristic (FROC) curve analysis to assess diagnostic and localization accuracy. They found that the best algorithm performed similarly to a pathologist working without a time constraint (Figure 11).

FIGURE 11

FROC Curves of the Top 5 Performing Algorithms vs Pathologist WOTC for the Metastases Identification Task (Task 1) From the CAMELYON16 Competition

CAMELYON16 indicates Cancer Metastases in Lymph Nodes Challenge 2016; CULab, Chinese University Lab; FROC, free-response receiver operator characteristic; HMS, Harvard Medical School; MGH, Massachusetts General Hospital; MIT, Massachusetts Institute of Technology; WOTC, without time constraint. The range on the x-axis is linear between 0 and 0.125 (blue) and base 2 logarithmic scale between 0.125 and 8. Teams were those organized in the CAMELYON16 competition. Task 1 was measured on the 129 whole-slide images in the test data set, of which 49 contained metastatic regions. The pathologist did not produce any false-positives and achieved a true-positive fraction of 0.724 for detecting and localizing metastatic regions.

Five ascending free-response receiver operator characteristic curves are graphed to depict the performing algorithms vs pathologist WOTC The horizontal axis depicts a mean no. of false-positives per whole-slide image with a linear range between 0 to 0.125 and base 2 logarithmic scale between 0.125 and 8. The vertical axis depicts a metastases detection sensitivity with a range between 0 to 1.0. The Diagnostic machine-learning algorithm teams are plotted in a linear ascending curve in the graph.
View Full Size||Download Slide (.ppt)

Free-response operating characteristic analysis assesses the ability of a medical test to identify abnormalities on an image. Examples include identifying tumors in radiographs or foci of malignancy on histological slides. There are similarities between FROC analysis and the more commonly used receiver operating characteristic (ROC) curve analysis.2,3 Conventional ROC curves, however, evaluate the accuracy of a test for detecting the presence or absence of disease but do not evaluate whether a test correctly identifies the location.

WHY ARE FROC CURVES USED?

When trying to characterize how well a test determines the location of disease and decide if one test is better than another at this task, it is necessary to account for variations in the appearances of the lesions and the fact that lesions may be located anywhere on an image. One approach that can be used for this purpose is called free-response analysis, meaning that a person or machine reading the image assesses the entire image, marks the portions of the image that look abnormal and may be diseased, and makes a determination regarding the probability that the marked areas represent disease. A single image may have several locations with the disease entity.

When performing this sort of analysis, a rating (either continuous or ordinal) is given regarding the likelihood that there is disease in any marked spot. Lesions identified ...

Pop-up div Successfully Displayed

This div only appears when the trigger link is hovered over. Otherwise it is hidden from view.

This site uses cookies to provide, maintain and improve your experience. MH Privacy Center Close