This JAMA Guide to Statistics and Methods discusses the use of free-response receiver operating characteristic curves to test the accuracy of computer algorithms to detect the localization of disease on pathology slide images.
In a machine learning study, Ehteshami Bejnordi et al1 evaluated and compared the ability of 32 computer algorithms to identify the presence and location of metastatic lesions on pathology slide images of sentinel axillary lymph nodes from women with breast cancer. The authors used free-response receiver operating characteristic (FROC) curve analysis to assess diagnostic and localization accuracy. They found that the best algorithm performed similarly to a pathologist working without a time constraint (Figure 11).
FROC Curves of the Top 5 Performing Algorithms vs Pathologist WOTC for the Metastases Identification Task (Task 1) From the CAMELYON16 Competition
CAMELYON16 indicates Cancer Metastases in Lymph Nodes Challenge 2016; CULab, Chinese University Lab; FROC, free-response receiver operator characteristic; HMS, Harvard Medical School; MGH, Massachusetts General Hospital; MIT, Massachusetts Institute of Technology; WOTC, without time constraint. The range on the x-axis is linear between 0 and 0.125 (blue) and base 2 logarithmic scale between 0.125 and 8. Teams were those organized in the CAMELYON16 competition. Task 1 was measured on the 129 whole-slide images in the test data set, of which 49 contained metastatic regions. The pathologist did not produce any false-positives and achieved a true-positive fraction of 0.724 for detecting and localizing metastatic regions.
Free-response operating characteristic analysis assesses the ability of a medical test to identify abnormalities on an image. Examples include identifying tumors in radiographs or foci of malignancy on histological slides. There are similarities between FROC analysis and the more commonly used receiver operating characteristic (ROC) curve analysis.2,3 Conventional ROC curves, however, evaluate the accuracy of a test for detecting the presence or absence of disease but do not evaluate whether a test correctly identifies the location.
WHY ARE FROC CURVES USED?
When trying to characterize how well a test determines the location of disease and decide if one test is better than another at this task, it is necessary to account for variations in the appearances of the lesions and the fact that lesions may be located anywhere on an image. One approach that can be used for this purpose is called free-response analysis, meaning that a person or machine reading the image assesses the entire image, marks the portions of the image that look abnormal and may be diseased, and makes a determination regarding the probability that the marked areas represent disease. A single image may have several locations with the disease entity.
When performing this sort of analysis, a rating (either continuous or ordinal) is given regarding the likelihood that there is disease in any marked spot. Lesions identified ...