A recent study published in the journal Radiology found that radiologists were better than artificial intelligence tools at identifying or ruling out three common lung diseases from more than 2,000 chest X-rays. Radiologists surpassed artificial intelligence in accurately detecting three common lung diseases from chest X-rays, according to a study in the journal Radiology. The AI tool, while sensitive, produced more false positives, making it less reliable for autonomous diagnosis but useful for second opinions.
In a study of more than 2,000 chest X-rays, radiologists outperformed artificial intelligence in accurately identifying the presence or absence of three common lung diseases, according to a study published September 26 in Radiology, the journal of the Radiological Society of North America (RSNA).
The role of radiography
"Chest X-rays are a common diagnostic tool, but correctly interpreting the results requires extensive training and experience," said lead researcher Louis L. Plesner, MD, a resident physician and doctoral researcher in the Department of Radiology at Herlev and Gentofte Hospital in Copenhagen, Denmark.
Although there are FDA-approved AI tools on the market to assist radiologists, the clinical application of deep learning-based AI tools for radiological diagnosis is still in its infancy. "While more and more artificial intelligence tools are being approved for use in radiology, there is an unmet need to further test these tools in real clinical scenarios," said Dr. Plesner. "AI tools can assist radiologists in interpreting chest X-rays, but their actual diagnostic accuracy remains unclear."
(A) Posteroanterior chest radiograph of a 71-year-old male patient who was referred for radiologic examination because of progressive dyspnea showing bilateral fibrosis (arrow B). Posteroanterior chest radiograph of a 31-year-old female patient who was referred for radiological examination because of a one-month-old cough shows subtle air gap opacity at the right heart border (arrow). (C) Anterior chest radiograph of a 78-year-old male patient referred after central venous catheter placement shows a right skin fold (arrow). (D) Posteroanterior chest radiograph of a 78-year-old male patient referred for exclusion of pneumothorax shows a very subtle pneumothorax (arrow) on the right apex. (E) Posterior anteroposterior chest radiograph showing chronic rounding of the costophrenic angle (arrow) in a 72-year-old male patient who was referred for radiological examination for no particular reason. (F) Anterior chest X-ray of a 76-year-old female patient referred because of suspected congestion and/or pneumonia shows a very small effusion on the left side of the chest (arrow) that was missed by all three AI tools that analyze pleural effusions on anterior chest X-rays. Source: Radiological Society of North America
Research results
Dr. Plesner and the research team compared the performance of four commercially available artificial intelligence tools and 72 radiologists in interpreting 2,040 adult chest X-rays taken consecutively over two years at four hospitals in Denmark in 2020. The median age of the patient population was 72 years. Among the chest X-ray samples, 669 (32.8%) had at least one target finding.
Chest X-rays were evaluated for three common findings: air cell disease (a pattern of chest X-rays caused by things like pneumonia or pulmonary edema), pneumothorax (collapse of the lung), and pleural effusion (accumulation of water around the lungs).
The AI tool had a sensitivity of 72% to 91% for airway disease, 63% to 90% for pneumothorax, and 62% to 95% for pleural effusion.
"AI tools showed moderate to high sensitivity comparable to radiologists in detecting airspace disease, pneumothorax, and pleural effusion on chest X-rays," he said. "However, they produced more false-positive results (predicting disease when it was not present) than radiologists, and their performance degraded when multiple findings and smaller targets were present."
Predicted value comparison
For pneumothorax, the AI system's positive predictive value - the probability that a patient who screened positive actually had the disease - ranged from 56% to 86%, compared with 96% for radiologists.
"AI performed worst at identifying pneumothorax disease, with a positive predictive value between 40 and 50 percent," Dr. Plesner said. "In this difficult sample of elderly patients, the AI predicted non-existent air gap disease 5 to 6 times out of 10. You couldn't have an AI system working on its own at that speed."
The goal of radiologists is to strike a balance between the ability to detect and rule out disease, both to avoid significant disease being overlooked and to avoid overdiagnosis. "AI systems appear to be very good at detecting disease, but not as good as radiologists at determining the absence of disease, especially when chest X-rays are more complex," he said. "Too many false-positive diagnoses can lead to unnecessary imaging, radiation exposure and increased costs."
Most studies typically tend to evaluate AI's ability to determine the presence or absence of a single disease, which is much easier than in real-life situations where patients often suffer from multiple diseases. In many of the previous studies that claimed AI was superior to radiologists, the radiologists only viewed the images without knowledge of the patient's clinical history and previous imaging studies. In daily practice, a radiologist's interpretation of an imaging examination is a composite of these three data points. The researchers speculate that the next generation of AI tools could become even more powerful if they could also perform this kind of synthesis, but no such system currently exists.
"Our research shows that radiologists generally outperform AI in real-world scenarios with a wide variety of patients," he said. "While AI systems can effectively identify normal chest X-rays, AI should not make diagnoses autonomously."
Dr. Plesner noted that these artificial intelligence tools can increase radiologists' confidence in their diagnosis by taking a second look at chest X-rays.