Microsoft recently published a study that explores the capabilities and limitations of GPT-4 in radiology.
Working with a radiologist and Nuance, a Microsoft company whose PowerScribe solution is used by more than 80 percent of radiologists in the U.S., the research team created a comprehensive evaluation and defect analysis framework.
Within this framework, the team evaluated GPT-4's ability to process radiology reports, including understanding natural language and generating radiology tasks such as classifying diseases and summarizing findings. For the tasks, the team was careful to emphasize complex and challenging real-world radiology scenarios.
GPT-4 outperforms even human radiologists
The study found that GPT-4 demonstrated new state-of-the-art performance on some tasks, outperforming existing systems by up to ten percent. Although GPT-4 occasionally fails to surface domain knowledge, it has "substantial capability in processing and analysis of radiology text, achieving near-ceiling performance in many tasks," the paper states.
In some cases, the radiology report summaries generated by GPT-4 were even more accurate and provided more detailed findings than the reports generated by experienced radiologists.
Another promising aspect of GPT-4 is its ability to automatically structure radiology reports, which are often complex and unstructured. Studies have shown that structured reports can improve standardization and consistency in the description of disease.
This facilitates interpretation by other healthcare providers and makes them more searchable for research and quality improvement initiatives.
GPT-4 could help improve real-world data (RWD) and its use for real-world evidence (RWE) to complement clinical trials and accelerate the translation of research into clinical practice.
Are large language models better specialists?
The results are encouraging, but need to be confirmed by further research and clinical trials, the researchers wrote. "When used with human oversight, GPT-4 also has the potential to transform radiology by assisting professionals in their day-to-day tasks."
Earlier in August, Microsoft researchers published study results showing that generalist AI models pre-trained with large amounts of data, such as GPT-4, can outperform specialized medical models.