UPJ INSIGHT Urology Research and Patient Understanding: Large Language Models to Generate Layperson Summaries

By: Michael B. Eppler, BA, Keck School of Medicine, University of Southern California, Los Angeles; Conner Ganjavi, BS, Keck School of Medicine, University of Southern California, Los Angeles; J. Everett Knudsen, BS, Keck School of Medicine, University of Southern California, Los Angeles; Ryan J. Davis, BS, Keck School of Medicine, University of Southern California, Los Angeles; Oluwatobiloba Ayo-Ajibola, BS, Keck School of Medicine, University of Southern California, Los Angeles; Aditya Desai, BA, Keck School of Medicine, University of Southern California, Los Angeles; Lorenzo Storino Ramacciotti, MD, Keck School of Medicine, University of Southern California, Los Angeles; Andrew Chen, MD, Keck School of Medicine, University of Southern California, Los Angeles; Andre De Castro Abreu, MD, Keck School of Medicine, University of Southern California, Los Angeles; Mihir M. Desai, MD, Keck School of Medicine, University of Southern California, Los Angeles; Inderbir S. Gill, MD, Keck School of Medicine, University of Southern California, Los Angeles; Giovanni E. Cacciamani, MD, Keck School of Medicine, University of Southern California, Los Angeles | Posted on: 25 Oct 2023

Eppler MB, Ganjavi C, Knudsen JE, et al. Bridging the gap between urological research and patient understanding: the role of large language models in automated generation of layperson’s summaries. Urol Pract. 2023;10(5):436-443.

Study Need and Importance

There is currently a global focus on translating complicated research into simple language for the general public, as recommended by the European Union’s 2014 Clinical Trials Regulation and the “Good Lay Summary Practice” guidelines (2021). Clear and concise summaries are important to ensure understanding by a wide audience. To optimize these summaries, strategies such as shorter sentences, fewer syllables, and less use of passive verbs can be employed. Innovative natural language processors like ChatGPT have the potential to create comprehensive and understandable summaries, but their effectiveness and usefulness should be studied first.

What We Found

ChatGPT-generated patient summary outputs were produced in a short amount of time (less than 20 seconds) with an improvement in multiple readability metrics (Global Readability Score, Flesch Kincade Reading Ease, Flesch Kincaid Grade Level, Gunning Fog Score, Smog Index, Coleman Liau Index, and Automated Readability Index) when compared to both original abstracts and original patient summaries. Furthermore, physicians independently rated the ChatGPT patient summaries with a high correctness rate (>85%) and clarity score.

Limitations

Study limitations primarily relate to uncertainties associated with ChatGPT’s capabilities. Although ChatGPT has been shown to be reliable, there are still questions regarding its consistency and accuracy. Furthermore, while ChatGPT is currently accessible for free, newer and superior versions may eventually become restricted to paying users. It is essential to note that this study’s focus was on urology and urological literature, and therefore its findings should be validated across surgical and medical specialties.

Interpretation for Patient Care

Discussions are ongoing about the various uses of large language models, such as ChatGPT, in patient care. Potential applications include simplifying and customizing discharge summaries and at-home care instructions for patients. Summarizing medical research to improve patient comprehension has already proven beneficial to patient care, and this research reveals a possible tool to assist in that process.