Attention: Restrictions on use of AUA, AUAER, and UCF content in third party applications, including artificial intelligence technologies, such as large language models and generative AI.
You are prohibited from using or uploading content you accessed through this website into external applications, bots, software, or websites, including those using artificial intelligence technologies and infrastructure, including deep learning, machine learning and large language models and generative AI.
UPJ INSIGHT Comparison of ChatGPT and Traditional Patient Education Materials for Men’s Health
By: Yash B. Shah, BS, Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, Pennsylvania; Anushka Ghosh, BS, Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, Pennsylvania; Aaron R. Hochberg, BS, Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, Pennsylvania; Eli Rapoport, MD, NYU Langone, New York, New York; Costas D. Lallas, MD, Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, Pennsylvania; Mihir S. Shah, MD, Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, Pennsylvania; Seth D. Cohen, MD, MPH, NYU Langone, New York, New York | Posted on: 19 Jan 2024
Shah YB, Ghosh A, Hochberg AR, et al. Comparison of ChatGPT and traditional patient education materials for men’s health. Urol Pract. 2024;11(1):87-94.
Study Need and Importance
As patients increasingly seek health information online, especially for sensitive topics like sexual health, we sought to investigate a potential role for artificial intelligence (AI). Our study explores whether ChatGPT might carry risks similar to social media, where misinformation has been consistently documented in urology content. We evaluate ChatGPT’s ability to enhance accessibility and quality of men’s health education and compare its responses with the gold-standard Urology Care Foundation™ (UCF) resources.
What We Found
ChatGPT generated longer and more complex responses compared to UCF, indicating a higher volume of information and reduced understandability. Unfortunately, both ChatGPT and UCF responses exceeded the recommended sixth to eighth grade readability level. When prompted to adjust for user literacy, readability improved significantly, even surpassing UCF for many topics (Figure). Quality analysis showed that ChatGPT responses paralleled UCF in accuracy and demonstrated better comprehensiveness. This suggests potential value in AI’s ability to increase accessibility of provider-written content to laypersons when the technology is used appropriately.
Limitations
The study sample focused on UCF articles and ChatGPT responses and may not represent all physician-created online resources. Readability formulas couldn’t account for multimedia content, such as videos or images. Additionally, the evaluation of accuracy, quality, and comprehensiveness relied on subjective assessments by 2 urologists, introducing potential bias.
Interpretation for Patient Care
The study underscores the poor readability of online men’s health resources, including those from a trusted patient platform like UCF. However, it shows promise in using AI-enabled improvement by prompting ChatGPT to adapt to the appropriate literacy level. AI models have the potential to play a crucial role in patient education but need further refinement. Health care providers and organizations could leverage AI to simplify language and create accessible educational materials. Future iterations of AI models should focus on specialized medical information and prioritize short, accessible advice. Collaboration between expert stakeholders, medical associations, and AI companies can lead to improved patient-oriented tools, ultimately benefiting patient education in men’s health and other fields.
advertisement
advertisement