Within a week of its release in November 2022, OpenAI’s ChatGPT large language model (LLM) had already provided millions of internet users with unprecedented access to a transformative artificial intelligence (AI) tool.¹ While prior advanced LLMs had existed for some time, ChatGPT’s novelty was its user-focused interface and conversational communication style—one no longer needed to have a background in computer programming to meaningfully use the technology. The ChatGPT chatbot platform is remarkable for its ability to provide confident-sounding answers to almost any query about almost any topic, including health care and urology. ChatGPT met the needs of our interconnected world, providing us with information that aligned with the persistent desire for instantaneous access.

Health care providers quickly began using this technology and testing its ability to perform rote tasks such as creating call schedules, drafting prior authorization letters to insurance companies, and formatting replies to messages in the electronic medical record. When a global cohort of 456 urologists was surveyed in April 2023 about their LLM use and experiences, 48% of them reported using LLMs in academic practice for tasks such as idea generation.² Nearly 20% of respondents also reported using ChatGPT in clinical practice, primarily for patient education applications. Interestingly, while equal numbers of respondents (29.6%) reported either trusting or not trusting LLMs to provide accurate information, 78% and 56% of respondents believed that LLMs could play an important role in academic and clinical practice, respectively. Considering the increased uptake among urologists, our patients are also likely to begin utilizing these AI tools. It is therefore important for us to understand the accuracy and limitations of current AI chatbot outputs, especially for urologic health care advice.

Table. Summary Table Containing Many of the Published Studies That Have Examined Large Language Models in Urology

ChatGPT (version 3.5) is the most studied AI chatbot model in our field (Table). Common response attributes evaluated by research groups have included appropriateness, comprehensiveness, clarity, and reproducibility. These assessments were performed using a variety of scoring instruments across the different studies. Appropriateness, although defined differently in different studies, is used by the researchers as a surrogate end point which combines an assessment of response accuracy, comprehensiveness, and clarity. Across studies, 52% to 78% of ChatGPT responses were deemed appropriate.^3-5 The most commonly reported reason for judging responses to be inappropriate was the absence of vital information, and response clarity was the least common reason.⁴ Concerningly, the one study which examined response reproducibility within a single chatbot (ChatGPT version 3.5) by posing the same question in 3 independent chat instances found substantial response inconsistency—25% of question sets produced responses with dissimilar appropriateness ratings.³ The LLM was also noted by multiple groups to provide responses at a difficult reading level, rated as college level or higher.^4-6 This body of research highlights that despite the hype and excitement about this new tool, the information currently provided by chatbots to urologic health care queries is generally of low quality and should not be considered actual, reliable medical advice.

Figure. The Chatbot will see you now. This image was created by the authors using ChatGPT 4 with DALL-E plugin, October 16, 2023 version.

One of the major limitations of the current body of research on LLMs in health care is that there does not currently exist a validated assessment instrument by which the models’ responses can be evaluated in a standard and reproducible fashion. Researchers so far have designed their own Likert scales or adapted existing evaluation tools which were originally designed for other purposes (eg, DISCERN, Brief DISCERN, QUEST). Given the rapid adoption and the degree to which LLMs have already become entrenched within society, as well as the potential for real harm from low-quality responses to potentially serious medical questions, there is an urgent need for the development of a standardized evaluation tool that is specifically designed to assess chatbot responses to medical questions. Such a tool will be critical for the performance and interpretation of high-quality future research in this emerging field.

In summary, chatbot responses to urologic health care queries are sometimes appropriate. While they are often written in clear language, they frequently provide information which is either not factual or not comprehensive. Responses are also limited in their practicality by being written at a very high reading level and often lacking actionable instructions for their end users, who may be patients. Nonetheless, it is increasingly apparent that AI and chatbots are here to stay (Figure). We as a field can choose to ignore the proverbial elephant in the room, or we can choose to participate meaningfully in their evaluation and evolution to ensure that future iterations of these tools wield their considerable computational power for healing and not for harm.