ChatGPT (Generative Pretrained Transformer) is an artificial intelligence (AI) large language model developed to mimic human conversation. The AI tool is able to complete a wide variety of complex tasks (eg, writing original music, language translation, or debugging computer code). While ChatGPT was not specifically designed for medicine, it made headlines when it passed the USMLE (United States Medical Licensing Examination) Step 1, 2, and 3 licensing exams.¹ Unsurprisingly, urologists have been eager to assess how ChatGPT can impact medical education in our field.

ChatGPT did not perform as well on the AUA Self-Assessment Study Program as it did on the USMLE exams. It answered only 42.3% and 30% of the 2021 and 2022 questions correctly, respectively. The AI performed very well on first-order information recall questions and worse on questions with multiple reasoning steps. This result is unsurprising as the AI chatbot was trained on massive amounts of writing on the internet and questions requiring high-order reasoning introduce more variables and increased chances for the AI to incorporate relevant but incorrect information. The authors found that ChatGPT provided a “highly logical and coherent rationale for its answer choice, regardless of whether it chose the correct answer.”² Nonetheless, this calls into question the readiness of ChatGPT as a primary teaching tool in urology.³

More interestingly, ChatGPT is not static. It can be trained. Researchers have built a ChatGPT trained on the 2023 European Association of Urology guidelines. This custom AI responded with more concise and precise answers compared to the untrained ChatGPT.⁴ Future custom GPTs may be built and validated as education and clinical decision support tools.

While we are happy to report that urologists won’t yet be replaced by AI in clinical decision-making, for now, like any new technology, the true potential for ChatGPT has yet to be realized. In terms of medical education, ChatGPT poses a huge upside for medical students, residents, fellows, and attending physicians. There are multiple tasks ChatGPT can do to elevate the medical education experience including teaching assistance, personalized learning with materials and study plans, research assistance, content creation, documentation, and patient interactions.⁵ Just like any complex tool, proper usage must be taught. While research on ChatGPT is still in its adolescence, one of the most important aspects of optimization of its use comes from understanding how to appropriately prompt and direct the AI. Studies show the accuracy and efficiency of the desired outcome depends on the user. With so many potential ways to utilize this technology, medical schools and residency programs need to focus on teaching trainees how to elicit the appropriate responses from ChatGPT to properly achieve their goal.^3,4 Similar to how medical students learn to properly perform a PubMed search or statistics, ChatGPT optimization should be taught in the classroom. By providing students with the right tools, they can best take full advantage of ChatGPT to improve their knowledge and, at the end of the day, improve patient care.