Attention: Restrictions on use of AUA, AUAER, and UCF content in third party applications, including artificial intelligence technologies, such as large language models and generative AI.
You are prohibited from using or uploading content you accessed through this website into external applications, bots, software, or websites, including those using artificial intelligence technologies and infrastructure, including deep learning, machine learning and large language models and generative AI.
Urology Faculty and Residents Outperform Artificial Intelligence in Prediction of Winning Teams
By: Max S. Yudovich, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Joseph Smith, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Joe O. Littlejohn Jr, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Jennifer A. Kane, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Blake R. Baer, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Thomas M. FitzGibbon, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Roderick K. Clark, MD, MSc, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Cassra B. Clark, MD, MS, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Austin K. Bramwell, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Matthew G. Kaag, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Ali M. Ziada, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Susan MacDonald, MD, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Tullika Garg, MD, MPH, Penn State Health Milton S. Hershey Medical Center, Pennsylvania; Jay D. Raman, MD, FACS, FRCS (Glasg), Penn State Health Milton S. Hershey Medical Center, Pennsylvania | Posted on: 19 Apr 2024
Introduction
The National Collegiate Athletic Association (NCAA) Division 1 men’s basketball tournament is one of the most popular sporting events in the US and is well known for the widespread creation of bracket pools in which participants predict the winning teams from each round of the tournament. ChatGPT is an artificial intelligence tool which has been the subject of numerous investigations regarding applicability within the field of urology. The objective of this study is to compare the prediction accuracy of ChatGPT to a pool of urology faculty and residents in brackets created for the 2024 basketball tournament.
Methods
Urology faculty (n = 8) and residents (n = 5) were invited to participate in a bracket pool for the 2024 NCAA men’s basketball tournament through CBS Sports. GPT4, using the NCAA Basketball GPT based on publicly available team performance statistics, was asked to predict the outcomes of all games in the tournament. For each round of the tournament, all matchups were provided to GPT4 in a single session. Based on historical trends, GPT4 was asked to select 9, 5, 3, and 2 upsets in the first, second, third, and fourth rounds, respectively. All selections were made prior to the start of the tournament. For each correct selection, 2, 2, 4, 8, 12, or 16 points were awarded in each corresponding tournament round.
Results
GPT4 correctly selected 32 (z-score −2.2) winning teams, while urologists selected on average 38.8 (standard deviation 3.1). GPT4 scored 76 points (z-score −5.8) over the course of the tournament compared to urologists with an average of 107 (standard deviation 19.2). GPT4 selected more upsets (n = 7, z-score 1.0) than average (4.5, standard deviation 2.4). All but 1 urologist scored more points than GPT4.
Conclusion
Urology faculty and residents outperform ChatGPT with respect to the selection of winning teams for the annual men’s college basketball tournament. Further training of the artificial intelligence model is required to improve its predictive ability.
advertisement
advertisement