The AUANews article in the May 2023 issue, “ChatGPT: A Time-saving Companion for Physicians,” by Gabrielson et al, is excellent and gives a good introduction to this emerging technology. The authors are completely correct about how using these new artificial intelligence (AI) tools can benefit physicians and researchers.

However, readers need to be aware that one of the limitations of the present ChatGPT system is that it does not give accurate citations of scholarly articles. No, correct that: It often lies, and it produces fake references that are scarily good.

To be fair, if you ask ChatGPT directly: “Give me scholarly articles,” it will respond by saying, “I apologize, but as an AI language model, I don’t have direct access to specific scholarly articles or the ability to browse the Internet. I can provide general information and answer questions based on my training up until September 2021.” Very polite, and very accurate.

But ChatGPT is not always careful in following its own stated limitations. When I asked ChatGPT to provide me with some older articles on healing in the renal papilla, it gave me a list (see Figure). The paper titles provided look great! But 2 of these citations are completely fake, and the other 2 citations have odd errors.

Figure. List of papers provided by ChatGPT when prompted to provide older articles on healing in the renal papilla (ChatGPT, personal communication, June 7, 2023).

If you go to Physiological Reviews and pull up volume 72 in 1992, the page numbers 693-732 overlap with existing papers. In PubMed, it is easy to find out that Drs Jacobson and Fogo have never actually published together, but they both work in the areas of renal biology and pathology, so their names are not randomly assigned to this fictitious paper. For the second citation, checking in the American Journal of Physiology-Renal Physiology shows that, again, the page numbers overlap with existing papers, but ChatGPT appropriately cites page numbers beginning with the letter F, as they always do in that journal. In checking the author names on this citation, there are too many published authors named Z Xu to be certain of that name, but Drs Ong and Moldovan are real biomedical researchers who also have never published together.

The third citation is not fake: The Dagher et al paper is in Journal of the American Society of Nephrology, but the ChatGPT citation gives the wrong issue number and leaves off the final author name. The fourth citation is also a real paper, but the ChatGPT citation is correct only with the first author, while the other author names are people who have published with the first author, but who are not actually listed on this paper.

So, the citations given to me by ChatGPT were not entirely fake, but even the partially correct ones make me uneasy. What is true is that the citations invented by ChatGPT look really good. In my experience on this I have found the following:

• The paper citations from ChatGPT always have real author names from the field, often even grouped as you might expect (German names in one paper, Chinese in another)
• The titles of the papers are very believable (and just what I am looking for!)
• The journals are real
• The volume numbers given match the year of the citation properly
• But many of the papers are fake. The page numbers given overlap existing papers in the given volume. Searches of the titles of the papers (on Google or PubMed) show that they do not exist

Frankly, I find all of this to be disturbing. When I am reviewing a paper or grant proposal now, do I need to check all the references to ensure that they are real? Maybe so.

But, for readers of AUANews, the important thing to note is that for all the value of ChatGPT in its present form, no references should be trusted. Indeed, in its present form, probably any facts it trots out should be verified. Of course, ChatGPT is free, so what should I expect? Well, I expect that it would at least not make things up. Surely the programmers can force AI bots to cite only real publications, and at least one can hope that this will be fixed in future versions.

But one thing you can say truthfully about ChatGPT as it is now: It is always very polite!