Introduction:

ChatGPT (OpenAI, San Francisco, CA) is a large language model-based artificial intelligence chatbot. ChatGPT can be used to answer unique, user-generated questions without direct training on specific content. In this study, we assessed the accuracy and utility of ChatGPT to generate responses to questions regarding renal masses and renal cell carcinoma.

Methods:

We reviewed three professional society guidelines for renal cell carcinoma to generate fifteen content questions. These were posed to ChatGPT and its text output recorded. We also generated pre- and post-content assessment questions regarding familiarity with and opinions on ChatGPT use in medical and patient education. Questions were then presented to four urologic oncologists and four non-clinical staff members and responses analyzed using basic statistics.

Results:

We found that all assessors rated ChatGPT highly on the accuracy and usefulness of information provided with overall mean scores of 4.64 (± 0.62 SD) and 3.58 (± 0.75) out of 5, respectively. Clinicians and non-clinicians did not differ in their scoring of responses (p=0.378). Completing content assessment improved confidence in the accuracy of ChatGPT’s information (p=0.011) and increased agreement that it should be used for medical education (p=0.007, Figure 1). Attitudes towards use for patient education did not change (p=0.305)

Conclusion:

ChatGPT has significant potential utility in medical education if it can provide accurate and useful information. We have found it to be a useful adjunct to expert human guidance both for medical trainee and, less so, for patient education. Further work is needed to validate these results before adoption.

Funding: N/A