Introduction:

Generative Artificial Intelligence (AI) has become prominent in healthcare, particularly as an oncology aid in clinical documentation, which has implications for treatment decisions.¹ National guidelines favor synoptic reporting of prostate biopsy specimens to aid in proper risk stratification of prostate cancer (PCa).² Thus, AI is a potential tool to analyze synoptic reports and reduce the variation in algorithmic risk stratification. We questioned whether a trained AI model could review pathology reports, along with clinical vignettes, for PCa to provide an accurate risk assessment and recommendations.

1.         Sorin, V, Barash, Y, Konin, E., et al.. Deep-learning natural language processing for oncological applications. The Lancet Oncology. 2020 Dec 2020;21(12)doi:10.1016/S1470-2045(20)30615-X

2.         https://www.nccn.org/professionals/physician_gls/pdf/prostate_blocks.pdf

Methods:

Retrospective review of localized PCa patients from January 2022 – January 2023 was performed. Following initial prostate biopsy, all patients were consulted and risk stratified based on NCCN guidelines by a urologic oncologist. We provided a generative AI model, ChatGPT³, the PCa risk stratification algorithm from NCCN Version 1.2023. Then queried the model to ensure accurate recall of the algorithm rules. Clinical features (PSA, cT stage) and histologic features (synoptic path report) from each case were provided and the model was prompted to “risk stratify the following patient.” In addition, we requested additional evaluation (ex: imaging) and initial therapy recommendations were also queried. 10 patients from each NCCN risk category were reviewed.

3.         ChatGPT. https://chat.openai.com

Results:

60 patients were reviewed. The generative AI model correctly risk stratified 39/60 patients (65.0%). The AI model incorrectly risk stratified 21/60 patients, however, it did assign them to an adjacent risk group (Figure 1).  Notably, 14/20 patients were correctly assigned as intermediate risk group, but the model required further prompting for categorization into favorable or unfavorable risk. Once prompted, only 12/20 were correctly risk stratified (Figure 1). Finally, when queried, the model was able to recommend treatment and imaging modalities appropriate to the stated risk group (ex: very low risk favor active surveillance, and high risk required imaging).

Conclusion:

Generative AI demonstrated poor performance in algorithmic risk stratification using standardized synoptic reports. The implications of inaccurate assessments need to be considered as more healthcare professionals and systems look to incorporating generative AI into clinical tools. A healthy amount of skepticism and close review is needed to vet any clinical tool. Ultimately when it comes to the nuance of the PCa, a trained provider is still needed to interpret the information and recommendations in line with the patient’s goals and preferences.

Funding: N/A