Automated risk stratification in localized prostate cancer using an AI-assisted framework.

Abstract

341 Background: Specialist clinicians (oncologists, urologists, radiation oncologists) read through free text reports of prostate MRI and biopsy reports to risk stratify patients during initial consultation of localized PCa. This laborious and time-consuming process is repeated for each day during the clinic. Herein, we propose a hybrid stratification framework utilizing a combination of large language model (LLM) and rule-based programming to automate this process. Methods: This retrospective study included patients with localized PCa (2004-2024) presenting to care at Mayo Clinic with at least one positive prostate biopsy and MRI report available. The state-of-the-art LLM – GPT4 – was utilized using a structured zeroshot prompt to extract relevant phenotypic variables (clinical T stage, Gleason patterns, grade group, extent of specimen/cores involved, prostate size/volume) from unstructured MRI and biopsy reports. Using a rule-based algorithm, the extracted variables and PSA at PCa diagnosis were used to categorize each patient into one of the six NCCN risk categories (very low [VL], low [L], favorable intermediate [FInt], unfavorable intermediate [UFInt], high risk (HR), very high risk [VHR]). Prompts were iteratively developed using ~5% of the total dataset and validated on ~10% of the dataset. Final performance was assessed against a held-out expert annotated test dataset using evaluation metrics (accuracy, F1-score, precision, recall). Additionally, manual annotation was conducted by two independent novice reviewers to compare machine annotated and novice-human annotated risk categorization. Results: A total of 397 patients were included in the evaluation. The median age at diagnosis was 64.8 (IQR: 59.7-68.5); majority of the men were White (n: 366; 92%) and non-Hispanic (n: 371; 94%). The most prevalent risk category was UFint (n: 151; 38%) followed by FInt (n: 80; 20.2%), HR (n: 76; 19.1%), VHR (n: 64; 16.1%), L (21: 5.3%), and VL (n: 5; 1.3%). Novice reviewers achieved an accuracy of 79%, precision of 80%, recall of 82% and a F1-score of 0.81. The hybrid stratification framework using GPT4 achieved an accuracy of 89%, precision of 89%, recall of 88% and a F1-score of 0.88. Among 43 errors by GPT4 risk stratification, majority of the errors were due to miscategorization of clinical T-stage (n: 21; 49%) followed by erroneous number of positive cores (n: 13; 30%). However, additional evaluation showed that GPT4 achieved numerically higher performance (accuracy: 90%; precision: 91%; recall: 90%; F1: 0.90) compared to novice reviewers (accuracy: 77%; precision: 89%; recall: 77%; F1: 0.82) for ascertaining clinical T-stage. Conclusions: Large language models exhibited superior performance than novice clinicians for risk stratification in patients with localized PCa. As the next step, we intend to validate our framework for automated risk classification in a prospective manner.