Using large language models to assess adherence to ASCO patient-oncologist communication standards.

Abstract

12105 Background: The American Society of Clinical Oncology (ASCO) convened a multidisciplinary panel resulting in patient-oncologist communication guidelines published in 2017. These guidelines contain recommendations across topics including goals of care, treatment selection, end-of-life care, facilitating family involvement, and clinician training in communication. Ideally, these conversations should be documented in the electronic health record (EHR), so that they can be referred to at future visits as a patient’s clinical course evolves. Tracking adherence to these communication guidelines may be beneficial for quality improvement efforts. However, manual chart review of unstructured free text notes is tedious and burdensome. The recent development of Large Language Models (LLMs) may represent a new computational approach that can capture such documentation more efficiently than chart review. To our knowledge, no prior study has used LLMs to capture such documentation in free text notes, validated against gold-standard manual chart review. Methods: As part of a larger study on development of LLMs for tracking palliative care quality measures, we randomly selected 30 patients with advanced cancer and clinical notes in the month following navigation to a poor prognosis treatment node. We used GPT-4o-2024-05-13 , our HIPAA-secure tool, to develop an LLM prompt for identifying 14 ASCO communication domains in clinical text. The LLM prompt required output to generate source text to support identification of a communication domain. A “hallucination score” was calculated for source text, which is a measure of evidence produced by LLMs not found in source text. We then compared to gold standard manual chart review using standard performance metrics. Results: Across communication domains, note-level LLM analysis achieved sensitivity ranging from 0.43-1.0, specificity ranging 0.32-0.99, and accuracy ranging 0.51-0.99. Examples of documentation identified by both the LLM and chart review include goals of care and prognosis (“recently informed that her disease had progressed with treatment. Currently on ‘last line’ of chemotherapy”), treatment options and clinical trials (“her oncologist recommended a potential trial treatment, and she is contemplating involvement in this”), end-of-life care (“if her cancer continues to progress with her current treatment, they will transition her care to home hospice for comfort measures only”), and cost of care (“financial insecurity - referred to resource specialist ” ). Average hallucination index for documentation identified by the LLM was low. LLM frequently identified information missed by annotators. The LLM extracted information relevant to communication domains in a fraction of the time required by manual chart review. Conclusions: LLMs can identify communication domains in EHRs, potentially contributing to quality improvement efforts.