LUCID: Turning clinical noise into signal, structuring oncology data at scale.

Abstract

e15507 Background: Most clinically meaningful oncology information is recorded in unstructured formats, limiting its consistent use. Northwell Health diagnoses approximately 26,000 new cancer patients annually and operates a data lake in which all patient-level digital information is accessible. LUCID was developed to convert unstructured oncology data into structured representations in order to enable reliable downstream clinical, research, and operational use. Methods: LUCID is a platform deployed within a secure, HIPAA compliant environment designed to convert unstructured oncology data into clinically meaningful variables. The architecture combines natural language processing pipelines with modular large language model agents using GPT-5-mini. The modules described here characterize cancer presence, pathologic features, surgical resection events, staging, and follow longitudinal radiographic disease status. Each module was evaluated against clinician-validated ground truth, with cases reviewed by at least two clinicians and disagreements adjudicated by a third reviewer. Performance was assessed using accuracy, precision, recall, and F1 score. Time per patient abstraction and computational cost were measured. Results: 30 patient records were randomly select amongst patients who had undergone a biopsy at Northwell Health within the last decade and had a GI malignancy-associated ICD code; 20 of these cases had malignancy identified and were then assessed for staging and radiographic changes over time. These cases represented real-world heterogeneity including >6 histologies, a range of TNM stages (I-IV), and a breadth of longitudinal imaging assessments (average 13 per patient). Across evaluated domains, LUCID achieved accuracy ranging from 93.0% to 100.0%, with F1 scores from 0.96 to 1 (Table 1). False positives (FP) describe identification of events such as progression or response, whereas false negatives (FN) represent no call detected for an event. Errors were typically attributed to clinical edge-cases such as pseudo-malignant cysts. Abstraction required a total of 85 seconds per patient and 3 cents in processing cost. Conclusions: This proof- of- principle shows that structured oncology information can be extracted efficiently and at scale from unstructured clinical data within a health-system data lake. Although LUCID is capable of generating longitudinal patient timelines, this evaluation focused on module-level performance to enable objective measurements. Ongoing work applies this architecture to expand clinical and research capabilities system-wide. LUCID Module performance. Domain N Accuracy Precision Recall F1 Error profile Cancer Presence 30 100% 100% 100% 100% Valid Resection 30 93% 100% 82% 90% 2 FN Histology and Differentiation 30 97% 95% 100% 98% 1 FP T stage 20 100% 100% 100% 100% N stage 20 100% 100% 100% 100% M stage 20 95% 95% 100% 97% 1 FP Radiographic Changes 106 96% 95% 98% 96% 3 FP