Next-generation U-Net Encoder: Decoder for accurate, automated CTC detection from images of peripheral blood nucleated cells stained with EPCAM and DAPI.

Abstract

3061 Background: Direct circulating tumor cell (CTC) detection is a promising biomarker for early cancer detection and monitoring. Traditional fluorescence microscopy and AI-driven methods have limitations such as subjectivity and labor-intensiveness. We developed a deep-learning pipeline using a U-Net–type encoder–decoder architecture for precise pixel-level CTC discrimination in peripheral blood nucleated cells (PBNCs). This method preserves morphological and fluorescence details, overcoming convolutional neural network (CNN) limitations by maintaining fine features through skip connections for better discrimination. We present specificity and sensitivity data from a case-control study. Methods: We collected 5 ml of peripheral blood in EDTA tubes from 1383 asymptomatic healthy volunteers (744, 54% male; 639, 46% females with mean age of 49 [(20 to 93) yrs], 38 individuals diagnosed with non-malignant conditions including prostatitis, PCOD and acute pancreatitis, and 143 individuals recently diagnosed with surgically resectable early stage cancers (Stage 1/ 2) - Head and Neck (N=50, 35%), Breast (N=31, 22%), Colorectal (N=17, 12%), Pancreas (N=8, 6%), Prostate (N=8, 6%), Lung (N=5, 3%), Ovary (N=5, 3%) others (N=19, 13%). Nucleated cells were isolated from the samples after RBC lysis and centrifugation and stained with EPCAM and DAPI and set on imaging slides. 60X images were obtained and processed by AI utilizing U-Net–Based Encoder–Decoder Architecture and context discrimination to detect CTCs. The customized U-Net pipeline encodes spatial information through successive convolutional and pooling layers, generating a highly compressed representation of cells in the bottleneck. By employing transposed convolutions in the decoder stage—and incorporating skip connections from the encoder layers—the AI model reconstructs a pixel-wise segmentation mask to identify potential CTCs with cell diameter >10 microns. This approach aims to surpass existing methods that rely on bounding-box–based detection by offering enhanced sensitivity and specificity through end-to-end learned feature extraction. Ground truth annotations were established via expert cytopathology review, and training procedures involved cross-validation to ensure generalizable performance. Results: Analysis of total 1564 samples showed that our U-Net–based model achieved a sensitivity of 89% (95% CI: 88.81) and specificity of 97% (95% CI: 97.98) for detecting CTCs. Performance remained consistent across solid tumors, highlighting the flexibility and adaptability of the architecture in various fluorescence staining conditions. Conclusions: Our U-Net pipeline uses pixel-level segmentation and skip connections to enhance CTC detection accuracy. Integrating fluorescence and morphology, it can streamline cancer screening and disease monitoring.