Evaluating Concordance Between Physicians' Decisions and AI-Generated Treatment Recommendations for Breast Cancer: A Retrospective Pilot Study
Müge Sönmez11Division of Medical Oncology, Department of Internal Medicine, Ordu University Faculty of Medicine, Ordu State Hospital, Ordu, Türkiye
Objectives: This study evaluated concordance between physicians’ decisions and recommendations generated by a large language model (LLM) in breast cancer management and explored factors associated with discordance. Methods: In this study, 101 patients with breast cancer were evaluated. Anonymized clinical data were entered into Chat Generative Pre-trained Transformer (GPT) 5.5 to generate treatment recommendations based on National Comprehensive Cancer Network (NCCN) and/or European Society for Medical Oncology (ESMO) guidelines. Concordance between physician decisions and artificial intelligence-generated (AI-generated) recommendations was classified as full, partial, or non-concordant. Descriptive statistics and exploratory multivariable logistic regression analyses were performed. Results: The median age was 53 years (IQR, 45–62). Disease stage was Stage I–II in 70 patients (69.3%), Stage III in 20 (19.8%), and metastatic in 11 (10.9%). Full concordance was observed in 83 patients (82.2%), while partial concordance occurred in 18 (17.8%); no cases of complete non-concordance were identified. Partial/non-concordance was more frequent in metastatic disease than in earlier-stage disease (45.5% vs. 12.9–20.0%; p=0.031). In multivariable analysis, HER2 (human epidermal growth factor receptor 2) positive disease (adjusted OR, 3.71; 95% CI (confidence interval), 1.09–12.65; p=0.036) and the 2024–2025 guideline period (adjusted OR, 4.73; 95% CI, 1.26–17.78; p=0.022) were independently associated with partial/non-concordance. Conclusion: High overall concordance was observed between physician decisions and AI recommendations. Discordance was more common in particularly metastatic disease and periods of evolving treatment guidelines. Further prospective validation is required before routine clinical implementation. Keywords: Artificial intelligence, breast cancer, ChatGPT, clinical decision support, large language model, treatment concordance
Cite This Article
Sönmez M. Evaluating Concordance Between Physicians' Decisions and AI-Generated Treatment Recommendations for Breast Cancer: A Retrospective Pilot Study. EJMI. 2026; 10(4): 480-490
Corresponding Author: