Consistency of CSCO AI with Multidisciplinary Clinical Decision-Making Teams in Breast Cancer: A Retrospective Study.

Background: The Chinese Society of Clinical Oncology Artificial Intelligence System (CSCO AI) serves as a clinical decision support system developed utilizing Chinese breast cancer data. Our study delved into the congruence between breast cancer treatment recommendations provided by CSCO AI and their practical application in clinical settings. Methods: A retrospective analysis encompassed 537 breast cancer patients treated at the Second Affiliated Hospital of Anhui Medical University between January 2017 and December 2022. Proficient senior oncology researchers manually input patient data into the CSCO AI system. "Consistent" and "Inconsistent" treatment categories were defined by aligning our treatment protocols with the classification system in the CSCO AI recommendations. Cases that initially showed inconsistency underwent a second evaluation by the Multi-Disciplinary Treatment (MDT) team at the hospital. Concordance was achieved when MDTs' treatment suggestions were in the 'Consistent' categories. Results: An impressive 80.4% concurrence was observed between actual treatment protocols and CSCO AI recommendations across all breast cancer patients. Notably, the alignment was markedly higher for stage I (85.02%) and stage III (88.46%) patients in contrast to stage II patients (76.06%, P=0.023). Moreover, there was a significant concordance between invasive ductal carcinoma and lobular carcinoma (88.46%). Interestingly, triple-negative breast cancer (TNBC) exhibited a high concordance rate (87.50%) compared to other molecular subtypes. When contrasting MDT-recommended treatments with CSCO AI decisions, an overall 92.4% agreement was established. Furthermore, a logistic multivariate analysis highlighted the statistical significance of age, menstrual status, tumor type, molecular subtype, tumor size, and TNM stage in influencing consistency. Conclusion: In the realm of breast cancer treatment, the alignment between recommendations offered by CSCO AI and those from MDT is predominant. CSCO AI can be a useful tool for breast cancer treatment decisions.

Assessing the decision quality of artificial intelligence and oncologists of different experience in different regions in breast cancer treatment.

Background: AI-based clinical decision support system (CDSS) has important prospects in overcoming the current informational challenges that cancer diseases faced, promoting the homogeneous development of standardized treatment among different geographical regions, and reforming the medical model. However, there are still a lack of relevant indicators to comprehensively assess its decision-making quality and clinical impact, which greatly limits the development of its clinical research and clinical application. This study aims to develop and application an assessment system that can comprehensively assess the decision-making quality and clinical impacts of physicians and CDSS. Methods: Enrolled adjuvant treatment decision stage early breast cancer cases were randomly assigned to different decision-making physician panels (each panel consisted of three different seniority physicians in different grades hospitals), each physician made an independent "Initial Decision" and then reviewed the CDSS report online and made a "Final Decision". In addition, the CDSS and guideline expert groups independently review all cases and generate "CDSS Recommendations" and "Guideline Recommendations" respectively. Based on the design framework, a multi-level multi-indicator system including "Decision Concordance", "Calibrated Concordance", " Decision Concordance with High-level Physician", "Consensus Rate", "Decision Stability", "Guideline Conformity", and "Calibrated Conformity" were constructed. Results: 531 cases containing 2124 decision points were enrolled; 27 different seniority physicians from 10 different grades hospitals have generated 6372 decision opinions before and after referring to the "CDSS Recommendations" report respectively. Overall, the calibrated decision concordance was significantly higher for CDSS and provincial-senior physicians (80.9%) than other physicians. At the same time, CDSS has a higher " decision concordance with high-level physician" (76.3%-91.5%) than all physicians. The CDSS had significantly higher guideline conformity than all decision-making physicians and less internal variation, with an overall guideline conformity variance of 17.5% (97.5% vs. 80.0%), a standard deviation variance of 6.6% (1.3% vs. 7.9%), and a mean difference variance of 7.8% (1.5% vs. 9.3%). In addition, provincial-middle seniority physicians had the highest decision stability (54.5%). The overall consensus rate among physicians was 64.2%. Conclusions: There are significant internal variation in the standardization treatment level of different seniority physicians in different geographical regions in the adjuvant treatment of early breast cancer. CDSS has a higher standardization treatment level than all physicians and has the potential to provide immediate decision support to physicians and have a positive impact on standardizing physicians' treatment behaviors.