Machine learning model for Asthma/COPD Differentiation Classification (ACDC) outperforms clinicians in a retrospective multiple-rater multiple-case review study

05 Aug 2021
Respiratory conditions
  • Asthma
  • COPD
Type of resource
Dublin 2021
Janwillem Kocks, General Practitioners Research Institute, Groningen, The Netherlands; University of Groningen, University Medical Center Groningen, GRIAC Research Institute, Groningen, The Netherlands; Observational and Pragmatic Research Institute, Singapore, Netherlands
Clinical Research Results Aim: Asthma/COPD Differentiation Classification (ACDC) program employs machine-learning (ML) model, based on 12 variables to aid physicians in diagnosing asthma, COPD, and asthma-COPD overlap (ACO). We compared diagnostic accuracy of ACDC in differentiating these conditions, compared to primary care physicians (PCPs) and pulmonologists (PULs) using multinational, multiple-rater, case review design.Methods: Clinical data of patients aged ≥35 years from FOCUS study1 were evaluated by seven PCPs and PULs to determine expert panel diagnosis (EPD). During 60-minute online web-based electronic case review, 360 PCPs and PULs from 9 countries each provided diagnosis for 30 randomly assigned EPD cases. EPD cases were analysed using ACDC ML models, which assigned a diagnosis. Primary endpoint: diagnostic accuracy of ACDC, PCPs and PULs in primary case set (EPD diagnoses of Asthma, COPD or ACO). Secondary endpoints included sensitivity and positive predictive values. Diagnostic accuracy was analysed using a Bayesian model that jointly modeled patient's EPD as categorical random variable and physician diagnoses using random effects multinomial logistic regression.Results: 116/119 cases (asthma:53, COPD:43, ACO:7, other diseases:13) received EPD and used in case review study. Diagnostic accuracy of ACDC (73%) was superior to PCPs (50%) (difference: 24%, 95% credible interval [CrI] 17-29; p<0.0001) and to PULs (61%) (difference: 12%, 95% CrI 6-17; p=0.0006). Sensitivity of ACDC was greater for asthma and COPD compared to PCPs and PULs; PCPs and PULs correctly classified more ACO cases. Positive predictive values were similar between ACDC and PULs for diagnosing asthma, performing slightly better than PCPs; PCPs and PULs were more precise in diagnosing ACO and COPD versus ACDC (Table).Conclusions: ACDC showed superior diagnostic accuracy versus PCPs and PULs in differential diagnosis of asthma and COPD. Diagnosis of ACO remains challenging for HCPs and ML models. ACDC appears useful to support physicians in differential diagnosis of asthma and COPD. Implementation Science/Service Development Research Ideas on Respiratory Conditions and Tobacco Dependency Abstract Declaration of Interest This study was funded by Novartis Pharma AG, Basel, Switzerland References and Clinical Trial Registry Information van de Hei SJ, et al. NPJ Prim Care Respir Med. 2020;30(1):22.