Performance Evaluation of a Supervised Machine Learning Pain Classification Model Developed by Neonatal Nurses.

BACKGROUND: Early-life pain is associated with adverse neurodevelopmental consequences; and current pain assessment practices are discontinuous, inconsistent, and highly dependent on nurses' availability. Furthermore, facial expressions in commonly used pain assessment tools are not associated with brain-based evidence of pain. PURPOSE: To develop and validate a machine learning (ML) model to classify pain. METHODS: In this retrospective validation study, using a human-centered design for Embedded Machine Learning Solutions approach and the Neonatal Facial Coding System (NFCS), 6 experienced neonatal intensive care unit (NICU) nurses labeled data from randomly assigned iCOPEvid (infant Classification Of Pain Expression video) sequences of 49 neonates undergoing heel lance. NFCS is the only observational pain assessment tool associated with brain-based evidence of pain. A standard 70% training and 30% testing split of the data was used to train and test several ML models. NICU nurses' interrater reliability was evaluated, and NICU nurses' area under the receiver operating characteristic curve (AUC) was compared with the ML models' AUC. RESULTS: Nurses weighted mean interrater reliability was 68% (63%-79%) for NFCS tasks, 77.7% (74%-83%) for pain intensity, and 48.6% (15%-59%) for frame and 78.4% (64%-100%) for video pain classification, with AUC of 0.68. The best performing ML model had 97.7% precision, 98% accuracy, 98.5% recall, and AUC of 0.98. IMPLICATIONS FOR PRACTICE AND RESEARCH: The pain classification ML model AUC far exceeded that of NICU nurses for identifying neonatal pain. These findings will inform the development of a continuous, unbiased, brain-based, nurse-in-the-loop Pain Recognition Automated Monitoring System (PRAMS) for neonates and infants.

Application of a human-centered design for embedded machine learning model to develop data labeling software with nurses: Human-to-Artificial Intelligence (H2AI).

BACKGROUND: Nurses are essential for assessing and managing acute pain in hospitalized patients, especially those who are unable to self-report pain. Given their role and subject matter expertise (SME), nurses are also essential for the design and development of a supervised machine learning (ML) model for pain detection and clinical decision support software (CDSS) in a pain recognition automated monitoring system (PRAMS). Our first step for developing PRAMS with nurses was to create SME-friendly data labeling software. PURPOSE: To develop an intuitive and efficient data labeling software solution, Human-to-Artificial Intelligence (H2AI). METHOD: The Human-centered Design for Embedded Machine Learning Solutions (HCDe-MLS) model was used to engage nurses. In this paper, HCDe-MLS will be explained using H2AI and PRAMS as illustrative cases. FINDINGS: Using HCDe-MLS, H2AI was developed and facilitated labeling of 139 videos (mean = 29.83 min) with 3189 images labeled (mean = 75 s) by 6 nurses. OpenCV was used for video-to-image pre-processing; and MobileFaceNet was used for default landmark placement on images. H2AI randomly assigned videos to nurses for data labeling, tracked labelers' inter-rater reliability, and stored labeled data to train ML models. CONCLUSIONS: Nurses' engagement in CDSS development was critical for ensuring the end-product addressed nurses' priorities, reflected nurses' cognitive and decision-making processes, and garnered nurses' trust for technology adoption.