Performance of a NLP Tool for Text Classification from Orthopaedic Operative Reports, Using Data from the Large Network of Clinical Data Warehouses of the West of France: The HACRO-HUGORTHO Project.

ABSTRACT

Electronic health data concerning implantable medical devices (IMD) opens opportunities for dynamic real-world monitoring to assess associated risks related to implanted materials. Due to population ageing and expanding demands, total hip, knee, and shoulder arthroplasties are increasing. Automating the collection and analysis of orthopedic device features could benefit physicians and public health policies enabling early issue detection, IMD monitoring and patient safety assessment. A machine learning tool using natural language processing (NLP) was developed for the automated extraction of operation information from medical reports in orthopedics. A corpus of 959 orthopaedic operative reports from 5 centres was manually annotated using the Prodigy software® with a strong inter-annotator agreement of 0.80. Data to extract concerned key clinical and procedure information (n= 9) selected by a multidisciplinary group based on the French health authority checklist. Performances parameters of the NLP model estimated an overall strong precision and recall of respectively 97.0 and 96.0 with a F1-score 96.3. Systematic monitoring of orthopedic devices could be ensured by an automated tool, leveraging clinical data warehouses. Traceability of medical devices with implantation modalities will allow detection of implant factors leading to complications. The evidence from real-world data could provide concrete and dynamic insights to surgeons and infectious disease specialists concerning implant follow-up, guiding therapeutic decision-making, and informing public health policymakers. The tool will be applied on clinical data warehouses to automate information extraction and presentation, providing feedback on mandatory information completion and contents of operative reports to support improvements, and thereafter implant research projects.