A practical guide to the implementation of AI in orthopaedic research, Part 6: How to evaluate the performance of AI research?

Artificial intelligence's (AI) accelerating progress demands rigorous evaluation standards to ensure safe, effective integration into healthcare's high-stakes decisions. As AI increasingly enables prediction, analysis and judgement capabilities relevant to medicine, proper evaluation and interpretation are indispensable. Erroneous AI could endanger patients; thus, developing, validating and deploying medical AI demands adhering to strict, transparent standards centred on safety, ethics and responsible oversight. Core considerations include assessing performance on diverse real-world data, collaborating with domain experts, confirming model reliability and limitations, and advancing interpretability. Thoughtful selection of evaluation metrics suited to the clinical context along with testing on diverse data sets representing different populations improves generalisability. Partnering software engineers, data scientists and medical practitioners ground assessment in real needs. Journals must uphold reporting standards matching AI's societal impacts. With rigorous, holistic evaluation frameworks, AI can progress towards expanding healthcare access and quality. Level of Evidence: Level V.

Prevalence and Predictors of Concomitant Meniscal Surgery During Pediatric and Adolescent ACL Reconstruction: Analysis of 4729 Patients Over 20 Years at a Tertiary-Care Regional Children's Hospital.

Background: The rate of concomitant meniscal procedures performed in conjunction with anterior cruciate ligament (ACL) reconstruction is increasing. Few studies have examined these procedures in high-risk pediatric cohorts. Hypotheses: That (1) the rates of meniscal repair compared with meniscectomy would increase throughout the study period and (2) patient-related factors would be able to predict the type of meniscal operation, which would differ according to age. Study Design: Cohort study (prevalence); Level of evidence, 2. Methods: Natural language processing was used to extract clinical variables from notes of patients who underwent ACL reconstruction between 2000 and 2020 at a single institution. Patients were stratified to pediatric (5-13 years) and adolescent (14-19 years) cohorts. Linear regression was used to evaluate changes in the prevalence of concomitant meniscal surgery during the study period. Logistic regression was used to determine predictors of the need for and type of meniscal procedure. Results: Of 4729 patients (mean age, 16 ± 2 years; 54.7% female) identified, 2458 patients (52%) underwent concomitant meniscal procedures (55% repair rate). The prevalence of lateral meniscal (LM) procedures increased in both pediatric and adolescent cohorts, whereas the prevalence of medial meniscal (MM) repair increased in the adolescent cohort (P = .02). In the adolescent cohort, older age was predictive of concomitant medial meniscectomy (P = .031). In the pediatric cohort, female sex was predictive of concomitant MM surgery and of undergoing lateral meniscectomy versus repair (P≤ .029). Female sex was associated with decreased odds of concomitant LM surgery in both cohorts (P≤ .018). Revision ACLR was predictive of concomitant MM surgery and of meniscectomy (medial and lateral) in the adolescent cohort (P < .001). Higher body mass index was associated with increased odds of undergoing medial meniscectomy versus repair in the pediatric cohort (P = .03). Conclusion: More than half of the young patients who underwent ACLR had meniscal pathology warranting surgical intervention. The prevalence of MM repair compared with meniscectomy in adolescents increased throughout the study period. Patients who underwent revision ACLR were more likely to undergo concomitant meniscal surgeries, which were more often meniscectomy. Female sex had mixed effects in both the pediatric and adolescent cohorts.

Prevalence and Predictors of Concomitant Meniscal and Ligamentous Injuries Associated With ACL Surgery: An Analysis of 20 Years of ACL Reconstruction at a Tertiary Care Children's Hospital.

BACKGROUND: There is an increasing rate of procedures being performed for concomitant injuries during anterior cruciate ligament (ACL) surgery. Few studies have examined risk factors for these associated injuries in young patients. HYPOTHESIS: There are patient-related factors predictive of concomitant knee pathology that differ between age-based cohorts. STUDY DESIGN: Cross-sectional study; Level of evidence, 3. METHODS: Natural language processing was used to extract clinical variables from available notes of patients undergoing ACL surgery between 2000 and 2020 at a single institution (5174 ACL surgeries; mean age, 17 ± 4 years; 53.1% female; accuracy, >98%). Patients were stratified to pediatric (5-13 years), adolescent (14-19 years), and young adult (20-35 years) cohorts. Logistic regression was used to determine predictors of concomitant injury to the menisci, medial collateral ligament (MCL), posterolateral corner (PLC), and posterior cruciate ligament (PCL). RESULTS: Between 2000 and 2020, 54% of pediatric, 71% of adolescent, and 70% of adult patients had ≥1 concomitant soft tissue injury. In children and adolescents, increased age was consistently predictive of sustaining a concomitant injury (P < .02). Female children had increased odds of concomitant medial meniscal injury, while female adults had decreased odds (P≤ .046). Adolescent and adult female patients had decreased odds of concomitant lateral meniscal injury (P≤ .027). Female children had increased odds of injury to the MCL (P = .015), whereas female children and adolescents had decreased odds of PCL injury (P≤ .044). Adolescents undergoing revision ACL surgery had increased odds of meniscal injury (P≤ .001) and decreased odds of concomitant MCL injury (P = .028). Increased body mass index (BMI) was associated with increased odds of concomitant medial meniscal injury in all cohorts (P≤ .041), lateral meniscal injury in adults (P = .045), and PLC injury in children (P = .016). Contact injuries were associated with increased odds of MCL injury in adolescents (P = .017) and PLC injury in adolescents and adults (P < .014). CONCLUSION: These findings support the hypothesis, as there were multiple factors that significantly affected the risk of concomitant injuries that differed between cohorts. Increased age, BMI, and contact injury history were generally associated with increased odds of sustaining a concomitant injury, whereas female sex and revision ACL surgery had mixed effects. Further studies are essential to investigate the sex-based differences in risk for concomitant injuries and to develop tailored treatment plans that minimize the risk of secondary ACL injury.

Editorial Commentary: Artificial Intelligence Models Using Machine Learning Can Improve Preoperative Identification of Subscapularis Pathology.

Subscapularis pathology is difficult to diagnose, in part because of decreased sensitivity and accuracy in identifying tears with magnetic resonance imaging (MRI) when compared to other cuff tendons. Artificial intelligence evaluation of patient physical examination and MRI data using a machine learning model shows that arthroscopically confirmed partial- or full-thickness subscapularis tears are highly associated with abnormal subscapularis tendon length, long head of the biceps tears, and subscapularis fatty atrophy, and on physical examination, with weakness with internal rotation and positive lift-off, belly press, and bear hug tests. Today, physicians may use machine learning as a tool, but this model may not currently be sufficient to drastically change practice. However, with continued research and development, which is occurring rapidly, similar models could aid physicians in timely identification of pathology and optimization of preoperative planning, as well as physician training and education.

Autologous Osteochondral Transfer of the Knee Demonstrates Continued High Rates of Return to Sport and Low Rates of Conversion to Arthroplasty at Long-Term Follow-Up: A Systematic Review.

PURPOSE: To perform a systematic review of the literature to evaluate (1) activity level and knee function, (2) reoperation and failure rates, and (3) risk factors for reoperation and failure of autologous osteochondral transfer (AOT) at long-term follow-up. METHODS: A comprehensive review of the long-term outcomes of AOT was performed. Studies reported on activity-based outcomes (Tegner Activity Scale) and clinical outcomes (Lysholm score and International Knee Documentation Committee score). Reoperation and failure rates as defined by the publishing authors were recorded for each study. Modified Coleman Methodology Scores were calculated to assess study methodological quality. RESULTS: Twelve studies with a total of 495 patients and an average age of 32.5 years at the time of surgery and a mean follow-up of 15.1 years (range, 10.4-18.0 years) were included. The mean defect size was 3.2 cm2 (range, 1.9-6.9 cm2). The mean duration of symptoms before surgery was 5.1 years. Return to sport rates ranged from 86% to 100%. Conversion to arthroplasty rates ranged from 0% to 16%. The average preoperative International Knee Documentation Committee scores ranged from 32.9 to 36.8, and the average postoperative International Knee Documentation Committee scores at final follow-up ranged from 66.3 to 77.3. The average preoperative Lysholm scores ranged from 44.5 to 56.0 and the average postoperative Lysholm scores ranged from 70.0 to 96.5. The average preoperative Tegner scores ranged from 2.5 to 3.0, and the average postoperative scores ranged from 4.1 to 7.0. CONCLUSIONS: AOT of the knee resulted in high rates of return to sport with correspondingly low rates of conversion to arthroplasty at long-term follow-up. In addition, AOT demonstrated significant improvements in long-term patient-reported outcomes from baseline. LEVEL OF EVIDENCE: Level IV, systematic review of Level I-IV studies.

Acetabular Rim Disorders: Epidemiology, Etiology, Management, and Outcomes.

¼ Anatomic disorders of the acetabular rim are a common, correctable source of hip pain in younger patients.¼ Some common conditions of involving abnormal acetabular rim morphology include developmental dysplasia of the hip, pincer-type femoroacetabular impingement, acetabular protrusion, and acetabular retroversion.¼ Treatment option for these conditions were historically limited to open osteotomy and osteoplasty procedures; however, there is increasing use of arthroscopic intervention for these patients.¼ Arthroscopic intervention has demonstrated short-term success in a variety of focal acetabular rim disorders; however, further research is needed to determine the long-term outcomes of these procedures and their utility in more global pathology.

Natural language processing in radiology: Clinical applications and future directions.

Natural language processing (NLP) is a wide range of techniques that allows computers to interact with human text. Applications of NLP in everyday life include language translation aids, chat bots, and text prediction. It has been increasingly utilized in the medical field with increased reliance on electronic health records. As findings in radiology are primarily communicated via text, the field is particularly suited to benefit from NLP based applications. Furthermore, rapidly increasing imaging volume will continue to increase burden on clinicians, emphasizing the need for improvements in workflow. In this article, we highlight the numerous non-clinical, provider focused, and patient focused applications of NLP in radiology. We also comment on challenges associated with development and incorporation of NLP based applications in radiology as well as potential future directions.

Survival analyses and their applications in orthopaedics.

Survival analyses are a powerful statistical tool used to analyse data when the outcome of interest involves the time until an event. There is an array of models fit for this goal; however, there are subtle differences in assumptions, as well as a number of pitfalls, that can lead to biased results if researchers are unaware of the subtleties. As larger amounts of data become available, and more survival analyses are published every year, it is important that healthcare professionals understand how to evaluate these models and apply them into their practice. Therefore, the purpose of this study was to present an overview of survival analyses, including required assumptions and important pitfalls, as well as examples of their use within orthopaedic surgery.

Supervised machine learning and associated algorithms: applications in orthopedic surgery.

Supervised learning is the most common form of machine learning utilized in medical research. It is used to predict outcomes of interest or classify positive and/or negative cases with a known ground truth. Supervised learning describes a spectrum of techniques, ranging from traditional regression modeling to more complex tree boosting, which are becoming increasingly prevalent as the focus on "big data" develops. While these tools are becoming increasingly popular and powerful, there is a paucity of literature available that describe the strengths and limitations of these different modeling techniques. Typically, there is no formal training for health care professionals in the use of machine learning models. As machine learning applications throughout medicine increase, it is important that physicians and other health care professionals better understand the processes underlying application of these techniques. The purpose of this study is to provide an overview of commonly used supervised learning techniques with recent case examples within the orthopedic literature. An additional goal is to address disparities in the understanding of these methods to improve communication within and between research teams.

Natural language processing: using artificial intelligence to understand human language in orthopedics.

Natural language processing (NLP) describes the broad field of artificial intelligence by which computers are trained to understand and generate human language. Within healthcare research, NLP is commonly used for variable extraction and classification/cohort identification tasks. While these tools are becoming increasingly popular and available as both open-source and commercial products, there is a paucity of the literature within the orthopedic space describing the key tasks within these powerful pipelines. Curation and navigation of the electronic medical record are becoming increasingly onerous, and it is important for physicians and other healthcare professionals to understand potential methods of harnessing this large data resource. The purpose of this study is to provide an overview of the tasks required to develop an NLP pipeline for orthopedic research and present recent examples of successful implementations.

The development and deployment of machine learning models.

Applications of artificial intelligence, specifically machine learning, are becoming increasingly popular in Orthopaedic Surgery, and medicine as a whole. This growing interest is shared by data scientists and physicians alike. However, there is an asymmetry of understanding of the developmental process and potential applications of machine learning. As new technology will undoubtedly affect clinical practice in the coming years, it is important for physicians to understand how these processes work. The purpose of this paper is to provide clarity and a general framework for building and assessing machine learning models.

The readability of online patient education materials for slipped capital femoral epiphysis.

Given the long-term complications of undiagnosed slipped capital femoral epiphysis (SCFE) and the importance of readable health information materials on positive, equitable health outcomes, the objective of this study was to determine if the online patient education materials regarding SCFE are written at or below accepted recommendations. The secondary objective was to determine whether the readability of these materials varied when stratified by the type of website. 'Slipped capital femoral epiphysis', 'SCFE', and 'slipped femoral head' were used as search queries in three common search engines. The readability of each website was evaluated using five established metrics, and the scores were compared by website type and by the complexity of the search query. In this study of 53 unique websites about SCFE, we demonstrated that only one of the web pages was written at the recommended sixth-grade level, and the mean reading level of the online material was above the 10th-grade level. Post hoc testing showed that only websites associated with pediatric academic institutions were written at a significantly lower grade level than general health websites [P < 0.05 for all, range (0.003, 0.04)]. The materials about SCFE that are available to patients and their families continue to be written at an inappropriate level. To increase accessibility and allow for equitable long-term health outcomes, physicians, universities, hospitals and medical societies must ensure that they produce readable education to increase patients' understanding of SCFE, its symptoms and available treatment options. Future studies evaluating progress regarding these metrics are warranted.