The Number of Patients Lost to Follow-Up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model.

PURPOSE: To (1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF), and (2) determine whether RCTs with relatively "low" FIs are truly as sensitive to patients LTF as previously portrayed in the literature. METHODS: All FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements-defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome-were determined by 2 authors. A theoretical RCT with a sample size of 100, P = .006, and FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (FI) and adding patients LTF back to the trial to show the true sensitivity of RCTs to patients LTF. RESULTS: Of the 39 FI-based studies, 37 (95%) directly compared the FI with the number of patients LTF. Of these 37 studies, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal. CONCLUSIONS: The claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was shown to be greater than the FI, suggesting that RCTs with relatively low FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT. CLINICAL RELEVANCE: Surgeons may benefit from re-examining their interpretation of prior FI reviews that have made claims of substantial RCT fragility based on comparisons between the FI and patients LTF; it is possible the results are more robust than previously believed.

Currently Available Large Language Models Do Not Provide Musculoskeletal Treatment Recommendations That are Concordant With Evidence-Based Clinical Practice Guidelines.

PURPOSE: To determine whether leading, commercially-available LLMs provide treatment recommendations concordant with evidenced-based clinical practice guidelines (CPGs) developed by the American Academy of Orthopedic Surgeons (AAOS). METHODS: All CPGs concerning the management of rotator cuff tears(n=33) and anterior cruciate ligament (ACL) injuries(n=15) were extracted from the AAOS. Treatment recommendations from Chat-generative pretrained transformer version-4 [ChatGPT-4; OpenAI], Gemini (Google), Mistral-7B (Mistral AI), and Claude-3 (Anthropic) were graded by two blinded physicians as being "concordant," "discordant," or "indeterminate" (i.e., neutral response without definitive recommendation) with respect to AAOS CPGs. The overall concordance between LLM and AAOS recommendations were quantified, while the comparative overall concordance of recommendations amongst the four LLMs was evaluated through the Fischer's-exact test. RESULTS: Overall 135(70.3%) responses were concordant, 43(22.4%) indeterminate, and 14(7.3%) discordant. Inter-rater reliability for concordance classification was excellent (Kappa=0.92). Concordance with AAOS CPGs was most frequently observed with ChatGPT-4 (n=38, 79.2%), and least frequently with Mistral-7B (n=28,58.3%). Indeterminate recommendations were most frequently observed with Mistral-7B (n=17,35.4%) and least frequently with Claude-3 (n=8, 6.7%). Discordant recommendations were most frequently observed with Gemini (n=6,12.5%) and least frequently with ChatGPT-4 (n=1,2.1%). Overall, no statistically significant differences in concordant recommendations was observed across LLMs (p=0.12). Only 20 (10.4%) of all recommendations were transparent and provided references with full bibliographic details or links to specific peer-reviewed content to support recommendations. CONCLUSION: Among leading commercially-available LLMs, more than one-in-four recommendations concerning the evaluation and management of rotator cuff and ACL injuries do not reflect current evidenced-based CPGs. Although ChatGPT-4 demonstrated the highest performance, clinically significant rates of recommendations without concordance or supporting evidence were observed. Only 10% of responses by LLMs were transparent, precluding users from fully interpreting the sources from which recommendations were provided. CLINICAL RELEVANCE: While leading LLMs generally provide recommendations concordant with CPGs, a substantial error-rate exists, and the proportion of recommendations that do not align with these CPGs suggest that LLMs are not trustworthy clinical support tools at this time. Each off-the-shelf, closed-source LLM has strengths and weaknesses. Future research should evaluate and compare multiple LLMs to avoid bias associated with narrow evaluation of few models as observed in current literature.

The Large Language Model ChatGPT-4 Exhibits Excellent Triage Capabilities and Diagnostic Performance for Patients Presenting With Various Causes of Knee Pain.

PURPOSE: To provide a proof-of-concept analysis of the appropriateness and performance of ChatGPT-4 to triage, synthesize differential diagnoses, and generate treatment plans concerning common presentations of knee pain. METHODS: Twenty knee complaints warranting triage and expanded scenarios were input into ChatGPT-4, with memory cleared prior to each new input to mitigate bias. For the 10 triage complaints, ChatGPT-4 was asked to generate a differential diagnosis that was graded for accuracy and suitability in comparison to a differential created by 2 orthopaedic sports medicine physicians. For the 10 clinical scenarios, ChatGPT-4 was prompted to provide treatment guidance for the patient, which was again graded. To test the higher-order capabilities of ChatGPT-4, further inquiry into these specific management recommendations was performed and graded. RESULTS: All ChatGPT-4 diagnoses were deemed appropriate within the spectrum of potential pathologies on a differential. The top diagnosis on the differential was identical between surgeons and ChatGPT-4 for 70% of scenarios, and the top diagnosis provided by the surgeon appeared as either the first or second diagnosis in 90% of scenarios. Overall, 16 of 30 diagnoses (53.3%) in the differential were identical. When provided with 10 expanded vignettes with a single diagnosis, the accuracy of ChatGPT-4 increased to 100%, with the suitability of management graded as appropriate in 90% of cases. Specific information pertaining to conservative management, surgical approaches, and related treatments was appropriate and accurate in 100% of cases. CONCLUSIONS: ChatGPT-4 provided clinically reasonable diagnoses to triage patient complaints of knee pain due to various underlying conditions that were generally consistent with differentials provided by sports medicine physicians. Diagnostic performance was enhanced when providing additional information, allowing ChatGPT-4 to reach high predictive accuracy for recommendations concerning management and treatment options. However, ChatGPT-4 may show clinically important error rates for diagnosis depending on prompting strategy and information provided; therefore, further refinements are necessary prior to implementation into clinical workflows. CLINICAL RELEVANCE: Although ChatGPT-4 is increasingly being used by patients for health information, the potential for ChatGPT-4 to serve as a clinical support tool is unclear. In this study, we found that ChatGPT-4 was frequently able to diagnose and triage knee complaints appropriately as rated by sports medicine surgeons, suggesting that it may eventually be a useful clinical support tool.

Defining clinically relevant outcome thresholds for pain and function after osteochondral autograft transplantation of the knee.

PURPOSE: To define the minimal clinically important difference (MCID) for measures of pain and function at 2, 5 and 10 years after osteochondral autograft transplantations (OATs). METHODS: Patients undergoing OATs of the knee were identified from a prospectively maintained cartilage surgery registry. Baseline demographic, injury and surgical factors were collected. Patient-reported outcome scores (PROMs) were collected at baseline, 2-, 5- and 10-year follow-up, including the International Knee Documentation Committee (IKDC) score, Knee Outcome Survey Activities of Daily Living Scale (KOS-ADLS), Marx activity scale and Visual Analogue Scale (VAS) for pain. The MCIDs were quantified for each metric utilizing a distribution-based method equivalent to one-half the standard deviation of the mean change in outcome score. The percentage of patients achieving MCID as a function of time was assessed. RESULTS: Of 63 consecutive patients who underwent OATs, 47 (74.6%) patients were eligible for follow-up (surgical date before October 2021) and had fully completed preoperative PROMs. A total of 39 patients (83%) were available for a minimum 2-year follow-up, with a mean (±standard deviation) follow-up of 5.8 ± 3.4 years. The MCIDs were determined to be 9.3 for IKDC, 2.5 for Marx, 7.4 for KOS-ADLS and 12.9 for pain. At 2 years, 78.1% of patients achieved MCID for IKDC, 77.8% for Marx, 75% for KOS-ADLS and 57.9% for pain. These results were generally maintained through 10-year follow-ups, with 75% of patients achieving MCID for IKDC, 80% for Marx, 80% for KOS-ADLS and 69.8% for pain. CONCLUSIONS: The majority of patients achieved a clinically relevant outcome improvement after OATs of the knee, with results sustained through 10-year follow-up. Patients who experience clinically relevant outcome improvement after OATs in the short term continue to experience sustained benefits at longer-term follow-up. These data provide valuable prognostic information when discussing patient candidacy and the expected trajectory of recovery. LEVEL OF EVIDENCE: Level III.

Contemporary Patterns of Financial Toxicity Among Patients With Rheumatologic Disease in the United States.

BACKGROUND/OBJECTIVE: Rheumatologic diseases encompass a group of disabling conditions that often require expensive clinical treatments and limit an individual's ability to work and maintain a steady income. The purpose of this study was to evaluate contemporary patterns of financial toxicity among patients with rheumatologic disease and assess for any associated demographic factors. METHODS: The cross-sectional National Health Interview Survey was queried from 2013 to 2018 for patients with rheumatologic disease. Patient demographics and self-reported financial metrics were collected or calculated including financial hardship from medical bills, financial distress, food insecurity, and cost-related medication (CRM) nonadherence. Multivariable logistic regressions were used to assess for factors associated with increased financial hardship. RESULTS: During the study period, 20.2% of 41,502 patients with rheumatologic disease faced some degree of financial hardship due to medical bills, 55.0% of whom could not pay those bills. Rheumatologic disease was associated with higher odds of financial hardship from medical bills (adjusted odds ratio, 1.29; 95% confidence interval, 1.22-1.36; p < 0.001) with similar trends for patients suffering from financial distress, food insecurity, and CRM nonadherence (p < 0.001 for all). Financial hardship among patients with rheumatologic disease was associated with being younger, male, Black, and uninsured ( p < 0.001 for all). CONCLUSION: In this nationally representative study, we found that a substantial proportion of adults with rheumatologic disease in the United States struggled with paying their medical bills and suffered from food insecurity and CRM nonadherence. National health care efforts and guided public policy should be pursued to help ease the burden of financial hardship for these patients.

Current Concepts on the Management of Shoulder Instability in Throwing Athletes.

PURPOSE OF REVIEW: The management of shoulder instability in throwing athletes remains a challenge given the delicate balance between physiologic shoulder laxity facilitating performance and the inherent need for shoulder stability. This review will discuss the evaluation and management of a throwing athlete with suspected instability with a focus on recent findings and developments. RECENT FINDINGS: The vast majority of throwing athletes with shoulder instability experience subtle microinstability as a result of repetitive microtrauma rather than episodes of gross instability. These athletes may present with arm pain, dead arms or reduced throwing velocity. Recent literature reinforces the fact that there is no "silver bullet" for the management of these athletes and an individualized, tailored approach to treatment is required. While initial nonoperative management remains the hallmark for treatment, the results of rehabilitation protocols are mixed, and some patients will ultimately undergo surgical stabilization. In these cases, it is imperative that the surgeon be judicious with the extent of surgical stabilization as overtightening of the glenohumeral joint is possible, which can adversely affect athlete performance. Managing shoulder instability in throwing athletes requires a thorough understanding of its physiologic and biomechanical underpinnings. Inconsistent results seen with surgical stabilization has led to a focus on nonoperative management for these athletes with surgery reserved for cases that fail to improve non-surgically. Overall, more high quality studies into the management of this challenging condition are warranted.

Platelet-Rich Plasma Versus Alternative Injections for Osteoarthritis of the Knee: A Systematic Review and Statistical Fragility Index-Based Meta-analysis of Randomized Controlled Trials.

BACKGROUND: Based in part on the results of randomized controlled trials (RCTs) that suggest a beneficial effect over alternative treatment options, the use of platelet-rich plasma (PRP) for the management of knee osteoarthritis (OA) is widespread and increasing. However, the extent to which these studies are vulnerable to slight variations in the outcomes of patients remains unknown. PURPOSE: To evaluate the statistical fragility of conclusions from RCTs that reported outcomes of patients with knee OA who were treated with PRP versus alternative nonoperative management strategies. STUDY DESIGN: Systematic review and meta-analysis; Level of evidence, 2. METHODS: All RCTs comparing PRP with alternative nonoperative treatment options for knee OA were identified. The fragility index (FI) and reverse FI were applied to assess the robustness of conclusions regarding the efficacy of PRP for knee OA. Meta-analyses were performed to determine the minimum number of patients from ≥1 trials included in the meta-analysis for which a modification on the event status would change the statistical significance of the pooled treatment effect. RESULTS: In total, this analysis included outcomes from 1993 patients with a mean ± SD age of 58.0 ± 3.8 years. The mean number of events required to reverse significance of individual RCTs (FI) was 4.57 ± 5.85. Based on random-effects meta-analyses, PRP demonstrated a significantly higher rate of successful outcomes when compared with hyaluronic acid (P = .002; odds ratio [OR], 2.19; 95% CI, 1.33-3.62), as well as higher rates of patient-reported symptom relief (P = .019; OR, 1.55; 95% CI, 1.07-2.24), not requiring a reintervention after the initial injection treatment (P = .002; OR, 2.17; 95% CI, 1.33-3.53), and achieving the minimal clinically important difference (MCID) for pain improvement (P = .007; OR, 6.19; 95% CI, 1.63-23.42) when compared with all alternative nonoperative treatments. Overall, the mean number of events per meta-analysis required to change the statistical significance of the pooled treatment effect was 8.67 ± 4.50. CONCLUSION: Conclusions drawn from individual RCTs evaluating PRP for knee OA demonstrated slight robustness. On meta-analysis, PRP demonstrated a significant advantage over hyaluronic acid as well as improved symptom relief, lower rates of reintervention, and more frequent achievement of the MCID for pain improvement when compared with alternative nonoperative treatment options. Statistically significant pooled treatment effects evaluating PRP for knee OA are more robust than approximately half of all comparable meta-analyses in medicine and health care. Future RCTs and meta-analyses should consider reporting FIs and fragility quotients to facilitate interpretation of results in their proper context.

Understanding How ChatGPT May Become a Clinical Administrative Tool Through an Investigation on the Ability to Answer Common Patient Questions Concerning Ulnar Collateral Ligament Injuries.

Background: The consumer availability and automated response functions of chat generator pretrained transformer (ChatGPT-4), a large language model, poise this application to be utilized for patient health queries and may have a role in serving as an adjunct to minimize administrative and clinical burden. Purpose: To evaluate the ability of ChatGPT-4 to respond to patient inquiries concerning ulnar collateral ligament (UCL) injuries and compare these results with the performance of Google. Study Design: Cross-sectional study. Methods: Google Web Search was used as a benchmark, as it is the most widely used search engine worldwide and the only search engine that generates frequently asked questions (FAQs) when prompted with a query, allowing comparisons through a systematic approach. The query "ulnar collateral ligament reconstruction" was entered into Google, and the top 10 FAQs, answers, and their sources were recorded. ChatGPT-4 was prompted to perform a Google search of FAQs with the same query and to record the sources of answers for comparison. This process was again replicated to obtain 10 new questions requiring numeric instead of open-ended responses. Finally, responses were graded independently for clinical accuracy (grade 0 = inaccurate, grade 1 = somewhat accurate, grade 2 = accurate) by 2 fellowship-trained sports medicine surgeons (D.W.A, J.S.D.) blinded to the search engine and answer source. Results: ChatGPT-4 used a greater proportion of academic sources than Google to provide answers to the top 10 FAQs, although this was not statistically significant (90% vs 50%; P = .14). In terms of question overlap, 40% of the most common questions on Google and ChatGPT-4 were the same. When comparing FAQs with numeric responses, 20% of answers were completely overlapping, 30% demonstrated partial overlap, and the remaining 50% did not demonstrate any overlap. All sources used by ChatGPT-4 to answer these FAQs were academic, while only 20% of sources used by Google were academic (P = .0007). The remaining Google sources included social media (40%), medical practices (20%), single-surgeon websites (10%), and commercial websites (10%). The mean (± standard deviation) accuracy for answers given by ChatGPT-4 was significantly greater compared with Google for the top 10 FAQs (1.9 ± 0.2 vs 1.2 ± 0.6; P = .001) and top 10 questions with numeric answers (1.8 ± 0.4 vs 1 ± 0.8; P = .013). Conclusion: ChatGPT-4 is capable of providing responses with clinically relevant content concerning UCL injuries and reconstruction. ChatGPT-4 utilized a greater proportion of academic websites to provide responses to FAQs representative of patient inquiries compared with Google Web Search and provided significantly more accurate answers. Moving forward, ChatGPT has the potential to be used as a clinical adjunct when answering queries about UCL injuries and reconstruction, but further validation is warranted before integrated or autonomous use in clinical settings.