Methodology for Online Reliability Studies: A Primer for Orthopedic Surgeons.

In orthopedic surgery, there is an increasing number of papers about online studies on the reliability of classification systems. Useful classification systems need to be reliable and valid. Measurement of validity can be variable and is prone to observer bias. These online collaboratives derive adequate power to study reliability by having a large group of trained surgeons review a small number of cases instead of the "classic" reliability studies in which a small number of observers evaluate many cases. Large online interobserver studies have advantages (i.e., less than 15 minutes to complete the survey, the ability to randomize, and the ability to study factors associated with reliability, accuracy, or decision-making). This 'handbook' paper gives an overview of current methods for online reliability studies. We discuss the study design, sample size calculation, statistical analyses of results, pitfalls, and limitations of the study design.

Dorsal Subluxation of the Proximal Interphalangeal Joint After Volar Base Fracture of the Middle Phalanx.

Background: Treatment decisions regarding volar base fractures of the middle phalanx depend on whether the proximal interphalangeal (PIP) joint is reduced. Our aim was to study the agreement among hand surgeons in determining whether the PIP joint fractures are subluxated and to study the factors associated with subluxation of these fractures. Methods: In this retrospective chart review, 413 volar base fractures of the middle phalanx were included. Demographic and injury-related factors were gathered from medical records and radiographs. Using a Web-based survey, interobserver agreement was determined among 105 hand surgeons on the assessment of PIP joint subluxation of a series of 26 cases. Using the cohort of 413 fractures, a threshold for percent articular involvement and relative fracture displacement that corresponds with subluxation of the PIP joint was analyzed. Results: We found moderate to substantial agreement between hand surgeons on subluxation (κ = 0.59, P < .0001) and an overall percent agreement of 85%. Percent articular involvement and relative fracture displacement were independently associated with subluxation of the PIP joint (P < .001). Percent articular involvement of 35% had a specificity of 90% and a negative predicting value (NPV) of 92% for joint subluxation. Relative fracture displacement of 35% had a specificity of 92% and an NPV of 94% for joint subluxation. Conclusions: Surgeons generally agree on whether a PIP joint is subluxated. Percent articular involvement and relative fragment displacement are objective measurements that can help characterize joint stability and assist with decision-making.

Variation in Treatment for Trapeziometacarpal Arthrosis.

BACKGROUND: Treatment recommendations for trapeziometacarpal (TMC) arthrosis are highly variable from surgeon to surgeon. This study addressed the influence of viewing radiographs on a decision to offer surgery for TMC arthrosis. METHODS: In an online survey, 92 hand surgeons viewed clinical scenarios and were asked if they would offer surgery to 30 patients with TMC arthrosis. Forty-two observers were randomly assigned to review clinical information alone and 50 to review clinical information as well as radiographs. The degree of limitation of daily activities, time since diagnosis, prior treatment, pain with grind, crepitation with grind, and metacarpal adduction with metacarpophalangeal hyperextension were randomized for each patient scenario to determine the influence of these factors on offers of surgery. A cross-classified binary logistic multilevel regression analysis identified factors associated with surgeon offer of surgery. RESULTS: Surgeons were more likely to offer surgery when they viewed radiographs (42% vs. 32%, P = 0.01). Other factors associated variation in offer of surgery included greater limitation of daily activities, symptoms for a year, prior splint or injection, deformity of the metacarpophalangeal joint. Factors not associated included limb dominance, prominence of the TMC joint, crepitation with the grind test, and pinch and grip strength. CONCLUSION: Surgeons that view radiographs are more likely to offer surgery to people with TMC arthrosis. urgeons are also more likely to offer surgery when people do not adapt with time and nonoperative treatment. Given the notable influence of surgeon bias, and the potential for surgeon and patient impatience with the adaptation process, methods for increasing patient participation in the decision-making process merit additional attention and study.

Symptoms of Burnout Among Surgeons Are Correlated with a Higher Incidence of Perceived Medical Errors.

BACKGROUND: Nearly 44% of practicing physicians in the USA report symptoms of burnout. Psychological distress and loss of joy in medicine are associated with malpractice lawsuits and attrition from medical practice and may correlate with the rate of perceived medical errors. QUESTIONS/PURPOSES: We sought to answer two questions: (1) What physician factors are associated with the number of perceived medical errors among practicing surgeons in the prior 3 months? (2) What characteristics are associated with symptoms of burnout among practicing surgeons? METHODS: We created a cross-sectional survey and invited members of the Science of Variation Group to respond between December 2018 and January 2019. Participating surgeons completed the Abbreviated Maslach Burnout Inventory, the two-item Patient Health Questionnaire (PHQ-2), and information about practice characteristics and demographics. We created a negative binomial and a multivariable linear regression model to seek factors independently associated with the number of perceived medical errors and symptoms of burnout. RESULTS: A greater level of emotional exhaustion was associated with a greater number of perceived medical errors, while practice location in Europe was associated with fewer perceived errors. A higher PHQ-2 score was independently associated with symptoms of burnout. CONCLUSION: It is possible that symptoms of burnout cause surgeons to be more likely to perceive an imperfection as an error or that burnout distracts surgeons, contributing to a greater likelihood of a verifiable error. Additional studies are merited to investigate a potential causal relationship between symptoms of burnout and medical errors.

Do Patient Preferences Influence Surgeon Recommendations for Treatment?

BACKGROUND: When the best treatment option is uncertain, a patient's preference based on personal values should be the source of most variation in diagnostic and therapeutic interventions. Unexplained surgeon-to-surgeon variation in treatment for hand and upper extremity conditions suggests that surgeon preferences have more influence than patient preferences. METHODS: A total of 184 surgeons reviewed 18 fictional scenarios of upper extremity conditions for which operative treatment is discretionary and preference sensitive, and recommended either operative or non-operative treatment. To test the influence of six specific patient preferences the preference was randomly assigned to each scenario in an affirmative or negative manner. Surgeon characteristics were collected for each participant. RESULTS: Of the six preferences studied, four influenced surgeon recommendations. Surgeons were more likely to recommend non-operative treatment when patients; preferred the least expensive treatment (adjusted OR, 0.82; 95% CI, 0.71 - 0.94; P=0.005), preferred non-operative treatment (adjusted OR, 0.82; 95% CI, 0.72 - 0.95; P=0.006), were not concerned about aesthetics (adjusted OR, 1.15; 95% CI, 1.0 - 1.3; P=0.046), and when patients only preferred operative treatment if there is consensus among surgeons that operative treatment is a useful option (adjusted OR, 0.78; 95% CI, 0.68 - 0.89; P<0.001). CONCLUSION: Patient preferences were found to have a measurable influence on surgeon treatment recommendations though not as much as we expected-and surgeons on average interpreted surgery as more aesthetic. This emphasizes the importance of strategies to help patients reflect on their values and ensure their preferences are consistent with those values (e.g. use of decision-aids).

Reliability of Diagnosis of Partial Union of Scaphoid Waist Fractures on Computed Tomography.

Computed tomography (CT) is increasingly used not just to diagnose union but also to estimate the percentage of the fracture gap that is bridged by healing bone. This study tested the primary null hypothesis that there is no agreement between observers on the extent of union of a scaphoid waist fracture on CT. CT scans of 13 nondisplaced scaphoid waist fractures treated nonoperatively were rated by 145 observers. CT scans were done 10 to 12 weeks after injury. Observers were asked to "eyeball" measure percentage of union. We found that there was a moderate agreement on the categorical degree of partial union of a scaphoid waist fracture on CT (k = 0.34). Agreement on the location of bony bridging was slight (k = 0.31). We concluded that there is limited reliability of diagnosis of partial union of a scaphoid waist fracture on CT and that this should be taken into account in both patient care and research. This is a Level III, diagnostic study.

Interobserver Variability of the Diagnosis of Scaphoid Proximal Pole Fractures.

Purpose Fractures of the proximal pole of the scaphoid are prone to adverse outcomes such as nonunion and avascular necrosis. Distinction of scaphoid proximal pole fractures from waist fractures is important for management but it is unclear if the distinction is reliable. Methods A consecutive series of 29 scaphoid fractures from one tertiary hospital was collected consisting of 5 scaphoid proximal pole and 24 scaphoid waist fractures. Fifty-seven members of the Science of Variation Group (SOVG) were randomized to diagnose fracture location and displacement by using radiographs alone or radiographs and a computed tomography (CT) scan. Results Observers reviewing radiographs alone and observers reviewing radiographs and CT scans both had substantial agreement on fracture location (κ = 0.82 and κ = 0.80, respectively; p = 0.54). Both groups had only fair agreement on fracture displacement (κ = 0.28 and κ = 0.35, respectively; p = 0.029). Conclusion Proximal pole fractures are sufficiently distinct from proximal waist fractures that CT does not improve reliability of diagnosis. Level of Evidence Level IV interobserver reliability case-control study.

How Reliable is the Radiographic Diagnosis of Mild Madelung Deformity?

Background Patients with Madelung deformity exhibit a spectrum of mild to severe deformity and distortion of wrist geometry. It may be difficult to reliably distinguish mild Madelung deformity from normal. Purpose This study thus tested the reliability of the diagnosis of mild Madelung deformity on a single posteroanterior (PA) radiograph. Materials and Methods An online survey was sent to hand and wrist surgeons of the Science of Variation Study Group for evaluation of 25 PA wrist radiographs comprising five adults with suspected mild Madelung deformity and 20 radiographs without any evident wrist pathology. Interobserver agreement was evaluated both via average percent agreement and Fleiss' kappa. To evaluate the relationship of rater characteristics and accuracy, a linear regression model was computed. Results The interobserver agreement among the 69 participating surgeons was low ( Κ = 0.12). The overall sensitivity, specificity, and accuracy were 0.30, 0.86, and 0.75, respectively. The mean confidence was 7.4 ± 0.4 for mild Madelung and 7.8 ± 0.5 for normal ( p = 0.112). The observers' confidence level was the only factor which had a mild but significant effect on the accuracy of the ratings. Conclusion The diagnosis of mild Madelung deformity on a single PA radiograph is unreliable. Level of Evidence The level of evidence is II, diagnostic study.

Interobserver Agreement in Diagnosing Early-Stage Kienböck Disease on Radiographs and Magnetic Resonance Imaging.

BACKGROUND: The appearance of early Kienböck disease on radiographs and magnetic resonance imaging (MRI) may be difficult to distinguish from other conditions that affect the lunate. We aimed to assess the interobserver agreement in the diagnosis of early Kienböck disease when evaluated on different imaging modalities. METHODS: Forty-three hand surgeon members of the Science of Variation Group were randomized to evaluate radiographs and 35 hand surgeons to evaluate radiographs and MRI scans of 26 patients for the presence of Kienböck disease, the lunate type, and the ulnar variance. We used Fleiss' kappa analysis to assess the interobserver agreement for categorical variables and compared the κ values between the 2 groups. RESULTS: We found that agreement on the diagnosis of early Kienböck disease was fair (κ, 0.36) among observers who evaluated radiographs alone and moderate (κ, 0.58) among observers who evaluated MRI scans in addition to radiographs, and that the difference in κ values was not statistically significant ( P = .057). Agreement did not differ between observers based on imaging modality with regard to the assessment of the lunate type ( P = .75) and ulnar variance ( P = .15). CONCLUSIONS: We found, with the numbers evaluated, a notable but nonsignificant difference in agreement in favor of observers who evaluated MRI scans in addition to radiographs compared with radiographs alone. Surgeons should be aware that the diagnosis of Kienböck disease in the precollapse stages is not well defined, as evidenced by the substantial interobserver variability.

Does a Comparison View Improve the Reliability of Staging Wrist Osteoarthritis?

BACKGROUND: Radiological grading of wrist osteoarthritis associated with scaphoid nonunion advanced collapse (SNAC) can be difficult. A comparison radiograph of the contralateral healthy wrist and an educational training in the various SNAC stages may improve reliability. Our purposes were to evaluate the difference in the reliability: (1) between observers who rate SNAC wrists with and without a comparison radiograph; and (2) between observers who receive training prior to ratings and those who do not. METHODS: In this cross-sectional survey study, 82 fully trained orthopedic or hand surgeons rated anteroposterior radiographs of 19 patient wrists following a scaphoid nonunion based on SNAC stages 0 to 4. Observers were randomized online in 4 groups: one group rated unilateral views without training, a second group unilateral views with training, a third group bilateral views without training, and a fourth group bilateral views with training. Training included a 1-page clarification of the SNAC stages. Interobserver agreement was calculated using kappa statistics. RESULTS: There was no significant difference between agreement between observers who rated unilateral radiographs (κ = 0.55) and who rated bilateral radiographs (κ = 0.58) ( P = .14), nor between agreement between observers who received training (κ = 0.59) and who did not (κ = 0.54) ( P = .058). CONCLUSIONS: The use of an additional comparison view and/or training does not seem to be clinically relevant in SNAC staging. There is room for improvement in the way we assess patients with SNAC wrists.

How do Orthopaedic Surgeons Address Psychological Aspects of Illness?

BACKGROUND: Orthopaedic surgeons have a pivotal role in transitioning the care of orthopedic patients from a biomedical to a biopsychosocial model. In an effort to foster this transition, we designed a study aimed to determine surgeons' attitudes and practice of noticing, screening, discussing psychological illness with patients, as well as making referrals to address psychosocial issues in patients in need. Additionally, we asked surgeons to rank order potential barriers to and reasons for referrals to psychosocial treatment. METHODS: Orthopaedic surgeons members of the Science and Variation Group and Ankle Platform (N =350) completed demographics, and a 4-part survey assessing the degree to which surgeons notice, assess, screen and refer for psychological treatments, as well ranked ordered barriers to engaging in these processes. RESULTS: As a group surgeons were neutral to referral for psychological treatment and formal screening of psychological factors, and somewhat likely to notice and discuss psychological factors. Surgeons were more likely to refer for psychological treatment if they engaged in research, or if they reside in South America as opposed to North America. The highest ranked barriers to screening, noticing, discussing and referring for psychological treatment were lack of time, stigma and feeling uncomfortable. CONCLUSION: Overall surgeons are likely to notice and discuss psychological factors, but less likely to formally screen or refer for psychological treatment. Transition to biopsychosocial models should focus on problem solving these barriers by teaching surgeons communication skills to increase comfort with discussing psychoemotional factors associated with orthopedic problems. The use of empathic communication can be very helpful in normalizing the difficulty of coping with an orthopedic condition, and may facilitate referral.

Online Studies on Variation in Orthopedic Surgery: Computed Tomography in MPEG4 Versus DICOM Format.

The purpose of this study was to compare the observer participation and satisfaction as well as interobserver reliability between two online platforms, Science of Variation Group (SOVG) and Traumaplatform Study Collaborative, for the evaluation of complex tibial plateau fractures using computed tomography in MPEG4 and DICOM format. A total of 143 observers started with the online evaluation of 15 complex tibial plateau fractures via either the SOVG or Traumaplatform Study Collaborative websites using MPEG4 videos or a DICOM viewer, respectively. Observers were asked to indicate the absence or presence of four tibial plateau fracture characteristics and to rate their satisfaction with the evaluation as provided by the respective online platforms. The observer participation rate was significantly higher in the SOVG (MPEG4 video) group compared to that in the Traumaplatform Study Collaborative (DICOM viewer) group (75 and 43%, respectively; P < 0.001). The median observer satisfaction with the online evaluation was seven (range, 0-10) using MPEG4 video compared to six (range, 1-9) using DICOM viewer (P = 0.11). The interobserver reliability for recognition of fracture characteristics in complex tibial plateau fractures was higher for the evaluation using MPEG4 video. In conclusion, observer participation and interobserver reliability for the characterization of tibial plateau fractures was greater with MPEG4 videos than with a standard DICOM viewer, while there was no difference in observer satisfaction. Future reliability studies should account for the method of delivering images.

The Effect of Two Factors on Interobserver Reliability for Proximal Humeral Fractures.

INTRODUCTION: The purpose of this study was to assess whether training observers and simplifying proximal humeral fracture classifications improve interobserver reliability among a large number of orthopaedic surgeons. METHODS: One hundred eighty-five observers were randomized to receive training or no training in a simple classification for proximal humeral fractures before evaluating preoperative radiographs of a consecutive series of 30 patients who were treated with open reduction and internal fixation. RESULTS: The overall interobserver reliability of the simple proximal humeral fracture classification system was low and not significantly different between the training and the no training group (κ = 0.20 and κ = 0.18, respectively; P = 0.10). Subgroup analyses showed that training improved the agreement among surgeons who have been in independent practice ≤5 years (κ = 0.23 versus κ = 0.14; P < 0.001), surgeons from the United States (κ = 0.23 versus κ = 0.16; P = 0.002), and general orthopaedic surgeons (κ = 0.42 versus κ = 0.15; P = 0.021). DISCUSSION: Simplifying classifications and training observers did not improve the interobserver reliability for the diagnosis of proximal humeral fractures. However, training observers improved interobserver reliability of a simple proximal humeral fracture classification system among surgeons from the United States and, in particular, younger and less specialized surgeons. This finding may suggest that our interpretations of radiographic information might become more fixed and immutable with experience.

Assessment of Decisional Conflict about the Treatment of Trigger Finger, Comparing Patients and Physicians.

BACKGROUND: As an early step in the development of a decision aid for idiopathic trigger finger (TF) we were interested in the level of decisional conflict experienced by patients and hand surgeons. This study tested the null hypothesis that there is no difference in decisional conflict between patients with one or more idiopathic trigger fingers and hand surgeons. Secondary analyses address the differences between patients and surgeons regarding the influence of the DCS-subcategories on the level of decisional conflict, as well as the influence of patient and physician demographics, the level of self-efficacy, and satisfaction with care on decisional conflict. METHODS: One hundred and five hand surgeon-members of the Science of Variation Group (SOVG) and 84 patients with idiopathic TF completed the survey regarding the Decisional Conflict Scale. Patients also filled out the Pain Self-efficacy Questionnaire (PSEQ) and the Patient Doctor Relationship Questionnaire (PDRQ-9). RESULTS: On average, patients had decisional conflict comparable to physicians, but by specific category patients felt less informed and supported than physicians. The only factors associated with greater decisional conflict was the relationship between the patient and doctor. CONCLUSIONS: There is a low, but measurable level of decisional conflict among patients and surgeons regarding idiopathic trigger finger. Studies testing the ability of decision aids to reduce decisional conflict and improve patient empowerment and satisfaction with care are merited.

Coaching of patients with an isolated minimally displaced fracture of the radial head immediately increases range of motion.

STUDY DESIGN: Prospective cohort. INTRODUCTION: Elbow stiffness is the most common adverse event after isolated radial head fractures. PURPOSE OF THE STUDY: To assess the effect of coaching on elbow motion during the same office visit in patients with such fractures. METHODS: We enrolled 49 adult patients with minimally displaced radial head fractures, within 14 days of injury. After diagnosis, we measured demographics, catastrophic thinking, health anxiety, symptoms of depression, upper extremity-specific symptoms and disability, pain, and elbow and wrist motion. The patient was taught to apply an effective stretch in spite of the pain to limit stiffness, and elbow motion was measured again. RESULTS: With the exception of radial deviation and pronation, motion measures improved slightly but significantly on average immediately after coaching. Elbow flexion improved from 79% (110° ± 22°) of the uninjured side to 88% (122° ± 18°) after coaching (P < .001); elbow extension improved from 71% (29° ± 14°) to 78% (22° ± 15°) (P = .0012). DISCUSSION: Instruction that stretching exercises are healthy even when painful resulted in immediate improvements in motion. Prospective studies comparing different strategies for coaching patients regarding painful stretches might help clarify the optimal approach. LEVEL OF EVIDENCE: Therapeutic level 4.

Interpretation of Post-operative Distal Humerus Radiographs After Internal Fixation: Prediction of Later Loss of Fixation.

PURPOSE: Stable fixation of distal humerus fracture fragments is necessary for adequate healing and maintenance of reduction. The purpose of this study was to measure the reliability and accuracy of interpretation of postoperative radiographs to predict which implants will loosen or break after operative treatment of bicolumnar distal humerus fractures. We also addressed agreement among surgeons regarding which fracture fixation will loosen or break and the influence of years in independent practice, location of practice, and so forth. METHODS: A total of 232 orthopedic residents and surgeons from around the world evaluated 24 anteroposterior and lateral radiographs of distal humerus fractures on a Web-based platform to predict which implants would loosen or break. Agreement among observers was measured using the multi-rater kappa measure. RESULTS: The sensitivity of prediction of failure of fixation of distal humerus fracture on radiographs was 63%, specificity was 53%, positive predictive value was 36%, the negative predictive value was 78%, and accuracy was 56%. There was fair interobserver agreement (κ = 0.27) regarding predictions of failure of fixation of distal humerus fracture on radiographs. Interobserver variability did not change when assessed for the various subgroups. CONCLUSIONS: When experienced and skilled surgeons perform fixation of type C distal humerus fracture, the immediate postoperative radiograph is not predictive of fixation failure. Reoperation based on the probability of failure might not be advisable. TYPE OF STUDY/LEVEL OF EVIDENCE: Diagnostic III.

Among Musculoskeletal Surgeons, Job Dissatisfaction Is Associated With Burnout.

BACKGROUND: Burnout is common in professions such as medicine in which employees have frequent and often stressful interpersonal interactions where empathy and emotional control are important. Burnout can lead to decreased effectiveness at work, negative health outcomes, and less job satisfaction. A relationship between burnout and job satisfaction is established for several types of physicians but is less studied among surgeons who treat musculoskeletal conditions. QUESTIONS/PURPOSES: We asked: (1) For surgeons treating musculoskeletal conditions, what risk factors are associated with worse job dissatisfaction? (2) What risk factors are associated with burnout symptoms? METHODS: Two hundred ten (52% of all active members of the Science of Variation Group [SOVG]) surgeons who treat musculoskeletal conditions (94% orthopaedic surgeons and 6% trauma surgeons; in Europe, general trauma surgeons do most of the fracture surgery) completed the Global Job Satisfaction instrument, Shirom-Malamed Burnout Measure, and provided practice and surgeon characteristics. Most surgeons were male (193 surgeons, 92%) and most were academically employed (186 surgeons, 89%). Factors independently associated with job satisfaction and burnout were identified with multivariable analysis. RESULTS: Greater symptoms of burnout (ß, -7.13; standard error [SE], 0.75; 95% CI, -8.60 to -5.66; p < 0.001; adjusted R(2), 0.33) was the only factor independently associated with lower job satisfaction. Having children (ß, -0.45; SE, 0.0.21; 95% CI, -0.85 to -0.043; p = 0.030; adjusted R(2), 0.046) was the only factor independently associated with fewer symptoms of burnout. CONCLUSIONS: Among an active research group of largely academic surgeons treating musculoskeletal conditions, most are satisfied with their job. Efforts to limit burnout and job satisfaction by optimizing engagement in and deriving meaning from the work are effective in other settings and merit attention among surgeons. LEVEL OF EVIDENCE: Level II, prognostic study.

6-week radiographs unsuitable for diagnosis of suspected scaphoid fractures.

INTRODUCTION: Six week follow-up radiographs are a common reference standard for the diagnosis of suspected scaphoid fractures. The main purpose of this study was to evaluate the interobserver reliability and diagnostic performance characteristics of 6-weeks radiographs for the detection of scaphoid fractures. In addition, two online techniques for evaluating radiographs were compared. MATERIALS AND METHODS: A total of 81 orthopedic surgeons affiliated with the Science of Variation Group assessed initial and 6-week scaphoid-specific radiographs of a consecutive series of 34 patients with suspected scaphoid fractures. They were randomized in two groups for evaluation, one used a standard website showing JPEG files and one a more sophisticated image viewer (DICOM). The goal was to identify the presence or absence of a (consolidated) scaphoid fracture. Interobserver reliability was calculated using the multirater kappa measure. Diagnostic performance characteristics were calculated according to standard formulas with CT and MRI upon presentation in the emergency department as reference standards. RESULTS: The interobserver agreement of 6-week radiographs for the diagnosis of scaphoid fractures was slight for both JPEG and DICOM (k = 0.15 and k = 0.14, respectively). The sensitivity (range 42-79 %) and negative predictive value (range 79-94 %) were significantly higher using a DICOM viewer compared to JPEG images. There were no differences in specificity (range 53-59 %), accuracy (range 53-58 %), and positive predictive value (range 14-26 %) between the groups. CONCLUSIONS: Due to low agreement between observers for the recognition of scaphoid fractures and poor diagnostic performance, 6-week radiographs are not adequate for evaluating suspected scaphoid fractures. The online evaluation of radiographs using a DICOM viewer seem to improve diagnostic performance characteristics compared to static JPEG images and future reliability and diagnostic studies should account for variation due to the method of delivering medical images. LEVEL OF EVIDENCE: Diagnostic level II.