The Social Media Index as an Indicator of Quality for Emergency Medicine Blogs: A METRIQ Study.

STUDY OBJECTIVE: Online educational resources such as blogs are increasingly used for education by emergency medicine clinicians. The Social Media Index was developed to quantify their relative impact. The Medical Education Translational Resources: Indicators of Quality (METRIQ) study was conducted in part to determine the association between the Social Media Index score and quality as measured by gestalt and previously derived quality instruments. METHODS: Ten blogs were randomly selected from a list of emergency medicine and critical care Web sites. The 2 most recent clinically oriented blog posts published on these blogs were evaluated with gestalt, the Academic Life in Emergency Medicine Approved Instructional Resources (ALiEM AIR) score, and the METRIQ-8 score. Volunteer raters (including medical students, emergency medicine residents, and emergency medicine attending physicians) were identified with a multimodal recruitment methodology. The Social Media Index was calculated in February 2016, November 2016, April 2017, and December 2017. Pearson's correlations were calculated between the Social Media Index and the average rater gestalt, ALiEM AIR score, and METRIQ-8 score. RESULTS: A total of 309 of 330 raters completed all ratings (93.6%). The Social Media Index correlated moderately to strongly with the mean rater gestalt ratings (range 0.69 to 0.76) and moderately with the mean rater ALiEM AIR score (range 0.55 to 0.61) and METRIQ-8 score (range 0.53 to 0.57) during the month of the blog post's selection and for 2 years after. CONCLUSION: The Social Media Index's correlation with multiple quality evaluation instruments over time supports the hypothesis that it is associated with overall Web site quality. It can play a role in guiding individuals to high-quality resources that can be reviewed with critical appraisal techniques.

Quality Evaluation Scores are no more Reliable than Gestalt in Evaluating the Quality of Emergency Medicine Blogs: A METRIQ Study.

Construct: We investigated the quality of emergency medicine (EM) blogs as educational resources. PURPOSE: Online medical education resources such as blogs are increasingly used by EM trainees and clinicians. However, quality evaluations of these resources using gestalt are unreliable. We investigated the reliability of two previously derived quality evaluation instruments for blogs. APPROACH: Sixty English-language EM websites that published clinically oriented blog posts between January 1 and February 24, 2016, were identified. A random number generator selected 10 websites, and the 2 most recent clinically oriented blog posts from each site were evaluated using gestalt, the Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) score, and the Medical Education Translational Resources: Impact and Quality (METRIQ-8) score, by a sample of medical students, EM residents, and EM attendings. Each rater evaluated all 20 blog posts with gestalt and 15 of the 20 blog posts with the ALiEM AIR and METRIQ-8 scores. Pearson's correlations were calculated between the average scores for each metric. Single-measure intraclass correlation coefficients (ICCs) evaluated the reliability of each instrument. RESULTS: Our study included 121 medical students, 88 EM residents, and 100 EM attendings who completed ratings. The average gestalt rating of each blog post correlated strongly with the average scores for ALiEM AIR (r = .94) and METRIQ-8 (r = .91). Single-measure ICCs were fair for gestalt (0.37, IQR 0.25-0.56), ALiEM AIR (0.41, IQR 0.29-0.60) and METRIQ-8 (0.40, IQR 0.28-0.59). CONCLUSION: The average scores of each blog post correlated strongly with gestalt ratings. However, neither ALiEM AIR nor METRIQ-8 showed higher reliability than gestalt. Improved reliability may be possible through rater training and instrument refinement.

Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study.

STUDY OBJECTIVE: Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. METHODS: We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. RESULTS: One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). CONCLUSION: The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.

Hip surgery and radiology reporting for children with cerebral palsy prior to initiation of a hip surveillance program.

PURPOSE: Hip displacement impacts quality of life for many children with cerebral palsy (CP). While early detection can help avoid dislocation and late-stage surgery, formalized surveillance programs are not ubiquitous. This study aimed to examine: 1) surgical practices around pediatric hip displacement for children with CP in a region without formalized hip surveillance; and 2) utility of MP compared to traditional radiology reporting for quantifying displacement. METHODS: A retrospective chart review examined hip displacement surgeries performed on children with CP between 2007-2016. Surgeries were classified as preventative, reconstructive, or salvage. Pre- and post-operative migration percentage (MP) was calculated for available radiographs using a mobile application and compared using Wilcoxon Signed Ranks test. MPs were also compared with descriptions in the corresponding radiology reports using directed and conventional content analyses. RESULTS: Data from 67 children (115 surgical hips) were included. Primary surgery types included preventative (63.5% hips), reconstructive (36.5%), or salvage (0%). For the 92 hips with both radiology reports and radiographs available, reports contained a range of descriptors that inconsistently reflected the retrospectively-calculated MPs. CONCLUSION: Current radiology reporting practices do not appear to effectively describe hip displacement for children with CP. Therefore, standardized reporting of MP is recommended.

Leveraging a Virtual Community of Practice to Participate in a Survey-based Study: A Description of the METRIQ Study Methodology.

OBJECTIVES: To power the METRIQ (Medical Education Translational Resources: Impact and Quality) Study adequately, we aimed to recruit > 200 medical students, residents, and attendings to complete a 90- to 120-minute survey by leveraging a virtual community of practice (vCoP). METHODS: Participants were recruited using personal (conference campaign and e-mails) and online (a study website and social media campaign utilizing Twitter, Facebook, blogs, podcasts, an infographic, and a YouTube video) techniques that leveraged relationships within a virtual community or practice. Participants received weekly survey reminders for 4 weeks and at the end of the rating period. Survey completion rates were calculated. RESULTS: A total of 380 potential participants completed an intake form (139 medical students, 120 residents, 121 attendings), 330 consented to participate, and 309 (81.3% of interested and 93.9% of consenting participants) completed the full survey (121, 88, and 100, respectively). The required sample size was achieved. CONCLUSIONS: The METRIQ Study utilized a multimodal recruitment campaign that targeted a vCoP. It recruited large numbers of participants with high completion rates. Response rates could not be calculated given the uncertainty surrounding the number of individuals invited to participate.