Quality Evaluation Scores are no more Reliable than Gestalt in Evaluating the Quality of Emergency Medicine Blogs: A METRIQ Study.

Construct: We investigated the quality of emergency medicine (EM) blogs as educational resources. PURPOSE: Online medical education resources such as blogs are increasingly used by EM trainees and clinicians. However, quality evaluations of these resources using gestalt are unreliable. We investigated the reliability of two previously derived quality evaluation instruments for blogs. APPROACH: Sixty English-language EM websites that published clinically oriented blog posts between January 1 and February 24, 2016, were identified. A random number generator selected 10 websites, and the 2 most recent clinically oriented blog posts from each site were evaluated using gestalt, the Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) score, and the Medical Education Translational Resources: Impact and Quality (METRIQ-8) score, by a sample of medical students, EM residents, and EM attendings. Each rater evaluated all 20 blog posts with gestalt and 15 of the 20 blog posts with the ALiEM AIR and METRIQ-8 scores. Pearson's correlations were calculated between the average scores for each metric. Single-measure intraclass correlation coefficients (ICCs) evaluated the reliability of each instrument. RESULTS: Our study included 121 medical students, 88 EM residents, and 100 EM attendings who completed ratings. The average gestalt rating of each blog post correlated strongly with the average scores for ALiEM AIR (r = .94) and METRIQ-8 (r = .91). Single-measure ICCs were fair for gestalt (0.37, IQR 0.25-0.56), ALiEM AIR (0.41, IQR 0.29-0.60) and METRIQ-8 (0.40, IQR 0.28-0.59). CONCLUSION: The average scores of each blog post correlated strongly with gestalt ratings. However, neither ALiEM AIR nor METRIQ-8 showed higher reliability than gestalt. Improved reliability may be possible through rater training and instrument refinement.

Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study.

STUDY OBJECTIVE: Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs. METHODS: We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality. RESULTS: One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8). CONCLUSION: The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.

Helicobacter pylori status among patients undergoing gastroscopy in rural northern Alberta.

OBJECTIVE: To determine the Helicobacter pylori status of patients who underwent gastroscopy. DESIGN: Retrospective chart review. SETTING: Peace River Community Health Centre in rural northwestern Alberta. PARTICIPANTS: Data were collected from patients who had a gastroscopy performed by either of 2 family physicians between January 1, 2011, and December 31, 2012. MAIN OUTCOME MEASURES: The proportion of patients who had positive test results for H pylori overall and among first-time gastroscopy patients. For first-time gastroscopy patients, the associations between H pylori infection and patient age, sex, residence, and procedural indications and findings were explored. RESULTS: A total of 251 gastroscopies were conducted in 229 unique patients during the study period. Overall, 12.4% (95% CI 8.3% to 16.4%) of patients had positive results for H pylori and among the 159 first-time gastroscopy patients, 17.6% (95% CI 11.7% to 23.5%) had positive test results for H pylori. Helicobacter pylori status did not differ significantly by geography, sex, or age. The prevalence of H pylori was higher among patients with H pylori-related indications for gastroscopy (such as dyspepsia and upper gastrointestinal tract bleeding) than among patients with other indications; however, H pylori infection was not statistically significantly greater in patients diagnosed with peptic ulcer disease. CONCLUSION: The prevalence of H pylori infection among patients undergoing gastroscopy in rural northern Alberta appears lower than other Canadian estimates. In regions with low H pylori rates, patients with dyspepsia might be better served by acid suppression and nonsteroidal anti-inflammatory drug cessation before investigating for H pylori infection. Population-based research is required to further describe regional differences in H pylori rates.

Keeping up with the literature: A current practice survey and qualitative needs assessment among emergency physicians.

Purpose: Physicians face the challenge of staying current with a rapidly growing body of evidence and applying it to their practice. How emergency physicians (EPs) do so is unknown. The authors sought to describe and assess needs around EP patterns of evidence-based medicine (EBM) and continuing medical education (CME) resource use. Methods: The authors conducted a multicenter, cross-sectional study in 2019 across 12 tertiary care, community, and suburban emergency department (ED) sites in the greater area of Edmonton. Information on EBM/CME resource use along with barriers and facilitators to staying current was gathered using a rigorously developed survey tool, distributed electronically and by mail. Responses were tabulated and subgroups analyzed using MANOVA and ANOVA tests. Thematic analysis of comments used a phenomenological lens. Results: A total of 118 EPs (40.1%) completed the survey. Listening to podcasts, attending EM conferences, and subscription-based resources were preferred for staying current. Resource use differed by years in practice but not by age, sex, training background, or site type. EBM had an important impact on respondents' practice (average rating 3.8 out of 5, with 5 indicating "practice changing"). Time was an important barrier. Most (62.7%) felt that they did not spend enough time, despite spending a median of 4 to 5 h monthly on EBM. Facilitators (including journal club summaries or lists of practice-relevant papers) had only moderate impacts. Thematic analysis found three themes (importance of EBM, implementation challenges, and dissemination of EBM) and 13 subthemes. Conclusion: EPs preferentially chose podcasts, conferences, and subscription-based resources to stay current with EBM; time was the biggest barrier. These findings help ED leads and educators tailor CME to physician learning preferences to maximize application of EBM to clinical practice. The next steps include developing/curating resources and disseminating the survey on a larger scale to identify opportunities for shared virtual resources.

The revised Approved Instructional Resources score: An improved quality evaluation tool for online educational resources.

BACKGROUND: Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations. OBJECTIVES: We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool. METHODS: As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability. RESULTS: Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI = 0.77 to 0.95). CONCLUSIONS: We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.

Evidence-based Medicine Simulation: A Novel and Practice-relevant Approach to Teaching Real-time Literature Searching to Emergency Medicine Residents.

INTRODUCTION: Evidence-based medicine (EBM) and literature searching skills are competencies within the emergency medicine (EM) residency curriculum. Previously in our residency program, a librarian taught literature searching instruction, including a classroom-based overview of search engines. Learners reported low engagement and poor retention. To improve engagement, interest, and skill retention, we used a novel approach: simulation to teach real-time literature searching. METHODS: Based on a needs assessment of our EM residents, we created a literature searching workshop using a flipped classroom approach and high-fidelity simulation. Goals of the session were to be interactive, engaging, and practice-relevant. With a librarian, we developed a brief list of EM-relevant databases, including tips for searching and links to sites/apps. Prereadings also covered the hierarchy of evidence and formulating a good clinical (PICO) question. Residents (12 junior residents) participated in a high-fidelity simulation involving a stable patient whose management required a literature search to inform decisions. Feedback was collected on the simulation experience. RESULTS: Residents received the list of EM-relevant databases 7 days prior and were instructed to set up and test the resources on their smartphones. The day of the session, one resident volunteered to lead the simulation; all residents participated in the search on their smart phones. Collectively, it took 4.5 minutes to find a study that adequately addressed the clinical question and to manage the patient accordingly. Feedback on the simulation was positive. Students found it "very real and practical" and "immediately institutable into practice." It helped residents learn to efficiently and effectively search the literature while managing a stable patient. CONCLUSION: A flipped-classroom simulation-based teaching strategy made learning literature searching more interesting, engaging, and applicable to EM practice. Based on popular demand, we will continue to use this teaching method.

The Revised METRIQ Score: A Quality Evaluation Tool for Online Educational Resources.

BACKGROUND: With the rapid proliferation of online medical education resources, quality evaluation is increasingly critical. The Medical Education Translational Resources: Impact and Quality (METRIQ) study evaluated the METRIQ-8 quality assessment instrument for blogs and collected feedback to improve it. METHODS: As part of the larger METRIQ study, participants rated the quality of five blog posts on clinical emergency medicine topics using the eight-item METRIQ-8 score. Next, participants used a 7-point Likert scale and free-text comments to evaluate the METRIQ-8 score on ease of use, clarity of items, and likelihood of recommending it to others. Descriptive statistics were calculated and comments were thematically analyzed to guide the development of a revised METRIQ (rMETRIQ) score. RESULTS: A total of 309 emergency medicine attendings, residents, and medical students completed the survey. The majority of participants felt the METRIQ-8 score was easy to use (mean ± SD = 2.7 ± 1.1 out of 7, with 1 indicating strong agreement) and would recommend it to others (2.7 ± 1.3 out of 7, with 1 indicating strong agreement). The thematic analysis suggested clarifying ambiguous questions, shortening the 7-point scale, specifying scoring anchors for the questions, eliminating the "unsure" option, and grouping-related questions. This analysis guided changes that resulted in the rMETRIQ score. CONCLUSION: Feedback on the METRIQ-8 score contributed to the development of the rMETRIQ score, which has improved clarity and usability. Further validity evidence on the rMETRIQ score is required.

Assessment of emergency medicine residents: a systematic review.

BACKGROUND: Competency-based medical education is becoming the new standard for residency programs, including Emergency Medicine (EM). To inform programmatic restructuring, guide resources and identify gaps in publication, we reviewed the published literature on types and frequency of resident assessment. METHODS: We searched MEDLINE, EMBASE, PsycInfo and ERIC from Jan 2005 - June 2014. MeSH terms included "assessment," "residency," and "emergency medicine." We included studies on EM residents reporting either of two primary outcomes: 1) assessment type and 2) assessment frequency per resident. Two reviewers screened abstracts, reviewed full text studies, and abstracted data. Reporting of assessment-related costs was a secondary outcome. RESULTS: The search returned 879 articles; 137 articles were full-text reviewed; 73 met inclusion criteria. Half of the studies (54.8%) were pilot projects and one-quarter (26.0%) described fully implemented assessment tools/programs. Assessment tools (n=111) comprised 12 categories, most commonly: simulation-based assessments (28.8%), written exams (28.8%), and direct observation (26.0%). Median assessment frequency (n=39 studies) was twice per month/rotation (range: daily to once in residency). No studies thoroughly reported costs. CONCLUSION: EM resident assessment commonly uses simulation or direct observation, done once-per-rotation. Implemented assessment systems and assessment-associated costs are poorly reported. Moving forward, routine publication will facilitate transitioning to competency-based medical education.