A validated search assessment tool: assessing practice-based learning and improvement in a residency program.

OBJECTIVE: The objective of this study was to validate an assessment instrument for MEDLINE search strategies at an academic medical center. METHOD: Two approaches were used to investigate if the search assessment tool could capture performance differences in search strategy construction. First, data from an evaluation of MEDLINE searches from a pediatric resident's longitudinal assessment were investigated. Second, a cross-section of search strategies from residents in one incoming class was compared with strategies of residents graduating a year later. MEDLINE search strategies formulated by faculty who had been identified as having search expertise were used as a gold standard comparison. Participants were presented with a clinical scenario and asked to identify the search question and conduct a MEDLINE search. Two librarians rated the blinded search strategies. RESULTS: Search strategy scores were significantly higher for residents who received training than the comparison group with no training. There was no significant difference in search strategy scores between senior residents who received training and faculty experts. CONCLUSION: The results provide evidence for the validity of the instrument to evaluate MEDLINE search strategies. This assessment tool can measure improvements in information-seeking skills and provide data to fulfill Accreditation Council for Graduate Medical Education competencies.

Real-time, evidence-based medicine instruction: a randomized controlled trial in a neonatal intensive care unit.

PURPOSE: The study assesses potential for improving residents' evidence-based medicine searching skills in MEDLINE through real-time librarian instruction. SUBJECTS: Ten residents on a rotation in a neonatal intensive care unit participated. METHODOLOGY: Residents were randomized into an instruction and a non-instruction group. Residents generated questions from rounds and searched MEDLINE for answers. Data were collected through observation, search strategy analysis, and surveys. Librarians observed searches and collected data on questions, searching skills, search problems, and the test group's instruction topics. Participants performed standardized searches before, after, and six-months after intervention and were scored using a search strategy analysis tool (1 representing highest score and 5 representing lowest score). Residents completed pre- and post-intervention surveys to measure opinions about MEDLINE and search satisfaction. RESULTS: Post-intervention, the test group formulated better questions, used limits more effectively, and reported greater confidence in using MEDLINE. The control group expressed less satisfaction with retrieval and demonstrated more errors when limiting. The test and control groups had the following average search scores respectively: 3.0 and 3.5 (pre-intervention), 3.3 and 3.4 (post-intervention), and 2.0 and 3.8 (six-month post-intervention). CONCLUSION: Data suggest that measurable learning outcomes were achieved. Residents receiving instruction improved and retained searching skills six-months after intervention.