Applying Kane's validity framework to a simulation based assessment of clinical competence.

Assessment of clinical competence is complex and inference based. Trustworthy and defensible assessment processes must have favourable evidence of validity, particularly where decisions are considered high stakes. We aimed to organize, collect and interpret validity evidence for a high stakes simulation based assessment strategy for certifying paramedics, using Kane's validity framework, which some report as challenging to implement. We describe our experience using the framework, identifying challenges, decisions points, interpretations and lessons learned. We considered data related to four inferences (scoring, generalization, extrapolation, implications) occurring during assessment and treated validity as a series of assumptions we must evaluate, resulting in several hypotheses and proposed analyses. We then interpreted our findings across the four inferences, judging if the evidence supported or refuted our proposed uses of the assessment data. Data evaluating "Scoring" included: (a) desirable tool characteristics, with acceptable inter-item correlations (b) strong item-total correlations (c) low error variance for items and raters, and (d) strong inter-rater reliability. Data evaluating "Generalizability" included: (a) a robust sampling strategy capturing the majority of relevant medical directives, skills and national competencies, and good overall and inter-station reliability. Data evaluating "Extrapolation" included: low correlations between assessment scores by dimension and clinical errors in practice. Data evaluating "Implications" included low error rates in practice. Interpreting our findings according to Kane's framework, we suggest the evidence for scoring, generalization and implications supports use of our simulation-based paramedic assessment strategy as a certifying exam; however, the extrapolation evidence was weak, suggesting exam scores did not predict clinical error rates. Our analysis represents a worked example others can follow when using Kane's validity framework to evaluate, and iteratively develop and refine assessment strategies.

Paramedic contact to balloon in less than 90 minutes: a successful strategy for st-segment elevation myocardial infarction bypass to primary percutaneous coronary intervention in a canadian emergency medical system.

INTRODUCTION: Few systems worldwide have achieved the benchmark time of less than 90 minutes from emergency medical services (EMS) contact to balloon inflation (E2B) for patients sustaining ST-segment elevation myocardial infarction (STEMI). We describe a successful EMS systems approach using a combination of paramedic and 12-lead electrocardiogram (ECG) software interpretation to activate a STEMI bypass protocol. OBJECTIVES: To determine the proportion of patients who met the benchmark of E2B in less than 90 minutes after institution of a regional paramedic activated STEMI bypass to primary PCI protocol. METHODS: We conducted a before-and-after observational cohort study over a 24-month period ending December 31, 2009. Included were all patients diagnosed with STEMI by paramedics trained in ECG acquisition and interpretation and transported by EMS. In the "before" phase of the study, paramedics gave emergency departments (EDs) advance notification of the arrival of STEMI patients and took the patients to the ED of the PCI center. In the "after" phase of the study, paramedics activated a STEMI bypass protocol in which STEMI patients were transported directly to the PCI suite, bypassing the local hospital EDs. Transmission of ECGs did not occur in either phase of the study. RESULTS: We compared the times for 95 STEMI patients in the before phase with the times for 80 STEMI patients in the after phase. The proportion for whom E2B was less than 90 minutes increased from 28.4% before to 91.3% after (p < 0.001). Median E2B time decreased from 107 minutes (interquartile range [IQR] = 30) before to 70 minutes (IQR = 24) after. Median D2B time decreased from 83 minutes (IQR = 34) before to 35 minutes (IQR = 19) after. Median E2D time increased from 21 minutes (IQR = 8) before to 32 minutes (IQR = 17) after. Median differences between phases were significant at p < 0.001. The rate of false-positive PCI laboratory activation during the after phase of the study was 12.4%. CONCLUSIONS: The proportion of patients with E2B times less than 90 minutes significantly improved through the implementation of a paramedic-activated STEMI bypass protocol. Further study is required to determine whether these benefits are reproducible in other EMS systems.