Teaching and assessing intra-operative consultations in competency-based medical education: development of a workplace-based assessment instrument.

Competency-based medical education (CBME) is being implemented worldwide. In CMBE, residency training is designed around competencies required for unsupervised practice and use entrustable professional activities (EPAs) as workplace "units of assessment". Well-designed workplace-based assessment (WBA) tools are required to document competence of trainees in authentic clinical environments. In this study, we developed a WBA instrument to assess residents' performance of intra-operative pathology consultations and conducted a validity investigation. The entrustment-aligned pathology assessment instrument for intra-operative consultations (EPA-IC) was developed through a national iterative consultation and used clinical supervisors to assess residents' performance at an anatomical pathology program. Psychometric analyses and focus groups were conducted to explore the sources of evidence using modern validity theory: content, response process, internal structure, relations to other variables, and consequences of assessment. The content was considered appropriate, the assessment was feasible and acceptable by residents and supervisors, and it had a positive educational impact by improving performance of intra-operative consultations and feedback to learners. The results had low reliability, which seemed to be related to assessment biases, and supervisors were reluctant to fully entrust trainees due to cultural issues. With CBME implementation, new workplace-based assessment tools are needed in pathology. In this study, we showcased the development of the first instrument for assessing resident's performance of a prototypical entrustable professional activity in pathology using modern education principles and validity theory.

Going Paperless? Issues in Converting a Surgical Assessment Tool to an Electronic Version.

UNLABELLED: CONSTRUCT: The competence of a trainee to perform a surgical procedure was assessed using an electronic tool. BACKGROUND: "Going paperless" in healthcare has received significant attention over the past decades given the numerous potential benefits of converting to electronic health records. Not surprisingly, medical educators have also considered the potential benefits of electronic assessments for their trainees. What literature exists on the transition from paper-based to electronic-based assessments suggests a positive outcome. In contrast, work done examining the transition to and implementation of electronic health records has noted that hospitals who have implemented these systems have not gone paperless despite the benefits of doing so. APPROACH: This study sought to transition a paper-based assessment tool, the Ottawa Surgical Competency Operating Room Evaluation (which has strong evidence for validity) to an electronic version, in three surgical specialties (Orthopedic Surgery, Urology, General Surgery). However, as the project progressed, it became necessary to change the focus of the study to explore the issues of transitioning to a paperless assessment tool as we identified an extremely low participation rate. RESULTS: Over the first 3 months 440 assessment cases were logged. However, only a small portion of these cases were assessed using the electronic tool (Orthopedic Surgery = 16%, Urology = 5%, General Surgery = 0%). Participants identified several barriers in using the electronic assessment tool such as increased time compared to the paper version and technological issues related to the log-in procedure. CONCLUSIONS: Essentially, users want the tool to be as convenient as paper. This is consistent with research on electronic health records implementation but different from previous work in medical education. Thus, we believe our study highlights an important finding. Transitioning from a paper-based assessment tool to an electronic one is not necessarily a neutral process. Consideration of potential barriers and finding solutions to these barriers will be necessary in order to realize the many benefits of electronic assessments.