Using program evaluation to support knowledge translation in an interprofessional primary care team: a case study.

BACKGROUND: Evaluation is a fundamental component in building quality primary care and is ideally situated to support individual, team and organizational learning by offering an accessible form of participatory inquiry. The evaluation literature has begun to recognize the unique features of KT evaluations and has described attributes to consider when evaluating KT activities. While both disciplines have focused on the evaluation of KT activities neither has explored the role of evaluation in KT. The purpose of the paper is to examine how participation in program evaluation can support KT in a primary care setting. METHODS: A mixed methods case study design was used, where evaluation was conceptualized as a change process and intervention. A Memory Clinic at an interprofessional primary care clinic was the setting in which the study was conducted. An evaluation framework, Pathways of Influence provided the theoretical foundation to understand how program evaluation can facilitate the translation of knowledge at the level of the individual, inter-personal (Memory Clinic team) and the organization. Data collection included questionnaires, interviews, evaluation log and document analysis. Questionnaires and interviews were administered both before and after the evaluation: Pattern matching was used to analyze the data based on predetermined propositions. RESULTS: Individuals gained program knowledge that resulted in changes to both individual and program practices. One of the key themes was the importance clinicians placed on local, program based knowledge. The evaluation had less influence on the broader health organization. CONCLUSIONS: Program evaluation facilitated individual, team and organizational learning. The use of evaluation to support KT is ideally suited to a primary care setting by offering relevant and applicable knowledge to primary care team members while being sensitive to local context.

Examiners and content and site: Oh My! A national organization's investigation of score variation in large-scale performance assessments.

Examiner effects and content specificity are two well known sources of construct irrelevant variance that present great challenges in performance-based assessments. National medical organizations that are responsible for large-scale performance based assessments experience an additional challenge as they are responsible for administering qualification examinations to physician candidates at several locations and institutions. This study explores the impact of site location as a source of score variation in a large-scale national assessment used to measure the readiness of internationally educated physician candidates for residency programs. Data from the Medical Council of Canada's National Assessment Collaboration were analyzed using Hierarchical Linear Modeling and Rasch Analyses. Consistent with previous research, problematic variance due to examiner effects and content specificity was found. Additionally, site location was also identified as a potential source of construct irrelevant variance in examination scores.

Cross-national trends in perceived school pressure by gender and age from 1994 to 2010.

BACKGROUND: Pressure within school can be a critical component in understanding how the school experience influences young people's intellectual development, physical and mental health and future educational decisions. METHODS: Data from five survey rounds (1993/1994, 1997/1998, 2001/2002, 2005/2006 and 2009/2010) were used to examine time-, age- and gender-related trends in the amounts of reported school pressure among 11-, 13- and 15-year-olds, in five different regions (North America, Great Britain, Eastern Europe, Nordic and Germanic countries). RESULTS: Across the regions the reported perceptions of school pressure did not change between 1994 and 2010, despite a temporary increase in 2002 and 2006. With the exception of children at 11 years of age, girls reported higher levels of school pressure than boys (Cohen's d from 0.12 to 0.58) and school pressure was higher in older age groups. These findings were consistent across countries. Regionally, children in North America reported the highest levels of school pressure, and students in the Germanic countries the lowest. CONCLUSION: Factors associated with child development and differences in societal expectations and structures, along with the possible, albeit, differential impact of the Programme for International Student Assessment (PISA), may partially explain the differences and trends found in school pressure. School pressure increases alongside the onset of adolescence and the shift from elementary school to the higher demanding expectations of secondary education. Time-related increases in school pressure occurred in the years following the release of the PISA results, and were larger in those regions in which results were less positive.

Psychometric properties of the multiple mini-interview used for medical admissions: findings from generalizability and Rasch analyses.

The multiple mini-interview (MMI) has become an increasingly popular admissions method for selecting prospective students into professional programs (e.g., medical school). The MMI uses a series of short, labour intensive simulation stations and scenario interviews to more effectively assess applicants' non-cognitive qualities such as empathy, critical thinking, integrity, and communication. MMI data from 455 medical school applicants were analyzed using: (1) Generalizability Theory to estimate the generalizability of the MMI and identify sources of error; and (2) the Many-Facet Rasch Model, to identify misfitting examinees, items and raters. Consistent with previous research, our results support the reliability of MMI process. However, it appears that the non-cognitive qualities are not being measured as unique constructs across stations.

Competency-based education calls for programmatic assessment: But what does this look like in practice?

RATIONALE, AIMS, AND OBJECTIVES: Programmatic assessment has been identified as a system-oriented approach to achieving the multiple purposes for assessment within Competency-Based Medical Education (CBME, i.e., formative, summative, and program improvement). While there are well-established principles for designing and evaluating programs of assessment, few studies illustrate and critically interpret, what a system of programmatic assessment looks like in practice. This study aims to use systems thinking and the 'two communities' metaphor to interpret a model of programmatic assessment and to identify challenges and opportunities with operationalization. METHOD: An interpretive case study was used to investigate how programmatic assessment is being operationalized within one competency-based residency program at a Canadian university. Qualitative data were collected from residents, faculty, and program leadership via semi-structured group and individual interviews conducted at nine months post-CBME implementation. Data were analyzed using a combination of data-based inductive analysis and theory-derived deductive analysis. RESULTS: In this model, Academic Advisors had a central role in brokering assessment data between communities responsible for producing and using residents' performance information for decision making (i.e., formative, summative/evaluative, and program improvement). As system intermediaries, Academic Advisors were in a privileged position to see how the parts of the assessment system contributed to the functioning of the whole and could identify which system components were not functioning as intended. Challenges were identified with the documentation of residents' performance information (i.e., system inputs); use of low-stakes formative assessments to inform high-stakes evaluative judgments about the achievement of competence standards; and gaps in feedback mechanisms for closing learning loops. CONCLUSIONS: The findings of this research suggest that program stakeholders can benefit from a systems perspective regarding how their assessment practices contribute to the efficacy of the system as a whole. Academic Advisors are well positioned to support educational development efforts focused on overcoming challenges with operationalizing programmatic assessment.

Comparison of faculty versus structured peer-feedback for acquisitions of basic and intermediate-level surgical skills.

PURPOSE: Video feedback and faculty feedback has been shown to improve surgical performance; however, consistent access to faculty is challenging. We studied the utility of structured peer-feedback (PF) compared to faculty-feedback (FF) during acquisition of basic and intermediate surgical skills. METHODOLOGY: Two randomized non-inferiority trials were conducted with 1st (n = 30) and 2nd year (n = 29) medical students learning skin-lesion excision and closure (S), and single-layer hand-sewn bowel anastomosis (B), respectively. Five attempts were performed. PF participants used an Objective Structured Assessment of Technical Skills tool to guide feedback. Blinded raters assessed video-recorded performance, time and Integrity of the completed task were also assessed. RESULTS: For both tasks performance by PF was comparable to FF (P = 0.111). Both groups improved significantly: performance (B:P < 0.0001, S:P = 0.035), time (B:P = 0.043, S:P < 0.0001) and integrity (B:P < 0.0001, S:P < 0.032). CONCLUSION: Structured peer-feedback is equivalent to faculty-feedback in the acquisition of basic and intermediate surgical skills, giving students freedom to practice independently.

Comparison of Personal Video Technology for Teaching and Assessment of Surgical Skills.

BACKGROUND: Improvements in personal technology have made video recording for teaching and assessment of surgical skills possible. OBJECTIVE: This study compared 5 personal video-recording devices based on their utility (image quality, hardware, mounting options, and accessibility) in recording open surgical procedures. METHODS: Open procedures in a simulated setting were recorded using smartphones and tablets (MOB), laptops (LAP), sports cameras such as GoPro (SC), single-lens reflex cameras (DSLR), and spy camera glasses (SPY). Utility was rated by consensus between 2 investigators trained in observation of technology using a 5-point Likert scale (1, poor, to 5, excellent). RESULTS: A total of 150 hours of muted video were reviewed with a minimum 1 hour for each device. Image quality was good (3.8) across all devices, although this was influenced by the device-mounting requirements (4.2) and its proximity to the area of interest. Device hardware (battery life and storage capacity) was problematic for long procedures (3.8). Availability of devices was high (4.2). CONCLUSIONS: Personal video-recording technology can be used for assessment and teaching of open surgical skills. DSLR and SC provide the best images. DSLR provides the best zoom capability from an offset position, while SC can be placed closer to the operative field without impairing sterility. Laptops provide best overall utility for long procedures due to video file size. All devices require stable recording platforms (eg, bench space, dedicated mounting accessories). Head harnesses (SC, SPY) provide opportunities for "point-of-view" recordings. MOB and LAP can be used for multiple concurrent recordings.

Derivation of some contemporary scales to measure adolescent risk-taking in Canada.

OBJECTIVES: To derive a contemporary series of composite indicators of adolescent risk-taking, inspired by the US CDC Framework and Problem Behaviour Theory. METHODS: Factor analyses were performed on 28-risk behaviours in a nationally representative sample of 30,096 Grades 6-10 students from the 2014 Canadian Health Behaviour in School-aged Children study. RESULTS: Three composite indicators emerged from our analysis: (1) Overt Risk-Taking (i.e., substance use, caffeinated energy drink consumption, fighting, and risky sexual behaviour), (2) Aversion to a Healthy Lifestyle (i.e., physical inactivity and low fruit and vegetable consumption), and (3) Screen Time Syndrome (i.e., abnormally high screen time use combined with unhealthy snacking). These three composite indicators of risk-taking were observed consistently with strong psychometric properties across different grade groups (6-8, 9-10). CONCLUSIONS: The three composite indicators of adolescent risk-taking each draw from multiple domains within the CDC framework, and support a novel, empirically directed approach of conceptualizing multiple risk behaviours among adolescents. The measures also highlight the breadth and diversity of risk behaviour engagement among Canadian adolescents. Research and preventive interventions should simultaneously consider the related behaviours within each of these composite indicators.

Mixed Messages or Miscommunication? Investigating the Relationship Between Assessors' Workplace-Based Assessment Scores and Written Comments.

PURPOSE: The shift toward broader, programmatic assessment has revolutionized the approaches that many take in assessing medical competence. To understand the association between quantitative and qualitative evaluations, the authors explored the relationships that exist among assessors' checklist scores, task ratings, global ratings, and written comments. METHOD: The authors collected and analyzed, using regression analyses, data from the McMaster Modular Assessment Program. The data were from emergency medicine residents in their first or second year of postgraduate training from 2012 through 2014. Additionally, using content analysis, the authors analyzed narrative comments corresponding to the "done" and "done, but needs attention" checklist score options. RESULTS: The regression analyses revealed that the task ratings, provided by faculty assessors, are associated with the use of the "done, but needs attention" checklist score option. Analyses also identified that the "done, but needs attention" option is associated with a narrative comment that is balanced, providing both strengths and areas for improvement. Analysis of qualitative comments revealed differences in the type of comments provided to higher- and lower-performing residents. CONCLUSIONS: This study highlights some of the relationships that exist among checklist scores, rating scales, and written comments. The findings highlight that task ratings are associated with checklist options while global ratings are not. Furthermore, analysis of written comments supports the notion of a "hidden code" used to communicate assessors' evaluation of medical competence, especially when communicating areas for improvement or concern. This study has implications for how individuals should interpret information obtained from qualitative assessments.

Competency-based simulation assessment of resuscitation skills in emergency medicine postgraduate trainees - a Canadian multi-centred study.

BACKGROUND: The use of high-fidelity simulation is emerging as a desirable method for competency-based assessment in postgraduate medical education. We aimed to demonstrate the feasibility and validity of a multi-centre simulation-based Objective Structured Clinical Examination (OSCE) of resuscitation competence with Canadian Emergency Medicine (EM) trainees. METHOD: EM postgraduate trainees (n=98) from five Canadian academic centres participated in a high fidelity, 3-station simulation-based OSCE. Expert panels of three emergency physicians evaluated trainee performances at each centre using the Queen's Simulation Assessment Tool (QSAT). Intraclass correlation coefficients were used to measure the inter-rater reliability, and analysis of variance was used to measure the discriminatory validity of each scenario. A fully crossed generalizability study was also conducted for each examination centre. RESULTS: Inter-rater reliability in four of the five centres was strong with a median absolute intraclass correlation coefficient (ICC) across centres and scenarios of 0.89 [0.65-0.97]. Discriminatory validity was also strong (p < 0.001 for scenarios 1 and 3; p < 0.05 for scenario 2). Generalizability studies found significant variations at two of the study centres. CONCLUSIONS: This study demonstrates the successful pilot administration of a multi-centre, 3-station simulation-based OSCE for the assessment of resuscitation competence in post-graduate Emergency Medicine trainees.

Queen's simulation assessment tool: development and validation of an assessment tool for resuscitation objective structured clinical examination stations in emergency medicine.

INTRODUCTION: The use of high-fidelity simulation is emerging as an effective approach to competency-based assessment in medical education. We aimed to develop and validate a modifiable anchored global assessment scoring tool for simulation-based Objective Structured Clinical Examinations (OSCEs) of resuscitation competence in postgraduate emergency medicine (EM) trainees. METHODS: The Queen's Simulation Assessment Tool was developed using a modified Delphi technique with a panel of EM physicians. Ten standardized resuscitation OSCE scenarios were administered to EM trainees, and their video-recorded performances were scored by 3 independent and blinded EM attending physicians using the Queen's Simulation Assessment Tool. Correlational analyses and analysis of variance were applied to measure the discriminatory capabilities and interrater reliability of each scenario. A fully crossed generalizability study was conducted for each examination. RESULTS: Emergency medicine postgraduate trainees at Queen's University (n = 19-25 per station) participated in the study over 3 years. Interrater reliability showed acceptable levels of agreement for each scenario (mean Spearman ρ = 0.75 [0.63-0.87]; mean interclass correlation coefficient, 0.69 [0.58-0.87]). Discriminatory validity was strong, with senior residents outperforming junior residents in all but 1 of the 10 scenarios. Generalizability studies found the trainee and trainee by scenario interactions as the largest contributors to variance, with G coefficients ranging from 0.67 to 0.84. Resident trainees reported comfort being assessed in the simulation environment (3.8/5) and found the simulation-based examination valuable to their learning (4.6/5). CONCLUSIONS: This study describes the development and validation of a novel modifiable anchored global assessment scoring tool for simulation-based OSCE assessment of resuscitation competence in postgraduate EM trainees.

School differences in adolescent health and wellbeing: findings from the Canadian Health Behaviour in School-aged Children Study.

The goal of this study was to assess the relationship between student- and school-level factors and student health and wellbeing outcomes, and to estimate the variability present at each of the student and school levels for each of three selected health-related outcomes. The data are from the 2006 Canadian Health Behaviour in School-aged children (HBSC) study in which Grades 6-10 students (N=9670) and administrators (N=187) were surveyed. The three outcome measures are Self-Rated Health (SRH), Emotional Wellbeing (EWB), and Subjective Health Complaints (SHC). Individual and school-level effects on the three outcomes were estimated using multi-level modeling. Both individual and school-level factors were associated with students' health. Gender, family wealth, family structure, academic achievement and neighbourhood were significant student-level predictors. We identified random associations between the student-level variables and reported health outcomes. These random effects indicate that the relationships between these student variables and health are not consistent across schools. Student Problem Behaviours at the school were significant predictors of SRH and SHC, while Student Aggression and the school's average socioeconomic standing were significant school-level predictors of EWB. Findings suggest that the environment and disciplinary climate in schools can predict student health and wellbeing outcomes, and may have important implications for school initiatives aimed at students who are struggling both emotionally and academically.

The relationship of schools to emotional health and bullying.

OBJECTIVES: To examine the extent to which school climate and school pressure could predict other aspects of adolescents' lives, most particularly their emotional health and bullying. Furthermore, the study sought to investigate if these relationships were consistent across countries. METHODS: Participants were 11-, 13-, and 15-year-olds from 26 European countries/regions, Canada, the United States, and Israel. Participants completed surveys focusing on health behaviours and lifestyles, using a contextual framework. Using cluster analytic techniques, three clusters were created varying on school pressure and perceived school climate. These clusters were then examined using variables not used in the clustering. RESULTS: Students in the cluster having the most positive relationships to school outcomes, including academic achievement, truancy, teacher and peer support, also had the most positive emotional health and the lowest incidence of bullying. Similarly, those in the poorest cluster in terms of school also had the poorest outcomes in terms of emotional health and bullying. CONCLUSIONS: These relatively small but significant associations suggest that schools may have a small role in supporting children's emotional well-being and ameliorate the presence of bullying.