Examiners and content and site: Oh My! A national organization's investigation of score variation in large-scale performance assessments.

Examiner effects and content specificity are two well known sources of construct irrelevant variance that present great challenges in performance-based assessments. National medical organizations that are responsible for large-scale performance based assessments experience an additional challenge as they are responsible for administering qualification examinations to physician candidates at several locations and institutions. This study explores the impact of site location as a source of score variation in a large-scale national assessment used to measure the readiness of internationally educated physician candidates for residency programs. Data from the Medical Council of Canada's National Assessment Collaboration were analyzed using Hierarchical Linear Modeling and Rasch Analyses. Consistent with previous research, problematic variance due to examiner effects and content specificity was found. Additionally, site location was also identified as a potential source of construct irrelevant variance in examination scores.

Cross-national trends in perceived school pressure by gender and age from 1994 to 2010.

BACKGROUND: Pressure within school can be a critical component in understanding how the school experience influences young people's intellectual development, physical and mental health and future educational decisions. METHODS: Data from five survey rounds (1993/1994, 1997/1998, 2001/2002, 2005/2006 and 2009/2010) were used to examine time-, age- and gender-related trends in the amounts of reported school pressure among 11-, 13- and 15-year-olds, in five different regions (North America, Great Britain, Eastern Europe, Nordic and Germanic countries). RESULTS: Across the regions the reported perceptions of school pressure did not change between 1994 and 2010, despite a temporary increase in 2002 and 2006. With the exception of children at 11 years of age, girls reported higher levels of school pressure than boys (Cohen's d from 0.12 to 0.58) and school pressure was higher in older age groups. These findings were consistent across countries. Regionally, children in North America reported the highest levels of school pressure, and students in the Germanic countries the lowest. CONCLUSION: Factors associated with child development and differences in societal expectations and structures, along with the possible, albeit, differential impact of the Programme for International Student Assessment (PISA), may partially explain the differences and trends found in school pressure. School pressure increases alongside the onset of adolescence and the shift from elementary school to the higher demanding expectations of secondary education. Time-related increases in school pressure occurred in the years following the release of the PISA results, and were larger in those regions in which results were less positive.

Psychometric properties of the multiple mini-interview used for medical admissions: findings from generalizability and Rasch analyses.

The multiple mini-interview (MMI) has become an increasingly popular admissions method for selecting prospective students into professional programs (e.g., medical school). The MMI uses a series of short, labour intensive simulation stations and scenario interviews to more effectively assess applicants' non-cognitive qualities such as empathy, critical thinking, integrity, and communication. MMI data from 455 medical school applicants were analyzed using: (1) Generalizability Theory to estimate the generalizability of the MMI and identify sources of error; and (2) the Many-Facet Rasch Model, to identify misfitting examinees, items and raters. Consistent with previous research, our results support the reliability of MMI process. However, it appears that the non-cognitive qualities are not being measured as unique constructs across stations.

Competency-based education calls for programmatic assessment: But what does this look like in practice?

RATIONALE, AIMS, AND OBJECTIVES: Programmatic assessment has been identified as a system-oriented approach to achieving the multiple purposes for assessment within Competency-Based Medical Education (CBME, i.e., formative, summative, and program improvement). While there are well-established principles for designing and evaluating programs of assessment, few studies illustrate and critically interpret, what a system of programmatic assessment looks like in practice. This study aims to use systems thinking and the 'two communities' metaphor to interpret a model of programmatic assessment and to identify challenges and opportunities with operationalization. METHOD: An interpretive case study was used to investigate how programmatic assessment is being operationalized within one competency-based residency program at a Canadian university. Qualitative data were collected from residents, faculty, and program leadership via semi-structured group and individual interviews conducted at nine months post-CBME implementation. Data were analyzed using a combination of data-based inductive analysis and theory-derived deductive analysis. RESULTS: In this model, Academic Advisors had a central role in brokering assessment data between communities responsible for producing and using residents' performance information for decision making (i.e., formative, summative/evaluative, and program improvement). As system intermediaries, Academic Advisors were in a privileged position to see how the parts of the assessment system contributed to the functioning of the whole and could identify which system components were not functioning as intended. Challenges were identified with the documentation of residents' performance information (i.e., system inputs); use of low-stakes formative assessments to inform high-stakes evaluative judgments about the achievement of competence standards; and gaps in feedback mechanisms for closing learning loops. CONCLUSIONS: The findings of this research suggest that program stakeholders can benefit from a systems perspective regarding how their assessment practices contribute to the efficacy of the system as a whole. Academic Advisors are well positioned to support educational development efforts focused on overcoming challenges with operationalizing programmatic assessment.

Derivation of some contemporary scales to measure adolescent risk-taking in Canada.

OBJECTIVES: To derive a contemporary series of composite indicators of adolescent risk-taking, inspired by the US CDC Framework and Problem Behaviour Theory. METHODS: Factor analyses were performed on 28-risk behaviours in a nationally representative sample of 30,096 Grades 6-10 students from the 2014 Canadian Health Behaviour in School-aged Children study. RESULTS: Three composite indicators emerged from our analysis: (1) Overt Risk-Taking (i.e., substance use, caffeinated energy drink consumption, fighting, and risky sexual behaviour), (2) Aversion to a Healthy Lifestyle (i.e., physical inactivity and low fruit and vegetable consumption), and (3) Screen Time Syndrome (i.e., abnormally high screen time use combined with unhealthy snacking). These three composite indicators of risk-taking were observed consistently with strong psychometric properties across different grade groups (6-8, 9-10). CONCLUSIONS: The three composite indicators of adolescent risk-taking each draw from multiple domains within the CDC framework, and support a novel, empirically directed approach of conceptualizing multiple risk behaviours among adolescents. The measures also highlight the breadth and diversity of risk behaviour engagement among Canadian adolescents. Research and preventive interventions should simultaneously consider the related behaviours within each of these composite indicators.

Mixed Messages or Miscommunication? Investigating the Relationship Between Assessors' Workplace-Based Assessment Scores and Written Comments.

PURPOSE: The shift toward broader, programmatic assessment has revolutionized the approaches that many take in assessing medical competence. To understand the association between quantitative and qualitative evaluations, the authors explored the relationships that exist among assessors' checklist scores, task ratings, global ratings, and written comments. METHOD: The authors collected and analyzed, using regression analyses, data from the McMaster Modular Assessment Program. The data were from emergency medicine residents in their first or second year of postgraduate training from 2012 through 2014. Additionally, using content analysis, the authors analyzed narrative comments corresponding to the "done" and "done, but needs attention" checklist score options. RESULTS: The regression analyses revealed that the task ratings, provided by faculty assessors, are associated with the use of the "done, but needs attention" checklist score option. Analyses also identified that the "done, but needs attention" option is associated with a narrative comment that is balanced, providing both strengths and areas for improvement. Analysis of qualitative comments revealed differences in the type of comments provided to higher- and lower-performing residents. CONCLUSIONS: This study highlights some of the relationships that exist among checklist scores, rating scales, and written comments. The findings highlight that task ratings are associated with checklist options while global ratings are not. Furthermore, analysis of written comments supports the notion of a "hidden code" used to communicate assessors' evaluation of medical competence, especially when communicating areas for improvement or concern. This study has implications for how individuals should interpret information obtained from qualitative assessments.

The relationship of schools to emotional health and bullying.

OBJECTIVES: To examine the extent to which school climate and school pressure could predict other aspects of adolescents' lives, most particularly their emotional health and bullying. Furthermore, the study sought to investigate if these relationships were consistent across countries. METHODS: Participants were 11-, 13-, and 15-year-olds from 26 European countries/regions, Canada, the United States, and Israel. Participants completed surveys focusing on health behaviours and lifestyles, using a contextual framework. Using cluster analytic techniques, three clusters were created varying on school pressure and perceived school climate. These clusters were then examined using variables not used in the clustering. RESULTS: Students in the cluster having the most positive relationships to school outcomes, including academic achievement, truancy, teacher and peer support, also had the most positive emotional health and the lowest incidence of bullying. Similarly, those in the poorest cluster in terms of school also had the poorest outcomes in terms of emotional health and bullying. CONCLUSIONS: These relatively small but significant associations suggest that schools may have a small role in supporting children's emotional well-being and ameliorate the presence of bullying.