The power of subjectivity in competency-based assessment.

With the introduction of competency-based undergraduate curriculum in India, a paradigm shift in the assessment methods and tools will be the need of the hour. Competencies are complex combinations of various attributes, many of which being not assessable by objective methods. Assessment of affective and communication domains has always been neglected for want of objective methods. Areas like professionalism, ethics, altruism, and communication-so vital for being an Indian Medical Graduate, can be assessed longitudinally applying subjective means only. Though subjectivity has often been questioned as being biased, it has been proven time and again that a subjective assessment in expert hands gives comparable results as that of any objective assessment. By insisting on objectivity, we may compromise the validity of the assessment and deprive the students of enriched subjective feedback and judgement also. This review highlights the importance of subjective assessment in competency-based assessment and ways and means of improving the rigor of subjective assessment, with particular emphasis on the development and use of rubrics.

Rising to the Challenge: Residency Programs' Experience With Implementing Milestones-Based Assessment.

BACKGROUND: Changes to assessment efforts following the shift to milestones-based assessment in the ACGME Next Accreditation System have not been fully characterized. OBJECTIVE: This study describes themes in initial milestones-based assessment practices with the goal of informing continued implementation and optimization of milestones-based assessment. METHODS: Semistructured interviews were conducted with 15 residency program leaders in 6 specialties at 8 academic medical centers between August and December 2016. We explored what was retained, what was added, and what was changed from pre-milestones assessment efforts. We also examined the perceived impact of the shift to milestones-based assessment on the programs. Thematic analysis began after the first 5 interviews and ended once thematic sufficiency was reached. Two additional authors reviewed the codes, offered critical input, and informed the formation and naming of the final themes. RESULTS: Three themes were identified: (1) program leaders faced challenges to effective implementation; (2) program leaders focused on adaptability and making milestones work in what felt like a less than ideal situation for them; and (3) despite challenges, program leaders see value and utility in their efforts to move to milestones-based assessment. We describe a number of strategies that worked for programs during the transition, with perceived benefits acknowledged. CONCLUSIONS: While adaptation to milestones has occurred and benefits are noted, negative impacts and challenges (eg, perceived lack of implementation guidance and faculty development resources) persist. There are important lessons learned (eg, utilizing implementation experiences formatively to improve curricula and assessment) in the transition to milestones-based assessment.

Moving Toward Summative Competency Assessment to Individualize the Postclerkship Phase.

In the move toward competency-based medical education, leaders have called for standardization of learning outcomes and individualization of the learning process. Significant progress has been made in establishing defined expectations for the knowledge, skills, attitudes, and behaviors required for successful transition to residency training, but individualization of educational processes to assist learners in reaching these competencies has been predominantly conceptual to date. The traditional time-based structure of medical education has posed a challenge to individualization within the curriculum and has led to more attention on innovations that facilitate transition from medical school to residency. However, a shift of focus to the clerkship-to-postclerkship transition point in the undergraduate curriculum provides an opportunity to determine how longitudinal competency-based assessments can be used to facilitate intentional and individualized structuring of the long-debated fourth year.This Perspective demonstrates how 2 institutions-the University of Virginia School of Medicine and the University of Michigan Medical School-are using competency assessments and applying standardized outcomes in decisions about individualization of the postclerkship learning process. One institution assesses Core Entrustable Professional Activities for Entering Residency, whereas the other has incorporated Accreditation Council for Graduate Medical Education core competencies and student career interests to determine degrees of flexibility in the postclerkship phase. Individualization in addition to continued assessment of performance presents an opportunity for intentional use of curriculum time to develop each student to be competently prepared for the transition to residency.

Competency-Based Medical Education implementation: Are we transforming the culture of assessment?

Purpose: Adopting CBME is challenging in medicine. It mandates a change in processes and approach, ultimately a change in institutional culture with stakeholders ideally embracing and valuing the new processes. Adopting the transformational change model, this study describes the shift in assessment culture by Academic Advisors (AAs) and preceptors over three years of CBME implementation in one Department of Family Medicine. Methods: A qualitative grounded theory method was used for this two-part study. Interviews were conducted with 12 AAs in 2013 and nine AAs in 2016 using similar interview questions. Data were analyzed through a constant comparative method. Results: Three overarching themes emerged from the data: (1) specific identified shifts in assessment culture, (2) factors supporting the shifts in culture, and (3) outcomes related to the culture shift. Conclusions: In both parts of the study, participants noted that assessment took more time and effort. In Part 2, however, the effort was mitigated by a sense of value for all stakeholders. With support from the mandate of regulatory bodies, local leadership, department, faculty development and an electronic platform, a cultural transformation occurred in assessment that enhanced learning and teaching, use of embedded standards for performance decisions, and tracking and documentation performance.

Designing programmes of assessment: A participatory approach.

Programmatic approaches to assessment provide purposeful and meaningful assessment yet few examples of their development exist. The aim of this study was to describe the development of a programme of assessment using a participatory action research (PAR) approach. Nine work-based assessors together with three academics met on six occasions to explore the current approach to competency-based assessment in the placement component of a dietetics university course, the findings of which were used to design a programme of assessment. Findings revealed disconnect between current assessment approaches and best practice. The PAR methodology fostered a shared vision for the design of a programmatic approach to assessment and strong leadership was essential. Participants experienced a philosophical shift in their views towards assessment, supporting the implementation of a new assessment programme. This paper is the first to describe a PAR approach as a feasible and effective way forward in the design of programmatic assessment. The approach engaged stakeholders to strengthen their abilities as work-based assessors and produced champions for best practice assessment.

Visions and reality: the idea of competence-oriented assessment for German medical students is not yet realised in licensing examinations.

Objective: Competence orientation, often based on the CanMEDS model, has become an important goal for modern curricula in medical education. The National Competence Based Catalogue of Learning Objectives for Undergraduate Medical Education (NKLM) has been adopted in Germany. However, it is currently unknown whether the vision of competence orientation has also reached the licensing examination procedures. Methods: Therefore, a prospective, descriptive, single-centre, exemplary study design was applied to evaluate 4051 questions/tasks (from 28 examiners at 7 two-day licensing oral-practical exams) for undergraduate medical students at the University of Ulm. The oral and practical questions/tasks as well as the real bedside assessment were assigned to specific competence roles (NKLM section I), categories (NKLM section II) and taxonomy levels of learning domains. Results: Numerous questions/tasks were set per candidate (day 1/2: 70±24/86±19 questions) in the licensing oral-practical exam. Competence roles beyond the "medical expert" were scarcely considered. Furthermore, practical and communication skills at the bedside were hardly addressed (less than 3/15 min). Strikingly, there was a significant predominance of questions with a low-level taxonomy. Conclusions: The data indicate a misalignment of competence-oriented frameworks and the "real world" licensing practical-oral medical exam, which needs improvement in both evaluation and education processes.

Commitment to Change and Challenges to Implementing Changes After Workplace-Based Assessment Rater Training.

PURPOSE: Faculty development for clinical faculty who assess trainees is necessary to improve assessment quality and impor tant for competency-based education. Little is known about what faculty plan to do differently after training. This study explored the changes faculty intended to make after workplace-based assessment rater training, their ability to implement change, predictors of change, and barriers encountered. METHOD: In 2012, 45 outpatient internal medicine faculty preceptors (who supervised residents) from 26 institutions participated in rater training. They completed a commitment to change form listing up to five commitments and ranked (on a 1-5 scale) their motivation for and anticipated difficulty implementing each change. Three months later, participants were interviewed about their ability to implement change and barriers encountered. The authors used logistic regression to examine predictors of change. RESULTS: Of 191 total commitments, the most common commitments focused on what faculty would change about their own teaching (57%) and increasing direct observation (31%). Of the 183 commitments for which follow-up data were available, 39% were fully implemented, 40% were partially implemented, and 20% were not implemented. Lack of time/competing priorities was the most commonly cited barrier. Higher initial motivation (odds ratio [OR] 2.02; 95% confidence interval [CI] 1.14, 3.57) predicted change. As anticipated difficulty increased, implementation became less likely (OR 0.67; 95% CI 0.49, 0.93). CONCLUSIONS: While higher baseline motivation predicted change, multiple system-level barriers undermined ability to implement change. Rater-training faculty development programs should address how faculty motivation and organizational barriers interact and influence ability to change.

A workforce in crisis: a case study to expand allied ophthalmic personnel.

OBJECTIVE: To examine how the development of allied ophthalmic personnel training programs affects human resource capacity. DESIGN: Using a qualitative case study method conducted at a single Ontario institution, this article describes 6 years of establishing a 2-tiered allied ophthalmic personnel training program. PARTICIPANTS: The Kingston Ophthalmic Training Centre participated in the study with 8 leadership and program graduate interviews. METHODS: To assess regional eye health workforce needs, a case study and iterative process used triangulations of the literature, case study, and qualitative interviews with stakeholders. This research was used to develop a model for establishing allied ophthalmic personnel training programs that would result in expanding human resource capacity. RESULTS: Current human resource capacity development and deployment is inadequate to provide the needed eye care services in Canada. A competency-based curriculum and accreditation model as the platform to develop formal academic training programs is essential. Access to quality eye care and patient services can be met by task-shifting from ophthalmologists to appropriately trained allied ophthalmic personnel. CONCLUSION: Establishing formal training programs is one important strategy to supplying a well-skilled, trained, and qualified ophthalmic workforce. This initiative meets the criteria required for quality, relevance, equity, and cost-effectiveness to meet the future demands for ophthalmic patient care.

Introducing 'The Diverse Nature of Defence Healthcare' university module for DMS personnel.

Over the past 10 years the UK Defence Medical Services has deployed healthcare personnel to a variety of operational areas in support of UK Operations. The unique nature of every operational deployment, in conjunction with the wide variety of roles which healthcare staff undertake, necessitates bespoke educational preparation of the military healthcare force. This paper explores the creation and development of one of the four modules which comprise the BSc (Hons) in Defence Health Care studies, entitled 'The Diverse Nature of Defence Healthcare'. It demonstrates the unique contribution that the Defence School of Healthcare Education makes towards Generation and Preparation of the Force for deployment.

Conceptualizing Interprofessional Teams as Multi-Team Systems-Implications for Assessment and Training.

UNLABELLED: SGEA 2015 CONFERENCE ABSTRACT (EDITED). Evaluating Interprofessional Teamwork During a Large-Scale Simulation. Courtney West, Karen Landry, Anna Graham, and Lori Graham. CONSTRUCT: This study investigated the multidimensional measurement of interprofessional (IPE) teamwork as part of large-scale simulation training. BACKGROUND: Healthcare team function has a direct impact on patient safety and quality of care. However, IPE team training has not been the norm. Recognizing the importance of developing team-based collaborative care, our College of Nursing implemented an IPE simulation activity called Disaster Day and invited other professions to participate. The exercise consists of two sessions: one in the morning and another in the afternoon. The disaster scenario is announced just prior to each session, which consists of team building, a 90-minute simulation, and debriefing. Approximately 300 Nursing, Medicine, Pharmacy, Emergency Medical Technicians, and Radiology students and over 500 standardized and volunteer patients participated in the Disaster Day event. To improve student learning outcomes, we created 3 competency-based instruments to evaluate collaborative practice in multidimensional fashion during this exercise. APPROACH: A 20-item IPE Team Observation Instrument designed to assess interprofessional team's attainment of Interprofessional Education Collaborative (IPEC) competencies was completed by 20 faculty and staff observing the Disaster Day simulation. One hundred sixty-six standardized patients completed a 10-item Standardized Patient IPE Team Evaluation Instrument developed from the IPEC competencies and adapted items from the 2014 Henry et al. PIVOT Questionnaire. This instrument assessed the standardized or volunteer patient's perception of the team's collaborative performance. A 29-item IPE Team's Perception of Collaborative Care Questionnaire, also created from the IPEC competencies and divided into 5 categories of Values/Ethics, Roles and Responsibilities, Communication, Teamwork, and Self-Evaluation, was completed by 188 students including 99 from Nursing, 43 from Medicine, 6 from Pharmacy, and 40 participants who belonged to more than one component, were students at another institution, or did not indicate their institution. The team instrument was designed to assess each team member's perception of how well the team and him- or herself met the competencies. Five of the items on the team perceptions questionnaire mirrored items on the standardized patient evaluation: demonstrated leadership practices that led to effective teamwork, discussed care and decisions about that care with patient, described roles and responsibilities clearly, worked well together to coordinate care, and good/effective communication. RESULTS: Internal consistency reliability of the IPE Team Observation Instrument was 0.80. In 18 of the 20 items, more than 50% of observers indicated the item was demonstrated. Of those, 6 of the items were observed by 50% to 75% of the observers, and the remaining 12 were observed by more than 80% of the observers. Internal consistency reliability of the IPE Team's Perception of Collaborative Care Instrument was 0.95. The mean response score-1 (strongly disagree) to 4 (strongly agree)-was calculated for each section of the instrument. The overall mean score was 3.57 (SD = .11). Internal consistency reliability of the Standardized Patient IPE Team Evaluation Instrument was 0.87. The overall mean score was 3.28 (SD = .17). The ratings for the 5 items shared by the standardized patient and team perception instruments were compared using independent sample t tests. Statistically significant differences (p < .05) were present in each case, with the students rating themselves higher on average than the standardized patients did (mean differences between 0.2 and 0.6 on a scale of 1-4). CONCLUSIONS: Multidimensional, competency-based instruments appear to provide a robust view of IPE teamwork; however, challenges remain. Due to the large scale of the simulation exercise, observation-based assessment did not function as well as self- and standardized patient-based assessment. To promote greater variation in observer assessments during future Disaster Day simulations, we plan to adjust the rating scale from "not observed," "observed," and "not applicable" to a 4-point scale and reexamine interrater reliability.

Introducing a competency based Fellowship programme for psychiatry in Australia and New Zealand.

OBJECTIVES: The aim of this paper is to summarise the new psychiatry Fellowship programme and its rationale, highlighting the new inclusions, revised assessment structure, the benefits and structure of the programme. CONCLUSIONS: The 2012 Fellowship programme is based on the CanMEDs educational framework. The Royal Australian and New Zealand College of Psychiatrists (RANZCP) underwent a comprehensive process, adapting the CanMEDs competencies to a psychiatric framework and mapping the curriculum to Fellowship competencies, learning outcomes and developmental descriptors of the various stages of training. The 2012 Fellowship programme introduced summative entrustable professional activities (EPAs), formative workplace-based assessments (WBAs) and revised external assessments.

Formação do terapeuta ocupacional para a gestão / Occupational therapy formation for management

INTRODUÇÃO: Por vários anos, a formação do terapeuta ocupacional esteve voltada para a assistência, mascom as mudanças no mercado de trabalho, nas políticas sociais e de saúde, diferentes espaços e cargos tornaram-se possíveis para esse profissional, dentre eles o de gestor. No entanto, a produção de conhecimento em Terapia Ocupacional na área deGestão é escassa e traz uma lacuna ao longo de sua história no Brasil. OBJETIVO: investigar sobre a formação de terapeutas ocupacionais para a prática de gestão. MÉTODO: trata-se de uma pesquisa descritiva e transversal cuja amostra foi composta por 278 terapeutas ocupacionais de diferentes instituições do Brasil. O instrumento de coleta foi um questionário com perguntas abertas e fechadas. Os dados da pesquisa foram analisados através de estatística descritiva simples e pelo programa IBM – Many Eyes. RESULTADOS: a importância do conteúdo sobre gestão durante a formação acadêmica foi apontada pelos participantes. Cerca de 86% (n=239) dos profissionais relataram dificuldades em relação ao conhecimento do conteúdo de gestão em sua formação.CONCLUSÃO: acredita-se que os resultados obtidos com essa investigação possam servir para a formulação de propostas curriculares que contemplem a formação para a gestão, uma vez que, pelas respostas identificadas, tal formação ainda se mostra bastante incipiente.

INTRODUCTION: for many years, undergraduate courses have focused on occupational therapy as clinicalpractitioners, but with changes in the labour market and social policies, the professional has begun to occupy different posts, including managerial ones. OBJECTIVE: to investigate thetraining of occupational therapists to practise in management. METHOD: this is a descriptive and cross sectional study with a sample of 278 occupational therapists from different institutions in Brazil. The method of data collection was through a questionnaire with both open and closed questions. The survey´s data was analyzed using the software IBM - Many Eyes. OUTCOMES: the importance of management study during undergraduate courses was reported by all participants. Around 86% (n=239) of participants reported diffi culties related to the management contentof their undergraduate courses. CONCLUSION: it is believed that the results obtained from this research may help future proposals that address curricular content, to include management, since the participants’ responses show currently such undergraduate courses still lack content in this area.

From the community to the classroom: the Aboriginal health curriculum at the Northern Ontario School of Medicine.

More undergraduate medical education programs are including curricula concerning the health, culture and history of Aboriginal people. This is in response to growing international recognition of the large divide in health status between Aboriginal and non-Aboriginal people, and the role medical education may play in achieving health equity. In this paper, we describe the development and delivery of the Aboriginal health curriculum at the Northern Ontario School of Medicine (NOSM). We describe a process for curriculum development and delivery, which includes ongoing engagement with Aboriginal communities as well as faculty expertise. Aboriginal health is delivered as a core curriculum, and learning is evaluated in summative assessments. Aboriginal health objectives are present in 4 of 5 required courses, primarily in years 1 and 2. Students attend a required 4-week Aboriginal cultural immersion placement at the end of year 1. Resources of Aboriginal knowledge are integrated into learning. In this paper, we reflect on the key challenges encountered in the development and delivery of the Aboriginal health curriculum. These include differences in Aboriginal and non-Aboriginal knowledge; risk of reinforcing stereotypes in case presentations; negotiation of curricular time; and faculty readiness and development. An organizational commitment to social accountability and the resulting community engagement model have been instrumental in creating a robust, sustainable program in Aboriginal health at NOSM.

Criterion validity of a competency-based assessment center in medical education--a 4-year follow-up study.

INTRODUCTION: Core competencies have progressively gained importance in medical education. In other contexts, especially personnel selection and development, assessment centers (ACs) are used to assess competencies, but there is only a limited number of studies on competency-based ACs in medical education. To the best of our knowledge, the present study provides the first data on the criterion-related validity of a competency-based AC in medical education. METHODS: We developed an AC tailored to measure core competencies relevant to medical education (social-ethical, communicative, self, and teaching) and tested its validity in n=30 first-year medical students using 3- to 4-year follow-up measures such as (a) objective structured clinical examinations (OSCE) on basic clinical skills (n=26), (b) OSCE on communication skills (n=21), and (c) peer feedback (n=18). The AC contained three elements: interview, group discussion, and role play. Additionally, a self-report questionnaire was provided as a basis for the interview. RESULTS: Baseline AC average score and teaching competency correlated moderately with the communication OSCE average score (r=0.41, p=0.03, and r=0.38, p=0.04, respectively). Social-ethical competency in the AC showed a very strong convergent association with the communication OSCE average score (r=0.60, p<0.01). The AC total score also showed a moderate correlation with the overall peer feedback score provided in Year 4 (r=0.38, p=0.06). In addition, communicative competency correlated strongly with the overall peer feedback (r=0.50, p=0.02). We found predominantly low and insignificant correlations between the AC and the OSCE on basic clinical skills (r=-0.33 to 0.30, all p's>0.05). CONCLUSION: The results showed that competency-based ACs can be used at a very early stage of medical training to successfully predict future performance in core competencies.