Assessment of Plastic Surgery Residency Applications without United States Medical Licensing Examination Step 1 Scores.

SUMMARY: The United States Medical Licensing Examination announced the changing of Step 1 score reporting from a three-digit number to pass/fail beginning on January 1, 2022. Plastic surgery residency programs have traditionally used United States Medical Licensing Examination Step 1 scores to compare plastic surgery residency applicants. Without a numerical score, the plastic surgery residency application review process will likely change. This article discusses advantages, disadvantages, and steps forward for residency programs related to the upcoming change. The authors encourage programs to continue to seek innovative methods of objectively and holistically evaluating applications.

Medical Education's Wicked Problem: Achieving Equity in Assessment for Medical Learners.

Despite a lack of intent to discriminate, physicians educated in U.S. medical schools and residency programs often take actions that systematically disadvantage minority patients. The approach to assessment of learner performance in medical education can similarly disadvantage minority learners. The adoption of holistic admissions strategies to increase the diversity of medical training programs has not been accompanied by increases in diversity in honor societies, selective residency programs, medical specialties, and medical school faculty. These observations prompt justified concerns about structural and interpersonal bias in assessment. This manuscript characterizes equity in assessment as a "wicked problem" with inherent conflicts, uncertainty, dynamic tensions, and susceptibility to contextual influences. The authors review the underlying individual and structural causes of inequity in assessment. Using an organizational model, they propose strategies to achieve equity in assessment and drive institutional and systemic improvement based on clearly articulated principles. This model addresses the culture, systems, and assessment tools necessary to achieve equitable results that reflect stated principles. Three components of equity in assessment that can be measured and evaluated to confirm success include intrinsic equity (selection and design of assessment tools), contextual equity (the learning environment in which assessment occurs), and instrumental equity (uses of assessment data for learner advancement and selection and program evaluation). A research agenda to address these challenges and controversies and demonstrate reduction in bias and discrimination in medical education is presented.

Who is failing who? A survey exploration of the barriers & enablers to accurate decision making when nursing students' competence is below required standards.

Healthcare practice supports the achievement of programme learning outcomes for nursing students internationally. Within this context the issue of reluctance to fail nursing students, when warranted, is extensively examined within the literature with few definitive solutions emerging. Little is known about the perceived barriers that exacerbate this reluctance, or about factors that might enable and support nurses to fail students. To address this issue our study employed a non-experimental cohort study of nurses (n = 365) from two large teaching hospitals in the Republic of Ireland (ROI). Preceptors in this study clearly enjoyed their role with students, and received positive feedback from students. However, findings suggest that preceptors require more support from colleagues in providing comprehensive evidence in assessment. They also require additional training for providing negative feedback to students. A collaborative approach to competence assessment is recommended, whereby the professional burden can be borne by the team, and where students can engage in a holistic, reflective, learning experience that will ultimately improve their competence.

Evaluation of an undergraduate nursing entrustable professional activities framework: An exploratory qualitative research.

BACKGROUND: Entrustable professional activities (EPA) are mainly used in graduate medical education and professional development and have not been widely implemented in nursing undergraduate settings. Nursing EPAs were therefore developed by the Alice Lee Centre of Nursing to translate theoretical nursing competencies into clinical practice and as forms of standardized clinical assessment tools. Feedback from various stakeholders is required to further refine the framework. OBJECTIVES: To explore the perceptions and experiences of using the new EPA framework in nursing students and hospital and university clinical instructors. DESIGN: An exploratory qualitative study using focus group interviews. PARTICIPANTS: Seven year-one nursing undergraduates, 12 year-two undergraduates, seven university clinical instructors, and 18 hospital clinical instructors participated in this study. METHODS: The students formed five groups, while the clinical instructors formed seven groups, each consisting of three to four participants. Semi-structured interviews were conducted to explore stakeholders' experiences and perceptions of the EPA assessment framework. Thematic analysis was used to analyze the data. RESULTS: Despite initial mixed reactions of confusion and relief, many students and clinical instructors commended the comprehensive and structured EPA framework and its emphasis on holistic patient-centered care. EPAs also allow flexible assessment methods, encourage critical thinking among students, and promote team-based care and peer teaching. However, the assessment using a two-dimensional matrix with multiple competencies for the EPAs, subjective assessment criteria, the lack of standardization using checklists, subjective assessment criteria, and the lack of, manpower, and time may potentially affect the accuracy of the clinical assessments. The effectiveness of the EPA framework was also dependent on the quality of feedback, students' intrinsic motivations, and learning environments. The refinement of EPAs and entrustment levels, a physical checklist, and an incorporation of EPAs into school curricula were recommended to improve practitioners' learning experiences. CONCLUSIONS: The use of EPAs in nursing education continues to be a novel and evolving process. There remains a need for a further refinement of the EPA framework to tailor to instructors' expectations and students' capabilities.

Maximizing Effectiveness of the Holistic Admission Process: Implementing the Multiple Mini Interview Model.

BACKGROUND: Nursing programs have responsibility to become more holistic in admission practices, ensuring they admit applicants based on more than grades and test scores to determine the best fit for their program. PROBLEM: One frequently cited holistic admissions barrier is incorporating qualitative measurement of applicant characteristics through interviews. Finding the best interview model is a challenge, depending on a program's intent to learn about applicants. APPROACH: This article describes the planning and implementation process for incorporating the Multiple Mini Interview (MMI) model into an existing holistic nursing program admission process. OUTCOMES: MMI model implementation resulted in more congruency in student attributes with the college vision, mission, and values and increased diversity. CONCLUSION: An efficient, well-organized MMI interview model was implemented, allowing achievement of overall admission goals and ensuring selected applicants demonstrate attributes aligned with college values and vision and increased diversity.

Australian midwifery student's perceptions of the benefits and challenges associated with completing a portfolio of evidence for initial registration: Paper based and ePortfolios.

Portfolios are used in midwifery education to provide students with a central place to store their accumulative evidence of clinical experience for initial registration in Australia. Portfolio formats can be paper-based or electronic. Anecdotal discussion between midwifery students in Queensland debated the best format to document the requirements for the Australian Nursing and Midwifery Accreditation Council (ANMAC) standard 8.11. Midwifery students using paper-based portfolios envisioned that an ePortfolio would be streamline, simple, safe to use, and able to be used anywhere with WIFI, while some students using an ePortfolio expressed a desire to have a paper-based portfolio as a hard copy. This situation called for evidence of a comparison to resolve the debate. The aim of this study was to investigate midwifery students' experiences of the benefits and challenges between paper-based and ePortfolios when compiling evidence to meet the requirements for initial registration as a midwife in Australia (ANMAC, 2014).

Effectiveness of three-dimensional visualisation on undergraduate nursing and midwifery students' knowledge and achievement in pharmacology: A mixed methods study.

BACKGROUND: Historically nursing and midwifery students have reported difficulty understanding the concept-based science underpinning the interactions between drugs and their targets. This knowledge is crucial for the administration and monitoring of the therapeutic and adverse effects of medications. Immersive three-dimensional technology is reported to enhance understanding of complex scientific concepts but the physical effects of motion sickness may limit its use. OBJECTIVES: This project compared the effectiveness of three-dimensional immersive visualisation technology with two-dimensional visualisation technology as a teaching method to improve student understanding of a pharmacological concept, and to assess levels of student discomfort and satisfaction associated with the experience. DESIGN: Traditional lecture content and presentation about drug-receptor binding was followed by exposure to either a two- or three-dimensional artifact visualising ß-adrenoceptor binding. Two student groups were compared by type of exposure: Group 1 watched the artifact via a three-dimensional immersive facility and Group 2 on a wide, two-dimensional screen. SETTINGS: School of Nursing and Midwifery in a regional university in Southeast Queensland, Australia. PARTICIPANTS: Two hundred and two second year undergraduate nursing and midwifery students. METHODS: The study used mixed methods methodology. Pre- and post- testing of student knowledge was collected using five multiple-choice questions. A post-intervention survey elicited students' self-assessed perceptions of discomfort and satisfaction with the learning experience. RESULTS: The three-dimensional immersive learning experience was comparable to the two-dimensional experience in terms of satisfaction and comfort but resulted in statistically significant improvements in post-test scores. CONCLUSIONS: The three-dimensional experience improved understanding when compared to two-dimensional viewing, satisfied students leaning needs, and caused minimal discomfort. The results are encouraging in terms of using three-dimensional technology to enhance student knowledge of pharmacological concepts necessary for competency in medication management.

The Osteopathic Applicant.

The number of osteopathic students choosing emergency medicine (EM) as a specialty is continuously increasing. However, EM remains a competitive specialty. Accordingly, in this article we guide osteopathic students interested in EM through the Comprehensive Osteopathic Medical Licensing Examination (COMLEX), the United States Medical Licensing Examination (USMLE), third- and fourth-year rotations, and the match process. Additionally, we provide tips on the process of applying to allopathic programs and we discuss the timeline of both the allopathic and osteopathic match. Finally, we discuss the effect of the Single Accreditation System and the Memorandum of Understanding, an agreement to merge the allopathic and osteopathic graduate medical education systems into a single graduate medical education accreditation system. This is expected to be completed as of July 1, 2020. Therefore, we elucidate the expectations for osteopathic applicants (particularly with regards to the USMLE and COMLEX examinations).

Beyond the United States Medical Licensing Examination Score: Assessing Competence for Entering Residency.

Assessments of physician learners during the transition from undergraduate to graduate medical education generate information that may inform their learning and improvement needs, determine readiness to move along the medical education continuum, and predict success in their residency programs. To achieve a constructive transition for the learner, residency program, and patients, high-quality assessments should provide meaningful information regarding applicant characteristics, academic achievement, and competence that lead to a suitable match between the learner and the residency program's culture and focus.The authors discuss alternative assessment models that may correlate with resident physician clinical performance and patient care outcomes. Currently, passing the United States Medical Licensing Examination Step examinations provides one element of reliable assessment data that could inform judgments about a learner's likelihood for success in residency. Yet, learner capabilities in areas beyond those traditionally valued in future physicians, such as life experiences, community engagement, language skills, and leadership attributes, are not afforded the same level of influence when candidate selections are made.While promising new methods of screening and assessment-such as objective structured clinical examinations, holistic assessments, and competency-based assessments-have attracted increased attention in the medical education community, currently they may be expensive, be less psychometrically sound, lack a national comparison group, or be complicated to administer. Future research and experimentation are needed to establish measures that can best meet the needs of programs, faculty, staff, students, and, more importantly, patients.

Examining Demographics, Prior Academic Performance, and United States Medical Licensing Examination Scores.

PURPOSE: To examine whether demographic differences exist in United States Medical Licensing Examination (USMLE) scores and the extent to which any differences are explained by students' prior academic achievement. METHOD: The authors completed hierarchical linear modeling of data for U.S. and Canadian allopathic and osteopathic medical graduates testing on USMLE Step 1 during or after 2010, and completing USMLE Step 3 by 2015. Main outcome measures were computer-based USMLE examinations: Step 1, Step 2 Clinical Knowledge, and Step 3. Test-taker characteristics included sex, self-identified race, U.S. citizenship status, English as a second language, and age at first Step 1 attempt. Covariates included composite Medical College Admission Test (MCAT) scores, undergraduate grade point average (GPA), and previous USMLE scores. RESULTS: A total of 45,154 examinees from 172 medical schools met the inclusion criteria. The sample was 67% white and 48% female; 3.7% non-U.S. citizens; and 7.4% with English as a second language. Hierarchical linear models examined demographic variables with and without covariates including MCAT scores and GPA. All Step examinations showed significant differences by gender after adding covariates, varying by Step. Racial differences were observed for each Step, attenuated by the addition of covariates. CONCLUSIONS: Demographic differences in USMLE performance were tempered by previous examination performance and undergraduate performance. Additional research is required to identify factors that contribute to demographic differences, can aid educators' identification of students who would benefit from assistance preparing for USMLE, and can assist residency program directors in assessing performance measures while meeting diversity goals.

From opening the 'black box' to looking behind the curtain: cognition and context in assessor-based judgements.

The increasing use of direct observation tools to assess routine performance has resulted in the growing reliance on assessor-based judgements in the workplace. However, we have a limited understanding of how assessors make judgements and formulate ratings in real world contexts. The current research on assessor cognition has largely focused on the cognitive domain but the contextual factors are equally important, and both are closely interconnected. This study aimed to explore the perceived cognitive and contextual factors influencing Mini-CEX assessor judgements in the Emergency Department setting. We used a conceptual framework of assessor-based judgement to develop a sequential mixed methods study. We analysed and integrated survey and focus group results to illustrate self-reported cognitive and contextual factors influencing assessor judgements. We used situated cognition theory as a sensitizing lens to explore the interactions between people and their environment. The major factors highlighted through our mixed methods study were: clarity of the assessment, reliance on and variable approach to overall impression (gestalt), role tension especially when giving constructive feedback, prior knowledge of the trainee and case complexity. We identified prevailing tensions between participants (assessors and trainees), interactions (assessment and feedback) and setting. The two practical implications of our research are the need to broaden assessor training to incorporate both cognitive and contextual domains, and the need to develop a more holistic understanding of assessor-based judgements in real world contexts to better inform future research and development in workplace-based assessments.

National survey: Developing a common approach to grading of practice in pre-registration midwifery.

This paper presents the final phase of a national project exploring grading of practice in programmes leading to registration as a midwife in the United Kingdom. The aim was to develop a generic framework for grading practice, enhancing standardisation while enabling flexibility in application of current and new educational standards. A mixed method on-line survey considered existing practice assessment tools, factors contributing to robust and reliable assessment and perceptions of two assessment tools developed by the research team: a 'Lexicon Framework' and 'Rubric', which were tested through scenarios. Participants included 170 midwifery and nursing academics, clinicians and students, representing 20 universities in the UK. Seven key themes emerged, from which an 'Evidence Based Model for Professional Practice Assessment' was developed. The proposed tools were overall positively evaluated and demonstrated a good level of reliability. A national tool to standardise midwifery practice assessment is recommended, and scope for transferability of our tools to all midwifery programmes and to nursing was identified. Other recommendations include engagement of key stakeholders in development of practice assessment documentation, and maintaining the professional purpose of grading practice as central to the process. A set of key principles for assessing practice is presented.

Preparing student midwives for professional practice: Evaluation of a student e-portfolio assessment item.

Capstone experiences facilitate consolidation and application of previous learning, strengthening professional identity and competency. This study evaluates the effectiveness and acceptability of a capstone assessment item, in the form of an e-portfolio, designed to identify and demonstrate preparedness for professional midwifery practice. A sequential explanatory strategy occurred within a mixed method research design. Final year Bachelor of Midwifery students at an Australian university, having completed the e-portfolio assessment, participated in two phases of data collection; an initial online-survey, followed by in-depth exploration of emergent concepts within a focus group. Analysis of the quantitative data identified completing the e-portfolio assessment increased students' skills, knowledge and confidence and promoted reflection and critical thinking. Three themes emerged from the qualitative data; acknowledging growth and development; transitioning to practice, and knowing 'who I am and where I am going'. The e-portfolio assessment meets the aims of a capstone assessment and provides an appropriate framework and authentic opportunity for students to identify and demonstrate their level of preparedness for professional practice, determine their ongoing learning needs and develop strategies for achieving them. The assessment item provides an opportunity to develop and articulate a personal practice philosophy and embeds the principles of lifelong learning.

Development and implementation of a competency-based clinical evaluation tool for midwifery education.

BACKGROUND: The learning goals and evaluation strategies of competency-based midwifery programs must be explicit and well-defined. In the US, didactic learning is evaluated through a standardized certification examination, but standardized clinical competence evaluation is lacking. DESCRIPTION OF THE TOOL: The Midwifery Competency Assessment Tool (MCAT) has been adapted from the International Confederation of Midwives' (ICM) "Essential Competencies" and from the American College of Nurse-Midwives' (ACNM) "Core Competencies", with student self-evaluation based on Benner's Novice-to-Expert theory. The MCAT allows for the measurement and monitoring of competence development in all domains of full-scope practice over the course of the midwifery program. APPLICATION AND EVALUATION OF THE TOOL: Strengths of the MCAT are that it provides clear learning goals and performance evaluations for students, ensures and communicates content mapping across a curriculum, and highlights strengths and gaps in clinical opportunities at individual clinical sites and for entire programs. Challenges of the MCAT lie in balancing the number of competency items to be measured with the tedium of form completion, in ensuring the accuracy of student self-evaluation, and in determining "adequate" competence achievement when particular clinical opportunities are limited. CONCLUSIONS: Use of the MCAT with competency-based clinical education may facilitate a more standardized approach to clinical evaluation, as well as a more strategic approach to clinical site development and use.

Evaluation of the McMahon Competence Assessment Instrument for Use with Midwifery Students During a Simulated Shoulder Dystocia.

INTRODUCTION: Intrapartum emergencies occur infrequently but require a prompt and competent response from the midwife to prevent morbidity and mortality of the woman, fetus, and newborn. Simulation provides the opportunity for student midwives to develop competence in a safe environment. The purpose of this study was to determine the inter-rater reliability of the McMahon Competence Assessment Instrument (MCAI) for use with student midwives during a simulated shoulder dystocia scenario. METHODS: A pilot study using a nonprobability convenience sample was used to evaluate the MCAI. Content validity indices were calculated for the individual items and the overall instrument using data from a panel of expert reviewers. Fourteen student midwives consented to be video recorded while participating in a simulated shoulder dystocia scenario. Three faculty raters used the MCAI to evaluate the student performance. These quantitative data were used to determine the inter-rater reliability of the MCAI. RESULTS: The intraclass correlation coefficient (ICC) was used to assess the inter-rater reliability of MCAI scores between 2 or more raters. The ICC was 0.86 (95% confidence interval, 0.60-0.96). Fleiss's kappa was calculated to determine the inter-rater reliability for individual items. Twenty-three of the 42 items corresponded to excellent strength of agreement. DISCUSSION: This study demonstrates a method to determine the inter-rater reliability of a competence assessment instrument to be used with student midwives. Data produced by this study were used to revise and improve the instrument. Additional research will further document the inter-rater reliability and can be used to determine changes in student competence. Valid and reliable methods of assessment will encourage the use of simulation to efficiently develop the competence of student midwives.

The influence of MCAT and GPA preadmission academic metrics on interview scores.

Medical school admissions interviews are used to assess applicants' nonacademic characteristics as advocated by the Association of American Medical Colleges' Advancing Holistic Review Initiative. The objective of this study is to determine whether academic metrics continue to significantly influence interviewers' scores in holistic processes by blinding interviewers to applicants' undergraduate grade point averages (uGPA) and Medical College Admission Test (MCAT). This study examines academic and demographic predictors of interview scores for two applicant cohorts at the University of Michigan Medical School. In 2012, interviewers were provided applicants' uGPA and MCAT scores; in 2013, these academic metrics were withheld from interviewers' files. Hierarchical regression analysis was conducted to examine the influence of academic and demographic variables on overall cohort interview scores. When interviewers were provided uGPA and MCAT scores, academic metrics explained more variation in interview scores (7.9%) than when interviewers were blinded to these metrics (4.1%). Further analysis showed a statistically significant interaction between cohort and uGPA, indicating that the association between uGPA and interview scores was significantly stronger for the 2012 unblinded cohort compared to the 2013 blinded cohort (ß = .573, P < .05). By contrast, MCAT scores had no interactive effects on interviewer scores. While MCAT scores accounted for some variation in interview scores for both cohorts, only access to uGPA significantly influenced interviewers' scores when looking at interaction effects. Withholding academic metrics from interviewers' files may promote assessment of nonacademic characteristics independently from academic metrics.

Validation of the Australian Midwifery Standards Assessment Tool (AMSAT): A tool to assess midwifery competence.

BACKGROUND: There is no current validated clinical assessment tool to measure the attainment of midwifery student competence in the midwifery practice setting. The lack of a valid assessment tool has led to a proliferation of tools and inconsistency in assessment of, and feedback on student learning. OBJECTIVE: This research aimed to develop and validate a tool to assess competence of midwifery students in practice-based settings. DESIGN: A mixed-methods approach was used and the study implemented in two phases. Phase one involved the development of the AMSAT tool with qualitative feedback from midwifery academics, midwife assessors of students, and midwifery students. In phase two the newly developed AMSAT tool was piloted across a range of midwifery practice settings and ANOVA was used to compare scores across year levels, with feedback being obtained from assessors. FINDINGS: Analysis of 150 AMSAT forms indicate the AMSAT as: reliable (Cronbach alpha greater than 0.9); valid-data extraction loaded predominantly onto one factor; and sensitivity scores indicating level of proficiency increased across the three years. Feedback evaluation forms (n=83) suggest acceptance of this tool for the purpose of both assessing and providing feedback on midwifery student's practice performance and competence. CONCLUSION: The AMSAT is a valid, reliable and acceptable midwifery assessment tool enables consistent assessment of midwifery student competence. This assists benchmarking across midwifery education programs.

Learning theories and tools for the assessment of core nursing competencies in simulation: A theoretical review.

AIM: To identify the theories used to explain learning in simulation and to examine how these theories guided the assessment of learning outcomes related to core competencies in undergraduate nursing students. BACKGROUND: Nurse educators face the challenge of making explicit the outcomes of competency-based education, especially when competencies are conceptualized as holistic and context dependent. DESIGN: Theoretical review. DATA SOURCES: Research papers (N = 182) published between 1999-2015 describing simulation in nursing education. REVIEW METHODS: Two members of the research team extracted data from the papers, including theories used to explain how simulation could engender learning and tools used to assess simulation outcomes. Contingency tables were created to examine the associations between theories, outcomes and tools. RESULTS: Some papers (N = 79) did not provide an explicit theory. The 103 remaining papers identified one or more learning or teaching theories; the most frequent were the National League for Nursing/Jeffries Simulation Framework, Kolb's theory of experiential learning and Bandura's social cognitive theory and concept of self-efficacy. Students' perceptions of simulation, knowledge and self-confidence were the most frequently assessed, mainly via scales designed for the study where they were used. Core competencies were mostly assessed with an observational approach. CONCLUSION: This review highlighted the fact that few studies examined the use of simulation in nursing education through learning theories and via assessment of core competencies. It also identified observational tools used to assess competencies in action, as holistic and context-dependent constructs.