Racial and Gender Influences on Pass Rates for the UK and Ireland Specialty Board Examinations.

INTRODUCTION: We explored effects of gender, ethnic origin, first language, and training status on scores in the Intercollegiate Specialty Board examinations in the UK and Ireland across the computer-marked written section and in the face-to-face oral and clinical section. METHODS: Demographic characteristics and examination results from 9987 attempts across 177 sittings from 2009 to 2013 were analyzed in an analysis of variance by training status, gender, ethnic origin, first language, and section (computer-marked multiple-choice examination vs face-to-face oral and clinical examination). RESULTS: We found increasing alignment between examiner and candidate characteristics during this period, with a 50% increase in examiners of Asian ethnic origin and a 60% increase in examiners whose first language is not English. The strongest factor in the analysis of variance was training status (F[2, 9818] = 27.67, p < 0.001), with candidates in training significantly outperforming others. Within "core candidates" (first attempt, in training), we found significant main effects for ethnic origin (F[5, 4809] = 2.36, p = 0.04), and first language (F[2, 4809] = 5.29, p = 0.003), but no interaction effects between these factors and section (both F < 1, p > 0.05). CONCLUSIONS: Training status was the most important factor in candidates' results. Although the analysis showed significant effects of ethnic origin and first language within "core candidates," these differences were statistically indistinguishable between the 2 sections of the examination, suggesting that the differential attainment by these factors cannot be attributed to examiner bias in a face-to-face examination.

Constructing and evaluating a validity argument for the final-year ward simulation exercise.

The authors report final-year ward simulation data from the University of Dundee Medical School. Faculty who designed this assessment intend for the final score to represent an individual senior medical student's level of clinical performance. The results are included in each student's portfolio as one source of evidence of the student's capability as a practitioner, professional, and scholar. Our purpose in conducting this study was to illustrate how assessment designers who are creating assessments to evaluate clinical performance might develop propositions and then collect and examine various sources of evidence to construct and evaluate a validity argument. The data were from all 154 medical students who were in their final year of study at the University of Dundee Medical School in the 2010-2011 academic year. To the best of our knowledge, this is the first report on an analysis of senior medical students' clinical performance while they were taking responsibility for the management of a simulated ward. Using multi-facet Rasch measurement and a generalizability theory approach, we examined various sources of validity evidence that the medical school faculty have gathered for a set of six propositions needed to support their use of scores as measures of students' clinical ability. Based on our analysis of the evidence, we would conclude that, by and large, the propositions appear to be sound, and the evidence seems to support their proposed score interpretation. Given the body of evidence collected thus far, their intended interpretation seems defensible.

Improving student selection using multiple mini-interviews with multifaceted Rasch modeling.

PURPOSE: The authors report multiple mini-interview (MMI) selection process data at the University of Dundee Medical School; staff, students, and simulated patients were examiners and investigated how effective this process was in separating candidates for entry into medical school according to the attributes measured, whether the different groups of examiners exhibited systematic differences in their rating patterns, and what effect such differences might have on candidates' scores. METHOD: The 452 candidates assessed in 2009 rotated through the same 10-station MMI that measured six noncognitive attributes. Each candidate was rated by one examiner in each station. Scores were analyzed using Facets software, with candidates, examiners, and stations as facets. The computer program calculated fair average scores that adjusted for examiner severity/leniency and station difficulty. RESULTS: The MMI reliably (0.89) separated the candidates into four statistically distinct levels of noncognitive ability. The Rasch measures accounted for 31.69% of the total variance in the ratings (candidates 16.01%, examiners 11.32%, and stations 4.36%). Students rated more severely than staff and also had more unexpected ratings. Adjusting scores for examiner severity/leniency and station difficulty would have changed the selection outcomes for 9.6% of the candidates. CONCLUSIONS: The analyses highlighted the fact that quality control monitoring is essential to ensure fairness when ranking candidates according to scores obtained in the MMI. The results can be used to identify examiners needing further training, or who should not be included again, as well as stations needing review. "Fair average" scores should be used for ranking the candidates.

The multiple mini-interview in the U.K. context: 3 years of experience at Dundee.

BACKGROUND: The multiple mini-interview (MMI) is a new interview process that Dundee Medical School has recently adopted to assess entrants into its undergraduate medicine course. This involves an 'objective structured clinical examination' like rotational approach in which candidates are assessed on specific attributes at a number of stations. AIMS: To present methodological, questionnaire and psychometric data on the transitional process from traditional interviews to MMIs over a 3-year period and discuss the implications for those considering making this transition. METHODS: To facilitate the transition, a four-station MMI was piloted in 2007. Success encouraged consideration of desirable attributes which were used to develop a full 10-station process which was implemented in 2009 with assessors being recruited from staff, students and simulated patients. A questionnaire was administered to all assessors and candidates who participated in the 2009 MMIs. Cronbach's alpha and Pearson's r and analysis of variances were used to determine the MMI's psychometric properties. Multi-faceted Rasch modelling (MFRM) was modelled to control for assessor leniency/stringency and the impact of using 'fair scores' determined. Analysis was conducted using SPSS 17 and FACETS 3.65.0. RESULTS: The questionnaire confirmed that the process was acceptable to all parties. Cronbach's alpha reliability was satisfactory and consistent. Graduates/mature candidates outperformed U.K. school-leavers and overseas candidates. Using MFRM fair scores would change the selection outcome of 6.2% and 9.6% of candidates in 2009 and 2010, respectively. Students were less lenient, made more use of the full range of the rating scales and were just as reliable as staff. CONCLUSIONS: The strategy of generating institutional support through staged introduction proved effective. The MMI in Dundee was shown to be feasible and displayed sound psychometric properties. Student assessors appeared to perform at least as well as staff. Despite a considerable intellectual and logistical challenge MMIs were successfully introduced and deemed worthwhile.

Curriculum reform in a public health course at a chiropractic college: are we making progress toward improving clinical relevance?

Improving education in health promotion and prevention has been identified as a priority for all accredited professional health care training programs, an issue recently addressed by a collaboration of stakeholders in chiropractic education who developed a model course outline for public health education. Using a course evaluation questionnaire, the authors surveyed students in the public health course at the Canadian Memorial Chiropractic College (CMCC) before and after the implementation of new course content based on the model course outline. Following the new course, there were significant improvements in perceived relevance to chiropractic practice and motivation to learn the material as a foundation for clinical practice. Changes made to the content and delivery of the course based on the model course outline were well received in the short term.

Research in chiropractic education: an update.

PURPOSE: The objectives of this review were to update the 1997 report on research in chiropractic education from the Research Agenda Conference in 1996 and to provide recommendations for the future direction of this research. METHODS: We conducted a review of the medical and chiropractic literature from March 1997 to March 2005 on the 7 thematic areas identified in the Research Agenda Conference's 1997 article (ie, curriculum, assessment, instructional methods, admissions, faculty development and evaluation, postgraduate and continuing education, and patient centeredness). We searched the following databases: MEDLINE, CINAHL, Alt HealthWatch, Index to Chiropractic Literature, MANTIS, and ERIC. We hand searched conference proceedings and relevant journals along with the reference lists of retrieved articles. DISCUSSION: Although research studies in chiropractic education have been conducted on the 7 cited thematic areas, much have focused on instruction, curriculum, assessment, and faculty development. Research studies on areas recommended in the 1997 article, such as patient centeredness and continuing education, are less evident. CONCLUSIONS: Researchers in chiropractic education need to focus their efforts on key areas of importance and interest. They should consider collaborating across the professions with individuals who share the same interest and who have a productive and successful research track record. Individual researchers should be willing to share and acknowledge the work of other colleagues in and across professions. One such area of research that meets these criteria and that which is recommended for consideration is professionalism.

Predictors of performance of students from the Canadian Memorial Chiropractic College on the licensure examinations of the Canadian Chiropractic Examining Board.

PURPOSE: The purpose of this study was to determine if the Canadian Memorial Chiropractic College (CMCC) structured admissions interview and other student measures predict success on the Canadian Chiropractic Examining Board (CCEB) examinations. METHODS: An independent researcher combined CMCC and CCEB data by student name and then anonymized the data and destroyed the linking information. Backward stepwise multiple linear regressions were applied to external dependent variables from the CCEB and internal independent variables from the CMCC. RESULTS: Forty percent of the variance in the basic science examination (R2 = 0.405) was explained by the independent variables of third year objective structured clinical examination practical and second-year grade point average (chi rhoGPA). Forty-five percent of the variance in the applied science examination (R2 = 0.448) was explained by the fourth year final examination and chi rhoGPA for years 4, 2, and 3. Twenty-four percent of the variance in the clinical decision making examination (R2 = 0.239) was explained by chi rhoGPAs of years 4, 1, 2, and 3. Sixteen percent of the variance in the clinical skills examination (R2 = 0.160) was explained by the third year objective structured clinical examination theory and chi rhoGPA for year 2. CONCLUSION: For this educational institution, the admissions interview is not a predictor of success on the outcome measures of the CCEB. Student chi rhoGPA in year 2 is critical to success on CCEB licensure examinations. This information infers that students should only continue in their education after they have demonstrated a mastery level at the end of year 2.

Climate studies: can students' perceptions of the ideal educational environment be of use for institutional planning and resource utilization?

As educational climate strongly affects student achievement, satisfaction and success, it is important to get regular feedback from students on how they experience the educational environment. The Dundee Ready Education Environment Measure (DREEM) Inventory was administered on the same day to all first-, second- and third-year students at the Canadian Memorial Chiropractic College (CMCC) and the students were requested to complete the questionnaire as they were actually experiencing the educational environment at CMCC, and then to say what they would have wanted, or preferred it to be like. Valid returns were received from 146 (95%) first-, 123 (82%) second- and 73 (48%) third-year students (n = 342). The results indicated that the DREEM Inventory used in the Ideal mode, together with the responses in the Actual mode, could be used effectively to determine the dissonance between what they had and what they would have liked to have. It was found that there was a strong similarity in the areas of the educational environment that the different groups of students indicated as falling short of their ideal. The results of this study provided a useful basis for strategic planning and resource utilization.

Identifying the perceived weaknesses of a new curriculum by means of the Dundee Ready Education Environment Measure (DREEM) Inventory.

The Canadian Memorial Chiropractic College is in the fourth year of implementing a new four-year Integrative Curriculum. In order to determine how the students experienced the educational environment generated by the new curriculum, the Dundee Ready Education Environment (DREEM) Inventory was administered on the same day to 148 (96%) first-year, 131 (87%) second-year, and 128 (84%) third-year students (n = 407). Resultant scores indicated many areas of concern. However, bimodal and large numbers of 'uncertain' responses also occurred. In order to establish and compare areas of most concern between the three year groups, responses were calculated as percentages indicating for each item those who agree, disagree or are uncertain. This result clearly indicated areas of joint concern for each of the five domains addressed by the DREEM Inventory. Analysis of these concerns formed the basis of strategic planning in order to institute remedial action, and on which to focus institutional resources.