In-course and career outcomes predicted by medical school selection procedures based on personal qualities.

This paper reports a 40-year follow-up of 57 graduates from the initial intake to an Australian medical school who were selected on the basis of either academic criteria alone or desirable personal qualities as assessed by non-cognitive tests and an interview (with a third small group satisfying both criteria). Both students and teaching staff have remained blind to the basis for selection until the present day. Analysis of their under- and post-graduate careers indicates that 'academic' entrants were more likely to complete an intercalated BMedSci degree and to become specialists, while 'personal qualities' entrants were more likely to graduate with honours, become GPs, and win higher degrees after graduation. However, gender more significantly predicted these outcomes, with female results similar to 'personal qualities' entrants and males' similar to 'academic.' The results are interpreted with reference to the aims and structure of the Newcastle medicine curriculum.

Why is it so hard to consider personal qualities when selecting medical students?

Having 'good' doctors is important to everybody. How to select medical students better has been discussed repeatedly for more than seventy years, implying that prevailing methods could be improved. A significant body of research exists about selection methods and their application in medicine. Yet most medical schools world-wide continue to use prior academic performance and cognitive ability as their major criteria for selection, with minor or no consideration of personal qualities and interpersonal skills (possibly assuming they will develop naturally during training and practice). We describe the main methods available for assessing personal qualities of applicants to medical school and have attempted to identify some reasons and systemic disincentives working against their adoption.

Reliability of personality and values tests: The effects of "high stakes" selection conditions, and of four years in medical school.

This study investigated if scores on tests of personal qualities are affected by whether they will determine selection decisions ("high stakes") or not; and whether they are stable for individuals and groups across a four-year medical course. Two tests, one assessing values and one assessing components of personality, were administered either at the same time as a medical university entrance exam (first cohort; N = 216), or after entry was confirmed (second cohort; N = 142). Both cohorts took the tests again after four years of medical school. Analysis of variance was used to compare group mean scores and interactions, and correlation coefficients to measure temporal reliability. The high stakes cohort initially presented themselves in a significantly more positive light on the personality test. After four years of medical school scores on both tests changed significantly, towards more communitarian values and less empathic attitudes. Thus, personality scores were affected by both the conditions under which the initial tests were conducted and by the passage of time, but values only by the passage of time. Before and after scores were significantly correlated.

Selecting medical students: An unresolved challenge.

Despite the abundant supply of academically outstanding applicants to medical schools in most countries the regularly recurring debate in the academic literature, and indeed sometimes in the popular media, implies that admissions committees are still getting it wrong in a significant number of instances. How can this be so when our procedures are directed unashamedly at selecting the most highly academically and intellectually qualified students in the expectation that they will make the best doctors? Perhaps it is time for a radical change in emphasis. Instead of endeavouring to differentiate among the top ranks of a pool of outstandingly qualified applicants, the selection effort might be better focused on identifying those potentially unsuitable in terms of their non-academic personal qualities to ensure they do not gain entry. The account that follows is an analysis of the problems of medical student selection and offers a potential solution - a solution that was first suggested in the medical literature 70 years ago, but not adopted. It is the present author's contention that the cycle of debate will continue to recur unless such an approach is pursued.

Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study.

BACKGROUND: Over the past 70 years, there has been a recurring debate in the literature and in the popular press about how best to select medical students. This implies that we are still not getting it right: either some students are unsuited to medicine or the graduating doctors are considered unsatisfactory, or both. AIM: To determine whether particular variables at the point of selection might distinguish those more likely to become satisfactory professional doctors, by following a complete intake cohort of students throughout medical school and analysing all the data used for the students' selection, their performance on a range of other potential selection tests, academic and clinical assessments throughout their studies, and records of professional behaviour covering the entire five years of the course. METHODS: A longitudinal database captured the following anonymised information for every student (n = 146) admitted in 2007 to the Hull York Medical School (HYMS) in the UK: demographic data (age, sex, citizenship); performance in each component of the selection procedure; performance in some other possible selection instruments (cognitive and non-cognitive psychometric tests); professional behaviour in tutorials and in other clinical settings; academic performance, clinical and communication skills at summative assessments throughout; professional behaviour lapses monitored routinely as part of the fitness-to-practise procedures. Correlations were sought between predictor variables and criterion variables chosen to demonstrate the full range of course outcomes from failure to complete the course to graduation with honours, and to reveal clinical and professional strengths and weaknesses. RESULTS: Student demography was found to be an important predictor of outcomes, with females, younger students and British citizens performing better overall. The selection variable "HYMS academic score", based on prior academic performance, was a significant predictor of components of Year 4 written and Year 5 clinical examinations. Some cognitive subtest scores from the UK Clinical Aptitude Test (UKCAT) and the UKCAT total score were also significant predictors of the same components, and a unique predictor of the Year 5 written examination. A number of the non-cognitive tests were significant independent predictors of Years 4 and 5 clinical performance, and of lapses in professional behaviour. First- and second-year tutor ratings were significant predictors of all outcomes, both desirable and undesirable. Performance in Years 1 and 2 written exams did not predict performance in Year 4 but did generally predict Year 5 written and clinical performance. CONCLUSIONS: Measures of a range of relevant selection attributes and personal qualities can predict intermediate and end of course achievements in academic, clinical and professional behaviour domains. In this study HYMS academic score, some UKCAT subtest scores and the total UKCAT score, and some non-cognitive tests completed at the outset of studies, together predicted outcomes most comprehensively. Tutor evaluation of students early in the course also identified the more and less successful students in the three domains of academic, clinical and professional performance. These results may be helpful in informing the future development of selection tools.

Trial use of the Personal Qualities Assessment (PQA) in the entrance examination of a Japanese medical university: similarities to the results in western countries.

BACKGROUND: The Personal Qualities Assessment (PQA), developed by the University of Newcastle, Australia to assess the aptitude of future medical professionals, has been used in Western countries. PURPOSES: The objective was to investigate whether the PQA is appropriate for Japanese medical school applicants. METHODS: Two of the PQA tests, Libertarian-Dual-Communitarian moral orientations (Mojac) and Narcissism, Aloofness, Confidence, and Empathy (NACE), were translated into Japanese, and administered at the Tokyo Women's Medical University entrance examinations from 2007 to 2009. RESULTS: The distributions of the applicants' Mojac and NACE scores were close to the normal distribution, and the mean scores did not exhibit a large difference from those in Western countries. The only significant difference was that the mean score of the NACE test was slightly lower than the Western norm. CONCLUSIONS: The translated PQA tests may be appropriate for use with Japanese applicants, though further research considering cultural differences is required.

The desirable qualities of future doctors--a study of medical student perceptions.

BACKGROUND: There is a lack of consensus regarding the qualities possessed by the ideal doctor, and very limited research regarding the views of medical students on these qualities. AIMS: To investigate the views of commencing medical students regarding the desirable qualities of doctors. METHODS: A survey containing a set of proposed desirable qualities of doctors identified from the existing literature was completed by 158 first-year medical students. RESULTS: The survey had a 75% response rate. Students rated the individual qualities of empathy, motivation to be a doctor, good verbal communication, ethically sound, integrity and honesty as the most important. A factor analysis identified six categories of qualities: methodical processing, cognitive capacity, people skills, generic work ethic, role certainty and warmth. Significant differences in factor scores were found across subgroups of students (international and domestic students, with and without prior tertiary studies) on the following factors: methodical processing, which was scored highest by domestic students with prior tertiary studies, cognitive capacity, which was scored highest by domestic students without prior tertiary studies and generic work ethic, which was scored highest by international students. CONCLUSIONS: Medical students identified a range of desirable personal qualities of a doctor which varied according to student characteristics, including their prior educational experience. Future research aiming to define such desirable qualities should include a broader range of stakeholders, including students at different training levels and institutions.

Comparing the cognitive, personality and moral characteristics of high school and graduate medical entrants to the Tehran University of Medical Sciences in Iran.

BACKGROUND: Tehran University of Medical Sciences has two streams of medical student admission: an established high school entry (HSE) route and an experimental graduate entry (GE) route. AIM: To compare the cognitive skills, personality traits and moral characteristics of HSE and GE students admitted to this university. METHODS: The personal qualities assessment tool (PQA; www.pqa.net.au ) was translated from English to Persian and then back-translated. Afterwards 35 individuals from the GE and 109 individuals from the 2007 to 2008 HSE completed the test. The results were compared by t-test and Chi-square. RESULTS: The HSE students showed significantly higher ability in the cognitive skills tests (p < 0.001). They were also more libertarian (p = 0.022), but had lower ability to confront stress and unpleasant events (p < 0.001), and had lower self-awareness and self-control (p < 0.001). CONCLUSION: On the basis of their personal qualities, the GE students had more self-control and strength when coping with stress than the HSE students, but the latter had superior cognitive abilities. Hence it may be useful to include cognitive tests in GE students' entry exam and include tests of personal qualities to exclude those with unsuitable characteristics.

Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study.

BACKGROUND: Over two-thirds of UK medical schools are augmenting their selection procedures for medical students by using the United Kingdom Clinical Aptitude Test (UKCAT), which employs tests of cognitive and non-cognitive personal qualities, but clear evidence of the tests' predictive validity is lacking. This study explores whether academic performance and professional behaviours that are important in a health professional context can be predicted by these measures, when taken before or very early in the medical course. METHODS: This prospective cohort study follows the progress of the entire student cohort who entered Hull York Medical School in September 2007, having taken the UKCAT cognitive tests in 2006 and the non-cognitive tests a year later. This paper reports on the students' first and second academic years of study. The main outcome measures were regular, repeated tutor assessment of individual students' interpersonal skills and professional behaviour, and annual examination performance in the three domains of recall and application of knowledge, evaluation of data, and communication and practical clinical skills. The relationships between non-cognitive test scores, cognitive test scores, tutor assessments and examination results were explored using the Pearson product-moment correlations for each group of data; the data for students obtaining the top and bottom 20% of the summative examination results were compared using Analysis of Variance. RESULTS: Personal qualities measured by non-cognitive tests showed a number of statistically significant relationships with ratings of behaviour made by tutors, with performance in each year's objective structured clinical examinations (OSCEs), and with themed written summative examination marks in each year. Cognitive ability scores were also significantly related to each year's examination results, but seldom to professional behaviours. The top 20% of examination achievers could be differentiated from the bottom 20% on both non-cognitive and cognitive measures. CONCLUSIONS: This study shows numerous significant relationships between both cognitive and non-cognitive test scores, academic examination scores and indicators of professional behaviours in medical students. This suggests that measurement of non-cognitive personal qualities in applicants to medical school could make a useful contribution to selection and admission decisions. Further research is required in larger representative groups, and with more refined predictor measures and behavioural assessment methods, to establish beyond doubt the incremental validity of such measures over conventional cognitive assessments.

Predictive validity of the Biomedical Admissions Test: an evaluation and case study.

There has been an increase in the use of pre-admission selection tests for medicine. Such tests need to show good psychometric properties. Here, we use a paper by Emery and Bell [2009. The predictive validity of the Biomedical Admissions Test for pre-clinical examination performance. Med Educ 43:557-564] as a case study to evaluate and comment on the reporting of psychometric data in the field of medical student selection (and the comments apply to many papers in the field). We highlight pitfalls when reliability data are not presented, how simple zero-order associations can lead to inaccurate conclusions about the predictive validity of a test, and how biases need to be explored and reported. We show with BMAT that it is the knowledge part of the test which does all the predictive work. We show that without evidence of incremental validity it is difficult to assess the value of any selection tests for medicine.

Predictive validity of the personal qualities assessment for selection of medical students in Scotland.

BACKGROUND: The Personal Qualities Assessment (PQA) was developed to enhance medical student selection by measuring a range of non-cognitive attributes in the applicants to medical school. Applicants to the five Scottish medical schools were invited to pilot the test in 2001 and 2002. AIMS: To evaluate the predictive validity of PQA for selecting medical students. METHODS: A longitudinal cohort study was conducted in which PQA scores were compared with senior year medical school performance. RESULTS: Consent to access performance markers was obtained from 626 students (61.6% of 1017 entrants in 2002-2003). Linkable Foundation Year (4th) rankings were available for 411 (66%) students and objective structured clinical examination (OSCE) rankings for 335 (54%) of those consenting. Both samples were representative of the original cohort. No significant correlations were detected between separate elements of the PQA assessment and student performance. However, using the algorithm advocated by Powis et al. those defined as 'non-extreme' (<±1.5 SD from the cohort mean scores; SD, standard deviation) character types on the involved-detached and on the libertarian-communitarian moral orientation scales were ranked higher in OSCEs (average of 7.5% or 25 out of 335, p = 0.049). CONCLUSIONS: This study was limited by high attrition and basic outcome markers which are insensitive to relevant non-cognitive characteristics. However, it is the largest currently available study of predictive validity for the PQA assessment. There was one finding of significance: that those students who were identified by PQA as 'not extreme' on the two personal characteristics scales performed better in an OSCE measure of professionalism. Futures studies are required since psychometric testing for both cognitive and non-cognitive attributes are increasingly used in admission process and these should include more and better measures of professionalism against which to correlate non-cognitive traits.

Assessment for selection for the health care professions and specialty training: consensus statement and recommendations from the Ottawa 2010 Conference.

Assessment for selection in medicine and the health professions should follow the same quality assurance processes as in-course assessment. The literature on selection is limited and is not strongly theoretical or conceptual. For written testing, there is evidence of the predictive validity of Medical College Admission Test (MCAT) for medical school and licensing examination performance. There is also evidence for the predictive validity of grade point average, particularly in combination with MCAT for graduate entry but little evidence about the predictive validity of school leaver scores. Interviews have not been shown to be robust selection measures. Studies of multiple mini-interviews have indicated good predictive validity and reliability. Of other measures used in selection, only the growing interest in personality testing appears to warrant future work. Widening access to medical and health professional programmes is an increasing priority and relates to the social accountability mandate of medical and health professional schools. While traditional selection measures do discriminate against various population groups, there is little evidence on the effect of non-traditional measures in widening access. Preparation and outreach programmes show most promise. In summary, the areas of consensus for assessment for selection are small in number. Recommendations for future action focus on the adoption of principles of good assessment and curriculum alignment, use of multi-method programmatic approaches, development of interdisciplinary frameworks and utilisation of sophisticated measurement models. The social accountability mandate of medical and health professional schools demands that social inclusion, workforce issues and widening of access are embedded in the principles of good assessment for selection.

A comprehensive model for the selection of medical students.

BACKGROUND: Medical schools have a need to select their students from an excess of applicants. Selection procedures have evolved piecemeal: Academic thresholds have risen, written tests have been incorporated and interview protocols are developed. AIM: To develop and offer for critical review and, ultimately, present for adoption by medical schools, an evidence-based and defensible model for medical student selection. METHODS: We have described here a comprehensive model for selecting medical students which is grounded on the theoretical and empirical selection and assessment literature, and has been shaped by our own research and experience. RESULTS: The model includes the following selection criteria: Informed self-selection, academic achievement, general cognitive ability (GCA) and aspects of personality and interpersonal skills. A psychometrically robust procedure by which cognitive and non-cognitive test scores can be used to make selection decisions is described. Using de-identified data (n = 1000) from actual selection procedures, we demonstrate how the model and the procedure can be used in practice. CONCLUSION: The model presented is based on a currently best-practice approach and uses measures and methods that maximise the probability of making accurate, fair and defensible selection decisions.

Graduate entry to medicine: widening psychological diversity.

BACKGROUND: At Nottingham University more than 95% of entrants to the traditional 5-year medical course are school leavers. Since 2003 we have admitted graduate entrants (GEM) to a shortened (4-year) course to 'widen access to students from more disadvantaged backgrounds'. We have recently shown that the GEM course widens academic and socio-demographic diversity of the medical student population. This study explored whether GEM students also bring psychological diversity and whether this could be beneficial. METHODS: We studied: a) 217 and 96 applicants to the Nottingham 5- and 4-year courses respectively, applying in the 2002-3 UCAS cycle, and, b) 246 school leavers starting the 5-year course and 39 graduate entrants to the 4-year course in October 2003. The psychological profiles of the two groups of applicants and two groups of entrants were compared using their performance in the Goldberg 'Big 5' Personality test, the Personal Qualities Assessment (PQA; measuring interpersonal traits and interpersonal values), and the Lovibond and Lovibond measure of depression, anxiety and stress. For the comparison of the Entrants we excluded the 33 school leavers and seven graduates who took the tests as Applicants.Statistical analyses were undertaken using SPSS software (version 16.0). RESULTS: Graduate applicants compared to school leaver applicants were significantly more conscientious, more confident, more self controlled, more communitarian in moral orientation and less anxious. Only one of these differences was preserved in the entrants with graduates being less anxious. However, the graduate entrants were significantly less empathetic and conscientious than the school leavers. CONCLUSION: This study has shown that school leaver and graduate entrants to medical school differ in some psychological characteristics. However, if confirmed in other studies and if they were manifest in the extreme, not all the traits brought by graduates would be desirable for someone aiming for a medical career.

Graduate entry to medicine: widening academic and socio-demographic access.

OBJECTIVE: This study aimed to examine whether a graduate entry course widens access to medicine. METHODS: We carried out a retrospective study at the University of Nottingham Medical School of socio-demographic and academic data collected by the Universities and Colleges Admission Service (UCAS). Study participants comprised all applicants to the 5-year (mainly school-leavers) and 4-year (graduates only) medical courses for admission in October 2003. RESULTS: In 2002-03, there were 2392 applicants for the 5-year course and 1235 applicants for the 4-year course. Significantly, applicants for the 4-year course comprised more males, were more socio-economically deprived and had lower UCAS tariff point scores compared with applicants for the 5-year course. These differences were preserved in those students who eventually started on either of the 2 courses in October 2003 (254 and 94 students for the 5- and 4-year courses, respectively). Comparing entrants with non-entrants (mainly rejected students), those joining the 5-year course were younger, were less socio-economically deprived, were more likely to be White and had higher UCAS tariff point scores than non-entrants, but there were no significant gender differences. Those joining the 4-year course were more likely to be White and had higher UCAS tariff point scores than non-entrants. By contrast with entrants to the 5-year course, entrants to the 4-year course were significantly older than non-entrants. CONCLUSIONS: Graduate entrants to medicine widen academic and socio-demographic diversity in the medical school student population.

Have we got the selection process right? The validity of selection tools for predicting academic performance in the first year of undergraduate medicine.

This article was migrated. The article was marked as recommended. Content: There remains much debate over the 'best' method for selecting students in to medicine. This study aimed to assess the predictive validity of four different selection tools with academic performance outcomes in first-year undergraduate medical students. Methods: Regression analyses were conducted between admission scores on previous academic performance - the Australian Tertiary Admission Rank (ATAR), the Undergraduate Medicine and Health Sciences Admission Test (UMAT), Multiple-Mini Interview (MMI) and the Personal Qualities Assessment (PQA) with student performance in first-year assessments of Multiple Choice Questions, Short Answer Questions, Objective Structured Clinical Examinations (OSCE) and Problem-Based Learning (PBL) Tutor ratings in four cohorts of students (N = 604, 90%). Results: All four selection tools were found to have significant predictive associations with one or more measures of student performance in Year One of undergraduate medicine. UMAT, ATAR and MMI scores consistently predicted first year performance on a number of outcomes. ATAR was the only selection tool to predict the likelihood of making satisfactory progress overall. Conclusions: All four selection tools play a contributing role in predicting academic performance in first year medical students. Further research is required to assess the validity of selection tools in predicting performance in the later years of medicine.

Assessment of personal qualities in selection of medical radiation science students.

UNLABELLED: It is increasingly acknowledged that, in addition to prior academic achievement, there is a need to seek evidence for the abilities and personal qualities of applicants to health professional programs at a university. The aim of this study was to determine the specific abilities and personal qualities required for excellence in practice in the relevant professional domains of medical radiation science (MRS). METHODS: A focus group, consisting of MRS academic staff, developed a questionnaire. The questionnaire was sent to senior MRS practitioners throughout Australia and 213 were returned for analysis. Respondents were asked to rate 40 specific abilities and qualities (referred to as "elements") on a 5-point scale. RESULTS: Two hundred thirteen completed questionnaires were returned, a 53% response rate. One hundred twelve respondents (52%) indicated they currently worked in diagnostic radiography (DR), 57 (27%) worked in radiation therapy (RT), and 44 (21%) worked in nuclear medicine (NM). The duration (mean +/- SD) of the respondents' professional practice in MRS was 14.5 +/- 10 y, with durations ranging from 1 to 43 y. Raw scores and mean scores were examined for any influence of the variable "Number of Years in Practice." DISCUSSION: No major differences were found between the ratings provided by the practitioners from the 3 different MRS professional domains of NM, RT, and DR. Factor analysis indicated the existence of 3 orthogonal factors in the questionnaire data: (a) treat others professionally and ethically, (b) engage with and be open to others, and (c) problem-solving ability. Qualitative analysis of the respondents' comments provided similar themes: (a) the need for professional competence (knowledge and abilities), (b) ethical behavior, (c) the need for a technology and a people orientation, and (d) MRS should be the first choice of MRS students and not a second choice to other professional degrees. CONCLUSION: Senior medical radiation scientists identified professionalism, ethical behavior, engagement with and openness to others, intrinsic specific motivation, and an orientation to people and technology as nonacademic qualities required for excellence in the practice of the professions embraced by MRS.

The influence of critical thinking skills on performance and progression in a pre-registration nursing program.

BACKGROUND: The importance of developing critical thinking skills in preregistration nursing students is recognized worldwide. Yet, there has been limited exploration of how students' critical thinking skill scores on entry to pre-registration nursing education influence their academic and clinical performance and progression. AIM: The aim of this study was to: i) describe entry and exit critical thinking scores of nursing students enrolled in a three year bachelor of nursing program in Australia in comparison to norm scores; ii) explore entry critical thinking scores in relation to demographic characteristics, students' performance and progression. METHOD: This longitudinal correlational study used the Health Sciences Reasoning Test (HSRT) to measure critical thinking skills in a sample (n=134) of students, at entry and exit (three years later). A one sample t-test was used to determine if differences existed between matched student critical thinking scores between entry and exit points. Academic performance, clinical performance and progression data were collected and correlations with entry critical thinking scores were examined. RESULTS: There was a significant relationship between critical thinking scores, academic performance and students' risk of failing, especially in the first semester of study. Critical thinking scores were predictive of program completion within three years. The increase in critical thinking scores from entry to exit was significant for the 28 students measured. In comparison to norm scores, entry level critical thinking scores were significantly lower, but exit scores were comparable. Critical thinking scores had no significant relationship to clinical performance. CONCLUSION: Entry critical thinking scores significantly correlate to academic performance and predict students risk of course failure and ability to complete a nursing degree in three years. Students' critical thinking scores are an important determinant of their success and as such can inform curriculum development and selection strategies.