Exploring UK medical school differences: the MedDifs study of selection, teaching, student and F1 perceptions, postgraduate outcomes and fitness to practise.

BACKGROUND: Medical schools differ, particularly in their teaching, but it is unclear whether such differences matter, although influential claims are often made. The Medical School Differences (MedDifs) study brings together a wide range of measures of UK medical schools, including postgraduate performance, fitness to practise issues, specialty choice, preparedness, satisfaction, teaching styles, entry criteria and institutional factors. METHOD: Aggregated data were collected for 50 measures across 29 UK medical schools. Data include institutional history (e.g. rate of production of hospital and GP specialists in the past), curricular influences (e.g. PBL schools, spend per student, staff-student ratio), selection measures (e.g. entry grades), teaching and assessment (e.g. traditional vs PBL, specialty teaching, self-regulated learning), student satisfaction, Foundation selection scores, Foundation satisfaction, postgraduate examination performance and fitness to practise (postgraduate progression, GMC sanctions). Six specialties (General Practice, Psychiatry, Anaesthetics, Obstetrics and Gynaecology, Internal Medicine, Surgery) were examined in more detail. RESULTS: Medical school differences are stable across time (median alpha = 0.835). The 50 measures were highly correlated, 395 (32.2%) of 1225 correlations being significant with p < 0.05, and 201 (16.4%) reached a Tukey-adjusted criterion of p < 0.0025. Problem-based learning (PBL) schools differ on many measures, including lower performance on postgraduate assessments. While these are in part explained by lower entry grades, a surprising finding is that schools such as PBL schools which reported greater student satisfaction with feedback also showed lower performance at postgraduate examinations. More medical school teaching of psychiatry, surgery and anaesthetics did not result in more specialist trainees. Schools that taught more general practice did have more graduates entering GP training, but those graduates performed less well in MRCGP examinations, the negative correlation resulting from numbers of GP trainees and exam outcomes being affected both by non-traditional teaching and by greater historical production of GPs. Postgraduate exam outcomes were also higher in schools with more self-regulated learning, but lower in larger medical schools. A path model for 29 measures found a complex causal nexus, most measures causing or being caused by other measures. Postgraduate exam performance was influenced by earlier attainment, at entry to Foundation and entry to medical school (the so-called academic backbone), and by self-regulated learning. Foundation measures of satisfaction, including preparedness, had no subsequent influence on outcomes. Fitness to practise issues were more frequent in schools producing more male graduates and more GPs. CONCLUSIONS: Medical schools differ in large numbers of ways that are causally interconnected. Differences between schools in postgraduate examination performance, training problems and GMC sanctions have important implications for the quality of patient care and patient safety.

Fitness to practise sanctions in UK doctors are predicted by poor performance at MRCGP and MRCP(UK) assessments: data linkage study.

BACKGROUND: The predictive validity of postgraduate examinations, such as MRCGP and MRCP(UK) in the UK, is hard to assess, particularly for clinically relevant outcomes. The sanctions imposed on doctors by the UK's General Medical Council (GMC), including erasure from the Medical Register, are indicators of serious problems with fitness to practise (FtP) that threaten patient safety or wellbeing. This data linkage study combined data on GMC sanctions with data on postgraduate examination performance. METHODS: Examination results were obtained for UK registered doctors taking the MRCGP Applied Knowledge Test (AKT; n = 27,561) or Clinical Skills Assessment (CSA; n = 17,365) at first attempt between 2010 and 2016 or taking MRCP(UK) Part 1 (MCQ; n = 37,358), Part 2 (MCQ; n = 28,285) or Practical Assessment of Clinical Examination Skills (PACES; n = 27,040) at first attempt between 2001 and 2016. Exam data were linked with GMC actions on a doctor's registration from September 2008 to January 2017, sanctions including Erasure, Suspension, Conditions on Practice, Undertakings or Warnings (ESCUW). Examination results were only considered at first attempts. Multiple logistic regression assessed the odds ratio for ESCUW in relation to examination results. Multiple imputation was used for structurally missing values. RESULTS: Doctors sanctioned by the GMC performed substantially less well on MRCGP and MRCP(UK), with a mean Cohen's d across the five exams of - 0.68. Doctors on the 2.5th percentile of exam performance were about 12 times more likely to have FtP problems than those on the 97.5th percentile. Knowledge assessments and clinical assessments were independent predictors of future sanctions, with clinical assessments predicting ESCUW significantly better. The log odds of an FtP sanction were linearly related to examination marks over the entire range of performance, additional performance increments lowering the risk of FtP sanctions at all performance levels. CONCLUSIONS: MRCGP and MRCP(UK) performance are valid predictors of professionally important outcomes that transcend simple knowledge or skills and the GMC puts under the headings of conduct and trust. Postgraduate examinations may predict FtP sanctions because the psychological processes involved in successfully studying, understanding and practising medicine at a high level share similar mechanisms to those underlying conduct and trust.

Maternal depressive symptoms and adolescent academic attainment: Testing pathways via parenting and self-control.

Maternal depression is associated with reduced academic attainment in children, however, it is not clear how this association comes about. Depressive symptoms are associated with impairment in social roles including parenting. Children's self-control is an important contributor to academic attainment and is influenced by parenting. We therefore hypothesised that impaired parenting and children's self-control may mediate links between maternal depression and children's academic attainment. Data were from a brief longitudinal study (3 waves) of UK children aged 11-12 years and their mothers. Higher maternal depressive symptoms at baseline were associated with lower academic attainment in children assessed one year later. There was evidence to support an indirect effect of maternal depressive symptoms on children's academic attainment through the mother-child and the father-child relationship which, in turn, reduced children's self-control. These influences were independent of socio-economic deprivation. A direct effect of maternal depression on children's academic attainment was also observed.

Impact of accelerated, graduate-entry medicine courses: a comparison of profile, success, and specialty destination between graduate entrants to accelerated or standard medicine courses in UK.

BACKGROUND: Little research has compared the profile, success, or specialty destinations of graduates entering UK medical schools via accelerated, 4-yr, standard 5-yr and 6-yr programmes. Four research questions directed this investigation:- What are the success rates for graduates entering graduate-entry vs. undergraduate medicine courses? How does the sociodemographic and educational profile differ between these two groups? Is success - in medical school and foundation training - dependent on prior degree, demographic factors, or aptitude test performance at selection? What specialty do graduate entry medicine students subsequently enter? METHODS: The data from two cohorts of graduates entering medical school in 2007 and 2008 (n = 2761) in the UKMED (UK Medical Education Database) database were studied: 1445 taking 4-yr and 1150 taking 5-yr medicine courses, with smaller numbers following other programmes. RESULTS: Completion rates for degree programmes were high at 95%, with no significant difference between programme types. 4-yr entrants were older, less likely to be from Asian communities, had lower HESA (Higher Education Statistics Agency) tariff scores, but higher UKCAT (UK Clinical Aptitude Test) and GAMSAT (Graduate Medical School Admissions Test) scores, than 5-yr entrants. Higher GAMSAT scores, black or minority ethnicity (BME), and younger age were independent predictors of successful completion of medical school. Foundation Programme (FPAS) selection measures (EPM - educational performance measure; SJT - situational judgment test) were positively associated with female sex, but negatively with black or minority ethnicity. Higher aptitude test scores were associated with EPM and SJT, GAMSAT with EPM, UKCAT with SJT. Prior degree subject, class of degree, HESA tariff, and type of medicine programme were not related to success. CONCLUSIONS: The type of medicine programme has little effect on graduate entrant completion, or EPM or SJT scores, despite differences in student profile. Aptitude test score has some predictive validity, as do sex, age and BME, but not prior degree subject or class. Further research is needed to disentangle the influences of BME.

Would changing the selection process for GP trainees stem the workforce crisis? A cohort study using multiple-imputation and simulation.

BACKGROUND: There is currently a shortage of qualified GPs in the UK and not all of the training posts available each year are filled. Changing the way in which GP trainees are selected could help increase the training post fill rate and the number of new entrants to the GP Register. The aim of this study was to model the impact of changing the selection process for GP training on the number of trainees obtaining GP Registration, either with or without extensions. METHOD: This was a cohort study using UK applications for GP training in 2011-14. Application data were linked using GMC numbers to training outcome data where available, and imputed using multiple imputation where missing. The number of trainees appointed and GP Registrations within three and five years' full-time-equivalent were estimated for four different selection processes. RESULTS: The cut scores used in the actual 2015 selection process makes it impossible to fill all training posts. Random selection is the worst option, but the difference between this and other processes modelled falls as more trainees are selected. There are large marginal effects on outcomes: those with the highest selection scores are more likely to obtain GP Registration than those with the lowest scores. CONCLUSIONS: Changing the selection process alone would have a small impact on the number of GP Registrations; reducing/removing cut scores would have a much larger impact. This would also increase the number of trainees requiring extensions and being released from training which would have adverse consequences for the profession.

Using differential item functioning to evaluate potential bias in a high stakes postgraduate knowledge based assessment.

BACKGROUND: Fairness is a critical component of defensible assessment. Candidates should perform according to ability without influence from background characteristics such as ethnicity or sex. However, performance differs by candidate background in many assessment environments. Many potential causes of such differences exist, and examinations must be routinely analysed to ensure they do not present inappropriate progression barriers for any candidate group. By analysing the individual questions of an examination through techniques such as Differential Item Functioning (DIF), we can test whether a subset of unfair questions explains group-level differences. Such items can then be revised or removed. METHODS: We used DIF to investigate fairness for 13,694 candidates sitting a major international summative postgraduate examination in internal medicine. We compared (a) ethnically white UK graduates against ethnically non-white UK graduates and (b) male UK graduates against female UK graduates. DIF was used to test 2773 questions across 14 sittings. RESULTS: Across 2773 questions eight (0.29%) showed notable DIF after correcting for multiple comparisons: seven medium effects and one large effect. Blinded analysis of these questions by a panel of clinician assessors identified no plausible explanations for the differences. These questions were removed from the question bank and we present them here to share knowledge of questions with DIF. These questions did not significantly impact the overall performance of the cohort. Group-level differences in performance between the groups we studied in this examination cannot be explained by a subset of unfair questions. CONCLUSIONS: DIF helps explore fairness in assessment at the question level. This is especially important in high-stakes assessment where a small number of unfair questions may adversely impact the passing rates of some groups. However, very few questions exhibited notable DIF so differences in passing rates for the groups we studied cannot be explained by unfairness at the question level.

Faking handedness: Individual differences in ability to fake handedness, social cognitions of the handedness of others, and a forensic application using Bayes' theorem.

People usually describe their handedness honestly, but that need not necessarily be the case. A legal case is described of a murder said by the pathologist to be committed by a left-hander but the defendant claimed to be right-handed, and the first author assessed the defendant's handedness as an expert witness. We know of no previous work on faking handedness, and so we tested 30 right-handers and 25 left-handers on various handedness tasks, and then asked the participants to repeat the tasks while faking being of opposite handedness. Social cognitions of handedness were assessed from participants' knowledge of how other right- and left-handers would answer handedness questionnaires. Fake handedness was best differentiated using cursive lower-case sentence writing, upper-case written letters being less good at distinguishing, as also were simple motor tasks. Participants differed in social cognitions of handedness, and those with more accurate social cognitions were better able to fake. Personality measures did not predict faking ability. For forensic purposes a Bayesian analysis was carried out to evaluate the likelihood of right and left hand performance being true rather than faked, and the cursive lower-case writing provided strong posterior odds that, as claimed, the particular defendant was a true right-hander.

Higher cognitive ability buffers stress-related depressive symptoms in adolescent girls.

Stress has been shown to have a causal effect on risk for depression. We investigated the role of cognitive ability as a moderator of the effect of stressful life events on depressive symptoms and whether this varied by gender. Data were analyzed in two adolescent data sets: one representative community sample aged 11-12 years (n = 460) and one at increased familial risk of depression aged 9-17 years (n = 335). In both data sets, a three-way interaction was found whereby for girls, but not boys, higher cognitive ability buffered the association between stress and greater depressive symptoms. The interaction was replicated when the outcome was a diagnosis of major depressive disorder. This buffering effect in girls was not attributable to coping efficacy. However, a small proportion of the variance was accounted for by sensitivity to environmental stressors. Results suggest that this moderating effect of cognitive ability in girls is largely attributable to greater available resources for cognitive operations that offer protection against stress-induced reductions in cognitive processing and cognitive control which in turn reduces the likelihood of depressive symptomatology.

A longitudinal study of self-control at the transition to secondary school: Considering the role of pubertal status and parenting.

Higher self-control in children and adolescents is associated with a range of positive outcomes in adulthood. However, little is known about the naturalistic development of self-control during early adolescence and the factors that affect this. We examined the role of puberty and parenting style as theoretically important influences on stability and change in self-control. A longitudinal (3 waves), multiple-informant dataset of children entering early adolescence (M = 11 years) was used to explore longitudinal change in self-control using latent growth curve modelling. Children's self-control declined during the one-year study period and declines were associated with children's behavioural and social functioning. Associations with self-control were found for pubertal status and parental warmth and hostility, but not for parental discipline. The findings suggest that during early adolescence, when children make the transition to secondary school, self-control declines. This is particularly the case for those experiencing puberty earlier than their peers. Parent warmth influences the trajectory of self-control during this period.

The Tapley and Bryden test of performance differences between the hands: The original data, newer data, and the relation to pegboard and other tasks.

Tapley and Bryden (T&B)'s 1985 circle-marking task is a group-administered task assessing performance differences between the hands. The bimodal distribution clearly separates self-described right- and left-handers. Using Phil's original datafiles we analyse the test in more detail, providing raw scores for each hands which are useful forensically, and we provide reliability estimates. Van Horn's unpublished 1992 PhD thesis studied T&B tasks and Annett pegboards varying in difficulty. A striking finding, that Phil Bryden called "the Van Horn problem," was that hand differences (R - L) were unrelated to task difficulty. That result was the starting point for Pamela Bryden's 1998 thesis, firstly replicating Van Horn, but then showing that task difficulty did relate to hand differences for Grooved pegboards. Pamela Bryden's model for those effects is presented here. Comparing across tasks, the T&B and pegboard tasks showed almost complete consistency for direction of handedness. Likewise, within each task, degree of handedness intercorrelated strongly across variants. In strong contrast, degree of handedness for T&B tasks showed minimal correlation with degree of handedness for pegboards. At the highest level, therefore, direction of handedness is consistent within individuals (conventional right and left handedness), but there are separable processes determining dominant-non-dominant hands differences for each particular task.

Assessment at UK medical schools varies substantially in volume, type and intensity and correlates with postgraduate attainment.

BACKGROUND: In the United Kingdom (UK), medical schools are free to develop local systems and policies that govern student assessment and progression. Successful completion of an undergraduate medical degree results in the automatic award of a provisional licence to practice medicine by the General Medical Council (GMC). Such a licensing process relies heavily on the assumption that individual schools develop similarly rigorous assessment policies. Little work has evaluated variability of undergraduate medical assessment between medical schools. That absence is important in the light of the GMC's recent announcement of the introduction of the UKMLA (UK Medical Licensing Assessment) for all doctors who wish to practise in the UK. The present study aimed to quantify and compare the volume, type and intensity of summative assessment across medicine (A100) courses in the United Kingdom, and to assess whether intensity of assessment correlates with the postgraduate attainment of doctors from these schools. METHODS: Locally knowledgeable students in each school were approached to take part in guided-questionnaire interviews via telephone or Skype(TM). Their understanding of assessment at their medical school was probed, and later validated with the assessment department of the respective medical school. We gathered data for 25 of 27 A100 programmes in the UK and compared volume, type and intensity of assessment between schools. We then correlated these data with the mean first-attempt score of graduates sitting MRCGP and MRCP(UK), as well as with UKFPO selection measures. RESULTS: The median written assessment volume across all schools was 2000 min (mean = 2027, SD = 586, LQ = 1500, UQ = 2500, range = 1000-3200) and 1400 marks (mean = 1555, SD = 463, LQ = 1200, UQ = 1800, range = 1100-2800). The median practical assessment volume was 400 min (mean = 472, SD = 207, LQ = 400, UQ = 600, range = 200-1000). The median intensity (minutes per mark ratio) of summative written assessment was 1.24 min per mark (mean = 1.28, SD = 0.30, LQ = 1.11, UQ = 1.37, range = 0.85-2.08). An exploratory analysis suggested a significant correlation of total assessment time with mean first-attempt score on both the knowledge and the clinical assessments of MRCGP and of MRCP(UK). CONCLUSIONS: There are substantial differences in the volume, format and intensity of undergraduate assessment between UK medical schools. These findings suggest a potential for differences in the reliability of detecting poorly performing students, or differences in identifying and stratifying academically equivalent students for ranking in the Foundation Programme Application System (FPAS). Furthermore, these differences appear to directly correlate with performance in postgraduate examinations. Taken together, our findings highlight highly variable local assessment procedures that warrant further investigation to establish their potential impact on students.

Cross-comparison of MRCGP & MRCP(UK) in a database linkage study of 2,284 candidates taking both examinations: assessment of validity and differential performance by ethnicity.

BACKGROUND: MRCGP and MRCP(UK) are the main entry qualifications for UK doctors entering general [family] practice or hospital [internal] medicine. The performance of MRCP(UK) candidates who subsequently take MRCGP allows validation of each assessment. In the UK, underperformance of ethnic minority doctors taking MRCGP has had a high political profile, with a Judicial Review in the High Court in April 2014 for alleged racial discrimination. Although the legal challenge was dismissed, substantial performance differences between white and BME (Black and Minority Ethnic) doctors undoubtedly exist. Understanding ethnic differences can be helped by comparing the performance of doctors who take both MRCGP and MRCP(UK). METHODS: We identified 2,284 candidates who had taken one or more parts of both assessments, MRCP(UK) typically being taken 3.7 years before MRCGP. We analyzed performance on knowledge-based MCQs (MRCP(UK) Parts 1 and 2 and MRCGP Applied Knowledge Test (AKT)) and clinical examinations (MRCGP Clinical Skills Assessment (CSA) and MRCP(UK) Practical Assessment of Clinical Skills (PACES)). RESULTS: Correlations between MRCGP and MRCP(UK) were high, disattenuated correlations for MRCGP AKT with MRCP(UK) Parts 1 and 2 being 0.748 and 0.698, and for CSA and PACES being 0.636. BME candidates performed less well on all five assessments (P < .001). Correlations disaggregated by ethnicity were complex, MRCGP AKT showing similar correlations with Part1/Part2/PACES in White and BME candidates, but CSA showing stronger correlations with Part1/Part2/PACES in BME candidates than in White candidates. CSA changed its scoring method during the study; multiple regression showed the newer CSA was better predicted by PACES than the previous CSA. CONCLUSIONS: High correlations between MRCGP and MRCP(UK) support the validity of each, suggesting they assess knowledge cognate to both assessments. Detailed analyses by candidate ethnicity show that although White candidates out-perform BME candidates, the differences are largely mirrored across the two examinations. Whilst the reason for the differential performance is unclear, the similarity of the effects in independent knowledge and clinical examinations suggests the differences are unlikely to result from specific features of either assessment and most likely represent true differences in ability.

Genome-wide association study of handedness excludes simple genetic models.

Handedness is a human behavioural phenotype that appears to be congenital, and is often assumed to be inherited, but for which the developmental origin and underlying causation(s) have been elusive. Models of the genetic basis of variation in handedness have been proposed that fit different features of the observed resemblance between relatives, but none has been decisively tested or a corresponding causative locus identified. In this study, we applied data from well-characterised individuals studied at the London Twin Research Unit. Analysis of genome-wide SNP data from 3940 twins failed to identify any locus associated with handedness at a genome-wide level of significance. The most straightforward interpretation of our analyses is that they exclude the simplest formulations of the 'right-shift' model of Annett and the 'dextral/chance' model of McManus, although more complex modifications of those models are still compatible with our observations. For polygenic effects, our study is inadequately powered to reliably detect alleles with effect sizes corresponding to an odds ratio of 1.2, but should have good power to detect effects at an odds ratio of 2 or more.

Implementing statistical equating for MRCP(UK) Parts 1 and 2.

BACKGROUND: The MRCP(UK) exam, in 2008 and 2010, changed the standard-setting of its Part 1 and Part 2 examinations from a hybrid Angoff/Hofstee method to statistical equating using Item Response Theory, the reference group being UK graduates. The present paper considers the implementation of the change, the question of whether the pass rate increased amongst non-UK candidates, any possible role of Differential Item Functioning (DIF), and changes in examination predictive validity after the change. METHODS: Analysis of data of MRCP(UK) Part 1 exam from 2003 to 2013 and Part 2 exam from 2005 to 2013. RESULTS: Inspection suggested that Part 1 pass rates were stable after the introduction of statistical equating, but showed greater annual variation probably due to stronger candidates taking the examination earlier. Pass rates seemed to have increased in non-UK graduates after equating was introduced, but was not associated with any changes in DIF after statistical equating. Statistical modelling of the pass rates for non-UK graduates found that pass rates, in both Part 1 and Part 2, were increasing year on year, with the changes probably beginning before the introduction of equating. The predictive validity of Part 1 for Part 2 was higher with statistical equating than with the previous hybrid Angoff/Hofstee method, confirming the utility of IRT-based statistical equating. CONCLUSIONS: Statistical equating was successfully introduced into the MRCP(UK) Part 1 and Part 2 written examinations, resulting in higher predictive validity than the previous Angoff/Hofstee standard setting. Concerns about an artefactual increase in pass rates for non-UK candidates after equating were shown not to be well-founded. Most likely the changes resulted from a genuine increase in candidate ability, albeit for reasons which remain unclear, coupled with a cognitive illusion giving the impression of a step-change immediately after equating began. Statistical equating provides a robust standard-setting method, with a better theoretical foundation than judgemental techniques such as Angoff, and is more straightforward and requires far less examiner time to provide a more valid result. The present study provides a detailed case study of introducing statistical equating, and issues which may need to be considered with its introduction.

The Academic Backbone: longitudinal continuities in educational achievement from secondary school and medical school to MRCP(UK) and the specialist register in UK medical students and doctors.

BACKGROUND: Selection of medical students in the UK is still largely based on prior academic achievement, although doubts have been expressed as to whether performance in earlier life is predictive of outcomes later in medical school or post-graduate education. This study analyses data from five longitudinal studies of UK medical students and doctors from the early 1970s until the early 2000s. Two of the studies used the AH5, a group test of general intelligence (that is, intellectual aptitude). Sex and ethnic differences were also analyzed in light of the changing demographics of medical students over the past decades. METHODS: Data from five cohort studies were available: the Westminster Study (began clinical studies from 1975 to 1982), the 1980, 1985, and 1990 cohort studies (entered medical school in 1981, 1986, and 1991), and the University College London Medical School (UCLMS) Cohort Study (entered clinical studies in 2005 and 2006). Different studies had different outcome measures, but most had performance on basic medical sciences and clinical examinations at medical school, performance in Membership of the Royal Colleges of Physicians (MRCP(UK)) examinations, and being on the General Medical Council Specialist Register. RESULTS: Correlation matrices and path analyses are presented. There were robust correlations across different years at medical school, and medical school performance also predicted MRCP(UK) performance and being on the GMC Specialist Register. A-levels correlated somewhat less with undergraduate and post-graduate performance, but there was restriction of range in entrants. General Certificate of Secondary Education (GCSE)/O-level results also predicted undergraduate and post-graduate outcomes, but less so than did A-level results, but there may be incremental validity for clinical and post-graduate performance. The AH5 had some significant correlations with outcome, but they were inconsistent. Sex and ethnicity also had predictive effects on measures of educational attainment, undergraduate, and post-graduate performance. Women performed better in assessments but were less likely to be on the Specialist Register. Non-white participants generally underperformed in undergraduate and post-graduate assessments, but were equally likely to be on the Specialist Register. There was a suggestion of smaller ethnicity effects in earlier studies. CONCLUSIONS: The existence of the Academic Backbone concept is strongly supported, with attainment at secondary school predicting performance in undergraduate and post-graduate medical assessments, and the effects spanning many years. The Academic Backbone is conceptualized in terms of the development of more sophisticated underlying structures of knowledge ('cognitive capital' and 'medical capital'). The Academic Backbone provides strong support for using measures of educational attainment, particularly A-levels, in student selection.

The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools.

BACKGROUND: Most UK medical schools use aptitude tests during student selection, but large-scale studies of predictive validity are rare. This study assesses the United Kingdom Clinical Aptitude Test (UKCAT), and its four sub-scales, along with measures of educational attainment, individual and contextual socio-economic background factors, as predictors of performance in the first year of medical school training. METHODS: A prospective study of 4,811 students in 12 UK medical schools taking the UKCAT from 2006 to 2008 as a part of the medical school application, for whom first year medical school examination results were available in 2008 to 2010. RESULTS: UKCAT scores and educational attainment measures (General Certificate of Education (GCE): A-levels, and so on; or Scottish Qualifications Authority (SQA): Scottish Highers, and so on) were significant predictors of outcome. UKCAT predicted outcome better in female students than male students, and better in mature than non-mature students. Incremental validity of UKCAT taking educational attainment into account was significant, but small. Medical school performance was also affected by sex (male students performing less well), ethnicity (non-White students performing less well), and a contextual measure of secondary schooling, students from secondary schools with greater average attainment at A-level (irrespective of public or private sector) performing less well. Multilevel modeling showed no differences between medical schools in predictive ability of the various measures. UKCAT sub-scales predicted similarly, except that Verbal Reasoning correlated positively with performance on Theory examinations, but negatively with Skills assessments. CONCLUSIONS: This collaborative study in 12 medical schools shows the power of large-scale studies of medical education for answering previously unanswerable but important questions about medical student selection, education and training. UKCAT has predictive validity as a predictor of medical school outcome, particularly in mature applicants to medical school. UKCAT offers small but significant incremental validity which is operationally valuable where medical schools are making selection decisions based on incomplete measures of educational attainment. The study confirms the validity of using all the existing measures of educational attainment in full at the time of selection decision-making. Contextual measures provide little additional predictive value, except that students from high attaining secondary schools perform less well, an effect previously shown for UK universities in general.

Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: meta-regression of six UK longitudinal studies.

BACKGROUND: Measures used for medical student selection should predict future performance during training. A problem for any selection study is that predictor-outcome correlations are known only in those who have been selected, whereas selectors need to know how measures would predict in the entire pool of applicants. That problem of interpretation can be solved by calculating construct-level predictive validity, an estimate of true predictor-outcome correlation across the range of applicant abilities. METHODS: Construct-level predictive validities were calculated in six cohort studies of medical student selection and training (student entry, 1972 to 2009) for a range of predictors, including A-levels, General Certificates of Secondary Education (GCSEs)/O-levels, and aptitude tests (AH5 and UK Clinical Aptitude Test (UKCAT)). Outcomes included undergraduate basic medical science and finals assessments, as well as postgraduate measures of Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) performance and entry in the Specialist Register. Construct-level predictive validity was calculated with the method of Hunter, Schmidt and Le (2006), adapted to correct for right-censorship of examination results due to grade inflation. RESULTS: Meta-regression analyzed 57 separate predictor-outcome correlations (POCs) and construct-level predictive validities (CLPVs). Mean CLPVs are substantially higher (.450) than mean POCs (.171). Mean CLPVs for first-year examinations, were high for A-levels (.809; CI: .501 to .935), and lower for GCSEs/O-levels (.332; CI: .024 to .583) and UKCAT (mean = .245; CI: .207 to .276). A-levels had higher CLPVs for all undergraduate and postgraduate assessments than did GCSEs/O-levels and intellectual aptitude tests. CLPVs of educational attainment measures decline somewhat during training, but continue to predict postgraduate performance. Intellectual aptitude tests have lower CLPVs than A-levels or GCSEs/O-levels. CONCLUSIONS: Educational attainment has strong CLPVs for undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year performance. Such CLPVs justify the use of educational attainment measure in selection, but also raise a key theoretical question concerning the remaining 35% of variance (and measurement error, range restriction and right-censorship have been taken into account). Just as in astrophysics, 'dark matter' and 'dark energy' are posited to balance various theoretical equations, so medical student selection must also have its 'dark variance', whose nature is not yet properly characterized, but explains a third of the variation in performance during training. Some variance probably relates to factors which are unpredictable at selection, such as illness or other life events, but some is probably also associated with factors such as personality, motivation or study skills.

Neurobehavioral problems following low-level exposure to organophosphate pesticides: a systematic and meta-analytic review.

Meta-analysis was carried out to determine the neurotoxic effects of long-term exposure to low levels of organophosphates (OPs) in occupational settings. Concern about the effects of OPs on human health has been growing as they are increasingly used throughout the world for a variety of agricultural, industrial and domestic purposes. The neurotoxic effects of acute poisoning are well established but the possibility that low-level exposure causes ill health is controversial. It is important to get a clear answer to this question as more individuals are at risk of low-level exposure than acute poisoning. Although a number of reviews on this topic have been published in the past, authors have come to conflicting conclusions. To date, none of these reviews have attempted quantitative evaluation of study findings using meta-analysis. This paper reviews the available evidence concerning the neurotoxicity of low-level occupational exposure to OPs and goes on to report the results of a meta-analysis of 14 studies which fulfilled criteria for this type of statistical analysis (means and standard deviations of dependant variables reported). Data were assimilated from more than 1600 participants. The majority of well designed studies found a significant association between low-level exposure to OPs and impaired neurobehavioral function which is consistent, small to moderate in magnitude and concerned primarily with cognitive functions such as psychomotor speed, executive function, visuospatial ability, working and visual memory. Unresolved issues in the literature which should become the focus of further studies are highlighted and discussed.

Investigating possible ethnicity and sex bias in clinical examiners: an analysis of data from the MRCP(UK) PACES and nPACES examinations.

BACKGROUND: Bias of clinical examiners against some types of candidate, based on characteristics such as sex or ethnicity, would represent a threat to the validity of an examination, since sex or ethnicity are 'construct-irrelevant' characteristics. In this paper we report a novel method for assessing sex and ethnic bias in over 2000 examiners who had taken part in the PACES and nPACES (new PACES) examinations of the MRCP(UK). METHOD: PACES and nPACES are clinical skills examinations that have two examiners at each station who mark candidates independently. Differences between examiners cannot be due to differences in performance of a candidate because that is the same for the two examiners, and hence may result from bias or unreliability on the part of the examiners. By comparing each examiner against a 'basket' of all of their co-examiners, it is possible to identify examiners whose behaviour is anomalous. The method assessed hawkishness-doveishness, sex bias, ethnic bias and, as a control condition to assess the statistical method, 'even-number bias' (i.e. treating candidates with odd and even exam numbers differently). Significance levels were Bonferroni corrected because of the large number of examiners being considered. RESULTS: The results of 26 diets of PACES and six diets of nPACES were examined statistically to assess the extent of hawkishness, as well as sex bias and ethnicity bias in individual examiners. The control (odd-number) condition suggested that about 5% of examiners were significant at an (uncorrected) 5% level, and that the method therefore worked as expected. As in a previous study (BMC Medical Education, 2006, 6:42), some examiners were hawkish or doveish relative to their peers. No examiners showed significant sex bias, and only a single examiner showed evidence consistent with ethnic bias. A re-analysis of the data considering only one examiner per station, as would be the case for many clinical examinations, showed that analysis with a single examiner runs a serious risk of false positive identifications probably due to differences in case-mix and content-specificity. CONCLUSIONS: In examinations where there are two independent examiners at a station, our method can assess the extent of bias against candidates with particular characteristics. The method would be far less sensitive in examinations with only a single examiner per station as examiner variance would be confounded with candidate performance variance. The method however works well when there is more than one examiner at a station and in the case of the current MRCP(UK) clinical examination, nPACES, found possible sex bias in no examiners and possible ethnic bias in only one.

Compellingly negative: Bayesian analysis shows handedness and dementia are not associated.

This scientific commentary refers to "The role of hand preference in cognition and neuropsychiatric symptoms in neurodegenerative diseases" by Saari & Vuoksimaa (https://doi.org/10.1093/braincomms/fcad137).