Exploring reasons for differences in performance between UK and international medical graduates in the Membership of the Royal College of General Practitioners Applied Knowledge Test: a cognitive interview study.

OBJECTIVES: International medical graduates (IMGs) perform less well in national postgraduate licensing examinations compared with UK graduates, even in computer-marked multiple-choice licensing examinations. We aimed to investigate thought processes of candidates answering multiple- choice questions, considering possible reasons for differential attainment between IMGs and UK graduates. DESIGN: We employed a semistructured qualitative design using cognitive interviews. Systematic grounded theory was used to analyse data from 'think aloud' interviews of general practitioner specialty trainees (GPSTs) while answering up to 15 live questions from the UK Membership of the Royal College of General Practitioners Applied Knowledge Test (AKT). SETTING: East Midlands, UK. PARTICIPANTS: 21 GPSTs including 13IMGs and 8 UK-trained doctors. OUTCOMES: Perceptions of participants on how they answered AKT questions together with strategies used or difficulties experienced. RESULTS: We interviewed 21 GPSTs (8 female, 13 male, 13 IMGs, 14 from black and minority ethnic groups, age 24-64 years) in years 1-3 of training between January and April 2017. Four themes were identified. 'Theoretical versus real-life clinical experience': participants reported difficulties recalling information and responding to questions from theoretical learning compared with clinical exposure; rote learning helped some IMGs recall rare disease patterns. Recency, frequency, opportunity and relevance: participants reported greater difficulty answering questions not recently studied, less frequently encountered or perceived as less relevant. Competence versus insight: some participants were over optimistic about their performance despite answering incorrectly. Cultural barriers: for IMGs included differences in undergraduate experience, lack of familiarity with UK guidelines and language barriers which overlapped with the other themes. CONCLUSIONS: The difficulties we identified in candidates when answering AKT questions may be addressed through training. IMGs face additional difficulties which impede examination success due to differences in educational experience, content familiarity and language, which are also potentially amenable to additional training support.

Has there been a change in the knowledge of GP registrars between 2011 and 2016 as measured by performance on common items in the Applied Knowledge Test?

The aim of this study was to assess whether the absolute standard of candidates sitting the MRCGP Applied Knowledge Test (AKT) between 2011 and 2016 had changed. It is a descriptive study comparing the performance on marker questions of a reference group of UK graduates taking the AKT for the first time between 2011 and 2016. Using aggregated examination data, the performance of individual 'marker' questions was compared using Pearson's chi-squared tests and trend-line analysis. Binary logistic regression was used to analyse changes in performance over the study period. Changes in performance of individual marker questions using Pearson's chi-squared test showed statistically significant differences in 32 of the 49 questions included in the study. Trend line analysis showed a positive trend in 29 questions and a negative trend in the remaining 23. The magnitude of change was small. Logistic regression did not demonstrate any evidence for a change in the performance of the question set over the study period. However, candidates were more likely to get items on administration wrong compared with clinical medicine or research. There was no evidence of a change in performance of the question set as a whole.

Performance of candidates disclosing dyslexia with other candidates in a UK medical licensing examination: cross-sectional study.

PURPOSE OF THE STUDY: The aim of this study was to compare performance of candidates who declared an expert-confirmed diagnosis of dyslexia with all other candidates in the Applied Knowledge Test (AKT) of the Membership of the Royal College of General Practitioners licensing examination. STUDY DESIGN: We used routinely collected data from candidates who took the AKT on one or more occasions between 2010 and 2015. Multivariate logistic regression was used to analyse performance of candidates who declared dyslexia with all other candidates, adjusting for candidate characteristics known to be associated with examination success including age, sex, ethnicity, country of primary medical qualification, stage of training, number of attempts and time spent completing the test. RESULTS: The analysis included data from 14 examinations involving 14 801 candidates of which 2.6% (379/14 801) declared dyslexia. The pass rate for candidates who declared dyslexia was 83.6% compared with 95.0% for other candidates. After adjusting for covariates linked to examination success including age, sex, ethnicity, country of primary medical qualification, stage of training, number of attempts and time spent completing the test dyslexia was not significantly associated with pass rates in the AKT. Candidates declaring dyslexia after initially failing the AKT were more likely to have a primary medical qualification outside the UK. CONCLUSIONS: Performance was similar in AKT candidates disclosing dyslexia with other candidates once covariates associated with examination success were adjusted for. Candidates declaring dyslexia after initially failing the AKT were more likely to have a primary medical qualification outside the UK.

Quality assurance of the Applied Knowledge Test (AKT) of the MRCGP examination - an immediate post-test questionnaire evaluation of the candidates' views.

The Applied Knowledge Test (AKT) of the Membership of the Royal College of General Practitioners (MRCGP) examination is a computer-based assessment delivered three times a year. A computerised questionnaire, administered immediately after the test, sought candidates' views as part of the test evaluation. Of 1681 candidates taking the test 1418 (84%) responded. Most candidates believed that the test assessed their knowledge of problems relevant to general practice. Their feedback highlighted areas where improvements could be made. Candidates' views of postgraduate specialty medical examinations in the UK are rarely sought or published. We are not aware of other published evidence. The use of computer-based testing enables immediate candidate feedback and can be used routinely to evaluate the test validity and formats. The views of candidates are an important component of quality assurance in reviewing the content, format and educational experience of a high-stakes examination.

Comparing performance among male and female candidates in sex-specific clinical knowledge in the MRCGP.

BACKGROUND: Patients often seek doctors of the same sex, particularly for sex-specific complaints and also because of a perception that doctors have greater knowledge of complaints relating to their own sex. Few studies have investigated differences in knowledge by sex of candidate on sex-specific questions in medical examinations. AIM: The aim was to compare the performance of males and females in sex-specific questions in a 200-item computer-based applied knowledge test for licensing UK GPs. DESIGN AND SETTING: A cross-sectional design using routinely collected performance and demographic data from the first three versions of the Applied Knowledge Test, MRCGP, UK. METHOD: Questions were classified as female specific, male specific, or sex neutral. The performance of males and females was analysed using multiple analysis of covariance after adjusting for sex-neutral score and demographic confounders. RESULTS: Data were included from 3627 candidates. After adjusting for sex-neutral score, age, time since qualification, year of speciality training, ethnicity, and country of primary medical qualification, there were differences in performance in sex-specific questions. Males performed worse than females on female-specific questions (-4.2%, 95% confidence interval [CI] = -5.7 to -2.6) but did not perform significantly better than females on male-specific questions (0.3%, 95% CI = -2.6 to 3.2%. CONCLUSION: There was evidence of better performance by females in female-specific questions but this was small relative to the size of the test. Differential performance of males and females in sex-specific questions in a licensing examination may have implications for vocational and post-qualification general practice training.

Performance and views of examiners in the Applied Knowledge Test for the nMRCGP licensing examination.

BACKGROUND: A new computer-based Applied Knowledge Test (AKT) has been developed for the licensing examination for general practice administered by the Royal College of General Practitioners. AIM: The aim of this evaluation was to assess the acceptability, feasibility, and validity of the test as well as its transfer to a computerised format at local test centres. DESIGN OF STUDY: Computer-based test and postal questionnaire. PARTICIPANTS AND SETTING: Panel of examiners, Membership of the Royal College of General Practitioners (MRCGP) examination, UK. METHOD: Self-administered postal questionnaires were sent to examiners not involved with the development of the test after completing it. Their performance scores were compared with those of candidates. RESULTS: The majority of participants (80.9%) were satisfied with the new computer-based test. Responses relating to content and attitudes to the test were also positive overall, but some problems with content were highlighted. Fewer examiners (61.9%) were positive about the physical comfort of the test centre, including seating, heating, and lighting. Examiners had significantly higher scores (mean 83.3%, range 69 to 93%, 95% confidence interval [CI] = 81.9 to 84.7%) than 'real' candidates (mean 75.0%, range 45 to 94%, 95% CI = 74.6 to 75.5%), who subsequently took an identical test. CONCLUSION: The new computer-based licensing test (the AKT) was found to be acceptable to the majority of examiners. The pass-fail standard, determined by routine methods including an Angoff procedure, was supported by the higher success rate of examiners compared with candidates. The use of selected groups to assess high-stakes (licensing) examinations can be useful for assessing test validity.