Dutch postgraduate GP selection procedure; reliability of interview assessments.

BACKGROUND: Semi- structured interviews are the core of the Dutch selection procedure for postgraduate general practice (GP) training. A staff member, trainer and trainee independently assess personal qualities. Aiming to improve the selection procedure we were interested in the reliability aspects of these interviews. We investigated the inter-rater reliability of the interview for groups of two or three assessors and the degree to which candidates' characteristics and qualities assessed during interviews explained admission into GP training, controlled for differences between those who apply for the first versus the second or third application. METHODS: An observational study was conducted of all candidates who entered the Utrecht selection procedure between April 2008 and 2010. Candidates' characteristics and qualities were collected. Inter-rater reliability of different compositions of the interview group per quality was estimated. Factors associated with admission into GP training were assessed. RESULTS: The study population included 394 candidates. Twenty-six candidates were rejected based on their application letter (4.4%). Three candidates who applied more than 3 times were excluded. Ultimately, 206 of the 365 candidates were admitted to the GP training (56,4%). The inter-rater reliability was satisfactory (ICC: 0.78 - 0 .84). Reduction from three to two assessors slightly reduces the ICC. The candidates' qualities independently explained admission to GP training, whereas individual characteristics did not. These results did not differ for candidates who applied for the first time versus candidates applying for the second or third time. CONCLUSION: Selection interviews with two assessors yielded a satisfactory level of reliability. Individual characteristics were not associated with admission, whereas scores related to candidate qualities did show such an association. The results of those applying for the second or third time were similar.

Poor Performance Among Trainees in a Dutch Postgraduate GP Training Program.

BACKGROUND AND OBJECTIVES: Poor performance among trainees is an important issue, for patient safety and economic reasons. While early identification might enhance remediation measures, we explored the frequency, nature, and risk factors of poor performance in a Dutch postgraduate general practitioner (GP) training program. METHODS: All trainees who started the GP training between 2005 and 2007 were included. Multivariate logistic regression analysis was applied to examine associations between individual characteristics; early assessments of competencies and knowledge, training process characteristics (eg, illness, maternal leave), and the outcome poor performance; sub-analyses were performed for each year. RESULTS: A total of 215 trainees started the 3-year GP program, and 49 (22.8%) exhibited poor performance (in one or more years). In the first and second years, problem areas among poor performers were equally distributed across the roles of "medical expert," "communicator," and "professional." In the third year, shortcomings in "professionalism" were the most common problem. Increasing age was a risk factor for poor performance as were insufficient scores in communication and knowledge. Poor performance in the previous year was a risk factor for poor performance in the second and third years; OR=4.20 (CI=1.31--13.47) and OR=5.40 (CI=1.58--18.47), respectively. CONCLUSIONS: Poor performance is prevalent but primarily occurring within a single training year. This finding suggests that trainees are capable of solving trainee problems. Increasing age, insufficient assessment scores early in the training, and poor performance in a previous year constitute risk factors for poor performance.

A competency based selection procedure for Dutch postgraduate GP training: a pilot study on validity and reliability.

UNLABELLED: Abstract Background: Historically, semi-structured interviews (SSI) have been the core of the Dutch selection for postgraduate general practice (GP) training. This paper describes a pilot study on a newly designed competency-based selection procedure that assesses whether candidates have the competencies that are required to complete GP training. OBJECTIVES: The objective was to explore reliability and validity aspects of the instruments developed. METHODS: The new selection procedure comprising the National GP Knowledge Test (LHK), a situational judgement tests (SJT), a patterned behaviour descriptive interview (PBDI) and a simulated encounter (SIM) was piloted alongside the current procedure. Forty-seven candidates volunteered in both procedures. Admission decision was based on the results of the current procedure. RESULTS: Study participants did hardly differ from the other candidates. The mean scores of the candidates on the LHK and SJT were 21.9 % (SD 8.7) and 83.8% (SD 3.1), respectively. The mean self-reported competency scores (PBDI) were higher than the observed competencies (SIM): 3.7(SD 0.5) and 2.9(SD 0.6), respectively. Content-related competencies showed low correlations with one another when measured with different instruments, whereas more diverse competencies measured by a single instrument showed strong to moderate correlations. Moreover, a moderate correlation between LHK and SJT was found. The internal consistencies (intraclass correlation, ICC) of LHK and SJT were poor while the ICC of PBDI and SIM showed acceptable levels of reliability. CONCLUSION: Findings on content validity and reliability of these new instruments are promising to realize a competency based procedure. Further development of the instruments and research on predictive validity should be pursued.

Selection for Dutch postgraduate GP training; time for improvement.

BACKGROUND: In the Netherlands we select candidates for the postgraduate GP training by assessing personal qualities in interviews. Because of differences in the ratio of number of candidates and number of vacancies between the eight departments of GP training we questioned whether the risk of being rejected diverged amongst them. OBJECTIVE: The research question of this study was to which degree department of choice, candidates' characteristics and qualities assessed during interviews explain admission into GP training. METHODS: A nationwide observational study was conducted of all candidates who applied for postgraduate GP training in 2009/ 2010. Application ratio per department, candidates' characteristics (gender, age, region of medical school and times of application) and qualities (motivation, orientation on the job, personal attributes and learning needs) were collected. Outcome measures were admission to interview and admission to GP training. RESULTS: The study population addressed 542 candidates. Sixty three candidates were rejected on application letter (11.6%). So 479 candidates were admitted to the interview, of which 340 were admitted to the GP training (71%). Gender and region of medical school outside North West Europe were associated with admission to the interview. Department of choice had a strong association with admission in both stages (RR: 0.30 to 0.74; 0.20 to 0.79 respectively), while candidates' qualities explained admission (RR: 1.09- 1.25) as well. CONCLUSION: The influence of department of choice yields doubts about fairness of the procedure. So advantages and disadvantages of a national procedure are discussed as well as those of a competency based procedure.

[Attrition and poor performance in general practice training: age, competence and knowledge play a role]. / Uitval en stagnatie in de opleiding tot huisarts. Leeftijd, competenties en kennis spelen een rol.

OBJECTIVE: To investigate which determinants are related to poor performance and forced attrition in the first year residency in general practice (GP). DESIGN: Observational retrospective cohort study. METHOD: We collected data relating to personal characteristics such as age, sex and clinical experience from residents who started the GP training in Utrecht, the Netherlands, in the period March 2005-August 2007. We also collected competence scores from the domains 'medical expertise', 'doctor-patient communication' and 'professionalism', as well as scores on a national GP knowledge test. The outcome measures were 'poor performance' and 'forced attrition'. Multivariate logistic regression was used to analyse correlations between personal characteristics, competence scores on the 3 domains and knowledge scores in the first trimester on the one hand and poor performance or forced attrition on the other. RESULTS: 215 residents started the GP training. In the first trimester a quarter of the residents had an insufficient score in 1 or more of the domains. Competence scores were mutually correlated, but did not correlate with the knowledge score. 18 residents showed poor performance and 3 were forced to stop their training. Poor performance and forced attrition were correlated with age (adjusted odds ratio (OR): 1.1; 95% CI: 1.0-1.3), insufficient knowledge (adjusted OR: 8.9; 3.0-26.3) and medical expertise (adjusted OR: 2.1; 1.1-4.0) at the beginning of the training. CONCLUSION: Age, insufficient knowledge of general practice, and insufficient competence in the domain of 'medical expertise' at the beginning of the training are risk factors for poor performance by residents and attrition from their GP training.

Lessons learnt from incidents reported by postgraduate trainees in Dutch general practice. A prospective cohort study.

BACKGROUND: There is an inherent tension between allowing trainees in general practice (GP) to feel comfortable to report and learn from errors in a blame-free environment while still assuring high-quality and safe patient care. Unfortunately, little is known about the types and potential severity of incidents that may confront GP trainees. Furthermore, incident reporting by resident trainees is hindered by their concern that such transparency might result in more negative performance evaluations. OBJECTIVE: To explore the number and nature of incidents that were reported by GP trainees and to determine whether there were differences between the reporters and non-reporters based on their performance evaluations. DESIGN: Prospective cohort study. METHODS: Confidential and voluntary incident reporting was implemented in GP vocational training of the University Medical Center Utrecht, the Netherlands. Seventy-nine GP trainees were asked to report incidents over 6 months. Mixed methods were used to analyse the data. RESULTS: 24 trainees reported a total of 44 incidents. 23 incidents concerned the work process and 17 concerned problems with diagnosis or therapy. Three-quarters (34/44) of incidents were determined to be not specifically related to the inexperience of the GP trainees. While actual patient harm was determined to be minimal or absent in two-thirds of incidents (29/44), the potential for moderate, major, or catastrophic harm was 89% (39/44). Trainees performing best on their performance assessment in the domain of clinical expertise reported incidents more often (43% vs 18%, p<0.03) than those who performed at a lower level. CONCLUSIONS: GP trainees rated highly by their faculty voluntarily reported incidents in the delivery of clinical care when given a safe, blame-free, and confidential reporting process. Most incidents were not found to be directly related to the inexperience of the trainee, but were caused by failing organisational processes in the healthcare delivery system. Moreover, the trainees who tended to report these incidents were those whose performance was highly evaluated in the domain of clinical expertise.