Perceived clinical relevance and retention of basic sciences across the medical education continuum.

Medical programs are under pressure to maintain currency with scientific and technical advances, as well as prepare graduates for clinical work and a wide range of postgraduate careers. The value of the basic sciences in primary medical education was assessed by exploring the perceived clinical relevance and test performance trends among medical students, interns, residents, and experienced clinicians. A pilot study conducted in 2014 involved administration of a voluntary 60-item multiple-choice question test to 225 medical students and 4 interns. These participants and 26 teaching clinicians rated the items for clinical relevance. In 2016, a similarly constructed test (main study) was made a mandatory formative assessment, attempted by 563 students in years 2, 4, and 6 and by 120 commencing general practice residents. Test scores, performance trends, clinical relevance ratings, and correlations were assessed using relevant parametric and nonparametric tests. Rank order and pass-fail decisions were also reviewed. The mean test scores were 57% (SD 7.1) and 52% (SD 6.1) for the pilot and main studies, respectively. Highest scores were observed in pathology and social sciences. Overall performance increased with increasing year of study. Test scores were positively correlated with perceived relevance. There were moderate correlations (r = 0.50-0.63; P < 0.001) between participants' scores in the basic science and summative exams. Assessments may be key to fostering relevance and integration of the basic sciences. Benchmarking knowledge retention and result comparisons across topics are useful in program evaluation.

Validity of the scan of postgraduate educational environment domains (SPEED) questionnaire in a rural general practice training setting.

BACKGROUND: The educational environment is critical to learning and is determined by social interactions. Trainee satisfaction translates to career commitment, retention and a positive professional attitude as well as being an important factor in assessing the impact of the training program. This study aimed to validate the Scan of Postgraduate Educational Environment Domain (SPEED) tool and assess its appropriateness in evaluating the quality of General Practice (GP) rural postgraduate educational environment. METHODS: A questionnaire containing the 15-item SPEED tool was administered to GP registrars to examine their perceptions of the educational environment. Principal component analysis (PCA) and exploratory factor analysis (EFA) were used to gather evidences of the validity of the instrument based on its internal structure. Additional validity evidence and reliability estimates were obtained using many-facet Rasch model analysis (MFRM). RESULTS: The survey was completed by 351 registrars with a response rate of 60%. Parallel analysis performed using principal component analysis and exploratory factor analysis suggests that the SPEED tool is unidimensional. The MFRM analysis demonstrated an excellent degree of infit and outfit for items and training sites, but not for persons. The MFRM analysis also estimated high reliability levels for items (0.98), training sites (0.95) and persons within training sites (ranging from 0.87 to 0.93 in each training sites). Overall, the registrars agreed that the educational environment had high quality, with most (13 out of 15) of the items rated above 4 out of 5. CONCLUSIONS: This study demonstrated a high degree of validity and reliability of the SPEED tool for the measurement of the quality of the educational environment in a rural postgraduate GP training context. However, when applied in a new setting, the tool may not function as a multidimensional tool consistent with its theoretical grounding.

Benefits of simulated General Practice clinics in the preparation of medical students for primary healthcare.

The James Cook University (JCU) medical school has a mission to produce doctors who are willing to work across northern Australia and may choose generalist rather than specialist careers. In addition to real-life placements in primary healthcare settings, the medical school has developed simulated General Practice (GP) clinics (simGPclinic) for Year 5 (Y5) students. This study compares the simGPclinic with actual GP placements for authenticity, teaching clinical skills, and preparation for real-life primary healthcare settings. Y5 students were administered a survey following their simGPclinic (n = 65; response rate = 97%). Students rated the simGPclinic's authenticity as 77 out of 100, and were more likely to rate the simGPclinic as being 'better' than their real-life GP placement in teaching them to: 'formulate a medical management plan and order correct pathology tests'; 'rule out the "red flags" for the key clinical problem'; 'undertake a patient-centred history and examination'; 'make a differential diagnosis for the key clinical problem'; and, 'develop communications skills'. The simGPclinic provided medical students with authentic and positive learning experiences in primary healthcare that were at least as beneficial as those provided in real-life settings, as well as being more reliable and better structured.

Improving assessment practice through cross-institutional collaboration: An exercise on the use of OSCEs.

BACKGROUND: This study was undertaken to improve assessment practice on OSCEs through collaboration across geographically dispersed medical schools in Australia. METHODS: A total of eleven OSCE stations were co-developed by four medical schools and used in summative 2011 and 2012 examinations for the assessment of clinical performance in the early clinical and exit OSCEs in each school's medical course. Partial Credit Rasch Model was used to evaluate the psychometric properties of the shared OSCE data. Evaluation of the quality assurance reports was used to determine the beneficial impact of the collaborative benchmarking exercise on learning and teaching outcomes. RESULTS: The data for each examination demonstrated sufficient fit to the Rasch model with infit mean square values ranging from 0.88 to 0.99. Person separation (1.25-1.63) indices indicated good reliability. Evaluation of perceived benefits showed that the benchmarking process was successful as it highlighted common curriculum areas requiring specific focus and provided comparable data on the quality of teaching at the participating medical schools. CONCLUSION: This research demonstrates the validity of the psychometric data and benefits of evaluating clinical competence across medical schools without the enforcement of a prescriptive national curriculum or assessment.

Key elements of effective postgraduate GP educational environments: a mixed methods study.

OBJECTIVES: Evidence in the literature suggests that satisfaction with postgraduate general practice (GP) training is associated with the quality of the educational environment. This study aimed to examine GP registrars' level of satisfaction with a distributed model of training in a regional educational environment and investigate the relationship between satisfaction and academic performance. STUDY DESIGN: A longitudinal 3-year study was conducted among GP registrars at James Cook University using a sequential explanatory mixed methods research design. GP registrars' satisfaction was obtained using the scan of postgraduate educational environment domains tool. A focus group discussion was conducted to explore GP registrars' perceptions of satisfaction with the educational environment. SETTING: James Cook University General Practice Training (JCU GPT) programme. PARTICIPANTS: Six hundred and fifty one (651) GP registrars enrolled between 2016 and 2018 at JCU GPT programme. RESULTS: 651 registrars completed the satisfaction survey between 2016 and 2018. Overall, 92% of the registrars were satisfied with the educational training environment. Registrars who had become fellows reported higher satisfaction levels compared with those who were still in training (mean=4.39 vs 4.20, p=0.001). However, academic performance had no impact on level of satisfaction with the educational environment. Similarly, practice location did not influence registrars' satisfaction rates. Four themes (rich rural/remote educational environment, supportive learning environment, readiness to continue with rural practice and practice culture) emerged from the thematic data analysis. CONCLUSION: A clinical learning environment that focuses on and supports individual learning needs is vital for effective postgraduate medical training. This study suggests that JCU GPT programme's distributed model fostered a satisfying and supportive training environment with rich educational experiences that enhance retention of GP registrars in rural/remote North Queensland, Australia. The findings of this study may be applicable to other settings with similar training models.

Role of formative assessment in predicting academic success among GP registrars: a retrospective longitudinal study.

OBJECTIVES: The James Cook University General Practice Training (JCU GPT) programme's internal formative exams were compared with the Royal Australian College of General Practitioners (RACGP) pre-entry exams to determine ability to predict final performance in the RACGP fellowship exams. DESIGN: A retrospective longitudinal study. SETTING: General Practice (GP) trainees enrolled between 2016 and 2019 at a Registered Training Organisation in regional Queensland, Australia. PARTICIPANTS: 376 GP trainees enrolled in the training programme. EXPOSURE MEASURES: The pre-entry exams were Multiple-Mini Interviews (MMI), Situational Judgement Test (SJT) and Candidate Assessment and Applied Knowledge Test. The internal formative exams comprised multiple choice questions (MCQ1 and MCQ2), short answer questions, clinical skills and clinical reasoning. PRIMARY OUTCOME MEASURE: The college exams were Applied Knowledge Test (AKT), Key Feature Problems (KFP) and Objective Structured Clinical Examination (OSCE). RESULTS: Correlations (r), coefficients of determination (R2) and OR were used as parameters for estimating strength of relationship and precision of predictive accuracy. SJT and MMI were moderately (r=0.13 to 0.31) and MCQ1 and MCQ2 highly (r=0.37 to 0.53) correlated with all college exams (p<0.05 to p<0.01), with R2 ranging from 0.070 to 0.376. MCQ1 was predictive of failure in all college exams (AKT: OR=2.32, KFP: OR=3.99; OSCE: OR=3.46); while MCQ2 predicted failure in AKT (OR=2.83) and KFP (OR=3.15). CONCLUSION: We conclude that the internal MCQ formative exams predict performance in the RACGP fellowship exams. We propose that our formative assessment tools could be used as academic markers for early identification of potentially struggling trainees.