Improving the validity of script concordance testing by optimising and balancing items.

BACKGROUND: A script concordance test (SCT) is a modality for assessing clinical reasoning. Concerns had been raised about the plausible validity threat to SCT scores if students deliberately avoided the extreme answer options to obtain higher scores. The aims of the study were firstly to investigate whether students' avoidance of the extreme answer options could result in higher scores, and secondly to determine whether a 'balanced approach' by careful construction of SCT items (to include extreme as well as median options as model responses) would improve the validity of an SCT. METHODS: Using the paired sample t-test, the actual average student scores for 10 SCT papers from 2012-2016 were compared with simulated scores. The latter were generated by recoding all '-2' responses to '-1' and '+2' responses to '+1' for the whole and bottom 10% of the cohort (simulation 1), and scoring as if all students had chosen '0' for their responses (simulation 2). The actual average and simulated average scores in 2012 (before the 'balanced approach') were compared with those from 2013-2016, when papers had a good balance of modal responses from the expert reference panel. RESULTS: In 2012, a score increase was seen in simulation 1 in the third-year cohort, from 50.2 to 55.6% (t [10] = 4.818; p = 0.001). Since 2013, with the 'balanced approach', the actual SCT scores (57.4%) were significantly higher than scores in both simulation 1 and simulation 2 (46.7% and 23.9% respectively). CONCLUSIONS: When constructing SCT examinations, apart from the rigorous pre-examination optimisation, it is desirable to achieve a balance between items that attract extreme responses and those that attract median response options. This could mitigate the validity threat to SCT scores, especially for the low-performing students who have previously been shown to only select median responses and avoid the extreme responses.

Examining response process validity of script concordance testing: a think-aloud approach.

OBJECTIVES: This study investigated whether medical student responses to Script Concordance Testing (SCT) items represent valid clinical reasoning. Using a think-aloud approach students provided written explanations of the reasoning that underpinned their responses, and these were reviewed for concordance with an expert reference panel. METHODS: A set of 12, 11 and 15 SCT items were administered online to Year 3 (2018), Year 4 (2018) and Year 3 (2019) medical students respectively. Students' free-text descriptions of the reasoning supporting each item response were analysed, and compared with those of the expert panel. Response process validity was quantified as the rate of true positives (percentage of full and partial credit responses derived through correct clinical reasoning); and true negatives (percentage of responses with no credit derived through faulty clinical reasoning). RESULTS: Two hundred and nine students completed the online tests (response rate = 68.3%). The majority of students who had chosen the response which attracted full or partial credit also provided justifications which were concordant with the experts (true positive rate of 99.6% for full credit; 99.4% for partial credit responses). Most responses that attracted no credit were based on faulty clinical reasoning (true negative of 99.0%). CONCLUSIONS: The findings provide support for the response process validity of SCT scores in the setting of undergraduate medicine. The additional written think-aloud component, to assess clinical reasoning, provided useful information to inform student learning. However, SCT scores should be validated on each testing occasion, and in other contexts.

Does an incremental approach to implementing programmatic assessment work? Reflections on the change process.

This article was migrated. The article was marked as recommended. In 2017, the School of Medicine (Fremantle) of the University of Notre Dame Australia began moving towards programmatic assessment. Programmatic assessment seeks to achieve robust assessment validity through the assessment of a large number of low-stakes activities or data points. These data points exemplify assessment as learning by valuing feedback, discussion and reflection, ultimately leading to deeper student engagement without compromising credible decision-making on student progress. The School adopted an incremental approach to implementing programmatic assessment that included first establishing data-informed mentoring, and then activating a continuous assessment program that contributed simultaneously to student learning and School decision-making. Action research helped understand the impact of the initiative. Re-engineering continuous assessment as an incremental step towards programmatic assessment proved to be problematic. Some ideas are proposed to draw the strands of programmatic assessment together that may be useful for others to chart a more fruitful path.

Construct validity of script concordance testing: progression of scores from novices to experienced clinicians.

OBJECTIVES: To investigate the construct validity of Script Concordance Testing (SCT) scores as a measure of the clinical reasoning ability of medical students and practising General Practitioners with different levels of clinical experience. METHODS: Part I involved a cross-sectional study, where 105 medical students, 19 junior registrars and 13 experienced General Practitioners completed the same set of SCT questions, and their mean scores were compared using one-way ANOVA. In Part II, pooled and matched SCT scores for 5 cohorts of students (2012 to 2017) in Year 3 (N=584) and Year 4 (N=598) were retrospectively analysed for evidence of significant progression. RESULTS: A significant main effect of clinical experience was observed [F(2, 136)=6.215, p=0.003]. The mean SCT score for General Practitioners (M=70.39, SD=4.41, N=13) was significantly higher (p=0.011) than that of students (M = 64.90, SD = 6.30, N=105). Year 4 students (M=68.90, SD= 7.79, N=584) scored a significantly higher mean score [t(552)=12.78, p<0.001] than Year 3 students (M = 64.03, SD=7.98, N=598). CONCLUSIONS: The findings that candidate scores increased with increasing level of clinical experience add to current evidence in the international literature in support of the construct validity of Script Concordance Testing. Prospective longitudinal studies with larger sample sizes are recommended to further test and build confidence in the construct validity of SCT scores.

International medical electives in selected African countries: a phenomenological study on host experience.

OBJECTIVES: To explore the host experience on international medical electives at a selection of hospitals in low- and middle-income countries in Africa. Outcomes of the study may inform and improve the preparation of global health curriculum, pre-elective training and debriefing for international medical electives. METHODS: A descriptive phenomenological study was undertaken, involving semi-structured interviews with ten elective hosts at seven study sites in three African countries. Purposive convenience sampling augmented by snowballing was utilised to recruit study participants. The data were thematically analysed and interpreted with reflexivity to generate an accurate aggregate of the experience of participants in hosting international medical electives. RESULTS: Six main themes emerged from the thematic analysis of interview data:  international medical student contribution to host hospitals, host professional and personal fulfiment, barriers to student learning experience, international medical student preparedness, hope for reciprocity and barriers to cultural immersion and patient care. CONCLUSIONS: Study participants described the experience of hosting international medical elective students as overwhelmingly positive. However, they highlighted issues such as barriers to students' learning experience and the lack of reciprocity between host and sending institutions as areas which could be addressed to optimize the experience for both hosts and international medical students. An understanding of the host experience provides stakeholders with a clearer idea of what is important in preparation, organisation and evaluation of the elective experience. This study provides the impetus for further research to examine the effectiveness of introducing appropriate pre-departure training and post-elective debriefing to students embarking on international medical electives.

Clinical audit project in undergraduate medical education curriculum: an assessment validation study.

OBJECTIVES: To evaluate the merit of the Clinical Audit Project (CAP) in an assessment program for undergraduate medical education using a systematic assessment validation framework. METHODS: A cross-sectional assessment validation study at one medical school in Western Australia, with retrospective qualitative analysis of the design, development, implementation and outcomes of the CAP, and quantitative analysis of assessment data from four cohorts of medical students (2011- 2014). RESULTS: The CAP is fit for purpose with clear external and internal alignment to expected medical graduate outcomes.  Substantive validity in students' and examiners' response processes is ensured through relevant methodological and cognitive processes. Multiple validity features are built-in to the design, planning and implementation process of the CAP.  There is evidence of high internal consistency reliability of CAP scores (Cronbach's alpha > 0.8) and inter-examiner consistency reliability (intra-class correlation>0.7). Aggregation of CAP scores is psychometrically sound, with high internal consistency indicating one common underlying construct.  Significant but moderate correlations between CAP scores and scores from other assessment modalities indicate validity of extrapolation and alignment between the CAP and the overall target outcomes of medical graduates.  Standard setting, score equating and fair decision rules justify consequential validity of CAP scores interpretation and use. CONCLUSIONS: This study provides evidence demonstrating that the CAP is a meaningful and valid component in the assessment program. This systematic framework of validation can be adopted for all levels of assessment in medical education, from individual assessment modality, to the validation of an assessment program as a whole.