Evaluation of the Prescribing Skills Assessment implementation, performance and medical student experience in Australia and New Zealand.

AIMS: The UK Prescribing Safety Assessment was modified for use in Australia and New Zealand (ANZ) as the Prescribing Skills Assessment (PSA). We investigated the implementation, student performance and acceptability of the ANZ PSA for final-year medical students. METHODS: This study used a mixed-method approach involving student data (n = 6440) for 2017-2019 (PSA overall score and 8 domain subscores). Data were also aggregated by medical school and included student evaluation survey results. Quantitative data were analysed using descriptive and multivariate analyses. The pass rate was established by a modified Angoff method. Thematic analyses of open-ended survey comments were conducted. RESULTS: The average pass rate was slightly higher in 2017 (89%) which used a different examination to 2018 (85%) and 2019 (86%). Little difference was identified between schools for the PSA overall performance or domain subscores. There was low intercorrelation between subscores. Most students provided positive feedback about the PSA regarding the interface and clarity of questions, but an average of 35% reported insufficient time for completion. Further, 70% on average felt unprepared by their school curricula for the PSA, which is in part explained by the low prescribing experience; 69% reported completing ≤10 prescriptions during training. CONCLUSION: The ANZ PSA was associated with high pass rates and acceptability, although student preparedness was highlighted as a concern for further investigation. We demonstrate how a collaboration of medical schools can adapt a medical education assessment resource (UK PSA) as a means for fulfilling an unmet need.

Student perspectives on programmatic assessment in a large medical programme: A critical realist analysis.

BACKGROUND: Fundamental challenges exist in researching complex changes of assessment practice from traditional objective-focused 'assessments of learning' towards programmatic 'assessment for learning'. The latter emphasise both the subjective and social in collective judgements of student progress. Our context was a purposively designed programmatic assessment system implemented in the first year of a new graduate entry curriculum. We applied critical realist perspectives to unpack the underlying causes (mechanisms) that explained student experiences of programmatic assessment, to optimise assessment practice for future iterations. METHODS: Data came from 14 in-depth focus groups (N = 112/261 students). We applied a critical realist lens drawn from Bhasker's three domains of reality (the actual, empirical and real) and Archer's concept of structure and agency to understand the student experience of programmatic assessment. Analysis involved induction (pattern identification), abduction (theoretical interpretation) and retroduction (causal explanation). RESULTS: As a complex educational and social change, the assessment structures and culture systems within programmatic assessment provided conditions (constraints and enablements) and conditioning (acceptance or rejection of new 'non-traditional' assessment processes) for the actions of agents (students) to exercise their learning choices. The emergent underlying mechanism that most influenced students' experience of programmatic assessment was one of balancing the complex relationships between learner agency, assessment structures and the cultural system. CONCLUSIONS: Our study adds to debates on programmatic assessment by emphasising how the achievement of balance between learner agency, structure and culture suggests strategies to underpin sustained changes (elaboration) in assessment practice. These include; faculty and student learning development to promote collective reflexivity and agency, optimising assessment structures by enhancing integration of theory with practice, and changing learning culture by both enhancing existing and developing new social structures between faculty and the student body to gain acceptance and trust related to the new norms, beliefs and behaviours in assessing for and of learning.

Situational judgement tests for selection: Traditional vs construct-driven approaches.

CONTEXT: Historically, situational judgement tests (SJTs) have been widely used for personnel selection. Their use in medical selection in Europe is growing, with plans for further expansion into North America and Australasia, in an attempt to measure and select on 'non-academic' personal attributes. However, there is a lack of clarity regarding what such tests actually measure and how they should be designed, scored and implemented within the medical and health education selection process. In particular, the theoretical basis from which such tests are developed will determine the scoring options available, influencing their psychometric properties and, ultimately, their validity. METHODS: The aim of this article is to create an awareness of the previous theory and practice that has informed SJT development. We describe the emerging interest in the use of the SJT format to measure specific constructs (eg 'resilience', 'dependability', etc.), drawing on the tradition of 'individual differences' psychology. We compare and contrast this newer 'construct-driven' method with the traditional, pragmatic approach to SJT creation, often employed by organisational psychologists. Making reference to measurement theory, we highlight how the anticipated psychometric properties of traditional vs construct-driven SJTs are likely to differ. CONCLUSIONS: Compared to traditional SJTs, construct-driven SJTs have a strong theoretical basis, are uni- rather than multidimensional, and may behave more like personality self-report instruments. Emerging evidence also suggests that construct-driven SJTs have comparable predictive validity for workplace performance, although they may be more prone to 'faking' effects. It is possible that construct-driven approaches prove more appropriate at early stages of medical selection, where candidates have little or no health care work experience. Conversely, traditional SJTs may be more suitable for specialty recruitment, where a range of hypothetical workplace scenarios can be sampled in assessments.

Standard setting in Australian medical schools.

BACKGROUND: Standard setting of assessment is critical in quality assurance of medical programs. The aims of this study were to identify and compare the impact of methods used to establish the passing standard by the 13 medical schools who participated in the 2014 Australian Medical Schools Assessment Collaboration (AMSAC). METHODS: A survey was conducted to identify the standard setting procedures used by participating schools. Schools standard setting data was collated for the 49 multiple choice items used for benchmarking by AMSAC in 2014. Analyses were conducted for nine schools by their method of standard setting and key characteristics of 28 panel members from four schools. RESULTS: Substantial differences were identified between AMSAC schools that participated in the study, in both the standard setting methods and how particular techniques were implemented. The correlation between the item standard settings data by school ranged from - 0.116 to 0.632. A trend was identified for panel members to underestimate the difficulty level of hard items and overestimate the difficulty level of easy items for all methods. The median derived cut-score standard across schools was 55% for the 49 benchmarking questions. Although, no significant differences were found according to panel member standard setting experience or clinicians versus scientists, panel members with a high curriculum engagement generally had significantly lower expectations of borderline candidates (p = 0.044). CONCLUSION: This study used a robust assessment framework to demonstrate that several standard setting techniques are used by Australian medical schools, which in some cases use different techniques for different stages of their program. The implementation of the most common method, the Modified Angoff standard setting approach was found to vary markedly. The method of standard setting used had an impact on the distribution of expected minimally competent student performance by item and overall, with the passing standard varying by up to 10%. This difference can be attributed to the method of standard setting because the ASMSAC items have been shown over time to have consistent performance levels reflecting similar cohort ability. There is a need for more consistency in the method of standard setting used by medical schools in Australia.

The Australian Medical Schools Assessment Collaboration: benchmarking the preclinical performance of medical students.

OBJECTIVES: To report the level of participation of medical schools in the Australian Medical Schools Assessment Collaboration (AMSAC); and to measure differences in student performance related to medical school characteristics and implementation methods. DESIGN: Retrospective analysis of data using the Rasch statistical model to correct for missing data and variability in item difficulty. Linear model analysis of variance was used to assess differences in student performance. SETTING AND PARTICIPANTS: 6401 preclinical students from 13 medical schools that participated in AMSAC from 2011 to 2013. MAIN OUTCOME MEASURES: Rasch estimates of preclinical basic and clinical science knowledge. RESULTS: Representation of Australian medical schools and students in AMSAC more than doubled between 2009 and 2013. In 2013 it included 12 of 19 medical schools and 68% of medical students. Graduate-entry students scored higher than students entering straight from school. Students at large schools scored higher than students at small schools. Although the significance level was high (P < 0.001), the main effect sizes were small (4.5% and 2.3%, respectively). The time allowed per multiple choice question was not significantly associated with student performance. The effect on performance of multiple assessments compared with the test items as part of a single end-of-year examination was negligible. The variables investigated explain only 12% of the total variation in student performance. CONCLUSIONS: An increasing number of medical schools are participating in AMSAC to monitor student performance in preclinical sciences against an external benchmark. Medical school characteristics account for only a small part of overall variation in student performance. Student performance was not affected by the different methods of administering test items.

Internet-based multiple mini-interviews for candidate selection for graduate entry programmes.

CONTEXT: Multiple mini-interviews (MMIs) have been used by The University of Sydney graduate medical and dental programmes since 2006. In 2011, interviews with international candidates were conducted using Skype (iMMI), whereas interviews with local candidates were conducted in person. We determined whether the MMI scores derived from both methods were comparable. We describe the feasibility, acceptability and cost-effectiveness of the iMMI. METHODS: We compared 2011 international student internet-based iMMI results with data from 2009 international student MMIs and 2011 local student MMIs. Analyses of variance (anovas) were used to investigate equivalence of the two formats by exploring whether the medium of interviewing resulted in significantly different mean scores and variance for the in-person MMI and the iMMI. Acceptability of the process was informed by feedback surveys from interviewers and candidates, and cost savings were estimated. RESULTS: No significant difference was found between the 2011 iMMI scores for international candidates and MMI scores in 2009 (p > 0.05). There was no significant difference between the MMI scores for local and international candidates in 2011 (p > 0.05); the MMI scores for international candidates had greater variation (p < 0.01). Using generalisability theory, the reliability of the nine-question iMMI was 0.76 and for the MMI was 0.70. Delivery of the iMMI occurred smoothly and candidates and interviewers gave positive feedback on its format and delivery. Cost savings have been estimated to be over AU$50 000, representing an 84% saving. CONCLUSIONS: We believe this is the first study reporting an internet-based MMI for a high stakes interview. We have shown that interviewers were able to make valid and reliable decisions about candidates through the iMMI in a process that was acceptable to participants, producing comparable results to the in-person MMI with a saving of resources. The slightly wider variance in iMMI scores warrants further investigation.

A workshop to promote interprofessional care.

BACKGROUND: Paediatric incontinence has traditionally been managed through a discipline-specific approach by doctors, nurses, physiotherapists and psychologists. We evaluated a workshop aiming to increase health professionals' knowledge, confidence and willingness to involve other health professionals when managing paediatric incontinence. METHODS: Our 1-day workshop focused on paediatric bedwetting, daytime incontinence, faecal incontinence and neurogenic bladder. Attendees completed surveys before and after the workshop, and 6 months later. Differences between health professionals and changes in attitudes, knowledge, confidence and experience before and after the workshop were analysed using descriptive analysis. Qualitative data analysis was undertaken using content analysis. RESULTS: The workshop was attended by 77 clinicians (41 doctors, 19 allied health professionals and 17 nurses). Over two-thirds of respondents acknowledged having average or above average knowledge of and/or confidence in managing bedwetting, daytime urinary incontinence (DUI) and faecal incontinence, with lower baseline knowledge and confidence for neurogenic bladder. Participants who completed the initial and final surveys reported increased knowledge, improved ability to manage patients, better appreciation of the contribution of other disciplines and willingness to refer patients to other disciplines. Of those who saw patients with incontinence, most reported that the workshop had improved their management: 33/35 for bedwetting; 31/35 for DUI; 21/29 for faecal incontinence; and 16/19 for neurogenic bladder. Paediatric incontinence has traditionally been managed through a discipline-specific approach by doctors, nurses, physiotherapists and psychologists DISCUSSION: This study demonstrates that an interprofessional workshop conducted by clinicians from different health professional backgrounds increases clinicians' awareness of the roles of other disciplines and promotes the development of an interdisciplinary approach or multiprofessional collaboration, enabling a more effective, patient-centred, collaborative care model.

Surgical teaching program for our senior medical students: room for improvement.

BACKGROUND: To ensure the quality of surgical teaching within our graduate entry medical program, a distinctive surgical teaching program has been developed at Sydney Medical School-Central. Spanning 2 years, the program includes lectures, small group surgical clinical tutorials, and formal student surgical grand rounds presentations, plus clinical placements and attendance in operating theaters. We sought to evaluate the effectiveness of the program. METHODS: In 2013, at the completion of year 4, all graduating students (n=54) were asked to complete an open and closed-ended questionnaire regarding their experience of the surgical program. RESULTS: A total of 44/54 (81%) students completed the questionnaire. Students reported a high level of engagement with their experience in clinical tutorials, and a moderate level of engagement in surgical lectures. Students found the clinical attachment to be the least useful method of teaching, with the surgical grand rounds presentation also eliciting a poor response from students. CONCLUSION: While both large group lectures and small group learner-centered teaching methods were highly valued by students, changes are needed to enhance clinical attachments for students in surgical wards. The benefits of students being made to feel part of a team during their surgical clinical attachments, along with adequate inpatient contact and formative feedback, should not be underestimated.

Bayer-activities of daily living scale in mild and moderate dementia of the Alzheimer type.

The main purpose of this study is to examine the reliability of the Bayer-Activities of Daily Living (B-ADL) scale when used as a cognitive screening instrument for mild and moderate dementia of the Alzheimer type. This is a retrospective study of 66 patients with dementia. The B-ADL scale was completed by the caregiver or the family member at the first encounter. The internal consistency was found to be 0.94 for the 27 patients that completed all 25 questions in the scale. Significant correlation and receiver-operating characteristic curve analysis were found for the B-ADL total score and subscale 1 (tasks requiring short- and long-term memory) for Clinical Dementia Rating scale. Severity of dementia by the B-ADL scale is statistically similar but not the same as Mini-Mental State Examination. Our findings confirm that B-ADL scale is a valid indicator of the cognitive status of patients with Alzheimer's disease.