Expanding Kane's argument-based validity framework: What can validation practices in language assessment offer health professions education?

CONTEXT: One central consideration in health professions education (HPE) is to ensure we are making sound and justifiable decisions based on the assessment instruments we use on health professionals. To achieve this goal, HPE assessment researchers have drawn on Kane's argument-based framework to ascertain the validity of their assessment tools. However, the original four-inference model proposed by Kane - frequently used in HPE validation research - has its limitations in terms of what each inference entails and what claims and sources of backing are housed in each inference. The under-specification in the four-inference model has led to inconsistent practices in HPE validation research, posing challenges for (i) researchers who want to evaluate the validity of different HPE assessment tools and/or (ii) researchers who are new to test validation and need to establish a coherent understanding of argument-based validation. METHODS: To address these identified concerns, this article introduces the expanded seven-inference argument-based validation framework that is established practice in the field of language testing and assessment (LTA). We explicate (i) why LTA researchers experienced the need to further specify the original four Kanean inferences; (ii) how LTA validation research defines each of their seven inferences and (iii) what claims, assumptions and sources of backing are associated with each inference. Sampling six representative validation studies in HPE, we demonstrate why an expanded model and a shared disciplinary validation framework can facilitate the examination of the validity evidence in diverse HPE validation contexts. CONCLUSIONS: We invite HPE validation researchers to experiment with the seven-inference argument-based framework from LTA to evaluate its usefulness to HPE. We also call for greater interdisciplinary dialogue between HPE and LTA since both disciplines share many fundamental concerns about language use, communication skills, assessment practices and validity in assessment instruments.

Do students' self-reflections of performance align with their graded performance in objective structured clinical exams?

INTRODUCTION: Self-awareness of strengths and weaknesses through self-reflection are important for life-long learning and development. The aim of this study was to assess the alignment in third-year undergraduate pharmacy students' self-reflections of their objective structured clinical exam (OSCE) performance to their actual OSCE scores and explore the most common aspects students reflected on as markers of perceived performance. METHODS: Students completed a three-station OSCE and a written self-reflection about their performance. These reflections were coded using a latent pattern content analysis, with categories defined as "doing well (≥ 50% on exam)" and "not doing well (< 50% on exam)" and compared to their actual OSCE exam scores, to determine the degree of alignment. RESULTS: Two hundred sixty-nine students completed the OSCE and reflection. Students had a low degree of alignment between their self-reflections and actual OSCE performance. Low alignment was overwhelmingly prevalent and significant in high-achieving students with OSCE scores of ≥90%. Most common aspects students reflected on as indicators of performance were finishing on time and communicating effectively. High-achieving students reflected on aspects such as empathy, systematic questioning, and patient teach-back as aspects of their performance. CONCLUSIONS: Student reflections on exam performance do not align with their actual performance, particularly amongst the high-achieving students. High-achieving students were more aware of the different aspects that affected their performance. To ensure that high-achieving students are aware of their strengths, educators should provide more targeted feedback mechanisms and positive reassurances to help these students become more confident in their decision-making skills.

What's been trending with OSCEs in pharmacy education over the last 20 years? A bibliometric review and content analysis.

BACKGROUND: Objective structured clinical examinations (OSCEs) remain an integral part of pharmacy education. This study aimed to characterize key researchers, areas, and themes in pharmacy education OSCEs using a bibliometric review with content analysis. METHODS: A bibliometric review was conducted on literature from over 23 years from January 2000 to May 2023. Articles focusing on any type of OSCE research in pharmacy education in both undergraduate and postgraduate sectors were included. Articles were excluded if they were not original articles or not published in English. A summative content analysis was also conducted to identify key topics. RESULTS: A total of 192 articles were included in the analysis. There were 242 institutions that contributed to the OSCE literature in pharmacy education, with the leading country being Canada. Most OSCE research came from developed countries and were descriptive studies based on single institution data. The top themes emerging from content analysis were student perceptions on OSCE station styles (n = 98), staff perception (n = 19), grade assessment of OSCEs (n = 145), interprofessional education (n = 11), standardized patients (n = 12), and rubric development and standard setting (n = 8). IMPLICATIONS: There has been a growth in virtual OSCEs, interprofessional OSCEs, and artificial intelligence OSCEs. Communication rubrics and minimizing assessor variability are still trending research areas. There is scope to conduct more research on evaluating specific types of OSCEs, when best to hold an OSCE, and comparing OSCEs to other assessments.

Using Politeness Theory to Deconstruct How Preregistrant Pharmacists Approach Disagreement in the Workplace.

OBJECTIVE: Using politeness theory, this study investigates how preregistrant pharmacists engage in workplace disputes. METHODS: Overall, 56 students participated in 2 mock job judgment scenarios. In scenario A (n = 25), the conflict was with a colleague of lower hierarchical status, while in scenario B (n = 32), the conflict was with a colleague of equal hierarchical status. Using politeness theory, responses were coded into 3 tiers: (1) engagement in the face-threatening act (FTA); (2) use of on/off-record approaches; and (3) communicative strategies. For tier 1, reasons for engaging in the FTA were also coded. RESULTS: 89% of the participants indicated they would unequivocally engage in the FTA. For scenario A, the pharmacist's roles and responsibilities (40%) and for scenario B, situational urgency/safety (65%) were the key drivers. Scenario A participants were more likely to use an on-record approach (80%) than those in scenario B (68%). Empathetic approaches (56%) were more common in scenario A, while explanatory approaches (64%) were more frequent in scenario B. CONCLUSION: This study shows that preregistrant pharmacists are aware of the principles of face-work. While the study only reflects what participants believe they would do, not their actual ability in professional disputes, it supports current research that students need simulated low-stakes opportunities to practice communication skills before entering the workplace, especially where situational urgency is present. Integration of politeness theory in workshop and feedback design could help students to link awareness to actual interaction, although the theory should be expanded to integrate urgency as a factor impacting all levels of interactional decision-making.

Opportunities and Barriers to Pediatric Antimicrobial Stewardship by Community Pharmacists.

Community Pharmacists (CPs) are easily accessible and can advocate for the appropriate use of antibiotics in children. Semi-structured interviews were conducted with 47 CPs and 46 parents/caregivers. Both groups expressed challenges to intervening when antibiotics have already been prescribed and highlighted the need for more support for CPs to make informed decisions.

Shifting to Authentic Assessments? A Systematic Review of Student Perceptions of High-Fidelity Assessments in Pharmacy.

OBJECTIVE: Despite becoming increasingly popular, there is no comprehensive review on high-fidelity assessments in pharmacy education that has a focus on the perceptions and experiences of students. This systematic review investigates the student acceptability of high-fidelity simulation for summative assessments in pharmacy education and provides recommendations regarding high-fidelity simulation practices. FINDINGS: The search yielded 37 studies in total. The articles fell into 3 distinct categories: objective structured clinical examinations (N = 25); face-to-face simulation assessments (N = 9), and augmented reality assessments (N = 3). Most high-fidelity assessments were well received with most students agreeing they had a vital role in assessing the application of clinical knowledge, even though they are stressful. Students prefer high-fidelity assessments to be face-to-face rather than online and also prefer when simulated patients whom they are unfamiliar with are used. Students also expressed a need to be well-prepared for the assessment regarding the logistics of the exam and technology use. SUMMARY: High-fidelity simulation is likely to be increasingly important in the assessment of the knowledge and skills of pharmacy students and student perception is an important factor to consider when developing such assessments. Reducing stress associated with high-fidelity assessments could include familiarizing students with task logistics or technology used prior to the assessment, using external simulated patients, and having face-to-face assessments and practice sessions.

Investigating How Preregistrant Pharmacists Respond in Job Interviews.

OBJECTIVE: To investigate what competencies and attributes preregistrant pharmacists draw upon in job interviews. METHODS: We used a virtual mock job interview assessment asking preregistrant pharmacists to apply for an entry-level pharmacist position. Data were analyzed using a team-based framework analysis using an inductive and deductive approach and mapping responses to the National Competency Standards. RESULTS: A total of 143 interview transcripts were included in the analysis. The top skills mentioned were leadership of self (98.6%) and communication and collaboration (96.5%). Despite graduating from a course with an integrated research curriculum, participants rarely reflected on research skills (31.5%) and no participant discussed any expertise in clinical topics or knowledge of specific professional services. Responses generally lacked specific skills and skills were spoken about broadly without relating to evidence/experience and were often not targeted to the job description. A proposal for educators aligned with competency standards was also developed based on the findings. CONCLUSION: Preregistrant pharmacists perceive experience within the workforce and communication and collaboration as the most desired by employers for entry-level pharmacy positions. Education and research competencies were seen as least useful to the job. There was a disconnect between skills gained in university and translation to practice. Academics could enhance the better preregistrant pharmacists' reflection of the skills and competencies they have developed employability by (1) providing portfolio management from the beginning of the course that collects evidence and maps to competencies; (2) integrating learning opportunities across all competencies; and (3) regular skills coaching/mentoring from practicing pharmacists to ensure students are aware of current needs in the job market.

Asthma during pregnancy: the experiences, concerns and views of pregnant women with asthma.

OBJECTIVE: To investigate how pregnant women manage their asthma during pregnancy and factors influencing their behavior. METHODS: In-depth interviews (telephone or face-to-face) with a purposive sample of 23 asthmatic women at various stages of pregnancy and with varying severity of asthma. RESULTS: Five major themes were discerned relating to health behavior of pregnant women with asthma. Many of the participants decreased or discontinued their asthma medications themselves and refrained from taking doses when necessary during pregnancy without consulting their doctors. Reasons behind their decisions revolved around lack of support and information about what to do, concerns about the safety of the medications, past experiences, and desire for an "all natural" pregnancy. Asthma monitoring during pregnancy was seen as a low priority for some women and their doctors. Communication between pregnant women and health professionals regarding asthma management was poor. The health behavior of pregnant women with asthma could be explained using the Health Beliefs Model. CONCLUSIONS: Pregnant women are not well supported in managing asthma during pregnancy, despite being concerned about outcomes. Interventions, education, and more support are warranted and wanted by pregnant women with asthma to optimize pregnancy and neonatal outcomes.

A comparison of the academic performance of graduate entry and undergraduate entry pharmacy students at the course exit level.

INTRODUCTION: Graduate entry (GE) pharmacy students are trained in a shorter timeframe than undergraduate entry (UE) students. This study compares the academic performance of GE and UE pharmacy students at the course exit point. METHODS: A retrospective analysis of final exam grades in written and objective structured clinical examination (OSCE) was performed between GE and UE students from three graduating cohorts. Final written examination contained clinical case study questions, whereas OSCE involved role play with simulated patients or doctors. Statistical analyses were performed by t-test and one-way analysis of variance at .05 significance level and Pearson's correlation coefficient. RESULTS: No significant difference in academic performance was seen between GE and UE groups at course exit (P > .05). There was a trend for GE students performing marginally better in OSCE than UE students. Females showed better performances in verbal communication than males. GE males showed significantly lower empathy scores than all other groups. No significant difference was seen in problem-solving scores amongst all groups. Both UE and GE groups scored significantly better in written examinations compared with OSCE. CONCLUSIONS: Graduate entry pharmacy students from accelerated learning pathway and UE students performed similarly at the course exit point, providing empirical support for non-traditional graduate entry pathway as a viable option.

Is Technology Enhanced Learning Cost-effective to Improve Skills?: The Monash Objective Structured Clinical Examination Virtual Experience.

INTRODUCTION: Objective Structured Clinical Examinations (OSCEs) are an accepted technique for evaluation of clinical competence in healthcare. However, the economic imperative requires faculty to control cost, using innovative educational strategies such as virtual simulation. The objective of this study was to evaluate the cost implications of implementing an online interactive learning module [Monash OSCE Virtual Experience (MOVE)]. METHODS: All fourth-year pharmacy students enrolled in Monash University in 2017 were provided access to MOVE. Cost-minimization analyses were performed to evaluate the cost of introducing MOVE in the pharmacy course using the smallest cohort size (Malaysia campus) of 40 students as the base case. We also determined under what circumstances MOVE would be more cost-effective, considering the different operational situations such as when student numbers increased or when the number of simulation modules created were increased. RESULTS: The overall cost of setup and implementation of MOVE in the first year of implementation among 40 students was US $94.38 per student. In comparison, the face-to-face workshop cost was US $64.14 per student. On the second year of implementation, the ongoing cost of operation of MOVE was US $32.86 per student compared with US $58.97 per student using face-to-face workshop. A net benefit using MOVE was observed after the third year of implementation. Larger savings were noted when the cohort size extends larger than 100 students. CONCLUSIONS: Monash OSCE Virtual Experience was a flexible and cost-effective approach to aid students in preparation for an OSCE and enhanced students' learning experience. The wider applicability of these findings will need to be explored in other settings.

Management of asthma in pregnant women by general practitioners: a cross sectional survey.

BACKGROUND: Poorly controlled asthma can lead to maternal and fetal complications. Despite the known risks of poorly controlled asthma during pregnancy and the need for stepping up therapy when appropriate, there are concerns that management is suboptimal in primary care.Our objective was to investigate the management of asthma during pregnancy by general practitioners providing shared maternity care. METHODS: A pre-piloted, anonymous mail survey was sent to all general practitioners (n = 842) involved in shared maternity care at six maternity hospitals in Victoria, Australia. Respondents were asked about their perceived safety of individual asthma medications during pregnancy. Approach to asthma management during pregnancy was further explored using scenarios of pregnant women with stable and deteriorating asthma and poor medication adherence. RESULTS: Inhaled corticosteroids (ICS) were perceived to be the safest and were the preferred preventive medication in first trimester (74.1%), whilst leukotriene receptor antagonists were the least preferred (2.9%). A quarter (25.8%) of respondents would stop or decrease patients' ICS doses during pregnancy, even when their asthma was well controlled by current therapy. In addition, 12.1% of respondents were not sure how to manage deteriorating asthma during pregnancy and opted to refer to another health professional. Almost half the respondents (48.9%) reported encountering medication nonadherence during pregnancy. CONCLUSION: A lack of confidence and/or knowledge among general practitioners in managing deteriorating asthma in pregnancy was observed despite a good understanding of the safety of asthma medications during pregnancy, compliance with evidence-based guidelines in the selection of preventive medications, and self reported good asthma knowledge.

Pharmacy Students' Perceptions and Performance on the Use of an Online Virtual Experience Tool for Practicing Objective Structured Clinical Examinations.

Objective. To examine pharmacy students' performance on and perceptions regarding the use of an interactive online tool for practicing to take objective structured clinical examinations (OSCEs).Methods. The Monash OSCE Virtual Experience (MOVE), an online module consisting of 20 pharmacy case scenarios with virtual patients, was piloted with final-year pharmacy students at Monash University campuses in Australia and Malaysia. A mixed methods approach that included reviewing user attempts and comparing grades, collecting student-administered questionnaires, and holding focus groups was used to examine students' perception and performance.Results. More than 99% of all students attempted at least one online case scenario in preparation for their final in-person OSCE, and 81% attempted all 20 scenarios two or more times. Ninety percent of students at the Malaysia campus and 70% of students at the Australia campus reported that MOVE was a helpful study tool for their OSCE preparation. However, a raw comparison of user attempts and OSCE grades did not find a direct correlation between online module attempts and assessment grades. Self-administered questionnaire and focus group results indicated that MOVE prepared students for targeted and time-restricted history-taking and problem-solving skills. Overall, students perceived MOVE to be a useful learning tool and a less overwhelming learning experience than were face-to-face sessions. Nevertheless, students still preferred face-to-face OSCE practice with simulated patients over online practice with virtual patients.Conclusion. The Monash OSCE Virtual Experience was perceived by our students as a flexible and useful online learning aid in preparing for their final-year OSCE However, there was no direct correlation between online practice attempts and students' exam grades.

A comparative study of quality of life, functional and bone outcomes in osteogenesis imperfecta with bisphosphonate therapy initiated in childhood or adulthood.

Bisphosphonates have been used for treatment of bone fragility disorders for over 25 years to increase bone mineral density (BMD). Anecdotally, bisphosphonate-treated Osteogenesis Imperfecta (OI) has a different trajectory to the natural history of untreated OI in terms of fracture incidence, quality of life and physical function, with minimal published evidence to support this clinical observation. This study describes functional outcomes of a cohort of adults with OI, stratified according to severity and treated with intravenous bisphosphonates as children. Reported outcomes included fracture incidence before and after puberty, mobility and BMD outcomes of this cohort. The cohort was compared to adults with OI who were never treated as children. All participants completed four questionnaires: a study specific questionnaire addressing fracture and treatment history, WHOQOL-BREF (quality of life), SF-36 (musculoskeletal function) and IPAQ (physical activity), and medical records were reviewed. Fifty-two adults with OI (80% response rate) completed the questionnaires; 33 of whom were treated with bisphosphonates in childhood. The childhood treated cohort had higher lumbar spine BMD than the adult treated cohort (z-score - 0.4 at mean age 21.3 years versus -2.1 at mean age 40.9 years; p = 0.003). Pre-pubertal fracture incidence was reduced for all severities of OI in the childhood treated cohort (less severe OI, p = 0.01; more severe OI, p < 0.001), but post-pubertal fracture incidence was higher for less severe OI (p < 0.001). In less severe OI, childhood treated individuals had higher levels of physical activity (p = 0.004) and physical functioning (p = 0.01) than adult treated individuals. Incidence of scoliosis was not different between cohorts. There were no differences in quality of life scores between the two cohorts. Improvements in BMD do not appear to influence the prevalence of scoliosis. Results suggest that treatment with bisphosphonates at an earlier age improves physical activity, particularly in less severe forms of OI but may not alter quality of life.

Developing a Framework for Objective Structured Clinical Examinations Using the Nominal Group Technique.

Objective. To use the nominal group technique to develop a framework to improve existing and develop new objective structured clinical examinations (OSCEs) within a four-year bachelor of pharmacy course. Design. Using the nominal group technique, a unique method of group interview that combines qualitative and quantitative data collection, focus groups were conducted with faculty members, practicing pharmacists, and undergraduate pharmacy students. Five draft OSCEs frameworks were suggested and participants were asked to generate new framework ideas. Assessment. Two focus groups (n=9 and n=7) generated nine extra frameworks. Two of these frameworks, one from each focus group, ranked highest (mean scores of 4.4 and 4.1 on a 5-point scale) and were similar in nature. The project team used these two frameworks to produce the final framework, which includes an OSCE in every year of the course, earlier implementation of teaching OSCEs, and the use of independent simulated patients who are not examiners. Conclusions. The new OSCE framework provides a consistent structure from course entry to exit and ensures graduates meet internship requirements.

Multidisciplinary Approach to Management of Maternal Asthma (MAMMA): a randomized controlled trial.

BACKGROUND: Uncontrolled asthma during pregnancy is associated with maternal and perinatal hazards. A pharmacist-led intervention directed at improving maternal asthma control, involving multidisciplinary care, education, and regular monitoring to help reduce these risks, was developed and evaluated. METHODS: A randomized controlled trial was carried out in the antenatal clinics of two major Australian maternity hospitals. Sixty pregnant women < 20 weeks gestation who had used asthma medications in the previous year were recruited. Participants were randomized to either an intervention or a usual care group and followed prospectively throughout pregnancy. The primary outcome was Asthma Control Questionnaire (ACQ) score. Mean changes in ACQ scores from baseline were compared between groups at 3 and 6 months to evaluate intervention efficacy. RESULTS: The ACQ score in the intervention group (n = 29) decreased by a mean ± SD of 0.46 ± 1.05 at 3 months and 0.89 ± 0.98 at 6 months. The control group (n = 29) had a mean decrease of 0.15 ± 0.63 at 3 months and 0.18 ± 0.73 at 6 months. The difference between groups, adjusting for baseline, was -0.22 (95% CI, -0.54 to 0.10) at 3 months and -0.60 (95% CI, -0.85 to -0.36) at 6 months. The difference at 6 months was statistically significant (P < .001) and clinically significant (> 0.5). No asthma-related oral corticosteroid use, hospital admissions, emergency visits, or days off from work were reported during the trial. CONCLUSIONS: A multidisciplinary model of care for asthma management involving education and regular monitoring could potentially improve maternal asthma outcomes and be widely implemented in clinical practice. TRIAL REGISTRY: Australian and New Zealand Clinical Trials Registry; No.: ACTRN12612000681853; URL: www.anzctr.org.au.