Exploring the use of ChatGPT to analyze student course evaluation comments.

BACKGROUND: Since the release of ChatGPT, numerous positive applications for this artificial intelligence (AI) tool in higher education have emerged. Faculty can reduce workload by implementing the use of AI. While course evaluations are a common tool used across higher education, the process of identifying useful information from multiple open-ended comments is often time consuming. The purpose of this study was to explore the use of ChatGPT in analyzing course evaluation comments, including the time required to generate themes and the level of agreement between instructor-identified and AI-identified themes. METHODS: Course instructors independently analyzed open-ended student course evaluation comments. Five prompts were provided to guide the coding process. Instructors were asked to note the time required to complete the analysis, the general process they used, and how they felt during their analysis. Student comments were also analyzed through two independent Open-AI ChatGPT user accounts. Thematic analysis was used to analyze the themes generated by instructors and ChatGPT. Percent agreement between the instructor and ChatGPT themes were calculated for each prompt, along with an overall agreement statistic between the instructor and two ChatGPT themes. RESULTS: There was high agreement between the instructor and ChatGPT results. The highest agreement was for course-related topics (range 0.71-0.82) and lowest agreement was for weaknesses of the course (range 0.53-0.81). For all prompts except themes related to student experience, the two ChatGPT accounts demonstrated higher agreement with one another than with the instructors. On average, instructors took 27.50 ± 15.00 min to analyze their data (range 20-50). The ChatGPT users took 10.50 ± 1.00 min (range 10-12) and 12.50 ± 2.89 min (range 10-15) to analyze the data. In relation to reviewing and analyzing their own open-ended course evaluations, instructors reported feeling anxiety prior to the process, satisfaction during the process, and frustration related to findings. CONCLUSIONS: This study offers valuable insights into the potential of ChatGPT as a tool for analyzing open-ended student course evaluation comments in health professions education. However, it is crucial to ensure ChatGPT is used as a tool to assist with the analysis and to avoid relying solely on its outputs for conclusions.

There's no time for no stress! Exploring the relationship between pharmacy student stress and time use.

BACKGROUND: Health professions students experience significantly more stress than students 20 years ago. While prior studies have explored student time use and other studies have begun to explore factors influencing student stress, little is known about the relationship between student time use and stress. As more efforts are employed to promote student wellness and better understand student stress, it is imperative to recognize the implications of time as a finite resource. Thus, it is important to understand if and how time use relates to student stress so the two can be better managed. METHODS: A mixed methods approach exploring the challenge-hindrance stressor framework was used to collect and analyze student stress and time use. First, second, and third year pharmacy students were invited to participate. Participants completed the Perceived Stress Scale (PSS10), a week-long daily time logging activity, and daily stress questionnaire. After the week-long daily time logging, students participated in a semi-structured focus group. Descriptive statistics were used to analyze quantitative data, and inductive coding along with creation of summary reports were created for the qualitative data. RESULTS: Students reported moderate stress on the PSS10, and spending the majority of their time on activities of daily life and academic activities. Students shared that academics, co-curriculars, and working for pay increased their stress, whereas discretionary activities such as socializing and exercising alleviated stress. Finally, students reported feeling overwhelmed due to too little time to complete all necessary activities daily, including limited time to invest in discretionary activities to support their wellbeing. CONCLUSIONS: Increased stress levels among students is a concerning trend that affects students' mental health and therefore limits their ability to perform to their greatest potential. Better understanding of the relationship between time use and stress is critical to improve the quality of life of students in the health professions. These findings provide critical insight into factors contributing towards student stress that can inform curricular strategies that support wellness within health professions education.

Development and implementation of a formative instructional coaching program using the Teaching Practices Inventory within a health professions program.

BACKGROUND: A growing body of literature describes teaching practices that are positively associated with student achievement. Observing, characterizing, and providing feedback on these teaching practices is a necessary, yet significant challenge to improving teaching quality. This study describes the design, implementation, and evaluation of an instructional coaching program created to provide formative feedback to instructors based on their use of evidence-based teaching practices. METHODS: The program was designed for formative purposes utilizing an instrument adapted from the Teaching Practices Inventory. All faculty were invited to participate in the program on a voluntary basis when the program launched in Fall 2019. Program coaches included any School personnel who completed required training. Two rounds of instrument development were conducted with multiple observers and assessed using Krippendorff's Alpha. The program was evaluated using an anonymous post-session survey. RESULTS: Interrater reliability of the form improved over two rounds of piloting and no differences were found in scoring between trainees and education professionals. Seventeen observations were completed by nine coaches. Instructors indicated that feedback was practical, timely, specific, and collegial, suggesting that including student perspectives (e.g., focus groups, student course evaluations) in the coaching program might be helpful. CONCLUSIONS: Creating programs that emphasize and foster the use of evidence-based teaching are critical for health professions education. Additional research is needed to further develop coaching programs that ensure teaching practices in the health professions are optimizing student learning.

Situational judgment test validity: an exploratory model of the participant response process using cognitive and think-aloud interviews.

BACKGROUND: Situational judgment tests (SJTs) are used in health sciences education to measure examinee knowledge using case-based scenarios. Despite their popularity, there is a significant gap in the validity research on the response process that demonstrates how SJTs measure their intended constructs. A model of SJT response processes has been proposed in the literature by Robert Ployhart; however, few studies have explored and expanded the factors. The purpose of this study was to describe the factors involved in cognitive processes that examinees use as they respond to SJT items in a health professions education context. METHODS: Thirty participants-15 student pharmacists and 15 practicing pharmacists-completed a 12-item SJT designed to measure empathy. Each participant engaged in a think-aloud interview while completing the SJT, followed by a cognitive interview probing their decision-making processes. Interviews were transcribed and independently coded by three researchers to identify salient factors that contributed to response processes. RESULTS: The findings suggest SJT response processes include all four stages (comprehension, retrieval, judgment, and response selection) as initially proposed by Ployhart. The study showed factors from other published research were present, including job-specific knowledge and experiences, emotional intelligence, and test-taking. The study also identified new factors not yet described, including identifying a task objective in the scenario, assumptions about the scenario, perceptions about the scenario, and the setting of the item. CONCLUSIONS: This study provides additional SJT validity evidence by exploring participants' response processes through cognitive and think-aloud interviews. It also confirmed the four-stage model previously described by Ployhart and identified new factors that may influence SJT response processes. This study contributes to the literature with an expanded SJT response process model in a health professions education context and offers an approach to evaluate SJT response processes in the future.

Assessing student academic time use: assumptions, predictions and realities.

CONTEXT: In an era of medical education reform and increasing accountability at all levels of higher education, there is a need to understand how the time in which students engage in academic activities can inform evidence-based quality improvement of the curriculum. Time logging provides an opportunity to quantify student use of academic time and guide data-informed decision making in curriculum design. OBJECTIVES: This study aimed to evaluate faculty staff and student predictions of students' academic time use and to assess students' reported academic time use. METHODS: Graduate-level professional students engaged in a time use exercise during the first semester of Year 1 (autumn 2015) and second semester of Year 2 (spring 2017) of a redesigned curriculum launched in autumn 2015. This exercise involved three key activities: (i) prediction of time use; (ii) time logging, and (iii) reflection on time use. Key faculty staff predicted students' weekday time use in both semesters. RESULTS: Students' predictions of academic time use strongly correlated with their reported academic time use during both the first semester of Year 1 and second semester of Year 2 (r = 0.55 and r = 0.53, respectively). Faculty members' predictions of academic time use did not correlate with student academic time use during either semester. Although 63.8% of Year-1 students reported the time use exercise motivated them to change their time use, students reported spending similar amounts of time on academic activities during the first semester of Year 1 (7.8 ± 1.5 hours per weekday) and the second semester of Year 2 (7.9 ± 2.0 hours per weekday). Most students reported that the exercise had been useful and indicated that their logged time accurately reflected their actual time use. CONCLUSIONS: Although curriculum reform efforts may always require that some assumptions be made, time logging can quantify students' academic use of time. Although students predict their use of time more accurately than do faculty staff, negligible changes in students' academic time use despite reported desires to make changes indicate that students' academic time use may remain inelastic. Educators must consider these findings as they design curricula, identify academic rigour, and establish student expectations of academic time use.

Using the multiple mini interview as an assessment strategy within the first year of a health professions curriculum.

BACKGROUND: The multiple mini-interview (MMI) is a common assessment strategy used in student selection. The MMI as an assessment strategy within a health professions curriculum, however, has not been previously studied. This study describes the integration of a 5-station MMI as part of an end-of-year capstone following the first year of a health professions curriculum. The goal of the capstone MMI was to assess professional competencies of students and to offer formative feedback to prepare students for their upcoming clinical practice experiences. The purpose of this study was to evaluate the psychometric properties of an MMI integrated into a health professions curriculum. METHODS: Five capstone MMI stations were designed to each evaluate a single construct assessed by one rater. A principal component analysis (PCA) was used to evaluate the structure of the model and its ability to distinguish 5 separate constructs. A Multifaceted Rasch Measurement (MFRM) model assessed student performance and estimated the sources of measurement error attributed to 3 facets: student ability, rater stringency, and station difficulty. At the conclusion, students were surveyed about the capstone MMI experience. RESULTS: The PCA confirmed the MMI reliably assessed 5 unique constructs and performance on each station was not strongly correlated with one another. The 3-facet MFRM analysis explained 58.79% of the total variance in student scores. Specifically, 29.98% of the variance reflected student ability, 20.25% reflected rater stringency, and 8.56% reflected station difficulty. Overall, the data demonstrated an acceptable fit to the MFRM model. The majority of students agreed the MMI allowed them to effectively demonstrate their communication (80.82%), critical thinking (78.77%), and collaboration skills (70.55%). CONCLUSIONS: The MMI can be a valuable assessment strategy of professional competence within a health professions curriculum. These findings suggest the MMI is well-received by students and can produce reliable results. Future research should explore the impact of using the MMI as a strategy to monitor longitudinal competency development and inform feedback approaches.

Scoping Review of Programmatic Well-Being Interventions and Outcomes to Support Pharmacy Students.

OBJECTIVES: Recent literature highlights various well-being initiatives implemented across pharmacy programs; however, there is much heterogeneity in their implementation and limited studies assessing the impact and success of these initiatives on pharmacy students' well-being. The purpose of this scoping review was to identify the effectiveness of strategies implemented by pharmacy schools to improve the well-being of pharmacy students. FINDINGS: In total, 23 studies were included, and well-being strategies were categorized into 3 groups: organizational wellness programs, curriculum design and educational course activities, and specific relaxation or meditation activities. Strategies included yoga meditation, structured mindfulness courses, and self-directed mindfulness with digital smartphone applications. The majority of studies assessed outcomes of stress, burnout, and mindfulness, though other well-being domains such as resilience and belonging emerged. Some also assessed perceptions and acceptability of the interventions through qualitative approaches. SUMMARY: Despite the heterogeneity of interventions and outcome measures, this review provides a comprehensive scope of well-being domains, assessment tools, interventions, and approaches targeted at pharmacy students and highlights the strongest evidence for interventions on reducing stress. Pharmacy programs can benefit from approaches at both the organization level and those fostering individual accountability by exposing students to a variety of self-help well-being strategies that develop protective factors and motivate them to sustain well-being practices themselves as a shared approach. This scoping review addresses a critical gap by gaining an understanding of the current landscape of well-being initiatives and their effectiveness to better guide pharmacy programs on strategies that are most likely to improve student well-being.

Understanding pathways to health-system pharmacy administration: A journey mapping study.

PURPOSE: Pharmacy administrators play a critical role in the success of health systems, yet pathways to pharmacy administration are incompletely understood. The purpose of this study was to explore and describe pathways to becoming a health-system pharmacy administrator, including challenges experienced and resources needed. METHODS: Participants (N = 21) individually completed one or more journey maps detailing the pathway(s) to pharmacy administration and then engaged in small group discussion to identify common themes. A large group debrief was used to share and identify findings across small groups. Thematic analysis was utilized to identify common ideas shared. RESULTS: There was variation across the maps, illustrating many steps one might follow to become a pharmacy administrator. In the 23 maps completed, pathways most frequently began with pharmacy school (n = 9, 39.1%), high school (n = 6, 26.1%), or undergraduate training (n = 4, 17.4%). Most included residency training or dual master's of science in health-system pharmacy administration/residency training (n = 16, 69.6%). Common challenges included distractions, time constraints, grades/performance, financial constraints, and stress. Common resources needed included exposure to diverse career paths, pipeline and outreach support, mentoring, and leadership. Themes included debate about the definition of a pharmacy administrator, questions regarding the core competencies of the pharmacy administrator, discussion regarding ways to enter pharmacy administration, and reflection on the role of serendipity and networking. CONCLUSION: Understanding the pathways to pharmacy administration is critical to fostering a sustainable and competent workforce. Work must be done to identify strategies and develop processes for identifying, equipping, and retaining talented health-system pharmacy administrators and leaders.

Assessing Workplace Factors That Influence Burnout and Identifying Recommendations to Support Well-being in Pharmacy Faculty.

OBJECTIVE: Creating environments that promote well-being is critical as studies have shown that burnout, and thus poor well-being, results in poor health outcomes and declining workplace retention. While studies have measured the prevalence of burnout in pharmacy faculty, a gap exists regarding workplace factors that impact faculty well-being. The purpose of this study was to assess factors influencing pharmacy faculty burnout and identify recommendations to improve faculty well-being. METHODS: Full-time pharmacy faculty were invited to participate in this exploratory study. Focus groups were divided by faculty rank (ie, assistant, associate, full professor). A semistructured interview format was used to identify factors contributing to faculty burnout and fulfillment as well as solicit suggestions for strategies to improve well-being. Inductive coding was used to identify themes. RESULTS: Eleven faculty participated in 4 focus groups: 6 assistant professors in 2 focus groups (n = 3 faculty/focus group), 3 associate professors in 1 focus group, and 2 full professors in 1 focus group. Common factors influencing faculty burnout included overwhelming workload, workplace inefficiencies, and unexpected factors. Factors contributing to faculty fulfillment included impact, having a passion for work, and connecting with others. Recommendations for improving well-being included school-level well-being initiatives from leadership, encouraging work-life balance, and implementing intentional well-being initiatives and resources. CONCLUSION: Participants identified several workplace factors that contribute to burnout and faculty fulfillment. This study fills a critical gap in the literature regarding factors that influence pharmacy faculty well-being and burnout and can inform strategies for fostering faculty well-being in pharmacy education.

Initiatives to improve lung and sleep health in children: Delphi consensus from the pediatrics, pulmonary, and sleep conference.

BACKGROUND AND OBJECTIVES: The lung and sleep health of adults is heavily influenced by early factors, both genetic and environmental; therefore, optimizing respiratory health begins in childhood. Multiple barriers impede improvements in lung and sleep health for children. First, the traditional siloing between general pediatric care in the community, pediatric pulmonary and sleep subspecialty care, and the research community limits the translation of knowledge into practice. Additionally, identifying and addressing health disparities remains a challenge. The 2021 NHLBI-sponsored workshop "Defining and Promoting Pediatric Pulmonary Health (DAP3H)" was a first step in defining critical gaps in our current healthcare system in identifying and optimizing lung and sleep health in children. The workshop identified key opportunities including measuring pulmonary function in young children, sleep-focused outcomes, developing biomarkers, and longitudinal research cohorts. To expand on the work of DAP3H and continue initiatives to improve childhood lung and sleep health, the Pediatrics & Pulmonary Network: Improving Health Together conference was held in 2023. STUDY DESIGN: A modified Delphi process was applied to form consensus surrounding gaps, barriers, and action items, with the goal of identifying the most urgent opportnities for improving childhood lung and sleep health. RESULTS: Cross-cutting foundational principles were identified as: (1) Authentic Stakeholder Collaboration & Engagement, (2) Reach & Implementation in Real World Settings, (3) Understanding Current Landscape & Resources and (4) Purposeful Diversity, Equity, & Inclusion Initiatives. CONCLUSIONS: To improve lung and sleep health in children, these principles should be the foundation for research design, development, and implementation.

Pharmacy Student Stress and Time Use in Pre-Clinical and Clinical Students.

OBJECTIVE: Literature has identified high levels of stress in pharmacy students; however, more information is needed to understand how student stress relates to time use. This study explored the causes of stress and its relationship to time use in pre-clinical and clinical pharmacy students, which were compared, as previous literature has found differences in time management and stress in these groups. METHODS: In this observational, mixed methods study, pre-Advanced Pharmacy Practice Experience students completed a baseline and final stress assessment, recorded daily time use and stress levels for 1 week, and participated in a semi-structured focus group. Predetermined time use categories were used to collect and analyze time use data. Inductive coding was used to determine themes from focus group transcripts. RESULTS: Pre-clinical students were found to have overall higher baseline and final stress scores and spent more time on stress-inducing activities (ie, academics) than clinical students. Both groups reported spending more time on activities related to pharmacy school during the week and increased time on activities of daily life and discretionary activities during the weekend. Common sources of stress for both groups included academics, cocurriculars, and ineffective stress management techniques. CONCLUSIONS: Our findings support the hypothesis that time use and stress are linked. Pharmacy students acknowledged having many responsibilities and too little time to participate in stress-relieving activities. Understanding sources of student stress, including demands on students' time, and the relationship between the two is needed to support student stress management and academic success of both pre-clinical and clinical pharmacy students.

"Put me in Coach!" Engaging Faculty to Form a Winning Assessment Team.

This commentary uses the metaphor of an athletic team model to provide guidance when managing a successful assessment committee and assessment processes. To become a winning team, a joint effort must be exerted by players, coaches, and the athletic director. The topics of developing a team of productive members, creating, and implementing an assessment plan, forming a positive culture, and establishing leadership are discussed. Examples and advice are provided to assist with promoting an engaged faculty to form a well-rounded and productive assessment committee with clearly defined roles and responsibilities.

Implementation and evaluation of pharmaly houses as learning communities to promote community, wellness, and advising.

INTRODUCTION: Although medical education has demonstrated learning communities promote wellness and community while supporting advising and mentoring, its application in pharmacy is limited. This study explored faculty and student perceptions of pharmacy learning communities (PLC), including structure, member interactions, and curriculum topics. METHODS: PLCs involving first-year (PY1) student advisees, third-year (PY3) peer mentors, and faculty coaches were implemented. The PLC model included Pharmaly Houses, which were compromised of multiple Pharmalies, with each Pharmaly consisting of one faculty coach, two PY3 peer mentors, and approximately 10 PY1 advisees. PLC members were invited to participate in semi-structured focus groups, and transcripts were analyzed using inductive coding. RESULTS: Three major themes and related subthemes emerged: strengths of the PLC structure (e.g., multiple perspectives, community, networking); interactions of Pharmaly members (i.e. faculty coaches, PY3 peer mentors, PY1 advisees); and feedback on PLC curriculum topics (e.g., topic consultation, continued learning). Participants indicated the multiple perspectives provided by Pharmaly members was a strength and highlighted the PLC structure facilitated a sense of community and belonging among Pharmaly members while promoting expanded networking within the House. While the PLC expanded networks, faculty and PY1 advisees desired more one-to-one interactions. Further, students reflected on how current PLC curriculum topics (e.g., wellness) prompted continued learning, reflection, and discussion outside the House or Pharmaly meeting. CONCLUSIONS: Participants indicated PLCs can support student access to mentoring and advising, promote wellness, and foster a sense of community and belonging. The unique peer mentoring model further promoted community.

Development of a system-wide pharmacy operational weighted workload model at a large academic health system.

PURPOSE: The purpose of this study is to develop a standard operational and distributional weighted workload model that is applicable across an integrated, diverse healthcare system. This model aims to not only demonstrate the operational intensity of pharmacy practice but also to inform opportunities to decrease waste, increase efficiency, facilitate growth, and demonstrate value across operational and distributional pharmacy services. SUMMARY: Time studies were conducted at 8 hospitals within the UNC Health system to objectively measure time spent within each operational process in order to create a system-wide weighted workload model. Time study results informed the development of a system-wide weighted workload model. Data from December 29, 2019, through December 26, 2020, was then applied to this weighted workload model. With this model, acute care hospital and infusion center operational areas were compared in thousands of combinations within single operational areas and across any and all operational areas by dispense code, weighted work, and ratio of weighted work to total sum of dispenses at each site. CONCLUSION: The model successfully achieved the objective to develop a standard operational weighted workload model that is applicable across the integrated, diverse care system. This model provides a foundation for UNC Health to further productivity measurement and fills a gap in the literature by offering a novel method of developing a system-level operational workload model that can be used to evaluate and compare operational workloads across health-system sites.

Identifying health-system pharmacy operational process categories and corresponding tasks across a diverse health system using a modified Delphi process.

PURPOSE: The purpose of this study was to identify and build consensus on operational tasks that occur within a health-system pharmacy. METHODS: An expert panel of 8 individuals was invited to participate in a 3-round modified Delphi process. In the first round, the expert panel independently reviewed an initial list and provided feedback. All feedback was incorporated into the second round and then reviewed and discussed as a group. The expert panel reviewed an updated list based on feedback from the second round and reached consensus on a final list of operational processes and corresponding tasks. RESULTS: All 8 participants agreed to serve on the Delphi expert panel and reviewed an initial list of 9 process categories (hazardous intravenous [IV] medications, nonhazardous IV medications, hazardous oral medications, nonhazardous oral medications, controlled substances, total parenteral nutrition [TPN]/fluid preparations, distribution and delivery, clinical tasks, and miscellaneous operational tasks) and 44 corresponding tasks. Through the Delphi process, 72 new tasks were identified in the first round, while 34 new tasks were identified in the second round. In the third and final round, the expert panel reviewed the updated list of 9 process categories and 150 corresponding tasks, made additional edits, and reached consensus on a final list of 9 processes and 138 corresponding tasks that represented operational work within a health-system pharmacy. CONCLUSION: The modified Delphi process effectively identified operational processes and corresponding tasks occurring within hospital pharmacies in a diverse health system. This process facilitated consensus building, and the findings may inform development of an operational workload model.

Exploring the Role of Item Scenario Features on Situational Judgment Test Response Selections.

Objective. To explore pharmacists' and pharmacy students' perceptions regarding the significance of changing the features of test item scenario (eg, switching from a health care to a non-health care context) on their situational judgment test (SJT) responses.Methods. Fifteen Doctor of Pharmacy students and 15 pharmacists completed a 12-item SJT intended to measure empathy. The test included six scenarios in a health care context and six scenarios in a non-health care context; participants had to rank potential response options in order of appropriateness and no two items could be of equal rank. Qualitative data were collected individually from participants using think-aloud and cognitive interview techniques. During the cognitive interview, participants were asked how they selected their final responses for each item and whether they would have changed their answer if features of the scenario were switched (eg, changed to a non-health care context if the original item was in a health care context). Interviews were transcribed and a thematic analysis was conducted to identify the features of the scenario for each item that were perceived to impact response selections.Results. Participants stated that they would have changed their responses on average 51.3% of the time (range 20%-100%) if the features of the scenario for an item were changed. Qualitative analysis identified four pertinent scenario features that may influence response selections, which included information about the examinee, the actors in the scenario, the relationship between examinee and actors, and details about the situation. There was no discernible pattern linking scenario features to the component of empathy being measured or participant type.Conclusion. Results from this study suggest that the features of the scenario described in an SJT item could influence response selections. These features should be considered in the SJT design process and require further research to determine the extent of their impact on SJT performance.

Exploring career development through a student-directed practicum to provide individualized learning experiences.

INTRODUCTION: The primary objective was to assess the impact of an optional student-directed career development program on career and professional development between the second and third year of a doctor of pharmacy curriculum. METHODS: Students who completed the student-directed practicum (SDP) completed a nine item pre-/post-survey assessing its impact on their career development within the learning constructs of discovery, application, and reflection. Additional open-ended questions assessed why students participated in the SDP and their perceived benefits of doing so. An additional questionnaire regarding the impact of the practicum was conducted at graduation. Quantitative data were analyzed with Wilcoxon signed-rank test for the pre-/post-survey and descriptive statistics for the graduation survey while qualitative data used a two-cycle open coding process. RESULTS: Statistically significant increases were noted with those responding with "agree" or "strongly agree" to each of the nine statements in the pre-/post-survey. Students participated to further explore career options, and believed doing so resulted in enhanced career direction and future employability. The graduation survey showed those participating in the SDP had a higher rate of: postgraduate placement (86% vs. 77%), receiving ≥ five residency interview offers (81% vs. 69%), and first or second choice residency match (92% vs. 85%) compared to those who completed an internship outside of the SDP. Students commented that the SDP was a valuable part of their education and helped them take the next steps for their career. CONCLUSIONS: A student-directed career development experience can positively impact student's career direction and professional development.

Role of Knowledge and Experience in Situational Judgment Test Responses of Pharmacists and Pharmacy Students.

Objective. To describe the role of examinee knowledge and experience in situational judgment test (SJT) response processes.Methods. Thirty participants (15 students and 15 pharmacists) completed a 12-item SJT on empathy. Each participant completed a think-aloud interview followed by a cognitive interview to elicit their understanding of the items and factors that influenced their response selections. Interviews were coded to identify references to general and job-specific knowledge and experiences. Utterances were quantified to explore differences in the occurrence based on the individual item, item setting (ie, health care or non-health care setting), participant type (ie, student or pharmacists), and empathy component being assessed (ie, affective or cognitive empathy).Results. Participants made 480 references to knowledge and experiences: 45.2% were job-specific knowledge or experiences, 27.5% were general knowledge or experiences, 17.9% related to a lack of experience, and 9.4% were nondescript and could not be distinguished. There were significant differences in the reference to general and job-specific knowledge or experiences based on whether the item scenario occurred in a health care or non-health care setting and the component of empathy being assessed. Experience references often included comments about location, actors, task, similarity, specificity, and recency; knowledge references were classified by information, strategies, and skills.Conclusion. Findings from this study suggest general and job-specific knowledge and experiences influence response processes in SJTs.

A Scoping Review of the Pharmacy Curriculum Outcomes Assessment Literature.

Objective. To identify themes, gaps, and sources of evidence from the literature regarding the Pharmacy Curriculum Outcomes Assessment (PCOA) to inform practice and additional areas for research in pharmacy education.Findings. Nineteen articles describing the administration and use of PCOA were identified. Since PCOA was made a curricular requirement by the Accreditation Council of Pharmacy Education in 2016, the focus of literature related to the PCOA has shifted from administration practices (four articles published before 2016 vs two articles published since) to determining models that may predict student performance on the assessment (two vs five articles) or how the examination might be used to predict future performance (one vs seven articles), especially on the North American Pharmacist Licensing Examination. While there is a growing body of literature focused on the PCOA's utility for measuring performance, few variables have been consistently used.Summary. This review found no studies with objectives that aligned with the initial intended use of the PCOA as defined by the National Association of Boards of Pharmacy, which included tracking individual student performance throughout the curriculum, benchmarking programs against other programs, and evaluating whether a program was meeting their desired outcomes. Additionally, no consensus across the Academy was found as to the proper use of the PCOA, and a paucity of literature exists regarding how the PCOA informs schools and colleges about the effectiveness of their curriculum. There is a need for the Academy to establish a uniform application for the PCOA in pharmacy schools, assess the resources that programs need to administer this required assessment, and determine the utility of the PCOA to measure curricular effectiveness and/or student performance.

Does the ability to identify the construct on an empathy situational judgment test relate to performance? Exploring a new concept in assessment.

INTRODUCTION: The ability to identify criteria (ATIC) refers to an examinee's capacity to distinguish the construct being evaluated. Previous research indicates ATIC can be predictive of performance on some assessments. This exploratory study investigated the relationship between a participant's ability to identify criteria and their performance on an empathy situational judgment test (SJT), an assessment format used to measure social and behavioral attributes. METHODS: A 12-item empathy SJT was completed by 15 students and 15 pharmacists. During a cognitive interview, participants were asked what they believed each exam question measured. Responses were coded to determine whether participants stated "empathy" (indication of ATIC). The point-biserial correlation coefficient was calculated to explore the relationship of ATIC (correctly or not correctly identifying the item measured empathy) and performance on the SJT (total score). RESULTS: Participants identified empathy 33.3% of the time, and it was the construct most often identified. Pharmacists (27.5%) identified empathy less often than students (39.2%). When empathy was identified as the construct, it was most often reported for items in a non-healthcare setting (56.3%) rather than a healthcare setting (43.7%) and for questions targeting affective empathy (71.3%) rather than cognitive empathy (28.7%). There were no statistically significant relationships with correctly identifying the construct and performance on individual items and the overall test. CONCLUSIONS: There is inconclusive evidence that ATIC relates to performance on an empathy SJT. Additional research is needed to evaluate the role of ATIC and assessment performance to corroborate study results.