Academic conference posters: Describing visual impression in pharmacy education.

Background: Academic conference posters are a key communication before journal articles. Attention to visual attributes can enhance academic poster communication. Objective: This investigation's purpose was to create a visual impression measurement instrument, and then to describe and compare visual impression among scientific posters from an academic conference. Methods: A mixed-approach rubric was created to quickly measure visual impression of academic posters. Then, posters from a pharmacy education conference were retrospectively reviewed and scored. Visual impression was compared for traditional versus contemporary poster-formats. Various poster characteristics (poster-format, summary statement presence, abstract presence, wordiness, QR-code presence, logical sequencing, visuals) that might have impacted visual communication were coded. These characteristics were regressed onto visual impression scores. Results: Three-hundred seventy-eight posters were scored with sound inter-rater reliability. Contemporary poster-format scored significantly higher than traditional. Poster-format, abstract absence, lack of wordiness, QR-code presence, logical sequencing, and number of visuals were significant when regressed. Conclusion: Posters at one academic conference had varied visual impression. While a contemporary poster-format appeared more helpful, it was not a panacea; variation from poor through exemplary was seen with both poster-formats. Posters are not text-filled articles; displaying a combination of visuals/text clearly and concisely can help effective communication with academic posters.

Interacting within an asynchronous online interprofessional education workshop focused on social determinants of health.

BACKGROUND: Meaningful interprofessional education (IPE) involves students from at least two professions interacting to learn with, about, and from one another. Our objective was to describe a novel online approach used to create meaningful IPE within a social determinants of health (SDoH) workshop. INTERPROFESSIONAL EDUCATION ACTIVITY: This online workshop integrated four different professions' perspectives on SDoH (social-work, public-health, nursing, and pharmacy). Each six-student interprofessional team was assigned a local neighborhood. This week-long workshop had numerous activities (pre- and post-workshop quizzes, a SDoH-primer video, video self-introduction to teammates, a windshield questionnaire with two subsequent clinical cases, a post-workshop reflection, and post-workshop evaluation). For discussion, asynchronous video-based responses were used instead of traditional text-based discussion-boards. DISCUSSION: Quantitatively comparing quiz scores, students' SDoH knowledge increased with this workshop. Qualitatively from evaluations, most students found this workshop helpful and meaningful. Supporting use of video-based responses, many students' favorite aspect was interacting and collaborating within their interprofessional teams, although some students desired synchronous activities instead. Faculty facilitators confirmed that meaningful IPE interactions occurred. IMPLICATIONS: In short, students from multiple health-professions learned SDoH-content and, using video-based responses, interacted asynchronously during this online workshop. This report demonstrated one tool available to help facilitate meaningful IPE asynchronously. This asynchronous, online IPE workshop appears to be a promising format to be integrated with other in-person IPE sessions.

Using Rasch measurement for instrument rating scale refinement.

OUR SITUATION: Rasch measurement is an analysis tool that can provide validity evidence for instruments that attempt to measure student learning or other psychosocial behaviors, regardless if tools are newly created, modified, or previously developed. Rating scales are exceedingly common among psychosocial instruments and properly functioning rating scales are critical to effective measurement. Rasch measurement can help investigate this. METHODOLOGICAL LITERATURE REVIEW: Aside from using Rasch measurement from the beginning to help create rigorous new measurement instruments, researchers can also benefit from employing Rasch measurement on previously developed instruments that had not included Rasch measurement during development. This article is focused on Rasch measurement's unique analysis of rating scales. That is, Rasch measurement can uniquely help examine if and how an instrument's rating scale is functioning among newly studied respondents (who will likely differ from the originally researched sample). OUR RECOMMENDATIONS AND THEIR APPLICATION: After reviewing this article, the reader should be able to describe Rasch measurement, including how it is focused on fundamental measurement and how it differs from classical test theory and item-response theory, and reflect on situations in their own research where a Rasch measurement analysis might be helpful for generating validation evidence with a previously developed instrument. POTENTIAL IMPACT: In the end, Rasch measurement can offer a helpful, unique, rigorous approach to further developing instruments that scientifically measure, accurately and precisely.

Last Matter: Introducing infographics to Methodology Matters.

Visual summaries are gaining momentum in the health sciences literature. The Journal is introducing a new article type-Last Matter (LM). These will consist of infographics that quickly summarize and visually describe topics typically addressed in more detail within Methodology Matters reviews. The primary goal is to provide readers with clear guidance related to one or two common issues, pitfalls, or points of confusion when conducting pharmacy education scholarship. In addition to a graphical summary, a key element of each LM is a list of recommended resources for readers interested in more detailed information. The first Last Matter published in this issue summarizes key concepts related to quality in qualitative research. The Journal hopes these infographics may be helpful to for readers to comprehend and share, as well as to influence future contributions to the pharmacy education literature.

Comparing Data Submitted by Public and Private Pharmacy Schools to the Opioid-Related Activities Database.

Objective. This investigation compared similarities and differences in education on opioids and opioid abuse between public and private US schools and colleges of pharmacy.Methods. The American Association of Colleges of Pharmacy has created and maintains an Opioid-Related Activities database for schools and colleges of pharmacy in the United States. With data from 2019, a mixed-methods design was used to triangulate quantitative analysis with a concurrent qualitative analysis. After describing, the data were compared to national statistics of schools and colleges of pharmacy (ie, number, type of school, and program structure). Data from the database on opioid activity types (ie, education, service, practice, research, and advocacy) were compared between private and public institutions, both quantitatively and qualitatively. The quantitative analysis used odds-ratios (for effect-size) and chi-square (for statistical significance), while the qualitative analysis employed word clouds to explore opioid-related activities descriptors.Results. One-hundred-seven of 144 US schools and colleges of pharmacy (74% response rate) provided their opioid-related activities information to AACP. The institutions (55 private, 52 public) had entered 436 unique opioid-related activities in the AACP database. Results of the quantitative and qualitative analyses triangulated that private institutions focused more on education-opioid-related activities, while public institutions offered more activities that involving research. Magnified to education-type opioid-related activities, faculty from private institutions often focused narrowly on an education event alone, while faculty from institutions often focused more broadly on education and other aspects such as funding, research and published articles.Conclusion. Overall, private and public US schools and colleges of pharmacy widely engaged in combatting the US opioid epidemic by training student pharmacists in this important area.

A Teachable Moment Matters for: Striving for inter-rater reliability with a patient counseling assessment rubric: Lessons learned.

The Teachable Moments Matter (TMM) category of articles is designed to offer readers insight into a methodological issue identified within a companion article. Written in collaboration with one of these authors, these articles provide an opportunity to focus on a challenge experienced by the authors and, in the process, provide one or more perspectives as to how to successfully navigate this issue. The current TMM is focused on issues and pitfalls in validation. The Journal hopes this case-based approach will help highlight the nuance of a topic in context, something that might get "lost" in the entirety of a full-length article.

Introducing Teachable Moments Matters.

The Teachable Moments Matter (TMM) category of articles is designed to offer readers insight into a methodological issue identified within a companion article. Written in collaboration with one of the companion article authors, these articles provide an opportunity to focus on a challenge experienced by the authors and, in the process, provide one or more perspectives as to how to successfully navigate this issue. The Journal hopes this case-based approach will help to highlight an issue nuance in context, something that might get "lost" in the entirety of a full-length article.

Providing Validation Evidence for a Clinical-Science Module: Improving Testing Reliability with Quizzes.

DESCRIPTION OF THE PROBLEM: High-stakes decision-making should have sound validation evidence; reliability is vital towards this. A short exam may not be very reliable on its own within didactic courses, and so supplementing it with quizzes might help. But how much? This study's objective was to understand how much reliability (for the overall module-grades) could be gained by adding quiz data to traditional exam data in a clinical-science module. THE INNOVATION: In didactic coursework, quizzes are a common instructional strategy. However, individual contexts/instructors can vary quiz use formatively and/or summatively. Second-year PharmD students took a clinical-science course, wherein a 5-week module focused on cardiovascular therapeutics. Generalizability Theory (G-Theory) combined seven quizzes leading to an exam into one module-level reliability, based on a model where students were crossed with items nested in eight fixed testing occasions (mGENOVA used). Furthermore, G-Theory decision-studies were planned to illustrate changes in module-grade reliability, where the number of quiz-items and relative-weighting of quizzes were altered. CRITICAL ANALYSIS: One-hundred students took seven quizzes and one exam. Individually, the exam had 32 multiple-choice questions (MCQ) (KR-20 reliability=0.67), while quizzes had a total of 50MCQ (5-9MCQ each) with most individual quiz KR-20s less than or equal to 0.54. After combining the quizzes and exam using G-Theory, estimated reliability of module-grades was 0.73; improved from the exam alone. Doubling the quiz-weight, from the syllabus' 18% quizzes and 82% exam, increased the composite-reliability of module-grades to 0.77. Reliability of 0.80 was achieved with equal-weight for quizzes and exam. NEXT STEPS: Expectedly, more items lent to higher reliability. However, using quizzes predominantly formatively had little impact on reliability, while using quizzes more summatively (i.e., increasing their relative-weight in module-grade) improved reliability further. Thus, depending on use, quizzes can add to a course's rigor.

Initial Validation Evidence for Clinical Case Presentations by Student Pharmacists.

OBJECTIVE: There is a paucity of validation evidence for assessing clinical case-presentations by Doctor of Pharmacy (PharmD) students. Within Kane's Framework for Validation, evidence for inferences of scoring and generalization should be generated first. Thus, our objectives were to characterize and improve scoring, as well as build initial generalization evidence, in order to provide validation evidence for performance-based assessment of clinical case-presentations. DESIGN: Third-year PharmD students worked up patient-cases from a local hospital. Students orally presented and defended their therapeutic care-plan to pharmacist preceptors (evaluators) and fellow students. Evaluators scored each presentation using an 11-item instrument with a 6-point rating-scale. In addition, evaluators scored a global-item with a 4-point rating-scale. Rasch Measurement was used for scoring analysis, while Generalizability Theory was used for generalization analysis. FINDINGS: Thirty students each presented five cases that were evaluated by 15 preceptors using an 11-item instrument. Using Rasch Measurement, the 11-item instrument's 6-point rating-scale did not work; it only worked once collapsed to a 4-point rating-scale. This revised 11-item instrument also showed redundancy. Alternatively, the global-item performed reasonably on its own. Using multivariate Generalizability Theory, the g-coefficient (reliability) for the series of five case-presentations was 0.76 with the 11-item instrument, and 0.78 with the global-item. Reliability was largely dependent on multiple case-presentations and, to a lesser extent, the number of evaluators per case-presentation. CONCLUSIONS: Our pilot results confirm that scoring should be simple (scale and instrument). More specifically, the longer 11-item instrument measured but had redundancy, whereas the single global-item provided measurement over multiple case-presentations. Further, acceptable reliability can be balanced between more/fewer case-presentations and using more/fewer evaluators.

Validation Evidence using Generalizability Theory for an Objective Structured Clinical Examination.

OBJECTIVES: Performance-based assessments, including objective structured clinical examinations (OSCEs), are essential learning assessments within pharmacy education. Because important educational decisions can follow from performance-based assessment results, pharmacy colleges/schools should demonstrate acceptable rigor in validation of their learning assessments. Though G-Theory has rarely been reported in pharmacy education, it would behoove pharmacy educators to, using G-Theory, produce evidence demonstrating reliability as a part of their OSCE validation process. This investigation demonstrates the use of G-Theory to describes reliability for an OSCE, as well as to show methods for enhancement of the OSCE's reliability. INNOVATION: To evaluate practice-readiness in the semester before final-year rotations, third-year PharmD students took an OSCE. This OSCE included 14 stations over three weeks. Each week had four or five stations; one or two stations were scored by faculty-raters while three stations required students' written responses. All stations were scored 1-4. For G-Theory analyses, we used G_Strings and then mGENOVA. CRITICAL ANALYSIS: Ninety-seven students completed the OSCE; stations were scored independently. First, univariate G-Theory design of students crossed with stations nested in weeks (p × s:w) was used. The total-score g-coefficient (reliability) for this OSCE was 0.72. Variance components for test parameters were identified. Of note, students accounted for only some OSCE score variation. Second, a multivariate G-Theory design of students crossed with stations (p• × s°) was used. This further analysis revealed which week(s) were weakest for the reliability of test-scores from this learning assessment. Moreover, decision-studies showed how reliability could change depending on the number of stations each week. For a g-coefficient >0.80, seven stations per week were needed. Additionally, targets for improvements were identified. IMPLICATIONS: In test validation, evidence of reliability is vital for the inference of generalization; G-Theory provided this for our OSCE. Results indicated that the reliability of scores was mediocre and could be improved with more stations. Revision of problematic stations could help reliability as well. Within this need for more stations, one practical insight was to administer those stations over multiple weeks/occasions (instead of all stations in one occasion).

Validation Evidence from using Generalizability Theory in a Basic-Science Course: Reliability of Course-Grades from Multiple Examinations.

DESCRIPTION OF THE PROBLEM: Reliability is critical validation evidence on which to base high-stakes decision-making. Many times, one exam in a didactic course may not be acceptably reliable on its own. But how much might multiple exams add when combined together? THE INNOVATION: To improve validation evidence towards high-stakes decision-making, Generalizability Theory (G-Theory) can combine reliabilities from multiple exams into one composite-reliability (G_String IV software). Further, G-Theory decision-studies can illustrate changes in course-grade reliability, depending on the number of exams and exam-items. CRITICAL ANALYSIS: 101 first-year PharmD students took two midterm-exams and one final-exam in a pharmaceutics course. Individually, Exam1 had 50MCQ (KR-20=0.69), Exam2 had 43MCQ (KR-20=0.65), and Exam3 had 67MCQ (KR-20=0.67). After combining exam occasions using G-Theory, the composite-reliability was 0.71 for overall course-grades-better than any exam alone. Remarkably, increased numbers of exam occasions showed fewer items per exam were needed, and fewer items over all exams, to obtain an acceptable composite-reliability. Acceptable reliability could be achieved with different combinations of number of MCQs on each exam and number of exam occasions. IMPLICATIONS: G-Theory provided reliability critical validation evidence towards high-stakes decision-making. Final course-grades appeared quite reliable after combining multiple course exams-though this reliability could and should be improved. Notably, more exam occasions allowed fewer items per exam and fewer items over all the exams. Thus, one added benefit of more exam occasions for educators is developing fewer items per exam and fewer items over all exams.

Moving beyond Cronbach's Alpha and Inter-Rater Reliability: A Primer on Generalizability Theory for Pharmacy Education.

BACKGROUND: When available, empirical evidence should help guide decision-making. Following each administration of a learning assessment, data becomes available for analysis. For learning assessments, Kane's Framework for Validation can helpfully categorize evidence by inference (i.e., scoring, generalization, extrapolation, implications). Especially for test-scores used within a high-stakes setting, generalization evidence is critical. While reporting Cronbach's alpha, inter-rater reliability, and other reliability coefficients for a single measurement error are somewhat common in pharmacy education, dealing with multiple concurrent sources of measurement error within complex learning assessments is not. Performance-based assessments (e.g., OSCEs) that use raters, are inherently complex learning assessments. PRIMER: Generalizability Theory (G-Theory) can account for multiple sources of measurement error. G-Theory is a powerful tool that can provide a composite reliability (i.e., generalization evidence) for more complex learning assessments, including performance-based assessments. It can also help educators explore ways to make a learning assessment more rigorous if needed, as well as suggest ways to better allocate resources (e.g., staffing, space, fiscal). A brief review of G-Theory is discussed herein focused on pharmacy education. MOVING FORWARD: G-Theory has been common and useful in medical education, though has been used rarely in pharmacy education. Given the similarities in assessment methods among health-professions, G-Theory should prove helpful in pharmacy education as well. Within this Journal and accompanying this Idea Paper, there are multiple reports that demonstrate use of G-Theory in pharmacy education.

Use of reflective writing within interprofessional education: a mixed-methods analysis.

Nurturing student's development of interprofessional collaboration is fundamental. Assessment-For-Learning can use reflection as one technique to support students' growth. Thus, we investigated using reflective-writing within an interprofessional education (IPE) course using an exploratory mixed-methods design. In 2015, student-nurses, student-pharmacists, and student-physicians participated in an IPE course and completed self-assessments of student learning objectives (SLOs). In 2016, new cohorts of student-nurses, student-pharmacists, and student-physicians participated in the course and completed their self-assessments of SLOs; however, student-nurses and student-pharmacists also reflectively-wrote. Quantitatively comparing SLOs from 2015 cohorts with 2016 cohorts, we found that the effect-sizes (magnitude of difference) for those who reflectively-wrote (student-nurses and student-pharmacists) grew more than historical controls, whereas the effect-sizes remained unchanged for a control group (student-physicians) who did not reflectively-write. Qualitatively, initial and final reflective-writings were explored using content analysis. Initial reflective-writings helped students create a baseline for their final reflective-writings. In final reflective-writings, most students discussed their growth in understanding roles/responsibilities and communication, though limited growth was discussed for teams/teamwork and values/ethics. Thus, initial and final reflective-writings appeared useful within this IPE course. Initial reflective-writing further enhanced students' self-assessed IPE improvement and recorded students' baseline perceptions for later review, while final reflective-writings documented students' self-actualized IPE development.

Updating conceptions of validity and reliability.

Measurement validity is important when conducting research. This is as true for sociobehavioral research as for clinical research. Although the importance of validity is not new, its conceptualization has changed substantially in the past few decades. In the literature, there is a lack of consistency in how validity is presented. This may stem from a lack of awareness of the relatively recent changes in conceptualization of validity, the continued use of a historical framework in some educational texts, and/or the continued use of a historical framework in some training programs. This article presents a brief history of the conceptualization of validity including the progression from a perspective of related concepts of reliability and validity, to multiple types of validity, to a view of validity as a unitary concept supported by different types of evidence. This article closes by raising some important considerations about promoting use of a contemporary validity framework and associated terminology in current research, as well as in the education of future health-sciences researchers.

Guidance for high-stakes testing within pharmacy educational assessment.

BACKGROUND: The term "high-stakes testing" is widely used among pharmacy educators, but the term often seems misused or used incompletely. This Teachable Moments Matter (TMM) focuses on the importance of scientific-rigor when assessing learners' abilities. This article discusses high-stakes testing - what it is and what it is not. This TMM is not meant as an extensive review of the topic. IMPACT: As imperative for ethically-fair high-stakes testing, we will focus on defining and explaining high-stake testing, to include: evidence for validation, development of cut-scores, magnitudes of reliability coefficients, and other reliability measurement tools such as Generalizability Theory and Item-Response Theory. TEACHABLE MOMENT: From our perspectives as educational psychometricians, we hope that this discussion will help foster scientifically-rigorous use and reporting of high-stakes testing in pharmacy education and research.