Does a sudden shift of testing format from closed-book to open-book change the characteristics of test scores on summative final exams?

INTRODUCTION: In response to the COVID-19 pandemic, most universities in North America transitioned to online instruction and assessment in March 2020. Undergraduate pharmacy students in years one to three of two four-year entry-to-practice programs at a university in Canada were administered open-book examinations to complete their didactic winter-term courses in pharmaceutical sciences; behavioural, social, and administrative sciences; and pharmacotherapeutics. The impacts of the switch to open-book examinations on final exam characteristics are examined. METHODS: The ratios and correlations of final exam and midterm grades in 2020, where final exams were open-book, and in 2019, where finals were closed-book, were calculated and compared. RESULTS: In 2020, the ratio of final exam to midterm exam scores for five out of seven courses were significantly larger than they were in 2019. Alternatively, for all but one course, the correlations between midterm and final examination grades showed no significant difference from 2019 to 2020. CONCLUSIONS: Compared to 2019 when finals were administered in a closed-book format, a sudden shift to an open-book format for final exams in 2020 appears to be associated with the final exams becoming easier relative to midterms. However, when considering how final and midterm exam grades correlate year over year, in all but one class, there was no significant difference. These findings suggest that changing exams to be open-book may change how they can be used to inform criterion-referenced or absolute grading decisions but not norm-referenced or rank-based decisions.

Guidance for high-stakes testing within pharmacy educational assessment.

BACKGROUND: The term "high-stakes testing" is widely used among pharmacy educators, but the term often seems misused or used incompletely. This Teachable Moments Matter (TMM) focuses on the importance of scientific-rigor when assessing learners' abilities. This article discusses high-stakes testing - what it is and what it is not. This TMM is not meant as an extensive review of the topic. IMPACT: As imperative for ethically-fair high-stakes testing, we will focus on defining and explaining high-stake testing, to include: evidence for validation, development of cut-scores, magnitudes of reliability coefficients, and other reliability measurement tools such as Generalizability Theory and Item-Response Theory. TEACHABLE MOMENT: From our perspectives as educational psychometricians, we hope that this discussion will help foster scientifically-rigorous use and reporting of high-stakes testing in pharmacy education and research.

Community pharmacists' clinical reasoning: a protocol analysis.

Background There are two fundamental approaches to clinical reasoning, intuitive and analytical. These approaches have yet to be well explored to describe how pharmacists make decisions to determine medication appropriateness. Objective (1) to identify the cognitive actions (i.e., operators) that pharmacists employ when they move from one cue (i.e., concept) to another, and (2) to describe the overall clinical reasoning approach taken by pharmacists when checking for medication appropriateness. Setting Pharmacists from a chain pharmacy in Canada were invited to participate in this study. Method Data was collected in private rooms using video recordings to capture simulated patient-pharmacist interactions of a new prescription medication. A simulated case scenario was used to gather two types of verbal reports, concurrent think-aloud and structured retrospective think-aloud from pharmacists. All verbal reports were video-recorded, transcribed verbatim, and analyzed using protocol analysis. Main outcome measure Pharmacists' reasoning approaches when making medication appropriateness decisions. Results A total of 17 pharmacists participated. Pharmacists were most likely to use analytical clinical reasoning approaches when checking prescriptions and three used no clinical reasoning. When the pharmacists were asked specific questions regarding the decision-making model for pharmacy (i.e., check for indication, efficacy, safety, and adherence), 50% reported using analytical decision-making approaches, with a third of the decisions being made in hindsight. Conclusion The majority of the pharmacists followed an analytical decision-making approach to clinical reasoning. When the pharmacists were asked prompting questions about their medication-related decisions, they employed a combination of intuitive and analytical approaches. The pharmacists had the competency to check for medication appropriateness; though this knowledge was mostly restructured during the process of hindsight reasoning.

Admissions processes in North American pharmacy schools: To what extent are characteristics of practice success measured?

INTRODUCTION: The goal of preparing competent pharmacy practitioners starts with the admissions process, whereby skills and abilities can be screened to identify those individuals with a strong potential for success. This study sought to describe the admissions processes of top pharmacy schools in North America and identify the extent to which characteristics associated with professional success are measured. METHODS: An observational survey design was used to collect information to help characterize the admissions processes of 60 pharmacy schools. Online and centralized Pharmacy College Application Service data of admissions requirements were confirmed and further details were collected through a telephone or email cross-sectional survey. Data were analyzed using descriptive statistics. RESULTS: Online data indicate that every school (n = 60) requires prerequisite courses; 53 (88%) consider grade point average (GPA) and 45 (75%) consider the Pharmacy College Admission Test (PCAT). Most schools utilize interviews during admissions (94%), with 67% using a personal interview, 20% the Multiple Mini Interview (MMI), and 7% a combination of both. Of 60 schools contacted, 42% completed the survey. Most schools (92%) assess critical thinking and professional motivation; the personality traits of agreeableness and conscientiousness were considered by 83%. Personal interview was the most frequently cited method for assessing applicant characteristics. CONCLUSION: Current admissions practices in the top North American pharmacy schools are varied and continue to rely on traditional components including GPA, PCAT, and interviews. However, there is also a movement toward using standardized and validated measures of non-cognitive aspects potentially more predictive of success in pharmacy school and subsequent practice.

Examining the relationship between prerequisite grades and types of academic performance in pharmacy school.

PURPOSE: To investigate relationships between different prerequisite course grades and grade point average (GPA) of different types of academic performance in a Canadian entry-to-practice pharmacy program while controlling for important demographic variables. METHODS: Data from eight years of recently admitted students (2007-2014) were used to conduct a series of multiple linear regression analyses to assess relationships between prerequisite course GPA and eight different pharmacy school academic performance variables including: GPA in each of the first three years of the program, overall Y1-Y3 GPA, and GPA in pharmaceutical science, clinical science, clinical practice, and behavioral, social, and administrative (BSA) science courses. Demographic predictor variables including gender, mature status, and whether students attended ranked versus non-ranked universities were included as control variables. RESULTS: Analysis reveals that Biology and Biochemistry prerequisite GPA consistently predicts all eight academic performance variables while prerequisite English GPA was found to predict only clinical practice and BSA GPA. Being female and attending ranked universities were revealed as positively associated with most types of performance. Being classified as a mature student generally predicted lower academic performance. CONCLUSIONS: The consistent relationship between biology-based prerequisites and academic performance warrants consideration for increasing their weight in admissions GPA calculations. The fact that the set of prerequisites and demographic variables are weaker predictors of clinical practice and BSA performance than pharmaceutical science performance provides empirical support for recent moves to include non-traditional admission criteria.

Measuring social science concepts in pharmacy education research: From definition to item analysis of self-report instruments.

MY ISSUE: Interpreting results from quantitative research can be difficult when measures of concepts are constructed poorly, something that can limit measurement validity. METHODS: Social science steps for defining concepts, guidelines for limiting construct-irrelevant variance when writing self-report questions, and techniques for conducting basic item analysis are reviewed to inform the design of instruments to measure social science concepts in pharmacy education research. MY RECOMMENDATIONS AND THEIR APPLICATIONS: Based on a review of the literature, four main recommendations emerge: These include: (1) employ a systematic process of conceptualization to derive nominal definitions; (2) write exact and detailed operational definitions for each concept, (3) when creating self-report questionnaires, write statements and select scales to avoid introducing construct-irrelevant variance (CIV); and (4) use basic item analysis results to inform instrument revision. POTENTIAL IMPACT: Employing recommendations that emerge from this review will strengthen arguments to support measurement validity which in turn will support the defensibility of study finding interpretations. An example from pharmacy education research is used to contextualize the concepts introduced.

Patient Satisfaction with Pharmacist-Led Collaborative Follow-Up Care in an Ambulatory Rheumatology Clinic.

OBJECTIVES: Patient satisfaction is known to increase with pharmacist intervention in general outpatient clinics and with nurse-led care in rheumatology clinics. The aim of the present study was to describe and compare patient satisfaction with two different types of care: a pharmacist physician collaborative model and a traditional physician model in a rheumatology clinic setting. METHODS: A cross-sectional survey of inflammatory arthritis patients seen during a follow-up visit in Edmonton, Alberta, Canada, was conducted over a ten-week period. Patient satisfaction was measured using a modified version of the validated Leeds Satisfaction Questionnaire, which uses a five-point Likert scale to measure six dimensions of satisfaction, and compared between the collaborative care and traditional physician models. RESULTS: A total of 62 patients completed the questionnaire (21 collaborative care and 41 traditional physician model). The average age of respondents was 52 years and the majority were female. The mean score for satisfaction across the six dimensions was 4.56 in the collaborative care group and 4.30 in the traditional physician group (p = 0.02). Patient satisfaction in the collaborative care group was consistently higher across all dimensions. No difference was noted between participants seen for the first time compared with those seen two or more times by the pharmacist. CONCLUSIONS: A collaborative care model can exceed the already high expectations for care of patients with inflammatory arthritis. Our findings support the role of pharmacists using a collaborative care approach to care for patients in rheumatology clinics.

Improving ability measurement in surveys by following the principles of IRT: The Wordsum vocabulary test in the General Social Survey.

Survey researchers often administer batteries of questions to measure respondents' abilities, but these batteries are not always designed in keeping with the principles of optimal test construction. This paper illustrates one instance in which following these principles can improve a measurement tool used widely in the social and behavioral sciences: the GSS's vocabulary test called "Wordsum". This ten-item test is composed of very difficult items and very easy items, and item response theory (IRT) suggests that the omission of moderately difficult items is likely to have handicapped Wordsum's effectiveness. Analyses of data from national samples of thousands of American adults show that after adding four moderately difficult items to create a 14-item battery, "Wordsumplus" (1) outperformed the original battery in terms of quality indicators suggested by classical test theory; (2) reduced the standard error of IRT ability estimates in the middle of the latent ability dimension; and (3) exhibited higher concurrent validity. These findings show how to improve Wordsum and suggest that analysts should use a score based on all 14 items instead of using the summary score provided by the GSS, which is based on only the original 10 items. These results also show more generally how surveys measuring abilities (and other constructs) can benefit from careful application of insights from the contemporary educational testing literature.