Exploring the Validity Based on Internal Structure of the Oldenburg Burnout Inventory - Medical Student (OLBI-MS).

CONSTRUCT: The study gathers validity evidence for the use of the Oldenburg Burnout Inventory - Medical Student (OLBI-MS), a 16-item scale used to measure medical student burnout. The 16 items on the OLBI-MS are split to form two subscales, disengagement and exhaustion. BACKGROUND: Medical student burnout has been empirically linked to several detrimental professional and personal consequences. In recognition of the high prevalence of medical student burnout, one recommendation has been to regularly measure burnout using standardized measures that have strong validity evidence for their intended use. The OLBI-MS, a frequently used measure of medical student burnout, was adapted from the Oldenburg Burnout Inventory (OLBI). The OLBI has been studied in many occupational settings and been found to have a two-factor solution in majority of these populations, but there is limited validity evidence available that supports the use of the OLBI-MS subscales in a medical student population. APPROACH: Two years of Association of American Medical College Year 2 Questionnaire data (n = 24,008) were used in the study for a series of exploratory and confirmatory factor analyses. The data from the first year (n = 11,586) was randomly split into a confirmatory and exploratory sample, with the data from the second year (n = 12,422) used as a secondary confirmatory sample. Because the questionnaire is administered to medical students during their second year of undergraduate medical education, we consider this a study as providing validity evidence specifically for the measure's use with that population. FINDINGS: The two-factor structure of the OLBI-MS was not empirically supported in the second year medical-student population. Several of the items had low inter-item correlations and/or moderate correlations with unexpected items. Three modified versions of the OLBI-MS were tested using subsets of the original items. Two of the modified versions were adequate statistical explanations of the relationships in the data. However, it is unclear if these revised scales appropriately measure all aspects of the construct of burnout and additional validity evidence is needed prior to their use. CONCLUSIONS: The use of the OLBI-MS is not recommended for measuring second-year medical student burnout. It is unclear if the OLBI-MS is appropriate for medical students at all, or if different measures are necessary at different stages in a medical student's professional development. Additional research is needed to either improve the OLBI-MS or use it as a foundation for a new measure.Supplemental data for this article is available online at at www.tandfonline.com/htlm .

Comparing computer adaptive testing stopping rules under the generalized partial-credit model.

An important consideration of any computer adaptive testing (CAT) program is the criterion used for ending item administration-the stopping rule, which ensures that all examinees are assessed to the same standard. Although various stopping rules exist, none of them have been compared under the generalized partial-credit model (Muraki in Applied Psychological Measurement, 16, 159-176, 1992). In this simulation study we compared the performance of three variable-length stopping rules-standard error (SE), minimum information (MI), and change in theta (CT)-both in isolation and in combination with requirements of minimum and maximum numbers of items, as well as a fixed-length stopping rule. Each stopping rule was examined under two termination criteria-one a more lenient requirement (SE = 0.35, MI = 0.56, CT = 0.05), and one more stringent (SE = 0.30, MI = 0.42, CT = 0.02). The simulation design also included content-balancing and exposure controls, aspects of CAT that have been excluded in previous research comparing variable-length stopping rules. The minimum-information stopping rule produced biased theta estimates and varied greatly in measurement quality across the theta distribution. The absolute-change-in-theta stopping rule had strong performance when paired with a lower criterion and a minimum test length. The standard error stopping rule consistently provided the best balance of measurement precision and operational efficiency and was based on the fewest number of administered items necessary to obtain accurate and precise theta estimates, particularly when it was paired with a maximum-number-of-items stopping rule.

Comparing Imputation Methods for Trait Estimation Using the Rating Scale Model.

This study examined the performance of four methods of handling missing data for discrete response options on a questionnaire: (1) ignoring the missingness (using only the observed items to estimate trait levels); (2) nearest-neighbor hot deck imputation; (3) multiple hot deck imputation; and (4) semi-parametric multiple imputation. A simulation study examining three questionnaire lengths (41-, 20-, and 10-item) crossed with three levels of missingness (10, 25, and 40 percent) was conducted to see which methods best recovered trait estimates when data were missing completely at random and the polytomous items were scored with Andrich's (1978) rating scale model. The results showed that ignoring the missingness and semi-parametric imputation best recovered known trait levels across all conditions, with the semi-parametric technique providing the most precise trait estimates. This study demonstrates the power of specific objectivity in Rasch measurement, as ignoring the missingness leads to generally unbiased trait estimates.

SHARP (SHort Answer, Rationale Provision): A New Item Format to Assess Clinical Reasoning.

PROBLEM: Many non-workplace-based assessments do not provide good evidence of a learner's problem representation or ability to provide a rationale for a clinical decision they have made. Exceptions include assessment formats that require resource-intensive administration and scoring. This article reports on research efforts toward building a scalable non-workplace-based assessment format that was specifically developed to capture evidence of a learner's ability to provide a justification for a clinical decision that they had made. APPROACH: The authors developed a 2-step item format called SHARP (SHort Answer, Rationale Provision), referring to the 2 tasks that comprise the item. In collaboration with physician-educators, the authors integrated short-answer questions into a patient medical record-based item starting in October 2021 and arrived at an innovative item format in December 2021. In this format, a test-taker interprets patient medical record data to make a clinical decision, types in their response, and pinpoints medical record details that justify their answers. In January 2022, a total of 177 fourth-year medical students, representing 20 U.S. medical schools, completed 35 SHARP items in a proof-of-concept study. OUTCOMES: Primary outcomes were item timing, difficulty, reliability, and scoring ease. There was substantial variability in item difficulty, with the average item answered correctly by 44% of students (range, 4%-76%). The estimated reliability (Cronbach α) of the set of SHARP items was 0.76 (95% CI, 0.70-0.80). Item scoring is fully automated, minimizing resource requirements. NEXT STEPS: A larger study is planned to gather additional validity evidence about the item format. This study will allow comparisons between performance on SHARP items and other examinations, the examination of group differences in performance, and possible use cases for formative assessment purposes. Cognitive interviews are also planned to better understand the thought processes of medical students as they work through the SHARP items.

"Cephalgia" or "migraine"? Solving the headache of assessing clinical reasoning using natural language processing.

In this op-ed, we discuss the advantages of leveraging natural language processing (NLP) in the assessment of clinical reasoning. Clinical reasoning is a complex competency that cannot be easily assessed using multiple-choice questions. Constructed-response assessments can more directly measure important aspects of a learner's clinical reasoning ability, but substantial resources are necessary for their use. We provide an overview of INCITE, the Intelligent Clinical Text Evaluator, a scalable NLP-based computer-assisted scoring system that was developed to measure clinical reasoning ability as assessed in the written documentation portion of the now-discontinued USMLE Step 2 Clinical Skills examination. We provide the rationale for building a computer-assisted scoring system that is aligned with the intended use of an assessment. We show how INCITE's NLP pipeline was designed with transparency and interpretability in mind, so that every score produced by the computer-assisted system could be traced back to the text segment it evaluated. We next suggest that, as a consequence of INCITE's transparency and interpretability features, the system may easily be repurposed for formative assessment of clinical reasoning. Finally, we provide the reader with the resources to consider in building their own NLP-based assessment tools.

Cut-Score Operating Function Extensions: Penalty-Based Errors and Uncertainty in Standard Settings.

We model pass/fail examinations aiming to provide a systematic tool to minimize classification errors. We use the method of cut-score operating functions to generate specific cut-scores on the basis of minimizing several important misclassification measures. The goal of this research is to examine the combined effects of a known distribution of examinee abilities and uncertainty in the standard setting on the optimal choice of the cut-score. In addition, we describe an online application that allows others to utilize the cut-score operating function for their own standard settings.

One Size Doesn't Fit All: Using Factor Analysis to Gather Validity Evidence When Using Surveys in Your Research.

Across all sciences, the quality of measurements is important. Survey measurements are only appropriate for use when researchers have validity evidence within their particular context. Yet, this step is frequently skipped or is not reported in educational research. This article briefly reviews the aspects of validity that researchers should consider when using surveys. It then focuses on factor analysis, a statistical method that can be used to collect an important type of validity evidence. Factor analysis helps researchers explore or confirm the relationships between survey items and identify the total number of dimensions represented on the survey. The essential steps to conduct and interpret a factor analysis are described. This use of factor analysis is illustrated throughout by a validation of Diekman and colleagues' goal endorsement instrument for use with first-year undergraduate science, technology, engineering, and mathematics students. We provide example data, annotated code, and output for analyses in R, an open-source programming language and software environment for statistical computing. For education researchers using surveys, understanding the theoretical and statistical underpinnings of survey validity is fundamental for implementing rigorous education research.

Effects of Discovery, Iteration, and Collaboration in Laboratory Courses on Undergraduates' Research Career Intentions Fully Mediated by Student Ownership.

Course-based undergraduate research experiences (CUREs) provide a promising avenue to attract a larger and more diverse group of students into research careers. CUREs are thought to be distinctive in offering students opportunities to make discoveries, collaborate, engage in iterative work, and develop a sense of ownership of their lab course work. Yet how these elements affect students' intentions to pursue research-related careers remain unexplored. To address this knowledge gap, we collected data on three design features thought to be distinctive of CUREs (discovery, iteration, collaboration) and on students' levels of ownership and career intentions from ∼800 undergraduates who had completed CURE or inquiry courses, including courses from the Freshman Research Initiative (FRI), which has a demonstrated positive effect on student retention in college and in science, technology, engineering, and mathematics. We used structural equation modeling to test relationships among the design features and student ownership and career intentions. We found that discovery, iteration, and collaboration had small but significant effects on students' intentions; these effects were fully mediated by student ownership. Students in FRI courses reported significantly higher levels of discovery, iteration, and ownership than students in other CUREs. FRI research courses alone had a significant effect on students' career intentions.

The Math-Biology Values Instrument: Development of a Tool to Measure Life Science Majors' Task Values of Using Math in the Context of Biology.

In response to calls to improve the quantitative training of undergraduate biology students, there have been increased efforts to better integrate math into biology curricula. One challenge of such efforts is negative student attitudes toward math, which are thought to be particularly prevalent among biology students. According to theory, students' personal values toward using math in a biological context will influence their achievement and behavioral outcomes, but a validated instrument is needed to determine this empirically. We developed the Math-Biology Values Instrument (MBVI), an 11-item college-level self--report instrument grounded in expectancy-value theory, to measure life science students' interest in using math to understand biology, the perceived usefulness of math to their life science career, and the cost of using math in biology courses. We used a process that integrates multiple forms of validity evidence to show that scores from the MBVI can be used as a valid measure of a student's value of math in the context of biology. The MBVI can be used by instructors and researchers to help identify instructional strategies that influence math-biology values and understand how math-biology values are related to students' achievement and decisions to pursue more advanced quantitative-based courses.

Do Biology Students Really Hate Math? Empirical Insights into Undergraduate Life Science Majors' Emotions about Mathematics.

Undergraduate life science majors are reputed to have negative emotions toward mathematics, yet little empirical evidence supports this. We sought to compare emotions of majors in the life sciences versus other natural sciences and math. We adapted the Attitudes toward the Subject of Chemistry Inventory to create an Attitudes toward the Subject of Mathematics Inventory (ASMI). We collected data from 359 science and math majors at two research universities and conducted a series of statistical tests that indicated that four AMSI items comprised a reasonable measure of students' emotional satisfaction with math. We then compared life science and non-life science majors and found that major had a small to moderate relationship with students' responses. Gender also had a small relationship with students' responses, while students' race, ethnicity, and year in school had no observable relationship. Using latent profile analysis, we identified three groups-students who were emotionally satisfied with math, emotionally dissatisfied with math, and neutral. These results and the emotional satisfaction with math scale should be useful for identifying differences in other undergraduate populations, determining the malleability of undergraduates' emotional satisfaction with math, and testing effects of interventions aimed at improving life science majors' attitudes toward math.

Race and Gender Differences in Undergraduate Research Mentoring Structures and Research Outcomes.

Participating in undergraduate research with mentorship from faculty may be particularly important for ensuring the persistence of women and minority students in science. Yet many life science undergraduates at research universities are mentored by graduate or postdoctoral researchers (i.e., postgraduates). We surveyed a national sample of undergraduate life science researchers about the mentoring structure of their research experiences and the outcomes they realized from participating in research. We observed two common mentoring structures: an open triad with undergraduate-postgraduate and postgraduate-faculty ties but no undergraduate-faculty tie, and a closed triad with ties among all three members. We found that men and underrepresented minority (URM) students are significantly more likely to report a direct tie to their faculty mentors (closed triad) than women, white, and Asian students. We also determined that mentoring structure was associated with differences in student outcomes. Women's mentoring structures were associated with their lower scientific identity, lower intentions to pursue a science, technology, engineering, and mathematics (STEM) PhD, and lower scholarly productivity. URM students' mentoring structures were associated with higher scientific identity, greater intentions to pursue a STEM PhD, and higher scholarly productivity. Asian students reported lower scientific identity and intentions to pursue a STEM PhD, which were unrelated to their mentoring structures.

The Laboratory Course Assessment Survey: A Tool to Measure Three Dimensions of Research-Course Design.

Course-based undergraduate research experiences (CUREs) are increasingly being offered as scalable ways to involve undergraduates in research. Yet few if any design features that make CUREs effective have been identified. We developed a 17-item survey instrument, the Laboratory Course Assessment Survey (LCAS), that measures students' perceptions of three design features of biology lab courses: 1) collaboration, 2) discovery and relevance, and 3) iteration. We assessed the psychometric properties of the LCAS using established methods for instrument design and validation. We also assessed the ability of the LCAS to differentiate between CUREs and traditional laboratory courses, and found that the discovery and relevance and iteration scales differentiated between these groups. Our results indicate that the LCAS is suited for characterizing and comparing undergraduate biology lab courses and should be useful for determining the relative importance of the three design features for achieving student outcomes.