An experimental comparison of multiple-choice and short-answer questions on a high-stakes test for medical students.

Recent advances in automated scoring technology have made it practical to replace multiple-choice questions (MCQs) with short-answer questions (SAQs) in large-scale, high-stakes assessments. However, most previous research comparing these formats has used small examinee samples testing under low-stakes conditions. Additionally, previous studies have not reported on the time required to respond to the two item types. This study compares the difficulty, discrimination, and time requirements for the two formats when examinees responded as part of a large-scale, high-stakes assessment. Seventy-one MCQs were converted to SAQs. These matched items were randomly assigned to examinees completing a high-stakes assessment of internal medicine. No examinee saw the same item in both formats. Items administered in the SAQ format were generally more difficult than items in the MCQ format. The discrimination index for SAQs was modestly higher than that for MCQs and response times were substantially higher for SAQs. These results support the interchangeability of MCQs and SAQs. When it is important that the examinee generate the response rather than selecting it, SAQs may be preferred. The results relating to difficulty and discrimination reported in this paper are consistent with those of previous studies. The results on the relative time requirements for the two formats suggest that with a fixed testing time fewer SAQs can be administered, this limitation more than makes up for the higher discrimination that has been reported for SAQs. We additionally examine the extent to which increased difficulty may directly impact the discrimination of SAQs.

The Associations Between United States Medical Licensing Examination Performance and Outcomes of Patient Care.

PURPOSE: The United States Medical Licensing Examination (USMLE) comprises a series of assessments required for the licensure of U.S. MD-trained graduates as well as those who are trained internationally. Demonstration of a relationship between these examinations and outcomes of care is desirable for a process seeking to provide patients with safe and effective health care. METHOD: This was a retrospective cohort study of 196,881 hospitalizations in Pennsylvania over a 3-year period (January 1, 2017 to December 31, 2019) for 5 primary diagnoses: heart failure, acute myocardial infarction, stroke, pneumonia, or chronic obstructive pulmonary disease. The 1,765 attending physicians for these hospitalizations self-identified as family physicians or general internists. A converted score based on USMLE Step 1, Step 2 Clinical Knowledge, and Step 3 scores was available, and the outcome measures were in-hospital mortality and log length of stay (LOS). The research team controlled for characteristics of patients, hospitals, and physicians. RESULTS: For in-hospital mortality, the adjusted odds ratio was 0.94 (95% confidence interval [CI] = 0.90, 0.99; P < .02). Each standard deviation increase in the converted score was associated with a 5.51% reduction in the odds of in-hospital mortality. For log LOS, the adjusted estimate was 0.99 (95% CI = 0.98, 0.99; P < .001). Each standard deviation increase in the converted score was associated with a 1.34% reduction in log LOS. CONCLUSIONS: Better provider USMLE performance was associated with lower in-hospital mortality and shorter log LOS for patients, although the magnitude of the latter is unlikely to be of practical significance. These findings add to the body of evidence that examines the validity of the USMLE licensure program.

Near-miss narratives from the fire service: a Bayesian analysis.

BACKGROUND: In occupational safety research, narrative text analysis has been combined with coded surveillance, data to improve identification and understanding of injuries and their circumstances. Injury data give, information about incidence and the direct cause of an injury, while near-miss data enable the, identification of various hazards within an organization or industry. Further, near-miss data provide an, opportunity for surveillance and risk reduction. The National Firefighter Near-Miss Reporting System, (NFFNMRS) is a voluntary reporting system that collects narrative text data on near-miss and injurious, events within the fire and emergency services industry. In recent research, autocoding techniques, using Bayesian models have been used to categorize/code injury narratives with up to 90% accuracy, thereby reducing the amount of human effort required to manually code large datasets. Autocoding, techniques have not yet been applied to near-miss narrative data. METHODS: We manually assigned mechanism of injury codes to previously un-coded narratives from the, NFFNMRS and used this as a training set to develop two Bayesian autocoding models, Fuzzy and Naïve. We calculated sensitivity, specificity and positive predictive value for both models. We also evaluated, the effect of training set size on prediction sensitivity and compared the models' predictive ability as, related to injury outcome. We cross-validated a subset of the prediction set for accuracy of the model, predictions. RESULTS: Overall, the Fuzzy model performed better than Naïve, with a sensitivity of 0.74 compared to 0.678., Where Fuzzy and Naïve shared the same prediction, the cross-validation showed a sensitivity of 0.602., As the number of records in the training set increased, the models performed at a higher sensitivity, suggesting that both the Fuzzy and Naïve models were essentially "learning". Injury records were, predicted with greater sensitivity than near-miss records. CONCLUSION: We conclude that the application of Bayesian autocoding methods can successfully code both near misses, and injuries in longer-than-average narratives with non-specific prompts regarding injury. Such, coding allowed for the creation of two new quantitative data elements for injury outcome and injury, mechanism.

A dissonant scale: stress recognition in the SAQ.

BACKGROUND: Our previous analyses using the Stress Recognition subscale of the Safety Attitudes Questionnaire (SAQ) resulted in significant effect estimates with equally opposing explanations. We suspected construct validity issues and investigated such using our own data and correlation matrices of previous published studies. METHODS: The correlation matrices for each of the SAQ subscales from two previous studies by Speroff and Taylor were replicated and compared. The SAS Proc Factor procedure and the PRIORS = SMC option were used to perform Common Factor Analysis. RESULTS: The correlation matrices of both studies were very similar. Teamwork, Safety Climate, Job Satisfaction, Perceptions of Management and Working Conditions were well-correlated. The correlations ranged from 0.53 to 0.76. For Stress Recognition correlations ranged from -0.15 to 0.03. Common Factor Analysis confirmed the isolation of Stress Recognition. CFA returned a strong one-factor model that explained virtually all of the communal variance. Stress Recognition loaded poorly on this factor in both instances, and the CFA indicated that 96.4-100.0% of the variance associated with Stress Recognition was unique to that subscale, and not shared with the other 5 subscales. CONCLUSIONS: We conclude that the Stress Recognition subscale does not fit into the overall safety climate construct the SAQ intended to reflect. We recommend that this domain be omitted from overall safety climate scale score calculations, and clearly identified as an important yet distinct organizational construct. We suggest that this subscale be investigated for its true meaning, characterized as such, and findings conveyed to SAQ end users. We make no argument against Stress Recognition as an important organizational metric, rather we suggest that as a stand-alone construct its current packaging within the SAQ may be misleading for those intent on intervention development and evaluation in healthcare settings if they interpret Stress Recognition results as emblematic of safety climate.

Using inpatient hospital discharge data to monitor patient safety events.

The development of systematic and sustainable surveillance systems is necessary for the creation of patient safety prevention programs and the evaluation of improvement resulting from innovations. To that end, inpatient hospital discharges collected by the Pennsylvania Health Care Cost Containment Council were used to investigate patient safety events (PSEs) in Pennsylvania in 2006. PSEs were identified using external cause of injury codes (E-codes) in combination with the Agency for Healthcare Research and Quality's patient safety indicators (PSIs). Encounters with and without PSEs were compared with regard to patient age, sex, race, length of stay, and cost. Approximately 9% of all Pennsylvania inpatient discharges had a PSE in 2006. Patients with a PSE were on average older, male, and white. The average length of stay for a PSE was 3 days longer and $35 000 more expensive than a non-PSE encounter. It was concluded that E-codes and PSIs were useful tools for the surveillance of PSEs in Pennsylvania, and that administrative data from healthcare organizations provide a consistent source of standardized data related to patient encounters, creating an opportunity to describe PSEs at the population level.

What U.S. data should be used to measure the price elasticity of demand for alcohol?

This paper examines how estimates of the price elasticity of demand for beer vary with the choice of alcohol price series examined. Our most important finding is that the commonly used ACCRA price data are unlikely to reliably indicate alcohol demand elasticities-estimates obtained from this source vary drastically and unpredictably. As an alternative, researchers often use beer taxes to proxy for alcohol prices. While the estimated beer taxes elasticities are more stable, there are several problems with using taxes, including difficulties in accounting for cross-price effects. We believe that the most useful estimates reported in this paper are obtained using annual Uniform Product Code (UPC) "barcode" scanner data on grocery store alcohol prices. These estimates suggest relatively low demand elasticity, probably around -0.3, with evidence that the elasticities are considerably overstated in models that control for beer but not wine or spirits prices.