Nurturing Untapped Integration Expertise of MS4 Assessment Writers.

Creating original, integrated multiple-choice questions (MCQs) is time-consuming and onerous for basic science and clinical faculty. We demonstrate that medical students are co-experts to overcome assessment challenges of the faculty. We recruited, trained, and motivated medical students to write 10,000 high-quality MCQs for use in the foundational courses of medical education. These students were ideal because they possessed integrated knowledge (basic sciences and clinical experience). We taught them how to write high-quality MCQs using a writing template and continuous monitoring and support by an item bank curator. The students themselves also benefitted personally and pedagogically from the experience.

Assessing clinical competence: a multitrait-multimethod matrix construct validity study.

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Race and Gender Bias in Clerkship Grading.

Phenomenon: Existing literature, as well as anecdotal evidence, suggests that tiered clinical grading systems may display systematic demographic biases. This study aimed to investigate these potential inequities in-depth. Specifically, this study attempted to address the following gaps in the literature: (1) studying grades actually assigned to students (as opposed to self-reported ones), (2) using longitudinal data over an 8-year period, providing stability of data, (3) analyzing three important, potentially confounding covariates, (4) using a comprehensive multivariate statistical design, and (5) investigating not just the main effects of gender and race, but also their potential interaction. Approach: Participants included 1,905 graduates (985 women, 51.7%) who received the Doctor of Medicine degree between 2014 and 2021. Most of the participants were white (n = 1,310, 68.8%) and about one-fifth were nonwhite (n = 397, 20.8%). There were no reported race data for 10.4% (n = 198). To explore potential differential grading, a two-way multivariate analysis of covariance was employed to examine the impact of race and gender on grades in eight required clerkships, adjusting for prior academic performance. Findings: There were two significant main effects, race and gender, but no interaction effect between gender and race. Women received higher grades on average on all eight clerkships, and white students received higher grades on average on four of the eight clerkships (Medicine, Pediatrics, Surgery, Obstetrics/Gynecology). These relationships held even when accounting for prior performance covariates. Insights: These findings provide additional evidence that tiered grading systems may be subject to systematic demographic biases. It is difficult to tease apart the contributions of various factors to the observed differences in gender and race on clerkship grades, and the interactions that produce these biases may be quite complex. The simplest solution to cut through the tangled web of grading biases may be to move away from a tiered grading system altogether.

Stability and Differences in Empathy Between Men and Women Medical Students: a Panel Design Study.

PHENOMENON: Empathy is central to the physician-patient relationship, and affects clinical outcomes. There is uncertainty about the stability of empathy in medical students over the course of medical school, as well as differences in empathy between men and women. APPROACH: A panel study design was used to follow first year through fourth year medical students (MS1-4) during the 2018-2019 school year. Empathy was measured using the interpersonal reactivity index (IRI), a self-report scale that separates empathy into a cognitive perspective taking (PT) and affective empathic concern (EC) component. FINDINGS: A total of 631 (359 women and 272 men) from 970 students (65% response rate) responded to a baseline survey, and a total of 536 students (300 women and 236 men) from 970 students (55% response rate) responded to surveys throughout the year. At baseline, women had significantly higher EC scores than men (p < 0.0001), with no significant PT difference between men and women (p > 0.05). These differences were stable for all MS cohorts. INSIGHTS: Women had self-reported higher affective empathy (EC component) than men, while there were no differences in cognitive empathy (PT component). We discuss these data in the context of defining gender vs. sex, socialized gender stereotypes, and implications for future research.

Impact of the stringency of lockdown measures on covid-19: A theoretical model of a pandemic.

BACKGROUND: How effective have lockdowns been at reducing the covid-19 infection and mortality rates? Lockdowns influence contact among persons within or between populations including restricting travel, closing schools, prohibiting public gatherings, requiring workplace closures, all designed to slow the contagion of the virus. The purpose of the present study was to assess the impact of lockdown measures on the spread of covid-19 and test a theoretical model of the covid-19 pandemic employing structural equation modelling. METHODS: Lockdown variables, population demographics, mortality rates, infection rates, and health were obtained for eight countries: Austria, Belgium, France, Germany, Italy, Netherlands, Spain, and the United Kingdom. The dataset, owid-covid-data.csv, was downloaded on 06/01/2020 from: https://github.com/owid/covid-19-data/tree/master/public/data. Infection spread and mortality data were depicted as logistic growth and analyzed with stepwise multiple regression. The overall structure of the covid-19 data was explored through factor analyses leading to a theoretical model that was tested using latent variable path analysis. RESULTS: Multiple regression indicated that the time from lockdown had a small but significant effect (ß = 0.112, p< 0.01) on reducing the number of cases per million. The stringency index produced the most important effect for mortality and infection rates (ß = 0.588,ß = 0.702, ß = 0.518, ß = 0.681; p< 0.01). Exploratory and confirmatory analyses resulted in meaningful and cohesive latent variables: 1) Mortality, 2) Infection Spread, 3) Pop Health Risk, and 4) Health Vulnerability (Comparative Fit Index = 0.91; Standardized Root Mean Square Residual = 0.08). DISCUSSION: The stringency index had a large impact on the growth of covid-19 infection and mortality rates as did percentage of population aged over 65, median age, per capita GDP, diabetes prevalence, cardiovascular death rates, and ICU hospital beds per 100K. The overall Latent Variable Path Analysis is theoretically meaningful and coherent with acceptable fit indices as a model of the covid-19 pandemic.

Validity Evidence for Assessing Entrustable Professional Activities During Undergraduate Medical Education.

PURPOSE: To explore validity evidence for the use of entrustable professional activities (EPAs) as an assessment framework in medical education. METHOD: Formative assessments on the 13 Core EPAs for entering residency were collected for 4 cohorts of students over a 9- to 12-month longitudinal integrated clerkship as part of the Education in Pediatrics Across the Continuum pilot at the University of Minnesota Medical School. The students requested assessments from clinical supervisors based on direct observation while engaging in patient care together. Based on each observation, the faculty member rated the student on a 9-point scale corresponding to levels of supervision required. Six EPAs were included in the present analyses. Student ratings were depicted as curves describing their performance over time; regression models were employed to fit the curves. The unit of analyses for the learning curves was observations rather than individual students. RESULTS: (1) Frequent assessments on EPAs provided a developmental picture of competence consistent with the negative exponential learning curve theory; (2) This finding was true across a variety of EPAs and across students; and (3) The time to attain the threshold level of performance on the EPA for entrustment varied by student and EPA. CONCLUSIONS: The results provide validity evidence for an EPA-based program of assessment. Students assessed using multiple observations performing the Core EPAs for entering residency demonstrate classic developmental progression toward the desired level of competence resulting in entrustment decisions. Future work with larger data samples will allow further psychometric analyses of assessment of EPAs.

Assessing anaesthesiology and intensive care specialty physicians: An Italian language multisource feedback system.

BACKGROUND: Physician professionalism, including anaesthesiologists and intensive care doctors, should be continuously assessed during training and subsequent clinical practice. Multi-source feedback (MSF) is an assessment system in which healthcare professionals are assessed on several constructs (e.g., communication, professionalism, etc.) by multiple people (medical colleagues, coworkers, patients, self) in their sphere of influence. MSF has gained widespread acceptance for both formative and summative assessment of professionalism for reflecting on how to improve clinical practice. METHODS: Instrument development and psychometric analysis (feasibility, reliability, construct validity via exploratory factor analysis) for MSF questionnaires in a postgraduate specialty training in Anaesthesiology and intensive care in Italy. Sixty-four residents at the Università del Piemonte Orientale (Italy) Anesthesiology Residency Program. Main outcomes assessed were: development and psychometric testing of 4 questionnaires: self, medical colleague, coworker and patient assessment. RESULTS: Overall 605 medical colleague questionnaires (mean of 9.3 ±1.9) and 543 coworker surveys (mean 8.4 ±1.4) were collected providing high mean ratings for all items (> 4.0 /5.0). The self-assessment item mean score ranged from 3.1 to 4.3. Patient questionnaires (n = 308) were returned from 31 residents (40%; mean 9.9 ± 6.2). Three items had high percentages of "unable to assess" (> 15%) in coworker questionnaires. Factor analyses resulted in a two-factor solution: clinical management with leadership and accountability accounting for at least 75% of the total variance for the medical colleague and coworker's survey with high internal consistency reliability (Cronbach's α > 0.9). Patient's questionnaires had a low return rate, a limited exploratory analysis was performed. CONCLUSIONS: We provide a feasible and reliable Italian language MSF instrument with evidence of construct validity for the self, coworkers and medical colleague. Patient feedback was difficult to collect in our setting.

A Study of the Validity of the New MCAT Exam.

PURPOSE: To conduct a study of the validity of the new Medical College Admission Test (MCAT). METHOD: Deidentified data for first- and second-year medical students (185 women, 54.3%; 156 men, 45.7%) who matriculated in 2016 and 2017 to the University of Minnesota Medical School-Twin Cities were included. Of those students, 220 (64.5%) had taken the new MCAT exam and 182 (53.4%) had taken the old MCAT exam (61 [17.9%] had taken both). The authors calculated descriptive statistics and Pearson product moment correlations (r) between new and old MCAT section scores. They conducteda regression analysis of MCAT section scores with Step 1 scores and with preclerkship course performance. They also conducted an exploratory factor analysis (principal component analysis with varimax rotation) of MCAT scores, undergraduate grade point average, Step 1 scores, and course performance. RESULTS: The new MCAT exam section mean score percentiles ranged from 72 to 78 (mean composite score percentile of 80). The old MCAT exam section mean score percentiles ranged from 84 to 88 (mean composite score percentile of 83). The pattern of correlations among and between new and old MCAT exam section scores (range of r: 0.03-0.67; P < .01) provided evidence of both divergent and convergent validities. Backward multiple regression of new MCAT exam section scores and Step 1 scores resulted in a multiple R of .440; the same analysis with Human Behavior course performance as the dependent variable provided a similar solution with the expected sections of the new MCAT exam (multiple R = .502). The factor analysis resulted in 4 cohesive, theoretically meaningful factors: biomedical knowledge, basic science concepts, cognitive reasoning, and general achievement. CONCLUSIONS: This study provided empirical evidence of multiple types of validity for the new MCAT exam.

Cross-Cultural Validity Study of a Medical Education Leadership Competencies Instrument in Latin American Physicians: A Multinational Study.

PURPOSE: Physicians rarely receive formal training in leadership skills. Çitaku and colleagues have identified a set of leadership competencies (LCs) providing validity evidence in North American (NA) and European Union (EU) medical education institutions. We aim to apply this same survey to a sample of Latin American (LA) medical leaders from the oncology community and related areas, compare the results with those of the previous survey, and perform subgroup analyses within the LA cohort. METHODS: The survey was sent to nearly 8,000 physicians of participating professional organizations. In addition to the 63 questions, we also collected data on the type of institution, country, specialty, sex, age, years of experience in oncology, and leadership position. RESULTS: The 217 LA respondents placed the highest value on task management competencies (91.37% reported these as important or very important v 87.0% of NA/EU respondents; P < .0001), followed by self-management (87.45% of LA respondents v 87.55% of NA/EU respondents; P = not significant [NS]), social responsibility (86.83% of LA respondents v 87.48% of NA/EU respondents; P = NS), innovation (86.69% of LA respondents v 85.31% of NA/EU respondents; P = NS), and leading others (83.31% of LA respondents v 84.71% of NA/EU respondents; P = NS). Social responsibility, which was first in importance in the NA/EU survey, was only third in the LA survey. Subgroup analyses showed significant variations in the ratings of specific LCs within the LA population. CONCLUSION: LCs valued by LA leaders somewhat differ from those valued by their NA and EU counterparts, implying that cultural aspects might influence the perception of desired LCs. We also detected variations in the responses within the LA population. Our data indicate that current physician leadership training programs should be tailored to suit specific needs and cultural aspects of each region. Further validity studies of this instrument with other samples and cultures are warranted.

Does Clerkship Rotation Sequence Affect Performance on National Board of Medical Examiners (NBME) Clinical Subject Examinations and United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) Examination?

PURPOSE: To investigate whether starting the clerkship year in family medicine (FM), internal medicine (IM), pediatrics, or surgery influences NBME shelf and USMLE Step 2 CK examination performance. METHODS: USMLE Step 1, Step 2 CK, and shelf examination scores for FM, IM, pediatrics, and surgery were collected. Sequences were selected on the following assignment criteria: rotation 1 (either FM or IM), rotation 5 (pediatrics), rotation 8 (surgery), rotation 1 (either pediatrics or surgery), and rotation 8 (IM). Multivariate analysis of covariance, with Step 1 as the covariate, was used to investigate rotation sequence on examination performance. RESULTS: Wilks's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in FM or IM) on the pediatrics, surgery, and Step 2 CK examinations (Λ = .95, F[3,51] = .93, p ≤ .432). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .488, F[3,51] = 17.827, p ≤ .001), indicating the two groups differ on Step 1 performance. Wilk's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in pediatrics or surgery) on the IM and Step 2 CK results (Λ = .925, F[2,75] = 3.036, p ≤ .054). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .309, F[2,75] = 83.915, p ≤ .001) indicating that the two groups differ on Step 1 performance. CONCLUSION: Starting the clerkship year in FM, IM, pediatrics, or surgery does not influence subsequent performance on shelf examinations or on Step 2 CK.

A longitudinal study in learning preferences and academic performance in first year medical school.

This is a longitudinal study of first year medical students that investigates the relationship between the pattern change of the learning preferences and academic performance. Using the visual, auditory, reading-writing, and kinesthetic inventory at the beginning of the first and second year for the same class, it was found that within the first year, 36% of the class remained unimodal (single) modality learners (SS), 14% changed from unimodal to multimodality learners (SM), 27% changed from multimodality to unimodal modality learners (MS) and 21% remained as multimodality learners (MM). Among the academic performance through subsequent didactic blocks from Clinical Anatomy, Cell and Subcellular Processes to Medical Neuroscience during first year, the SM group made more significant improvement compared to the SS group. Semi-structured interview results from the SM group showed that students made this transition between the Clinical Anatomy course and the middle of the Medical Neuroscience course, in an effort to improve their performance. This study suggests that the transition from unimodal to multimodality learning among academically struggling students improved their academic performance in the first year of medical school. Therefore, this may be considered as part of academic advising tools for struggling students to improve their academic performances. Anat Sci Educ 11: 488-495. © 2017 American Association of Anatomists.

How do physicians become medical experts? A test of three competing theories: distinct domains, independent influence and encapsulation models.

The distinction between basic sciences and clinical knowledge which has led to a theoretical debate on how medical expertise is developed has implications for medical school and lifelong medical education. This longitudinal, population based observational study was conducted to test the fit of three theories-knowledge encapsulation, independent influence, distinct domains-of the development of medical expertise employing structural equation modelling. Data were collected from 548 physicians (292 men-53.3%; 256 women-46.7%; mean age = 24.2 years on admission) who had graduated from medical school 2009-2014. They included (1) Admissions data of undergraduate grade point average and Medical College Admission Test sub-test scores, (2) Course performance data from years 1, 2, and 3 of medical school, and (3) Performance on the NBME exams (i.e., Step 1, Step 2 CK, and Step 3). Statistical fit indices (Goodness of Fit Index-GFI; standardized root mean squared residual-SRMR; root mean squared error of approximation-RSMEA) and comparative fit [Formula: see text] of three theories of cognitive development of medical expertise were used to assess model fit. There is support for the knowledge encapsulation three factor model of clinical competency (GFI = 0.973, SRMR = 0.043, RSMEA = 0.063) which had superior fit indices to both the independent influence and distinct domains theories ([Formula: see text] vs [Formula: see text] [[Formula: see text]] vs [Formula: see text] [[Formula: see text]], respectively). The findings support a theory where basic sciences and medical aptitude are direct, correlated influences on clinical competency that encapsulates basic knowledge.

Identifying a parsimonious model for predicting academic achievement in undergraduate medical education: A confirmatory factor analysis.

OBJECTIVES: This study was conducted to adduce evidence of validity for admissions tests and processes and for identifying a parsimonious model that predicts students' academic achievement in Medical College. METHODS: Psychometric study done on admission data and assessment scores for five years of medical studies at Aga Khan University Medical College, Pakistan using confirmatory factor analysis (CFA) and structured equation modeling (SEM). Sample included 276 medical students admitted in 2003, 2004 and 2005. RESULTS: The SEM supported the existence of covariance between verbal reasoning, science and clinical knowledge for predicting achievement in medical school employing Maximum Likelihood (ML) estimations (n=112). Fit indices: χ2 (21) = 59.70, p =<.0001; CFI=.873; RMSEA = 0.129; SRMR = 0.093. CONCLUSIONS: This study shows that in addition to biology and chemistry which have been traditionally used as major criteria for admission to medical colleges in Pakistan; mathematics has proven to be a better predictor for higher achievements in medical college.

The effectiveness of physical models in teaching anatomy: a meta-analysis of comparative studies.

There are various educational methods used in anatomy teaching. While three dimensional (3D) visualization technologies are gaining ground due to their ever-increasing realism, reports investigating physical models as a low-cost 3D traditional method are still the subject of considerable interest. The aim of this meta-analysis is to quantitatively assess the effectiveness of such models based on comparative studies. Eight studies (7 randomized trials; 1 quasi-experimental) including 16 comparison arms and 820 learners met the inclusion criteria. Primary outcomes were defined as factual, spatial and overall percentage scores. The meta-analytical results are: educational methods using physical models yielded significantly better results when compared to all other educational methods for the overall knowledge outcome (p < 0.001) and for spatial knowledge acquisition (p < 0.001). Significantly better results were also found with regard to the long-retention knowledge outcome (p < 0.01). No significance was found for the factual knowledge acquisition outcome. The evidence in the present systematic review was found to have high internal validity and at least an acceptable strength. In conclusion, physical anatomical models offer a promising tool for teaching gross anatomy in 3D representation due to their easy accessibility and educational effectiveness. Such models could be a practical tool to bring up the learners' level of gross anatomy knowledge at low cost.

Clinical teaching as part of continuing professional development: Does teaching enhance clinical performance?

INTRODUCTION: Physicians identify teaching as a factor that enhances performance, although existing data to support this relationship is limited. PURPOSE: To determine whether there were differences in clinical performance scores as assessed through multisource feedback (MSF) data based on clinical teaching. METHODS: MSF data for 1831 family physicians, 1510 medical specialists, and 542 surgeons were collected from physicians' medical colleagues, co-workers (e.g., nurses and pharmacists), and patients and examined in relation to information about physician teaching activities including percentage of time spent teaching during patient care and academic appointment. Multivariate analysis of variance, partial eta squared effect sizes, and Tukey's HSD post hoc comparisons were used to determine between group differences in total MSF mean and subscale mean performance scores by teaching and academic appointment data. RESULTS: Higher clinical performance scores were associated with holding any academic appointment and generally with any time teaching versus no teaching during patient care. This was most evident for data from medical colleagues, where these differences existed across all specialty groups. CONCLUSION: More involvement in teaching was associated with higher clinical performance ratings from medical colleagues and co-workers. These results may support promoting teaching as a method to enhance and maintain high-quality clinical performance.

Developing Professionalism via Multisource Feedback in Team-Based Learning.

UNLABELLED: CGEA 2015 CONFERENCE ABSTRACT (EDITED). A Novel Approach to Assessing Professionalism in Preclinical Medical Students Using Paired Self- and Peer Evaluations. Amanda R. Emke, Steven Cheng, and Carolyn Dufault. CONSTRUCT: This study sought to assess the professionalism of 2nd-year medical students in the context of team-based learning. BACKGROUND: Professionalism is an important attribute for physicians and a core competency throughout medical education. Preclinical training often focuses on individual knowledge acquisition with students working only indirectly with faculty assessors. As such, the assessment of professionalism in preclinical training continues to present challenges. We propose a novel approach to preclinical assessment of medical student professionalism to address these challenges. APPROACH: Second-year medical students completed self- and peer assessments of professionalism in two courses (Pediatrics and Renal/Genitourinary Diseases) following a series of team-based learning exercises. Assessments were composed of nearly identical 9-point rating scales. Correlational analysis and linear regression were used to examine the associations between self- and peer assessments and the effects of predictor variables. Four subgroups were formed based on deviation from the median ratings, and logistic regression was used to assess stability of subgroup membership over time. A missing data analysis was conducted to examine differences between average peer-assessment scores as a function of selective nonparticipation. RESULTS: There was a significant positive correlation (r = .62, p < .0001) between self-assessments completed alone and those completed at the time of peer assessment. There was also a significant positive correlation between average peer-assessment and self-assessment alone (r = .19, p < .0002) and self-assessment at the time of peer assessment (r = .27, p < .0001). Logistic regression revealed that subgroup membership was stable across measurement at two time points (T1 and T2) for all groups, except for members of the high self-assessment/low peer assessment at T1, who were significantly more likely to move to a new group at T2, χ(2)(3, N = 129) = 7.80, p < .05. Linear regression revealed that self-assessment alone and course were significant predictors of self-assessment at the time of peer assessment (Fself_alone = 144.74, p < .01 and Fcourse = 4.70, p < .05), whereas average peer rating, stage (T1, T2) and academic year (13-14, 14-15) were not. Linear regression also revealed that students who completed both self-assessments had significantly higher average peer assessment ratings (average peer rating in students with both self-assessments = 8.42, no self-assessments = 8.10, self_at_peer = 8.37, self_alone = 8.28) compared to students who completed one or no self-assessments (F = 5.34, p < .01). CONCLUSIONS: When used as a professionalism assessment within team-based learning, stand-alone and simultaneous peer and self-assessments are highly correlated within individuals across different courses. However, although self-assessment alone is a significant predictor of self-assessment made at the time of assessing one's peers, average peer assessment does not predict self-assessment. To explore this lack of predictive power, we classified students into four subgroups based on relative deviation from median peer and self-assessment scores. Group membership was found to be stable for all groups except for those initially sorted into the high self-assessment/low peer assessment subgroup. Members of this subgroup tended to move into the low self-assessment/low peer assessment group at T2, suggesting they became more accurate at self-assessing over time. A small group of individuals remained in the group that consistently rated themselves highly while their peers rated them poorly. Future studies will track these students to see if similar deviations from accurate professional self-assessment persist into the clinical years. In addition, given that students who fail to perform self-assessments had significantly lower peer assessment scores than their counterparts who completed self-assessments in this study, these students may also be at risk for similar professionalism concerns in the clinical years; follow-up studies will examine this possibility.

Standard Setting Methods for Pass/Fail Decisions on High-Stakes Objective Structured Clinical Examinations: A Validity Study.

UNLABELLED: CONSTRUCT: Authentic standard setting methods will demonstrate high convergent validity evidence of their outcomes, that is, cutoff scores and pass/fail decisions, with most other methods when compared with each other. BACKGROUND: The objective structured clinical examination (OSCE) was established for valid, reliable, and objective assessment of clinical skills in health professions education. Various standard setting methods have been proposed to identify objective, reliable, and valid cutoff scores on OSCEs. These methods may identify different cutoff scores for the same examinations. Identification of valid and reliable cutoff scores for OSCEs remains an important issue and a challenge. APPROACH: Thirty OSCE stations administered at least twice in the years 2010-2012 to 393 medical students in Years 2 and 3 at Aga Khan University are included. Psychometric properties of the scores are determined. Cutoff scores and pass/fail decisions of Wijnen, Cohen, Mean-1.5SD, Mean-1SD, Angoff, borderline group and borderline regression (BL-R) methods are compared with each other and with three variants of cluster analysis using repeated measures analysis of variance and Cohen's kappa. RESULTS: The mean psychometric indices on the 30 OSCE stations are reliability coefficient = 0.76 (SD = 0.12); standard error of measurement = 5.66 (SD = 1.38); coefficient of determination = 0.47 (SD = 0.19), and intergrade discrimination = 7.19 (SD = 1.89). BL-R and Wijnen methods show the highest convergent validity evidence among other methods on the defined criteria. Angoff and Mean-1.5SD demonstrated least convergent validity evidence. The three cluster variants showed substantial convergent validity with borderline methods. CONCLUSIONS: Although there was a high level of convergent validity of Wijnen method, it lacks the theoretical strength to be used for competency-based assessments. The BL-R method is found to show the highest convergent validity evidences for OSCEs with other standard setting methods used in the present study. We also found that cluster analysis using mean method can be used for quality assurance of borderline methods. These findings should be further confirmed by studies in other settings.

A meta-analysis of the educational effectiveness of three-dimensional visualization technologies in teaching anatomy.

Many medical graduates are deficient in anatomy knowledge and perhaps below the standards for safe medical practice. Three-dimensional visualization technology (3DVT) has been advanced as a promising tool to enhance anatomy knowledge. The purpose of this review is to conduct a meta-analysis of the effectiveness of 3DVT in teaching and learning anatomy compared to all teaching methods. The primary outcomes were scores of anatomy knowledge tests expressed as factual or spatial knowledge percentage means. Secondary outcomes were perception scores of the learners. Thirty-six studies met the inclusion criteria including 28 (78%) randomized studies. Based on 2,226 participants including 2,128 from studies with comparison groups, 3DVTs (1) resulted in higher (d = 0.30, 95%CI: 0.02-0.62) factual knowledge, (2) yielded significant better results (d = 0.50, 95%CI: 0.20-0.80) in spatial knowledge acquisition, and (3) produced significant increase in user satisfaction (d = 0.28, 95%CI = 0.12-0.44) and in learners' perception of the effectiveness of the learning tool (d = 0.28, 95%CI = 0.14-0.43). The total mean scores (out of five) and ±SDs for QUESTS's Quality and Strength dimensions were 4.38 (±SD 1.3) and 3.3 (±SD 1.7), respectively. The results have high internal validity, for the improved outcomes of 3DVTs compared to other methods of anatomy teaching. Given that anatomy teaching and learning in the modern medical school appears to be approaching a crisis, 3DVT can be a potential solution to the problem of inadequate anatomy pedagogy.