Application of MCAT questions as a testing tool and evaluation metric for knowledge graph-based reasoning systems.

"Knowledge graphs" (KGs) have become a common approach for representing biomedical knowledge. In a KG, multiple biomedical data sets can be linked together as a graph representation, with nodes representing entities, such as "chemical substance" or "genes," and edges representing predicates, such as "causes" or "treats." Reasoning and inference algorithms can then be applied to the KG and used to generate new knowledge. We developed three KG-based question-answering systems as part of the Biomedical Data Translator program. These systems are typically tested and evaluated using traditional software engineering tools and approaches. In this study, we explored a team-based approach to test and evaluate the prototype "Translator Reasoners" through the application of Medical College Admission Test (MCAT) questions. Specifically, we describe three "hackathons," in which the developers of each of the three systems worked together with a moderator to determine whether the applications could be used to solve MCAT questions. The results demonstrate progressive improvement in system performance, with 0% (0/5) correct answers during the first hackathon, 75% (3/4) correct during the second hackathon, and 100% (5/5) correct during the final hackathon. We discuss the technical and sociologic lessons learned and conclude that MCAT questions can be applied successfully in the context of moderated hackathons to test and evaluate prototype KG-based question-answering systems, identify gaps in current capabilities, and improve performance. Finally, we highlight several published clinical and translational science applications of the Translator Reasoners.

The change from UMAT to UCAT for undergraduate medical school applicants: impact on selection outcomes.

OBJECTIVES: To assess whether the change from the Undergraduate Medical and Health Sciences Admissions Test (UMAT; 1991-2019) to the University Clinical Aptitude Test (UCAT) for the 2020 New South Wales undergraduate medical degree intake was associated with changes in the impact of sex, socio-economic status and remoteness of residence, and professional coaching upon selection for interview. DESIGN, SETTING, PARTICIPANTS: Cross-sectional study of applicants for the three NSW undergraduate medical programs for entry in 2019 (4114 applicants) or 2020 (4270); 703 people applied for both intakes. Applicants selected for interview were surveyed about whether they had received professional coaching for the selection test. MAIN OUTCOME MEASURES: Scores on the three sections of the UMAT (2019 entry cohort) and the five subtests of the UCAT (2020 entry); total UMAT and UCAT scores. RESULTS: Mean scores for UMAT 1 and 3 and for all four UCAT cognitive subtests were higher for men than women; the differences were statistically significant after adjusting for age, socio-economic status, and remoteness. The effect size for sex was 0.24 (95% CI, 0.18-0.30) for UMAT total score, 0.38 (95% CI, 0.32-0.44) for UCAT total score. For the 2020 intake, 2303 of 4270 applicants (53.9%) and 476 of 1074 interviewees (44.3%) were women. The effect size for socio-economic status was 0.47 (95% CI, 0.39-0.54) for UMAT, 0.43 (95% CI, 0.35-0.50) for UCAT total score; the effect size for remoteness was 0.54 (95% CI, 0.45-0.63) for UMAT, 0.48 (95% CI, 0.39-0.58) for UCAT total score. The impact of professional coaching on UCAT performance was not statistically significant among those accepted for interview. CONCLUSIONS: Women and people from areas outside major cities or of lower socio-economic status perform less well on the UCAT than other applicants. Reviewing the test and applicant quotas may be needed to achieve selection equity.

Admissions Is Not Enough: The Racial Achievement Gap in Medical Education.

The achievement gap is a disparity in academic and standardized test performance that exists between White and underrepresented minority (URM) students that begins as early as preschool and worsens as students progress through the educational system. Medical education is not immune to this inequality. URM medical students are more likely to experience delayed graduation and course failure, even after accounting for science grade point average and Medical College Admission Test performance. Moreover, URM students are more likely to earn lower scores on licensing examinations, which can have a significant impact on their career trajectory, including specialty choice and residency competitiveness. After the release of preliminary recommendations from the Invitational Conference on USMLE Scoring (InCUS) and public commentary on these recommendations, the National Board of Medical Examiners and Federation of State Medical Boards announced that the United States Medical Licensing Examination (USMLE) Step 1 would transition from a 3-digit numeric score to pass/fail scoring. Given that another of InCUS's recommendations was to "minimize racial demographic differences that exist in USMLE performance," it is paramount to consider the impact of this scoring change on URM medical students specifically. Holistic admissions are a step in the right direction of acknowledging that URM students often travel a further distance to reach medical school. However, when residency programs emphasize USMLE performance (or any standardized test score) despite persistent test score gaps, medical education contributes to the disproportionate harm URM students face and bolsters segregation across medical specialties. This Perspective provides a brief explanation of the achievement gap, its psychological consequences, and its consequences in medical education; discusses the potential effect of the Step 1 scoring change on URM medical students; and provides a review of strategies to redress this disparity.

Predictive Ability And Stakeholders' Perceptions Of The Selection Tools For Mbbs In Women Medical College: A Mixed Methods Study.

BACKGROUND: The selection criteria for entry into the MBBS programme used by Women Medical College (WMC) includes previous academic achievements, namely Secondary School Certificate (SSC) and Higher Secondary Certificate (HSC), and the Medical and Dental College Admission Test (MDCAT). This study determined the predictive validity of these selection tools and explored the perceptions of stakeholders regarding the selection process and the use of selection tools in WMC. METHODS: This study utilized both quantitative and qualitative methods. Quantitative methods involved a retrospective cohort study design to determine the statistical correlation between the performance of candidates in the selection tools and their subsequent academic achievements at medical college. This consisted of data collected from three cohorts (n=186) of students who graduated in 2014, 2015 and 2016. Qualitative methods of the study explored the perceptions of stakeholders through purposive sampling using face-to-face semi-structured interviews, which were analysed using thematic analysis. RESULTS: The study showed very weak correlations of SSC with performance in the fourth professional examination undertaken by the students and HSC with performance in the first and second professional examinations. MDCAT did not correlate with any professional examination. Qualitative analysis identified three emerging themes; a) lack of standardization, b) fairness of selection criteria, and c) assessment of noncognitive attributes. CONCLUSIONS: The selection tools showed poor predictive ability for the performance of students in the medical college. Standardizing the selection tools and including an assessment of non-cognitive attributes in the selection criteria is suggested.

Selecting top candidates for medical school selection interviews- a non-compensatory approach.

BACKGROUND: Medical schools apply a range of selection methods to ensure that admitted students succeed in the program. In Australia, selection tools typically include measures of academic achievement (e.g. the Australian Tertiary Admission Rank - ATAR) and aptitude tests (e.g. the Undergraduate Medicine and Health Sciences Admissions Test - UMAT). These are most commonly used to determine which applicants are invited for additional selection processes, such as interviews. However, no previous study has examined the efficacy of the first part of the selection process. In particular, are compensatory or non-compensatory approaches more effective in evaluating the outcomes of cognitive and aptitude tests, and do they affect the demographics of students selected for interview? METHODS: This study utilised data from consecutive cohorts of mainstream domestic students who applied to enter the UNSW Medicine program between 2013 to 2018. A compensatory ranked selection model was compared with a non-compensatory ranked model. Initially, ATAR marks and UMAT scores for each applicant were ranked within each cohort. In the compensatory model, the mean of the ATAR and UMAT ranks were used to determine the outcome. In the non-compensatory model, the lowest rank of ATAR and UMAT determined the outcome for each applicant. The impact of each model on the gender and socioeconomic status of applicants selected to interview was evaluated across all cohorts. RESULTS: The non-compensatory ranked selection model resulted in substantially higher ATAR and UMAT thresholds for invitation to interview, with no significant effect on the socioeconomic status of the selected applicants. CONCLUSIONS: These results are important, demonstrating that it is possible to raise the academic threshold for selection to medicine without having any negative impact on applicants from low socioeconomic backgrounds. Overall, the evidence gathered in this study suggests that a non-compensatory model is preferable for selecting applicants for medical student selection interview.

Assessment of English language performance scores and academic performance in an English-based curriculum for pharmacy students with English as a second language.

BACKGROUND: The primary objective was to determine if there is a relationship between English language performance and graduating grade point average (GPA) in pharmacy students with English as a second language (ESL). METHODS: Students graduating from a four-year pharmacy program in 2016-2018 were invited to participate in the study. We compared pharmacy students' pre-admission ESL scores to their cumulative GPA at graduation in each of the three graduating cohorts and also determined if these results held true for both genders. Correlation of GPA to individual mathematics, chemistry, and Chinese language scores were used as points of reference to compare the degree of correlation. RESULTS: There were 148 students screened for the study with 31 students not meeting the inclusion criteria and four students declining to participate. Statistical analyses show an overall weak correlation. There was a statistically significant stronger correlation between pre-admission ESL scores and cumulative graduating GPA (r = 0.273) as compared to the correlation of GPA with mathematics (r = 0.187), chemistry (r = 0.181), or Chinese language scores (r = 0.059). The results did not change when the study population was separated by gender as English score still had the strongest correlation as compared to the other subjects. CONCLUSION: This study provides evidence that English language scores correlate more strongly with academic performance than mathematics, chemistry, or Chinese language scores in ESL pharmacy students. Also, this English language correlation is stronger for females than males.

The impact of penalties for wrong answers on the gender gap in test scores.

Multiple-choice examinations play a critical role in university admissions across the world. A key question is whether imposing penalties for wrong answers on these examinations deters guessing from women more than men, disadvantaging female test-takers. We consider data from a large-scale, high-stakes policy change that removed penalties for wrong answers on the national college entry examination in Chile. The policy change reduced a large gender gap in questions skipped. It also narrowed gender gaps in performance, primarily among high-performing test-takers, and in the fields of math, social science, and chemistry.

Academic Achievements of College Graduates With Learning Disabilities Vis-a-Vis Admission Criteria and Academic Support.

Postsecondary entrance examination scores are generally low predictors of college achievement (grade point average [GPA]) for students with learning disabilities (LD). The difficulties with meeting academic requirements have raised the awareness of the needs of students with LD for support services. The present study examined the adequacy of entrance criteria to academic studies for students with LD and the effectiveness of three support levels during their academic studies in increasing their academic gains. Data were collected for 315 college graduates with LD and 955 graduates who do not have LD (NLD) who completed their BA studies in a college in Israel. Although the admission scores of graduates with LD were lower than those of NLD graduates, their GPA was higher than the GPA of NLD graduates. The psychometric score had a low correlation with final college GPA of graduates with LD. There was no difference in the final GPA of the graduates with LD in the different support programs. The findings suggest that academic support can be an effective way to help students with LD to eliminate or close the gap between them and NLD students and to earn an academic degree, regardless of what if any admission criteria were applied to them.

Effects of Moving the United States Medical Licensing Examination Step 1 After Core Clerkships on Step 2 Clinical Knowledge Performance.

PURPOSE: To investigate the effect of a change in the United States Medical Licensing Examination Step 1 timing on Step 2 Clinical Knowledge (CK) scores, the effect of lag time on Step 2 CK performance, and the relationship of incoming Medical College Admission Test (MCAT) score to Step 2 CK performance pre and post change. METHOD: Four schools that moved Step 1 after core clerkships between academic years 2008-2009 and 2017-2018 were analyzed. Standard t tests were used to examine the change in Step 2 CK scores pre and post change. Tests of differences in proportions were used to evaluate whether Step 2 CK failure rates differed between curricular change groups. Linear regressions were used to examine the relationships between Step 2 CK performance, lag time and incoming MCAT score, and curricular change group. RESULTS: Step 2 CK performance did not change significantly (P = .20). Failure rates remained highly consistent (pre change: 1.83%; post change: 1.79%). The regression indicated that lag time had a significant effect on Step 2 CK performance, with scores declining with increasing lag time, with small but significant interaction effects between MCAT and Step 2 CK scores. Students with lower incoming MCAT scores tended to perform better on Step 2 CK when Step 1 was after clerkships. CONCLUSIONS: Moving Step 1 after core clerkships appears to have had no significant impact on Step 2 CK scores or failure rates, supporting the argument that such a change is noninferior to the traditional model. Students with lower MCAT scores benefit most from the change.

The Consequences of Structural Racism on MCAT Scores and Medical School Admissions: The Past Is Prologue.

Those in medical education have a responsibility to prepare a physician workforce that can serve increasingly diverse communities, encourage healthy changes in patients, and advocate for the social changes needed to advance the health of all. The authors of this Perspective discuss many of the likely causes of the observed differences in mean Medical College Admission Test (MCAT) scores between students from groups well represented in medicine and those from groups underrepresented in medicine. The lower mean MCAT scores of underrepresented groups can present challenges to diversifying the physician workforce if medical schools only admit those applicants with the highest MCAT scores. The authors review the psychometric literature, which showed no evidence of bias in the exam, and note that the differences in mean MCAT scores between racial and ethnic groups are similar to those in other measures of academic achievement and performance on high-stakes tests.The authors then describe the ways in which structural racism in the United States has contributed to differences in achievement for underrepresented students compared with well-represented students. These differences are not due to differences in aptitude but to differences in opportunities. The authors describe the widespread consequences of structural racism on economic success, educational opportunity, and bias in the educational environment. They close with 3 recommendations for medical schools that may mitigate the consequences of structural racism while maintaining academic standards and admitting students likely to succeed. Adopting these recommendations may help the medical profession build the diverse physician workforce needed to serve communities today.

The Diversity and Success of Medical School Applicants With Scores in the Middle Third of the MCAT Score Scale.

Admissions officers assemble classes of medical students with different backgrounds and experiences who can contribute to their institutions' service, leadership, and research goals. While schools' local interests vary, they share a common goal: meeting the health needs of an increasingly diverse population. Despite the well-known benefits of diversity, the physician workforce does not yet reflect the nation's diversity by socioeconomic status, race/ethnicity, or other background characteristics.The authors reviewed the Medical College Admission Test (MCAT) scores and backgrounds of 2017 applicants, accepted applicants, and matriculants to U.S. MD-granting schools to explore avenues for increasing medical school class diversity. They found that schools that accepted more applicants with midrange MCAT scores had more diverse matriculating classes. Many schools admitting the most applicants with scores in the middle of the MCAT score scale were public, community-based, and primary care-focused institutions; those admitting the fewest of these applicants tended to be research-focused institutions and to report pressure to accept applicants with high MCAT scores to maintain or improve their national rankings.The authors argue that reexamining the use of MCAT scores in admissions provides an opportunity to diversify the physician workforce. Despite evidence that most students with midrange MCAT scores succeed in medical school, there is a tendency to overlook these applicants in favor of those with higher scores. To improve the health of all, the authors call for admitting more students with midrange MCAT scores and studying the learning environments that enable students with a wide range of MCAT scores to thrive.

The Validity of Scores From the New MCAT Exam in Predicting Student Performance: Results From a Multisite Study.

PURPOSE: The new Medical College Admission Test (MCAT) was introduced in April 2015. This report presents findings from the first study of the validity of scores from the new MCAT exam in predicting student performance in the first year of medical school (M1). METHOD: The authors analyzed data from the national population of 2016 matriculants with scores from the new MCAT exam (N = 7,970) and the sample of 2016 matriculants (N = 955) from 16 medical schools who volunteered to participate in the validity research. They examined correlations of students' MCAT total scores and total undergraduate grade point averages (UGPAs), alone and together, with their summative performance in M1, and the success rate of students with different MCAT scores in their on-time progression to the second year of medical school (M2). They assessed whether MCAT scores provided comparable prediction of performance in M1 by students' race/ethnicity, socioeconomic background, and gender. RESULTS: Correlations of MCAT scores with summative performance in M1 ranged from medium to large. Although MCAT scores and UGPAs provided similar prediction of performance in M1, using both metrics provided better prediction than either alone. Additionally, students with a wide range of MCAT scores progressed to M2 on time. Finally, MCAT scores provided comparable prediction of performance in M1 for students from different sociodemographic backgrounds. CONCLUSIONS: This study provides early evidence that scores from the new MCAT exam predict student performance in M1. Future research will examine the validity of MCAT scores in predicting performance in later years.

A Study of the Validity of the New MCAT Exam.

PURPOSE: To conduct a study of the validity of the new Medical College Admission Test (MCAT). METHOD: Deidentified data for first- and second-year medical students (185 women, 54.3%; 156 men, 45.7%) who matriculated in 2016 and 2017 to the University of Minnesota Medical School-Twin Cities were included. Of those students, 220 (64.5%) had taken the new MCAT exam and 182 (53.4%) had taken the old MCAT exam (61 [17.9%] had taken both). The authors calculated descriptive statistics and Pearson product moment correlations (r) between new and old MCAT section scores. They conducteda regression analysis of MCAT section scores with Step 1 scores and with preclerkship course performance. They also conducted an exploratory factor analysis (principal component analysis with varimax rotation) of MCAT scores, undergraduate grade point average, Step 1 scores, and course performance. RESULTS: The new MCAT exam section mean score percentiles ranged from 72 to 78 (mean composite score percentile of 80). The old MCAT exam section mean score percentiles ranged from 84 to 88 (mean composite score percentile of 83). The pattern of correlations among and between new and old MCAT exam section scores (range of r: 0.03-0.67; P < .01) provided evidence of both divergent and convergent validities. Backward multiple regression of new MCAT exam section scores and Step 1 scores resulted in a multiple R of .440; the same analysis with Human Behavior course performance as the dependent variable provided a similar solution with the expected sections of the new MCAT exam (multiple R = .502). The factor analysis resulted in 4 cohesive, theoretically meaningful factors: biomedical knowledge, basic science concepts, cognitive reasoning, and general achievement. CONCLUSIONS: This study provided empirical evidence of multiple types of validity for the new MCAT exam.

Is College Completion Associated with Better Cognition in Later Life for People Who Are the Least, or Most, Likely to Obtain a Bachelor's Degree?

OBJECTIVES: Drawing on insights from theorizing on cumulative dis/advantage (CDA), we aimed to advance understanding of educational attainment as a protective factor for later-life cognition by examining whether associations between obtaining a bachelor's degree and later-life cognition differ according to individuals' likelihood of completing college based on characteristics in adolescence. METHODS: We conducted a propensity score analysis with data from the Wisconsin Longitudinal Study (WLS). Measures to predict college completion were assessed prospectively in adolescence, and a global measure of later-life cognition was based on cognitive assessments at age 65. RESULTS: College completion by age 25 (vs high school only) was associated with better later-life cognition for both men and women. Among men specifically, associations were stronger for those who were less likely as adolescents to complete college. DISCUSSION: Results indicate the utility of a CDA perspective for investigating the implications of interconnected early life risk and protective factors for later-life cognition, as well as ways in which college education can both contribute to, as well as mitigate, processes of CDA.

Use of the United States Medical Licensing Examination Step-1 Score as a Screening Tool for Orthopaedic Surgery Away Rotations.

BACKGROUND: Orthopaedic surgery has become an increasingly competitive specialty for medical students applying to residency. Aside from residency interviews, away rotations are one of the only opportunities for residency programs to qualitatively evaluate an applicant. The purpose of this study was to determine if residency program directors (PDs) use a minimum United States Medical Licensing Examination (USMLE) Step-1 score as a screening tool for students applying for away rotations at their institution. METHODS: An anonymous 12-question survey about residency selection criteria and which criteria are used to select applicants for away rotations was distributed to PDs at U.S. allopathic orthopaedic surgery residency programs who are members of the Council of Orthopaedic Residency Directors (CORD). Questions included information on minimum USMLE Step-1 scores, the number of students who complete away rotations at the program each year, and how applicants are selected for rotations. RESULTS: Survey responses were received from 87 (58%) of 149 PDs. Seventy-one (82%) of these PDs reported that their program uses a minimum Step-1 score for residency applications, with the most common cutoff score being between 231 and 240 (as answered by 33 [46%] of 72 PDs). Twenty-seven (31%) of 87 PDs reported that if a student does not meet their residency minimum Step-1 score, he or she is not offered the opportunity to rotate at the program. Eleven (25%) of 44 PDs reported that they do not inform students when their score is below the program's minimum. CONCLUSIONS: A substantial proportion of orthopaedic surgery residency PDs use the USMLE Step-1 score as a screening tool for students interested in doing away rotations at their programs. If a student's Step-1 score does not meet the program's minimum, the majority of programs still will allow the student to rotate at their institution. Students should take this information into account when selecting away rotations in order to maximize their chances of matching into an orthopaedic surgery program.

An Analysis of the Optometric Applicant Pool Relative to Matriculants.

SIGNIFICANCE: The interplay of applicants to optometry school and matriculants has not been explored systematically. It is vital that the profession examines these trends to ensure a viable pipeline of future doctors of optometry. PURPOSE: The purpose of this study was to describe the demographics and academic qualifications of entering optometry classes from autumn 2010 through autumn 2018 of U.S.-based optometric institutions' application pool and matriculants (enrollees). METHODS: Data were gathered from reports generated from accredited schools and colleges of optometry in the United States and compiled by the Association of Schools and Colleges of Optometry (publicly available) and the Optometry Centralized Application Service. Metrics included the annual number of verified applicants, the annual number of matriculants, the home region of U.S.-based applicants, and the Optometry Admission Test (OAT) performance and grade point average of verified applicants. RESULTS: The number of verified applicants for autumn 2018 was 0.95% higher than that for autumn 2010, yet the number of matriculants in 2018 compared with 2010 increased by 11.2% with an applicant-to-matriculant ratio in 2010 of 1.53 compared with 1.39 in 2018. Grade point average and academic average OAT scores were stable from 2010 to 2018. The ratios of verified applicants with an academic average OAT score of at least 300 to matriculants were 0.87 for autumn 2018 and 0.92 for autumn 2010. The ratios of verified applicants with a grade point average of at least 3.00 to matriculants were 1.13 for autumn 2018 and 1.23 for autumn 2015. CONCLUSIONS: Evidence supports the conclusion that the applicant pool has remained essentially flat for the last decade, whereas the number of matriculants has increased substantially; thus, the number of qualified applicants to matriculants has logically decreased. In the last 2 years, optometric programs have responded by decreasing their institution's number of matriculants to accommodate the national trends.

A model to predict NAPLEX outcomes and identify students needing additional preparation.

INTRODUCTION: Studies have been conducted to identify factors that may predict North American Pharmacist Licensure Examination (NAPLEX) outcomes, but there is no proposed single or combination of predictors that can be implemented reliably in academia. We aimed to develop a NAPLEX outcomes predictive model that could be practical, measurable, and reliable. METHODS: The study cohort consisted of students who graduated from 2012 to 2016 who had taken NAPLEX and whose first-attempt examination scores were available to the school of pharmacy. Students were considered to have poor performance on NAPLEX if they received an overall score of less than or equal to 82. Linear and logistic regression analysis were utilized to identify independent predictors. RESULTS: Seventy of 433 (16.2%) students were identified as poor performers. Independent factors that were associated with a poor outcome on NAPLEX were: age >28 years at graduation, Pharmacy College Admission Test scaled score <74, High Risk Drug Knowledge Assessment score <90, third-year Pharmacy Curriculum Outcome Assessment scaled score <349, and grades of <74 in more than three courses. These predictors were utilized to stratify students into four risk groups: Low, Intermediate-1, Intermediate-2, and High. Mean NAPLEX scores for these groups were 106.4, 97.4, 87.1, and 75.1, respectively. CONCLUSIONS: The model can be used as a practical tool to identify students who are at risk for poor performance on NAPLEX. Four of the five predictors in the model could be generalizable to other schools of pharmacy.

Examining the Association of GPA and PCAT Scores on Objective Structured Clinical Examination Scores.

Objective. To examine the association between certain demographic and admission measures and Objective Structured Clinical Examination (OSCE) performance in a cohort of pharmacy students. Methods. A retrospective review of demographic characteristics, admissions data [cumulative and science admission Grade Point Average (GPA), Pharmacy College Admissions Test (PCAT) scores], and OSCE scores was performed for the Class of 2017 at the University of Tennessee College of Pharmacy. Results. Female students scored significantly higher than male students on the Warfarin OSCE - Standardized Patient (SP) rated General Communication Skills and on the Warfarin OSCE - Faculty rated Patient Interviewing Skills. Age was significantly, inversely correlated with Warfarin OSCE - Faculty rated Therapeutic Knowledge score. Warfarin OSCE - SP rated General Communication Skills score was significantly, positively related to PCAT composite score and PCAT reading comprehension score. PCAT composite score was significantly, inversely correlated to Warfarin OSCE - Faculty rated Patient Interviewing Skills score. Warfarin OSCE - Faculty rated General Communication Skills score was significantly, positively related to cumulative admission GPA and admission science GPA. Conclusion. Eight statistically significant correlations were found between demographic and admissions measures and specific OSCE scores. Regression models were significant but explained a low percentage of the variance in OSCE scores, suggesting other factors not included in the study have a greater effect on scores. Such factors may include knowledge acquired in pharmacy school courses such as the therapeutics course series.