A Multi-institutional Study of the Feasibility and Reliability of the Implementation of Constructed Response Exam Questions.

PROBLEM: Some medical schools have incorporated constructed response short answer questions (CR-SAQs) into their assessment toolkits. Although CR-SAQs carry benefits for medical students and educators, the faculty perception that the amount of time required to create and score CR-SAQs is not feasible and concerns about reliable scoring may impede the use of this assessment type in medical education. INTERVENTION: Three US medical schools collaborated to write and score CR-SAQs based on a single vignette. Study participants included faculty question writers (N = 5) and three groups of scorers: faculty content experts (N = 7), faculty non-content experts (N = 6), and fourth-year medical students (N = 7). Structured interviews were performed with question writers and an online survey was administered to scorers to gather information about their process for creating and scoring CR-SAQs. A content analysis was performed on the qualitative data using Bowen's model of feasibility as a framework. To examine inter-rater reliability between the content expert and other scorers, a random selection of fifty student responses from each site were scored by each site's faculty content experts, faculty non-content experts, and student scorers. A holistic rubric (6-point Likert scale) was used by two schools and an analytic rubric (3-4 point checklist) was used by one school. Cohen's weighted kappa (κw) was used to evaluate inter-rater reliability. CONTEXT: This research study was implemented at three US medical schools that are nationally dispersed and have been administering CR-SAQ summative exams as part of their programs of assessment for at least five years. The study exam question was included in an end-of-course summative exam during the first year of medical school. IMPACT: Five question writers (100%) participated in the interviews and twelve scorers (60% response rate) completed the survey. Qualitative comments revealed three aspects of feasibility: practicality (time, institutional culture, teamwork), implementation (steps in the question writing and scoring process), and adaptation (feedback, rubric adjustment, continuous quality improvement). The scorers' described their experience in terms of the need for outside resources, concern about lack of expertise, and value gained through scoring. Inter-rater reliability between the faculty content expert and student scorers was fair/moderate (κw=.34-.53, holistic rubrics) or substantial (κw=.67-.76, analytic rubric), but much lower between faculty content and non-content experts (κw=.18-.29, holistic rubrics; κw=.59-.66, analytic rubric). LESSONS LEARNED: Our findings show that from the faculty perspective it is feasible to include CR-SAQs in summative exams and we provide practical information for medical educators creating and scoring CR-SAQs. We also learned that CR-SAQs can be reliably scored by faculty without content expertise or senior medical students using an analytic rubric, or by senior medical students using a holistic rubric, which provides options to alleviate the faculty burden associated with grading CR-SAQs.

Youth Enjoy Science Program at the Case Comprehensive Cancer Center: Increasing Engagement and Opportunity for Underrepresented Minority Students.

The Youth Enjoy Science (YES) Program at the Case Comprehensive Cancer Center is a National Cancer Institute (NCI) R25-funded training grant, designed to increase the pipeline of underrepresented minority (URM) students entering college and pursuing biomedical research and health care careers in the Cleveland Metropolitan and surrounding school districts. The three components of the program include: Learn to Beat Cancer, engaging middle school students and their families; Research to Beat Cancer, designed for high school students and college undergraduates; and Teach to Beat Cancer, focused on enhancing science, technology, engineering, and mathematics (STEM) teaching capacity among high school teachers. This study focuses on Research to Beat Cancer, which, in 2018 enrolled 36 URM students as paid summer scholars. Students were assigned to a faculty mentor, were taught laboratory safety, responsible conduct of research and the scientific method, and then immersed in full-time laboratory cancer research during an eight-week period. Twice each week, students participated in Lunch and Learn Seminars where faculty members provided combined motivational and scientific guidance lectures. In a capstone poster session at the end of the program, students presented their research to peers, medical and graduate students, family members, faculty, community members and leaders. Students' perceptions of the program were reported using descriptive statistics and qualitative thematic analyses. Twenty-four of the 2018 YES students (67%) and 19 (53%) mentors completed the online post-program survey. Opportunity was a major qualitative theme from student and mentor responses. Future research will investigate the long-term impacts of YES, including college enrollment.