Assessing student engagement and self-regulated learning in a medical gross anatomy course.

In courses with large enrollment, faculty members sometimes struggle with an understanding of how individual students are engaging in their courses. Information about the level of student engagement that instructors would likely find most useful can be linked to: (1) the learning strategies that students are using; (2) the barriers to learning that students are encountering; and (3) whether the course materials and activities are yielding the intended learning outcomes. This study drew upon self-regulated learning theory (SRL) to specify relevant information about learning engagement, and how the measures of particular scales might prove useful for student/faculty reflection. We tested the quality of such information as collected via the Motivated Strategies for Learning Questionnaire (MSLQ). MSLQ items were administered through a web-based survey to 150 students in a first-year medical gross anatomy course. The resulting 66 responses (44% response rate) were examined for information quality (internal reliability and predictive validity) and usefulness of the results to the course instructor. Students' final grades in the course were correlated with their MSLQ scale scores to assess the predictive validity of the measures. These results were consistent with the course design and expectations, showing that greater use of learning strategies such as elaboration and critical thinking was associated with higher levels of performance in the course. Motivation subscales for learning were also correlated with the higher levels of performance in the course. The extent to which these scales capture valid and reliable information in other institutional settings and courses needs further investigation.

Assessing effective teaching: what medical students value when developing evaluation instruments.

PURPOSE: To investigate what criteria medical students would value and use in assessing teaching skills. METHOD: Fourth-year medical students at the University of Iowa Carver College of Medicine enrolled in a teaching elective course are required to design and use an evaluation instrument to assess effective teaching. Each class uses a similar process in developing their instruments. Since the first class in spring 2007, 193 medical students have created 36 different instruments. Three faculty evaluation experts conducted a thematic analysis of the instruments and coded the information according to what was being evaluated and what types of ratings were indicated. The data were submitted to a fourth faculty reviewer, who synthesized the information and adjusted the codes to better capture the data. Common themes and categories were detected. RESULTS: Four themes were identified: content (instructor knowledgeable, teaches at level of learner, practical information), learning environment, teacher personal attributes, and teaching methods. Thirty-two descriptors were distinguished across the 36 instruments. Thirteen descriptors were present in 50% or more of the instruments. The most common rating systems were Likert scales and open comments. CONCLUSIONS: Fourth-year medical students can offer an eclectic resource for evaluating teaching in the classroom and the clinic. Using the descriptors that were identified in greater than 50% of the evaluation instruments will provide effective measures that can be incorporated into medical teacher evaluation instruments.

A perspective on medical school admission research and practice over the last 25 years.

Over the last 25 years a large body of research has investigated how best to select applicants to study medicine. Although these studies have inspired little actual change in admission practice, the implications of this research are substantial. Five areas of inquiry are discussed: (1) the interview and related techniques, (2) admission tests, (3) other measures of personal competencies, (4) the decision process, and (5) defining and measuring the criterion. In each of these areas we summarize consequential developments and discuss their implication for improving practice. (1) The traditional interview has been shown to lack both reliability and validity. Alternatives have been developed that display promising measurement characteristics. (2) Admission test scores have been shown to predict academic and clinical performance and are generally the most useful measures obtained about an applicant. (3) Due to the high-stakes nature of the admission decision, it is difficult to support a logical validity argument for the use of personality tests. Although standardized letters of recommendation appear to offer some promise, more research is needed. (4) The methods used to make the selection decision should be responsive to validity research on how best to utilize applicant information. (5) Few resources have been invested in obtaining valid criterion measures. Future research might profitably focus on composite score as a method for generating a measure of a physician's career success. There are a number of social and organization factors that resist evidence-based change. However, research over the last 25 years does present important findings that could be used to improve the admission process.

Benchmarking the scholarly productivity of physician assistant educators: an update.

PURPOSE: Success in scholarship has long challenged physician assistant (PA) educators, most of whom enter academia with little experience in research or writing. Since most PA programs grant a professional graduate degree, and expectations for PA faculty typically focus on teaching and service rather than research, it is reasonable for promotion and tenure decisions to be based on comparisons from within the PA education realm. Providing such benchmarks is the focus of this report. Predictors of successful publication and trends over time are also explored briefly. METHODS: De-identified data from the 2010 Faculty Survey were obtained from the Physician Assistant Education Association (PAEA), including basic demographics, faculty rank and program role, degree, and number of peer-reviewed publications. PAEA distributed the online survey in March 2010 to all faculty associated with member programs. The response rate was 35%, N = 425. SPSS version 19 was used for data analysis. RESULTS: Respondents were 58.1% female. The mean number of publications reported by respondents over their entire career was 4.2, and over the last 3 years was 1.7. The respective median numbers of publications were one and zero. Logistic regression analysis identified three significant predictors of publication success: number of years in PA education, previous publications, and highest degree attained. CONCLUSIONS: This study seeks to provide rational benchmarks for PA program faculty seeking promotion or tenure. Previous publications and highest degree attained are key factors predicting successful publication. There is a continuing need for faculty development activities to help PA educators publish successfully.

Do preceptors with more rating experience provide more reliable assessments of medical student performance?

BACKGROUND: Although the existing psychometric literature provides guidance on the best method for acquiring a reliable clinical evaluation form (CEF)-based score, it also shows that a single CEF rating has very low reliability. PURPOSE: This study examines whether experience with rating students might act as a form of rater training and hence improve the quality of CEF ratings. METHODS: Preceptors were divided into two groups based on rater experience. The univariate and multivariate G study designs used were simple rater (r)-nested-within-person (p) [r : p and r(○) : p(•)] models, and in the univariate analysis was applied separately to CEFs completed by high and low experienced raters. RESULTS: The high experienced rater group yielded a substantially higher observed reliability in both the univariate and multivariate analyses. CONCLUSIONS: These results support the hypothesis that high experienced raters produce more reliable ratings of student performance and suggest methods for improving CEF ratings.

Assessing implicit gender bias in Medical Student Performance Evaluations.

For medical schools, the increasing presence of women makes it especially important that potential sources of gender bias be identified and removed from student evaluation methods. Our study looked for patterns of gender bias in adjective data used to inform our Medical Student Performance Evaluations (MSPEs). Multigroup Confirmatory Factor Analysis (CFA) was used to model the latent structure of the adjectives attributed to students (n = 657) and to test for systematic scoring errors by gender. Gender bias was evident in two areas: (a) women were more likely than comparable men to be described as ''compassionate,'' ''sensitive,'' and ''enthusiastic'' and (b) men were more likely than comparable women to be seen as ''quick learners.'' The gender gap in ''quick learner'' attribution grows with increasing student proficiency; men's rate of increase is over twice that of women's. Technical and nontechnical approaches for ameliorating the impact of gender bias on student recommendations are suggested.

Rater and occasion impacts on the reliability of pre-admission assessments.

CONTEXT: Some medical schools have recently replaced the medical school pre-admission interview (MSPI) with the multiple mini-interview (MMI), which utilises objective structured clinical examination (OSCE)-style measurement techniques. Their motivation for doing so stems from the superior reliabilities obtained with the OSCE-style measures. Other institutions, however, are hesitant to embrace the MMI format because of the time and costs involved in restructuring recruitment and admission procedures. OBJECTIVES: To shed light on the aetiology of the MMI's increased reliability and to explore the potential of an alternative, lower-cost interview format, this study examined the relative contributions of two facets (raters, occasions) to interview score reliability. METHODS: Institutional review board approval was obtained to conduct a study of all students who completed one or more MSPIs at a large Midwestern medical college during 2003-2007. Within this dataset, we identified 168 applicants who were interviewed twice in consecutive years and thus provided the requisite data for generalisability (G) and decision (D) studies examining these issues. RESULTS: Increasing the number of interview occasions contributed much more to score reliability than did increasing the number of raters. CONCLUSIONS: Replicating a number of interviews, each with one rater, is likely to be superior to the often recommended panel interview approach and may offer a practical, low-cost method for enhancing MSPI reliability. Whether such a method will ultimately enhance MSPI validity warrants further investigation.