Assessment Question Characteristics Predict Medical Student Performance in General Pathology.

CONTEXT.­: Evaluation of medical curricula includes appraisal of student assessments in order to encourage deeper learning approaches. General pathology is our institution's 4-week, first-year course covering universal disease concepts (inflammation, neoplasia, etc). OBJECTIVE.­: To compare types of assessment questions and determine which characteristics may predict student scores, degree of difficulty, and item discrimination. DESIGN.­: Item-level analysis was employed to categorize questions along the following variables: type (multiple choice question or matching answer), presence of clinical vignette (if so, whether simple or complex), presence of specimen image, information depth (simple recall or interpretation), knowledge density (first or second order), Bloom taxonomy level (1-3), and, for the final, subject familiarity (repeated concept and, if so, whether verbatim). RESULTS.­: Assessments comprised 3 quizzes and 1 final exam (total 125 questions), scored during a 3-year period, (total 417 students) for a total 52 125 graded attempts. Overall, 44 890 attempts (86.1%) were correct. In multivariate analysis, question type emerged as the most significant predictor of student performance, degree of difficulty, and item discrimination, with multiple choice questions being significantly associated with lower mean scores (P = .004) and higher degree of difficulty (P = .02), but also, paradoxically, poorer discrimination (P = .002). The presence of a specimen image was significantly associated with better discrimination (P = .04), and questions requiring data interpretation (versus simple recall) were significantly associated with lower mean scores (P = .003) and a higher degree of difficulty (P = .046). CONCLUSIONS.­: Assessments in medical education should comprise combinations of questions with various characteristics in order to encourage better student performance, but also obtain optimal degrees of difficulty and levels of item discrimination.

Design and validation of the biliary atresia research consortium histologic assessment system for cholestasis in infancy.

BACKGROUND & AIMS: Pathologists participating in the National Institutes of Health-sponsored Biliary Atresia Research Consortium (BARC) developed and then evaluated a standardized system for histologic reporting of liver biopsies from infants with cholestasis. METHODS: A set of 97 anonymous liver biopsy samples was sent to 10 pathologists at BARC centers. A semiquantitative scoring system that had 16 histologic features was developed and then used by the pathologists, who had no knowledge of clinical history, imaging results, or laboratory data. Interobserver agreement was evaluated statistically. Agreement on scoring of each feature and on the pathologists' diagnosis, compared with the final clinical diagnosis, was evaluated by using weighted kappa statistics. RESULTS: There was moderate to substantial interobserver agreement in identification of bile plugs in ducts, giant-cell transformation, extramedullary hematopoiesis, and bile duct proliferation. The pathologists' diagnosis of obstruction in clinically proven cases of biliary atresia (BA) ranged from 79%-98%, with a positive predictive value of 90.7%. Histologic features that best predicted BA, on the basis of logistic regression, included bile duct proliferation, portal fibrosis, and absence of sinusoidal fibrosis (each P<.0001). CONCLUSIONS: The BARC histologic assessment system identified features of liver biopsies from cholestatic infants, with good interobserver agreement, that might be used in diagnosis and determination of prognosis. The system diagnosed BA with a high level of sensitivity and identified infants with biliary obstruction with reasonable interobserver agreement. However, distinguishing between BA and disorders such as total parenteral nutrition-associated liver disease and alpha(1)-antitrypsin deficiency is not possible without adequate clinical information.