Twelve tips for clinicians dealing with uncertainty when assessing learners.

Clinician educators often experience distress caused by uncertainty regarding how effectively to participate in assessment practices in a way that supports both their programs and their students. Uncertainty is a common state for clinicians, particularly for those who see patients with early or ill-defined illness presentations. While clinicians often feel ill at ease when facing uncertainty in the clinical realm, becoming comfortable with uncertainty and learning to manage such states are now recognized as vital components of clinical practice. Clinicians, as a result, have adopted a series of strategies to lessen the unease that uncertainty can create. While similar experiences plague clinician educators placed in assessment roles, much less attention has been given to how we can support individuals in the education setting. Here, the distress of uncertainty may be greater due to clinician educators having less experience with assessment practices. Fortunately, strategies that are effective in the clinical domain can be translated into the assessment realm to accommodate uncertainty when assessing learners. In this 12 tips article we offer guidance on the translation of such strategies.

Development and testing of an assessment instrument for the formative peer review of significant event analyses.

AIM: To establish the content validity and specific aspects of reliability for an assessment instrument designed to provide formative feedback to general practitioners (GPs) on the quality of their written analysis of a significant event. METHODS: Content validity was quantified by application of a content validity index. Reliability testing involved a nested design, with 5 cells, each containing 4 assessors, rating 20 unique significant event analysis (SEA) reports (10 each from experienced GPs and GPs in training) using the assessment instrument. The variance attributable to each identified variable in the study was established by analysis of variance. Generalisability theory was then used to investigate the instrument's ability to discriminate among SEA reports. RESULTS: Content validity was demonstrated with at least 8 of 10 experts endorsing all 10 items of the assessment instrument. The overall G coefficient for the instrument was moderate to good (G>0.70), indicating that the instrument can provide consistent information on the standard achieved by the SEA report. There was moderate inter-rater reliability (G>0.60) when four raters were used to judge the quality of the SEA. CONCLUSIONS: This study provides the first steps towards validating an instrument that can provide educational feedback to GPs on their analysis of significant events. The key area identified to improve instrument reliability is variation among peer assessors in their assessment of SEA reports. Further validity and reliability testing should be carried out to provide GPs, their appraisers and contractual bodies with a validated feedback instrument on this aspect of the general practice quality agenda.

Assessing tutorial-based assessment.

Since the development of problem-based learning curricula, medical educators have viewed tutorials as an ideal context within which medical student competence can be assessed. Advantages of tutorial-based assessment include, (a) evaluation based on prolonged and intense interactions between students, peers, and tutors, (b) the opportunity to assess domains of competence that are not readily assessed by more traditional examinations (e.g., communication skills, scientific curiosity, and respect for peers), and (c) avoidance of the negative impact of more formal summative evaluations. However, in addition to psychometric weaknesses inherent in this form of evaluation, the interpersonal relationships that serve as one of the primary strengths of the tutorial have also proven to be one of the main defects in tutorial-based assessment. The current paper will review the evidence relevant to this assessment paradox, paying particular attention to the use of self-assessment.