Using MCQ response certainty to determine how aspects of self-monitoring develop through a medical course.

INTRODUCTION: Self-monitoring of clinical-decision-making is essential for health care professional practice. Using certainty in responses to assessment items could allow self-monitoring of clinical-decision-making by medical students to be tracked over time. This research introduces how aspects of insightfulness, safety and efficiency could be based on certainty in, and correctness of, multiple-choice question (MCQ) responses. We also show how these measures change over time. METHODS: With each answer on twice yearly MCQ progress tests, medical students provided their certainty of correctness. An insightful student would be more likely to be correct for those answers given with increasing certainty. A safe student would be expected to have a high probability of being correct for answers given with a high certainty. An efficient student would be expected to have a sufficiently low probability of being correct when they have no certainty. The system was developed using first principles and data from one cohort of students. A dataset from a second cohort was then used as an independent validation sample. RESULTS: The patterns of aspects of self-monitoring were similar for both cohorts. Almost all the students met the criteria for insightfulness on all tests. Most students had an undetermined outcome for the safety aspect. When a definitive result for safety was obtained, absence of safety was most prevalent in the middle of the course, while the presence of safety increased later. Most of the students met the criteria for efficiency, with the highest prevalence mid-course, but efficiency was more likely to be absent later. DISCUSSION: Throughout the course, students showed reassuring levels of insightfulness. The results suggest that students may balance safety with efficiency. This may be explained by students learning the positive implications of decisions before the negative implications, making them initially more efficient, but later being more cautious and safer.

Making accommodations for medical students' long-term conditions in assessments: An action research guided approach.

PURPOSE: There is an increase in medical students with long term conditions applying for accommodations in assessment. Medical school responses to such applications appear inconsistent, possibly reflecting insufficient guidance for policy. We aimed to inform an approach by developing guidance. METHODS: Within a New Zealand and Australian context, we used a four-stage action research methodology: discussion with medical educators and survey of all 21 medical schools, including responding to 22 hypothetical scenarios; developing an approach; applying the approach to the hypothetical scenarios; and seeking feedback from stakeholders on the proposed approach. RESULTS: Current practice varied among the 13 responding medical schools. Medical schools were consistent in their responses for 10 hypothetical scenarios but inconsistent in 12. An approach based on a matrix of authenticity to practice, including regulatory and employment factors, and feasibility to educational institutions was developed. This was applied to the hypothetical scenarios and highlighted how consistency could be better achieved, and where further discussion between regulators and employers might be needed. CONCLUSION: This approach and the matrix based on authenticity and feasibility provides guidance to consider assessment accommodation applications. It highlights the need for discussions among regulators, employers, educational institutions and the disability sector.

Randomised controlled trial of students access to resources in an examination.

INTRODUCTION: Assessment of healthcare professionals should be authentic to clinical practice. As clinicians regularly use resources in practice, similar resources should be available to those sitting assessment. There is limited information on the impacts of open-book (resource) assessments on standard setting for use in high-stakes assessments. This research aims to explore the effects on standard setting and student perceptions when open vs closed resources are available in high-stakes assessment of medical students. METHODS: Students sat multiple-choice question (MCQ) examinations under both closed- and open-resource conditions in a randomised crossover design. A standard setting panel set pass-marks for both closed- and open-resource conditions of delivery, and we compared these pass-marks with each other and with actual performance. Students responded to a survey on perceptions of open-resource assessments. RESULTS: The pass-mark was set higher when panellists considered open-resource conditions compared to closed conditions (59% vs 47%), but actual student performance showed no difference in scores between the two conditions. The net effect was that the pass rate was higher for closed than open-resource conditions (71% vs 34%). Open-resource conditions increased the time to complete the questions. The students perceived that open resource was more authentic but was more time-consuming and would require different preparation. Regarding the acceptability of including open resources in high-stakes assessment, the responses of students were mixed. DISCUSSION: Pass standards based solely on judgements by panellists experienced in closed-resource conditions might not be applicable under open conditions. Questions vary in how much time accessing resources may take and the degree of assistance in selecting the correct answer. A programme of assessment could be constructed to include both closed- and open-resource condition assessments depending on the questions' content and format. Open-resource conditions may promote assessment preparation that focuses more on seeking and evaluating resources rather than learning facts.

Medical student learner neglect in the clinical learning environment: Applying Glaser's theoretical model.

INTRODUCTION: Learner neglect is a relatively new concept in education, and no suitable framework for its exploration has been devised. The aim of this study was to determine whether an existing framework, Glaser's framework of child neglect, could be applied to learner neglect in clinical learning environments. This was a retrospective analysis of data obtained as part of a related study. METHOD: Six focus groups were conducted with medical students in their early clinical years to explore their views of what experiences in medical education were challenging and why they presented a challenge. The transcript data were analysed using inductive content analysis, within an interpretivist approach in the development of categories. The identified categories were cross referenced with Glaser's framework categories replacing the carer with the teacher and the child with the learner. RESULTS: Glaser's classifications of teacher (parent) behaviours were all identified in the negative aspects of medical learner clinical education including emotional unavailability/unresponsiveness, acting in a hostile manner, inappropriate inconsistent developmental interaction, failure to recognise individuality and failure to promote social adaption. Physical unavailability was identified as an additional category and is included in our proposed framework of learner neglect. DISCUSSION: Adapting Glaser's framework was useful in considering learner neglect. Medical schools have a role in ensuring learning experiences are positive across contexts and to make explicit to teachers any behaviours that may appear as learner neglect. Applying this framework has the potential to make more explicit any subtle undermining teacher behaviours. Once explicit, there is a greater likelihood that behaviours may be reappraised both by the teacher and learner and modified to promote a more effective clinical learning experience.

Towards a Definition of Distinction in Professionalism.

Phenomenon: Professionalism can be characterized by a particular set of attributes that clinicians demonstrate in practice. Although much has been described on those attributes that define acceptable professionalism, the characteristics that define distinction in professionalism have not yet been well defined. Approach: In this exploratory project, qualitative methods were used to triangulate three sources of data collected from three campuses of one medical school: student assessment summaries, teacher interviews, and an institutional policy. Findings: One hundred-thirty student assessment summaries, eight teacher interviews, and one institutional policy were analyzed. Three characteristics emerged that define distinction in professionalism: improvement of oneself, helping others learn, and teamwork. These characteristics are in addition to students demonstrating a clear minimum standard in all other aspects of professionalism. Insights: Findings from this project offer a first step toward a definition of distinction in professionalism for assessing student performance. The characteristics can be demonstrated by students to varying degrees of proficiency and are potentially achievable by all students. Finally, the characteristics would be required in addition to demonstrating a clear minimum standard of performance in all other aspects of professionalism and cannot be inferred by the absence of negative or unprofessional behaviors. Recognizing that conceptions of professionalism have contextual and cultural influences, the characteristics of distinction identified by this project expand the language available for teachers and learners to discuss professionalism. Teachers may use these characteristics to help inform their teaching, learning, and feedback practices. Students will gain clarity about the expectations regarding their professional behavior.

Defining and tracking medical student self-monitoring using multiple-choice question item certainty.

BACKGROUND: Self-monitoring is an important component of clinical practice. It is underpinned by the framework of self-efficacy which is concerned with judgments of how well one believes one can achieve or perform a task. This research aimed to develop criteria for adequate self-monitoring, then to measure patterns of self-monitoring, and to explore how these patterns relate to a student's year in a medical course and to patterns of knowledge. METHODS: Analysis of individuals' levels of correctness in answering assessment items and their certainty in correctness may be used to inform assessments of ability to self-monitor. Two criteria were proposed and applied to define adequate self-monitoring. Firstly, increasing proportions correct with increasing levels of certainty. Secondly, having a proportion correct for high certainty responses that was not lower than cohort levels. Student responses in progress tests comprising multiple-choice questions (MCQs) and associated certainty were analysed. Criteria for the presence of adequate self-monitoring and for adequate knowledge were applied to the results of each of four tests conducted over 2 years, and used to categorise patterns of self-monitoring and knowledge. RESULTS: Data from 3 year group cohorts totalling 737 students were analysed. The majority (58%) of students demonstrated adequate knowledge and met both criteria for adequate self-monitoring across all four tests. The most advanced year group cohort had the highest rates of adequate knowledge and the highest rates of meeting both self-monitoring criteria. The patterns of self-monitoring were the same as the patterns of knowledge across the four tests for 454 students, but for the remaining 283 the patterns of self-monitoring and knowledge differed. CONCLUSION: Analysis of responses to item level certainty has informed development of a definition of adequate self-monitoring that may be applied to individual student's responses from a single test, and to track the adequacy of a student's self-monitoring over time. Patterns of self-monitoring tend to match patterns of knowledge, but not in all cases, suggesting the self-monitoring measure could provide additional information about student ability.

Adding to the debate on the numbers of options for MCQs: the case for not being limited to MCQs with three, four or five options.

BACKGROUND: There is a significant body of literature that indicates that the number of options for single-best answer multiple choice questions (MCQs) can be reduced from five to three or four without adversely affecting the quality of the questions and tests. Three or four options equates to two or three distractors respectively. MAINTEXT: Whilst these arguments may be true when focusing on psychometric aspects of questions, we should also focus on educational and clinical authenticity aspects of questions. I present reasons for MCQs in tests to have a variable number of options which will usually be more than three, four, or five. These include: decisions related to broad clinical scenarios cannot be limited to a small number of options; options lists should include all possible combinations of option elements; and options that are rarely chosen can provide information regarding students and/or for students. CONCLUSION: Finally, given computer based delivery, longer option lists are not impractical for examinees. In the contexts that are appropriate, it is time to consider a move to adopting appropriate and variable numbers of MCQ options and not be limited to MCQs with three, four or five options.

Student progress decision-making in programmatic assessment: can we extrapolate from clinical decision-making and jury decision-making?

BACKGROUND: Despite much effort in the development of robustness of information provided by individual assessment events, there is less literature on the aggregation of this information to make progression decisions on individual students. With the development of programmatic assessment, aggregation of information from multiple sources is required, and needs to be completed in a robust manner. The issues raised by this progression decision-making have parallels with similar issues in clinical decision-making and jury decision-making. MAIN BODY: Clinical decision-making is used to draw parallels with progression decision-making, in particular the need to aggregate information and the considerations to be made when additional information is needed to make robust decisions. In clinical decision-making, diagnoses can be based on screening tests and diagnostic tests, and the balance of sensitivity and specificity can be applied to progression decision-making. There are risks and consequences associated with clinical decisions, and likewise with progression decisions. Both clinical decision-making and progression decision-making can be tough. Tough and complex clinical decisions can be improved by making decisions as a group. The biases associated with decision-making can be amplified or attenuated by group processes, and have similar biases to those seen in clinical and progression decision-making. Jury decision-making is an example of a group making high-stakes decisions when the correct answer is not known, much like progression decision panels. The leadership of both jury and progression panels is important for robust decision-making. Finally, the parallel between a jury's leniency towards the defendant and the failure to fail phenomenon is considered. CONCLUSION: It is suggested that decisions should be made by appropriately selected decision-making panels; educational institutions should have policies, procedures, and practice documentation related to progression decision-making; panels and panellists should be provided with sufficient information; panels and panellists should work to optimise their information synthesis and reduce bias; panellists should reach decisions by consensus; and that the standard of proof should be that student competence needs to be demonstrated.

Low performing students have insightfulness when they reflect-in-action.

CONTEXT: Measuring appropriateness of certainty of responses in a progress test using descriptors authentic to practice as reflection-in-action builds on existing theories of self-monitoring. Clinicians making decisions require the ability to accurately self-monitor, including certainty of being correct. Inappropriate certainty could lead to medical error. Self-assessment and certainty of assessment performance have been measured in a variety of ways. Previous work has shown that those with less experience are less accurate in self-assessment, but such studies looked at self-assessment using methods less authentic to clinical practice. This study investigates how correctness varies with certainty, allowing for experience and performance. METHODS: Students in Years 2-5 were certain of their responses to two iterations of a progress test during one calendar year. Analyses compared correctness for certainty of response, test number, student year cohort and performance level, defined by criterion scores. RESULTS: The odds of a correct response increased with student certainty for all subsets allowing for year group and ability, including student subsets with less experience and subsets in lower-performance groups. CONCLUSION: Unlike previous work showing poorer accuracy of self-assessment for those with less experience or ability, we postulate that our finding of similar increases in correctness with increasing certainty even in the less experienced and lower performance groups, relates to certainty descriptors being worded in a way that is authentic to clinical practice, and in turn related to reflection-in-action.

Decision-making bias in assessment: the effect of aggregating objective information and anecdote.

INTRODUCTION: Assessment decisions increasingly rely on synthesis of information from a variety of sources. It is known that aggregation of information to make decisions is open to a number of biases. The aim of this research was to investigate bias, accuracy and confidence of assessment decision making. METHODS: The participants were consultation skills assessors. A model for incremental information was developed with participants being shown results from purposefully selected, but authentic, data from the University's final summative 10-station Objective Structured Clinical Examination (OSCE). After each piece of information, participants gave a pass-fail decision and their confidence in that choice. Following the information from 10 OSCE stations the participants were given a discordant fictional anecdote and again participants gave a pass-fail decision and their confidence. RESULTS: When there is overwhelming evidence to support a pass or fail, participants were not as confident as the data would support. Participants were less confident to make a fail decision than a pass. Despite considerable evidence from multiple results some participants altered decisions based on isolated contradictory information from an anecdote. DISCUSSION: These findings are significant in understanding decision-making. Given equivalent levels of evidence, decision makers are less confident to fail than pass and less robust information can undermine more robust information.

Determining measures of insight and foresight from responses to multiple choice questions.

BACKGROUND: Clinical decisions require insight and foresight. When these are lacking, overconfidence and error can occur. AIMS: Multiple choice questions (MCQ) responses were used to estimate insight, as determined by confidence in responses, and foresight, as determined by the safety of incorrect responses. METHODS: An MCQ response system was developed which reflected confidence in, and safeness of, responses. An optional examination incorporating this system was offered to medical students at one university. RESULTS: A total of 372 students completed the study. There was evidence of insight. The proportion of responses that were correct rose from 32% for low confidence to 77% for high confidence. The proportion that was incorrect and not unsafe fell from 34% for low confidence to 7% for high confidence. There was less evidence of foresight. Of the responses that were incorrect, the proportion that was unsafe increased from 52% for low confidence to 68% for high confidence. Measures of insight and foresight increased with experience. Students with poorer knowledge chose more unsafe responses, but with lower confidence. Five students gave multiple highly unsafe responses held with high confidence. CONCLUSIONS: Students had insight but less foresight. Experience and knowledge altered measures of both. This response system identified students who appeared to lack both.

Joining the dots: conditional pass and programmatic assessment enhances recognition of problems with professionalism and factors hampering student progress.

BACKGROUND: Programmatic assessment that looks across a whole year may contribute to better decisions compared with those made from isolated assessments alone. The aim of this study is to describe and evaluate a programmatic system to handle student assessment results that is aligned not only with learning and remediation, but also with defensibility. The key components are standards based assessments, use of "Conditional Pass", and regular progress meetings. METHODS: The new assessment system is described. The evaluation is based on years 4-6 of a 6-year medical course. The types of concerns staff had about students were clustered into themes alongside any interventions and outcomes for the students concerned. The likelihoods of passing the year according to type of problem were compared before and after phasing in of the new assessment system. RESULTS: The new system was phased in over four years. In the fourth year of implementation 701 students had 3539 assessment results, of which 4.1% were Conditional Pass. More in-depth analysis for 1516 results available from 447 students revealed the odds ratio (95% confidence intervals) for failure was highest for students with problems identified in more than one part of the course (18.8 (7.7-46.2) p < 0.0001) or with problems with professionalism (17.2 (9.1-33.3) p < 0.0001). The odds ratio for failure was lowest for problems with assignments (0.7 (0.1-5.2) NS). Compared with the previous system, more students failed the year under the new system on the basis of performance during the year (20 or 4.5% compared with four or 1.1% under the previous system (p < 0.01)). CONCLUSIONS: The new system detects more students in difficulty and has resulted in less "failure to fail". The requirement to state conditions required to pass has contributed to a paper trail that should improve defensibility. Most importantly it has helped detect and act on some of the more difficult areas to assess such as professionalism.

Observed consultation: confidence and accuracy of assessors.

Judgments made by the assessors observing consultations are widely used in the assessment of medical students. The aim of this research was to study judgment accuracy and confidence and the relationship between these. Assessors watched recordings of consultations, scoring the students on: a checklist of items; attributes of consultation; a passmark scale and lastly their confidence in this last judgment. Then they were interviewed using stimulated recall to explain their rationale for scoring and confidence. Twenty-three staff assessors watched two consultations. Assessor confidence and accuracy were least for those student performances perceived near the passmark standard. The difference between confidence and accuracy, over-confidence, was greatest at this level, although at the extremes under-confidence was found. In the interviews the assessors were aware of a variety of factors that they perceived affected their scoring and confidence. As in other contexts confidence and accuracy vary and over-confidence increases the more difficult the judgments. However, this study also demonstrated under-confidence for less difficult judgments. The perception that more information would help, may improve confidence but not necessarily accuracy, so increasing over-confidence.

A randomized controlled trial comparing instructions regarding unsafe response options in a MCQ examination.

BACKGROUND: Marking of multiple choice type examinations often just takes account of the correct responses. This may encourage guessing of incorrect and potentially unsafe responses. Ideally responses should contain a high proportion of correct, unsafe and the use of 'don't know' response rather than incorrect. AIM: This study explored the effect of instructions on responses. METHODS: Fourth- and fifth-year students sitting for an optional multiple choice examination were randomized to receive one of four instruction options: number-correct marking (the control group); alert to unsafe; mark deduction for unsafe; or correction for guessing. RESULTS: A total of 210 students sat the test. For the fourth-year cohort, compared with the control group, being alerted to unsafe and mark deduction for unsafe or incorrect responses were associated with graduated increases in the use of 'don't know' and reductions in incorrect responses. For the fifth-year cohort, there were no differences in responses between options. The fifth-year cohort, gave more correct, and fewer incorrect, unsafe and 'don't know' responses. CONCLUSIONS: Both the year group and instructions had an effect. Being alerted to potentially unsafe responses, even when there is no mark deduction penalty, had an effect, although mark deduction still had the greatest effect. Assessment instructions may give subliminal messages that have important consequences.

Considering evidence for ethnicity bias using assessment case scenarios and medical student correctness and certainty.

AIMS: There is inequitable distribution of health risks, exposures, resources and outcomes by ethnicity. This may be contributed to by health professional bias. The aim of this study was to investigate the relationship between ethnicity of patients, within written assessment case scenarios, and medical students' response correctness and certainty. METHODS: Otago Medical School students sit a 150 MCQ progress test with item level response certainty. Patient ethnicity for 60 MCQ case scenarios was varied between two ethnic groups (New Zealand European, Maori) and none specified. Analysis of responses by patient ethnicity was undertaken to compare: odds of correctness; level of certainty; correctness for level of certainty and also by year groups and ability. RESULTS: One thousand one hundred and three students sat the test. There was no significant difference in odds of correctness or level of certainty by the ethnicity of the patient case scenario. These did not differ significantly by year groups or ability groups, or for correctness by level of certainty. CONCLUSIONS: No systematic differences in correctness or certainty of student responses to case scenarios by patient ethnicity were detected. Further exploration is warranted, including incorporating more ethnicity descriptors, analysis of incorrect answers, analyses for patterns responses over time by individual students, and selecting questions where varying patient ethnicity is expected to alter the correct response or difficulty.

Intensive care nurses' knowledge of pressure ulcers: development of an assessment tool and effect of an educational program.

BACKGROUND: Critically ill patients are at high risk for pressure ulcers. Successful prevention of pressure ulcers requires that caregivers have adequate knowledge of this complication. OBJECTIVE: To assess intensive care nurses' knowledge of pressure ulcers and the impact of an educational program on knowledge levels. METHODS: A knowledge assessment test was developed. A cohort of registered nurses in a tertiary referral hospital in New Zealand had knowledge assessed 3 times: before an educational program, within 2 weeks after the program, and 20 weeks later. Multivariate analysis was performed to determine if attributes such as length of time since qualifying or level of intensive care unit experience were associated with test scores. The content and results of the assessment test were evaluated. RESULTS: Completion of the educational program resulted in improved levels of knowledge. Mean scores on the assessment test were 84% at baseline and 89% following the educational program. The mean baseline score did not differ significantly from the mean 20-week follow-up score of 85%. No association was detected between demographic data and test scores. Content validity and standard setting were verified by using a variety of methods. CONCLUSION: Levels of knowledge to prevent and manage pressure ulcers were good initially and improved with an educational program, but soon returned to baseline.

Diagnostic testing and educational assessment.

BACKGROUND: Clinicians are familiar with making diagnostic decisions based on information gathered from history, clinical examination and diagnostic tests. Although many clinicians assess students, they may be less familiar with ways to assimilate assessment information to inform educational decisions. We draw parallels between the processes used to make a clinical diagnosis and the similar processes needed to make an educational decision. CONTEXT: There are several indices that describe the performance and utility of diagnostic tests, which we have extrapolated to educational assessment. INNOVATION: We provide a clinical diagnostic question and an education assessment question, and use examples of indices of performance and utility for both of these situations to explore: reliability, indeterminate results, certainty in decisions, acceptable levels of sensitivity and specificity, pre-test probability and dealing with limitations. Test reliability requires adequate sampling and consistency between observers. Seeking more information should be targeted to situations where decisions are not certain. Altering score cut-points alters test sensitivity and specificity, which in assessment will alter the numbers of falsely passing or falsely failing candidates. Just as the pre-test probability of a diagnosis influences how to interpret diagnostic tests, so too does the pre-test probability of failure alter the performance characteristics of assessments. In clinical situations, a 'wait and see' approach may be limited by clinical urgency. Likewise, in assessment the 'wait and see' approach may be limited by a duty to society. IMPLICATIONS: Clinicians familiar with the performance and utility of diagnostic tests can extrapolate that knowledge to make better interpretations of educational assessments.

Developing the One-Minute Preceptor.

BACKGROUND: Learning from experienced doctors in real clinical settings is very important for medical students. However, the busy and at times unpredictable nature of clinical work means that clinical work must take priority over teaching. What clinicians want is to be able to offer quality learning experiences for students without significant disruption to their clinical work. CONTEXT: In the context of medical education, students are learning in a variety of physical locations. These various locations require different sets of teaching skills. This article describes how as faculty educational developers we worked with clinicians to enhance their role as teachers within busy clinical contexts. More specifically, we will describe how we augmented an established programme of travelling workshops for clinical teachers by incorporating the key principles associated with the development of the One-Minute Preceptor. INNOVATION AND IMPLICATIONS: We combined classroom training with observation of teaching in the clinical area, and by doing so were more able to translate classroom theory into authentic workplace practice.