When guidelines don't guide: the effect of patient context on management decisions based on clinical practice guidelines.

PURPOSE: This study examines the influence of patient social context on physicians' adherence to clinical practice guidelines (CPGs). METHOD: Expert emergency medicine (EM) physicians and novice physicians (EM residents) were surveyed using an Internet-based program between January and July of 2013. Participants were presented clinical cases and were asked to indicate if they would order or prescribe a specified test or treatment. Cases were chosen from four domains where CPGs exist, and were constructed to include or exclude a "context variable" (CV). Both expert and novice physicians' CPG adherence rate in the CV condition was compared with that in the no CV condition. The CPG adherence rates in CV and no CV conditions were also compared between expert and novice EM physicians. RESULTS: Expert EM physicians (n = 28) were less likely to adhere to CPGs in the CV condition compared with the no CV condition (56% versus 80%, respectively; odds ratio [OR] = 0.32, 95% confidence interval [CI]: 0.17-0.53, P < .001). Experts were less likely to adhere to CPGs in the CV condition when compared with novice physicians (n = 28) (56% versus 67%; OR = 0.62, 95% CI: 0.39-1.0, P = .039). Expert and novice EM physicians did not differ in their adherence to CPGs in the no CV condition. CONCLUSIONS: Participants were sensitive to both the best clinical evidence of benefit, as recommended by CPGs, and patient context when determining how care should be managed.

"Precision" and "accuracy": two terms that are neither.

OBJECTIVE: In some publications, the terms "precision" and "accuracy" are used as if they were synonyms for "reliability" and "validity." METHODS AND RESULTS: This article shows that these terms are neither precise nor accurate when used in this way. Scales can demonstrate high test-retest or interrater reliability (i.e., they are "precise") but still be unreliable in certain circumstances; and "imprecise" scales can still show good reliability. Further, "accuracy" as a synonym for validity reflects an outdated conceptualization of validity, which has been superseded by one that emphasizes that validity tells us what conclusions can be drawn about a person based on a test result. CONCLUSION: The article ends with a call for the use of the more traditional terms as better reflecting the process of scale development and the uses to which they are put.

How can I know what I don't know? Poor self assessment in a well-defined domain.

As the rapidity with which medical knowledge is generated and disseminated becomes amplified, an increasing emphasis has been placed on the need for physicians to develop the skills necessary for life-long learning. One such skill is the ability to evaluate one's own deficiencies. A ubiquitous finding in the study of self-assessment, however, is that self-ratings are poorly correlated with other performance measures. Still, many educators view the ability to recognize and communicate one's deficiencies as an important component of adult learning. As a result, two studies have been performed in an attempt to improve upon this status quo. First, we tried to re-define the limits within which self-assessments should be used, using Rosenblit and Keil's argument that calibration between perceived and actual performance will be better within taxonomies that are regularly tested (e.g., factual knowledge) compared to those that are not (e.g., conceptual knowledge). Second, we tried to norm reference individuals based on both the performance of their colleagues and their own historical performance on McMaster's Personal Progress Inventory (a multiple choice question test of medical knowledge). While it appears that students are able to (a) make macro-level self-assessments (i.e., to recognize that third year students typically outperform first year students), and (b) judge their performance relatively accurately after the fact, students were unable to predict the percentage of questions they would answer correctly with a testing procedure in which they have had a substantial amount of feedback. Previous test score was a much better predictor of current test performance than were individuals' expectations.

An admissions OSCE: the multiple mini-interview.

CONTEXT: Although health sciences programmes continue to value non-cognitive variables such as interpersonal skills and professionalism, it is not clear that current admissions tools like the personal interview are capable of assessing ability in these domains. Hypothesising that many of the problems with the personal interview might be explained, at least in part, by it being yet another measurement tool that is plagued by context specificity, we have attempted to develop a multiple sample approach to the personal interview. METHODS: A group of 117 applicants to the undergraduate MD programme at McMaster University participated in a multiple mini-interview (MMI), consisting of 10 short objective structured clinical examination (OSCE)-style stations, in which they were presented with scenarios that required them to discuss a health-related issue (e.g. the use of placebos) with an interviewer, interact with a standardised confederate while an examiner observed the interpersonal skills displayed, or answer traditional interview questions. RESULTS: The reliability of the MMI was observed to be 0.65. Furthermore, the hypothesis that context specificity might reduce the validity of traditional interviews was supported by the finding that the variance component attributable to candidate-station interaction was greater than that attributable to candidate. Both applicants and examiners were positive about the experience and the potential for this protocol. DISCUSSION: The principles used in developing this new admissions instrument, the flexibility inherent in the multiple mini-interview, and its feasibility and cost-effectiveness are discussed.

The truly remarkable universality of half a standard deviation: confirmation through another look.

In this issue of Expert Review of Pharmacoeconomics and Outcomes Research, Farivar, Liu, and Hays present their findings in 'Another look at the half standard deviation estimate of the minimally important difference in health-related quality of life scores (hereafter referred to as 'Another look') . These researchers have re-examined the May 2003 Medical Care article 'Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation' (hereafter referred to as 'Remarkable') in the hope of supporting their hypothesis that the minimally important difference in health-related quality of life measures is undoubtedly closer to 0.3 standard deviations than 0.5. Nonetheless, despite their extensive wranglings with the exclusion of many articles that we included in our review; the inclusion of articles that we did not include in our review; and the recalculation of effect sizes using the absolute value of the mean differences, in our opinion, the results of the 'Another look' article confirm the same findings in the 'Remarkable' paper.

Self and peer assessment in tutorials: application of a relative-ranking model.

PURPOSE: While self assessment continues to be touted as being of paramount importance for continuing professional competence, problem-based learning curricula, and adult learning theory, techniques for ensuring valid judgments have proven elusive. This study tested the applicability of an innovative relative-ranking procedure to problem-based learning tutorials. METHOD: A total of 36 students in the McMaster University Faculty of Health Sciences' MD program were provided relative-ranking forms listing seven domains of competence along with their definitions. The student, two of the student's peers, and the student's tutor were asked to complete the ranking exercise after their second, fourth, and sixth tutorials. RESULTS: Combining each level of the time and rater variables generated 66 correlation coefficients, none of which was significantly different from zero. Re-performing the analysis on only the extreme domains did not improve this result. CONCLUSION: The relative-ranking instrument developed did not prove to be a reliable measure of tutorial performance. Ratings were inconsistent from one week to the next as well as across raters within a week.