Assessing the acceptability of script concordance testing: a nationwide study in otolaryngology.

Background: Script concordance testing (SCT) is an objective method to evaluate clinical reasoning that assesses the ability to interpret medical information under conditions of uncertainty. Many studies have supported its validity as a tool to assess higher levels of learning, but little is known about its acceptability to major stakeholders. The aim of this study was to determine the acceptability of SCT to residents in otolaryngology ­ head and neck surgery (OTL-HNS) and a reference group of experts. Methods: In 2013 and 2016, a set of SCT questions, as well a post-test exit survey, were included in the National In-Training Examination (NITE) for OTL-HNS. This examination is administered to all OTL-HNS residents across Canada who are in the second to fifth year of residency. The same SCT questions and survey were then sent to a group of OTL-HNS surgeons from 4 Canadian universities. Results: For 64.4% of faculty and residents, the study was their first exposure to SCT. Overall, residents found it difficult to adapt to this form of testing, thought that the clinical scenarios were not clear and believed that SCT was not useful for assessing clinical reasoning. In contrast, the vast majority of experts felt that the test questions reflected real-life clinical situations and would recommend SCT as an evaluation method in OTL-HNS. Conclusion: Views about the acceptability of SCT as an assessment tool for clinical reasoning differed between OTL-HNS residents and experts. Education about SCT and increased exposure to this testing method are necessary to improve residents' perceptions of SCT.

Contexte: Le test de concordance de script (TCS) est une méthode objective d'évaluation du raisonnement clinique qui mesure la capacité d'interpréter les renseignements médicaux en contexte d'incertitude. Beaucoup d'études en appuient la validité en tant qu'outil pour évaluer l'enseignement supérieur, mais on en sait peu sur son acceptabilité auprès des principales parties prenantes. Le but de cette étude était de déterminer l'acceptabilité du TCS chez les résidents en otorhinolaryngologie ­ chirurgie de la tête et du cou (ORL ­ chirurgie tête et cou) et un groupe de référence composé d'experts. Méthodes: En 2013 et 2016, une série de questions de TCS, de même qu'un questionnaire post-test, ont été inclus dans l'examen national en cours de formation NITE (National In-Training Examination) pour l'ORL ­ chirurgie tête et cou. Cet examen est administré à tous les résidents en ORL ­ chirurgie tête et cou au Canada qui sont entre leurs deuxième et cinquième années de résidence. Les mêmes questions de TCS ont été envoyées à un groupe de chirurgiens en ORL ­ chirurgie tête et cou de 4 université canadiennes. Résultats: Pour 64,4 % des membres facultaires et des résidents, l'étude était leur première exposition au TCS. Dans l'ensemble, les résidents ont trouvé difficile de s'adapter à cette forme de test, même si les scénarios cliniques étaient clairs, et ils ont estimé que le TCS était peu utile pour évaluer le raisonnement clinique. En revanche, la grande majorité des experts ont jugé que les questions du test reflétaient la réalité des cas cliniques et recommanderaient le TCS comme méthode d'évaluation en ORL ­ chirurgie tête et cou. Conclusion: Entre les résidents et les experts en ORL ­ chirurgie tête et cou, les points de vue quant à l'acceptabilité du TCS comme outil d'évaluation du raisonnement clinique ont différé et il faudrait y exposer les résidents davantage pour améliorer leur perception du TCS.

Rheumatoid Meningitis Presenting With Acute Parkinsonism and Protracted Non-convulsive Seizures: An Unusual Case Presentation and Review of Treatment Strategies.

Rheumatoid meningitis is a rare complication of rheumatoid arthritis (RA). It is associated with substantial morbidity and mortality. The condition may present in a variety of ways and is therefore diagnostically challenging. Uncertainty still exists regarding the optimal treatment strategy. Herein, we describe the case of a 74-year-old man with a history of well-controlled seropositive RA on low-dose prednisone, hydroxychloroquine, and methotrexate. The patient presented with a several-month history of multiple prolonged episodes of expressive aphasia, right hemiparesis, and encephalopathy. Although no epileptiform activity was recorded on repeated electroencephalography, the symptoms fully resolved following treatment with antiepileptic drugs. He subsequently developed acute asymmetrical parkinsonism of the right hemibody. Magnetic resonance imaging revealed subtle enhancement of the leptomeninges over the left frontoparietal convexity. Cerebrospinal fluid analysis revealed a mild lymphocytic pleocytosis and elevated proteins. Histopathologic analysis of a meningeal biopsy revealed nodular rheumatoid meningitis. The patient was treated with corticosteroids and cyclophosphamide, following which he incompletely recovered. This is the first description of rheumatoid meningitis manifesting with acute parkinsonism and protracted non-convulsive seizures. A summary of cases reported since 2005, including data on pathology, therapy and outcomes, along with a discussion on the efficacy of different treatment strategies are provided.

Optimization of answer keys for script concordance testing: should we exclude deviant panelists, deviant responses, or neither?

The Script Concordance Test (SCT) uses a panel-based, aggregate scoring method that aims to capture the variability of responses of experienced practitioners to particular clinical situations. The use of this type of scoring method is a key determinant of the tool's discriminatory power, but deviant answers could potentially diminish the reliability of scores by introducing measurement error. (1) to investigate the effects on SCT psychometrics of excluding from the test's scoring key either deviant panelists or deviant answers; (2) to propose a method for excluding either deviant panelists or deviant answers. Using an SCT in radiation oncology, we examined three methods for reducing panel response variability. One method ('outliers') entailed removing from the panel members with very low total scores. Two other methods ('distance-from-mode' and 'judgment-by-experts') excluded widely deviant responses to individual questions from the test's scoring key. We compared the effects of these methods on score reliability, correlations between original and adjusted scores, and between-group effect sizes (panel-residents; panel-students; and residents-students). With a large panel (n = 45), optimization methods have no effect on reliability of scores, correlation and effect size. With a smaller panel (n = 15) no significant effect of optimization methods were observed on reliability and correlation, but significant variation on effect size was observed across samples. Measurement error resulting from deviant panelist responses on SCTs is negligible, provided the panel size is sufficiently large (>15). However, if removal of deviant answers is judged necessary, the distance-from-mode strategy is recommended.

Assessment in the context of uncertainty using the script concordance test: more meaning for scores.

BACKGROUND: The Script Concordance Test (SCT) uses authentic, ill-defined clinical cases to compare medical learners' judgment skills with those of experienced physicians. SCT scores are meant to measure the degree of concordance between the performance of examinees and that of the reference panel. Raw test scores have meaning only if statistics (mean and standard deviation) describing the panel's performance are concurrently provided. PURPOSE: The purpose of this study is to suggest a method for reporting scores that standardizes panel mean and standard deviation, allowing examinees to immediately gauge their performance relative to panel members. METHODS: Based on a statistical method of standardization, a new method for computing SCT scores is described. According to this method, test raw scores are converted into a scale in which the panel mean is set as the value of reference, and the standard deviation of the panel serves as a yardstick by which examinee performance is measured. RESULTS: The effect of this transformation on four data sets obtained from SCTs in radio-oncology, surgery, neurology, and nursing is discussed. CONCLUSION: This transformation method proposes a common metric basis for reporting SCT scores and provides examinees with clear, interpretable insights into their performance relative to that of physicians of the field. We recommend reporting SCT scores with the mean and standard deviation of panel scores set at standard scores of 80 and 5, respectively. Beyond SCT, our transformation method may be generalizable to the scoring of other test formats in which the performance of examinees and those of a panel of reference undertaking the same cognitive tasks are compared.