Twelve tips for developing key-feature questions (KFQ) for effective assessment of clinical reasoning.

Clinical reasoning is the cognitive process that makes it possible for us to reach conclusions from clinical data. "A key feature (KF) is defined as a significant step in the resolution of a clinical problem. Examinations using key-feature questions (KFQs) focus on a challenging aspect in the diagnosis and management of a clinical problem where the candidates are most likely to make errors." KFs have been used at different levels of medical education and practice, from undergraduate to certification examinations. KFQs illuminate the strengths and limits of an individual's clinical problem-solving ability. These types of items are more likely than other forms of assessment to discriminate among stronger or weaker candidates in the area of clinical reasoning. The 12 tips in this article will provide guidance to faculty who wish to develop KFQs for their tests.

Key-feature questions for assessment of clinical reasoning: a literature review.

OBJECTIVES: Key-feature questions (KFQs) have been developed to assess clinical reasoning skills. The purpose of this paper is to review the published evidence on the reliability and validity of KFQs to assess clinical reasoning. METHODS: A literature review was conducted by searching MEDLINE (1946-2012) and EMBASE (1980-2012) via OVID and ERIC. The following search terms were used: key feature; question or test or tests or testing or tested or exam; assess or evaluation, and case-based or case-specific. Articles not in English were eliminated. RESULTS: The literature search resulted in 560 articles. Duplicates were eliminated, as were articles that were not relevant; nine articles that contained reliability or validity data remained. A review of the references and of citations of these articles resulted in an additional 12 articles to give a total of 21 for this review. Format, language and scoring of KFQ examinations have been studied and modified to maximise reliability. Internal consistency reliability has been reported as being between 0.49 and 0.95. Face and content validity have been shown to be moderate to high. Construct validity has been shown to be good using vector thinking processes and novice versus expert paradigms, and to discriminate between teaching methods. The very modest correlations between KFQ examinations and more general knowledge-based examinations point to differing roles for each. Importantly, the results of KFQ examinations have been shown to successfully predict future physician performance, including patient outcomes. CONCLUSIONS: Although it is inaccurate to conclude that any testing format is universally reliable or valid, published research supports the use of examinations using KFQs to assess clinical reasoning. The review identifies areas of further study, including all categories of evidence. Investigation into how examinations using KFQs integrate with other methods in a system of assessment is needed.

IMG candidates' demographic characteristics as predictors of CEHPEA CE1 results.

OBJECTIVE: To assess the extent to which demographic characteristics are related to international medical graduate (IMG) candidate performance on the Centre for the Evaluation of Health Professionals Educated Abroad General Comprehensive Clinical Examination 1 (CE1). DESIGN: Retrospective study. SETTING: Toronto, Ont. PARTICIPANTS: All IMG candidates who registered for and took the CE1 in 2007 (n = 430), 2008 (n = 480), and 2009 (n = 472) were included in this analysis. All candidates completed the Centre for the Evaluation of Health Professionals Educated Abroad CE1, a 12-station objective structured clinical examination. MAIN OUTCOME MEASURES: Mean (SD) examination scores for groups based on demographic variables (age, region of medical training, and Medical Council of Canada Qualifying Examination Part 1 [MCCQE1] score) were calculated. Analysis of variance was done using CE1 examination total scores as the dependent variables. RESULTS: Candidates from countries where both medical education and patient care are conducted in English and those from South America and Western Europe achieved the highest scores, while candidates from the Western Pacific region and Africa achieved the lowest scores. Younger candidates achieved higher scores than older candidates. These results were consistent across the 3 years of CE1 examination administration. There was a significant relationship between MCCQE1 and CE1 scores in 2 of the 3 years: 2007 (r = 0.218, P < .001) and 2008 (r = 0.23, P < .01). CONCLUSION: The CE1 includes an assessment of communication skills; hence it is reasonable that candidates with stronger English skills have the highest scores on the CE1. Age, as a proxy for time since graduation, also has a substantial effect on examination scores, possibly owing to those further from their training lacking some currency of knowledge or being in focused rather than general practices. It is reasonable that those who had higher scores on the written test (the MCCQE1) would also have higher scores on the clinical test (the CE1). Demographic characteristics appear to be related to performance on the CE1.

Validation of a large-scale clinical examination for international medical graduates.

OBJECTIVE: To evaluate a new examination process for international medical graduates (IMGs) to ensure that it is able to reliably assign candidates to 1 of 4 competency levels, and to determine if a global rating scale can accurately stratify examinees into 4 levels of learners: clerks, first-year residents, second-year residents, or practice ready. DESIGN: Validation study evaluating a 12-station objective structured clinical examination. SETTING: Ontario. PARTICIPANTS: A total of 846 IMGs, and an additional 63 randomly selected volunteers from 2 groups: third-year clinical clerks (n = 42) and first-year family medicine residents (n = 21). MAIN OUTCOME MEASURES: The accuracy of the stratification of the examinees into learner levels, the impact of the patient-encounter ratings and postencounter oral questions, and between-group differences in total score. RESULTS: Reliability of the patient-encounter scores, postencounter oral question scores, and the total between-group difference scores was 0.93, 0.88, and 0.76, respectively. Third-year clerks scored the lowest, followed by the IMGs. First-year residents scored highest for all 3 scores. Analysis of variance demonstrated significant between-group differences for all 3 scores (P < .05). Postencounter oral question scores differentiated among all 3 groups. CONCLUSION: Clinical examination scores were capable of differentiating among the 3 groups. As a group, the IMGs seemed to be less competent than the first-year family medicine residents and more competent than the third-year clerks. The scores generated by the postencounter oral questions were the most effective in differentiating between the 2 training levels and among the 3 groups of test takers.

Practical solutions for implementation of Transition to Practice curricula in a competency-based medical education model.

BACKGROUND: Although transition from residency to practice represents a critical learning stage, there is a paucity of literature to inform local curriculum development and implementation. OBJECTIVES: To describe local curriculum development for Transition to Practice (TTP) for use within a competency-based medical education model, including important content and suitable teaching and assessment strategies. DESIGN: We reviewed the literature to construct a definition and develop initial curriculum content for TTP. We then gathered local residency program directors' views on TTP content, teaching, and assessment via online survey and an international educational conference workshop. RESULTS: We identified 21 important TTP content areas in the literature and analyzed 35 survey responses, representing 33 residency programs. Survey participants viewed Further sophistication of clinical skills, How to set up a practice, and Time management skills as the three most important content areas. Views on content importance varied by program. For teaching and assessment strategies, most respondents preferred: assessing what residents could do, providing real-life practice opportunities, and offering workplace-based assessments. CONCLUSIONS: TTP curricula implementation should reflect nationally set, specialty-specific curriculum elements; locally developed priority content; and assessment and teaching strategies. Individual learner needs and imminent practice context should guide faculty approaches to curriculum delivery.Résumé.

CONTEXTE: Bien que la transition de la résidence à la pratique représente une étape d'apprentissage essentielle, on retrouve peu de littérature pour informer le développement et l'implantation des cursus locaux. OBJECTIFS: Décrire le développement d'un cursus local de Transition vers la pratique (TVP) pour un modèle d'éducation médicale par compétences, incluant le contenu important et des stratégies d'enseignement et d'évaluation adaptées. CONCEPTION: Nous avons recensé les écrits afin d'établir une définition et de développer le contenu initial du cursus de TVP. Nous avons ensuite recueilli l'opinion des directions de programmes de résidence locaux sur le contenu, l'enseignement et l'évaluation du cursus TVP par le biais d'un sondage en ligne et un atelier offert dans le cadre d'une conférence pédagogique internationale. RÉSULTATS: Nous avons identifié 21 sujets importants pour la TVP dans la littérature et nous avons analysé les réponses de 35 sondages représentant 33 programmes de résidence. Les trois sujets considérés les plus importants par les participants au sondage étaient le Perfectionnement avancé des compétences cliniques; Comment débuter sa pratique médicale et les Aptitudes de gestion du temps. Les opinions sur l'importance du contenu variaient selon les programmes. Pour ce qui est des stratégies d'apprentissage et d'enseignement, la plupart des personnes sondées ont exprimé les préférences suivantes : évaluer ce que les résidents sont en mesure de faire; fournir des occasions réelles de pratiquer ; offrir des évaluations en milieu de travail. CONCLUSIONS: La mise en œuvre des cursus TVP devrait refléter des éléments des curriculums propres aux spécialités et définis à l'échelle nationale; des sujets prioritaires développés localement; des stratégies d'enseignement et d'évaluation. Les besoins personnels des étudiants et le contexte imminent du début de pratique devraient orienter les approches du corps professoral utilise dans l'enseignement de ce cursus.

Ambulatory Toxicity Management (AToM) Pilot: results of a pilot study of a pro-active, telephone-based intervention to improve toxicity management during chemotherapy for breast cancer.

BACKGROUND: Chemotherapy is associated with a significant risk of toxicity, which often peaks between ambulatory visits to the cancer centre. Remote symptom management support is a tool to optimize self-management and healthcare utilization, including emergency department visits and hospitalizations (ED+H) during chemotherapy. We performed a single-arm pilot study to evaluate the feasibility, acceptability, and potential impact of a telephone symptom management intervention on healthcare utilization during chemotherapy for early stage breast cancer (EBC). METHODS: Women starting adjuvant or neoadjuvant chemotherapy for EBC at two cancer centres in Ontario, Canada, received standardized, nurse-led calls to assess common toxicities at two time points following each chemotherapy administration. Feasibility outcomes included patient enrollment, retention, RN adherence to delivering calls per the study schedule, and resource use associated with calls; acceptability was evaluated based on patient and provider feedback. Impact on acute care utilization was evaluated post hoc by linking individual patient records to provincial data holdings to examine ED+H patterns among participating patients compared to contemporaneous controls. RESULTS: Between September 2013 and December 2014, 77 women were enrolled (mean age 55 years). Most commonly used regimens were AC-paclitaxel (58%) and FEC-docetaxel (16%); 78% of patients received primary granulocyte colony-stimulating factor prophylaxis. 83.8% of calls were delivered per schedule; mean call duration was 9 min. The intervention was well received by both patients and clinicians. Comparison of ED+H rates among study participants versus controls showed that there were fewer ED visits in intervention patients [incidence rate ratio (IRR) (95% CI) = 0.54 (0.36, 0.81)] but no difference in the rate of hospitalizations [IRR (95% CI) = 1.02 (0.59, 1.77)]. Main implementation challenges included identifying eligible patients, fitting the calls into existing clinical responsibilities, and effective communication to the patient's clinical team. CONCLUSIONS: Telephone-based pro-active toxicity management during chemotherapy is feasible, perceived as valuable by clinicians and patients, and may be associated with lower rates of acute care use. However, attention must be paid to workflow issues for scalability. Larger scale evaluation of this approach is in progress.

Epidemiology of competence: a scoping review to understand the risks and supports to competence of four health professions.

OBJECTIVES: This study examined the risks and supports to competence discussed in the literature related to occupational therapists, pharmacists, physical therapists and physicians, using epidemiology as a conceptual model. DESIGN: Articles from a scoping literature review, published from 1975 to 2014 inclusive, were included if they were about a risk or support to the professional or clinical competence of one of four health professions. Descriptive and regression analyses identified potential associations between risks and supports to competence and the location of study, type of health profession, competence life-cycle and the domain(s) of competence (organised around the CanMEDS framework). RESULTS: A total of 3572 abstracts were reviewed and 943 articles analysed. Most focused on physicians (n=810, 86.0%) and 'practice' (n=642, 68.0%). Fewer articles discussed risks to competence (n=418, 44.3%) than supports (n=750, 79.5%). The top four risks, each discussed in over 15% of articles, were: transitions in practice, being an international graduate, lack of clinical exposure/experience (ie, insufficient volume of procedures or patients) and age. The top two supports (over 35%) were continuing education participation and educational information/programme features. About 60% of all the articles discussed medical expert and about 25% applied to all roles. Articles focusing on residents had a greater probability of reporting on risks. CONCLUSIONS: Articles about physicians were dominant. The majority of articles were written in the last decade and more discussed supports than risks to competence. An epidemiology-based conceptual model offers a helpful organising framework for exploring and explaining the competence of health professions.

The epidemiology of competence: protocol for a scoping review.

INTRODUCTION: Factors that are important to the competence of healthcare providers have important consequences for quality of healthcare. Although some previous research has discussed risks or supports to the competence of clinicians, a thorough exploration is currently lacking. The purpose of this review is to examine the literature examining risks and supports to clinical competence of healthcare practitioners and trainees engaged in field-based education. In this study, field-based education refers to teaching or training in a real-world/work-based setting. METHODS AND ANALYSIS: We will employ an established scoping review methodology. Eligible studies will include those that mention (1) a healthcare professional, (2) competence from field-based education throughout the lifespan/career and (3) a risk or support to competence. Four authors will independently apply the inclusion criteria to all studies, regardless of study design. Our data extraction will include information on study design, location and type of study and we will develop a comprehensive list of risks and supports that are discussed in the literature. ETHICS AND DISSEMINATION: Since this is a review of the literature, ethics approval is not indicated. We will disseminate the findings from this study in publications in peer-reviewed journals as well as presentations at relevant national and international conferences.

Psychometric properties of a peer-assessment program to assess continuing competence in physical therapy.

BACKGROUND: The College of Physiotherapists of Ontario implemented an Onsite Assessment to evaluate the continuing competence of physical therapists. OBJECTIVE: This study was undertaken to examine the reliability of the various tools used in the Onsite Assessment and to consider the relationship between the final decision and demographic factors. DESIGN: This was a psychometric study. METHODS: Trained peer assessors (n=63) visited randomly selected physical therapists (n=106) in their workplace. Fifty-three physical therapists were examined by 2 assessors simultaneously. The assessment included a review of practice issues, record keeping, billing practices, the physical therapist's professional portfolio, and a chart-stimulated recall process. The Quality Management Committee made the final decision regarding the physical therapist's performance using the assessor's summary report. Generalizability theory was used to examine the interrater reliability of the tools. Correlation coefficients and regression analyses were used to examine the relationships between demographic factors and performance. RESULTS: The majority of the physical therapists (88%) completed the program successfully, 11% required remediation, and 1% required further assessment. The interrater reliability of the components was above .70 for 2 raters' evaluations, with the exception of billing practices. There was no relationship between the final decision and age or years since graduation (r<.05). Limitations Limitations include a small sample and a lack of data on system-related factors that might influence performance. CONCLUSIONS: The vast majority of the physical therapists met the College of Physiotherapists of Ontario's professional standards. Reliability analysis indicated that the number of charts reviewed could be reduced. Strategies to improve the reliability of the various components must take into account feasibility issues related to financial and human resources. Further research to examine factors associated with failure to adhere to professional standards should be considered. These results can provide valuable information to regulatory agencies or managers considering similar continuing competence assessment programs.