Towards a program of assessment for health professionals: from training into practice.

Despite multifaceted attempts to "protect the public," including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critical lens on current assessment practices and to offer insights into ways in which they might be adapted to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved healthcare. Three dominant themes will be addressed: (1) The need to redress unintended consequences of competency-based assessment; (2) The potential to design assessment systems that facilitate performance improvement; and (3) The importance of ensuring authentic linkage between assessment and practice. Several principles cut across each of these themes and represent the foundational goals we would put forward as signposts for decision making about the continued evolution of assessment practices in the health professions: (1) Increasing opportunities to promote learning rather than simply measuring performance; (2) Enabling integration across stages of training and practice; and (3) Reinforcing point-in-time assessments with continuous professional development in a way that enhances shared responsibility and accountability between practitioners, educational programs, and testing organizations. Many of the ideas generated represent suggestions for strategies to pilot test, for infrastructure to build, and for harmonization across groups to be enabled. These include novel strategies for OSCE station development, formative (diagnostic) assessment protocols tailored to shed light on the practices of individual clinicians, the use of continuous workplace-based assessment, and broadening the focus of high-stakes decision making beyond determining who passes and who fails. We conclude with reflections on systemic (i.e., cultural) barriers that may need to be overcome to move towards a more integrated, efficient, and effective system of assessment.

Developing the role of big data and analytics in health professional education.

As we capture more and more data about learners, their learning, and the organization of their learning, our ability to identify emerging patterns and to extract meaning grows exponentially. The insights gained from the analyses of these large amounts of data are only helpful to the extent that they can be the basis for positive action such as knowledge discovery, improved capacity for prediction, and anomaly detection. Big Data involves the aggregation and melding of large and heterogeneous datasets while education analytics involves looking for patterns in educational practice or performance in single or aggregate datasets. Although it seems likely that the use of education analytics and Big Data techniques will have a transformative impact on health professional education, there is much yet to be done before they can become part of mainstream health professional education practice. If health professional education is to be accountable for its programs run and are developed, then health professional educators will need to be ready to deal with the complex and compelling dynamics of analytics and Big Data. This article provides an overview of these emerging techniques in the context of health professional education.

Expert validation of fit-for-purpose guidelines for designing programmes of assessment.

BACKGROUND: An assessment programme, a purposeful mix of assessment activities, is necessary to achieve a complete picture of assessee competence. High quality assessment programmes exist, however, design requirements for such programmes are still unclear. We developed guidelines for design based on an earlier developed framework which identified areas to be covered. A fitness-for-purpose approach defining quality was adopted to develop and validate guidelines. METHODS: First, in a brainstorm, ideas were generated, followed by structured interviews with 9 international assessment experts. Then, guidelines were fine-tuned through analysis of the interviews. Finally, validation was based on expert consensus via member checking. RESULTS: In total 72 guidelines were developed and in this paper the most salient guidelines are discussed. The guidelines are related and grouped per layer of the framework. Some guidelines were so generic that these are applicable in any design consideration. These are: the principle of proportionality, rationales should underpin each decisions, and requirement of expertise. Logically, many guidelines focus on practical aspects of assessment. Some guidelines were found to be clear and concrete, others were less straightforward and were phrased more as issues for contemplation. CONCLUSIONS: The set of guidelines is comprehensive and not bound to a specific context or educational approach. From the fitness-for-purpose principle, guidelines are eclectic, requiring expertise judgement to use them appropriately in different contexts. Further validation studies to test practicality are required.

Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference.

In this article, we outline criteria for good assessment that include: (1) validity or coherence, (2) reproducibility or consistency, (3) equivalence, (4) feasibility, (5) educational effect, (6) catalytic effect, and (7) acceptability. Many of the criteria have been described before and we continue to support their importance here. However, we place particular emphasis on the catalytic effect of the assessment, which is whether the assessment provides results and feedback in a fashion that creates, enhances, and supports education. These criteria do not apply equally well to all situations. Consequently, we discuss how the purpose of the test (summative versus formative) and the perspectives of stakeholders (examinees, patients, teachers-educational institutions, healthcare system, and regulators) influence the importance of the criteria. Finally, we offer a series of practice points as well as next steps that should be taken with the criteria. Specifically, we recommend that the criteria be expanded or modified to take account of: (1) the perspectives of patients and the public, (2) the intimate relationship between assessment, feedback, and continued learning, (3) systems of assessment, and (4) accreditation systems.

Sharing a household with children and risk of COVID-19: a study of over 300 000 adults living in healthcare worker households in Scotland.

OBJECTIVE: Children are relatively protected from COVID-19, due to a range of potential mechanisms. We investigated if contact with children also affords adults a degree of protection from COVID-19. DESIGN: Cohort study based on linked administrative data. SETTING: Scotland. STUDY POPULATION: All National Health Service Scotland healthcare workers and their household contacts as of March 2020. MAIN EXPOSURE: Number of young children (0-11 years) living in the participant's household. MAIN OUTCOMES: COVID-19 requiring hospitalisation, and any COVID-19 (any positive test for SARS-CoV-2) in adults aged ≥18 years between 1 March and 12 October 2020. RESULTS: 241 266, 41 198, 23 783 and 3850 adults shared a household with 0, 1, 2 and 3 or more young children, respectively. Over the study period, the risk of COVID-19 requiring hospitalisation was reduced progressively with increasing numbers of household children-fully adjusted HR (aHR) 0.93 per child (95% CI 0.79 to 1.10). The risk of any COVID-19 was similarly reduced, with the association being statistically significant (aHR per child 0.93; 95% CI 0.88 to 0.98). After schools reopened to all children in August 2020, no association was seen between exposure to young children and risk of any COVID-19 (aHR per child 1.03; 95% CI 0.92 to 1.14). CONCLUSION: Between March and October 2020, living with young children was associated with an attenuated risk of any COVID-19 and COVID-19 requiring hospitalisation among adults living in healthcare worker households. There was no evidence that living with young children increased adults' risk of COVID-19, including during the period after schools reopened.

Making self-assessment more effective.

Self-assessment has been held out as an important mechanism for lifelong learning and self-improvement for health care professionals. However, there is growing concern that individual learners often interpret the results inaccurately. This idea has led to skepticism that self-assessment in its current form can ever be truly useful for lifelong professional development. We examine the proposal that self-assessment can and should be made more effective. First, relevance should be improved. The process should be tied more explicitly to the individual's actual practice profile, rather than being loosely relevant to broader constructs around the permitted scope of practice (eg, certification or licensure). In addition, self-assessment should include not only knowledge and reasoning but also what is done every day in practice, thereby broadening from competence in simulated settings to performance in real settings. Second, the impact of self-assessment should be substantially strengthened by periodic external validation of self-assessment results, together with goals set as a result and plans for further improvement. This offers to the individual the very tangible benefit of satisfying external mandates (eg, licensure and certification). In addition, impact should be reinforced by linking the results of self-assessment to subsequent learning activities including Continuing Medical Education (CME). Although these enhancements individually may not cure all of what ails self-assessment, they might ensure greater effectiveness for the purposes of lifelong learning.

Evidence of peripheral large nerve involvement in fibromyalgia: a retrospective review of EMG and nerve conduction findings in 55 FM subjects.

OBJECTIVE: Whereas small fiber neuropathy (SFN) is now a recognized part of fibromyalgia (FM), surprisingly little attention has been paid to any findings of large fiber neuropathy (LFN) in this disorder. Since 90% to 95% of FM subjects seen in our outpatient facility routinely undergo EMG and nerve conduction studies (NCS) we elected to retrospectively review the EMG/NCS results garnered from a large cohort of unselected subjects in order to describe the electrodiagnostic features of LFN in FM. METHODS: Records from 100 consecutive, unselected clinic patients meeting the 1990 ACR criteria for FM, who had undergone EMG/NCS, were reviewed. The same electromyographer tested all subjects. After exclusion of FM patients with any other clinically relevant condition that might influence EMG results (e.g., familial neural degenerative conditions, diabetes mellitus, Vitamin B-12 deficiency, etc.) fifty-five FM subjects remained: 29 subjects with "FM Only," and 26 subjects with FM+Rheumatoid Arthritis ("FM+RA"). All subjects had also undergone ankle area skin biopsy for determination of epidermal nerve fiber density (ENFD). Fourteen other subjects, without FM or RA, examined by the same electromyographer, were used as an EMG/NCS comparison group. RESULTS: Ninety percent of the "FM Only" subjects demonstrated a demyelinating and/or axonal, sensorimotor polyneuropathy, and 63% had findings of SFN (ENFD ≤7 fibers/mm), suggesting a mixed fiber neuropathy in most. Furthermore, 61% of the "FM Only" subjects showed EMG findings suggestive of non-myotomal lower extremity axonal motor denervation, most likely due to a polyneuropathy, and 41% satisfied published criteria for "possible" chronic inflammatory demyelinating polyneuropathy (CIDP). There was surprisingly little difference in the EMG/NCS findings between the "FM Only" and the "FM+RA" groups. With the exception of carpal tunnel syndrome, our EMG/NCS comparison group showed few to none of these findings. CONCLUSION: Our review of the EMG/NCS results, gleaned from the largest FM cohort yet studied with these modalities, shows that electrodiagnostic features of polyneuropathy, muscle denervation, and CIDP are common in FM. Furthermore these electrodiagnostic findings are often seen coincident with SFN, and are not significantly influenced by the presence of RA. These results, particularly when taken as a whole, suggest that EMG/NCS may be clinically useful in detecting LFN in FM and help in better understanding the etiopathogenesis of this painful disorder.

The quality of in-house medical school examinations.

PURPOSE: Most medical schools test their students throughout the curriculum using in-house examinations written by the faculty who teach the courses. The authors assessed the quality of in-house examinations used in three U.S. medical schools. METHOD: In 1998, nine basic science examinations from the three schools were gathered and each question was subjected to quality assessment by three expert biomedical test developers, each of whom has had extensive experience in reviewing and evaluating questions for the United States Medical Licensing Examination (USMLE) Steps 1 and 2. Questions were rated on a five-point scale: 1 = tested recall only and was technically flawed to 5 = used a clinical or laboratory vignette, required reasoning to answer, and was free of technical flaws. Each rater made independent assessments, and the mean score for each question was calculated. Mean quality scores for National Board of Medical Examiners (NBME) who were trained question writers were compared with the mean scores for question writers without NBME training. The raters' quality assessments were made without knowledge of the test writers' training background or the study's hypothesis. RESULTS: A total of 555 questions were analyzed. The mean score for all questions was 2.39 +/- 1.21. The 92 questions written by NBME-trained question writers had a mean score of 4.24 +/- 0.85, and the 463 questions written by faculty without formal NBME training had a mean score of 2.03 +/- 0.90 (p <.01). CONCLUSIONS: The in-house examinations were of relatively low quality. The quality of examination questions can be significantly improved by providing question writers with formal training.

Continuing medical education effect on practice performance: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines.

BACKGROUND: There has been sizable debate and widespread skepticism about the effect of continuing medical education (CME) on the performance of physicians in the practice setting. This portion of the review was undertaken to examine that effect. METHODS: The guideline panel used data from a comprehensive review of the effectiveness of CME developed by The Johns Hopkins Evidence-based Practice Center, focusing on the effect of CME on clinical performance. RESULTS: The review found 105 studies, which evaluated the impact of CME on short- and long-term physician practice performance. Nearly 60% met objectives relative to changing clinical performance in prescribing; screening; counseling about smoking cessation, diet, and sexual practices; guideline adherence; and other topics. Single live and multiple media appeared to be generally positive in their effect, print media much less so. Multiple educational techniques were more successful at changing provider performance than single techniques. The amount or frequency of exposure to CME activities appeared to have little effect on behavior change. CONCLUSIONS: Overall, CME, especially using live or multiple media and multiple educational techniques, is generally effective in changing physician performance. More research, however, is needed that focuses on the specific types of media and educational techniques that lead to the greatest improvements in performance.

Continuing medical education effect on clinical outcomes: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines.

BACKGROUND: As opportunities for quality improvement become more visible, educational planners, health services researchers, and policymakers search for strategies that lead to better clinical outcomes. Continuing medical education (CME) is one such strategy, but the impact of CME is poorly defined in relation to clinical outcomes, and efforts to standardize definitions of clinical outcomes are in varied stages of development. METHODS: The Johns Hopkins University Evidence-based Practice Center conducted a systematic review of the effectiveness of CME for the Agency for Health Care Research and Quality. From the review, 37 studies were used by the guideline panel to answer questions about improvement in clinical outcomes. Recommendations were made using the American College of Chest Physicians guideline grading system. RESULTS: Multiple media, multiple techniques of instruction, and multiple exposures to content are suggested to meet instructional objectives intended to improve clinical outcomes. CONCLUSIONS: There are models to describe and guide the planning and evaluation of CME, and there are models to measure quality of care. Research and practice of CME must be defined in relation to guideline implementation and quality improvement and other interventions and systems intended to improve or measure clinical outcomes. Further research is required to identify the qualities essential for measuring causal linkages thought to exist among CME, physician behavior, and clinical outcomes.

The future of continuing medical education: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines.

To ensure that continuing medical education (CME) continues to evolve so that it offers educational activities that are relevant to physicians in keeping with the definition of CME, CME providers must respond to and prepare for emerging expectations. This article puts into context the impact of the current emphasis on lifelong learning in medicine, particularly the requirement for maintenance of certification and licensure, on CME. Further, the effect of changing needs assessments and the impact of the integration of new technology in CME is included. Finally, a discussion of the emerging unique needs of CME providers and organizations related to these changes are addressed in the following four broad categories: CME as a value center, resources in support of CME, research to further advance the field, and leadership to guide the profession.