Large-Scale Telemedicine Implementation for Outpatient Clinicians: Results From a Pandemic-Adapted Learning Collaborative.

Learning collaboratives are seldom used outside of health care quality improvement. We describe a condensed, 10-week learning collaborative ("Telemedicine Hack") that facilitated telemedicine implementation for outpatient clinicians early in the COVID-19 pandemic. Live attendance averaged 1688 participants per session. Of 1005 baseline survey respondents, 57% were clinicians with one-third identifying as from a racial/ethnic minoritized group. Practice characteristics included primary care (71%), rural settings (51%), and community health centers (28%). Of three surveys, a high of 438 (81%) of 540 clinicians had billed ≥1 video-based telemedicine visit. Our learning collaborative "sprint" is a promising model for scaling knowledge during emergencies and addressing health inequities.

Facilitating Real-Time, Multidirectional Learning for Clinicians in a Low-Evidence Pandemic Response.

As COVID-19 was declared a health emergency in March 2020, there was immense demand for information about the novel pathogen. This paper examines the clinician-reported impact of Project ECHO COVID-19 Clinical Rounds on clinician learning. Primary sources of study data were Continuing Medical Education (CME) Surveys for each session from the dates of March 24, 2020 to July 30, 2020 and impact surveys conducted in November 2020, which sought to understand participants' overall assessment of sessions. Quantitative analyses included descriptive statistics and Mann-Whitney testing. Qualitative data were analyzed through inductive thematic analysis. Clinicians rated their knowledge after each session as significantly higher than before that session. 75.8% of clinicians reported they would 'definitely' or 'probably' use content gleaned from each attended session and clinicians reported specific clinical and operational changes made as a direct result of sessions. 94.6% of respondents reported that COVID-19 Clinical Rounds helped them provide better care to patients. 89% of respondents indicated they 'strongly agree' that they would join ECHO calls again.COVID-19 Clinical Rounds offers a promising model for the establishment of dynamic peer-to-peer tele-mentoring communities for low or no-notice response where scientifically tested or clinically verified practice evidence is limited.

Virtual Peer-to-Peer Learning to Enhance and Accelerate the Health System Response to COVID-19: The HHS ASPR Project ECHO COVID-19 Clinical Rounds Initiative.

Tasked with identifying digital health solutions to support dynamic learning health systems and their response to COVID-19, the US Department of Health and Human Services Office of the Assistant Secretary for Preparedness and Response partnered with the University of New Mexico's Project ECHO and more than 2 dozen other organizations and agencies to create a real-time virtual peer-to-peer clinical education opportunity: the COVID-19 Clinical Rounds Initiative. Focused on 3 "pressure points" in the COVID-19 continuum of care-(1) the out-of-hospital and/or emergency medical services setting, (2) emergency departments, and (3) inpatient critical care environments-the initiative has created a massive peer-to-peer learning network for real-time information sharing, engaging participants in all 50 US states and more than 100 countries. One hundred twenty-five learning sessions had been conducted between March 24, 2020 and February 25, 2021, delivering more than 58,000 total learner-hours of contact in the first 11 months of operation.

DNA Barcoding Silver Butter Catfish (Schilbe intermedius) Reveals Patterns of Mitochondrial Genetic Diversity Across African River Systems.

The silver butter catfish (Schilbe intermedius) is widely distributed across African river systems. To date, information on its mitochondrial genetic diversity, population structure, and historical demography are not well-established. Herein, we combined newly generated mitochondrial cytochrome c oxidase (COI) subunit I gene sequences with previously published COI sequences in the global databases to reconstruct its phylogeography, population genetic structure, and historical demography. Results from the mtDNA phylogeography and species delimitation tests (Cluster algorithm - Species Identifier, Automatic Barcode Gap Discovery and Poison Tree Process model) revealed that S. intermedius comprises at least seven geographically defined matrilines. Although the overall haplotype diversity of S. intermedius was high (h = 0.90), results showed that East (Kenya) and West (Nigeria) African populations had low levels of haplotype diversity (h = ~0.40). In addition, population genetic polymorphism and historical demographics showed that S. intermedius populations in both East and West Africa underwent severe contractions as a result of biogeographic influences. The patterns of genetic diversity and population structure were consistent with adaptive responses to historical biogeographic factors and contemporary environmental variations across African river systems. This is suggestive of the influence of historical biogeographic factors and climatic conditions on population divergence of S. intermedius across African river systems. Given our discovery of previously underappreciated diversity within S. intermedius, we recommend that this species be considered for increased conservation and management.

Management practices associated with strongylid parasite prevalence on horse farms in rural counties of Kentucky.

Anthelmintic resistance among cyathostomin parasites is a wide-spread problem. The parasite control guidelines written by the American Association of Equine Practitioners (AAEP) encourages the preservation of anthelmintic efficacy by reducing treatment frequency, using targeted deworming, and implementing environmental management practices. While there is knowledge regarding parasite management practices of affluent horse farms in the United States, surveys rarely explore the rural and underserved regions. The purpose of this study was to observe the management practices of horse farms in rural regions Kentucky, including working Amish farms, and determine factors associated with strongyle prevalence. A total of 160 horses among 38 owners from 28 different farms were enrolled in this study. A questionnaire survey regarding equine information, farm management, and deworming history was performed with each owner. Fecal samples were collected to determine fecal egg counts, perform coprocultures for subsequent strongyle larvae identification, and Strongylus vulgaris specific PCR. Serum samples were collected for the S. vulgaris antibody specific ELISA. The mean number of deworming treatments given in the last year was 2.1 with a 95% confidence interval of 1.9-2.3 with ivermectin being the most common active used. Statistical analysis showed horses treated within the last three months with a macrocylic lactone (ML) drug had significantly lower egg counts than horses treated with a ML 7-9 months ago (p = .0005). Despite the AAEP recommendations to reduce the overall number of treatments by using a surveillance-based approach and to no longer rotate treatments, only 17 horses reportedly had a fecal sample submitted for a fecal egg count and 65 horses were dewormed in a rotational manner. Horses whose owners utilized an informative deworming source (i.e., veterinarian, internet, magazine, local feed store) also had significantly lower counts (p = .0026). All coprocultures were negative for S. vulgaris while five horses were PCR positive. Interestingly, 95 horses tested ELISA positive for S. vulgaris. The strongyle egg counts of the working Amish horses were not significantly different from the other horses in this study and deworming practices including the use of efficacious drugs and low treatment frequencies were in accordance with the AAEP guidelines. This study was the first to summarize deworming management practices of rural regions in Kentucky, including a working Amish community. Overall, horse owners employed deworming practices recommended by the AAEP, however rotational deworming is still commonly implemented and fecal egg counts are rarely used.

Making robust assessments of specialist trainees' workplace performance.

BACKGROUND: Workplace-based assessments should provide a reliable measure of trainee performance, but have met with mixed success. We proposed that using an entrustability scale, where supervisors scored trainees on the level of supervision required for the case would improve the utility of compulsory mini-clinical evaluation exercise (CEX) assessments in a large anaesthesia training program. METHODS: We analysed mini-CEX scores from all Australian and New Zealand College of Anaesthetists trainees submitted to an online database over a 12-month period. Supervisors' scores were adjusted for the expected supervision requirement for the case for trainees at different stages of training. We used generalisability theory to determine score reliability. RESULTS: 7808 assessments were available for analysis. Supervision requirements decreased significantly (P < 0.05) with increased duration and level of training, supporting validity. We found moderate reliability (G > 0.7) with a feasible number of assessments. Adjusting scores against the expected supervision requirement considerably improved reliability, with G > 0.8 achieved with only nine assessments. Three per cent of trainees generated average mini-CEX scores below the expected standard. CONCLUSIONS: Using an entrustment scoring system, where supervisors score trainees on the level of supervision required, mini-CEX scores demonstrated moderate reliability within a feasible number of assessments, and evidence of validity. When scores were adjusted against an expected standard, underperforming trainees could be identified, and reliability much improved. Taken together with other evidence on trainee ability, the mini-CEX is of sufficient reliability for inclusion in high stakes decisions on trainee progression towards independent specialist practice.

Accreditation of medical schools in Saudi Arabia: A qualitative study.

CONTEXT: The accreditation of undergraduate medical education is a universal undertaking. Despite the widespread adoption of accreditation processes and an increasing focus on accreditation as a mechanism to ensure minimum standards are met in various fields, there is little evidence to support the effectiveness of accreditation. Traditionally, accreditation has worked toward achieving two ends: assuring and improving quality. Many recent articles emphasize the need for continuous quality improvement mechanisms to work, as well as the quality assurance role of accreditation. METHODS: The aim of the study was to examine the purposes and outcomes of accreditation, and stakeholders' experience of accreditation in Saudi Arabia. Triangulation of data was achieved through literature review, analysis of accreditation documents, examined the outcome of accreditation process (pre and post) through stakeholders' experience of accreditation (learner, teacher, and academic leader perspectives). Data were interrogated using thematic analysis approach involving identifying, analyzing, and reporting repeated patterns (themes) of meaning within data. RESULTS: Three themes emerged from the three phase study: "Passing the exam" versus long-term benefit, generic versus specialized accreditation standards, and internal quality assurance and self-evaluation. The data revealed a number of strategies that stakeholders can employ to achieve a balance between an "accreditation threat" and a quality improvement approach that is likely to have a lasting effect on educational outcomes. DISCUSSION: This empirical study revealed strong parallels between assessment and accreditation purpose, engagement, and outcomes. Like an increasing number of commentaries in the literature, this study suggests that accreditation bodies would do well to shift toward a holistic approach to quality management in medical education; implementation of quality improvement by an external "other"-described by some participants as the "policeman approach"-is not ideal for promoting sustainable quality education. Sustainable accreditation for long-term education improvement is not presented as a method, but as a way of thinking about important, and often overlooked, aspects of accreditation practice. Sustainable accreditation means that there is a need to meet both the immediate accreditation standards ("the exam") as well as establishing a basis for continuing quality improvement.

Reliability of the direct observation of procedural skills assessment tool for ultrasound-guided regional anaesthesia.

The Direct Observation of Procedural Skills (DOPS) form is used as a workplace-based assessment tool in the current Australian and New Zealand College of Anaesthetists curriculum. The objective of this study was to evaluate the reliability of DOPS when used to score trainees performing ultrasound-guided regional anaesthesia. Reliability of an assessment tool is defined as the reproducibility of scores given by different assessors viewing the same trainee. Forty-nine anaesthetists were recruited to score two scripted videos of trainees performing a popliteal sciatic nerve block and an axillary brachial plexus block. Reliability, as measured by intraclass correlation coefficients, was -0.01 to 0.43 for the individual items in DOPS, and 0.15 for the 'Overall Performance for this Procedure' item. Assessors demonstrated consistency of scoring within DOPS, with significant correlation of sum of individual item scores with the 'Overall Performance for this Procedure' item (r=0.78 to 0.80, P<0.001), and with yes versus no responses to the 'Was the procedure completed satisfactorily?' item (W=24, P=0.0004, Video 1, and W=65, P=0.003, Video 2). While DOPS demonstrated a good degree of internal consistency in this setting, inter-rater reliability did not reach levels generally recommended for formative assessment tools. Feasibility of the form could be improved by removing the 'Was the procedure completed satisfactorily?' item without loss of information.

Can I leave the theatre? A key to more reliable workplace-based assessment.

BACKGROUND: The value of workplace-based assessments such as the mini-clinical evaluation exercise (mini-CEX), and clinicians' confidence and engagement in the process, has been constrained by low reliability and limited capacity to identify underperforming trainees. We proposed that changing the way supervisors make judgements about trainees would improve score reliability and identification of underperformers. Anaesthetists regularly make decisions about the level of trainee independence with a case, based on how closely they need to supervise them. We therefore used this as the basis for a new scoring system. METHODS: We analysed 338 mini-CEXs where supervisors scored trainees using the conventional system, and also scored trainee independence, based on the need for direct, or more distant, supervision. As supervisory requirements depend on case difficulty, we then compared the actual trainee independence score and the expected trainee independence score obtained externally. RESULTS: Compared with the conventional scoring system used in previous studies, reliability was very substantially improved using a system based on a trainee's level of independence with a case. Reliability improved further when this score was corrected for case difficulty. Furthermore, the new scoring system overcame the previously identified problem of assessor leniency and identified a number of trainees performing below expectations. CONCLUSIONS: Supervisors' judgements on trainee independence with a case, based on the need for direct or more distant supervision, can generate reliable scores of trainee ability without the need for an onerous number of assessments, identify trainees performing below expectations, and track trainee progress towards independent specialist practice.

Work-based assessment: qualitative perspectives of novice nutrition and dietetics educators.

BACKGROUND: The assessment of competence for health professionals including nutrition and dietetics professionals in work-based settings is challenging. The present study aimed to explore the experiences of educators involved in the assessment of nutrition and dietetics students in the practice setting and to identify barriers and enablers to effective assessment. METHODS: A qualitative research approach using in-depth interviews was employed with a convenience sample of inexperienced dietitian assessors. Interviews explored assessment practices and challenges. Data were analysed using a thematic approach within a phenomenological framework. Twelve relatively inexperienced practice educators were purposefully sampled to take part in the present study. RESULTS: Three themes emerged from these data. (i) Student learning and thus assessment is hindered by a number of barriers, including workload demands and case-mix. Some workplaces are challenged to provide appropriate learning opportunities and environment. Adequate support for placement educators from the university, managers and their peers and planning are enablers to effective assessment. (ii) The role of the assessor and their relationship with students impacts on competence assessment. (iii) There is a lack of clarity in the tasks and responsibilities of competency-based assessment. CONCLUSIONS: The present study provides perspectives on barriers and enablers to effective assessment. It highlights the importance of reflective practice and feedback in assessment practices that are synonymous with evidence from other disciplines, which can be used to better support a work-based competency assessment of student performance.

Approaches to learning for the ANZCA Final Examination and validation of the revised Study Process Questionnaire in specialist medical training.

When evaluating assessments, the impact on learning is often overlooked. Approaches to learning can be deep, surface and strategic. To provide insights into exam quality, we investigated the learning approaches taken by trainees preparing for the Australian and New Zealand College of Anaesthetists (ANZCA) Final Exam. The revised two-factor Study Process Questionnaire (R-SPQ-2F) was modified and validated for this context and was administered to ANZCA advanced trainees. Additional questions were asked about perceived value for anaesthetic practice, study time and approaches to learning for each exam component. Overall, 236 of 690 trainees responded (34%). Responses indicated both deep and surface approaches to learning with a clear preponderance of deep approaches. The anaesthetic viva was valued most highly and the multiple choice question component the least. Despite this, respondents spent the most time studying for the multiple choice questions. The traditionally low short answer questions pass rate could not be explained by limited study time, perceived lack of value or study approaches. Written responses suggested that preparation for multiple choice questions was characterised by a surface approach, with rote memorisation of past questions. Minimal reference was made to the ANZCA syllabus as a guide for learning. These findings indicate that, although trainees found the exam generally relevant to practice and adopted predominantly deep learning approaches, there was considerable variation between the four components. These results provide data with which to review the existing ANZCA Final Exam and comparative data for future studies of the revisions to the ANZCA curriculum and exam process.

Investigation of trainee and specialist reactions to the mini-Clinical Evaluation Exercise in anaesthesia: implications for implementation.

BACKGROUND: The mini-Clinical Evaluation Exercise (mini-CEX) is a workplace-based assessment which may be useful in anaesthesia training. However, its value depends on how supervisors use it with their trainees. This study analyses experience with the mini-CEX after its introduction into anaesthesia departments in our institution. METHODS: We conducted surveys, focus groups, and interviews with trainees and specialists. Data were recorded, transcribed, and entered into NVivo 8. Themes were identified and data coded into these themes. RESULTS: We identified six themes: assessor factors included skills needed to perform the assessments, influences on scoring decisions, and effects on the specialist-trainee relationship; trainee factors related to impact on trainee performance and value at the different training levels; teaching and learning included the effect of focused observation on structuring workplace learning; feedback described how the mini-CEX changed feedback and what was considered useful; mini-CEX process included implementation, initiation of assessments and case selection; and use in assessment included comparisons with existing assessments and the ability to identify poor performers. CONCLUSIONS: Mini-CEX formalized the supervisory relationship, promoting educational interactions. During the observation period, trainees took responsibility for decisions, and specialists learnt more about their abilities. The structured format broadened the scope of feedback and made it easier to address performance gaps. We identified factors that facilitated or hindered implementation, or limited effective feedback and the ability to address poor performance. From this analysis, we propose strategies for the implementation of mini-CEX, and recommendations for assessor training to improve the quality and value of the assessments.

Mini-clinical evaluation exercise in anaesthesia training.

BACKGROUND: The Mini-Clinical Evaluation Exercise (Mini-CEX) is a workplace-based assessment tool of potential value in anaesthesia to assess and improve clinical performance. Its reliability and positive educational impact have been reported in other specialities, but not, to date, in anaesthesia. In this study, we evaluated the psychometric characteristics, logistics of application, and impact on the quality of supervision of the Mini-CEX in anaesthesia training. METHODS: A Mini-CEX encounter consisted of a single specialist anaesthetist observing a trainee over a defined period of time, completing an online Mini-CEX form with the trainee, and providing written and verbal feedback. We sought trainee and supervisor perspectives on its value and ease of use and used Generalizability Theory to estimate reliability. RESULTS: We collected 331 assessments from 61 trainees and 58 assessors. Survey responses strongly supported the positive effect of the Mini-CEX on feedback, its relative feasibility, and acceptance as a potential assessment tool. In this cohort, we found variable assessor stringency and low trainee variation. However, a feasible sample of cases and assessors would produce sufficiently precise scores to decide that performance was satisfactory for each trainee with 95% confidence. To generate scores that could discriminate sufficiently between trainees to allow ranking, a much larger sample of cases would be needed. CONCLUSIONS: The Mini-CEX in anaesthesia has strengths and weaknesses. Strengths include: its perceived very positive educational impact and its relative feasibility. Variable assessor stringency means that large numbers of assessors are required to produce reliable scores.

Generalisability of behavioural skills in simulated anaesthetic emergencies.

A reliable assessment of clinical performance requires multiple cases, as performance varies between cases depending on previous experiences and knowledge of the case. However, behavioural attributes, including communication and teamwork, may be expected to be less dependent on specific case knowledge and thus be more stable across cases. This has implications for training and assessment design. In this study we measured the psychometric properties of assessment of behaviour in simulated anaesthetic emergencies. Twenty anaesthesia trainees were rated in three simulated emergencies by four assessors. The psychometric properties of scores for behaviour were determined and were compared with scores for medical management and overall performance. We found that scores for behaviour were less dependent on the specific clinical context than the scores for overall performance and medical management, implying transferability of crisis management behaviours between cases.

Assessing the technical skills of surgical trainees.

BACKGROUND: The aims were to determine whether tests of technical skill on simple simulations can predict competence in the operating theatre and whether objective assessment in the operating theatre by direct observation and video recording is feasible and reliable. METHODS: Thirty-three general surgical trainees undertook five simple skill simulations (knotting, skin incision and suturing, tissue dissection, vessel ligation and small bowel anastomosis). The operative competence of each trainee was then assessed during two or three saphenofemoral disconnections (SFDs) by a single surgeon. Video recordings of the operations were also assessed by two surgeons. RESULTS: The inter-rater reliability between direct observation and blinded videotape assessment was high (alpha = 0.96 (95 per cent confidence interval 0.92 to 0.98)). Backward stepwise regression analysis revealed that the best predictors of operative competence were the number of SFDs performed previously plus the simulation scores for dissection and ligation, the key components of SFD (64 per cent of variance explained; P = 0.001). CONCLUSION: Deconstruction of operations into their component parts enables trainees to practise on simple simulations representing each component, and be assessed as competent, before undertaking the actual operation. Assessment of surgical competence by direct observation and video recording is feasible and reliable; such assessments could be used for both formative and summative assessment.

Psychometric characteristics of simulation-based assessment in anaesthesia and accuracy of self-assessed scores.

The purpose of this study was to define the psychometric properties of a simulation-based assessment of anaesthetists. Twenty-one anaesthetic trainees took part in three highly standardised simulations of anaesthetic emergencies. Scenarios were videotaped and rated independently by four judges. Trainees also assessed their own performance in the simulations. Results were analysed using generalisability theory to determine the influence of subject, case and judge on the variance in judges' scores and to determine the number of cases and judges required to produce a reliable result. Self-assessed scores were compared to the mean score of the judges. The results suggest that 12-15 cases are required to rank trainees reliably on their ability to manage simulated crises. Greater reliability is gained by increasing the number of cases than by increasing the number of judges. There was modest but significant correlation between self-assessed scores and external assessors' scores (rho = 0.321; p = 0.01). At the lower levels of performance, trainees consistently overrated their performance compared to those performing at higher levels (p = 0.0001).