Multicenter randomized trial exploring effects of simulation-based ultrasound training on obstetricians' diagnostic accuracy: value for experienced operators.

OBJECTIVE: To explore the effects of simulation-based ultrasound training on the accuracy of fetal weight estimation in the third trimester among obstetricians with different levels of clinical experience. METHODS: This was a multicenter, randomized pre-post-test practical trial conducted between March 2016 and January 2018. Obstetricians with different levels of clinical experience were randomized to either simulation-based ultrasound training focusing on fetal weight scans or no intervention. Participants completed two scans in pregnant women at term to establish baseline accuracy of fetal weight estimation. Another two scans were performed at follow-up. Accuracy was defined by the percentage difference between estimated fetal weight and actual birth weight. Ultrasound image quality was rated by two expert raters. RESULTS: Seventy participants with different levels of clinical experience completed the study. Adjusting for clinical experience, the intervention group demonstrated an improvement in measurement accuracy of 31.9% (95% CI, 6.9-50.1%) (P = 0.02), whereas the control group did not improve (relative difference, 13.1% (95% CI, -17.9 to 55.9%); P = 0.45). The change in accuracy was significantly different between the groups (P = 0.02) and independent of clinical experience (P = 0.54). Image-quality scores improved by a mean of 1.2 (95% CI, 0.4-2.1) (P < 0.01) in the intervention group, with no change in the control group (mean difference, 0.1 (95% CI, -0.8 to 1.0); P = 0.78). There was a strong negative correlation between time spent using the simulator and clinical experience (r = -0.70, P = 0.0001). CONCLUSION: Simulation-based ultrasound training improved accuracy and image quality when performing fetal weight estimation in women at term, independent of obstetricians' clinical experience. Copyright © 2019 ISUOG. Published by John Wiley & Sons Ltd.

Long-term experiences of being a simulation-educator: A multinational interview study.

The long-term reactions, experiences and reflections of simulation educators have not been explored. In a semistructured, exploratory interview study, the experiences of simulation educators in either Advanced Life Support (ALS) or Crisis Resource Management (CRM) courses in Denmark, Norway and the USA were analyzed. Three overarching themes were identified: (1) general reflections on simulation-based teaching, (2) transfer of knowledge and skills from the simulation setting to clinical settings and (3) more overarching transformations in simulation educators, simulation participants, and the healthcare system. Where ALS was deemed as high on the efficiency dimension of learning, CRM courses were described as high on the innovation dimension. General reflections, transfer and transformations described were related to differences in course principles. The results are relevant for career planning, faculty development and understanding simulation as social practice.

Evaluating structured assessment of anaesthesiologists' non-technical skills.

BACKGROUND: Non-technical skills (NTS) are essential for safe and efficient anaesthesia. Assessment instruments with appropriate validity evidence can be used to ensure that anaesthesiologists possess the NTS necessary to deliver high-standard patient care. The aims were to collect validity evidence using a contemporary validity framework for the assessment instrument Anaesthesiologists' Non-Technical Skills in Denmark (ANTSdk) regarding response process and internal structure (including reliability), and to investigate the effect of rater training on these properties. METHODS: An explorative study was undertaken at the Danish Institute for Medical Simulation, Copenhagen, Denmark. In a 1-day session, using ANTSdk, a convenience sample of 19 anaesthesiologists rated trainee anaesthesiologists' NTS in nine video-recorded simulation scenarios before and after a 3-h training session. RESULTS: Response process evidence: participants considered ANTSdk useful and feasible for NTS assessment. Internal structure evidence: inter-rater reliability (single measures) largely expressed substantial agreement (ICC ≥ 0.55 and ICC ≥ 0.60 for pre- and post-training ratings respectively). Strong internal consistency of ratings was found (Spearman's correlation coefficient ≥ 0.82). Accuracy of participants' ratings compared with reference ratings (± 1 scale point) was notable (76% and 78% for pre- and post-training ratings, respectively). The results indicate that the elements 'Demonstrating self-awareness', 'Reassessing decisions', 'Assessing competencies', and 'Supporting others' need more attention in future rater training. CONCLUSION: The validity evidence collected on content, response process, and internal structure, suggests that ANTSdk is easy to use on video-recorded simulation scenarios, indicating that ANTSdk is a feasible instrument for NTS assessment during anaesthesia training.

Collecting Validity Evidence for the Assessment of Mastery Learning in Simulation-Based Ultrasound Training.

PURPOSE: To collect validity evidence for the assessment of mastery learning on a virtual reality transabdominal ultrasound simulator. MATERIALS AND METHODS: We assessed the validity evidence using Messick's framework for validity. The study included 20 novices and 9 ultrasound experts who all completed 10 obstetric training modules on a transabdominal ultrasound simulator that provided automated measures of performance for each completed module (i. e., simulator metrics). Differences in the performance of the two groups were used to identify simulator metrics with validity evidence for the assessment of mastery learning. The novices continued to practice until they had attained mastery learning level. RESULTS: One-third of the simulator metrics discriminated between the two groups. The median simulator scores from a maximum of 40 metrics were 17.5 percent (range 0 - 45.0 percent) for novices and 90.0 percent (range 85.0 - 97.5) for experts, p < 0.001. Internal consistency was high, with a Cronbach's alpha value of 0.98. The test/retest reliability gave an intra-class correlation coefficient (ICC) of 0.62 for novices who reached the mastery learning level twice. Novices reached the mastery learning level within a median of 4 attempts (range 3 - 8) corresponding to a median of 252 minutes of simulator training (range 211 - 394 minutes). CONCLUSION: This study found that validity evidence for the assessment of mastery learning in simulation-based ultrasound training can be demonstrated and that ultrasound novices can attain mastery learning levels with less than 5 hours of training. Only one-third of the standard simulator metrics discriminated between different levels of competence.

Sustained effect of simulation-based ultrasound training on clinical performance: a randomized trial.

OBJECTIVE: To study the effect of initial simulation-based transvaginal sonography (TVS) training compared with clinical training only, on the clinical performance of residents in obstetrics and gynecology (Ob-Gyn), assessed 2 months into their residency. METHODS: In a randomized study, new Ob-Gyn residents (n = 33) with no prior ultrasound experience were recruited from three teaching hospitals. Participants were allocated to either simulation-based training followed by clinical training (intervention group; n = 18) or clinical training only (control group; n = 15). The simulation-based training was performed using a virtual-reality TVS simulator until an expert performance level was attained, and was followed by training on a pelvic mannequin. After 2 months of clinical training, one TVS examination was recorded for assessment of each resident's clinical performance (n = 26). Two ultrasound experts blinded to group allocation rated the scans using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale. RESULTS: During the 2 months of clinical training, participants in the intervention and control groups completed an average ± SD of 58 ± 41 and 63 ± 47 scans, respectively (P = 0.67). In the subsequent clinical performance test, the intervention group achieved higher OSAUS scores than did the control group (mean score, 59.1% vs 37.6%, respectively; P < 0.001). A greater proportion of the intervention group passed a pre-established pass/fail level than did controls (85.7% vs 8.3%, respectively; P < 0.001). CONCLUSION: Simulation-based ultrasound training leads to substantial improvement in clinical performance that is sustained after 2 months of clinical training. © 2015 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of the International Society of Ultrasound in Obstetrics and Gynecology.

Reliable and valid assessment of ultrasound operator competence in obstetrics and gynecology.

OBJECTIVES: To explore the reliability and validity of a recently developed instrument for assessment of ultrasound operator competence, the Objective Structured Assessment of Ultrasound Skills (OSAUS). METHODS: Three groups of 10 doctors with different levels of ultrasound experience in obstetrics and gynecology were included. The novices had less than 1 month of experience, the intermediate group had 12-60 months of experience and the senior participants were all consultants. Fifteen participants performed transabdominal fetal biometry and the other 15 participants performed systematic transvaginal gynecological ultrasound scans. All scans were video-recorded and assessed by two blinded consultants using the OSAUS scale. The OSAUS scores were compared between the groups using the Kruskal-Wallis test, and pass/fail scores were determined using the contrasting-groups method of standard setting. RESULTS: For the transabdominal fetal biometry examinations, the mean ± SD OSAUS scores of the novices, intermediates and senior participants were 1.5 ± 0.4, 3.3 ± 0.6 and 4.4 ± 0.4, respectively (P = 0.003). For the systematic transvaginal scans, the mean ± SD OSAUS scores of the novices, intermediates and senior participants were 1.8 ± 0.2, 3.1 ± 0.1 and 3.9 ± 0.5, respectively (P = 0.003). Post-hoc comparisons showed significant differences between each of the groups for both types of scans. The pass/fail score was 2.5 for the transvaginal scan and 3.0 for the transabdominal biometry examinations. The inter-rater reliability was 0.89. CONCLUSIONS: Ultrasound competence can be assessed in a reliable and valid way using the OSAUS scale. The pass/fail scores may be used to help determine when trainees are qualified for independent practice.

Which factors are associated with trainees' confidence in performing obstetric and gynecological ultrasound examinations?

OBJECTIVE: To explore the association between clinical training characteristics and trainees' level of confidence in performing ultrasound scans independently. METHODS: A cross-sectional e-survey was distributed to members of the national societies of junior obstetricians/gynecologists in Denmark, Sweden and Norway (n = 973). Multiple linear regression models were used to explore the effect that amount of time spent in specialized ultrasound units and clinical experience had on trainees' confidence in performing ultrasonography independently. Exploratory factor analysis was used to identify factors that contributed to trainees' confidence in performing ultrasonography. Trainees' ultrasound confidence was finally compared with their expected levels of performance. RESULTS: Of the 682 respondents (response rate 70.1%), 621 met the inclusion criteria. Clinical experience and time spent in specialized ultrasound units were predictors of trainees' confidence in performing ultrasonography independently (P < 0.001). Trainees required more than 24 months of clinical experience and 12-24 days of training in specialized ultrasound units in order to feel confident about performing transvaginal and transabdominal ultrasound scans independently. Three factors were related to ultrasound confidence: technical aspects, image perception and integration of scan into patient care. There were significant differences between trainees' level of confidence and their expected levels of performance (P < 0.001). CONCLUSIONS: Clinical experience and time spent in specialized ultrasound units were predictors of trainees' confidence in performing ultrasonography independently. Discrepancies between trainees' confidence and their expected levels of performance raised concerns about the adequacy of current ultrasound training programs.

Assessment of performance measures and learning curves for use of a virtual-reality ultrasound simulator in transvaginal ultrasound examination.

OBJECTIVE: To assess the validity and reliability of performance measures, develop credible performance standards and explore learning curves for a virtual-reality simulator designed for transvaginal gynecological ultrasound examination. METHODS: A group of 16 ultrasound novices, along with a group of 12 obstetrics/gynecology (Ob/Gyn) consultants, were included in this experimental study. The first two performances of the two groups on seven selected modules on a high-fidelity ultrasound simulator were used to identify valid and reliable metrics. Performance standards were determined and novices were instructed to continue practicing until they attained the performance level of an expert subgroup (n = 4). RESULTS: All 28 participants completed the selected modules twice and all novices reached the expert performance level. Of 153 metrics, 48 were able to be used to discriminate between the two groups' performance. The ultrasound novices scored a median of 43.8% (range, 17.9-68.9%) and the Ob/Gyn consultants scored a median of 82.8% (range, 60.4-91.7%) of the maximum sum score (P < 0.001). The ultrasound novices reached the expert level (88.4%) within a median of five iterations (range, 5-6), corresponding to an average of 219 min (range, 150-251 min) of training. The test/retest reliability was high, with an intraclass correlation coefficient of 0.93. CONCLUSIONS: Competence in the performance of gynecological ultrasound examination can be assessed in a valid and reliable way using virtual-reality simulation. The novices' performance improved with practice and their learning curves plateaued at the level of expert performance, following between 3 and 4 h of simulator training.

Feasibility of self-directed learning in clerkships.

BACKGROUND: Self-directed learning has been well described in preclinical settings. However, studies report conflicting results when self-directed initiatives are implemented in clinical clerkships. AIM: To explore the feasibility of self-directed learning stimulated by clinical encounter-cards (CECs) in clinical clerkships. METHODS: Two focus groups of year-four and year-five students were interviewed about the usefulness of CECs to their learning in clerkships. The CECs were then introduced in two cohorts of 248 year-four and 250 year-five medical students and evaluated on a nine-point scale with regard to usefulness and feasibility. RESULTS: The pilot groups reported that the CECs had positive effects in terms of engaging in diagnostic reasoning, reflection on management plans, and professional identity formation. However, the two large cohorts of students rated the usefulness of the CECs on learning in clerkship low (year-four: mean 2.92, SD 1.54; year-five: mean 2.28, SD 1.06) along with preceptor support (year-four: mean 2.68, SD 1.62; year-five: mean 2.59, SD 1.78, p = 0.34). CONCLUSION: Self-directed CECs can have a positive effect on participation and clinical reasoning but are highly dependent on the context of use. Self-directed learning initiatives that aim to increase participation in communities of practice may not be feasible without major faculty development initiatives.

Reliable and valid assessment of competence in endoscopic ultrasonography and fine-needle aspiration for mediastinal staging of non-small cell lung cancer.

BACKGROUND AND STUDY AIMS: Fine-needle aspiration (FNA) guided by endoscopic ultrasonography (EUS) is important in mediastinal staging of non-small cell lung cancer (NSCLC). Training standards and implementation strategies of this technique are currently under discussion. The aim of this study was to explore the reliability and validity of a newly developed EUS Assessment Tool (EUSAT) designed to measure competence in EUS - FNA for mediastinal staging of NSCLC. PATIENTS AND METHODS: A total of 30 patients with proven or suspected NSCLC underwent EUS - FNA for mediastinal staging by three trainees and three experienced physicians. Their performances were assessed prospectively by three experts in EUS under direct observation and again 2 months later in a blinded fashion using digital video-recordings. Based on the assessments, intra-rater reliability, inter-rater reliability, and construct validity were explored. RESULTS: The intra-rater reliability was good (Cronbach's α = 0.80), but comparison of results based on direct observations and blinded video-recordings indicated a significant bias favoring consultants (P = 0.022). Inter-rater reliability was very good (Cronbach's α = 0.93). However, one rater assessing five procedures or two raters each assessing four procedures were necessary to secure a generalizability coefficient of 0.80. The assessment tool demonstrated construct validity by discriminating between trainees and experienced physicians (P = 0.034). CONCLUSIONS: Competency in mediastinal staging of NSCLC using EUS and EUS - FNA can be assessed in a reliable and valid way using the EUSAT assessment tool. Measuring and defining competency and training requirements could improve EUS quality and benefit patient care.

Construct validity and reliability of structured assessment of endoVascular expertise in a simulated setting.

OBJECTIVES: To study the construct validity and reliability of a novel endovascular global rating scale, Structured Assessment of endoVascular Expertise (SAVE). DESIGN: A Clinical, experimental study. MATERIALS: Twenty physicians with endovascular experiences ranging from complete novices to highly experienced operators performed a video-recorded simulated contra-lateral iliac-artery-stenting procedure. The virtual-patient case was a novel technically challenging procedure presenting the distal arteries below the knee. METHODS: Three experts assessed the performances blinded to operator identity. Validity was analysed by correlating experience with performance results. Reliability was analysed according to generalisability theory. RESULTS: The mean score on the 29 items of the SAVE scale correlated well with clinical experience (R = 0.84, P < 0.01) and was found discriminative even among the more experienced participants having performed up to 500 endovascular procedures in total. Only the most experienced participants (>5000 procedures) obtained maximum scores. The inter-rater reliability was high (G = 0.94 and G = 0.95). The procedure time (median 69 min, range 32-86) correlated moderately with clinical experience (R = -0.53, P < 0.05), whereas the fluoroscopy time and amount of contrast fluid did not correlate. CONCLUSIONS: The construct validity and reliability of assessment with the SAVE scale was high when applied to performances in a simulation setting with advanced realism. No ceiling effect was present in the assessment situation.

Capturing the essence of developing endovascular expertise for the construction of a global assessment instrument.

OBJECTIVES: To explore what characterises the development of endovascular expertise and to construct a novel global assessment instrument. DESIGN: Literature review and an experimental study. MATERIALS AND METHODS: The literature was searched for information regarding available global rating scales (GRSs); scientific societies' official statements on endovascular competence; and task analyses of endovascular procedures. In the experimental study, clinicians performed a video-recorded simulated iliac-artery stenting procedure. Subsequently, by using the method of retrospective verbalisation, the clinicians were interviewed while watching their performance on video commenting on key issues of the construct. Data from all sources were analysed, categorised and synthesised into a novel rating scale. RESULTS: Available GRSs primarily included technical aspects of performance, whereas the competence statements, task analyses and clinicians' perceptions added a range of non-technical aspects. The novel rating scale SAVE (Structured Assessment of endoVascular Expertise) differs from prior scales by including issues of pre-planning; prediction of challenges; preparation of tools; management of imaging presentation; distinction of technical skills into external and internal control according to operator focus of visual attention; adaptation of strategy; clinical decision making; use of assistant; complications; inter-personal skills; and post-procedural planning. CONCLUSIONS: The essence of developing endovascular expertise goes far beyond mere technical aspects.

The testing effect on skills learning might last 6 months.

In a recent study we found that testing as a final activity in a skills course increases the learning outcome compared to spending an equal amount of time practicing. Whether this testing effect measured as skills performance can be demonstrated on long-term basis is not known. The research question was: does testing as a final activity in a cardio-pulmonary resuscitation (CPR) skills course increase learning outcome when assessed after half a year, compared to spending an equal amount of time practicing? The study was an assessor-blinded randomised controlled trial. A convenient sample of 7th semester medical students attending a mandatory CPR course was randomised to intervention course or control course. Participants were taught in small groups. The intervention course included 3.5 h skills training plus 30 min of skills testing. The practice-only control course lasted 4 h. Both groups were invited to a retention assessment of CPR skills half a year later. Participants included 89/180 (50%) of those invited to participate in the study. Mean performance score was 75.9 (SD 11.0) in the intervention group (N = 48) and 70.3 (SD 17.1) in the control group, effect size 0.4. The difference between groups was not statistically significant, P = 0.06. This study suggests that testing as a final activity in a CPR skills course might have an effect on long-term learning outcome compared to spending an equal amount of time practicing the skills. Although this difference was not statistically significant, the identified effect size of 0.4 can have important clinical and educational implications.

Assessment of Advanced Life Support competence when combining different test methods--reliability and validity.

UNLABELLED: Robust assessment of Advanced Life Support (ALS) competence is paramount to the credibility of ALS-provider certification and for estimating the learning outcome and retention of ALS competence following the courses. The European Resuscitation Council (ERC) provides two sets of MCQs and four Cardiac Arrest Simulation Test (CASTest) scenarios for the assessments according to guidelines 2005. AIMS: To analyse the reliability and validity of the individual sub-tests provided by ERC and to find a combination of MCQ and CASTest that provides a reliable and valid single effect measure of ALS competence. METHODS: Two groups of participants were included in this randomised, controlled experimental study: a group of newly graduated doctors, who had not taken the ALS course (N=17) and a group of students, who had passed the ALS course 9 months before the study (N=16). Reliability in terms of inter-rater agreement and generalisability across skills scenarios were estimated. Validity was studied in terms of equality of test difficulty and ability to discriminate performance between the groups. RESULTS: Inter-rater agreement on checklist scores were generally high, Intraclass Correlation Coefficients between 0.766 and 0.977. Inter-rater agreements on pass/fail decisions were not perfect. The one MCQ test was significantly more difficult than the other. There were no significant differences between CASTests. Generalisability theory was use to identify a composite of MCQ and CASTest scenarios that possessed high reliability, equality of test sets, and ability to discriminate between the two groups of supposedly different ALS competence. CONCLUSIONS: ERC sub-tests of ALS competence possess sufficient reliability and validity. A combined ALS score with equal weighting of one MCQ and one CASTest can be used as a single measurement of ALS competence.

What is the role of the consultant responsible for postgraduate education in the clinical department?

BACKGROUND: The organisation of specialist training is complex and involves many clinical departments. The position of consultants responsible for education (CRE) in specialist training at department level is poorly defined in the literature. AIMS: The aim of the study was to explore expectations of stakeholders concerning the role and position of a CRE in specialist training. METHOD: The role and position of the CRE was explored using focus group and semi-structured individual interviews. RESULTS: Knowledge of tasks and responsibilities was limited in all stakeholders except among CREs. The expectations of stakeholders to the CRE varied according to their position in the hospital hierarchy. In general terms the CRE was expected to assume overall responsibility for specialist training, promote a positive educational climate and secure quality of specialist training along with numerous administrative tasks. All interviewees expressed a wish for a strong leader at the same time they did not consider the position of the CRE influential. CONCLUSION: Along with improved information about the role of the CRE, formal education, proper job-descriptions and clear leadership in the organisation concerning specialist training might increase the influence and power of CREs.

Long-term effect of a course on in-training assessment in postgraduate specialist education.

BACKGROUND: In-training assessment has become an important part of clinical teachers' responsibilities. One way to ensure that clinical teachers are qualified for this role is setting up a course. A "Teach the teachers" course focusing on in-training assessment was designed for anaesthesiologists in Denmark. AIMS: To evaluate short and longer term effects of a course on in-training assessment for clinical teachers in Anaesthesiology. METHOD: Fifty-one anaesthesiologists attended a 2-day interactive course about in-training assessment. Effects of the course on knowledge were assessed using identical pre- and post- tests. Longer- term effects were measured six months after the course using the same test. Self-reported use of in-training assessment methods was evaluated using supplemental questions in the follow-up test. RESULTS: There were significant increases in knowledge about in-training assessment immediately following the course (effect size, Cohens d = 1, 5). The knowledge was retained six months later. Knowledge about assessment by clinical structured observation and by written assignments showed further increases in the follow-up period. Participants used the various assessment methods in their daily practice during the six-month study period. CONCLUSION: A focused "Teach the teachers" course during the implementation phase of a new assessment programme increased participants' knowledge about in-training assessment.

Person-task-context: a model for designing curriculum and in-training assessment in postgraduate education.

Structured curricula for senior house officers have often been lacking. The aim of this study was to trial a person-task-context model in designing a curriculum and in-training assessment (ITA) programme for SHOs in internal medicine. A working group designed the programme based on triangulation of information from interviews with trainees and programme directors, analysis of patient case mix and national quality assurance data. The interview data showed that the main difference currently between trainee levels was in expected degree of responsibility for patient management rather than in actual tasks. Key learning needs were how to take a structured approach to the tasks and get an overview of situations. SHOs expressed a need for explicit learning goals and standards of performance. SHOs requested formal teaching in non-medical aspects of competence such as communication, interpersonal skills and professionalism. This article points out how consideration of the type of trainees involved, the tasks they must do and learn, and the context in which they work are important in designing postgraduate curricula. The person-task-context model can be used to tailor curricula and ITA that support learning and may be especially beneficial in promoting learning in non-dominant areas of a specialty.

Ventilation-perfusion distributions and central hemodynamics in chronic obstructive pulmonary disease. Effects of terbutaline administration.

We studied the effects of intravenous terbutaline on VA/Q distributions and central hemodynamics in 11 patients with mixed-type COPD. Terbutaline caused an increase in VA/Q inequality in patients having PaO2 values greater than 60 mm Hg which resulted in a moderate fall in the PaO2. Patients with PaO2 values less than 60 mm Hg, the highest mean PAPs and the poorest spirometric performances demonstrated no significant changes in VA/Q distributions or PaO2 after terbutaline. Cardiac output increased 40 to 60 percent in all patients after terbutaline with an increase in tissue oxygen delivery. Mean PAP did not change in any patient after terbutaline and pulmonary vasodilatation was indicated by a decrease of calculated static PVR. The decrease of PaO2 after terbutaline in COPD is related to a further deterioration of existing VA/Q relationships. The cause of these effects and lack of such responses in patients with more advanced disease are discussed.

Intestinal distension test, a method for evaluating intermittent visceral pain in the rabbit.

Behavioural response to intestinal distension was studied in 12 female New Zealand albino rabbits under various conditions. On increasing intraluminal pressures, the rabbits elicited uniform behavioural responses within discrete pressure ranges, notably a sudden pelvic withdrawal at 30-50 mmHg. The pressure provoking pelvic withdrawal was chosen as the test parameter and proved to be individually reproducible, irrespective of fasting/non-fasting or the time of day and with no signs of adaptation in six days consecutive measurements. Morphine modified the pressure response in a dose-dependent manner, whereas isotonic saline or pentobarbital had no effect. In conclusion, the intestinal distension test is reproducible and mimicks intermittent visceral pain in the rabbit. This allows for paired observations in small animal populations with a minimum of discomfort to the animals, which offers a major advantage when comparing with the existing visceral pain tests.