Validating the virtual: a deep dive into ultrasound simulator metrics in otorhinolaryngology.

PURPOSE: This study aimed to assess the validity of simulation-based assessment of ultrasound skills for thyroid ultrasound. METHODS: The study collected validity evidence for simulation-based ultrasound assessment of thyroid ultrasound skills. Experts (n = 8) and novices (n = 21) completed a test containing two tasks and four cases on a virtual reality ultrasound simulator (U/S Mentor's Neck Ultrasound Module). Validity evidence was collected and structured according to Messick's validity framework. The assessments being evaluated included built-in simulator metrics and expert-based evaluations using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale. RESULTS: Out of 64 built-in simulator metrics, 9 (14.1%) exhibited validity evidence. The internal consistency of these metrics was strong (Cronbach's α = 0.805) with high test-retest reliability (intraclass correlation coefficient = 0.911). Novices achieved an average score of 41.9% (SD = 24.3) of the maximum, contrasting with experts at 81.9% (SD = 16.7). Time comparisons indicated minor differences between experts (median: 359 s) and novices (median: 376.5 s). All OSAUS items differed significantly between the two groups. The correlation between correctly entered clinical findings and the OSAUS scores was 0.748 (p < 0.001). The correlation between correctly entered clinical findings and the metric scores was 0.801 (p < 0.001). CONCLUSION: While simulation-based training is promising, only 14% of built-in simulator metrics could discriminate between novices and ultrasound experts. Already-established competency frameworks such as OSAUS provided strong validity evidence for the assessment of otorhinolaryngology ultrasound competence.

NON-pharmacological Approach Less Invasive Surfactant Administration (NONA-LISA) trial: protocol for a randomised controlled trial.

INTRODUCTION: Using pre-procedure analgesia with the risk of apnoea may complicate the Less Invasive Surfactant Administration (LISA) procedure or reduce the effect of LISA. METHODS: The NONA-LISA trial (ClinicalTrials.gov, NCT05609877) is a multicentre, blinded, randomised controlled trial aiming at including 324 infants born before 30 gestational weeks, meeting the criteria for surfactant treatment by LISA. Infants will be randomised to LISA after administration of fentanyl 0.5-1 mcg/kg intravenously (fentanyl group) or isotonic saline solution intravenously (saline group). All infants will receive standardised non-pharmacological comfort care before and during the LISA procedure. Additional analgesics will be provided at the clinician's discretion. The primary outcome is the need for invasive ventilation, meaning mechanical or manual ventilation via an endotracheal tube, for at least 30 min (cumulated) within 24 h of the procedure. Secondary outcomes include the modified COMFORTneo score during the procedure, bronchopulmonary dysplasia at 36 weeks, and mortality at 36 weeks. DISCUSSION: The NONA-LISA trial has the potential to provide evidence for a standardised approach to relief from discomfort in preterm infants during LISA and to reduce invasive ventilation. The results may affect future clinical practice. IMPACT: Pre-procedure analgesia is associated with apnoea and may complicate procedures that rely on regular spontaneous breathing, such as Less Invasive Surfactant Administration (LISA). This randomised controlled trial addresses the effect of analgesic premedication in LISA by comparing fentanyl with a placebo (isotonic saline) in infants undergoing the LISA procedure. All infants will receive standardised non-pharmacological comfort. The NONA-LISA trial has the potential to provide evidence for a standardised approach to relief from discomfort or pain in preterm infants during LISA and to reduce invasive ventilation. The results may affect future clinical practice regarding analgesic treatment associated with the LISA procedure.

Surgical gestures can be used to assess surgical competence in robot-assisted surgery : A validity investigating study of simulated RARP.

To collect validity evidence for the assessment of surgical competence through the classification of general surgical gestures for a simulated robot-assisted radical prostatectomy (RARP). We used 165 video recordings of novice and experienced RARP surgeons performing three parts of the RARP procedure on the RobotiX Mentor. We annotated the surgical tasks with different surgical gestures: dissection, hemostatic control, application of clips, needle handling, and suturing. The gestures were analyzed using idle time (periods with minimal instrument movements) and active time (whenever a surgical gesture was annotated). The distribution of surgical gestures was described using a one-dimensional heat map, snail tracks. All surgeons had a similar percentage of idle time but novices had longer phases of idle time (mean time: 21 vs. 15 s, p < 0.001). Novices used a higher total number of surgical gestures (number of phases: 45 vs. 35, p < 0.001) and each phase was longer compared with those of the experienced surgeons (mean time: 10 vs. 8 s, p < 0.001). There was a different pattern of gestures between novices and experienced surgeons as seen by a different distribution of the phases. General surgical gestures can be used to assess surgical competence in simulated RARP and can be displayed as a visual tool to show how performance is improving. The established pass/fail level may be used to ensure the competence of the residents before proceeding with supervised real-life surgery. The next step is to investigate if the developed tool can optimize automated feedback during simulator training.

Improving Skin Cancer Diagnostics Through a Mobile App With a Large Interactive Image Repository: Randomized Controlled Trial.

BACKGROUND: Skin cancer diagnostics is challenging, and mastery requires extended periods of dedicated practice. OBJECTIVE: The aim of the study was to determine if self-paced pattern recognition training in skin cancer diagnostics with clinical and dermoscopic images of skin lesions using a large-scale interactive image repository (LIIR) with patient cases improves primary care physicians' (PCPs') diagnostic skills and confidence. METHODS: A total of 115 PCPs were randomized (allocation ratio 3:1) to receive or not receive self-paced pattern recognition training in skin cancer diagnostics using an LIIR with patient cases through a quiz-based smartphone app during an 8-day period. The participants' ability to diagnose skin cancer was evaluated using a 12-item multiple-choice questionnaire prior to and 8 days after the educational intervention period. Their thoughts on the use of dermoscopy were assessed using a study-specific questionnaire. A learning curve was calculated through the analysis of data from the mobile app. RESULTS: On average, participants in the intervention group spent 2 hours 26 minutes quizzing digital patient cases and 41 minutes reading the educational material. They had an average preintervention multiple choice questionnaire score of 52.0% of correct answers, which increased to 66.4% on the postintervention test; a statistically significant improvement of 14.3 percentage points (P<.001; 95% CI 9.8-18.9) with intention-to-treat analysis. Analysis of participants who received the intervention as per protocol (500 patient cases in 8 days) showed an average increase of 16.7 percentage points (P<.001; 95% CI 11.3-22.0) from 53.9% to 70.5%. Their overall ability to correctly recognize malignant lesions in the LIIR patient cases improved over the intervention period by 6.6 percentage points from 67.1% (95% CI 65.2-69.3) to 73.7% (95% CI 72.5-75.0) and their ability to set the correct diagnosis improved by 10.5 percentage points from 42.5% (95% CI 40.2%-44.8%) to 53.0% (95% CI 51.3-54.9). The diagnostic confidence of participants in the intervention group increased on a scale from 1 to 4 by 32.9% from 1.6 to 2.1 (P<.001). Participants in the control group did not increase their postintervention score or their diagnostic confidence during the same period. CONCLUSIONS: Self-paced pattern recognition training in skin cancer diagnostics through the use of a digital LIIR with patient cases delivered by a quiz-based mobile app improves the diagnostic accuracy of PCPs. TRIAL REGISTRATION: ClinicalTrials.gov NCT05661370; https://classic.clinicaltrials.gov/ct2/show/NCT05661370.

Acquisition and usage of robotic surgical data for machine learning analysis.

BACKGROUND: The increasing use of robot-assisted surgery (RAS) has led to the need for new methods of assessing whether new surgeons are qualified to perform RAS, without the resource-demanding process of having expert surgeons do the assessment. Computer-based automation and artificial intelligence (AI) are seen as promising alternatives to expert-based surgical assessment. However, no standard protocols or methods for preparing data and implementing AI are available for clinicians. This may be among the reasons for the impediment to the use of AI in the clinical setting. METHOD: We tested our method on porcine models with both the da Vinci Si and the da Vinci Xi. We sought to capture raw video data from the surgical robots and 3D movement data from the surgeons and prepared the data for the use in AI by a structured guide to acquire and prepare video data using the following steps: 'Capturing image data from the surgical robot', 'Extracting event data', 'Capturing movement data of the surgeon', 'Annotation of image data'. RESULTS: 15 participant (11 novices and 4 experienced) performed 10 different intraabdominal RAS procedures. Using this method we captured 188 videos (94 from the surgical robot, and 94 corresponding movement videos of the surgeons' arms and hands). Event data, movement data, and labels were extracted from the raw material and prepared for use in AI. CONCLUSION: With our described methods, we could collect, prepare, and annotate images, events, and motion data from surgical robotic systems in preparation for its use in AI.

Effect of Histopathological Explanations for Dermoscopic Criteria on Learning Curves in Skin Cancer Training: a Randomized Controlled Trial.

INTRODUCTION: Case-based training improves novices pattern recognition and diagnostic accuracy in skin cancer diagnostics. However, it is unclear how pattern recognition is best taught in conjunction with the knowledge needed to justify a diagnosis. OBJECTIVES: The aim of this study was to examine whether an explanation of the underlying histopathological reason for dermoscopic criteria improves skill acquisition and retention during case-based training in skin cancer diagnostics. METHODS: In this double-blinded randomized controlled trial, medical students underwent eight days of case-based training in skin cancer diagnostics, which included access to written diagnosis modules. The modules dermoscopic subsections differed between the study groups. All participants received a general description of the criteria, but the intervention group additionally received a histopathological explanation. RESULTS: Most participants (78%) passed a reliable test in skin cancer diagnostics, following a mean training time of 217 minutes. Access to histopathological explanations did not affect participants' learning curves or skill retention. CONCLUSIONS: The histopathological explanation did not affect the students, but the overall educational approach was efficient and scalable.

Simulation-based assessment of robotic cardiac surgery skills: An international multicenter, cross-specialty trial.

Objective: This study aimed to investigate the validity of simulation-based assessment of robotic-assisted cardiac surgery skills using a wet lab model, focusing on the use of a time-based score (TBS) and modified Global Evaluative Assessment of Robotic Skills (mGEARS) score. Methods: We tested 3 wet lab tasks (atrial closure, mitral annular stitches, and internal thoracic artery [ITA] dissection) with both experienced robotic cardiac surgeons and novices from multiple European centers. The tasks were assessed using 2 tools: TBS and mGEARS score. Reliability, internal consistency, and the ability to discriminate between different levels of competence were evaluated. Results: The results demonstrated a high internal consistency for all 3 tasks using mGEARS assessment tool. The mGEARS score and TBS could reliably discriminate between different levels of competence for the atrial closure and mitral stitches tasks but not for the ITA harvesting task. A generalizability study also revealed that it was feasible to assess competency of the atrial closure and mitral stitches tasks using mGEARS but not the ITA dissection task. Pass/fail scores were established for each task using both TBS and mGEARS assessment tools. Conclusions: The study provides sufficient evidence for using TBS and mGEARS scores in evaluating robotic-assisted cardiac surgery skills in wet lab settings for intracardiac tasks. Combining both assessment tools enhances the evaluation of proficiency in robotic cardiac surgery, paving the way for standardized, evidence-based preclinical training and credentialing. Clinical trial registry number: NCT05043064.

Using machine learning to identify quality-of-care predictors for emergency caesarean sections: a retrospective cohort study.

OBJECTIVES: Emergency caesarean sections (ECS) are time-sensitive procedures. Multiple factors may affect team efficiency but their relative importance remains unknown. This study aimed to identify the most important predictors contributing to quality of care during ECS in terms of the arrival-to-delivery interval. DESIGN: A retrospective cohort study. ECS were classified by urgency using emergency categories one/two and three (delivery within 30 and 60 min). In total, 92 predictor variables were included in the analysis and grouped as follows: 'Maternal objective', 'Maternal psychological', 'Fetal factors', 'ECS Indication', 'Emergency category', 'Type of anaesthesia', 'Team member qualifications and experience' and 'Procedural'. Data was analysed with a linear regression model using elastic net regularisation and jackknife technique to improve generalisability. The relative influence of the predictors, percentage significant predictor weight (PSPW) was calculated for each predictor to visualise the main determinants of arrival-to-delivery interval. SETTING AND PARTICIPANTS: Patient records for mothers undergoing ECS between 2010 and 2017, Nordsjællands Hospital, Capital Region of Denmark. PRIMARY OUTCOME MEASURES: Arrival-to-delivery interval during ECS. RESULTS: Data was obtained from 2409 patient records for women undergoing ECS. The group of predictors representing 'Team member qualifications and experience' was the most important predictor of arrival-to-delivery interval in all ECS emergency categories (PSPW 25.9% for ECS category one/two; PSPW 35.5% for ECS category three). In ECS category one/two the 'Indication for ECS' was the second most important predictor group (PSPW 24.9%). In ECS category three, the second most important predictor group was 'Maternal objective predictors' (PSPW 24.2%). CONCLUSION: This study provides empirical evidence for the importance of team member qualifications and experience relative to other predictors of arrival-to-delivery during ECS. Machine learning provides a promising method for expanding our current knowledge about the relative importance of different factors in predicting outcomes of complex obstetric events.

The concept of errors in medical education: a scoping review.

The purpose of this scoping review was to explore how errors are conceptualized in medical education contexts by examining different error perspectives and practices. This review used a scoping methodology with a systematic search strategy to identify relevant studies, written in English, and published before January 2021. Four medical education journals (Medical Education, Advances in Health Science Education, Medical Teacher, and Academic Medicine) and four clinical journals (Journal of the American Medical Association, Journal of General Internal Medicine, Annals of Surgery, and British Medical Journal) were purposively selected. Data extraction was charted according to a data collection form. Of 1505 screened studies, 79 studies were included. Three overarching perspectives were identified: 'understanding errors') (n = 31), 'avoiding errors' (n = 25), 'learning from errors' (n = 23). Studies that aimed at'understanding errors' used qualitative methods (19/31, 61.3%) and took place in the clinical setting (19/31, 61.3%), whereas studies that aimed at 'avoiding errors' and 'learning from errors' used quantitative methods ('avoiding errors': 20/25, 80%, and 'learning from errors': 16/23, 69.6%, p = 0.007) and took place in pre-clinical (14/25, 56%) and simulated settings (10/23, 43.5%), respectively (p < 0.001). The three perspectives differed significantly in terms of inclusion of educational theory: 'Understanding errors' studies 16.1% (5/31),'avoiding errors' studies 48% (12/25), and 'learning from errors' studies 73.9% (17/23), p < 0.001. Errors in medical education and clinical practice are defined differently, which makes comparisons difficult. A uniform understanding is not necessarily a goal but improving transparency and clarity of how errors are currently conceptualized may improve our understanding of when, why, and how to use and learn from errors in the future.

Standard Setting in Simulation-based Training of Surgical Procedures: A Systematic Review.

OBJECTIVE: This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures. SUMMARY OF BACKGROUND: Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods. METHODS: Systematic literature search was done in 4 databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with the application of a standard setting were included. Quality of evidence was appraised using GRADE. RESULTS: Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, setup, and expert level classification. The studies were appraised as having low and moderate evidence. CONCLUSION: Surgical education is shifting toward competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid, and reliable assessment.

Consensus on Training and Assessment of Competence in Performing Chorionic Villus Sampling and Amniocentesis: An International Delphi Survey.

INTRODUCTION: The aim of this study was to obtain expert consensus on the content of a curriculum for learning chorionic villus sampling (CVS) and amniocentesis (AC) and the items of an assessment tool to evaluate CVS and AC competence. METHODS: We used a 3-round iterative Delphi process. A steering committee supervised all processes. Seven international collaborators were identified to expand the breadth of the study internationally. The collaborators invited fetal medicine experts to participate as panelists. In the first round, the panelists suggested content for a CVS/AC curriculum and an assessment tool. The steering committee organized and condensed the suggested items and presented them to the panelists in round 2. In the second round, the panelists rated and commented on the suggested items. The results were processed by the steering committee and presented to the panelists in the third round, where final consensus was obtained. Consensus was defined as support by more than 80% of the panelists for an item. RESULTS: Eighty-six experts agreed to participate in the study. The panelists represented 16 countries across 4 continents. The final list of curricular content included 12 theoretical and practical items. The final assessment tool included 11 items, systematically divided into 5 categories: pre-procedure, procedure, post-procedure, nontechnical skills, and overall performance. These items were provided with behavioral scale anchors to rate performance, and an entrustment scale was used for the final overall assessment. CONCLUSION: We established consensus among international fetal medicine experts on content to be included in a CVS/AC curriculum and on an assessment tool to evaluate CVS/AC skills. These results are important to help transition current training and assessment methods from a time- and volume-based approach to a competency-based approach which is a key step in improving patient safety and outcomes for the 2 most common invasive procedures in fetal medicine.

Protocol for a randomised clinical trial of transoral ultrasound versus standard of care in the diagnosis of peritonsillar abscess.

INTRODUCTION: Peritonsillar infection is a common complication to acute tonsillitis in younger adults. If peritonsillar cellulitis progresses to a peritonsillar abscess (PTA), the primary treatment is surgical drainage. But distinguishing cellulitis from PTA on a standard clinical examination is difficult. This trial aims to explore whether point-of-care transoral ultrasound can improve diagnostic accuracy and guide successful needle aspiration in patients referred with PTA. METHODS: A randomised, controlled multicentre trial will be conducted at the departments of otorhinolaryngology, head and neck surgery at Rigshospitalet and Odense University Hospital. Patients referred with PTA will be randomised to either standard clinical examination (control) or standard clinical examination with supplemental transoral ultrasound (intervention). The diagnostic accuracy, the total number of performed needle aspirations and the proportion of successful needle aspirations will be compared between the two groups. The difference will be evaluated using binary logistic regression and a generalised estimating equation to adjust for clustering of data within each physician and each hospital. A total of 88 patients are necessary to measure the clinical effect of adding transoral ultrasound. CONCLUSIONS: This study will explore the clinical benefits of adding transoral ultrasound to the diagnostic work-up of patients with peritonsillar infections. FUNDING: The Rigshospitalet and Odense University Hospital Foundation. TRIAL REGISTRATION: Clinicaltrials NCT03824288.

The Effects of Simulation-based Transvaginal Ultrasound Training on Quality and Efficiency of Care: A Multicenter Single-blind Randomized Trial.

OBJECTIVE: To explore the effect of adding simulation-based transvaginal ultrasound training to trainees' clinical training compared with only clinical training on quality of and efficiency of care. BACKGROUND: Simulation-based ultrasound training may be an effective adjunct to clinical training, but no studies have examined its effects on quality and efficiency of care. METHODS: Trainees from 4 University Hospitals in East Denmark were included (N = 54). Participants were randomized to either simulation-based ultrasound training and clinical training (intervention group, n = 28), or to clinical training only (control group, n = 26).The primary outcome was patient-reported discomfort during transvaginal ultrasound examinations performed by study participants. Secondary outcomes included patient-reported perceived safety and confidence in ultrasound provider. Finally, the need for trainee supervision or repeated patient examinations was recorded. RESULTS: In total, 1150 patient ratings were collected. The intervention was associated with a reduction of patient discomfort by 18.5% [95% confidence interval (CI), 10.7-25.5; P < 0.001), and with a 7.9% (95% CI, 0.5-14.7; P = 0.04) increase in perceived safety. The intervention group participants received 11.1% (95% CI, 2.5-18.9) higher scores on patients' confidence compared with control group participants (P = 0.01). When the number of days of clinical training was doubled, the odds for trainee supervision or repeated patient examination was reduced by 45.3% (95% CI, 33.5-55.1) and 19.8% (95% CI, 4.1-32.9) in the intervention and control group, respectively (P = 0.005). CONCLUSIONS: Simulation-based ultrasound training improved quality of care and reduced the need for repeated patient examination and trainee supervision.

Training and Assessment of Hysteroscopic Skills: A Systematic Review.

OBJECTIVE: The aim of this systematic review was to identify studies on hysteroscopic training and assessment. DESIGN: PubMed, Excerpta Medica, the Cochrane Library, and Web of Science were searched in January 2015. Manual screening of references and citation tracking were also performed. Studies on hysteroscopic educational interventions were selected without restrictions on study design, populations, language, or publication year. A qualitative data synthesis including the setting, study participants, training model, training characteristics, hysteroscopic skills, assessment parameters, and study outcomes was performed by 2 authors working independently. Effect sizes were calculated when possible. Overall, 2 raters independently evaluated sources of validity evidence supporting the outcomes of the hysteroscopy assessment tools. RESULTS: A total of 25 studies on hysteroscopy training were identified, of which 23 were performed in simulated settings. Overall, 10 studies used virtual-reality simulators and reported effect sizes for technical skills ranging from 0.31 to 2.65; 12 used inanimate models and reported effect sizes for technical skills ranging from 0.35 to 3.19. One study involved live animal models; 2 studies were performed in clinical settings. The validity evidence supporting the assessment tools used was low. Consensus between the 2 raters on the reported validity evidence was high (94%). CONCLUSIONS: This systematic review demonstrated large variations in the effect of different tools for hysteroscopy training. The validity evidence supporting the assessment of hysteroscopic skills was limited.

Transfer from point-of-care Ultrasonography training to diagnostic performance on patients--a randomized controlled trial.

BACKGROUND: Clinicians are increasingly using point-of-care ultrasonography for bedside examinations of patients. However, proper training is needed in this technique, and it is unknown whether the skills learned from focused Ultrasonography courses are being transferred to diagnostic performance on patients. METHODS: Thirty-one physicians were randomized to participate in a focused Ultrasonography course or control circumstances before they examined 4 patients with different abdominal conditions by ultrasonography. Performance scores and diagnostic accuracy were compared using independent samples t test and binary logistic regression, respectively. RESULTS: There was a significant difference in the performance score between the intervention group (27.4%) and the control group (18.0%, P = .004) and the diagnostic accuracy between the intervention group (65%) and the control group (39%, P = .014). CONCLUSIONS: Clinicians could successfully transfer learning from an Ultrasonography course to improve diagnostic performance on patients. However, our results also indicate a need for more training when new technologies such as point-of-care ultrasonography are introduced.

Reliable and valid assessment of point-of-care ultrasonography.

OBJECTIVE: To explore the reliability and validity of the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale for point-of-care ultrasonography (POC US) performance. BACKGROUND: POC US is increasingly used by clinicians and is an essential part of the management of acute surgical conditions. However, the quality of performance is highly operator-dependent. Therefore, reliable and valid assessment of trainees' ultrasonography competence is needed to ensure patient safety. METHODS: Twenty-four physicians, representing novices, intermediates, and experts in POC US, scanned 4 different surgical patient cases in a controlled set-up. All ultrasound examinations were video-recorded and assessed by 2 blinded radiologists using OSAUS. Reliability was examined using generalizability theory. Construct validity was examined by comparing performance scores between the groups and by correlating physicians' OSAUS scores with diagnostic accuracy. RESULTS: The generalizability coefficient was high (0.81) and a D-study demonstrated that 1 assessor and 5 cases would result in similar reliability. The construct validity of the OSAUS scale was supported by a significant difference in the mean scores between the novice group (17.0; SD 8.4) and the intermediate group (30.0; SD 10.1), P = 0.007, as well as between the intermediate group and the expert group (72.9; SD 4.4), P = 0.04, and by a high correlation between OSAUS scores and diagnostic accuracy (Spearman ρ correlation coefficient = 0.76; P < 0.001). CONCLUSIONS: This study demonstrates high reliability as well as evidence of construct validity of the OSAUS scale for assessment of POC US competence. Hence, the OSAUS scale may be suitable for both in-training as well as end-of-training assessment.