Formative Assessment of Performance in Diagnostic Ultrasound Using Simulation and Quantitative and Objective Metrics.

BACKGROUND: We developed simulator-based tools for assessing provider competence in transthoracic echocardiography (TTE) and vascular duplex scanning. METHODS: Psychomotor (technical) skill in TTE image acquisition was calculated from the deviation angle of an acquired image from the anatomically correct view. We applied this metric for formative assessment to give feedback to learners and evaluate curricula.Psychomotor skill in vascular ultrasound was measured in terms of dexterity and image plane location; cognitive skill was assessed from measurements of blood flow velocity, parameter settings, and diagnosis. The validity of the vascular simulator was assessed from the accuracy with which experts can measure peak systolic blood flow velocity (PSV). RESULTS: In the TTE simulator, the skill metric enabled immediate feedback, formative assessment of curriculum efficacy, and comparison of curriculum outcomes. The vascular duplex ultrasound simulator also provided feedback, and experts' measurements of PSV deviated from actual PSV in the model by <10%. CONCLUSIONS: Skill in acquiring diagnostic ultrasound images of organs and vessels can be measured using simulation in an objective, quantitative, and standardized manner. Current applications are provision of feedback to learners to enable training without direct faculty oversight and formative assessment of curricula. Simulator-based metrics could also be applied for summative assessment.

Test of simulator-based assessment of psychomotor skill in transthoracic echo.

BACKGROUND: We developed a transthoracic echo simulator that measures technical skill in image acquisition in terms of the deviation angle between an acquired image and the anatomically correct plane for that view. We studied whether this metric reflects the clinical experience of providers. METHODS: Attendees at an echo course or at the annual meeting of the Swedish Heart Association were invited to test themselves on the simulator by scanning a mannequin and acquiring four views in 15 min: parasternal long axis (pLAX) in patient 1, apical four chamber and apical long axis (aLAX) in patient 2 and pLAX in patient 3. Their experience with echo was assessed from duration in years and procedure volume in the past year. Image acquisition error was assessed from the deviation angle. RESULTS: Of 61 participants, there were 37 physicians and 24 non-physicians (22 sonographers and two nurses). Non-physicians had higher procedure volume than physicians (850 ± 599 versus 312 ± 393 tests year-1 , P<0·001); both had similar duration of experience (9 ± 8 versus 12 ± 11 years, P = NS). The deviation angle for aLAX (55 ± 37 degrees) was higher than for any other view (P<0·00001). aLAX was the only view whose deviation angle correlated significantly with experience and only with procedure volume (r = -0·357, P = 0·008). CONCLUSIONS: These results demonstrate that deviation angle, a novel metric of technical skill in image acquisition, reflects clinical experience. Simulator-based testing provides objective and quantitative assessment that may be of value in the certification of trainees and for maintenance of certification.

Simulation for competency assessment in vascular and cardiac ultrasound.

Healthcare providers who use peripheral vascular and cardiac ultrasound require specialized training to develop the technical and interpretive skills necessary to perform accurate diagnostic tests. Assessment of competence is a critical component of training that documents a learner's progress and is a requirement for competency-based medical education (CBME) as well as specialty certification or credentialing. The use of simulation for CBME in diagnostic ultrasound is particularly appealing since it incorporates both the psychomotor and cognitive domains while eliminating dependency on the availability of live patients with a range of pathology. However, successful application of simulation in this setting requires realistic, full-featured simulators and appropriate standardized metrics for competency testing. The principal diagnostic parameter in peripheral vascular ultrasound is measurement of peak systolic velocity (PSV) on Doppler spectral waveforms, and simulation of Doppler flow detection presents unique challenges. The computer-based duplex ultrasound simulator developed at the University of Washington uses computational fluid dynamics modeling and presents real-time color-flow Doppler images and Doppler spectral waveforms along with the corresponding B-mode images. This simulator provides a realistic scanning experience that includes measuring PSV in various arterial segments and applying actual diagnostic criteria. Simulators for echocardiography have been available since the 1990s and are currently more advanced than those for peripheral vascular ultrasound. Echocardiography simulators are now offered for both transesophageal echo and transthoracic echo. These computer-based simulators have 3D graphic displays that provide feedback to the learner and metrics for assessment of technical skill that are based on transducer tracking data. Such metrics provide a motion-based or kinematic analysis of skill in performing cardiac ultrasound. The use of simulation in peripheral vascular and cardiac ultrasound can provide a standardized and readily available method for training and competency assessment.

Quantitative assessment of regional left ventricular function with cardiac MRI: three-dimensional centersurface method.

OBJECTIVES: The purpose of this study was to provide the first in vivo validation of a three-dimensional (3D) method to quantify regional left ventricular (LV) function with cardiac magnetic resonance (CMR) imaging after myocardial infarction (MI). BACKGROUND: Current cardiac methods to analyze LV function are limited by geometric assumptions and observer biases. METHODS: MI was induced percutaneously by 90-min proximal left circumflex artery balloon occlusion in 25 Yucatan minipigs. Cine and contrast-enhanced (CE) CMR imaging was performed at 5 days (n = 21) and 8 weeks (n = 22) post-MI. Twelve control animals without MI were also imaged. Regional wall thickening was measured orthogonal to the myocardial wall using the centersurface method. The left ventricle was divided into 16 segments (six basal 60 degrees , six middle 60 degrees , four apical 90 degrees ). Normal ranges for segmental wall thickness and wall thickening were defined as mean +/- 2D in control hearts. Hypokinesis was defined as a segmental thickening value below the normal range. RESULTS: Hypokinesis following MI was identified in the inferior, inferolateral and anterolateral segments when compared with controls and corresponded to areas of infarction by CE CMR. The aggregate wall thickening was also expressed as a percentage at 5 days (Infarct zone: 15% +/- 16% vs. NonInfarct zone 33% +/- 20%, P < 0.001) and 8 weeks (Infarct zone 20% +/- 20% vs. NonInfarct zone 32% +/- 22%, P < 0.001). CONCLUSIONS: The centersurface method can quantify regional wall thickening and spatially identify regions of abnormal function in 3D after MI without relying on geometric assumptions. This method may be a valuable tool to quantify regional LV function in the assessment of myocardial viability, ischemia, infarction, and the response to therapeutic interventions.

Validation of viability assessment by electromechanical mapping by three-dimensional reconstruction with dobutamine stress echocardiography in patients with coronary artery disease.

We evaluated the ability of electromechanical mapping (EMM) to discriminate between normal, viable, and nonviable (scarred) myocardium in patients with coronary artery disease versus dobutamine stress echocardiography (DSE) when the correspondence between the test and reference data sets is established via a common 3-dimensional reconstruction of the left ventricle. We studied 21 patients with coronary artery disease who underwent angiography, biplane ventriculography, and EMM within 1 month of DSE. A 3-dimensional left ventricular (LV) reconstruction was prepared from the ventriculogram and spatially aligned with EMM. EMM measurements of unipolar voltage, bipolar voltage, and local linear shortening were projected onto the three-dimensional left ventricle, averaged in each of 16 segments, and compared with DSE viability (normal, viable, scar) assessed at a core laboratory. All of the EMM measurements varied significantly (p <0.001) between the normal, viable, and scarred myocardium as assessed by DSE. Local linear shortening for normal, viable, and scarred segments was 10.4 +/- 6.5%, 7.8 +/- 5.6%, and 4.8 +/- 4.4%, respectively. In discriminating between these 3 groups, local linear shortening was more powerful than unipolar voltage or bipolar voltage (F = 20.765, F = 10.655, F = 4.795, respectively). Local linear shortening correlated best with viability, perhaps because it shares the same cognitive function as DSE. Three-dimensional analysis provides an anatomic framework that enables direct comparison of data from multiple imaging modalities rather than assuming segmental correspondence. Our results show that EMM provides significant on-line, diagnostic information on myocardial viability assessed by DSE on a segment-by-segment basis.

Three-dimensional assessment of two-dimensional technique for evaluation of right ventricular function by tricuspid annulus motion.

BACKGROUND: Measurement of tricuspid annulus motion (TAM) is an easy way to estimate right ventricular ejection fraction (RVEF). However the accuracy of two-dimensional (2-D) methods for analyzing the three-dimensional (3-D) structure of the tricuspid annulus has not been evaluated. OBJECTIVE: This study evaluated the accuracy with which 2-D measurements of TAM reflect RVEF using 3-D reconstructions of the heart at end diastole (ED) and end systole (ES). METHODS: 2-D echocardiographic studies were performed on 12 subjects and used to reconstruct the RV and tricuspid annulus in 3-D at ED and ES. Measurements of TAM from medial and lateral positions on the annulus were selected from the standard echocardiographic apical four-chamber view. The minimum and maximum possible TAM values, RV volumes, and movement of the apex of the heart along the trajectory of TAM were calculated from the 3-D reconstructions. RESULTS: TAM correlated highly with RVEF (r > or = 0.90). Values found by 2-D and 3-D techniques were not significantly different. Correcting TAM for apex motion did not improve correlation. Summation of medial and lateral TAM data increased correlation values slightly relative to lateral TAM alone. Regional aberrant contractility degraded the predictive value of TAM. CONCLUSION: Estimation of RVEF from 2-D echo measurement of TAM is accurate, especially when medial and lateral TAM are summed, except in patients with severe apical RV dysfunction.