Cannulation Skill Assessment Using Functional Data Analysis.

OBJECTIVE: A clinician's operative skill-the ability to safely and effectively perform a procedure-directly impacts patient outcomes and well-being. Therefore, it is necessary to accurately assess skill progression during medical training as well as develop methods to most efficiently train healthcare professionals. METHODS: In this study, we explore whether time-series needle angle data recorded during cannulation on a simulator can be analyzed using functional data analysis methods to (1) identify skilled versus unskilled performance and (2) relate angle profiles to degree of success of the procedure. RESULTS: Our methods successfully differentiated between types of needle angle profiles. In addition, the identified profile types were associated with degrees of skilled and unskilled behavior of subjects. Furthermore, the types of variability in the dataset were analyzed, providing particular insight into the overall range of needle angles used as well as the rate of change of angle as cannulation progressed in time. Finally, cannulation angle profiles also demonstrated an observable correlation with degree of cannulation success, a metric that is closely related to clinical outcome. CONCLUSION: In summary, the methods presented here enable rich assessment of clinical skill since the functional (i.e., dynamic) nature of the data is duly considered.

Relating process and outcome metrics for meaningful and interpretable cannulation skill assessment: A machine learning paradigm.

BACKGROUND AND OBJECTIVES: The quality of healthcare delivery depends directly on the skills of clinicians. For patients on hemodialysis, medical errors or injuries caused during cannulation can lead to adverse outcomes, including potential death. To promote objective skill assessment and effective training, we present a machine learning approach, which utilizes a highly-sensorized cannulation simulator and a set of objective process and outcome metrics. METHODS: In this study, 52 clinicians were recruited to perform a set of pre-defined cannulation tasks on the simulator. Based on data collected by sensors during their task performance, the feature space was then constructed based on force, motion, and infrared sensor data. Following this, three machine learning models- support vector machine (SVM), support vector regression (SVR), and elastic net (EN)- were constructed to relate the feature space to objective outcome metrics. Our models utilize classification based on the conventional skill classification labels as well as a new method that represents skill on a continuum. RESULTS: With less than 5% of trials misplaced by two classes, the SVM model was effective in predicting skill based on the feature space. In addition, the SVR model effectively places both skill and outcome on a fine-grained continuum (versus discrete divisions) that is representative of reality. As importantly, the elastic net model enabled the identification of a set of process metrics that highly impact outcomes of the cannulation task, including smoothness of motion, needle angles, and pinch forces. CONCLUSIONS: The proposed cannulation simulator, paired with machine learning assessment, demonstrates definite advantages over current cannulation training practices. The methods presented here can be adopted to drastically increase the effectiveness of skill assessment and training, thereby potentially improving clinical outcomes of hemodialysis treatment.

Simulator-Based Metrics for Quantifying Vascular Palpation Skill for Cannulation.

Palpation is essential for accurate diagnosis and treatment in many clinical examinations and procedures. Specifically, vascular palpation is used to diagnose cardiovascular health issues and identify anatomical landmarks in the peripheral vascular system. However, little attention has been given to quantifying what comprises skilled vascular palpation; therefore, this study aims to objectively quantify the differences between high performer (HP), mid performer (MP), and low performer (LP) behavior towards understanding vascular palpation skills. Eleven HPs, twenty-five MPs, and ten LPs completed sixteen trials on our simulator under various conditions. There were four fistulas, two skin thicknesses, and two motor vibration intensities. Finger force and location data were recorded for each trial on the simulator. We examined three types of palpation metrics: time, force, and location. All three types of metrics demonstrated statistically significant differences between HP and LP palpation behavior. Therefore, these metrics could be used for structured and standardized palpation skills training in the future, potentially improving patient outcomes.

Measuring Cannulation Skills for Hemodialysis: Objective Versus Subjective Assessment.

Lack of cannulation skill during hemodialysis treatments results in poor clinical outcomes due to infiltration and other cannulation-related trauma. Unfortunately, training of patient care technicians and nurses, specifically on the "technical" aspects of cannulation, has traditionally not received much attention. Simulators have been successfully deployed in many medical specialties for assessment and training of clinical skills. However, simulators have not been as widely used in nursing, especially in the context of training clinical personnel in the dialysis unit. We designed a state-of-the-art simulator for quantifying skill for hemodialysis cannulation. In this study, 52 nurses and patient care technicians with varying levels of clinical experience performed 16 cannulations on the simulator with different fistula properties. We formulated a composite metric for objectively measuring overall success of cannulation and compared this metric with subjective assessment by experts. In addition, we examined if years of clinical experience correlated with objective and subjective scores for cannulation skill. Results indicated that, while subjective and objective metrics generally correlated with each other, the objective metric was more precise and better suited for quantifying cannulation skill. Further, the simulator-based objective metric provides several advantages over subjective ratings, including providing fine-grained assessment of skill, consistency in measurement unaffected by subjective biases, and basing assessment on a more complete evaluation of performance. Years of clinical experience, however, demonstrated little correlation with either method of skill assessment. The methods presented for cannulation skill assessment in this study, if widely applied, could result in improved cannulation skill among our PCTs and nurses, which could positively impact patient outcomes in a tangible way.

Motion Smoothness Metrics for Cannulation Skill Assessment: What Factors Matter?

Medical training simulators have the potential to provide remote and automated assessment of skill vital for medical training. Consequently, there is a need to develop "smart" training devices with robust metrics that can quantify clinical skills for effective training and self-assessment. Recently, metrics that quantify motion smoothness such as log dimensionless jerk (LDLJ) and spectral arc length (SPARC) are increasingly being applied in medical simulators. However, two key questions remain about the efficacy of such metrics: how do these metrics relate to clinical skill, and how to best compute these metrics from sensor data and relate them with similar metrics? This study addresses these questions in the context of hemodialysis cannulation by enrolling 52 clinicians who performed cannulation in a simulated arteriovenous (AV) fistula. For clinical skill, results demonstrate that the objective outcome metric flash ratio (FR), developed to measure the quality of task completion, outperformed traditional skill indicator metrics (years of experience and global rating sheet scores). For computing motion smoothness metrics for skill assessment, we observed that the lowest amount of smoothing could result in unreliable metrics. Furthermore, the relative efficacy of motion smoothness metrics when compared with other process metrics in correlating with skill was similar for FR, the most accurate measure of skill. These results provide guidance for the computation and use of motion-based metrics for clinical skill assessment, including utilizing objective outcome metrics as ideal measures for quantifying skill.

Is Experience in Hemodialysis Cannulation Related to Expertise? A Metrics-based Investigation for Skills Assessment.

Cannulation is not only one of the most common medical procedures but also fraught with complications. The skill of the clinician performing cannulation directly impacts cannulation outcomes. However, current methods of teaching this skill are deficient, relying on subjective demonstrations and unrealistic manikins that have limited utility for skills training. Furthermore, of the factors that hinders effective continuing medical education is the assumption that clinical experience results in expertise. In this work, we examine if objective metrics acquired from a novel cannulation simulator are able to distinguish between experienced clinicians and established experts, enabling the measurement of true expertise. Twenty-two healthcare professionals, who practiced cannulation with varying experience, performed a simulated arteriovenous fistula cannulation task on the simulator. Four clinicians were peer-identified as experts while the others were designated to the experienced group. The simulator tracked the motion of the needle (via an electromagnetic sensor), rendered blood flashback function (via an infrared light sensor), and recorded pinch forces exerted on the needle (via force sensing elements). Metrics were computed based on motion, force, and other sensor data. Results indicated that, with near 80% of accuracy using both logistic regression and linear discriminant analysis, the objective metrics differentiated between experts and the experienced, including identifying needle motion and finger force as two prominent features that distinguished between the groups. Furthermore, results indicated that expertise was not correlated with years of experience, validating the central hypothesis of the study. These insights contribute to structured and standardized medical skills training by enabling a meaningful definition of expertise and could potentially lead to more effective skills training methods.

A Method for Segmenting the Process of Needle Insertion during Simulated Cannulation using Sensor Data.

Cannulation is a routine yet challenging medical procedure resulting in a direct impact on patient outcomes. While current training programs provide guidelines to learn this complex procedure, the lack of objective and quantitative feedback impedes learning this skill more effectively. In this paper, we present a simulator for performing hemodialysis cannulation that captures the process using multiple sensing modalities that provide a multi-faceted assessment of cannulation. Further, we describe an algorithm towards segmenting the cannulation process using specific events in the sensor data for detailed analysis. Results from three participants with varying levels of clinical cannulation expertise are presented along with a metric that successfully differentiates the three participants. This work could lead to sensor-based cannulation skill assessment and training in the future potentially resulting in improved patient outcomes.

Examining the Effect of Haptic Factors for Vascular Palpation Skill Assessment Using an Affordable Simulator.

GOAL: Simulators that incorporate haptic feedback for clinical skills training are increasingly used in medical education. This study addresses the neglected aspect of rendering simulated feedback for vascular palpation skills training by systematically examining the effect of haptic factors on performance. METHODS: A simulator-based approach to examine palpation skill is presented. Novice participants with and without minimal previous palpation training performed a palpation task on a simulator that rendered controlled vibratory feedback under various conditions. RESULTS: Five objective metrics were employed to analyze participants' performance that yielded key findings in quantifying palpation performance. Participants' palpation accuracy was influenced by all three haptic factors, ranging from moderate to statistically significant. Duration, Total Path Length and Ration of Correct Movement also demonstrated utility for quantifying performance. CONCLUSIONS: We demonstrate that our affordable simulator is capable of rendering controlled haptic feedback suitable for skills training. Further, metrics presented in this study can be used for structured palpation skills assessment and training, potentially improving healthcare delivery.

Extracting Subtask-specific Metrics Toward Objective Assessment of Needle Insertion Skill for Hemodialysis Cannulation.

About 80% of all in-hospital patients require vascular access cannulation for treatments. However, there is a high rate of failure for vascular access cannulation, with several studies estimating up to a 50% failure rate for these procedures. Hemodialysis cannulation (HDC) is arguably one of the most difficult of these procedures with a steep learning curve and an extremely high failure rate. In light of this, there is a critical need that clinicians performing HDC have requisite skills. In this work, we present a method that combines the strengths of simulator-based objective skill quantification and task segmentation for needle insertion skill assessment at the subtask level. The results from our experimental study with seven novice nursing students on the cannulation simulator demonstrate that the simulator was able to segment needle insertion into subtask phases. In addition, most metrics were significantly different between the two phases, indicating that there may be value in evaluating participants' behavior at the subtask level. Further, the outcome metric (risk of infiltrating the simulated blood vessel) was successfully predicted by the process metrics in both phases. The implications of these results for skill assessment and training are discussed, which could potentially lead to improved patient outcomes if more extensive validation is pursued.