Tracking and mapping in medical computer vision: A review.

As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

The current state of autonomous suturing: a systematic review.

BACKGROUND: Robotic technology is an important tool in surgical innovation, with robots increasingly being used in the clinical setting. Robots can be used to enhance accuracy, perform remote actions, or to automate tasks. One such surgical task is suturing, a repetitive, fundamental component of surgery that can be tedious and time consuming. Suturing is a promising automation target because of its ubiquity, repetitive nature, and defined constraints. This systematic review examines research to date on autonomous suturing. METHODS: A systematic review of the literature focused on autonomous suturing was conducted in accordance with PRISMA guidelines. RESULTS: 6850 articles were identified by searching PubMed, Embase, Compendex, and Inspec. Duplicates and non-English articles were removed. 4389 articles were screened and 4305 were excluded. Of the 84 remaining, 43 articles did not meet criteria, leaving 41 articles for final review. Among these, 34 (81%) were published after 2014. 31 (76%) were published in an engineering journal9 in a robotics journal, and 1 in a medical journal. The great majority of articles (33, 80%) did not have a specific clinical specialty focus, whereas 6 (15%) were focused on applications in MIS/laparoscopic surgery and 2 (5%) on applications in ophthalmology. Several suturing subtasks were identified, including knot tying, suture passing/needle insertion, needle passing, needle and suture grasping, needle tracking/kinesthesia, suture thread detection, suture needle shape production, instrument assignment, and suture accuracy. 14 articles were considered multi-component because they referred to several previously mentioned subtasks. CONCLUSION: In this systematic review exploring research to date on autonomous suturing, 41 articles demonstrated significant progress in robotic suturing. This summary revealed significant heterogeneity of work, with authors focused on different aspects of suturing and a multitude of engineering problems. The review demonstrates increasing academic and commercial interest in surgical automation, with significant technological advances toward feasibility.

A case series of appendicitis and pseudo-appendicitis in a paediatric intensive care unit.

INTRODUCTION: Appendicitis is a common childhood condition that can be diagnostically challenging. Severe cases may necessitate support in the critical or intensive care unit. These "critical appendicitis diagnoses" have rarely been described. CASE DESCRIPTION: We retrospective reviewed the PICU database of the Hong Kong Children's Hospital and identified cases of suspected and confirmed appendicitis. Clinical features, radiologic findings and final diagnosis of each case were summarized and reported in this case series. We review six anonymized cases of appendicitis managed in a paediatric intensive care unit (PICU) to illustrate the different age spectrum and clinical manifestations of the condition. Rupture of the inflamed appendix, peritonitis and pancreatitis were some of the complications encountered. Crohn disease was found in one case as an underlying diagnosis. Also, one girl clinically diagnosed with appendicitis was found to be a case of ruptured hepatoblastoma with no appendicitis (i.e., pseudoappendicitis). CONCLUSION: Prompt diagnosis, surgical removal of the inflamed appendix, and use of appropriate antimicrobials when indicated are essential in reducing mortality and morbidity associated with severe appendicitis. Significant premorbid conditions such as acute myeloid leukemia, mitochondrial encephalopathy lactic acidosis syndrome (MELAS), inflammatory bowel disease and complications may be present in patients needing intensive care as is illustrated in the present cases. Pseudoappendicitis is an important differential diagnosis. Imaging is crucial and useful in establishing and confirming the diagnosis of appendicitis and pseudo-appendicitis in these PICU cases.

Artificial intelligence meets medical robotics.

Artificial intelligence (AI) applications in medical robots are bringing a new era to medicine. Advanced medical robots can perform diagnostic and surgical procedures, aid rehabilitation, and provide symbiotic prosthetics to replace limbs. The technology used in these devices, including computer vision, medical image analysis, haptics, navigation, precise manipulation, and machine learning (ML) , could allow autonomous robots to carry out diagnostic imaging, remote surgery, surgical subtasks, or even entire surgical procedures. Moreover, AI in rehabilitation devices and advanced prosthetics can provide individualized support, as well as improved functionality and mobility (see the figure). The combination of extraordinary advances in robotics, medicine, materials science, and computing could bring safer, more efficient, and more widely available patient care in the future. -Gemma K. Alderton.

ORRN: An ODE-Based Recursive Registration Network for Deformable Respiratory Motion Estimation With Lung 4DCT Images.

OBJECTIVE: Deformable Image Registration (DIR) plays a significant role in quantifying deformation in medical data. Recent Deep Learning methods have shown promising accuracy and speedup for registering a pair of medical images. However, in 4D (3D + time) medical data, organ motion, such as respiratory motion and heart beating, can not be effectively modeled by pair-wise methods as they were optimized for image pairs but did not consider the organ motion patterns necessary when considering 4D data. METHODS: This article presents ORRN, an Ordinary Differential Equations (ODE)-based recursive image registration network. Our network learns to estimate time-varying voxel velocities for an ODE that models deformation in 4D image data. It adopts a recursive registration strategy to progressively estimate a deformation field through ODE integration of voxel velocities. RESULTS: We evaluate the proposed method on two publicly available lung 4DCT datasets, DIRLab and CREATIS, for two tasks: 1) registering all images to the extreme inhale image for 3D+t deformation tracking and 2) registering extreme exhale to inhale phase images. Our method outperforms other learning-based methods in both tasks, producing the smallest Target Registration Error of 1.24 mm and 1.26 mm, respectively. Additionally, it produces less than 0.001% unrealistic image folding, and the computation speed is less than 1 s for each CT volume. CONCLUSION: ORRN demonstrates promising registration accuracy, deformation plausibility, and computation efficiency on group-wise and pair-wise registration tasks. SIGNIFICANCE: It has significant implications in enabling fast and accurate respiratory motion estimation for treatment planning in radiation therapy or robot motion planning in thoracic needle insertion.

Mixed reality surgical mentoring of combat casualty care related procedures in a perfused cadaver model: Initial results of a randomized feasibility study.

BACKGROUND: Most telemedicine modalities have limited ability to enhance procedural and operative care. We developed a novel system to provide synchronous bidirectional expert mixed reality-enabled virtual procedural mentoring. In this feasibility study, we evaluated mixed reality mentoring of combat casualty care related procedures in a re-perfused cadaver model. METHODS: Novices received real-time holographic mentoring from experts using augmented reality via Hololens (Microsoft Inc, Redmond, WA). The experts maintained real-time awareness of the novice's operative environment using virtual reality via HTC-Vive (HTC Corp, Xindian District, Taiwan). Additional cameras (both environments) and novel software created the immersive, shared, 3-dimensional mixed reality environment in which the novice and expert collaborated. The novices were prospectively randomized to either mixed reality or audio-only mentoring. Blinded experts independently evaluated novice procedural videos using a 5-point Likert scale-based questionnaire. Nonparametric variables were evaluated using the Wilcoxon rank-sum test and comparisons using the χ2 analysis; significance was defined at P < .05. RESULTS: Surgeon and nonsurgeon novices (14) performed 69 combat casualty care-related procedures (38 mixed reality, 31 audio), including various vascular exposures, 4-compartment lower leg fasciotomy, and emergency neurosurgical procedures; 85% were performed correctly with no difference in either group. Upon video review, mixed reality-mentored novices showed no difference in procedural flow and forward planning (3.67 vs 3.28, P = .21) or the likelihood of performing individual procedural steps correctly (4.12 vs 3.59, P = .06). CONCLUSION: In this initial feasibility study, our novel mixed reality-based mentoring system successfully facilitated the performance of a wide variety of combat casualty care relevant procedures using a high fidelity re-perfused cadaver model. The small sample size and limited variety of novice types likely impacted the ability of holographically mentored novices to demonstrate improvement over the audio-only control group. Despite this, using virtual, augmented, and mixed reality technologies for procedural mentoring demonstrated promise, and further study is needed.

Remote telesurgery in humans: a systematic review.

BACKGROUND: Since the conception of robotic surgery, remote telesurgery has been a dream upon which incredible technological advances haven been built. Despite the considerable enthusiasm for, there have been few published studies of remote telesurgery on humans. METHODS: We performed a systematic review of the English literature (PubMed, EMbase, Inspec & Compendex and Web of Science) to report studies of remote telesurgery in humans. Keywords included telesurgery, remote surgery, long-distance surgery, and telerobotics. Subjects had to be human (live patients or cadavers). The operating surgeon had to be remote from the patient, separated by more than one kilometer. The article had to explicitly report the use of a long-distance telerobotic technique. Articles that focused on telepresence or tele-mentoring were excluded. RESULTS: The study included eight articles published from 2001 to 2020. One manuscript (1 subject) described remote surgery on a cadaver model, and the other seven were on live humans (72 subjects). Procedure types included percutaneous, endovascular, laparoscopic, and transoral. Communication methods varied, with the first report using a telephone line and the most recent studies using a 5G network. Six of the studies reported signal latency as a single value and it ranged from 28 ms to 280 ms. CONCLUSIONS: Few studies have described remote telesurgery in humans, and there is considerable variability in robotic and communication methods. Future efforts should work to improve reporting of signal latency and follow careful research methodology.

Compensatory motion scaling for time-delayed robotic surgery.

BACKGROUND: Round trip signal latency, or time delay, is an unavoidable constraint that currently stands as a major barrier to safe and efficient remote telesurgery. While there have been significant technological advancements aimed at reducing the time delay, studies evaluating methods of mitigating the negative effects of time delay are needed. Herein, we explored instrument motion scaling as a method to improve performance in time-delayed robotic surgery. METHODS: This was a robotic surgery user study using the da Vinci Research Kit system. A ring transfer task was performed under normal circumstances (no added time delay), and with 250 ms, 500 ms, and 750 ms delay. Robotic instrument motion scaling was modulated across a range of values (- 0.15, - 0.1, 0, + 0.1, + 0.15), with negative values indicating less instrument displacement for a given amount of operator movement. The primary outcomes were task completion time and total errors. Three-dimensional instrument movement was compared against different motion scales using dynamic time warping to demonstrate the effects of scaling. RESULTS: Performance declined with increasing time delay. Statistically significant increases in task time and number of errors were seen at 500 ms and 750 ms delay (p < 0.05). Total errors were positively correlated with task time on linear regression (R = 0.79, p < 0.001). Under 750 ms delay, negative instrument motion scaling improved error rates. Negative motion scaling trended toward improving task times toward those seen in non-delayed scenarios. Improvements in instrument path motion were seen with the implementation of negative motion scaling. CONCLUSIONS: Under time-delayed conditions, negative robotic instrument motion scaling yielded fewer surgical errors with slight improvement in task time. Motion scaling is a promising method of improving the safety and efficiency of time-delayed robotic surgery and warrants further investigation.

Clinical and radiographic outcomes with a posteriorly augmented glenoid for Walch B2, B3, and C glenoids in reverse total shoulder arthroplasty.

OBJECTIVE: The purpose of this study is to quantify the clinical and radiographic outcomes of patients with severe posterior glenoid wear who were treated with reverse total shoulder arthroplasty (rTSA) and a posterior augmented baseplate. METHODS: A total of 67 primary rTSA patients with osteoarthritis and posterior glenoid wear were treated with an 8° posterior augmented glenoid baseplate. All patients had a Walch B2, B3, or C glenoid, 2-year minimum follow-up, and mean follow-up of 40 months. All patients were scored preoperatively and at the latest follow-up using 5 clinical outcome metrics; active range of motion was also measured. A Student's 2-tailed, unpaired t-test quantified differences in outcomes, where P < .05 denoted significance. RESULTS: All patients experienced significant improvements in pain and function after primary rTSA with a posterior augmented glenoid baseplate. Three complications were reported for a rate of 4.5%; no cases of aseptic glenoid loosening occurred. A total of 90% of patients exceeded the minimal clinically important difference threshold, and 80% of patients exceeded the substantial clinical benefit threshold for each clinical outcome metric and range of motion measure. No differences in outcomes or complications were observed between Walch B2 and B3 patients, demonstrating that this full-wedge posterior augmented baseplate was equally good in each type of glenoid deformity. DISCUSSION: Primary rTSA patients with Walch B2, B3, and C glenoids who received an 8° posterior augmented glenoid baseplate experienced excellent clinical and radiographic outcomes with a low complication rate and no reports of aseptic glenoid loosening at a mean follow-up of 40 months.

Processing Homophones Interactively: Evidence from eye-movement data.

The question of how to process an ambiguous word in context has been long-studied in psycholinguistics and the present study examined this question further by investigating the spoken word recognition processes of Cantonese homophones (a common type of ambiguous word) in context. Sixty native Cantonese listeners were recruited to participate in an eye-tracking experiment. Listeners were instructed to listen carefully to a sentence ending with a Cantonese homophone and then look at different visual probes (either Chinese characters or line-drawing pictures) presented on the computer screen simultaneously. Two findings were observed. First, the results revealed that sentence context exerted an early effect on homophone processes. Second, visual probes that serve as phonological competitors only had a weak effect on the spoken word recognition processes. Consistent with previous studies, the patterns of eye-movement results appeared to support an interactive processing approach in homophone recognition.

Hemorrhagic Pericardial Cyst Diagnosis Accelerated by Emergency Physician Echocardiography: A Case Report.

BACKGROUND: The differential diagnosis for chest pain in the emergency department is broad and includes both benign and life-threatening conditions-with pericardial cyst as a rare example. Emergency physician-performed point-of-care focused cardiac ultrasound (FOCUS) is increasingly recognized as a useful modality in the evaluation of patients with chest pain. CASE REPORT: We report a case of hemorrhagic pericardial cyst in a young woman presenting with chest pain in which findings on FOCUS contradicted findings on chest x-ray study and thus, accelerated diagnosis and definitive treatment. We also comment on epidemiology, pathophysiology, clinical presentation, diagnosis, and management of this uncommon, potentially fatal cause of chest pain. WHY SHOULD AN EMERGENCY PHYSICIAN BE AWARE OF THIS?: This case report aims to bring an uncommon case to the attention of emergency providers and emphasize the importance of facility with FOCUS. Although definitive diagnosis and management were not accomplished at the bedside in this case, an abnormal finding on FOCUS prompted further investigation and timely treatment.

Routine Screening Ultrasound in Children With Cleft Palate and/or Lip: A Single Center Experience.

The aim of this study is to evaluate the incidence of associated congenital anomalies that is picked up by screening ultrasound of the heart, urinary system, and central nervous system (CNS) of children with cleft palate and/or lip deformities in a single regional center. A total of 225 children with cleft deformities were recruited in the study in 2009 to 2013, 91% of them underwent screening echocardiogram and of these, 98% of them were normal findings that do not require further follow-up ultrasound or treatment. Total 85% of the children underwent screening ultrasound of the urinary system and CNS, and of these, 99% were reported to have normal findings. Medium term follow-up of these children did not reveal any significant structural anomalies in the heart, brain, and urinary system that might have missed in the initial ultrasound screening. The authors suggest that in resource-limited settings, routine ultrasound screening of the heart, the CNS, and the urinary system may not be required in asymptomatic and nonsyndromic infants with cleft deformities and in those with "lip-only" cleft deformity.

A pilot study: application of hemoglobin and cortisol levels, and a memory test to evaluate the quality of life of breast cancer patients on chemotherapy.

BACKGROUND: Short-term memory (STM) decline in breast cancer patients resulting from chemotherapy was evaluated by means of blood biomarkers, a questionnaire, and a computerized STM test. METHODS: This study was conducted from January 2013 to June 2013, recruiting 90 subjects: 30 breast cancer patients beginning the 3rd of 4th cycles of docetaxel and cyclophosphamide chemotherapy, 30 recovered patients (who completed 4 cycles of docetaxel for a minimum of 6 months), and 30 healthy subjects (disease-free females). The levels of hemoglobin, red and white blood cells, and cortisol in serum, and a computerized STM test were analyzed to estimate the effects of chemotherapy on STM. A questionnaire was given to all subjects to assess quality of life. RESULTS: Statistically significant differences were observed for the blood parameters (hemoglobin, red and white blood cells, and cortisol levels) between healthy and on-treatment subjects (respectively 13.47 ± 0.96 g/dL vs 5.37 ± 0.38 g/dL, 4.58 ± 0.41 10(12)/L vs 2.07 ± 0.13 10(12)/L, and 6.15 ± 1.03 10(9)/L vs 0.86 ± 0.41 10(9)/L). Scores of the STM test were significantly lower for patients compared to healthy subjects. As indicated by the results of the questionnaire, breast cancer patients had a higher tendency to forget than healthy controls (X(2)=3.15; p<0.0001) and recovered subjects (X(2)=3.15; p<0.0001). CONCLUSION: We found depleted levels of hemoglobin, red and white blood cells as a result of chemotherapy, and elevated levels of stress correlated with poor performances in the computerized STM test. A higher cortisol level might be an important precursor of STM deterioration. Monitoring cortisol would be beneficial for evaluating the quality of life of breast cancer patients on chemotherapy.

Status of oral ulcerative mucositis and biomarkers to monitor posttraumatic stress disorder effects in breast cancer patients.

BACKGROUND: This study was designed to assess oral ulcerative mucositis, C-reactive protein, blood pressure, heart rate and thyroid function in breast cancer patients in relation to the occurrence of posttraumatic stress disorder (PTSD). METHODS: A total of 120 female breast cancer patients and women 100 healthy subjects were enrolled in this study. PTSD status was assessed by questionnaire. Before and after treatment (modified radical mastectomy and chemotherapy), serum samples were collected and measured for levels of triiodothyronine (T3), thyroxine (T4), thyroid stimulating hormone (TSH) and high-sensitivity C-reactive protein (hs-CRP) by ELISA. Oral ulcerative mucositis was evaluated by the number and duration of oral ulcers and the degree of pain. RESULTS: Breast cancer patients experienced long-term PTSD and had elevated serum T3 and T4 levels. Patients experienced more severe pain and longer duration of oral ulcers compared with the healthy group. Oral ulcers were significantly associated with PTSD score in terms of the number of ulcers (p=0.0025), the degree of pain (p<0.0001) and the duration of ulcers (p<0.0001). CONCLUSION: These findings support that thyroid function is altered in breast cancer patients with PTSD. Elevation of T3 and T4 and oral ulcerative mucositis might be indicative of the emotional status of breast cancer patients.

Registration of 3D ultrasound through an air-tissue boundary.

In this study we evaluated a new method for registering three-dimensional ultrasound (3DUS) data to external coordinate systems. First, 3DUS was registered to the stereo endoscope of a da Vinci Surgical System by placing a registration tool against an air-tissue boundary so that the 3DUS could image ultrasound fiducials while the stereo endoscope could image camera markers on the same tool. The common points were used to solve the registration between the 3DUS and camera coordinate systems. The target registration error (TRE) when imaging through a PVC tissue phantom ranged from 3.85 1.76 mm to 1.82 1.03 mm using one to four registration tool positions. TRE when imaging through an ex-vivo liver tissue sample ranged from 2.36 1.01 mm to 1.51 0.70 mm using one to four registration tool positions. Second, using a similar method, 3DUS was registered to the kinematic coordinate system of a da Vinci Surgical System by using the da Vinci surgical manipulators to identify common points on an air-tissue boundary. TRE when imaging through a PVC tissue phantom was 0.95 0.38 mm. This registration method is simpler and potentially more accurate than methods using commercial motion tracking systems. This method may be useful in the future in augmented reality systems for laparoscopic and robotic-assisted surgery.

Tissue tracking and registration for image-guided surgery.

Vision-based tracking of tissue is a key component to enable augmented reality during a surgical operation. Conven- tional tracking techniques in computer vision rely on identifying strong edge features or distinctive textures in a well-lit environ- ment; however endoscopic tissue images do not have strong edge features, are poorly lit and exhibit a high degree of specular reflection. Therefore, prior work in achieving densely populated 3D features for describing tissue surface profiles require complex image processing techniques and have been limited in providing stable, long-term tracking or real-time processing. In this paper, we present an integrated framework for ac- curately tracking tissue in surgical stereo-cameras at real-time speeds. We use a combination of the STAR feature detector and Binary Robust Independent Elementary Features to acquire salient features that can be persistently tracked at high frame rates. The features are then used to acquire a densely-populated map of the deformations of tissue surface in 3D. We evaluate the method against popular feature algorithms in in-vivo animal study video sequences, and we also apply the proposed method to human partial nephrectomy video sequences. We extend the salient feature framework to support region tracking in order to maintain the spatial correspondence of a tracked region of tissue or a medical image registration to the surrounding tissue. In-vitro tissue studies show registration accuracies of 1.3-3.3 mm using a rigid-body transformation method.

A robust uniaxial force sensor for minimally invasive surgery.

This paper presents a novel miniature uniaxial force sensor for use within a beating heart during mitral valve annuloplasty. The sensor measures 5.5 mm in diameter and 12 mm in length and provides a hollow core to pass instrumentation. A soft elastomer flexure design maintains a waterproof seal. Fiber optic transduction eliminates electrical circuitry within the heart, and acetal components minimize ultrasound-imaging artifacts. Calibration uses a nonlinear viscoelastic method, and in vitro tests demonstrate a 0-4-N force range with rms errors of 0.13 N (< 3.2%). In vivo tests provide the first endocardial measurements of tissue-minimally invasive surgery instrument interaction forces in a beating heart.

Indirect low-intensity ultrasonic stimulation for tissue engineering.

Low-intensity ultrasound (LIUS) treatment has been shown to increase mass transport, which could benefit tissue grafts during the immediate postimplant period, when blood supply to the implanted tissue is suboptimal. In this in vitro study, we investigated effects of LIUS stimulation on dye diffusion, proliferation, metabolism, and tropomyosin expression of muscle cells (C2C12) and on tissue viability and gene expression of human adipose tissue organoids. We found that LIUS increased dye diffusion within adjacent tissue culture wells and caused anisotropic diffusion patterns. This effect was confirmed by a hydrophone measurement resulting in acoustic pressure 150-341 Pa in wells. Cellular studies showed that LIUS significantly increased proliferation, metabolic activity, and expression of tropomyosin. Adipose tissue treated with LIUS showed significantly increased metabolic activity and the cells had similar morphology to normal unilocular adipocytes. Gene analysis showed that tumor necrosis factor-alpha expression (a marker for tissue damage) was significantly lower for stimulated organoids than for control groups. Our data suggests that LIUS could be a useful modality for improving graft survival in vivo.

Robotic force stabilization for beating heart intracardiac surgery.

The manipulation of fast moving, delicate tissues in beating heart procedures presents a considerable challenge to surgeons. We present a new robotic force stabilization system that assists surgeons by maintaining a constant contact force with the beating heart. The system incorporates a novel, miniature uniaxial force sensor that is mounted to surgical instrumentation to measure contact forces during surgical manipulation. Using this sensor in conjunction with real-time tissue motion information derived from 3D ultrasound, we show that a force controller with feed-forward motion terms can provide safe and accurate force stabilization in an in vivo contact task against the beating mitral valve annulus. This confers a 50% reduction in force fluctuations when compared to a standard force controller and a 75% reduction in fluctuations when compared to manual attempts to maintain the same force.