Confidence-aware self-supervised learning for dense monocular depth estimation in dynamic laparoscopic scene.

This paper tackles the challenge of accurate depth estimation from monocular laparoscopic images in dynamic surgical environments. The lack of reliable ground truth due to inconsistencies within these images makes this a complex task. Further complicating the learning process is the presence of noise elements like bleeding and smoke. We propose a model learning framework that uses a generic laparoscopic surgery video dataset for training, aimed at achieving precise monocular depth estimation in dynamic surgical settings. The architecture employs binocular disparity confidence information as a self-supervisory signal, along with the disparity information from a stereo laparoscope. Our method ensures robust learning amidst outliers, influenced by tissue deformation, smoke, and surgical instruments, by utilizing a unique loss function. This function adjusts the selection and weighting of depth data for learning based on their given confidence. We trained the model using the Hamlyn Dataset and verified it with Hamlyn Dataset test data and a static dataset. The results show exceptional generalization performance and efficacy for various scene dynamics, laparoscope types, and surgical sites.

Loss Weightings for Improving Imbalanced Brain Structure Segmentation Using Fully Convolutional Networks.

Brain structure segmentation on magnetic resonance (MR) images is important for various clinical applications. It has been automatically performed by using fully convolutional networks. However, it suffers from the class imbalance problem. To address this problem, we investigated how loss weighting strategies work for brain structure segmentation tasks with different class imbalance situations on MR images. In this study, we adopted segmentation tasks of the cerebrum, cerebellum, brainstem, and blood vessels from MR cisternography and angiography images as the target segmentation tasks. We used a U-net architecture with cross-entropy and Dice loss functions as a baseline and evaluated the effect of the following loss weighting strategies: inverse frequency weighting, median inverse frequency weighting, focal weighting, distance map-based weighting, and distance penalty term-based weighting. In the experiments, the Dice loss function with focal weighting showed the best performance and had a high average Dice score of 92.8% in the binary-class segmentation tasks, while the cross-entropy loss functions with distance map-based weighting achieved the Dice score of up to 93.1% in the multi-class segmentation tasks. The results suggested that the distance map-based and the focal weightings could boost the performance of cross-entropy and Dice loss functions in class imbalanced segmentation tasks, respectively.

Complex-Valued Wavelet Spectrum Analysis of Respiratory Conditions and Its Feasibility in the Detection of Low-Functional Respiration.

Respiratory monitoring is a significant issue to reduce patient risks and medical staff labor in postoperative care and epidemic infection, particularly after the COVID-19 pandemic. Oximetry is widely used for respiration monitoring in the clinic, but it sometimes fails to capture a low-functional respiratory condition even though a patient has breathing difficulty. Another approach is breathing-sound monitoring, but this is unstable due to the indirect measurement of lung volume. Kobayashi in our team is developing a sensor measuring temporal changes in lung volume with a displacement sensor attached across the sixth and eighth ribs. For processing these respiratory signals, we propose the combination of complex-valued wavelet transform and the correlation among spectrum sequences. We present the processing results and discuss its feasibility to detect a low-functional condition in respiration. The result for detecting low-functional respiration showed good performance with a sensitivity of 0.88 and specificity of 0.88 to 1 in its receiver operating characteristic (ROC) curve.

A Small and High-Speed Driving Mechanism for 3D Shape Measurement in Monocular Endoscopy.

Three-dimensional (3D) shape acquisition has been widely introduced to enrich quantitative analysis with the combination of object shape and texture, for example, surface roughness evaluation in industry and gastrointestinal endoscopy in medicine. Shape from focus is a promising technique to measure substance surfaces in 3D space because no occlusion problem appears in principle, as does with stereo shape measurement, which is another commonly used option. We have been developing endoscopic shape measurement devices and shape reconstruction algorithms. In this paper, we propose a mechanism for driving an image sensor reciprocated for the shape from focus of 3D shape measurement in monocular endoscopy. It uses a stepping motor and a planar-end cam, which transforms the motor rotation to imaging sensor reciprocation, to implement the shape from focus of 3D shape measurement in endoscopy. We test and discuss the device in terms of its driving accuracy and application feasibility for endoscopic 3D shape measurement.

Computational reproductions of external force field adaption without assuming desired trajectories.

Optimal feedback control is an established framework that is used to characterize human movement. However, it is not fully understood how the brain computes optimal gains through interactions with the environment. In the past study, we proposed a model of motor learning that identifies a set of feedback and feedforward controllers and a state predictor of the arm musculoskeletal system to control free reaching movements. In this study, we applied the model to force field adaptation tasks where normal reaching movements are disturbed by an external force imposed on the hand. Without a priori knowledge about the arm and environment, the model was able to adapt to the force field by generating counteracting forces to overcome it in a manner similar to what is reported in the behavioral literature. The kinematics of the movements generated by our model share characteristic features of human movements observed before and after force field adaptation. In addition, we demonstrate that the structure and learning algorithm introduced in our model induced a shift in the end-point's equilibrium position and a static force modulation, accompanied by a fast and a slow learning process. Importantly, our model does not require desired trajectories, yields movements without specifying movement duration, and predicts force generation patterns by exploring the environment. Our model demonstrates a possible mechanism through which the central nervous system may control and adapt a point-to-point reaching movement without specifying a desired trajectory by continuously updating the body's musculoskeletal model.

Label cleaning and propagation for improved segmentation performance using fully convolutional networks.

PURPOSE: In recent years, fully convolutional networks (FCNs) have been applied to various medical image segmentation tasks. However, it is difficult to generate a large amount of high-quality annotation data to train FCNs for medical image segmentation. Thus, it is desired to achieve high segmentation performances even from incomplete training data. We aim to evaluate performance of FCNs to clean noises and interpolate labels from noisy and sparsely given label images. METHODS: To evaluate the label cleaning and propagation performance of FCNs, we used 2D and 3D FCNs to perform volumetric brain segmentation from magnetic resonance image volumes, based on network training on incomplete training datasets from noisy and sparse annotation. RESULTS: The experimental results using pseudo-incomplete training data showed that both 2D and 3D FCNs could provide improved segmentation results from the incomplete training data, especially by using three orthogonal annotation images for network training. CONCLUSION: This paper presented a validation for label cleaning and propagation based on FCNs. FCNs might have the potential to achieve improved segmentation performances even from sparse annotation data including possible noises by manual annotation, which can be an important clue to more efficient annotation.

Suction-fixing surgical device for assisting liver manipulation with laparoscopic forceps.

PURPOSE: Noninvasiveness and stability are significant issues in laparoscopic liver resection. Inappropriate grasping force can cause damage or serious bleeding to the liver. In addition, instability of grasping can result unsafe operations or wavered cutting. We propose a surgical device to improve stability of liver manipulation. METHODS: A proposed device adheres to the liver surface with suction fixation, then tunes its stiffness to being hard and shapes like as a bulge on the liver surface to be grasped with laparoscopic forceps. It consists of two soft beams, a chamber sponge, membrane covering the device upper, suburb extrusion wing membrane, a vacuuming tube and to-be-grasped bars. The beams are designed as being non-stretchable and easy to bend. The device is connected to a medical vacuuming pump to vacuum air in the device and then gets hard to transfer forceps operation well. This stiffness tuning mechanism by pneumatic control features the device for achieving good liver shape followability and forceps operation propagation less invasively. The proposed device was tested with rubber phantoms and porcine livers on shape followability, stiffness transition, liver invasiveness and operational usability in the experiments. RESULTS: Performance of the proposed device was assessed in experiments. The device showed good object-shape followability. It held the liver with 2.43-N force for vertical lifting and 4.90-N shear force with - 80 kPa vacuuming pressure. Invasiveness was reduced to acceptable level of liver damage. In usability test, the device grasped the liver stably and transferred surgical forceps operations to the liver surface well. CONCLUSION: The proposed device showed effective performance to improve laparoscopic liver manipulation. It held the liver stably and less invasively and transferred forceps operation force to the liver surface well.

When 90% of the variance is not enough: residual EMG from muscle synergy extraction influences task performance.

Muscle synergies are usually identified via dimensionality reduction techniques, such that the identified synergies reconstruct the muscle activity to an accuracy level defined heuristically, often set to 90% of the variance. Here, we question the assumption that the residual muscle activity not explained by the synergies is due to noise. We hypothesize instead that the residual activity is not entirely random and can influence the execution of motor tasks. Young healthy subjects performed an isometric reaching task in which the surface electromyography of 10 arm muscles was mapped onto a two-dimensional force used to control a cursor. Three to five synergies explained 90% of the variance in muscle activity. We altered the muscle-force mapping via "hard" and "easy" virtual surgeries. Whereas in both surgeries the forces associated with synergies spanned the same dimension of the virtual environment, the muscle-force mapping was as close as possible to the initial mapping in the easy surgery; in contrast, it was as far as possible in the hard surgery. This design maximized potential differences in reaching errors attributable to residual activity. Results show that the easy surgery produced smaller directional errors than the hard surgery. Additionally, simulations of surgeries constructed with 1 to 10 synergies show that the errors in the easy and hard surgeries differ significantly for up to 8 synergies, which explains 98% of the variance on average. Our study thus indicates the need for cautious interpretations of results derived from synergy extraction techniques based on heuristics with lenient accuracy levels.NEW & NOTEWORTHY The muscle synergy hypothesis posits that the central nervous system simplifies motor control by grouping muscles into modules. Current techniques use dimensionality reduction, such that the identified synergies reconstruct 90% of the muscle activity. We show that residual muscle activity following such identification can have a large systematic effect on movements, even when the number of synergies approaches the number of muscles. Current synergy extraction techniques must therefore be updated to identify true physiological synergies.

Three-dimensional posture estimation of robot forceps using endoscope with convolutional neural network.

BACKGROUND: In recent years, there has been significant developments in surgical robots. Image-based sensing of surgical instruments, without the use of electric sensors, are preferred for easily washable robots. METHODS: We propose a method to estimate the three-dimensional posture of the tip of the forceps tip by using an endoscopic image. A convolutional neural network (CNN) receives the image of the tracked markers attached to the forceps as an input and outputs the posture of the forceps. RESULTS: The posture estimation results showed that the posture estimated from the image followed the electrical sensor. The estimated results of the external force calculated based on the posture also followed the measured values. CONCLUSION: The method which estimates the forceps posture from the image using CNN is effective. The mean absolute error of the estimated external force is smaller than the human detection limit.

Body ownership and agency altered by an electromyographically controlled robotic arm.

Understanding how we consciously experience our bodies is a fundamental issue in cognitive neuroscience. Two fundamental components of this are the sense of body ownership (the experience of the body as one's own) and the sense of agency (the feeling of control over one's bodily actions). These constructs have been used to investigate the incorporation of prostheses. To date, however, no evidence has been provided showing whether representations of ownership and agency in amputees are altered when operating a robotic prosthesis. Here we investigated a robotic arm using myoelectric control, for which the user varied the joint position continuously, in a rubber hand illusion task. Fifteen able-bodied participants and three trans-radial amputees were instructed to contract their wrist flexors/extensors alternately, and to watch the robotic arm move. The sense of ownership in both groups was extended to the robotic arm when the wrists of the real and robotic arm were flexed/extended synchronously, with the effect being smaller when they moved in opposite directions. Both groups also experienced a sense of agency over the robotic arm. These results suggest that these experimental settings induced successful incorporation of the prosthesis, at least for the amputees who took part in the present study.

Incorporation of prosthetic limbs into the body representation of amputees: Evidence from the crossed hands temporal order illusion.

Understanding how we consciously experience our bodies is a fundamental issue in both psychology and neuroscience. To date, the incorporation of nonbody objects into the body representation has been investigated extensively, and the incorporation of prosthetic arms in amputees has been demonstrated using the rubber hand illusion. In this study, we investigated the incorporation of prosthetic arms in amputees using the crossed hands illusion, in which successive somatosensory stimuli are delivered, one to each arm, at intervals of 300ms or less, and where arm crossing often causes inversion of perceived tactile temporal order. The induced reversal illusion was greater with a prosthetic limb than without in three amputees. With a shorter prosthetic arm (i.e., one that did not reach the contralateral limb), the illusion induced by vision of the short prosthetic arm was significantly reduced as compared to that seen when the long prosthetic arm crossed over the other arm. These results therefore suggest that the somatosensory signals were referred to the spatial location of the tips of the prosthetic arm, which was incorporated into the body representation by the amputees.

Affective Stimuli for an Auditory P300 Brain-Computer Interface.

Gaze-independent brain computer interfaces (BCIs) are a potential communication tool for persons with paralysis. This study applies affective auditory stimuli to investigate their effects using a P300 BCI. Fifteen able-bodied participants operated the P300 BCI, with positive and negative affective sounds (PA: a meowing cat sound, NA: a screaming cat sound). Permuted stimuli of the positive and negative affective sounds (permuted-PA, permuted-NA) were also used for comparison. Electroencephalography data was collected, and offline classification accuracies were compared. We used a visual analog scale (VAS) to measure positive and negative affective feelings in the participants. The mean classification accuracies were 84.7% for PA and 67.3% for permuted-PA, while the VAS scores were 58.5 for PA and -12.1 for permuted-PA. The positive affective stimulus showed significantly higher accuracy and VAS scores than the negative affective stimulus. In contrast, mean classification accuracies were 77.3% for NA and 76.0% for permuted-NA, while the VAS scores were -50.0 for NA and -39.2 for permuted NA, which are not significantly different. We determined that a positive affective stimulus with accompanying positive affective feelings significantly improved BCI accuracy. Additionally, an ALS patient achieved 90% online classification accuracy. These results suggest that affective stimuli may be useful for preparing a practical auditory BCI system for patients with disabilities.

Decoding finger movement in humans using synergy of EEG cortical current signals.

The synchronized activity of neuronal populations across multiple distant brain areas may reflect coordinated interactions of large-scale brain networks. Currently, there is no established method to investigate the temporal transitions between these large-scale networks that would allow, for example, to decode finger movements. Here we applied a matrix factorization method employing principal component and temporal independent component analyses to identify brain activity synchronizations. In accordance with previous studies investigating "muscle synergies", we refer to this activity as "brain activity synergy". Using electroencephalography (EEG), we first estimated cortical current sources (CSs) and then identified brain activity synergies within the estimated CS signals. A decoding analysis for finger movement in eight directions showed that such CS synergies provided more information for dissociating between movements than EEG sensor signals, EEG synergy, or CS signals, suggesting that temporal activation patterns of the synchronizing CSs may contain information related to motor control. A quantitative analysis of features selected by the decoders further revealed temporal transitions among the primary motor area, dorsal and ventral premotor areas, pre-supplementary motor area, and supplementary motor area, which may reflect transitions in motor planning and execution. These results provide a proof of concept for brain activity synergy estimation using CSs.

A hybrid BMI-based exoskeleton for paresis: EMG control for assisting arm movements.

OBJECTIVE: Brain-machine interface (BMI) technologies have succeeded in controlling robotic exoskeletons, enabling some paralyzed people to control their own arms and hands. We have developed an exoskeleton asynchronously controlled by EEG signals. In this study, to enable real-time control of the exoskeleton for paresis, we developed a hybrid system with EEG and EMG signals, and the EMG signals were used to estimate its joint angles. APPROACH: Eleven able-bodied subjects and two patients with upper cervical spinal cord injuries (SCIs) performed hand and arm movements, and the angles of the metacarpophalangeal (MP) joint of the index finger, wrist, and elbow were estimated from EMG signals using a formula that we derived to calculate joint angles from EMG signals, based on a musculoskeletal model. The formula was exploited to control the elbow of the exoskeleton after automatic adjustments. Four able-bodied subjects and a patient with upper cervical SCI wore an exoskeleton controlled using EMG signals and were required to perform hand and arm movements to carry and release a ball. MAIN RESULTS: Estimated angles of the MP joints of index fingers, wrists, and elbows were correlated well with the measured angles in 11 able-bodied subjects (correlation coefficients were 0.81 ± 0.09, 0.85 ± 0.09, and 0.76 ± 0.13, respectively) and the patients (e.g. 0.91 ± 0.01 in the elbow of a patient). Four able-bodied subjects successfully positioned their arms to adequate angles by extending their elbows and a joint of the exoskeleton, with root-mean-square errors <6°. An upper cervical SCI patient, empowered by the exoskeleton, successfully carried a ball to a goal in all 10 trials. SIGNIFICANCE: A BMI-based exoskeleton for paralyzed arms and hands using real-time control was realized by designing a new method to estimate joint angles based on EMG signals, and these may be useful for practical rehabilitation and the support of daily actions.

Feasibility of magnetic resonance elastography for the pancreas at 3T.

PURPOSE: 1) To assess the usefulness of an elastic belt bracing the upper abdomen for reducing the miscalculated areas of the pancreas on 3.0T magnetic resonance elastography (MRE); 2) to test whether MRE can detect difference of stiffness between normal pancreas and the focal pancreatic diseases. MATERIALS AND METHODS: Using an initial eight normal volunteers, miscalculated areas were compared between MRE with the elastic belt and without the belt on 3.0T MRI. Then, using the belt, MRE of the normal pancreas was measured using 14 volunteers and 11 patients with focal pancreatic lesions. RESULTS: The median (95% confidence interval [CI]) percentages of correctly calculated areas were 57.4% (32.9-63.0) with the elastic belt and 35.3% (11.4-60.4) without the belt (P = 0.0078). The stiffness of each pancreatic segment of the normal volunteers (mean ± SE) was 2.37 ± 0.16 kPa for the head, 2.46 ± 0.17 kPa for the body, and 2.58 ± 0.26 kPa for the tail. The stiffness of seven pancreatic cancers was 6.06 ± 0.49 kPa, which was higher than the overall pancreatic stiffness of the normal volunteers (2.47 ± 0.11 kPa, P < 0.0001). Stiffness of the pancreatic lesions in the head of 6.03 ± 0.42 kPa, body of 5.57 ± 0.82 kPa, and tail of 5.9 ± 1.9 kPa were also higher than those of corresponding segments of the normal volunteers (P = 0.0011, 0.0029, and 0.029, respectively). CONCLUSION: With the elastic belt, miscalculation of the pancreatic stiffness was reduced. MRE showed differences of stiffness between normal pancreas and pancreatic lesions.

Use of high-frequency visual stimuli above the critical flicker frequency in a SSVEP-based BMI.

OBJECTIVE: This study presents a new steady-state visual evoked potential (SSVEP)-based brain-machine interface (BMI) using flickering visual stimuli at frequencies greater than the critical flicker frequency (CFF). METHODS: We first asked participants to fixate on a green/blue flicker (30-70Hz), and SSVEP amplitude was evaluated. Participants were asked to indicate whether the stimulus was visibly flickering and to report their subjective level of discomfort. We then assessed visibly (41, 43, and 45Hz) vs. invisibly (61, 63, and 65Hz) flickering stimulus in an SSVEP-based BMI. Visual fatigue was assessed via the flicker test before and after operation of the BMI. RESULTS: Higher frequency stimuli reduced participants' subjective discomfort. Participants successfully controlled the SSVEP-based BMI using both the visibly and invisibly flickering stimuli (93.1% and 88.0%, respectively); the flicker test revealed a decrease in CFF (i.e., visual fatigue) under the visible condition only (-5.7%, P<0.001). CONCLUSIONS: The use of high-frequency visual stimuli above the CFF led to high classification accuracy and decreased visual fatigue in an SSVEP-based BMI. SIGNIFICANCE: High-frequency flicker stimuli above the CFF were able to induce SSVEPs and may prove useful in the development of BMI-based assistive products.

A BMI-based occupational therapy assist suit: asynchronous control by SSVEP.

A brain-machine interface (BMI) is an interface technology that uses neurophysiological signals from the brain to control external machines. Recent invasive BMI technologies have succeeded in the asynchronous control of robot arms for a useful series of actions, such as reaching and grasping. In this study, we developed non-invasive BMI technologies aiming to make such useful movements using the subject's own hands by preparing a BMI-based occupational therapy assist suit (BOTAS). We prepared a pre-recorded series of useful actions-a grasping-a-ball movement and a carrying-the-ball movement-and added asynchronous control using steady-state visual evoked potential (SSVEP) signals. A SSVEP signal was used to trigger the grasping-a-ball movement and another SSVEP signal was used to trigger the carrying-the-ball movement. A support vector machine was used to classify EEG signals recorded from the visual cortex (Oz) in real time. Untrained, able-bodied participants (n = 12) operated the system successfully. Classification accuracy and time required for SSVEP detection were ~88% and 3 s, respectively. We further recruited three patients with upper cervical spinal cord injuries (SCIs); they also succeeded in operating the system without training. These data suggest that our BOTAS system is potentially useful in terms of rehabilitation of patients with upper limb disabilities.

Implementation of a beam forming technique in real-time magnetoencephalography.

Real-time magnetoencephalography (rtMEG) is an emerging neurofeedback technology that could potentially benefit multiple areas of basic and clinical neuroscience. In the present study, we implemented voxel-based real-time coherence measurements in a rtMEG system in which we employed a beamformer to localize signal sources in the anatomical space prior to computing imaginary coherence. Our rtMEG experiment showed that a healthy subject could increase coherence between the parietal cortex and visual cortex when attending to a flickering visual stimulus. This finding suggests that our system is suitable for neurofeedback training and can be useful for practical brain-machine interface applications or neurofeedback rehabilitation.

The effect of temporal perception on weight perception.

A successful catch of a falling ball requires an accurate estimation of the timing for when the ball hits the hand. In a previous experiment in which participants performed ball-catching task in virtual reality environment, we accidentally found that the weight of a falling ball was perceived differently when the timing of ball load force to the hand was shifted from the timing expected from visual information. Although it is well known that spatial information of an object, such as size, can easily deceive our perception of its heaviness, the relationship between temporal information and perceived heaviness is still not clear. In this study, we investigated the effect of temporal factors on weight perception. We conducted ball-catching experiments in a virtual environment where the timing of load force exertion was shifted away from the visual contact timing (i.e., time when the ball hit the hand in the display). We found that the ball was perceived heavier when force was applied earlier than visual contact and lighter when force was applied after visual contact. We also conducted additional experiments in which participants were conditioned to one of two constant time offsets prior to testing weight perception. After performing ball-catching trials with 60 ms advanced or delayed load force exertion, participants' subjective judgment on the simultaneity of visual contact and force exertion changed, reflecting a shift in perception of time offset. In addition, timing of catching motion initiation relative to visual contact changed, reflecting a shift in estimation of force timing. We also found that participants began to perceive the ball as lighter after conditioning to 60 ms advanced offset and heavier after the 60 ms delayed offset. These results suggest that perceived heaviness depends not on the actual time offset between force exertion and visual contact but on the subjectively perceived time offset between them and/or estimation error in force timing.