Improving situation recognition using endoscopic videos and navigation information for endoscopic sinus surgery.

PURPOSE: Endoscopic sinus surgery (ESS) is widely used to treat chronic sinusitis. However, it involves the use of surgical instruments in a narrow surgical field in close proximity to vital organs, such as the brain and eyes. Thus, an advanced level of surgical skill is expected of surgeons performing this surgery. In a previous study, endoscopic images and surgical navigation information were used to develop an automatic situation recognition method in ESS. In this study, we aimed to develop a more accurate automatic surgical situation recognition method for ESS by improving the method proposed in our previous study and adding post-processing to remove incorrect recognition. METHOD: We examined the training model parameters and the number of long short-term memory (LSTM) units, modified the input data augmentation method, and added post-processing. We also evaluated the modified method using clinical data. RESULT: The proposed improvements improved the overall scene recognition accuracy compared with the previous study. However, phase recognition did not exhibit significant improvement. In addition, the application of the one-dimensional median filter significantly reduced short-time false recognition compared with the time series results. Furthermore, post-processing was required to set constraints on the transition of the scene to further improve recognition accuracy. CONCLUSION: We suggested that the scene recognition could be improved by considering the model parameter, adding the one-dimensional filter and post-processing. However, the scene recognition accuracy remained unsatisfactory. Thus, a more accurate scene recognition and appropriate post-processing method is required.

Brain activation measurement for motion gain decision of surgical endoscope manipulation.

BACKGROUND: Robotic surgery improves minimally invasive interventions. However, it is challenging to determine the best gain settings for control of the endoscope. Providing the surgeon with the ability to manipulate the endoscope at an appropriate speed will likely improve the surgery by reducing the surgeon's stress. In this study, we validated the feasibility of a gain-tuning method in which the operator's brain activity is measured and used to evaluate stress levels. METHODS: We developed an endoscope-manipulation simulator and used functional near-infrared spectroscopy to measure the prefrontal cortical activity, while participants controlled the simulator at different gain values. RESULTS: The brain activation levels in the prefrontal cortex exhibited significant differences under different viewpoint motion gain conditions. CONCLUSIONS: The stress-related brain activity was significantly reduced at specific gains, indicating that the brain activity evaluation would be useful to determine the parameters related to the operator's feelings.

Using Brain Activation to Evaluate Arrangements Aiding Hand-Eye Coordination in Surgical Robot Systems.

GOAL: To realize intuitive, minimally invasive surgery, surgical robots are often controlled using master-slave systems. However, the surgical robot's structure often differs from that of the human body, so the arrangement between the monitor and master must reflect this physical difference. In this study, we validate the feasibility of an embodiment evaluation method that determines the arrangement between the monitor and master. In our constructed cognitive model, the brain's intraparietal sulcus activates significantly when somatic and visual feedback match. Using this model, we validate a cognitively appropriate arrangement between the monitor and master. METHODS: In experiments, we measure participants' brain activation using an imaging device as they control the virtual surgical simulator. Two experiments are carried out that vary the monitor and hand positions. CONCLUSION: There are two common arrangements of the monitor and master at the brain activation's peak: One is placing the monitor behind the master, so the user feels that the system is an extension of his arms into the monitor; the other arranges the monitor in front of the master, so the user feels the correspondence between his own arm and the virtual arm in the monitor. SIGNIFICANCE: From these results, we conclude that the arrangement between the monitor and master impacts embodiment, enabling the participant to feel apparent posture matches in master-slave surgical robot systems.

Development of angle information system to facilitate the adjustment of needle-holding posture.

PURPOSE: Our purpose is to develop a system based on image processing methods that can inform users of the angular relationship between the needle and the forceps. The user thereby adjusts their needle grasping posture according to the angle information, which leads to an improvement in suturing accuracy. METHODS: The system prototype consists of a camera and an image processing computer. The image captured by the camera is input to the computer, and then, the angular relationship between the forceps and needle is calculated via image processing. Then, the system informs the user of the calculated angular relationship between the needle and forceps in real time. To evaluate whether the system improves suturing accuracy, we invited 12 participants to enroll in an experiment based on a suturing task. RESULTS: The experimental results illustrate that the system allows participants to easily adjust the positional relationship between the needle and the forceps and that this adjusted angular relationship leads to higher suturing accuracy. CONCLUSIONS: Adjustment to holding the needle at a right angle before insertion has a critical effect on suturing quality. Therefore, we developed a system that informs the user of the angular relationship between the needle and the forceps. The results of the evaluation show that the system significantly improves the suturing accuracy of participants via informing them of the angle.

Virtually transparent surgical instruments in endoscopic surgery with augmentation of obscured regions.

PURPOSE: We developed and evaluated a visual compensation system that allows surgeons to visualize obscured regions in real time, such that the surgical instrument appears virtually transparent. METHODS: The system consists of two endoscopes: a main endoscope to observe the surgical environment, and a supporting endoscope to render the region hidden from view by surgical instruments. The view captured by the supporting endoscope is transformed to simulate the view from the main endoscope, segmented to the shape of the hidden regions, and superimposed to the main endoscope image so that the surgical instruments look transparent. A prototype device was benchmarked for processing time and superimposition rendering error. Then, it was evaluated in a training environment with 22 participants performing a backhand needle driving task with needle exit point error as the criterion. Lastly, we conducted an in vivo study. RESULTS: In the benchmark, the mean processing time was 62.4 ms, which was lower than the processing time accepted in remote surgeries. The mean superimposition error of the superimposed image was 1.4 mm. In the training environment, needle exit point error with the system decreased significantly for experts compared with the condition without the system. This change was not significant for novices. In the in vivo study, our prototype enabled visualization of needle exit points during anastomosis. CONCLUSION: The benchmark suggests that the implemented system had an acceptable performance, and evaluation in the training environment demonstrated improved surgical task outcomes in expert surgeons. We will conduct a more comprehensive in vivo study in the future.

Brain activation in parietal area during manipulation with a surgical robot simulator.

PURPOSE: we present an evaluation method to qualify the embodiment caused by the physical difference between master-slave surgical robots by measuring the activation of the intraparietal sulcus in the user's brain activity during surgical robot manipulation. We show the change of embodiment based on the change of the optical axis-to-target view angle in the surgical simulator to change the manipulator's appearance in the monitor in terms of hand-eye coordination. The objective is to explore the change of brain activation according to the change of the optical axis-to-target view angle. METHODS: In the experiments, we used a functional near-infrared spectroscopic topography (f-NIRS) brain imaging device to measure the brain activity of the seven subjects while they moved the hand controller to insert a curved needle into a target using the manipulator in a surgical simulator. The experiment was carried out several times with a variety of optical axis-to-target view angles. RESULTS: Some participants showed a significant peak (P value = 0.037, F-number = 2.841) when the optical axis-to-target view angle was 75°. CONCLUSIONS: The positional relationship between the manipulators and endoscope at 75° would be the closest to the human physical relationship between the hands and eyes.

Visual and somatic sensory feedback of brain activity for intuitive surgical robot manipulation.

This paper presents a method to evaluate the hand-eye coordination of the master-slave surgical robot by measuring the activation of the intraparietal sulcus in users brain activity during controlling virtual manipulation. The objective is to examine the changes in activity of the intraparietal sulcus when the user's visual or somatic feedback is passed through or intercepted. The hypothesis is that the intraparietal sulcus activates significantly when both the visual and somatic sense pass feedback, but deactivates when either visual or somatic is intercepted. The brain activity of three subjects was measured by the functional near-infrared spectroscopic-topography brain imaging while they used a hand controller to move a virtual arm of a surgical simulator. The experiment was performed several times with three conditions: (i) the user controlled the virtual arm naturally under both visual and somatic feedback passed, (ii) the user moved with closed eyes under only somatic feedback passed, (iii) the user only gazed at the screen under only visual feedback passed. Brain activity showed significantly better control of the virtual arm naturally (p<;0.05) when compared with moving with closed eyes or only gazing among all participants. In conclusion, the brain can activate according to visual and somatic sensory feedback agreement.

Development of new measurement system of thoracic excursion with biofeedback: reliability and validity.

BACKGROUND: Respiratory rehabilitation reduces breathlessness from patient with respiratory dysfunction. Chest expansion score, which represents the circumference magnitude of the thoracic cage, is used for a target when treating patients with respiratory disease. However, it is often difficult for patients to understand the changes in the respiratory status and be motivated for therapy continuously. We developed a new measurement system with biofeedback named BREATH which shows chest expansion scores in real time. The purpose of this study was to determine the reliability and validity of the novel system in advance of clinical application. METHODS: Three evaluators measured chest expansion in 33 healthy individuals using tape measure, which is used for the measurement traditionally, and BREATH. The wire for BREATH system was threaded over the thoracic continuously and the data was recorded automatically; whereas the tape was winded and measured each maximal expiration and inspiration timing by evaluator. All participants were performed both measurement simultaneously for three times during deep breath. In this study, we studied chest expansion score without using biofeedback data of BREATH to check the validity of the result. To confirm intra- and inter-evaluator reliability, we computed intra-class correlations (ICCs). We used Pearson's correlation coefficient to evaluate the validity of measurement result by BREATH with reference to the tape measure results. RESULTS: The average (standard deviation) chest expansion scores for all, men and women by the tape measure were 5.53 (1.88), 6.40 (1.69) and 5.22 (1.39) cm, respectively, and those by BREATH were 3.89 (2.04), 4.36 (1.83) and 2.89 (1.66) cm, respectively. ICC within and among the three evaluators for BREATH and the tape measure were 0.90-0.94 and 0.85-0.94 and 0.85 and 0.82, respectively. The correlation coefficient between the two methods was 0.76-0.87. CONCLUSION: The novel measurement system, BREATH, has high intra- and inter-evaluator reliabilities and validity; therefore it can lead us more effective respiratory exercise. Using its biofeedback data, this system may help patients with respiratory disease to do exercises more efficiently and clinicians to assess the respiratory exercise more accurately.

Examination of a muscular activity estimation model using a Bayesian network for the influence of an ankle foot orthosis.

In the present paper, we examine the appropriateness of a new model to examine the activity of the foot in gait. We developed an estimation model for foot-ankle muscular activity in the design of an ankle-foot orthosis by means of a statistical method. We chose three muscles for measuring muscular activity and built a Bayesian network model to confirm the appropriateness of the estimation model. We experimentally examined the normal gait of a non-disabled subject. We measured the muscular activity of the lower foot muscles using electromyography, the joint angles, and the pressure on each part of the sole. From these data, we obtained the causal relationship at every 10% level for these factors and built models for the stance phase, control term, and propulsive term. Our model has three advantages. First, it can express the influences that change during gait because we use 10% level nodes for each factor. Second, it can express the influences of factors that differ for low and high muscular-activity levels. Third, we created divided models that are able to reflect the actual features of gait. In evaluating the new model, we confirmed it is able to estimate all muscular activity level with an accuracy of over 90%.

Intuitive operability evaluation of surgical robot using brain activity measurement to determine immersive reality.

Surgical robots have improved considerably in recent years, but intuitive operability, which represents user inter-operability, has not been quantitatively evaluated. Therefore, for design of a robot with intuitive operability, we propose a method to measure brain activity to determine intuitive operability. The objective of this paper is to determine the master configuration against the monitor that allows users to perceive the manipulator as part of their own body. We assume that the master configuration produces an immersive reality experience for the user of putting his own arm into the monitor. In our experiments, as subjects controlled the hand controller to position the tip of the virtual slave manipulator on a target in a surgical simulator, we measured brain activity through brain-imaging devices. We performed our experiments for a variety of master manipulator configurations with the monitor position fixed. For all test subjects, we found that brain activity was stimulated significantly when the master manipulator was located behind the monitor. We conclude that this master configuration produces immersive reality through the body image, which is related to visual and somatic sense feedback.

Thoracic ROM measurement system with visual bio-feedback: system design and biofeedback evaluation.

Patients with diseases such as chronic obstructive pulmonary disease (COPD) need to improve their thorax mobility. Thoracic ROM is one of the simplest and most useful indexes to evaluate the respiratory function. In this paper, we have proposed the prototype of a simple thoracic ROM measurement system with real-time visual bio-feedback in the chest expansion test. In this system, the thoracic ROM is measured using a wire-type linear encoder whose wire is wrapped around the thorax. In this paper, firstly, the repeatability and reliability of measured thoracic ROM was confirmed as a first report of the developed prototype. Secondly, we analyzed the effect of the bio-feedback system on the respiratory function. The result of the experiment showed that it was easier to maintain a large and stable thoracic ROM during deep breathing by using the real-time visual biofeedback system of the thoracic ROM.

Subtle grip force estimation from EMG and muscle stiffness--relationship between muscle character frequency and grip force.

A number of upper limb amputees experience difficulty in picking up a food bowl during a meal, because grip force estimation using EMG currently does not provide sufficient accuracy for this task. In this paper, we propose a grip force estimation system that allows amputees to pick up a bowl with a prosthetic hand by using the properties of muscle stiffness in addition to EMG. We have chosen a tray holding task to evaluate the proposed system. A weight is dropped on the tray and the subjects are expected to control the tray's attitude during the task. Actual grip force, EMG, and muscle stiffness are measured, and the actual measured grip force is compared with the estimated grip force for evaluation. As a result, the proposed algorithm is found to be able to estimate grip force with an error of just 18[N], which is 30% smaller than in the method that uses only EMG. From the result that the response time estimated by proposed system is even less than a human's mechanical reaction time, the effectiveness of the proposed method has been validated.

Pilot study on effectiveness of simulation for surgical robot design using manipulability.

Medical technology has advanced with the introduction of robot technology, which facilitates some traditional medical treatments that previously were very difficult. However, at present, surgical robots are used in limited medical domains because these robots are designed using only data obtained from adult patients and are not suitable for targets having different properties, such as children. Therefore, surgical robots are required to perform specific functions for each clinical case. In addition, the robots must exhibit sufficiently high movability and operability for each case. In the present study, we focused on evaluation of the mechanism and configuration of a surgical robot by a simulation based on movability and operability during an operation. We previously proposed the development of a simulator system that reproduces the conditions of a robot and a target in a virtual patient body to evaluate the operability of the surgeon during an operation. In the present paper, we describe a simple experiment to verify the condition of the surgical assisting robot during an operation. In this experiment, the operation imitating suturing motion was carried out in a virtual workspace, and the surgical robot was evaluated based on manipulability as an indicator of movability. As the result, it was confirmed that the robot was controlled with low manipulability of the left side manipulator during the suturing. This simulation system can verify the less movable condition of a robot before developing an actual robot. Our results show the effectiveness of this proposed simulation system.

Pilot study of design method for surgical robot using workspace reproduction system.

Recent development methods for surgical robots have an inherent problem. The user-friendliness of operating robot cannot be revealed until completion of the robot. To assist the design of a surgical robot that is user-friendly in terms of surgeon's operation, we propose a system that considers the operation manner of surgeon during the design phase of the robot. This system includes the following functionality: 1) a master manipulator that measures the operation manner of the surgeon (operator), and 2) a slave simulator in which the mechanical parameters can be configured freely. The operator can use the master manipulator to operate the slave simulator. Using this system, we investigate the necessity of considering the operator's manner when developing a surgical robot. In the experiment, we used three instruments with mechanisms that differed with respect to the length between bending joints and measured the trajectory of each instrument tip position during the surgical task. The results show that there are differences in the trajectories of each mechanism. Based on the results, changes in the mechanism of the surgical robot influenced the operator's manner. Therefore, when designing the mechanism for a surgical robot, there is a need to consider how this influences the operator's manner.

A detailed 3D ankle-foot model for simulate dynamics of lower limb orthosis.

The objective of this study is to develop a 3D ankle-foot model containing toe expression for designing an AFO (ankle-foot orthosis) with a training function. Two experiments were conducted to (1) show the influence of toes by comparing walking with and without an AFO, and (2) clarify the functions of toes during walking by correlating the activity of the major muscles controlling the ankle and the toes to the sole pressure data during walking. By analyzing the results of these two experiments, the necessary components and conditions of a detailed 3D foot-ankle model for developing an AFO with a training effect were clarified. A model was built and examined with empirical facts, and data were collected from the AFO simulation.

Pilot study on verification of effectiveness on operability of assistance system for robotic tele-surgery using simulation.

Tele-surgery enables medical care even in remote regions, and has been accomplished in clinical cases by means of dedicated communication lines. To make tele-surgery a more widespread method of providing medical care, a surgical environment needs to be made available using public lines of communication, such as the Internet. Moreover, a support system during surgery is required, as the use of surgical tools is performed in an environment subject to delay. In our research, we focus on the operability of specific tasks conducted by surgeons during a medical procedure, with the aim of clarifying, by means of a simulation, the optimum environment for robotic tele-surgery. In the study, we set up experimental systems using our proposed simulation system. In addition, we investigate the mental workloads on subjects and verify the effect of visual-assistance information as a pilot study. The operability of the task of gripping soft tissue was evaluated using a subjective workload assessment tool, the NASA Task Load Index. Results show that the tasks were completed, but the workload did not improve to less than 300ms and 400ms in the simulated environment. Verifying the effect of the support system was an important task under a more-than 200ms delay using this experiment, and future studies will evaluate the operability of the system under varying conditions of comfort. In addition, an intra-operative assistance system will be constructed using a simulation.

Development of an integrated needle insertion system with image guidance and deformation simulation.

OBJECTIVE: The purpose of our work was to develop an integrated system with image guidance and deformation simulation for the purpose of accurate needle insertion. METHODS: We designed an ultrasound-guided needle insertion manipulator and physical model to simulate liver deformation. We carried out an in vivo experiment using a porcine liver to verify the effectiveness of our manipulator and model. RESULTS: The results of the in vivo experiment showed that the needle insertion manipulator accurately positions the needle tip into the target. The experimental results also showed that the liver model accurately reproduces the nonlinear increase of force upon the needle during insertion. DISCUSSION: Based on these results, it is suggested that the needle insertion manipulator and the physical liver model developed and validated in this work are effective for accurate needle insertion.

Operability evaluation using an simulation system for gripping motion in robotic tele-surgery.

Tele-surgery enables medical care even in remote regions and has been accomplished in clinical cases by means of special communication lines. To make tele-surgery a more common method of medical care, the surgical environment must be made available using public lines of communication, such as the Internet. Moreover, a support system during operation is required as the use of surgical tools occurs in a delayed environment. In our research, we focus on the operability of certain tasks conducted by surgeons during a medical procedure, and aim to clarify the optimum environment for robotic tele-surgery using a simulation. In the present study, we conducted an experiment to evaluate this operability using a simulation system consisting of a virtual slave manipulator, network simulator and an organ deformation calculator. The operability of a task to grip soft tissue was evaluated using a subjective workload assessment tool, NASA Task Load Index (NASA-TLX). Results indicate that operability changed over a delay of 200 ms in the environment during the experiment. Future studies will focus on clarifying a comfortable tele-surgical environment using the present evaluation of operability. In addition, an intra-operative assistance system will be constructed using a simulation.

Development and validation of a viscoelastic and nonlinear liver model for needle insertion.

OBJECTIVE: The objective of our work is to develop and validate a viscoelastic and nonlinear physical liver model for organ model-based needle insertion, in which the deformation of an organ is estimated and predicted, and the needle path is determined with organ deformation taken into consideration. MATERIALS AND METHODS: First, an overview is given of the development of the physical liver model. The material properties of the liver considering viscoelasticity and nonlinearity are modeled based on the measured data collected from a pig's liver. The method to develop the liver model using FEM is also shown. Second, the experimental method to validate the model is explained. Both in vitro and in vivo experiments that made use of a pig's liver were conducted for comparison with the simulation using the model. RESULTS: Results of the in vitro experiment showed that the model reproduces nonlinear and viscoelastic response of displacement at an internally located point with high accuracy. For a force up to 0.45 N, the maximum error is below 1 mm. Results of the in vivo experiment showed that the model reproduces the nonlinear increase of load upon the needle during insertion. DISCUSSION: Based on these results, the liver model developed and validated in this work reproduces the physical response of a liver in both in vitro and in vivo situations.

Unimolecular and collision-induced dissociation of singly-charged mono-bromide silver clusters Ag(x)Br(+) (x = 2, 4, 6, 8, 10).

Relative intensities of singly-charged mono-bromide silver clusters Ag(x)Br(+) formed from sputtering of a pressed pellet of silver bromide were measured by mass spectrometry. The obtained results suggest that the Ag(x)Br(+) clusters have a structural formula of the form Ag(x-1)(+)(AgBr). The relative stability of Ag(x-1)(+)(AgBr) was determined by the intrinsic stability of the remaining metallic portion of the cluster (Ag(x-1)(+)) as predicted by the spherical jellium model (SJM). Unimolecular and high- energy collision-induced dissociation (CID) spectra of Ag(x)Br(+) (x = 2, 4, 6, 8, 10) clusters were also measured. In all of the spectra, the most intense fragment peaks were assigned to the Ag(x-1)(+) ions accompanying the loss of AgBr. The difference in the relative intensities of the Ag(x-1)(+) peaks between unimolecular dissociation and CID spectra led us to conclude that the weakest bond in the excited cluster Ag(x)Br(+*) is the Ag(x-1)(+)-AgBr bond and the structure of Ag(x)Br(+) is a metallic Ag(x-1)(+) ion cluster adduct with AgBr. The primary fragments observed in the CID spectra were also explained by the stabilities of the generated ion products and neutral fragments, both having even delocalized valence electrons. The present results were consistently explained by SJM. The dissociation behavior of Ag(2)Br(+) can be explained on the basis of the calculated thermochemical data.