Multimodal semi-supervised learning for online recognition of multi-granularity surgical workflows.

Purpose Surgical workflow recognition is a challenging task that requires understanding multiple aspects of surgery, such as gestures, phases, and steps. However, most existing methods focus on single-task or single-modal models and rely on costly annotations for training. To address these limitations, we propose a novel semi-supervised learning approach that leverages multimodal data and self-supervision to create meaningful representations for various surgical tasks. Methods Our representation learning approach conducts two processes. In the first stage, time contrastive learning is used to learn spatiotemporal visual features from video data, without any labels. In the second stage, multimodal VAE fuses the visual features with kinematic data to obtain a shared representation, which is fed into recurrent neural networks for online recognition. Results Our method is evaluated on two datasets: JIGSAWS and MISAW. We confirmed that it achieved comparable or better performance in multi-granularity workflow recognition compared to fully supervised models specialized for each task. On the JIGSAWS Suturing dataset, we achieve a gesture recognition accuracy of 83.3%. In addition, our model is more efficient in annotation usage, as it can maintain high performance with only half of the labels. On the MISAW dataset, we achieve 84.0% AD-Accuracy in phase recognition and 56.8% AD-Accuracy in step recognition. Conclusion Our multimodal representation exhibits versatility across various surgical tasks and enhances annotation efficiency. This work has significant implications for real-time decision-making systems within the operating room.

Real-time motion force-feedback system with predictive-vision for improving motor accuracy.

Many haptic guidance systems have been studied over the years; however, most of them have been limited to predefined guidance methods. Calculating guidance according to the operator's motion is important for efficient human motor adaptation and learning. In this study, we developed a system that haptically provides guidance trajectory by sequential weighting between the operator's trajectory and the ideal trajectory calculated from a predictive-vision system. We investigated whether motion completion with a predictive-vision system affects human motor accuracy and adaptation in time-constrained goal-directed reaching and ball-hitting tasks through subject experiments. The experiment was conducted with 12 healthy participants, and all participants performed ball-hitting tasks. Half of the participants get forceful guidance from the proposed system in the middle of the experiment. We found that the use of the proposed system improved the operator's motor performance. Furthermore, we observed a trend in which the improvement in motor performance using this system correlated with that after the washout of this system. These results suggest that the predictive-vision system effectively enhances motor accuracy to the target error in dynamic and time-constrained reaching and hitting tasks and may contribute to facilitating motor learning.

Intuitive Cell Manipulation Microscope System with Haptic Device for Intracytoplasmic Sperm Injection Simplification.

In recent years, the demand for effective intracytoplasmic sperm injection (ICSI) for the treatment of male infertility has increased. The ICSI operation is complicated as it involves delicate organs and requires a high level of skill. Several cell manipulation systems that do not require such skills have been proposed; notably, several automated methods are available for cell rotation. However, these methods are unfeasible for the delicate ICSI medical procedure because of safety issues. Thus, this study proposes a microscopic system that enables intuitive micropipette manipulation using a haptic device that safely and efficiently performs the entire ICSI procedure. The proposed system switches between field-of-view expansion and three-dimensional image presentation to present images according to the operational stage. In addition, the system enables intuitive pipette manipulation using a haptic device. Experiments were conducted on microbeads instead of oocytes. The results confirmed that the time required for the experimental task was improved by 52.6%, and the injection error was improved by 75.3% compared to those observed in the conventional system.

Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery.

In the context of Minimally Invasive Surgery, surgeons mainly rely on visual feedback during medical operations. In common procedures such as tissue resection, the automation of endoscopic control is crucial yet challenging, particularly due to the interactive dynamics of multi-agent operations and the necessity for real-time adaptation. This paper introduces a novel framework that unites a Hierarchical Quadratic Programming controller with an advanced interactive perception module. This integration addresses the need for adaptive visual field control and robust tool tracking in the operating scene, ensuring that surgeons and assistants have optimal viewpoint throughout the surgical task. The proposed framework handles multiple objectives within predefined thresholds, ensuring efficient tracking even amidst changes in operating backgrounds, varying lighting conditions, and partial occlusions. Empirical validations in scenarios involving single, double, and quadruple tool tracking during tissue resection tasks have underscored the system's robustness and adaptability. The positive feedback from user studies, coupled with the low cognitive and physical strain reported by surgeons and assistants, highlight the system's potential for real-world application.

The Heterogeneous Impact of Prediagnostic Folate Intake for Fluorouracil-Containing Induction Chemotherapy for Head and Neck Cancer.

Fluorouracil (FU) exerts its antitumor activity by inhibiting folate-mediated one-carbon metabolism. Evidence that folate may play a role in the carcinogenic process via folate-mediated one-carbon metabolism has given rise to the hypothesis that pre-diagnostic folate intake may induce heterogeneous chemosensitivity to FU-containing induction chemotherapy (IC) in head and neck cancer. To assess this hypothesis, we conducted a cohort study to investigate whether the association between prediagnostic dietary folate intake and cancer survival differed between treatment regimens with and without FU-containing IC in 504 cases of locally advanced (stage III/IV) HNSCC, using an epidemiologic database combined with clinical data. In total, 240 patients were treated with FU-containing IC followed by definitive treatment, and 264 patients were treated with definitive treatment alone. Definitive treatment is defined as (1) the surgical excision of a tumor with clear margins, with or without neck lymph node dissection; or (2) radiotherapy with or without chemotherapy. In the overall cohort of the FU-containing IC group, a higher folate intake was significantly associated with better overall survival (adjusted hazard ratios (HRs) for the highest compared to the lowest folate tertiles (HRT3-T1) = 0.42, 95%CI, 0.25-0.76, Ptrend = 0.003). Conversely, no apparent association between prediagnostic folate intake and survival was observed with definitive treatment alone (HRT3-T1: 0.83, 95%CI, 0.49-1.42, Ptrend = 0.491)). A consideration of the cumulative dose of FU-containing IC showed that the survival impact of prediagnostic folate intake differed statistically significantly by treatment regimen (Pinteraction = 0.012). In conclusion, an association between prediagnostic folate intake and HNSCC survival significantly differed by FU-containing IC. This finding indicates that in the carcinogenic process, folate status causes HNSCC to be heterogenous in terms of one-carbon metabolism.

Significance of timing of therapeutic line on effectiveness of nivolumab for metastatic renal cell carcinoma.

Objectives: This study aimed to clarify the significance of therapeutic timing on the effectiveness of nivolumab for treating metastatic renal cell carcinoma. Marterials and methods: Fifty-eight patients with metastatic renal cell carcinoma treated with nivolumab monotherapy were retrospectively studied. Patients who were treated with nivolumab as second-line therapy were included in the second-line group, while the others were included in the later-line group. The clinicopathological characteristics, effects of nivolumab, and prognoses of these groups were compared. Results: Twenty and thirty-eight patients were included in the second-line and later-line groups, respectively. There were no significant differences in the distribution of International Metastatic Renal Cell Carcinoma Database Consotium risk and other clinicopathological characteristics between the 2 groups. The proportion of patients whose objective best response was progressive disease in the second-line group was significantly lower than that in the later-line group (15% vs. 50%, p = 0.0090). The 50% progression-free survival with nivolumab in the second-line group was significantly better than that in the later-line group (not reached and 5 months, p = 0.0018). Multivariate analysis showed that the second-line setting was an independent predictive factor for better progression-free survival (p = 0.0028, hazard ratio = 0.108). The 50% overall survival after starting nivolumab in the second-line and later-line groups was not reached and 27.8 months, respectively (p = 0.2652). Conclusions: The therapeutic efficacy of nivolumab as second-line therapy is expected to be better than that of later therapy.

Gaze and Environmental Context-Guided Deep Neural Network and Sequential Decision Fusion for Grasp Intention Recognition.

Grasp intention recognition plays a crucial role in controlling assistive robots to aid older people and individuals with limited mobility in restoring arm and hand function. Among the various modalities used for intention recognition, the eye-gaze movement has emerged as a promising approach due to its simplicity, intuitiveness, and effectiveness. Existing gaze-based approaches insufficiently integrate gaze data with environmental context and underuse temporal information, leading to inadequate intention recognition performance. The objective of this study is to eliminate the proposed deficiency and establish a gaze-based framework for object detection and its associated intention recognition. A novel gaze-based grasp intention recognition and sequential decision fusion framework (GIRSDF) is proposed. The GIRSDF comprises three main components: gaze attention map generation, the Gaze-YOLO grasp intention recognition model, and sequential decision fusion models (HMM, LSTM, and GRU). To evaluate the performance of GIRSDF, a dataset named Invisible containing data from healthy individuals and hemiplegic patients is established. GIRSDF is validated by trial-based and subject-based experiments on Invisible and outperforms the previous gaze-based grasp intention recognition methods. In terms of running efficiency, the proposed framework can run at a frequency of about 22 Hz, which ensures real-time grasp intention recognition. This study is expected to inspire additional gaze-related grasp intention recognition works.

Indications for sentinel lymph node biopsy in node-negative oral cancers.

BACKGROUND: We aimed to define the indications for sentinel lymph node biopsy (SLNB), the third option for cervical treatment in oral cancer with negative cervical lymph nodes. METHODS: The greatest depth of invasion (DOI) and long diameter (LD) of the primary site were used as exposures. SLN metastasis was considered the outcome. RESULTS: In three trials conducted between 2009 and 2016, 158 patients were eligible and reassigned to this study group. The scatterplot based on the respective values of DOI and LD would eventually be divided into three sections. In cases of sections T1, T2, and T3, the proportions of SLN metastasis positivity were 21.3%, 35.3%, and 51.2%, respectively. In certain cases of T1 with 2 mm < DOI ≤ 5 mm and 8 mm < LD ≤ 20 mm, the proportion of SLN metastasis positivity was 40.9%. CONCLUSIONS: SLNB-navigated or assisted neck dissection can be added as an effective procedure for N0 neck control.

Predictive factors for dissection-free sentinel node micrometastases in early oral squamous cell carcinoma.

This sentinel node (SN) biopsy trial aimed to assess its effectiveness in identifying predictive factors of micrometastases and to determine whether elective neck dissection is necessary in oral squamous cell carcinoma. This retrospective study included 55 patients from three previous trials, with positive SNs. The relationship between the sizes of the metastatic focus and metastasis in non-sentinel node (NSN) was investigated. Four of the 55 largest metastatic focus were isolated tumor cells, and the remaining 51 were ranged from 0.2 to 15 mm, with a median of 2.6 mm. The difference of prevalence between 46 negative- and 9 positive-NSN was statistically significant with regard to age, long diameter of primary site and number of cases with regional recurrence. In comparing the size of largest metastatic focus dividing the number of positive SN, with metastaic focus range of < 3.0 mm in one-positive SN group, there were 18 (33%) negative-NSN and no positive-NSN. Regarding prognosis, 3-year overall survival rate of this group (n = 18) and other (n = 37) were 94% and 73% (p = 0.04), and 3-year recurrence free survival rate of this group and other were 94% and 51% (p = 0.03), respectively. Absolutely a further prospective clinical trial would be needed, micrometastases may be defined as solitary SN metastasis with < 3.0 mm of metastatic focus, and approximately 33% of neck dissections could be avoided using these criteria.

Natural Grasp Intention Recognition Based on Gaze in Human-Robot Interaction.

OBJECTIVE: While neuroscience research has established a link between vision and intention, studies on gaze data features for intention recognition are absent. The majority of existing gaze-based intention recognition approaches are based on deliberate long-term fixation and suffer from insufficient accuracy. In order to address the lack of features and insufficient accuracy in previous studies, the primary objective of this study is to suppress noise from human gaze data and extract useful features for recognizing grasp intention. METHODS: We conduct gaze movement evaluation experiments to investigate the characteristics of gaze motion. The target-attracted gaze movement model (TAGMM) is proposed as a quantitative description of gaze movement based on the findings. A Kalman filter (KF) is used to reduce the noise in the gaze data based on TAGMM. We conduct gaze-based natural grasp intention recognition evaluation experiments to collect the subject's gaze data. Four types of features describing gaze point dispersion ( fvar), gaze point movement ( fgm), head movement ( fhm), and distance from the gaze points to objects ( fdj) are then proposed to recognize the subject's grasp intentions. With the proposed features, we perform intention recognition experiments, employing various classifiers, and the results are compared with different methods. RESULTS: The statistical analysis reveals that the proposed features differ significantly across intentions, offering the possibility of employing these features to recognize grasp intentions. We demonstrated the intention recognition performance utilizing the TAGMM and the proposed features in within-subject and cross-subject experiments. The results indicate that the proposed method can recognize the intention with accuracy improvements of 44.26% (within-subject) and 30.67% (cross-subject) over the fixation-based method. The proposed method also consumes less time (34.87 ms) to recognize the intention than the fixation-based method (about 1 s). CONCLUSION: This work introduces a novel TAGMM for modeling gaze movement and a variety of practical features for recognizing grasp intentions. Experiments confirm the effectiveness of our approach. SIGNIFICANCE: The proposed TAGMM is capable of modeling gaze movements and can be utilized to process gaze data, and the proposed features can reveal the user's intentions. These results contribute to the development of gaze-based human-robot interaction.

A Concurrent Framework for Constrained Inverse Kinematics of Minimally Invasive Surgical Robots.

Minimally invasive surgery has undergone significant advancements in recent years, transforming various surgical procedures by minimizing patient trauma, postoperative pain, and recovery time. However, the use of robotic systems in minimally invasive surgery introduces significant challenges related to the control of the robot's motion and the accuracy of its movements. In particular, the inverse kinematics (IK) problem is critical for robot-assisted minimally invasive surgery (RMIS), where satisfying the remote center of motion (RCM) constraint is essential to prevent tissue damage at the incision point. Several IK strategies have been proposed for RMIS, including classical inverse Jacobian IK and optimization-based approaches. However, these methods have limitations and perform differently depending on the kinematic configuration. To address these challenges, we propose a novel concurrent IK framework that combines the strengths of both approaches and explicitly incorporates RCM constraints and joint limits into the optimization process. In this paper, we present the design and implementation of concurrent inverse kinematics solvers, as well as experimental validation in both simulation and real-world scenarios. Concurrent IK solvers outperform single-method solvers, achieving a 100% solve rate and reducing the IK solving time by up to 85% for an endoscope positioning task and 37% for a tool pose control task. In particular, the combination of an iterative inverse Jacobian method with a hierarchical quadratic programming method showed the highest average solve rate and lowest computation time in real-world experiments. Our results demonstrate that concurrent IK solving provides a novel and effective solution to the constrained IK problem in RMIS applications.

Impact of age for overall survival in head and neck sarcoma.

The purpose to the present study is to research the association between age at surgery and survival outcomes of patients with sarcoma in head and neck. Twenty-six patients with head and neck sarcoma who underwent by surgery from 2003 to 2017 were enrolled in the present observation study. Patients who did not undergo chemotherapy were significantly older age at surgery by Mann-Whitney U test. Fifty-five was the cutoff age that predicted death by receiver operating curve analysis. Shorter survival rates of overall, disease-specific, local recurrence-free and disease-free were associated with older age by log-rank test. Age (≥55 years/<55 years) was correlated with shorter overall survival by multivariate analysis of Cox's proportional hazards model adjusting with chemotherapy (absence/presence). In conclusion, older age predicts worse overall survival in head and neck sarcoma.

Macroscopic and multiple metastases in sentinel lymph node biopsy are respectively associated with poor prognosis in early oral cancer.

BACKGROUND: A multicenter, randomized controlled phase III trial was conducted on sentinel lymph node biopsy (SLNB) and elective neck dissection for T1 (depth of invasion ≥ 4 mm)-T2N0M0 oral cavity squamous cell carcinoma. This study identified factors associated with poor prognosis in patients who underwent SLNB based on a subgroup analysis of this trial. METHODS: We analyzed 418 sentinel lymph nodes (SLNs) from 132 patients who underwent SLNB. The metastatic SLNs were classified into three categories based on size-isolated tumor cells: < 0.2 mm, micrometastasis: ≥ 0.2 mm and < 2 mm, and macrometastasis: ≥ 2 mm. Three groups were formed based on the number of metastatic SLNs: no metastasis, 1 metastatic node, and ≥ 2 metastatic nodes. The size and number of metastatic SLNs on survival were evaluated using Cox proportional hazard models. RESULTS: Patients with macrometastasis and ≥ 2 metastatic SLNs had worse overall survival (OS) and disease-free survival (DFS) after adjustment for potential confounders (HR for OS: macrometastasis, 4.85; 95% CI 1.34-17.60; ≥ 2 metastatic SLN, 3.63; 95% CI 1.02-12.89; HR for DFS: macrometastasis, 2.94; 95% CI 1.16-7.44; ≥ 2 metastatic SLN, 2.97; 95% CI 1.18-7.51). CONCLUSIONS: In patients who underwent SLNB, a poorer prognosis was associated with macrometastasis or having ≥ 2 metastatic SLNs.

Estimation of CO2 Separation Performances through CHA-Type Zeolite Membranes Using Molecular Simulation.

Chabazite (CHA)-type zeolite membranes are a potential material for CO2 separations because of their small pore aperture, large pore volume, and low aluminum content. In this study, the permeation and separation properties were evaluated using a molecular simulation technique with a focus on improving the CO2 separation performance. The adsorption isotherms of CO2 and CH4 on CHA-type zeolite with Si/Al = 18.2 were predicted by grand canonical Monte Carlo, and the diffusivities in zeolite micropores were simulated by molecular dynamics. The CO2 separation performance of the CHA-type zeolite membrane was estimated by a Maxwell-Stefan equation, accounting for mass transfer through the support tube. The results indicated that the permeances of CO2 and CH4 were influenced mainly by the porosity of the support, with the CO2 permeance reduced due to preferential adsorption with increasing pressure drop. In contrast, it was important for estimation of the CH4 permeance to predict the amounts of adsorbed CH4. Using molecular simulation and the Maxwell-Stefan equation is shown to be a useful technique for estimating the permeation properties of zeolite membranes, although some problems such as predicting accurate adsorption terms remain.

Evaluation of FAU-type Zeolite Membrane Stability in Transesterification Reaction Conditions.

The transesterification conversion of methyl ether can be enhanced by the removal of the byproduct methanol using methanol permselective faujasite (FAU-type) zeolite membranes. However, the authors previously observed that the methanol flux during the transesterification reaction was lower than the predicted flux. Therefore, this study investigated the stability of FAU-type zeolite membranes in the presence of organic components associated with the transesterification reaction of methyl hexanoate and 1-hexanol. The stability was defined in terms of changes in methanol permeance and zeolite structure. The effect of reaction components (methanol, 1-hexanol, methyl hexanoate, and hexyl hexanoate) on the FAU-type zeolite structure and the methanol permeation performance of the FAU-type zeolite membranes were evaluated to find the component causing the lower methanol flux. From these results, two esters were found to adsorb strongly on the FAU-type zeolite. The methanol flux of the FAU-type zeolite membrane was examined after vapor exposure of each of the four reaction chemicals at 373 K for 8 h. In the case of methyl hexanoate and hexyl hexanoate vapor exposure, the methanol flux was reduced by about 75% compared to the initial flux of 15 kg m-2 h-1. These results indicated methanol permeation performance was inhibited by the adsorption of esters.

Impact of low and high body mass index on predicting therapeutic efficacy and prognosis in patients with metastatic renal cell carcinoma treated with first-line tyrosine kinase inhibitors.

OBJECTIVE: To evaluate the significance of both low and high body mass index (BMI) as a biomarker in first-line tyrosine kinase inhibitors (TKIs) for metastatic renal cell carcinoma (mRCC). METHODS: The oncological outcome of 235 patients with mRCC treated with TKI from 2007 to 2018 was reviewed retrospectively. All patients received first-line TKI as therapy. We analyzed the relationship between BMI (low and high) and disease control rate. The primary outcome was progression free survival and overall survival, and the association between BMI and survival prognosis was evaluated. RESULTS: The median BMI was 22.5 kg/m2 , and 25 patients (10.7%) had a low BMI (<18.5 kg/m2 ), 158 patients (67.2%) had a normal BMI (18.5-25 kg/m2 ), and 52 patients (22.1%) had a high BMI (≥ 25 kg/m2 ). Patients in the low BMI group had a significantly lower disease control rate, whereas patients in the high BMI group had a significantly higher disease control rate (p = 0.002 and p = 0.030, respectively). A log-rank test showed prognosis to be significantly poorer in the low BMI group and to be significantly better in the high BMI group than that in the normal BMI group. Multivariable Cox regression analysis showed that low BMI was an independent indicator of poor prognosis, whereas high BMI was an independent indicator of favorable prognosis. CONCLUSION: We showed the impact of both low and high BMI on predicting therapeutic efficacy and prognosis in mRCC patients treated with TKI.

Auditory Feedback for Enhanced Sense of Agency in Shared Control.

There is a growing need for robots that can be remotely controlled to perform tasks of one's own choice. However, the SoA (Sense of Agency: the sense of recognizing that the motion of an observed object is caused by oneself) is reduced because the subject of the robot motion is identified as external due to shared control. To address this issue, we aimed to suppress the decline in SoA by presenting auditory feedback that aims to blur the distinction between self and others. We performed the tracking task in a virtual environment under four different auditory feedback conditions, with varying levels of automation to manipulate the virtual robot gripper. Experimental results showed that the proposed auditory feedback suppressed the decrease in the SoA at a medium level of automation. It is suggested that our proposed auditory feedback could blur the distinction between self and others, and that the operator attributes the subject of the motion of the manipulated object to himself.

A Wirelessly Powered 4-Channel Neurostimulator for Reconstructing Walking Trajectory.

A wirelessly powered four-channel neurostimulator was developed for applying selective Functional Electrical Stimulation (FES) to four peripheral nerves to control the ankle and knee joints of a rat. The power of the neurostimulator was wirelessly supplied from a transmitter device, and the four nerves were connected to the receiver device, which controlled the ankle and knee joints in the rat. The receiver device had functions to detect the frequency of the transmitter signal from the transmitter coil. The stimulation site of the nerves was selected according to the frequency of the transmitter signal. The rat toe position was controlled by changing the angles of the ankle and knee joints. The joint angles were controlled by the stimulation current applied to each nerve independently. The stimulation currents were adjusted by the Proportional Integral Differential (PID) and feed-forward control method through a visual feedback control system, and the walking trajectory of a rat's hind leg was reconstructed. This study contributes to controlling the multiple joints of a leg and reconstructing functional motions such as walking using the robotic control technology.

Intelligent Gait Analysis and Evaluation System Based on Cane Robot.

Gait analysis and evaluation are vital for disease diagnosis and rehabilitation. Current gait analysis technologies require wearable devices or high-resolution vision systems within a limited usage space. To facilitate gait analysis and quantitative walking-ability evaluation in daily environments without using wearable devices, a mobile gait analysis and evaluation system is proposed based on a cane robot. Two laser range finders (LRFs) are mounted to obtain the leg motion data. An effective high-dimensional Takagi-Sugeno-Kang (HTSK) fuzzy system, which is suitable for high-dimensional data by solving the saturation problem caused by softmax function in defuzzification, is proposed to recognize the walking states using only the motion data acquired from LRFs. The gait spatial-temporal parameters are then extracted based on the gait cycle segmented by different walking states. Besides, a quantitative walking-ability evaluation index is proposed in terms of the conventional Tinetti scale. The plantar pressure sensing system records the walking states to label training data sets. Experiments were conducted with seven healthy subjects and four patients. Compared with five classical classification algorithms, the proposed method achieves the average accuracy rate of 96.57%, which is improved more than 10%, compared with conventional Takagi-Sugeno-Kang (TSK) fuzzy system. Compared with the gait parameters extracted by the motion capture system OptiTrack, the average errors of step length and gait cycle are only 0.02 m and 1.23 s, respectively. The comparison between the evaluation results of the robot system and the scores given by the physician also validates that the proposed method can effectively evaluate the walking ability.

Development of High-Cell-Density Tissue Method for Compressed Modular Bioactuator.

Bioactuators have been developed in many studies in the recent decade for actuators of micro-biorobots. However, bioactuators have not shown the same power as animal muscles. Centrifugal force was used in this study to increase the cell density of cultured muscle cells that make up the bioactuator. The effect of the centrifugal force on cells in the matrix gel before curing was investigated, and the optimal centrifugal force was identified to be around 450× g. The compressed modular bioactuator (C-MBA) fabricated in this study exhibited 1.71 times higher cell density than the conventional method. In addition, the contractile force per unit cross-sectional area was 1.88 times higher. The proposed method will contribute to new bioactuators with the same power as living muscles in animals.