Fully Organic Sensors for Continuous Real-Time Digestion Monitoring.

Monitoring the gastric digestive function is important for the diagnosis of gastric disorders and drug development. However, there is no report on the in situ and real-time monitoring of digestive functions. Herein, we report a flexible fully organic sensor to effectively monitor protein digestion in situ in a simulated gastric environment for the first time. The sensors are made of a blend of gluten that is a protein and poly(3,4-ethylenedioxythiophene):polystyrenesulfonate (PEDOT:PSS) that is a conducting polymer. During the protein digestion, the breakdown of the polypeptides increases the level of separation among the PEDOT chains, thereby increasing the resistance. The resistance variation is sensitive to various conditions, including the concentration of pepsin that is the enzyme for protein digestion, temperature, pH value, and digestive drugs. Hence, these sensors can provide real-time information about the digestion and efficacy of digestive drugs. In addition, the signals can be collected via a convenient wireless communication manner.

SimCol3D - 3D reconstruction during colonoscopy challenge.

Colorectal cancer is one of the most common cancers in the world. While colonoscopy is an effective screening technique, navigating an endoscope through the colon to detect polyps is challenging. A 3D map of the observed surfaces could enhance the identification of unscreened colon tissue and serve as a training platform. However, reconstructing the colon from video footage remains difficult. Learning-based approaches hold promise as robust alternatives, but necessitate extensive datasets. Establishing a benchmark dataset, the 2022 EndoVis sub-challenge SimCol3D aimed to facilitate data-driven depth and pose prediction during colonoscopy. The challenge was hosted as part of MICCAI 2022 in Singapore. Six teams from around the world and representatives from academia and industry participated in the three sub-challenges: synthetic depth prediction, synthetic pose prediction, and real pose prediction. This paper describes the challenge, the submitted methods, and their results. We show that depth prediction from synthetic colonoscopy images is robustly solvable, while pose estimation remains an open research question.

FARN: Fetal Anatomy Reasoning Network for Detection with Global Context Semantic and Local Topology Relationship.

Accurate recognition of fetal anatomical structure is a pivotal task in ultrasound (US) image analysis. Sonographers naturally apply anatomical knowledge and clinical expertise to recognizing key anatomical structures in complex US images. However, mainstream object detection approaches usually treat each structure recognition separately, overlooking anatomical correlations between different structures in fetal US planes. In this work, we propose a Fetal Anatomy Reasoning Network (FARN) that incorporates two kinds of relationship forms: a global context semantic block summarized with visual similarity and a local topology relationship block depicting structural pair constraints. Specifically, by designing the Adaptive Relation Graph Reasoning (ARGR) module, anatomical structures are treated as nodes, with two kinds of relationships between nodes modeled as edges. The flexibility of the model is enhanced by constructing the adaptive relationship graph in a data-driven way, enabling adaptation to various data samples without the need for predefined additional constraints. The feature representation is further enhanced by aggregating the outputs of the ARGR module. Comprehensive experimental results demonstrate that FARN achieves promising performance in detecting 37 anatomical structures across key US planes in tertiary obstetric screening. FARN effectively utilizes key relationships to improve detection performance, demonstrates robustness to small-scale, similar, and indistinct structures, and avoids some detection errors that deviate from anatomical norms. Overall, our study serves as a resource for developing efficient and concise approaches to model inter-anatomy relationships.

Photonic time-delayed reservoir computing based on series-coupled microring resonators with high memory capacity.

On-chip microring resonators (MRRs) have been proposed to construct time-delayed reservoir computing (RC) systems, which offer promising configurations available for computation with high scalability, high-density computing, and easy fabrication. A single MRR, however, is inadequate to provide enough memory for the computation task with diverse memory requirements. Large memory requirements are satisfied by the RC system based on the MRR with optical feedback, but at the expense of its ultralong feedback waveguide. In this paper, a time-delayed RC is proposed by utilizing a silicon-based nonlinear MRR in conjunction with an array of linear MRRs. These linear MRRs possess a high quality factor, providing enough memory capacity for the RC system. We quantitatively analyze and assess the proposed RC structure's performance on three classical tasks with diverse memory requirements, i.e., the Narma 10, Mackey-Glass, and Santa Fe chaotic timeseries prediction tasks. The proposed system exhibits comparable performance to the system based on the MRR with optical feedback, when it comes to handling the Narma 10 task, which requires a significant memory capacity. Nevertheless, the dimension of the former is at least 350 times smaller than the latter. The proposed system lays a good foundation for the scalability and seamless integration of photonic RC.

Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgery.

PURPOSE: Depth estimation in robotic surgery is vital in 3D reconstruction, surgical navigation and augmented reality visualization. Although the foundation model exhibits outstanding performance in many vision tasks, including depth estimation (e.g., DINOv2), recent works observed its limitations in medical and surgical domain-specific applications. This work presents a low-ranked adaptation (LoRA) of the foundation model for surgical depth estimation. METHODS: We design a foundation model-based depth estimation method, referred to as Surgical-DINO, a low-rank adaptation of the DINOv2 for depth estimation in endoscopic surgery. We build LoRA layers and integrate them into DINO to adapt with surgery-specific domain knowledge instead of conventional fine-tuning. During training, we freeze the DINO image encoder, which shows excellent visual representation capacity, and only optimize the LoRA layers and depth decoder to integrate features from the surgical scene. RESULTS: Our model is extensively validated on a MICCAI challenge dataset of SCARED, which is collected from da Vinci Xi endoscope surgery. We empirically show that Surgical-DINO significantly outperforms all the state-of-the-art models in endoscopic depth estimation tasks. The analysis with ablation studies has shown evidence of the remarkable effect of our LoRA layers and adaptation. CONCLUSION: Surgical-DINO shed some light on the successful adaptation of the foundation models into the surgical domain for depth estimation. There is clear evidence in the results that zero-shot prediction on pre-trained weights in computer vision datasets or naive fine-tuning is not sufficient to use the foundation model in the surgical domain directly.

Privacy-Preserving Synthetic Continual Semantic Segmentation for Robotic Surgery.

Deep Neural Networks (DNNs) based semantic segmentation of the robotic instruments and tissues can enhance the precision of surgical activities in robot-assisted surgery. However, in biological learning, DNNs cannot learn incremental tasks over time and exhibit catastrophic forgetting, which refers to the sharp decline in performance on previously learned tasks after learning a new one. Specifically, when data scarcity is the issue, the model shows a rapid drop in performance on previously learned instruments after learning new data with new instruments. The problem becomes worse when it limits releasing the dataset of the old instruments for the old model due to privacy concerns and the unavailability of the data for the new or updated version of the instruments for the continual learning model. For this purpose, we develop a privacy-preserving synthetic continual semantic segmentation framework by blending and harmonizing (i) open-source old instruments foreground to the synthesized background without revealing real patient data in public and (ii) new instruments foreground to extensively augmented real background. To boost the balanced logit distillation from the old model to the continual learning model, we design overlapping class-aware temperature normalization (CAT) by controlling model learning utility. We also introduce multi-scale shifted-feature distillation (SD) to maintain long and short-range spatial relationships among the semantic objects where conventional short-range spatial features with limited information reduce the power of feature distillation. We demonstrate the effectiveness of our framework on the EndoVis 2017 and 2018 instrument segmentation dataset with a generalized continual learning setting. Code is available at https://github.com/XuMengyaAmy/Synthetic_CAT_SD.

Lightweight Pneumatically Elastic Backbone Structure with Modular Construction and Nonlinear Interaction for Soft Actuators.

There has been a growing need for soft robots operating various force-sensitive tasks due to their environmental adaptability, satisfactory controllability, and nonlinear mobility unique from rigid robots. It is of desire to further study the system instability and strongly nonlinear interaction phenomenon that are the main influence factors to the actuations of lightweight soft actuators. In this study, we present a design principle on lightweight pneumatically elastic backbone structure (PEBS) with the modular construction for soft actuators, which contains a backbone printed as one piece and a common strip balloon. We build a prototype of a lightweight (<80 g) soft actuator, which can perform bending motions with satisfactory output forces (∼20 times self-weight). Experiments are conducted on the bending effects generated by interactions between the hyperelastic inner balloon and the elastic backbone. We investigated the nonlinear interaction and system instability experimentally, numerically, and parametrically. To overcome them, we further derived a theoretical nonlinear model and a numerical model. Satisfactory agreements are obtained between the numerical, theoretical, and experimental results. The accuracy of the numerical model is fully validated. Parametric studies are conducted on the backbone geometry and stiffness, balloon stiffness, thickness, and diameter. The accurate controllability, operation safety, modularization ability, and collaborative ability of the PEBS are validated by designing PEBS into a soft laryngoscope, a modularized PEBS library for a robotic arm, and a PEBS system that can operate remote surgery. The reported work provides a further applicability potential of soft robotics studies.

Novel miniature transendoscopic telerobotic system for endoscopic submucosal dissection (with videos).

BACKGROUND AND AIMS: The lack of tissue traction and instrument dexterity to allow for adequate visualization and effective dissection were the main issues in performing endoscopic submucosal dissection (ESD). Robot-assisted systems may provide advantages. In this study we developed a novel transendoscopic telerobotic system and evaluated its performance in ESD. METHODS: A miniature dual-arm robotic endoscopic assistant for minimally invasive surgery (DREAMS) was developed. The DREAMS system contained the current smallest robotic ESD instruments and was compatible with the commercially available dual-channel endoscope. After the system was established, a prospective randomized controlled study was conducted to validate the performance of the DREAMS-assisted ESD in terms of efficacy, safety, and workload by comparing it with the conventional technique. RESULTS: Two robotic instruments can achieve safe collaboration and provide sufficient visualization and efficient dissection during ESD. Forty ESDs in the stomach and esophagus of 8 pigs were completed by DREAMS-assisted ESD or conventional ESD. Submucosal dissection time was comparable between the 2 techniques, but DREAMS-assisted ESD demonstrated a significantly lower muscular injury rate (15% vs 50%, P = .018) and workload scores (22.30 vs 32.45, P < .001). In the subgroup analysis of esophageal ESD, DREAMS-assisted ESD showed significantly improved submucosal dissection time (6.45 vs 16.37 minutes, P = .002), muscular injury rate (25% vs 87.5%, P = .041), and workload (21.13 vs 40.63, P = .001). CONCLUSIONS: We developed a novel transendoscopic telerobotic system, named DREAMS. The safety profile and technical feasibility of ESD were significantly improved with the assistance of the DREAMS system, especially in the narrower esophageal lumen.

Rethinking pain communication of patients with Alzheimer's disease through E-textile interaction design.

Older individuals are easily prone to chronic pain. Due to the complexity of chronic pain, most elderly often have difficulty expressing pain to others to seek assistance, especially those with Alzheimer's disease (AD). The caregivers cannot instantly discover the patients' pain condition and provide timely pain management. This project applies physiological signal sensing technology to help AD patients express the presence of pain non-verbally. We embed sensors on patients' handkerchiefs to identify the patient's abnormal physical activity when pain occurs. Next, we translate the physiological signal into qualitative light alert to send to caregivers and indicate the pain occurrence condition. Then, utilizing multi-sensory stimulation intervention, we create an electronic textile (e-textile) tool to help caregivers effectively support patients in pain. And thus to create a two-way pain communication between caregivers and the patients. Pain perception can be independent of subjective expressions and tangibly perceived by others through our textile prototype. The e-textile handkerchiefs also bring up a new guide to facilitate communication for caregivers when their patients. We contribute the design insights of building a bio-sensing and e-textile system with considering the pain communication needs, patients' pain behaviors and preference of objects. Our e-textile system may contribute to pain communication bio-sensing tool design for special elderly groups, especially those with weakened cognition and communication abilities. We provide a new approach to dealing with the pain of AD patients for healthcare professionals.

Data augmentation using continuous conditional generative adversarial networks for regression and its application to improved spectral sensing.

Machine learning-assisted spectroscopy analysis faces a prominent constraint in the form of insufficient spectral samples, which hinders its effectiveness. Meanwhile, there is a lack of effective algorithms to simulate synthetic spectra from limited samples of real spectra for regression models in continuous scenarios. In this study, we introduced a continuous conditional generative adversarial network (CcGAN) to autonomously generate synthetic spectra. The labels employed for generating the spectral data can be arbitrarily selected from within the range of labels associated with the real spectral data. Our approach effectively produced spectra using a small spectral dataset obtained from a self-interference microring resonator (SIMRR)-based sensor. The generated synthetic spectra were subjected to evaluation using principal component analysis, revealing an inability to discern them from the real spectra. Finally, to enhance the DNN regression model, these synthetic spectra are incorporated into the original training dataset as an augmentation technique. The results demonstrate that the synthetic spectra generated by CcGAN exhibit exceptional quality and significantly enhance the predictive performance of the DNN model. In conclusion, CcGAN exhibits promising potential in generating high-quality synthetic spectra and delivers a superior data augmentation effect for regression tasks.

TransFSM: Fetal Anatomy Segmentation and Biometric Measurement in Ultrasound Images Using a Hybrid Transformer.

Biometric parameter measurements are powerful tools for evaluating a fetus's gestational age, growth pattern, and abnormalities in a 2D ultrasound. However, it is still challenging to measure fetal biometric parameters automatically due to the indiscriminate confusing factors, limited foreground-background contrast, variety of fetal anatomy shapes at different gestational ages, and blurry anatomical boundaries in ultrasound images. The performance of a standard CNN architecture is limited for these tasks due to the restricted receptive field. We propose a novel hybrid Transformer framework, TransFSM, to address fetal multi-anatomy segmentation and biometric measurement tasks. Unlike the vanilla Transformer based on a single-scale input, TransFSM has a deformable self-attention mechanism so it can effectively process multi-scale information to segment fetal anatomy with irregular shapes and different sizes. We devised a BAD to capture more intrinsic local details using boundary-wise prior knowledge, which compensates for the defects of the Transformer in extracting local features. In addition, a Transformer auxiliary segment head is designed to improve mask prediction by learning the semantic correspondence of the same pixel categories and feature discriminability among different pixel categories. Extensive experiments were conducted on clinical cases and benchmark datasets for anatomy segmentation and biometric measurement tasks. The experiment results indicate that our method achieves state-of-the-art performance in seven evaluation metrics compared with CNN-based, Transformer-based, and hybrid approaches. By Knowledge distillation, the proposed TransFSM can create a more compact and efficient model with high deploying potential in resource-constrained scenarios. Our study serves as a unified framework for biometric estimation across multiple anatomical regions to monitor fetal growth in clinical practice.

Two-stage contextual transformer-based convolutional neural network for airway extraction from CT images.

Accurate airway segmentation from computed tomography (CT) images is critical for planning navigation bronchoscopy and realizing a quantitative assessment of airway-related chronic obstructive pulmonary disease (COPD). Existing methods face difficulty in airway segmentation, particularly for the small branches of the airway. These difficulties arise due to the constraints of limited labeling and failure to meet clinical use requirements in COPD. We propose a two-stage framework with a novel 3D contextual transformer for segmenting the overall airway and small airway branches using CT images. The method consists of two training stages sharing the same modified 3D U-Net network. The novel 3D contextual transformer block is integrated into both the encoder and decoder path of the network to effectively capture contextual and long-range information. In the first training stage, the proposed network segments the overall airway with the overall airway mask. To improve the performance of the segmentation result, we generate the intrapulmonary airway branch label, and train the network to focus on producing small airway branches in the second training stage. Extensive experiments were performed on in-house and multiple public datasets. Quantitative and qualitative analyses demonstrate that our proposed method extracts significantly more branches and longer lengths of the airway tree while accomplishing state-of-the-art airway segmentation performance. The code is available at https://github.com/zhaozsq/airway_segmentation.

Three-Dimensional Collision Avoidance Method for Robot-Assisted Minimally Invasive Surgery.

In the robot-assisted minimally invasive surgery, if a collision occurs, the robot system program could be damaged, and normal tissues could be injured. To avoid collisions during surgery, a 3-dimensional collision avoidance method is proposed in this paper. The proposed method is predicated on the design of 3 strategic vectors: the collision-with-instrument-avoidance (CI) vector, the collision-with-tissues-avoidance (CT) vector, and the constrained-control (CC) vector. The CI vector demarcates 3 specific directions to forestall collision among the surgical instruments. The CT vector, on the other hand, comprises 2 components tailored to prevent inadvertent contact between the robot-controlled instrument and nontarget tissues. Meanwhile, the CC vector is introduced to guide the endpoint of the robot-controlled instrument toward the desired position, ensuring precision in its movements, in alignment with the surgical goals. Simulation results verify the proposed collision avoidance method for robot-assisted minimally invasive surgery. The code and data are available at https://github.com/cynerelee/collision-avoidance.

Rethinking exemplars for continual semantic segmentation in endoscopy scenes: Entropy-based mini-batch pseudo-replay.

Endoscopy is a widely used technique for the early detection of diseases or robotic-assisted minimally invasive surgery (RMIS). Numerous deep learning (DL)-based research works have been developed for automated diagnosis or processing of endoscopic view. However, existing DL models may suffer from catastrophic forgetting. When new target classes are introduced over time or cross institutions, the performance of old classes may suffer severe degradation. More seriously, data privacy and storage issues may lead to the unavailability of old data when updating the model. Therefore, it is necessary to develop a continual learning (CL) methodology to solve the problem of catastrophic forgetting in endoscopic image segmentation. To tackle this, we propose a Endoscopy Continual Semantic Segmentation (EndoCSS) framework that does not involve the storage and privacy issues of exemplar data. The framework includes a mini-batch pseudo-replay (MB-PR) mechanism and a self-adaptive noisy cross-entropy (SAN-CE) loss. The MB-PR strategy circumvents privacy and storage issues by generating pseudo-replay images through a generative model. Meanwhile, the MB-PR strategy can also correct the model deviation to the replay data and current training data, which is aroused by the significant difference in the amount of current and replay images. Therefore, the model can perform effective representation learning on both new and old tasks. SAN-CE loss can help model fitting by adjusting the model's output logits, and also improve the robustness of training. Extensive continual semantic segmentation (CSS) experiments on public datasets demonstrate that our method can robustly and effectively address the catastrophic forgetting brought by class increment in endoscopy scenes. The results show that our framework holds excellent potential for real-world deployment in a streaming learning manner.

Transformer-based 3D U-Net for pulmonary vessel segmentation and artery-vein separation from CT images.

Transformer-based methods have led to the revolutionizing of multiple computer vision tasks. Inspired by this, we propose a transformer-based network with a channel-enhanced attention module to explore contextual and spatial information in non-contrast (NC) and contrast-enhanced (CE) computed tomography (CT) images for pulmonary vessel segmentation and artery-vein separation. Our proposed network employs a 3D contextual transformer module in the encoder and decoder part and a double attention module in skip connection to effectively finish high-quality vessel and artery-vein segmentation. Extensive experiments are conducted on the in-house dataset and the ISICDM2021 challenge dataset. The in-house dataset includes 56 NC CT scans with vessel annotations and the challenge dataset consists of 14 NC and 14 CE CT scans with vessel and artery-vein annotations. For vessel segmentation, Dice is 0.840 for CE CT and 0.867 for NC CT. For artery-vein separation, the proposed method achieves a Dice of 0.758 of CE images and 0.602 of NC images. Quantitative and qualitative results demonstrated that the proposed method achieved high accuracy for pulmonary vessel segmentation and artery-vein separation. It provides useful support for further research associated with the vascular system in CT images. The code is available at https://github.com/wuyanan513/Pulmonary-Vessel-Segmentation-and-Artery-vein-Separation .

Domain adaptive Sim-to-Real segmentation of oropharyngeal organs.

Video-assisted transoral tracheal intubation (TI) necessitates using an endoscope that helps the physician insert a tracheal tube into the glottis instead of the esophagus. The growing trend of robotic-assisted TI would require a medical robot to distinguish anatomical features like an experienced physician which can be imitated by utilizing supervised deep-learning techniques. However, the real datasets of oropharyngeal organs are often inaccessible due to limited open-source data and patient privacy. In this work, we propose a domain adaptive Sim-to-Real framework called IoU-Ranking Blend-ArtFlow (IRB-AF) for image segmentation of oropharyngeal organs. The framework includes an image blending strategy called IoU-Ranking Blend (IRB) and style-transfer method ArtFlow. Here, IRB alleviates the problem of poor segmentation performance caused by significant datasets domain differences, while ArtFlow is introduced to reduce the discrepancies between datasets further. A virtual oropharynx image dataset generated by the SOFA framework is used as the learning subject for semantic segmentation to deal with the limited availability of actual endoscopic images. We adapted IRB-AF with the state-of-the-art domain adaptive segmentation models. The results demonstrate the superior performance of our approach in further improving the segmentation accuracy and training stability.

Sample Generation Method Based on Variational Modal Decomposition and Generative Adversarial Network (VMD-GAN) for Chemical Oxygen Demand (COD) Detection Using Ultraviolet Visible Spectroscopy.

Ultraviolet visible spectroscopy can realize the detection of chemical oxygen demand (COD), especially for low concentration levels due to its high sensitivity, but the issue of insufficient real water sample data has always been a challenge owing to the low probability of occurrence of actual water pollution events. However, in existing methods, generated absorption spectra do not conform to actual situations as the former neglect the actual spectral characteristics. On the other hand, the diversity and complexity are restricted because the information in one-dimensional data is not enough for direct spectral generation. This study proposed a spectral sample generation method based on the variational modal decomposition and generative adversarial network (VMD-GAN). First, the VMD algorithm was utilized to separate principal components and residuals of absorption spectra. Among them, the GAN was used to generate new principal components to ensure that the major spectral characteristics of actual water samples are not lost. The corresponding residuals were then obtained by adjusting the parameters of a three-order Gaussian fitting function, which is more beneficial than the direct use of GAN in the aspect of diversity and complexity. Based on the spectral reconstruction with new principal components and residuals, various absorption spectra were generated more coincident with actual situations. Finally, the effectiveness of this method was evaluated by establishing regression models and predicting COD for actual water samples. In all, the insufficient water sample data can be expanded for a better performance in modeling and analysis of water pollution using the proposed method.

Machine learning-assisted high-accuracy and large dynamic range thermometer in high-Q microbubble resonators.

Whispering gallery mode (WGM) resonators provide an important platform for fine measurement thanks to their small size, high sensitivity, and fast response time. Nevertheless, traditional methods focus on tracking single-mode changes for measurement, and a great deal of information from other resonances is ignored and wasted. Here, we demonstrate that the proposed multimode sensing contains more Fisher information than single mode tracking and has great potential to achieve better performance. Based on a microbubble resonator, a temperature detection system has been built to systematically investigate the proposed multimode sensing method. After the multimode spectral signals are collected by the automated experimental setup, a machine learning algorithm is used to predict the unknown temperature by taking full advantage of multiple resonances. The results show the average error of 3.8 × 10-3°C within the range from 25.00°C to 40.00°C by employing a generalized regression neural network (GRNN). In addition, we have also discussed the influence of the consumed data resource on its predicted performance, such as the amount of training data and the case of different temperate ranges between the training and test data. With high accuracy and large dynamic range, this work paves the way for WGM resonator-based intelligent optical sensing.

Federated Semi-Supervised Learning for Medical Image Segmentation via Pseudo-Label Denoising.

Distributed big data and digital healthcare technologies have great potential to promote medical services, but challenges arise when it comes to learning predictive model from diverse and complex e-health datasets. Federated Learning (FL), as a collaborative machine learning technique, aims to address the challenges by learning a joint predictive model across multi-site clients, especially for distributed medical institutions or hospitals. However, most existing FL methods assume that clients possess fully labeled data for training, which is often not the case in e-health datasets due to high labeling costs or expertise requirement. Therefore, this work proposes a novel and feasible approach to learn a Federated Semi-Supervised Learning (FSSL) model from distributed medical image domains, where a federated pseudo-labeling strategy for unlabeled clients is developed based on the embedded knowledge learned from labeled clients. This greatly mitigates the annotation deficiency at unlabeled clients and leads to a cost-effective and efficient medical image analysis tool. We demonstrated the effectiveness of our method by achieving significant improvements compared to the state-of-the-art in both fundus image and prostate MRI segmentation tasks, resulting in the highest Dice scores of 89.23% and 91.95% respectively even with only a few labeled clients participating in model training. This reveals the superiority of our method for practical deployment, ultimately facilitating the wider use of FL in healthcare and leading to better patient outcomes.

CholecTriplet2021: A benchmark challenge for surgical action triplet recognition.

Context-aware decision support in the operating room can foster surgical safety and efficiency by leveraging real-time feedback from surgical workflow analysis. Most existing works recognize surgical activities at a coarse-grained level, such as phases, steps or events, leaving out fine-grained interaction details about the surgical activity; yet those are needed for more helpful AI assistance in the operating room. Recognizing surgical actions as triplets of combination delivers more comprehensive details about the activities taking place in surgical videos. This paper presents CholecTriplet2021: an endoscopic vision challenge organized at MICCAI 2021 for the recognition of surgical action triplets in laparoscopic videos. The challenge granted private access to the large-scale CholecT50 dataset, which is annotated with action triplet information. In this paper, we present the challenge setup and the assessment of the state-of-the-art deep learning methods proposed by the participants during the challenge. A total of 4 baseline methods from the challenge organizers and 19 new deep learning algorithms from the competing teams are presented to recognize surgical action triplets directly from surgical videos, achieving mean average precision (mAP) ranging from 4.2% to 38.1%. This study also analyzes the significance of the results obtained by the presented approaches, performs a thorough methodological comparison between them, in-depth result analysis, and proposes a novel ensemble method for enhanced recognition. Our analysis shows that surgical workflow analysis is not yet solved, and also highlights interesting directions for future research on fine-grained surgical activity recognition which is of utmost importance for the development of AI in surgery.