Dense surface reconstruction using a learning-based monocular vSLAM model for laparoscopic surgery.

Augmented reality (AR) has seen increased interest and attention for its application in surgical procedures. AR-guided surgical systems can overlay segmented anatomy from pre-operative imaging onto the user's environment to delineate hard-to-see structures and subsurface lesions intraoperatively. While previous works have utilized pre-operative imaging such as computed tomography or magnetic resonance images, registration methods still lack the ability to accurately register deformable anatomical structures without fiducial markers across modalities and dimensionalities. This is especially true of minimally invasive abdominal surgical techniques, which often employ a monocular laparoscope, due to inherent limitations. Surgical scene reconstruction is a critical component towards accurate registrations needed for AR-guided surgery and other downstream AR applications such as remote assistance or surgical simulation. In this work, we utilize a state-of-the-art (SOTA) deep-learning-based visual simultaneous localization and mapping (vSLAM) algorithm to generate a dense 3D reconstruction with camera pose estimations and depth maps from video obtained with a monocular laparoscope. The proposed method can robustly reconstruct surgical scenes using real-time data and provide camera pose estimations without stereo or additional sensors, which increases its usability and is less intrusive. We also demonstrate a framework to evaluate current vSLAM algorithms on non-Lambertian, low-texture surfaces and explore using its outputs on downstream tasks. We expect these evaluation methods can be utilized for the continual refinement of newer algorithms for AR-guided surgery.

Augmented reality improving intraoperative navigation in minimally invasive liver surgery: an interplay between 3D reconstruction and indocyanine green.

Technology have helped surgeons to increase MILS feasibility, so that currently liver surgery evolution is strongly based on technological advances and the same trend is expected even further soon. Aim of the present technical report is to provide insights regarding the possible interplay between 3D reconstructions based on augmented reality and intraoperative navigation by indocyanine green fluorescence. Augmented reality methods based on reconstructions created through artificial intelligence interact synergistically. The better the understanding of the anatomy and characteristics of the lesion, the more accurate the preoperative planning may be scheduled. On the other hand, the better the intraoperative navigation, the more reproducible the preoperative planning becomes.

Reconstructing an ancient fish: Three-dimensional skeletal restoration of the head of Mawsonia (Sarcopterygii, Actinistia) using CT scan, and an adjusted model for body size estimation in fossil coelacanths.

Mawsonia constitutes one of the most conspicuous fossil coelacanth taxa, due to its unique anatomy and possible maximum body size. It typifies Mesozoic coelacanth morphology, before the putative disappearance of the group in the fossil record. In this work, the three-dimensional cranial anatomy and body size estimations of this genus are re-evaluated from a recently described specimen from Upper Jurassic deposits of Uruguay. The 3D restoration was performed directly on the material based on anatomical information provided by the living coelacanth Latimeria and previous two-dimensional restorations of the head of Mawsonia. The montage was then scanned with computed tomography and virtually adjusted to generate an interactive online resource for future anatomical, taxonomic and biomechanical research. In general terms, the model constitutes a tool to improve both the anatomical knowledge of this genus and its comparison with other coelacanths. It also facilitates the evaluation of possible evolutionary trends and the discussion of particular features with potential palaeobiological implications, such as the anterior position of the eye and the development of the pseudomaxillary fold. Regarding the body size, a previous model for body size estimation based on the gular plate was submitted to OLS, RMA, segmented linear and PGLS regressions (including the evaluation of regression statistics, variance analysis, t-tests and residual analysis). The results point to a power relationship between gular and total lengths showing a better support than a simple linear relationship. The new resulting equations were applied to the studied individual and are provided for future estimates. Although an isometric evolutionary growth cannot be rejected with the available evidence, additional models developed with other bones will be necessary to evaluate possible hidden evolutionary allometric trends in this group of fishes, thus avoiding overestimates.

BounTI (boundary-preserving threshold iteration): A user-friendly tool for automatic hard tissue segmentation.

X-ray Computed Tomography (CT) images are widely used in various fields of natural, physical, and biological sciences. 3D reconstruction of the images involves segmentation of the structures of interest. Manual segmentation has been widely used in the field of biological sciences for complex structures composed of several sub-parts and can be a time-consuming process. Many tools have been developed to automate the segmentation process, all with various limitations and advantages, however, multipart segmentation remains a largely manual process. The aim of this study was to develop an open-access and user-friendly tool for the automatic segmentation of calcified tissues, specifically focusing on craniofacial bones. Here we describe BounTI, a novel segmentation algorithm which preserves boundaries between separate segments through iterative thresholding. This study outlines the working principles behind this algorithm, investigates the effect of several input parameters on its outcome, and then tests its versatility on CT images of the craniofacial system from different species (e.g. a snake, a lizard, an amphibian, a mouse and a human skull) with various scan qualities. The case studies demonstrate that this algorithm can be effectively used to segment the craniofacial system of a range of species automatically. High-resolution microCT images resulted in more accurate boundary-preserved segmentation, nonetheless significantly lower-quality clinical images could still be segmented using the proposed algorithm. Methods for manual intervention are included in this tool when the scan quality is insufficient to achieve the desired segmentation results. While the focus here was on the craniofacial system, BounTI can be used to automatically segment any hard tissue. The tool presented here is available as an Avizo/Amira add-on, a stand-alone Windows executable, and a Python library. We believe this accessible and user-friendly segmentation tool can benefit the wider anatomical community.

Segmental Mandibulectomy and Mandibular Reconstruction with Fibula-Free Flap Using a 3D Template.

INTRODUCTION: The present study evaluates the influence of virtual surgical planning with a preoperative 3D resin model on aesthetic and functional outcomes in patients treated by segmental mandibulectomy and reconstruction with fibula-free flap for oral cancer. METHODS: All consecutive patients who underwent segmental mandibulectomy and mandibular reconstruction with a fibula-free flap using a 3D template at our department from January 2021 to January 2023 were included in the study. "Patients control" were patients treated by reconstruction with a fibula-free flap without using a 3D template. Three-dimensional modeling was performed by converting from preoperative computed tomography to a stereolithography format to obtain the resin 3D models. Qualitative analysis of anatomical and aesthetic results consisted of the evaluation of the patients' aesthetic and functional satisfaction and the symmetry of the mandibular contour observed at clinical examination. Quantitative analysis was based on the assessment of the accuracy and precision of the reconstruction by comparing preoperative and postoperative computed tomograms as objective indicators. RESULTS: Seven patients (five males and two females, mean age of 65.1 years) were included in the study. All patients showed a symmetric mandibular contour based on the clinical examination. After recovery, six patients (85.7%) considered themselves aesthetically satisfied. The quantitative analysis (assessed in six/seven patients) showed that the mean difference between preoperative and postoperative intercondylar distance, intergonial angle distance, anteroposterior dimension, and gonial angle improved in the 3D template-assisted group. CONCLUSION: The 3D-printed template for mandibular reconstruction with microvascular fibula-free flap can improve aesthetic outcomes in comparison with standard approaches.

Influence of Solute Drag Effect and Interphase Precipitation of Nb on Ferrite Transformation.

The significant impact of Nb on ferrite transformation, both in terms of solute drag effect (SDE) and interphase precipitation, was investigated quantitatively. Ferrite transformation kinetics were characterized using thermal expansion experiments and theoretical calculations. The microstructures were characterized using high-temperature confocal laser scanning microscopy (CLSM), a field-emission scanning electron microscope (FESEM), and a transmission electron microscope (TEM). Under a higher driving force, interphase precipitations were observed in the sample with a higher Nb content. A three-dimensional (3D) reconstruction method was used to convert the two-dimensional (2D) image of interphase precipitation into a three-dimensional model for a more typical view. The SDE and interphase precipitation had opposite effects on the kinetics of ferrite transformation. A lower Nb content showed a strong contribution to the SDE, which delayed ferrite transformation. A higher concentration of Nb was expected to enhance the SDE, but the inhibition effect was eliminated by the interphase precipitation of NbC during interfacial migration. Both the experimental results and theoretical calculations confirmed this phenomenon.

Fiberscopic pattern removal for optimal coverage in 3D bladder reconstructions of fiberscope cystoscopy videos.

Purpose: In the current clinical standard of care, cystoscopic video is not routinely saved because it is cumbersome to review. Instead, clinicians rely on brief procedure notes and still frames to manage bladder pathology. Preserving discarded data via 3D reconstructions, which are convenient to review, has the potential to improve patient care. However, many clinical videos are collected by fiberscopes, which are lower cost but induce a pattern on frames that inhibit 3D reconstruction. The aim of our study is to remove the honeycomb-like pattern present in fiberscope-based cystoscopy videos to improve the quality of 3D bladder reconstructions. Approach: Our study introduces an algorithm that applies a notch filtering mask in the Fourier domain to remove the honeycomb-like pattern from clinical cystoscopy videos collected by fiberscope as a preprocessing step to 3D reconstruction. We produce 3D reconstructions with the video before and after removing the pattern, which we compare with a metric termed the area of reconstruction coverage (ARC), defined as the surface area (in pixels) of the reconstructed bladder. All statistical analyses use paired t-tests. Results: Preprocessing using our method for pattern removal enabled reconstruction for all (n=5) cystoscopy videos included in the study and produced a statistically significant increase in bladder coverage (p=0.018). Conclusions: This algorithm for pattern removal increases bladder coverage in 3D reconstructions and automates mask generation and application, which could aid implementation in time-starved clinical environments. The creation and use of 3D reconstructions can improve documentation of cystoscopic findings for future surgical navigation, thus improving patient treatment and outcomes.

Ablation of Sam50 is associated with fragmentation and alterations in metabolism in murine and human myotubes.

The sorting and assembly machinery (SAM) Complex is responsible for assembling ß-barrel proteins in the mitochondrial membrane. Comprising three subunits, Sam35, Sam37, and Sam50, the SAM complex connects the inner and outer mitochondrial membranes by interacting with the mitochondrial contact site and cristae organizing system complex. Sam50, in particular, stabilizes the mitochondrial intermembrane space bridging (MIB) complex, which is crucial for protein transport, respiratory chain complex assembly, and regulation of cristae integrity. While the role of Sam50 in mitochondrial structure and metabolism in skeletal muscle remains unclear, this study aims to investigate its impact. Serial block-face-scanning electron microscopy and computer-assisted 3D renderings were employed to compare mitochondrial structure and networking in Sam50-deficient myotubes from mice and humans with wild-type (WT) myotubes. Furthermore, autophagosome 3D structure was assessed in human myotubes. Mitochondrial metabolic phenotypes were assessed using Gas Chromatography-Mass Spectrometry-based metabolomics to explore differential changes in WT and Sam50-deficient myotubes. The results revealed increased mitochondrial fragmentation and autophagosome formation in Sam50-deficient myotubes compared to controls. Metabolomic analysis indicated elevated metabolism of propanoate and several amino acids, including ß-Alanine, phenylalanine, and tyrosine, along with increased amino acid and fatty acid metabolism in Sam50-deficient myotubes. Furthermore, impairment of oxidative capacity was observed upon Sam50 ablation in both murine and human myotubes, as measured with the XF24 Seahorse Analyzer. Collectively, these findings support the critical role of Sam50 in establishing and maintaining mitochondrial integrity, cristae structure, and mitochondrial metabolism. By elucidating the impact of Sam50-deficiency, this study enhances our understanding of mitochondrial function in skeletal muscle.

Advances in Real-Time 3D Reconstruction for Medical Endoscopy.

This contribution is intended to provide researchers with a comprehensive overview of the current state-of-the-art concerning real-time 3D reconstruction methods suitable for medical endoscopy. Over the past decade, there have been various technological advancements in computational power and an increased research effort in many computer vision fields such as autonomous driving, robotics, and unmanned aerial vehicles. Some of these advancements can also be adapted to the field of medical endoscopy while coping with challenges such as featureless surfaces, varying lighting conditions, and deformable structures. To provide a comprehensive overview, a logical division of monocular, binocular, trinocular, and multiocular methods is performed and also active and passive methods are distinguished. Within these categories, we consider both flexible and non-flexible endoscopes to cover the state-of-the-art as fully as possible. The relevant error metrics to compare the publications presented here are discussed, and the choice of when to choose a GPU rather than an FPGA for camera-based 3D reconstruction is debated. We elaborate on the good practice of using datasets and provide a direct comparison of the presented work. It is important to note that in addition to medical publications, publications evaluated on the KITTI and Middlebury datasets are also considered to include related methods that may be suited for medical 3D reconstruction.

SimCol3D - 3D reconstruction during colonoscopy challenge.

Colorectal cancer is one of the most common cancers in the world. While colonoscopy is an effective screening technique, navigating an endoscope through the colon to detect polyps is challenging. A 3D map of the observed surfaces could enhance the identification of unscreened colon tissue and serve as a training platform. However, reconstructing the colon from video footage remains difficult. Learning-based approaches hold promise as robust alternatives, but necessitate extensive datasets. Establishing a benchmark dataset, the 2022 EndoVis sub-challenge SimCol3D aimed to facilitate data-driven depth and pose prediction during colonoscopy. The challenge was hosted as part of MICCAI 2022 in Singapore. Six teams from around the world and representatives from academia and industry participated in the three sub-challenges: synthetic depth prediction, synthetic pose prediction, and real pose prediction. This paper describes the challenge, the submitted methods, and their results. We show that depth prediction from synthetic colonoscopy images is robustly solvable, while pose estimation remains an open research question.

A modified position for optimized skeletal maturity assessment of AIS patients and its impact on 3D spinal and pelvic parameters.

PURPOSE: A hands-on-wall (HOW) position for low-dose stereoradiography of adolescent idiopathic scoliosis (AIS) patients would allow for skeletal maturity assessment of the hand and wrist. Our aims were twofold: confirm the reliability and validity of skeletal maturity assessment using the HOW radiographs and compare the spinal and pelvic 3D parameters to those of standard hands-on-cheeks (HOC) stereoradiographs. METHODS: Seventy AIS patients underwent two successive stereoradiographs and a standard hand and wrist radiograph on the same day. Patients were randomly assigned to begin with HOW and follow with HOC, or vice versa. Raters assessed digital skeletal age (DSA), Sanders Simplified Skeletal Maturity (SSMS) and Thumb Ossification Composite Index (TOCI). 3D reconstructions of the spine and pelvis bones were performed for each stereoradiograph to measure nine clinically relevant spinal and pelvic 3D parameters. RESULTS: Inter-rater and intra-rater reliabilities were excellent for DSA, SSMS and TOCI with both standard radiographs and HOW (ICC > 0.95). Strong correlation was found between ratings of both imaging types (ICC > 0.95). In the 3D reconstructions, kyphosis and sacral slope were slightly decreased in the HOW position, but within the clinical margin of error. All other parameters did not differ significantly between positions (p < 0.05). CONCLUSION: The results suggest that HOW stereoradiographs allow clinicians to assess skeletal maturity of the hand and wrist with adequate reliability and validity. We recommend that scoliosis clinics adopt the HOW position to assess skeletal maturity because there is no significant clinical impact on the spinal and pelvic evaluation, and on radiation exposure, cost or time.

Evaluation of women's aging influence on sperm passage inside the fallopian tube using 3D dynamic mechanical modeling.

The fallopian tubes play an important role in human fertility by facilitating the spermatozoa passage to the oocyte as well as later actively facilitating the fertilized oocyte transportation to the uterus cavity. The fallopian tubes undergo changes involving biological, physical, and morphological processes due to women aging, which may impair fertility. Here, we have modelled fallopian tubes of women at different ages and evaluated the chances of normal and pathological sperm cells reaching the fertilization site, the ampulla. By utilizing a unique combination of simulative tools, we implemented dynamic three-dimensional (3D) detailed geometrical models of many normal and pathological sperm cells swimming together in 3D geometrical models of three fallopian tubes associated with different women's age groups. By tracking the sperm cell swim, we found that for all age groups, the number of normal sperm cells in the ampulla is the largest, compared with the pathological sperm cells. On the other hand, the number of normal sperm cells in the fertilization site decreases due to the morphological and mechanical changes that occur in the fallopian tube with age. Moreover, in older ages, the normal sperm cells swim with lower velocities and for shorter distances inside the ampulla toward the ovary. Thus, the changes that the human fallopian tube undergoes due to women's aging have a significant influence on the human sperm cell motility. Our model of sperm cell motility through the fallopian tube in relation to the woman's age morphological changes provides a new scope for the investigation and treatment of diseases and infertility cases associated with aging, as well as a potential personalized medicine tool for evaluating the chances of a natural fertilization per specific features of a man's sperm and a woman's reproductive system.

FusionVision: A Comprehensive Approach of 3D Object Reconstruction and Segmentation from RGB-D Cameras Using YOLO and Fast Segment Anything.

In the realm of computer vision, the integration of advanced techniques into the pre-processing of RGB-D camera inputs poses a significant challenge, given the inherent complexities arising from diverse environmental conditions and varying object appearances. Therefore, this paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. Traditional computer vision systems face limitations in simultaneously capturing precise object boundaries and achieving high-precision object detection on depth maps, as they are mainly proposed for RGB cameras. To address this challenge, FusionVision adopts an integrated approach by merging state-of-the-art object detection techniques, with advanced instance segmentation methods. The integration of these components enables a holistic (unified analysis of information obtained from both color RGB and depth D channels) interpretation of RGB-D data, facilitating the extraction of comprehensive and accurate object information in order to improve post-processes such as object 6D pose estimation, Simultanious Localization and Mapping (SLAM) operations, accurate 3D dataset extraction, etc. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. Subsequently, FastSAM, an innovative semantic segmentation model, is applied to delineate object boundaries, yielding refined segmentation masks. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation, enhancing overall precision in 3D object segmentation.

Enhancing soccer goalkeepers penalty dive kinematics with instructional video and laterality insights in field conditions.

This study aimed to analyze the effect of laterality and instructional video on the soccer goalkeepers' dive kinematics in penalty. Eight goalkeepers from youth categories (U15, U17, U20) were randomly divided into control (CG) and video instruction groups (VG). The latter performed 20 penalty defense trials on the field with balls launched by a machine, ten before and after watching a video instruction to improve the diving kinematics. The CG only performed the dives. Three cameras recorded the collections. A markerless motion capture technique (OpenPose) was used for identification and tracking of joints and anatomical references on video. The pose data were used for 3D reconstruction. In the post-instruction situation, the VG presented differences in comparison to the CG in the: knee flexion/extension angle, time to reach peak resultant velocity, frontal step distance, and frontal departure angle, which generated greater acceleration during the dive. Non-dominant leg side dives had higher resultant velocity during 88.4 - 100% of the diving cycle, different knee flexion/extension angle, and higher values ​​in the frontal step distance. The instructional video generated an acute change in the diving movement pattern of young goalkeepers when comparing the control and the video instruction group in the post condition.

3D reconstruction from endoscopy images: A survey.

Three-dimensional reconstruction of images acquired through endoscopes is playing a vital role in an increasing number of medical applications. Endoscopes used in the clinic are commonly classified as monocular endoscopes and binocular endoscopes. We have reviewed the classification of methods for depth estimation according to the type of endoscope. Basically, depth estimation relies on feature matching of images and multi-view geometry theory. However, these traditional techniques have many problems in the endoscopic environment. With the increasing development of deep learning techniques, there is a growing number of works based on learning methods to address challenges such as inconsistent illumination and texture sparsity. We have reviewed over 170 papers published in the 10 years from 2013 to 2023. The commonly used public datasets and performance metrics are summarized. We also give a taxonomy of methods and analyze the advantages and drawbacks of algorithms. Summary tables and result atlas are listed to facilitate the comparison of qualitative and quantitative performance of different methods in each category. In addition, we summarize commonly used scene representation methods in endoscopy and speculate on the prospects of deep estimation research in medical applications. We also compare the robustness performance, processing time, and scene representation of the methods to facilitate doctors and researchers in selecting appropriate methods based on surgical applications.

3D reconstruction of the mouse cochlea from scRNA-seq data suggests morphogen-based principles in apex-to-base specification.

In the mammalian auditory system, frequency discrimination depends on numerous morphological and physiological properties of the organ of Corti, which gradually change along the apex-to-base (tonotopic) axis of the organ. For example, the basilar membrane stiffness changes tonotopically, thus affecting the tuning properties of individual hair cells. At the molecular level, those frequency-specific characteristics are mirrored by gene expression gradients; however, the molecular mechanisms controlling tonotopic gene expression in the mouse cochlea remain elusive. Through analyzing single-cell RNA sequencing (scRNA-seq) data from E12.5 and E14.5 time points, we predicted that morphogens, rather than a cell division-associated mechanism, confer spatial identity in the extending cochlea. Subsequently, we reconstructed the developing cochlea in 3D space from scRNA-seq data to investigate the molecular pathways mediating positional information. The retinoic acid (RA) and hedgehog pathways were found to form opposing apex-to-base gradients, and functional interrogation using mouse cochlear explants suggested that both pathways jointly specify the longitudinal axis.

A semiparametric Gaussian mixture model for chest CT-based 3D blood vessel reconstruction.

Computed tomography (CT) has been a powerful diagnostic tool since its emergence in the 1970s. Using CT data, 3D structures of human internal organs and tissues, such as blood vessels, can be reconstructed using professional software. This 3D reconstruction is crucial for surgical operations and can serve as a vivid medical teaching example. However, traditional 3D reconstruction heavily relies on manual operations, which are time-consuming, subjective, and require substantial experience. To address this problem, we develop a novel semiparametric Gaussian mixture model tailored for the 3D reconstruction of blood vessels. This model extends the classical Gaussian mixture model by enabling nonparametric variations in the component-wise parameters of interest according to voxel positions. We develop a kernel-based expectation-maximization algorithm for estimating the model parameters, accompanied by a supporting asymptotic theory. Furthermore, we propose a novel regression method for optimal bandwidth selection. Compared to the conventional cross-validation-based (CV) method, the regression method outperforms the CV method in terms of computational and statistical efficiency. In application, this methodology facilitates the fully automated reconstruction of 3D blood vessel structures with remarkable accuracy.

Laryngeal surface reconstructions from monocular endoscopic videos: a structure from motion pipeline for periodic deformations.

PURPOSE: Surface reconstructions from laryngoscopic videos have the potential to assist clinicians in diagnosing, quantifying, and monitoring airway diseases using minimally invasive techniques. However, tissue movements and deformations make these reconstructions challenging using conventional pipelines. METHODS: To facilitate such reconstructions, we developed video frame pre-filtering and featureless dense matching steps to enhance the Alicevision Meshroom SfM pipeline. Time and the anterior glottic angle were used to approximate the rigid state of the airway and to collect frames with different camera poses. Featureless dense matches were tracked with a correspondence transformer across subsets of images to extract matched points that could be used to estimate the point cloud and reconstructed surface. The proposed pipeline was tested on a simulated dataset under various conditions like illumination and resolution as well as real laryngoscopic videos. RESULTS: Our pipeline was able to reconstruct the laryngeal region based on 4, 8, and 16 images obtained from simulated and real patient exams. The pipeline was robust to sparse inputs, blur, and extreme lighting conditions, unlike the Meshroom pipeline which failed to produce a point cloud for 6 of 15 simulated datasets. CONCLUSION: The pre-filtering and featureless dense matching modules specialize the conventional SfM pipeline to handle the challenging laryngoscopic examinations, directly from patient videos. These 3D visualizations have the potential to improve spatial understanding of airway conditions.

Application of a novel deep learning-based 3D videography workflow to bat flight.

Studying the detailed biomechanics of flying animals requires accurate three-dimensional coordinates for key anatomical landmarks. Traditionally, this relies on manually digitizing animal videos, a labor-intensive task that scales poorly with increasing framerates and numbers of cameras. Here, we present a workflow that combines deep learning-powered automatic digitization with filtering and correction of mislabeled points using quality metrics from deep learning and 3D reconstruction. We tested our workflow using a particularly challenging scenario: bat flight. First, we documented four bats flying steadily in a 2 m3 wind tunnel test section. Wing kinematic parameters resulting from manually digitizing bats with markers applied to anatomical landmarks were not significantly different from those resulting from applying our workflow to the same bats without markers for five out of six parameters. Second, we compared coordinates from manual digitization against those yielded via our workflow for bats flying freely in a 344 m3 enclosure. Average distance between coordinates from our workflow and those from manual digitization was less than a millimeter larger than the average human-to-human coordinate distance. The improved efficiency of our workflow has the potential to increase the scalability of studies on animal flight biomechanics.