An analysis on the effect of body tissues and surgical tools on workflow recognition in first person surgical videos.

PURPOSE: Analysis of operative fields is expected to aid in estimating procedural workflow and evaluating surgeons' procedural skills by considering the temporal transitions during the progression of the surgery. This study aims to propose an automatic recognition system for the procedural workflow by employing machine learning techniques to identify and distinguish elements in the operative field, including body tissues such as fat, muscle, and dermis, along with surgical tools. METHODS: We conducted annotations on approximately 908 first-person-view images of breast surgery to facilitate segmentation. The annotated images were used to train a pixel-level classifier based on Mask R-CNN. To assess the impact on procedural workflow recognition, we annotated an additional 43,007 images. The network, structured on the Transformer architecture, was then trained with surgical images incorporating masks for body tissues and surgical tools. RESULTS: The instance segmentation of each body tissue in the segmentation phase provided insights into the trend of area transitions for each tissue. Simultaneously, the spatial features of the surgical tools were effectively captured. In regard to the accuracy of procedural workflow recognition, accounting for body tissues led to an average improvement of 3 % over the baseline. Furthermore, the inclusion of surgical tools yielded an additional increase in accuracy by 4 % compared to the baseline. CONCLUSION: In this study, we revealed the contribution of the temporal transition of the body tissues and surgical tools spatial features to recognize procedural workflow in first-person-view surgical videos. Body tissues, especially in open surgery, can be a crucial element. This study suggests that further improvements can be achieved by accurately identifying surgical tools specific to each procedural workflow step.

Two-stage video-based convolutional neural networks for adult spinal deformity classification.

Introduction: Assessment of human gait posture can be clinically effective in diagnosing human gait deformities early in life. Currently, two methods-static and dynamic-are used to diagnose adult spinal deformity (ASD) and other spinal disorders. Full-spine lateral standing radiographs are used in the standard static method. However, this is a static assessment of joints in the standing position and does not include information on joint changes when the patient walks. Careful observation of long-distance walking can provide a dynamic assessment that reveals an uncompensated posture; however, this increases the workload of medical practitioners. A three-dimensional (3D) motion system is proposed for the dynamic method. Although the motion system successfully detected dynamic posture changes, access to the facilities was limited. Therefore, a diagnostic approach that is facility-independent, has low practice flow, and does not involve patient contact is required. Methods: We focused on a video-based method to classify patients with spinal disorders either as ASD, or other forms of ASD. To achieve this goal, we present a video-based two-stage machine-learning method. In the first stage, deep learning methods are used to locate the patient and extract the area where the patient is located. In the second stage, a 3D CNN (convolutional neural network) device is used to capture spatial and temporal information (dynamic motion) from the extracted frames. Disease classification is performed by discerning posture and gait from the extracted frames. Model performance was assessed using the mean accuracy, F1 score, and area under the receiver operating characteristic curve (AUROC), with five-fold cross-validation. We also compared the final results with professional observations. Results: Our experiments were conducted using a gait video dataset comprising 81 patients. The experimental results indicated that our method is effective for classifying ASD and other spinal disorders. The proposed method achieved a mean accuracy of 0.7553, an F1 score of 0.7063, and an AUROC score of 0.7864. Additionally, ablation experiments indicated the importance of the first stage (detection stage) and transfer learning of our proposed method. Discussion: The observations from the two doctors were compared using the proposed method. The mean accuracies observed by the two doctors were 0.4815 and 0.5247, with AUROC scores of 0.5185 and 0.5463, respectively. We proved that the proposed method can achieve accurate and reliable medical testing results compared with doctors' observations using videos of 1 s duration. All our code, models, and results are available at https://github.com/ChenKaiXuSan/Walk_Video_PyTorch. The proposed framework provides a potential video-based method for improving the clinical diagnosis for ASD and non-ASD. This framework might, in turn, benefit both patients and clinicians to treat the disease quickly and directly and further reduce facility dependency and data-driven systems.

Integration of Independent Heat Transfer Mechanisms for Non-Contact Cold Sensation Presentation With Low Residual Heat.

Thermal sensation is crucial to enhancing our comprehension of the world and enhancing our ability to interact with it. Therefore, the development of thermal sensation presentation technologies holds significant potential, providing a novel method of interaction. Traditional technologies often leave residual heat in the system or the skin, affecting subsequent presentations. Our study focuses on presenting thermal sensations with low residual heat, especially cold sensations. To mitigate the impact of residual heat in the presentation system, we opted for a non-contact method, and to address the influence of residual heat on the skin, we present thermal sensations without significantly altering skin temperature. Specifically, we integrated two highly responsive and independent heat transfer mechanisms: convection via cold air and radiation via visible light, providing non-contact thermal stimuli. By rapidly alternating between perceptible decreases and imperceptible increases in temperature on the same skin area, we maintained near-constant skin temperature while presenting continuous cold sensations. In our experiments involving 15 participants, we observed that when the cooling rate was -0.2 to -0.24 °C/s and the cooling time ratio was 30 to 50%, more than 86.67% of the participants perceived only persistent cold without any warmth.

Development and Verification of Postural Control Assessment Using Deep-Learning-Based Pose Estimators: Towards Clinical Applications.

Occupational therapists evaluate various aspects of a client's occupational performance. Among these, postural control is one of the fundamental skills that need assessment. Recently, several methods have been proposed to estimate postural control abilities using deep-learning-based approaches. Such techniques allow for the potential to provide automated, precise, fine-grained quantitative indices simply by evaluating videos of a client engaging in a postural control task. However, the clinical applicability of these assessment tools requires further investigation. In the current study, we compared three deep-learning-based pose estimators to assess their clinical applicability in terms of accuracy of pose estimations and processing speed. In addition, we verified which of the proposed quantitative indices for postural controls best reflected the clinical evaluations of occupational therapists. A framework using deep-learning techniques broadens the possibility of quantifying clients' postural control in a more fine-grained way compared with conventional coarse indices, which can lead to improved occupational therapy practice.

Intensity-Adjustable Non-Contact Cold Sensation Presentation Based on the Vortex Effect.

Cold sensations of varying intensities are perceived when human skin is subject to diverse environments. The accurate presentation of temperature changes is important to elicit immersive sensations in applications such as virtual reality. In this article, we developed a method to elicit intensity-adjustable non-contact cold sensations based on the vortex effect. We applied this effect to generate cold air at approximately 0 °C and varied the skin temperature over a wide range. The perception of different temperatures can be elicited by adjusting the volume flow rate of the cold air. Additionally, we introduced a cooling model to relate the changes in skin temperature to various parameters such as the cold air volume flow rate and distance from the cold air outlet to the skin. For validation, we conducted measurement experiments and found that our model can estimate the change in skin temperature with a root mean-square error of 0.16 °C. Furthermore, we evaluated the performance of a prototype in psychophysical cold discrimination experiments based on the discrimination threshold. Thus, cold sensations of varying intensities can be generated by varying the parameters. These cold sensations can be combined with images, sounds, and other stimuli to create an immersive and realistic artificial environment.

Vision-based egg quality prediction in Pacific bluefin tuna (Thunnus orientalis) by deep neural network.

Closed-cycle aquaculture using hatchery produced seed stocks is vital to the sustainability of endangered species such as Pacific bluefin tuna (Thunnus orientalis) because this aquaculture system does not depend on aquaculture seeds collected from the wild. High egg quality promotes efficient aquaculture production by improving hatch rates and subsequent growth and survival of hatched larvae. In this study, we investigate the possibility of a simple, low-cost, and accurate egg quality prediction system based only on photographic images using deep neural networks. We photographed individual eggs immediately after spawning and assessed their qualities, i.e., whether they hatched normally and how many days larvae survived without feeding. The proposed system predicted normally hatching eggs with higher accuracy than human experts. It was also successful in predicting which eggs would produce longer-surviving larvae. We also analyzed the image aspects that contributed to the prediction to discover important egg features. Our results suggest the applicability of deep learning techniques to efficient egg quality prediction, and analysis of early developmental stages of development.

Looking represents choosing in toddlers: Exploring the equivalence between multimodal measures in forced-choice tasks.

In the two-alternative forced-choice (2AFC) paradigm, manual responses such as pointing have been widely used as measures to estimate cognitive abilities. While pointing measurements can be easily collected, coded, analyzed, and interpreted, absent responses are often observed particularly when adopting these measures for toddler studies, which leads to an increase of missing data. Although looking responses such as preferential looking can be available as alternative measures in such cases, it is unknown how well looking measurements can be interpreted as equivalent to manual ones. This study aimed to answer this question by investigating how accurately pointing responses (i.e., left or right) could be predicted from concurrent preferential looking. Using pre-existing videos of toddlers aged 18-23 months engaged in an intermodal word comprehension task, we developed models predicting manual from looking responses. Results showed substantial prediction accuracy for both the Simple Majority Vote and Machine Learning-Based classifiers, which indicates that looking responses would be reasonable alternative measures of manual ones. However, the further exploratory analysis revealed that when applying the created models for data of toddlers who did not produce clear pointing responses, the estimation agreement of missing pointing between the models and the human coders slightly dropped. This indicates that looking responses without pointing were qualitatively different from those with pointing. Bridging two measurements in forced-choice tasks would help researchers avoid wasting collected data due to the absence of manual responses and interpret results from different modalities comprehensively.

Development and User Evaluation of a Smartphone-Based System to Assess Range of Motion of Wrist Joint.

PURPOSE: Measuring range of motion (ROM) in the wrist joint is an essential part of hand and wrist functional evaluations, especially before and after surgery. However, accurate measurements require experience and time. To reduce patient and surgeon burdens related to ROM measurement, a smartphone-based system, which enables participants to measure the ROM of the wrist joint semiautomatically using self-taken pictures on a smartphone, was developed and evaluated in this study. METHODS: In the developed system, participants were asked to take a picture of their wrist by using the other hand to position the joint first into full flexion and then into full extension. The hand and arm regions were automatically extracted in the program, and the ROM was estimated after the area of the hand and forearm was cropped. To verify the accuracy of ROM measurements in this system, the proposed method was tested on 66 images of hands from 33 participants; measurements were compared with those taken by hand surgeons. A limit of agreement and an intraclass correlation coefficient (ICC) were used for evaluation. RESULTS: The smallest averages (95% limits of agreement) of flexion and extension were 11.32° (95% confidence interval [CI], 8.88° to 13.76°) and 11.01° (95% CI, 8.64° to 13.39°), respectively. The ICC (1,1) for 3 measurements taken by one assessor was 0.99 (95% CI, 0.986-0.992), and the ICC (2,1) for 2 measurements taken by both assessors was 0.97 (95% CI, 0.947-0.977). CONCLUSIONS: In this study, we developed a system to measure the semiautomatic ROM of the wrist joint using a smartphone image. Its accuracy was within a clinically usable error range that was comparable with that of a hand surgeon. CLINICAL RELEVANCE: This system can reduce the burden of ROM measurement for both patients and doctors.

Computer Vision-Based Approach for Quantifying Occupational Therapists' Qualitative Evaluations of Postural Control.

This study aimed to leverage computer vision (CV) technology to develop a technique for quantifying postural control. A conventional quantitative index, occupational therapists' qualitative clinical evaluations, and CV-based quantitative indices using an image analysis algorithm were applied to evaluate the postural control of 34 typically developed preschoolers. The effectiveness of the CV-based indices was investigated relative to current methods to explore the clinical applicability of the proposed method. The capacity of the CV-based indices to reflect therapists' qualitative evaluations was confirmed. Furthermore, compared to the conventional quantitative index, the CV-based indices provided more detailed quantitative information with lower costs. CV-based evaluations enable therapists to quantify details of motor performance that are currently observed qualitatively. The development of such precise quantification methods will improve the science and practice of occupational therapy and allow therapists to perform to their full potential.