OpenCap: Human movement dynamics from smartphone videos.

Measures of human movement dynamics can predict outcomes like injury risk or musculoskeletal disease progression. However, these measures are rarely quantified in large-scale research studies or clinical practice due to the prohibitive cost, time, and expertise required. Here we present and validate OpenCap, an open-source platform for computing both the kinematics (i.e., motion) and dynamics (i.e., forces) of human movement using videos captured from two or more smartphones. OpenCap leverages pose estimation algorithms to identify body landmarks from videos; deep learning and biomechanical models to estimate three-dimensional kinematics; and physics-based simulations to estimate muscle activations and musculoskeletal dynamics. OpenCap's web application enables users to collect synchronous videos and visualize movement data that is automatically processed in the cloud, thereby eliminating the need for specialized hardware, software, and expertise. We show that OpenCap accurately predicts dynamic measures, like muscle activations, joint loads, and joint moments, which can be used to screen for disease risk, evaluate intervention efficacy, assess between-group movement differences, and inform rehabilitation decisions. Additionally, we demonstrate OpenCap's practical utility through a 100-subject field study, where a clinician using OpenCap estimated musculoskeletal dynamics 25 times faster than a laboratory-based approach at less than 1% of the cost. By democratizing access to human movement analysis, OpenCap can accelerate the incorporation of biomechanical metrics into large-scale research studies, clinical trials, and clinical practice.

Assessing inertial measurement unit locations for freezing of gait detection and patient preference.

BACKGROUND: Freezing of gait, a common symptom of Parkinson's disease, presents as sporadic episodes in which an individual's feet suddenly feel stuck to the ground. Inertial measurement units (IMUs) promise to enable at-home monitoring and personalization of therapy, but there is a lack of consensus on the number and location of IMUs for detecting freezing of gait. The purpose of this study was to assess IMU sets in the context of both freezing of gait detection performance and patient preference. METHODS: Sixteen people with Parkinson's disease were surveyed about sensor preferences. Raw IMU data from seven people with Parkinson's disease, wearing up to eleven sensors, were used to train convolutional neural networks to detect freezing of gait. Models trained with data from different sensor sets were assessed for technical performance; a best technical set and minimal IMU set were identified. Clinical utility was assessed by comparing model- and human-rater-determined percent time freezing and number of freezing events. RESULTS: The best technical set consisted of three IMUs (lumbar and both ankles, AUROC = 0.83), all of which were rated highly wearable. The minimal IMU set consisted of a single ankle IMU (AUROC = 0.80). Correlations between these models and human raters were good to excellent for percent time freezing (ICC = 0.93, 0.89) and number of freezing events (ICC = 0.95, 0.86) for the best technical set and minimal IMU set, respectively. CONCLUSIONS: Several IMU sets consisting of three IMUs or fewer were highly rated for both technical performance and wearability, and more IMUs did not necessarily perform better in FOG detection. We openly share our data and software to further the development and adoption of a general, open-source model that uses raw signals and a standard sensor set for at-home monitoring of freezing of gait.

Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation.

Modeling human motor control and predicting how humans will move in novel environments is a grand scientific challenge. Researchers in the fields of biomechanics and motor control have proposed and evaluated motor control models via neuromechanical simulations, which produce physically correct motions of a musculoskeletal model. Typically, researchers have developed control models that encode physiologically plausible motor control hypotheses and compared the resulting simulation behaviors to measurable human motion data. While such plausible control models were able to simulate and explain many basic locomotion behaviors (e.g. walking, running, and climbing stairs), modeling higher layer controls (e.g. processing environment cues, planning long-term motion strategies, and coordinating basic motor skills to navigate in dynamic and complex environments) remains a challenge. Recent advances in deep reinforcement learning lay a foundation for modeling these complex control processes and controlling a diverse repertoire of human movement; however, reinforcement learning has been rarely applied in neuromechanical simulation to model human control. In this paper, we review the current state of neuromechanical simulations, along with the fundamentals of reinforcement learning, as it applies to human locomotion. We also present a scientific competition and accompanying software platform, which we have organized to accelerate the use of reinforcement learning in neuromechanical simulations. This "Learn to Move" competition was an official competition at the NeurIPS conference from 2017 to 2019 and attracted over 1300 teams from around the world. Top teams adapted state-of-the-art deep reinforcement learning techniques and produced motions, such as quick turning and walk-to-stand transitions, that have not been demonstrated before in neuromechanical simulations without utilizing reference motion data. We close with a discussion of future opportunities at the intersection of human movement simulation and reinforcement learning and our plans to extend the Learn to Move competition to further facilitate interdisciplinary collaboration in modeling human motor control for biomechanics and rehabilitation research.

Gene expression profiling of low-grade endometrial stromal sarcoma indicates fusion protein-mediated activation of the Wnt signaling pathway.

OBJECTIVE: Low-grade endometrial stromal sarcomas (LGESS) harbor chromosomal translocations that affect proteins associated with chromatin remodeling Polycomb Repressive Complex 2 (PRC2), including SUZ12, PHF1 and EPC1. Roughly half of LGESS also demonstrate nuclear accumulation of ß-catenin, which is a hallmark of Wnt signaling activation. However, the targets affected by the fusion proteins and the role of Wnt signaling in the pathogenesis of these tumors remain largely unknown. METHODS: Here we report the results of a meta-analysis of three independent gene expression profiling studies on LGESS and immunohistochemical evaluation of nuclear expression of ß-catenin and Lef1 in 112 uterine sarcoma specimens obtained from 20 LGESS and 89 LMS patients. RESULTS: Our results demonstrate that 143 out of 310 genes overexpressed in LGESS are known to be directly regulated by SUZ12. In addition, our gene expression meta-analysis shows activation of multiple genes implicated in Wnt signaling. We further emphasize the role of the Wnt signaling pathway by demonstrating concordant nuclear expression of ß-catenin and Lef1 in 7/16 LGESS. CONCLUSIONS: Based on our findings, we suggest that LGESS-specific fusion proteins disrupt the repressive function of the PRC2 complex similar to the mechanism seen in synovial sarcoma, where the SS18-SSX fusion proteins disrupt the mSWI/SNF (BAF) chromatin remodeling complex. We propose that these fusion proteins in LGESS contribute to overexpression of Wnt ligands with subsequent activation of Wnt signaling pathway and formation of an active ß-catenin/Lef1 transcriptional complex. These observations could lead to novel therapeutic approaches that focus on the Wnt pathway in LGESS.

Modeling longitudinal data using matrix completion.

In clinical practice and biomedical research, measurements are often collected sparsely and irregularly in time, while the data acquisition is expensive and inconvenient. Examples include measurements of spine bone mineral density, cancer growth through mammography or biopsy, a progression of defective vision, or assessment of gait in patients with neurological disorders. Practitioners often need to infer the progression of diseases from such sparse observations. A classical tool for analyzing such data is a mixed-effect model where time is treated as both a fixed effect (population progression curve) and a random effect (individual variability). Alternatively, researchers use Gaussian processes or functional data analysis, assuming that observations are drawn from a certain distribution of processes. While these models are flexible, they rely on probabilistic assumptions, require very careful implementation, and tend to be slow in practice. In this study, we propose an alternative elementary framework for analyzing longitudinal data motivated by matrix completion. Our method yields estimates of progression curves by iterative application of the Singular Value Decomposition. Our framework covers multivariate longitudinal data, and regression and can be easily extended to other settings. As it relies on existing tools for matrix algebra, it is efficient and easy to implement. We apply our methods to understand trends of progression of motor impairment in children with Cerebral Palsy. Our model approximates individual progression curves and explains 30% of the variability. Low-rank representation of progression trends enables identification of different progression trends in subtypes of Cerebral Palsy.

Spatially Segregated Macrophage Populations Predict Distinct Outcomes In Colon Cancer.

Tumor-associated macrophages are transcriptionally heterogeneous, but the spatial distribution and cell interactions that shape macrophage tissue roles remain poorly characterized. Here, we spatially resolve five distinct human macrophage populations in normal and malignant human breast and colon tissue and reveal their cellular associations. This spatial map reveals that distinct macrophage populations reside in spatially segregated micro-environmental niches with conserved cellular compositions that are repeated across healthy and diseased tissue. We show that IL4I1+ macrophages phagocytose dying cells in areas with high cell turnover and predict good outcome in colon cancer. In contrast, SPP1+ macrophages are enriched in hypoxic and necrotic tumor regions and portend worse outcome in colon cancer. A subset of FOLR2+ macrophages is embedded in plasma cell niches. NLRP3+ macrophages co-localize with neutrophils and activate an inflammasome in tumors. Our findings indicate that a limited number of unique human macrophage niches function as fundamental building blocks in tissue.

Smartphone videos of the sit-to-stand test predict osteoarthritis and health outcomes in a nationwide study.

Physical function decline due to aging or disease can be assessed with quantitative motion analysis, but this currently requires expensive laboratory equipment. We introduce a self-guided quantitative motion analysis of the widely used five-repetition sit-to-stand test using a smartphone. Across 35 US states, 405 participants recorded a video performing the test in their homes. We found that the quantitative movement parameters extracted from the smartphone videos were related to a diagnosis of osteoarthritis, physical and mental health, body mass index, age, and ethnicity and race. Our findings demonstrate that at-home movement analysis goes beyond established clinical metrics to provide objective and inexpensive digital outcome metrics for nationwide studies.

Session Introduction: Precision Medicine: Using Artificial Intelligence to Improve Diagnostics and Healthcare.

Precision medicine requires a deep understanding of complex biomedical and healthcare data, which is being generated at exponential rates and increasingly made available through public biobanks, electronic medical record systems and biomedical databases and knowledgebases. The complexity and sheer amount of data prohibit manual manipulation. Instead, the field depends on artificial intelligence approaches to parse, annotate, evaluate and interpret the data to enable applications to patient healthcare At the 2023 Pacific Symposium on Biocomputing (PSB) session entitled "Precision Medicine: Using Artificial Intelligence (AI) to improve diagnostics and healthcare", we spotlight research that develops and applies computational methodologies to solve biomedical problems.

A spatial map of human macrophage niches reveals context-dependent macrophage functions in colon and breast cancer.

Tumor-associated macrophages (TAMs) display heterogeneous phenotypes. Yet the exact tissue cues that shape macrophage functional diversity are incompletely understood. Here we discriminate, spatially resolve and reveal the function of five distinct macrophage niches within malignant and benign breast and colon tissue. We found that SPP1 TAMs reside in hypoxic and necrotic tumor regions, and a novel subset of FOLR2 tissue resident macrophages (TRMs) supports the plasma cell tissue niche. We discover that IL4I1 macrophages populate niches with high cell turnover where they phagocytose dying cells. Significantly, IL4I1 TAMs abundance correlates with anti-PD1 treatment response in breast cancer. Furthermore, NLRP3 inflammasome activation in NLRP3 TAMs correlates with neutrophil infiltration in the tumors and is associated with poor outcome in breast cancer patients. This suggests the NLRP3 inflammasome as a novel cancer immunetherapy target. Our work uncovers context-dependent roles of macrophage subsets, and suggests novel predictive markers and macrophage subset-specific therapy targets.

Generalized Matrix Factorization: efficient algorithms for fitting generalized linear latent variable models to large data arrays.

Unmeasured or latent variables are often the cause of correlations between multivariate measurements, which are studied in a variety of fields such as psychology, ecology, and medicine. For Gaussian measurements, there are classical tools such as factor analysis or principal component analysis with a well-established theory and fast algorithms. Generalized Linear Latent Variable models (GLLVMs) generalize such factor models to non-Gaussian responses. However, current algorithms for estimating model parameters in GLLVMs require intensive computation and do not scale to large datasets with thousands of observational units or responses. In this article, we propose a new approach for fitting GLLVMs to high-dimensional datasets, based on approximating the model using penalized quasi-likelihood and then using a Newton method and Fisher scoring to learn the model parameters. Computationally, our method is noticeably faster and more stable, enabling GLLVM fits to much larger matrices than previously possible. We apply our method on a dataset of 48,000 observational units with over 2,000 observed species in each unit and find that most of the variability can be explained with a handful of factors. We publish an easy-to-use implementation of our proposed fitting algorithm.

Personalization improves the biomechanical efficacy of foot progression angle modifications in individuals with medial knee osteoarthritis.

Modifying the foot progression angle during walking can reduce the knee adduction moment, a surrogate measure of medial knee loading. However, not all individuals reduce their knee adduction moment with the same modification. This study evaluates whether a personalized approach to prescribing foot progression angle modifications increases the proportion of individuals with medial knee osteoarthritis who reduce their knee adduction moment, compared to a non-personalized approach. Individuals with medial knee osteoarthritis (N=107) walked with biofeedback instructing them to toe-in and toe-out by 5° and 10° relative to their self-selected angle. We selected individuals' personalized foot progression angle as the modification that maximally reduced their larger knee adduction moment peak. Additionally, we used lasso regression to identify which secondary kinematic changes made a 10° toe-in gait modification more effective at reducing the first knee adduction moment peak. Seventy percent of individuals reduced their larger knee adduction moment peak by at least 5% with a personalized foot progression angle modification, which was more than (p≤0.002) the 23-57% of individuals who reduced it with a uniformly assigned 5° or 10° toe-in or toe-out modification. When toeing-in, greater reductions in the first knee adduction moment peak were related to an increased frontal-plane tibia angle (knee more medial than ankle), a more valgus knee abduction angle, reduced contralateral pelvic drop, and a more medialized center of pressure in the foot reference frame. In summary, personalization increases the proportion of individuals with medial knee osteoarthritis who may benefit from a foot progression angle modification.

Precision Medicine: Using Artificial Intelligence to Improve Diagnostics and Healthcare.

The continued generation of large amounts of data within healthcare-from imaging to electronic medical health records to genomics and multi-omics -necessitates tools and methods to parse and interpret these data to improve healthcare outcomes. Artificial intelligence, and in particular deep learning, has enabled researchers to gain new insights from large scale and multimodal data. At the 2022 Pacific Symposium on Biocomputing (PSB) session entitled "Precision Medicine: Using Artificial Intelligence to Improve Diagnostics and Healthcare", we showcase the latest research, influenced and inspired by the idea of using technology to build a more fair, tailored, and cost-effective healthcare system after the COVID-19 pandemic.

Self-Organizing Maps for Cellular In Silico Staining and Cell Substate Classification.

Cellular composition and structural organization of cells in the tissue determine effective antitumor response and can predict patient outcome and therapy response. Here we present Seg-SOM, a method for dimensionality reduction of cell morphology in H&E-stained tissue images. Seg-SOM resolves cellular tissue heterogeneity and reveals complex tissue architecture. We leverage a self-organizing map (SOM) artificial neural network to group cells based on morphological features like shape and size. Seg-SOM allows for cell segmentation, systematic classification, and in silico cell labeling. We apply the Seg-SOM to a dataset of breast cancer progression images and find that clustering of SOM classes reveals groups of cells corresponding to fibroblasts, epithelial cells, and lymphocytes. We show that labeling the Lymphocyte SOM class on the breast tissue images accurately estimates lymphocytic infiltration. We further demonstrate how to use Seq-SOM in combination with non-negative matrix factorization to statistically describe the interaction of cell subtypes and use the interaction information as highly interpretable features for a histological classifier. Our work provides a framework for use of SOM in human pathology to resolve cellular composition of complex human tissues. We provide a python implementation and an easy-to-use docker deployment, enabling researchers to effortlessly featurize digitalized H&E-stained tissue.

Wearable sensors enable personalized predictions of clinical laboratory measurements.

Vital signs, including heart rate and body temperature, are useful in detecting or monitoring medical conditions, but are typically measured in the clinic and require follow-up laboratory testing for more definitive diagnoses. Here we examined whether vital signs as measured by consumer wearable devices (that is, continuously monitored heart rate, body temperature, electrodermal activity and movement) can predict clinical laboratory test results using machine learning models, including random forest and Lasso models. Our results demonstrate that vital sign data collected from wearables give a more consistent and precise depiction of resting heart rate than do measurements taken in the clinic. Vital sign data collected from wearables can also predict several clinical laboratory measurements with lower prediction error than predictions made using clinically obtained vital sign measurements. The length of time over which vital signs are monitored and the proximity of the monitoring period to the date of prediction play a critical role in the performance of the machine learning models. These results demonstrate the value of commercial wearable devices for continuous and longitudinal assessment of physiological measurements that today can be measured only with clinical laboratory tests.

Open Source Software for Automatic Subregional Assessment of Knee Cartilage Degradation Using Quantitative T2 Relaxometry and Deep Learning.

OBJECTIVE: We evaluated a fully automated femoral cartilage segmentation model for measuring T2 relaxation values and longitudinal changes using multi-echo spin-echo (MESE) magnetic resonance imaging (MRI). We open sourced this model and developed a web app available at https://kl.stanford.edu into which users can drag and drop images to segment them automatically. DESIGN: We trained a neural network to segment femoral cartilage from MESE MRIs. Cartilage was divided into 12 subregions along medial-lateral, superficial-deep, and anterior-central-posterior boundaries. Subregional T2 values and four-year changes were calculated using a radiologist's segmentations (Reader 1) and the model's segmentations. These were compared using 28 held-out images. A subset of 14 images were also evaluated by a second expert (Reader 2) for comparison. RESULTS: Model segmentations agreed with Reader 1 segmentations with a Dice score of 0.85 ± 0.03. The model's estimated T2 values for individual subregions agreed with those of Reader 1 with an average Spearman correlation of 0.89 and average mean absolute error (MAE) of 1.34 ms. The model's estimated four-year change in T2 for individual subregions agreed with Reader 1 with an average correlation of 0.80 and average MAE of 1.72 ms. The model agreed with Reader 1 at least as closely as Reader 2 agreed with Reader 1 in terms of Dice score (0.85 vs. 0.75) and subregional T2 values. CONCLUSIONS: Assessments of cartilage health using our fully automated segmentation model agreed with those of an expert as closely as experts agreed with one another. This has the potential to accelerate osteoarthritis research.

Deep neural networks enable quantitative movement analysis using single-camera videos.

Many neurological and musculoskeletal diseases impair movement, which limits people's function and social participation. Quantitative assessment of motion is critical to medical decision-making but is currently possible only with expensive motion capture systems and highly trained personnel. Here, we present a method for predicting clinically relevant motion parameters from an ordinary video of a patient. Our machine learning models predict parameters include walking speed (r = 0.73), cadence (r = 0.79), knee flexion angle at maximum extension (r = 0.83), and Gait Deviation Index (GDI), a comprehensive metric of gait impairment (r = 0.75). These correlation values approach the theoretical limits for accuracy imposed by natural variability in these metrics within our patient population. Our methods for quantifying gait pathology with commodity cameras increase access to quantitative motion analysis in clinics and at home and enable researchers to conduct large-scale studies of neurological and musculoskeletal disorders.

The turning and barrier course reveals gait parameters for detecting freezing of gait and measuring the efficacy of deep brain stimulation.

Freezing of gait (FOG) is a devastating motor symptom of Parkinson's disease that leads to falls, reduced mobility, and decreased quality of life. Reliably eliciting FOG has been difficult in the clinical setting, which has limited discovery of pathophysiology and/or documentation of the efficacy of treatments, such as different frequencies of subthalamic deep brain stimulation (STN DBS). In this study we validated an instrumented gait task, the turning and barrier course (TBC), with the international standard FOG questionnaire question 3 (FOG-Q3, r = 0.74, p < 0.001). The TBC is easily assembled and mimics real-life environments that elicit FOG. People with Parkinson's disease who experience FOG (freezers) spent more time freezing during the TBC compared to during forward walking (p = 0.007). Freezers also exhibited greater arrhythmicity during non-freezing gait when performing the TBC compared to forward walking (p = 0.006); this difference in gait arrhythmicity between tasks was not detected in non-freezers or controls. Freezers' non-freezing gait was more arrhythmic than that of non-freezers or controls during all walking tasks (p < 0.05). A logistic regression model determined that a combination of gait arrhythmicity, stride time, shank angular range, and asymmetry had the greatest probability of classifying a step as FOG (area under receiver operating characteristic curve = 0.754). Freezers' percent time freezing and non-freezing gait arrhythmicity decreased, and their shank angular velocity increased in the TBC during both 60 Hz and 140 Hz STN DBS (p < 0.05) to non-freezer values. The TBC is a standardized tool for eliciting FOG and demonstrating the efficacy of 60 Hz and 140 Hz STN DBS for gait impairment and FOG. The TBC revealed gait parameters that differentiated freezers from non-freezers and best predicted FOG; these may serve as relevant control variables for closed loop neurostimulation for FOG in Parkinson's disease.

Pre-operative gastrocnemius lengths in gait predict outcomes following gastrocnemius lengthening surgery in children with cerebral palsy.

Equinus deformity is one of the most common gait deformities in children with cerebral palsy. We examined whether estimates of gastrocnemius length in gait could identify limbs likely to have short-term and long-term improvements in ankle kinematics following gastrocnemius lengthening surgery to correct equinus. We retrospectively analyzed data of 891 limbs that underwent a single-event multi-level surgery (SEMLS), and categorized outcomes based on the normalcy of ankle kinematics. Limbs with short gastrocnemius lengths that received a gastrocnemius lengthening surgery as part of a SEMLS (case limbs) were 2.2 times more likely than overtreated limbs (i.e., limbs who did not have short lengths, but still received a lengthening surgery) to have a good surgical outcome at the follow-up gait visit (good outcome rate of 71% vs. 33%). Case limbs were 1.2 times more likely than control limbs (i.e., limbs that had short gastrocnemius lengths but no lengthening surgery) to have a good outcome (71% vs. 59%). Three-fourths of the case limbs with a good outcome at the follow-up gait visit maintained this outcome over time, compared to only one-half of the overtreated limbs. Our results caution against over-prescription of gastrocnemius lengthening surgery and suggest gastrocnemius lengths can be used to identify good surgical candidates.

Deep longitudinal multiomics profiling reveals two biological seasonal patterns in California.

The influence of seasons on biological processes is poorly understood. In order to identify biological seasonal patterns based on diverse molecular data, rather than calendar dates, we performed a deep longitudinal multiomics profiling of 105 individuals over 4 years. Here, we report more than 1000 seasonal variations in omics analytes and clinical measures. The different molecules group into two major seasonal patterns which correlate with peaks in late spring and late fall/early winter in California. The two patterns are enriched for molecules involved in human biological processes such as inflammation, immunity, cardiovascular health, as well as neurological and psychiatric conditions. Lastly, we identify molecules and microbes that demonstrate different seasonal patterns in insulin sensitive and insulin resistant individuals. The results of our study have important implications in healthcare and highlight the value of considering seasonality when assessing population wide health risk and management.

Automated Classification of Radiographic Knee Osteoarthritis Severity Using Deep Neural Networks.

PURPOSE: To develop an automated model for staging knee osteoarthritis severity from radiographs and to compare its performance to that of musculoskeletal radiologists. MATERIALS AND METHODS: Radiographs from the Osteoarthritis Initiative staged by a radiologist committee using the Kellgren-Lawrence (KL) system were used. Before using the images as input to a convolutional neural network model, they were standardized and augmented automatically. The model was trained with 32 116 images, tuned with 4074 images, evaluated with a 4090-image test set, and compared to two individual radiologists using a 50-image test subset. Saliency maps were generated to reveal features used by the model to determine KL grades. RESULTS: With committee scores used as ground truth, the model had an average F1 score of 0.70 and an accuracy of 0.71 for the full test set. For the 50-image subset, the best individual radiologist had an average F1 score of 0.60 and an accuracy of 0.60; the model had an average F1 score of 0.64 and an accuracy of 0.66. Cohen weighted κ between the committee and model was 0.86, comparable to intraexpert repeatability. Saliency maps identified sites of osteophyte formation as influential to predictions. CONCLUSION: An end-to-end interpretable model that takes full radiographs as input and predicts KL scores with state-of-the-art accuracy, performs as well as musculoskeletal radiologists, and does not require manual image preprocessing was developed. Saliency maps suggest the model's predictions were based on clinically relevant information. Supplemental material is available for this article. © RSNA, 2020.