Federated learning with knowledge distillation for multi-organ segmentation with partially labeled datasets.

The state-of-the-art multi-organ CT segmentation relies on deep learning models, which only generalize when trained on large samples of carefully curated data. However, it is challenging to train a single model that can segment all organs and types of tumors since most large datasets are partially labeled or are acquired across multiple institutes that may differ in their acquisitions. A possible solution is Federated learning, which is often used to train models on multi-institutional datasets where the data is not shared across sites. However, predictions of federated learning can be unreliable after the model is locally updated at sites due to 'catastrophic forgetting'. Here, we address this issue by using knowledge distillation (KD) so that the local training is regularized with the knowledge of a global model and pre-trained organ-specific segmentation models. We implement the models in a multi-head U-Net architecture that learns a shared embedding space for different organ segmentation, thereby obtaining multi-organ predictions without repeated processes. We evaluate the proposed method using 8 publicly available abdominal CT datasets of 7 different organs. Of those datasets, 889 CTs were used for training, 233 for internal testing, and 30 volumes for external testing. Experimental results verified that our proposed method substantially outperforms other state-of-the-art methods in terms of accuracy, inference time, and the number of parameters.

Machine learning-based high-frequency neuronal spike reconstruction from low-frequency and low-sampling-rate recordings.

Recording neuronal activity using multiple electrodes has been widely used to understand the functional mechanisms of the brain. Increasing the number of electrodes allows us to decode more variety of functionalities. However, handling massive amounts of multichannel electrophysiological data is still challenging due to the limited hardware resources and unavoidable thermal tissue damage. Here, we present machine learning (ML)-based reconstruction of high-frequency neuronal spikes from subsampled low-frequency band signals. Inspired by the equivalence between high-frequency restoration and super-resolution in image processing, we applied a transformer ML model to neuronal data recorded from both in vitro cultures and in vivo male mouse brains. Even with the x8 downsampled datasets, our trained model reasonably estimated high-frequency information of spiking activity, including spike timing, waveform, and network connectivity. With our ML-based data reduction applicable to existing multichannel recording hardware while achieving neuronal signals of broad bandwidths, we expect to enable more comprehensive analysis and control of brain functions.

DeepFold: enhancing protein structure prediction through optimized loss functions, improved template features, and re-optimized energy function.

MOTIVATION: Predicting protein structures with high accuracy is a critical challenge for the broad community of life sciences and industry. Despite progress made by deep neural networks like AlphaFold2, there is a need for further improvements in the quality of detailed structures, such as side-chains, along with protein backbone structures. RESULTS: Building upon the successes of AlphaFold2, the modifications we made include changing the losses of side-chain torsion angles and frame aligned point error, adding loss functions for side chain confidence and secondary structure prediction, and replacing template feature generation with a new alignment method based on conditional random fields. We also performed re-optimization by conformational space annealing using a molecular mechanics energy function which integrates the potential energies obtained from distogram and side-chain prediction. In the CASP15 blind test for single protein and domain modeling (109 domains), DeepFold ranked fourth among 132 groups with improvements in the details of the structure in terms of backbone, side-chain, and Molprobity. In terms of protein backbone accuracy, DeepFold achieved a median GDT-TS score of 88.64 compared with 85.88 of AlphaFold2. For TBM-easy/hard targets, DeepFold ranked at the top based on Z-scores for GDT-TS. This shows its practical value to the structural biology community, which demands highly accurate structures. In addition, a thorough analysis of 55 domains from 39 targets with publicly available structures indicates that DeepFold shows superior side-chain accuracy and Molprobity scores among the top-performing groups. AVAILABILITY AND IMPLEMENTATION: DeepFold tools are open-source software available at https://github.com/newtonjoo/deepfold.

Deep 3D reconstruction of synchrotron X-ray computed tomography for intact lungs.

Synchrotron X-rays can be used to obtain highly detailed images of parts of the lung. However, micro-motion artifacts induced by such as cardiac motion impede quantitative visualization of the alveoli in the lungs. This paper proposes a method that applies a neural network for synchrotron X-ray Computed Tomography (CT) data to reconstruct the high-quality 3D structure of alveoli in intact mouse lungs at expiration, without needing ground-truth data. Our approach reconstructs the spatial sequence of CT images by using a deep-image prior with interpolated input latent variables, and in this way significantly enhances the images of alveolar structure compared with the prior art. The approach successfully visualizes 3D alveolar units of intact mouse lungs at expiration and enables us to measure the diameter of the alveoli. We believe that our approach helps to accurately visualize other living organs hampered by micro-motion.

One-shot Federated Learning on Medical Data using Knowledge Distillation with Image Synthesis and Client Model Adaptation.

One-shot federated learning (FL) has emerged as a promising solution in scenarios where multiple communication rounds are not practical. Notably, as feature distributions in medical data are less discriminative than those of natural images, robust global model training with FL is non-trivial and can lead to overfitting. To address this issue, we propose a novel one-shot FL framework leveraging Image Synthesis and Client model Adaptation (FedISCA) with knowledge distillation (KD). To prevent overfitting, we generate diverse synthetic images ranging from random noise to realistic images. This approach (i) alleviates data privacy concerns and (ii) facilitates robust global model training using KD with decentralized client models. To mitigate domain disparity in the early stages of synthesis, we design noise-adapted client models where batch normalization statistics on random noise (synthetic images) are updated to enhance KD. Lastly, the global model is trained with both the original and noise-adapted client models via KD and synthetic images. This process is repeated till global model convergence. Extensive evaluation of this design on five small- and three large-scale medical image classification datasets reveals superior accuracy over prior methods. Code is available at https://github.com/myeongkyunkang/FedISCA.

Time-Dependent Deep Image Prior for Dynamic MRI.

We propose a novel unsupervised deep-learning-based algorithm for dynamic magnetic resonance imaging (MRI) reconstruction. Dynamic MRI requires rapid data acquisition for the study of moving organs such as the heart. We introduce a generalized version of the deep-image-prior approach, which optimizes the weights of a reconstruction network to fit a sequence of sparsely acquired dynamic MRI measurements. Our method needs neither prior training nor additional data. In particular, for cardiac images, it does not require the marking of heartbeats or the reordering of spokes. The key ingredients of our method are threefold: 1) a fixed low-dimensional manifold that encodes the temporal variations of images; 2) a network that maps the manifold into a more expressive latent space; and 3) a convolutional neural network that generates a dynamic series of MRI images from the latent variables and that favors their consistency with the measurements in k -space. Our method outperforms the state-of-the-art methods quantitatively and qualitatively in both retrospective and real fetal cardiac datasets. To the best of our knowledge, this is the first unsupervised deep-learning-based method that can reconstruct the continuous variation of dynamic MRI sequences with high spatial resolution.

Grid-Free Localization Algorithm Using Low-rank Hankel Matrix for Super-Resolution Microscopy.

Localization microscopy, such as STORM / PALM, can reconstruct super-resolution images with a nanometer resolution through the iterative localization of fluorescence molecules. Recent studies in this area have focused mainly on the localization of densely activated molecules to improve temporal resolutions. However, higher density imaging requires an advanced algorithm that can resolve closely spaced molecules. Accordingly, sparsitydriven methods have been studied extensively. One of the major limitations of existing sparsity-driven approaches is the need for a fine sampling grid or for Taylor series approximation which may result in some degree of localization bias toward the grid. In addition, prior knowledge of the point-spread function (PSF) is required. To address these drawbacks, here we propose a true grid-free localization algorithm with adaptive PSF estimation. Specifically, based on the observation that sparsity in the spatial domain implies a low rank in the Fourier domain, the proposed method converts source localization problems into Fourier-domain signal processing problems so that a truly gridfree localization is possible. We verify the performance of the newly proposed method with several numerical simulations and a live-cell imaging experiment.

CNN-Based Projected Gradient Descent for Consistent CT Image Reconstruction.

We present a new image reconstruction method that replaces the projector in a projected gradient descent (PGD) with a convolutional neural network (CNN). Recently, CNNs trained as image-to-image regressors have been successfully used to solve inverse problems in imaging. However, unlike existing iterative image reconstruction algorithms, these CNN-based approaches usually lack a feedback mechanism to enforce that the reconstructed image is consistent with the measurements. We propose a relaxed version of PGD wherein gradient descent enforces measurement consistency, while a CNN recursively projects the solution closer to the space of desired reconstruction images. We show that this algorithm is guaranteed to converge and, under certain conditions, converges to a local minimum of a non-convex inverse problem. Finally, we propose a simple scheme to train the CNN to act like a projector. Our experiments on sparse-view computed-tomography reconstruction show an improvement over total variation-based regularization, dictionary learning, and a state-of-the-art deep learning-based direct reconstruction technique.

Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging.

For effective treatment of Alzheimer's disease (AD), it is important to identify subjects who are most likely to exhibit rapid cognitive decline. We aimed to develop an automatic image interpretation system based on a deep convolutional neural network (CNN) which can accurately predict future cognitive decline in mild cognitive impairment (MCI) patients using flurodeoxyglucose and florbetapir positron emission tomography (PET). PET images of 139 patients with AD, 171 patients with MCI and 182 normal subjects obtained from Alzheimer's Disease Neuroimaging Initiative database were used. Deep CNN was trained using 3-dimensional PET volumes of AD and normal controls as inputs. Manually defined image feature extraction such as quantification using predefined region-of-interests was unnecessary for our approach. Furthermore, it used minimally processed images without spatial normalization which has been commonly used in conventional quantitative analyses. Cognitive outcome of MCI subjects was predicted using this network. The prediction accuracy of the conversion of mild cognitive impairment to AD was compared with the conventional feature-based quantification approach. Accuracy of prediction (84.2%) for conversion to AD in MCI patients outperformed conventional feature-based quantification approaches. ROC analyses revealed that performance of CNN-based approach was significantly higher than that of the conventional quantification methods (p < 0.05). Output scores of the network were strongly correlated with the longitudinal change in cognitive measurements (p < 0.05). These results show the feasibility of deep learning as a practical tool for developing predictive neuroimaging biomarker.

MRI artifact correction using sparse + low-rank decomposition of annihilating filter-based hankel matrix.

PURPOSE: Magnetic resonance imaging (MRI) artifacts are originated from various sources including instability of an magnetic resonance (MR) system, patient motion, inhomogeneities of gradient fields, and so on. Such MRI artifacts are usually considered as irreversible, so additional artifact-free scan or navigator scan is necessary. To overcome these limitations, this article proposes a novel compressed sensing-based approach for removal of various MRI artifacts. THEORY: Recently, the annihilating filter based low-rank Hankel matrix approach was proposed. The annihilating filter based low-rank Hankel matrix exploits the duality between the low-rankness of weighted Hankel structured matrix and the sparsity of signal in a transform domain. Because MR artifacts usually appeared as sparse k-space components, the low-rank Hankel matrix from underlying artifact-free k-space data can be exploited to decompose the sparse outliers. METHODS: The sparse + low-rank decomposition framework using Hankel matrix was proposed for removal of MRI artifacts. Alternating direction method of multipliers algorithm was employed for the minimization of associated cost function with the initialized matrices from a factorization-based matrix completion. RESULTS: Experimental results demonstrated that the proposed algorithm can correct MR artifacts including herringbone (crisscross), motion, and zipper artifacts without image distortion. CONCLUSION: The proposed method may be a robust correction solution for various MRI artifacts that can be represented as sparse outliers. Magn Reson Med 78:327-340, 2017. © 2016 International Society for Magnetic Resonance in Medicine.

Fast and robust segmentation of the striatum using deep convolutional neural networks.

BACKGROUND: Automated segmentation of brain structures is an important task in structural and functional image analysis. We developed a fast and accurate method for the striatum segmentation using deep convolutional neural networks (CNN). NEW METHOD: T1 magnetic resonance (MR) images were used for our CNN-based segmentation, which require neither image feature extraction nor nonlinear transformation. We employed two serial CNN, Global and Local CNN: The Global CNN determined approximate locations of the striatum. It performed a regression of input MR images fitted to smoothed segmentation maps of the striatum. From the output volume of Global CNN, cropped MR volumes which included the striatum were extracted. The cropped MR volumes and the output volumes of Global CNN were used for inputs of Local CNN. Local CNN predicted the accurate label of all voxels. Segmentation results were compared with a widely used segmentation method, FreeSurfer. RESULTS: Our method showed higher Dice Similarity Coefficient (DSC) (0.893±0.017 vs. 0.786±0.015) and precision score (0.905±0.018 vs. 0.690±0.022) than FreeSurfer-based striatum segmentation (p=0.06). Our approach was also tested using another independent dataset, which showed high DSC (0.826±0.038) comparable with that of FreeSurfer. Comparison with existing method Segmentation performance of our proposed method was comparable with that of FreeSurfer. The running time of our approach was approximately three seconds. CONCLUSION: We suggested a fast and accurate deep CNN-based segmentation for small brain structures which can be widely applied to brain image analysis.

Reference-free single-pass EPI Nyquist ghost correction using annihilating filter-based low rank Hankel matrix (ALOHA).

PURPOSE: MR measurements from an echo-planar imaging (EPI) sequence produce Nyquist ghost artifacts that originate from inconsistencies between odd and even echoes. Several reconstruction algorithms have been proposed to reduce such artifacts, but most of these methods require either additional reference scans or multipass EPI acquisition. This article proposes a novel and accurate single-pass EPI ghost artifact correction method that does not require any additional reference data. THEORY AND METHODS: After converting a ghost correction problem into separate k-space data interpolation problems for even and odd phase encoding, our algorithm exploits an observation that the differential k-space data between the even and odd echoes is a Fourier transform of an underlying sparse image. Accordingly, we can construct a rank-deficient Hankel structured matrix, whose missing data can be recovered using an annihilating filter-based low rank Hankel structured matrix completion approach. RESULTS: The proposed method was applied to EPI data for both single and multicoil acquisitions. Experimental results using in vivo data confirmed that the proposed method can completely remove ghost artifacts successfully without prescan echoes. CONCLUSION: Owing to the discovery of the annihilating filter relationship from the intrinsic EPI image property, the proposed method successfully suppresses ghost artifacts without a prescan step. Magn Reson Med 76:1775-1789, 2016. © 2016 International Society for Magnetic Resonance in Medicine.

Acceleration of MR parameter mapping using annihilating filter-based low rank hankel matrix (ALOHA).

PURPOSE: MR parameter mapping is one of clinically valuable MR imaging techniques. However, increased scan time makes it difficult for routine clinical use. This article aims at developing an accelerated MR parameter mapping technique using annihilating filter based low-rank Hankel matrix approach (ALOHA). THEORY: When a dynamic sequence can be sparsified using spatial wavelet and temporal Fourier transform, this results in a rank-deficient Hankel structured matrix that is constructed using weighted k-t measurements. ALOHA then utilizes the low rank matrix completion algorithm combined with a multiscale pyramidal decomposition to estimate the missing k-space data. METHODS: Spin-echo inversion recovery and multiecho spin echo pulse sequences for T1 and T2 mapping, respectively, were redesigned to perform undersampling along the phase encoding direction according to Gaussian distribution. The missing k-space is reconstructed using ALOHA. Then, the parameter maps were constructed using nonlinear regression. RESULTS: Experimental results confirmed that ALOHA outperformed the existing compressed sensing algorithms. Compared with the existing methods, the reconstruction errors appeared scattered throughout the entire images rather than exhibiting systematic distortion along edges and the parameter maps. CONCLUSION: Given that many diagnostic errors are caused by the systematic distortion of images, ALOHA may have a great potential for clinical applications. Magn Reson Med 76:1848-1864, 2016. © 2016 International Society for Magnetic Resonance in Medicine.

Comparative study of iterative reconstruction algorithms for missing cone problems in optical diffraction tomography.

In optical tomography, there exist certain spatial frequency components that cannot be measured due to the limited projection angles imposed by the numerical aperture of objective lenses. This limitation, often called as the missing cone problem, causes the under-estimation of refractive index (RI) values in tomograms and results in severe elongations of RI distributions along the optical axis. To address this missing cone problem, several iterative reconstruction algorithms have been introduced exploiting prior knowledge such as positivity in RI differences or edges of samples. In this paper, various existing iterative reconstruction algorithms are systematically compared for mitigating the missing cone problem in optical diffraction tomography. In particular, three representative regularization schemes, edge preserving, total variation regularization, and the Gerchberg-Papoulis algorithm, were numerically and experimentally evaluated using spherical beads as well as real biological samples; human red blood cells and hepatocyte cells. Our work will provide important guidelines for choosing the appropriate regularization in ODT.

Annihilating Filter-Based Low-Rank Hankel Matrix Approach for Image Inpainting.

In this paper, we propose a patch-based image inpainting method using a low-rank Hankel structured matrix completion approach. The proposed method exploits the annihilation property between a shift-invariant filter and image data observed in many existing inpainting algorithms. In particular, by exploiting the commutative property of the convolution, the annihilation property results in a low-rank block Hankel structure data matrix, and the image inpainting problem becomes a low-rank structured matrix completion problem. The block Hankel structured matrices are obtained patch-by-patch to adapt to the local changes in the image statistics. To solve the structured low-rank matrix completion problem, we employ an alternating direction method of multipliers with factorization matrix initialization using the low-rank matrix fitting algorithm. As a side product of the matrix factorization, locally adaptive dictionaries can be also easily constructed. Despite the simplicity of the algorithm, the experimental results using irregularly subsampled images as well as various images with globally missing patterns showed that the proposed method outperforms existing state-of-the-art image inpainting methods.

High-speed terahertz reflection three-dimensional imaging using beam steering.

High-speed terahertz (THz) reflection three-dimensional (3D) imaging is demonstrated using electronically-controlled optical sampling (ECOPS) and beam steering. ECOPS measurement is used for scanning an axial range of 7.8 mm in free space at 1 kHz scan rate while a transverse range of 100 × 100 mm(2) is scanned using beam steering instead of moving an imaging target. Telecentric f-θ lenses with axial and non-axial symmetry have been developed for beam steering. It is experimentally demonstrated that the non-axially symmetric lens has better characteristics than the axially symmetric lens. The total scan time depends on the number of points in a transverse range. For example, it takes 40 s for 200 × 200 points and 10 s for 100 × 100 points. To demonstrate the application of the imaging technique to nondestructive testing, THz 3D tomographic images of a glass fiber reinforced polymer sample with artificial internal defects have been acquired using the lenses for comparison.

High-speed terahertz reflection three-dimensional imaging for nondestructive evaluation.

We demonstrate high-speed terahertz (THz) reflection three-dimensional (3D) imaging based on electronically controlled optical sampling (ECOPS). ECOPS enables scanning of an axial range of 9 mm in free space at 1 kHz. It takes 80 s to scan a transverse range of 100 mm × 100 mm along a zigzag trajectory that consists of 200 lines using translation stages. To show applicability of the imaging system to nondestructive evaluation, a THz reflection 3D image of an artificially made sample is obtained, which is made of glass fiber reinforced polymer composite material and has defects such as delamination and inclusion, and is compared with an ultrasonic reflection 3D image of the sample.

Terahertz substance imaging by waveform shaping.

Terahertz pulse shaping technique is used to adaptively design terahertz waveforms of enhanced spectral correlation to particular materials among a given set of materials. In a proof-of-principle experiment performed with a two-dimensional image target consisted of meta-materials of distinctive resonance frequencies, the as-designed waveforms are used to demonstrate terahertz substance imaging. It is hoped that this material-specific terahertz waveforms may enable single- or few-shot terahertz material classification when being used in conjunction with terahertz power measurement.

Enhancement of terahertz pulse emission by optical nanoantenna.

Bridging the gap between ultrashort pulsed optical waves and terahertz (THz) waves, the THz photoconductive antenna (PCA) is a major constituent for the emission or detection of THz waves by diverse optical and electrical methods. However, THz PCA still lacks employment of advanced breakthrough technologies for high-power THz emission. Here, we report the enhancement of THz emission power by incorporating optical nanoantennas with a THz photoconductive antenna. The confinement and concentration of an optical pump beam on a photoconductive substrate can be efficiently achieved with optical nanoantennas over a high-index photoconductive substrate. Both numerical and experimental results clearly demonstrate the enhancement of THz wave emission due to high photocarrier generation at the plasmon resonance of nanoantennas. This work opens up many opportunities for diverse integrated photonic elements on a single PCA at THz and optical frequencies.