On the Use of Minimum Penalties in Statistical Learning.

Modern multivariate machine learning and statistical methodologies estimate parameters of interest while leveraging prior knowledge of the association between outcome variables. The methods that do allow for estimation of relationships do so typically through an error covariance matrix in multivariate regression which does not generalize to other types of models. In this article we proposed the MinPen framework to simultaneously estimate regression coefficients associated with the multivariate regression model and the relationships between outcome variables using common assumptions. The MinPen framework utilizes a novel penalty based on the minimum function to simultaneously detect and exploit relationships between responses. An iterative algorithm is proposed as a solution to the non-convex optimization. Theoretical results such as high dimensional convergence rates, model selection consistency, and a framework for post selection inference are provided. We extend the proposed MinPen framework to other exponential family loss functions, with a specific focus on multiple binomial responses. Tuning parameter selection is also addressed. Finally, simulations and two data examples are presented to show the finite sample properties of this framework. Supplemental material providing proofs, additional simulations, code, and data sets are available online.

Gossip-based distributed stochastic mirror descent for constrained optimization.

This paper considers a distributed constrained optimization problem over a multi-agent network in the non-Euclidean sense. The gossip protocol is adopted to relieve the communication burden, which also adapts to the constantly changing topology of the network. Based on this idea, a gossip-based distributed stochastic mirror descent (GB-DSMD) algorithm is proposed to handle the problem under consideration. The performances of GB-DSMD algorithms with constant and diminishing step sizes are analyzed, respectively. When the step size is constant, the error bound between the optimal function value and the expected function value corresponding to the average iteration output of the algorithm is derived. While for the case of the diminishing step size, it is proved that the output of the algorithm uniformly approaches to the optimal value with probability 1. Finally, as a numerical example, the distributed logistic regression is reported to demonstrate the effectiveness of the GB-DSMD algorithm.

Gradient regularization of Newton method with Bregman distances.

In this paper, we propose a first second-order scheme based on arbitrary non-Euclidean norms, incorporated by Bregman distances. They are introduced directly in the Newton iterate with regularization parameter proportional to the square root of the norm of the current gradient. For the basic scheme, as applied to the composite convex optimization problem, we establish the global convergence rate of the order O(k-2) both in terms of the functional residual and in the norm of subgradients. Our main assumption on the smooth part of the objective is Lipschitz continuity of its Hessian. For uniformly convex functions of degree three, we justify global linear rate, and for strongly convex function we prove the local superlinear rate of convergence. Our approach can be seen as a relaxation of the Cubic Regularization of the Newton method (Nesterov and Polyak in Math Program 108(1):177-205, 2006) for convex minimization problems. This relaxation preserves the convergence properties and global complexities of the Cubic Newton in convex case, while the auxiliary subproblem at each iteration is simpler. We equip our method with adaptive search procedure for choosing the regularization parameter. We propose also an accelerated scheme with convergence rate O(k-3), where k is the iteration counter.

A fast continuous time approach for non-smooth convex optimization using Tikhonov regularization technique.

In this paper we would like to address the classical optimization problem of minimizing a proper, convex and lower semicontinuous function via the second order in time dynamics, combining viscous and Hessian-driven damping with a Tikhonov regularization term. In our analysis we heavily exploit the Moreau envelope of the objective function and its properties as well as Tikhonov regularization properties, which we extend to a nonsmooth case. We introduce the setting, which at the same time guarantees the fast convergence of the function (and Moreau envelope) values and strong convergence of the trajectories of the system to a minimal norm solution-the element of the minimal norm of all the minimizers of the objective. Moreover, we deduce the precise rates of convergence of the values for the particular choice of parameters. Various numerical examples are also included as an illustration of the theoretical results.

On the Determination of Lagrange Multipliers for a Weighted LASSO Problem Using Geometric and Convex Analysis Techniques.

Compressed Sensing (CS) encompasses a broad array of theoretical and applied techniques for recovering signals, given partial knowledge of their coefficients, cf. Candés (C. R. Acad. Sci. Paris, Ser. I 346, 589-592 (2008)), Candés et al. (IEEE Trans. Inf. Theo (2006)), Donoho (IEEE Trans. Inf. Theo. 52(4), (2006)), Donoho et al. (IEEE Trans. Inf. Theo. 52(1), (2006)). Its applications span various fields, including mathematics, physics, engineering, and several medical sciences, cf. Adcock and Hansen (Compressive Imaging: Structure, Sampling, Learning, p. 2021), Berk et al. (2019 13th International conference on Sampling Theory and Applications (SampTA) pp. 1-5. IEEE (2019)), Brady et al. (Opt. Express 17(15), 13040-13049 (2009)), Chan (Terahertz imaging with compressive sensing. Rice University, USA (2010)), Correa et al. (2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 7789-7793 (2014, May) IEEE), Gao et al. (Nature 516(7529), 74-77 (2014)), Liu and Kang (Opt. Express 18(21), 22010-22019 (2010)), McEwen and Wiaux (Mon. Notices Royal Astron. Soc. 413(2), 1318-1332 (2011)), Marim et al. (Opt. Lett. 35(6), 871-873 (2010)), Yu and Wang (Phys. Med. Biol. 54(9), 2791 (2009)), Yu and Wang (Phys. Med. Biol. 54(9), 2791 (2009)). Motivated by our interest in the mathematics behind Magnetic Resonance Imaging (MRI) and CS, we employ convex analysis techniques to analytically determine equivalents of Lagrange multipliers for optimization problems with inequality constraints, specifically a weighted LASSO with voxel-wise weighting. We investigate this problem under assumptions on the fidelity term Ax-b22, either concerning the sign of its gradient or orthogonality-like conditions of its matrix. To be more precise, we either require the sign of each coordinate of 2(Ax-b)TA to be fixed within a rectangular neighborhood of the origin, with the side lengths of the rectangle dependent on the constraints, or we assume ATA to be diagonal. The objective of this work is to explore the relationship between Lagrange multipliers and the constraints of a weighted variant of LASSO, specifically in the mentioned cases where this relationship can be computed explicitly. As they scale the regularization terms of the weighted LASSO, Lagrange multipliers serve as tuning parameters for the weighted LASSO, prompting the question of their potential effective use as tuning parameters in applications like MR image reconstruction and denoising. This work represents an initial step in this direction.

Feedback linearization control for uncertain nonlinear systems via generative adversarial networks.

This article presents a novel approach to leverage generative adversarial networks(GANs) techniques to learn a feedback linearization controller(FLC) for a class of uncertain nonlinear systems. By estimating uncertainty through the adversarial process, where ground truth samples are exclusively obtained from a predefined integral model, the feedback linearization controller, learned through a minimax two-player optimization framework, enhances the reference tracking performance of the input-output uncertain nonlinear system. Furthermore, we provide theoretical guarantee of convergence and stability, demonstrating the safe recovery of robust FLC. We also address the common challenge of mode collapse in GANs training through the strict convexity of our synthesized generator structure and an enhanced adversarial loss. Comprehensive simulations and practical experiments are conducted to underscore the superiority and efficacy of our proposed approach.

Simultaneous model prediction and data-driven control with relaxed assumption on the model.

This paper aims to design a Model Predictive Control (MPC) law based on the time series data gathered from the input and output of a system. An Auto-Regressive Integrated Moving Average (ARIMA) model with unknown parameters and an unknown sequence of controller signal are considered for the system. Based on a window of data, an optimization problem is formulated which can find the optimal unknown model parameters and controller sequence, simultaneously. This problem is a non-convex optimization problem with many non-convex constraints and difficult to solve. Therefore, a transformation is developed which can transfer the optimization problem to an equivalent problem with convex constraints and a non-convex objective function. This new problem is much easier to solve with the present solvers. The effectiveness of the overall approach is proved via several examples that reveal satisfaction and convincingness.

RKHS-based covariate balancing for survival causal effect estimation.

Survival causal effect estimation based on right-censored data is of key interest in both survival analysis and causal inference. Propensity score weighting is one of the most popular methods in the literature. However, since it involves the inverse of propensity score estimates, its practical performance may be very unstable, especially when the covariate overlap is limited between treatment and control groups. To address this problem, a covariate balancing method is developed in this paper to estimate the counterfactual survival function. The proposed method is nonparametric and balances covariates in a reproducing kernel Hilbert space (RKHS) via weights that are counterparts of inverse propensity scores. The uniform rate of convergence for the proposed estimator is shown to be the same as that for the classical Kaplan-Meier estimator. The appealing practical performance of the proposed method is demonstrated by a simulation study as well as two real data applications to study the causal effect of smoking on survival time of stroke patients and that of endotoxin on survival time for female patients with lung cancer respectively.

AdaSAM: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks.

Sharpness aware minimization (SAM) optimizer has been extensively explored as it can generalize better for training deep neural networks via introducing extra perturbation steps to flatten the landscape of deep learning models. Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step. In this paper, we try to analyze the convergence rate of AdaSAM in the stochastic non-convex setting. We theoretically show that AdaSAM admits a O(1/bT) convergence rate, which achieves linear speedup property with respect to mini-batch size b. Specifically, to decouple the stochastic gradient steps with the adaptive learning rate and perturbed gradient, we introduce the delayed second-order momentum term to decompose them to make them independent while taking an expectation during the analysis. Then we bound them by showing the adaptive learning rate has a limited range, which makes our analysis feasible. To the best of our knowledge, we are the first to provide the non-trivial convergence rate of SAM with an adaptive learning rate and momentum acceleration. At last, we conduct several experiments on several NLP tasks and the synthetic task, which show that AdaSAM could achieve superior performance compared with SGD, AMSGrad, and SAM optimizers.

Regularized parametric survival modeling to improve risk prediction models.

We propose to combine the benefits of flexible parametric survival modeling and regularization to improve risk prediction modeling in the context of time-to-event data. Thereto, we introduce ridge, lasso, elastic net, and group lasso penalties for both log hazard and log cumulative hazard models. The log (cumulative) hazard in these models is represented by a flexible function of time that may depend on the covariates (i.e., covariate effects may be time-varying). We show that the optimization problem for the proposed models can be formulated as a convex optimization problem and provide a user-friendly R implementation for model fitting and penalty parameter selection based on cross-validation. Simulation study results show the advantage of regularization in terms of increased out-of-sample prediction accuracy and improved calibration and discrimination of predicted survival probabilities, especially when sample size was relatively small with respect to model complexity. An applied example illustrates the proposed methods. In summary, our work provides both a foundation for and an easily accessible implementation of regularized parametric survival modeling and suggests that it improves out-of-sample prediction performance.

Scalable kernel balancing weights in a nationwide observational study of hospital profit status and heart attack outcomes.

Weighting is a general and often-used method for statistical adjustment. Weighting has two objectives: first, to balance covariate distributions, and second, to ensure that the weights have minimal dispersion and thus produce a more stable estimator. A recent, increasingly common approach directly optimizes the weights toward these two objectives. However, this approach has not yet been feasible in large-scale datasets when investigators wish to flexibly balance general basis functions in an extended feature space. To address this practical problem, we describe a scalable and flexible approach to weighting that integrates a basis expansion in a reproducing kernel Hilbert space with state-of-the-art convex optimization techniques. Specifically, we use the rank-restricted Nyström method to efficiently compute a kernel basis for balancing in nearly linear time and space, and then use the specialized first-order alternating direction method of multipliers to rapidly find the optimal weights. In an extensive simulation study, we provide new insights into the performance of weighting estimators in large datasets, showing that the proposed approach substantially outperforms others in terms of accuracy and speed. Finally, we use this weighting approach to conduct a national study of the relationship between hospital profit status and heart attack outcomes in a comprehensive dataset of 1.27 million patients. We find that for-profit hospitals use interventional cardiology to treat heart attacks at similar rates as other hospitals but have higher mortality and readmission rates.

Training multi-source domain adaptation network by mutual information estimation and minimization.

We address the problem of Multi-Source Domain Adaptation (MSDA), which trains a neural network using multiple labeled source datasets and an unlabeled target dataset, and expects the trained network to well classify the unlabeled target data. The main challenge in this problem is that the datasets are generated by relevant but different joint distributions. In this paper, we propose to address this challenge by estimating and minimizing the mutual information in the network latent feature space, which leads to the alignment of the source joint distributions and target joint distribution simultaneously. Here, the estimation of the mutual information is formulated into a convex optimization problem, such that the global optimal solution can be easily found. We conduct experiments on several public datasets, and show that our algorithm statistically outperforms its competitors. Video and code are available at https://github.com/sentaochen/Mutual-Information-Estimation-and-Minimization.

MATLAB-based innovative 3D finite element method simulator for optimized real-time hyperthermia analysis.

BACKGROUND AND OBJECTIVE: Owing to the significant role of hyperthermia in enhancing the efficacy of chemotherapy or radiotherapy for treating malignant tissues, this study introduces a real-time hyperthermia simulator (RTHS) based on the three-dimensional finite element method (FEM) developed using the MATLAB App Designer. METHODS: The simulator consisted of operator-defined homogeneous and heterogeneous phantom models surrounded by an annular phased array (APA) of eight dipole antennas designed at 915 MHz. Electromagnetic and thermal analyses were conducted using the RTHS. To locally raise the target temperature according to the tumor's location, a convex optimization algorithm (COA) was employed to excite the antennas using optimal values of the phases to maximize the electric field at the tumor and amplitudes to achieve the required temperature at the target position. The performance of the proposed RTHS was validated by comparing it with similar hyperthermia setups in the FEM-based COMSOL software and finite-difference time-domain (FDTD)-based Sim4Life software. RESULTS: The simulation results obtained using the RTHS were consistent, both for the homogeneous and heterogeneous models, with those obtained using commercially available tools, demonstrating the reliability of the proposed hyperthermia simulator. The effectiveness of the simulator was illustrated for target positions in five different regions for both homogeneous and heterogeneous phantom models. In addition, the RTHS was cost-effective and consumed less computational time than the available software. The proposed method achieved 94% and 96% accuracy for element sizes of λ/26 and λ/36, respectively, for the homogeneous model. For the heterogeneous model, the method demonstrated 93% and 95% accuracy for element sizes of λ/26 and λ/36, respectively. The accuracy can be further improved by using a more refined mesh at the cost of a higher computational time. CONCLUSIONS: The proposed hyperthermia simulator demonstrated reliability, cost-effectiveness, and reduced computational time compared to commercial software, making it a potential tool for optimizing hyperthermia treatment.

On Robustness of Individualized Decision Rules.

With the emergence of precision medicine, estimating optimal individualized decision rules (IDRs) has attracted tremendous attention in many scientific areas. Most existing literature has focused on finding optimal IDRs that can maximize the expected outcome for each individual. Motivated by complex individualized decision making procedures and the popular conditional value at risk (CVaR) measure, we propose a new robust criterion to estimate optimal IDRs in order to control the average lower tail of the individuals' outcomes. In addition to improving the individualized expected outcome, our proposed criterion takes risks into consideration, and thus the resulting IDRs can prevent adverse events. The optimal IDR under our criterion can be interpreted as the decision rule that maximizes the "worst-case" scenario of the individualized outcome when the underlying distribution is perturbed within a constrained set. An efficient non-convex optimization algorithm is proposed with convergence guarantees. We investigate theoretical properties for our estimated optimal IDRs under the proposed criterion such as consistency and finite sample error bounds. Simulation studies and a real data application are used to further demonstrate the robust performance of our methods. Several extensions of the proposed method are also discussed.

Efficient model selection for predictive pattern mining model by safe pattern pruning.

Predictive pattern mining is an approach used to construct prediction models when the input is represented by structured data, such as sets, graphs, and sequences. The main idea behind predictive pattern mining is to build a prediction model by considering unified inconsistent notation sub-structures, such as subsets, subgraphs, and subsequences (referred to as patterns), present in the structured data as features of the model. The primary challenge in predictive pattern mining lies in the exponential growth of the number of patterns with the complexity of the structured data. In this study, we propose the safe pattern pruning method to address the explosion of pattern numbers in predictive pattern mining. We also discuss how it can be effectively employed throughout the entire model building process in practical data analysis. To demonstrate the effectiveness of the proposed method, we conduct numerical experiments on regression and classification problems involving sets, graphs, and sequences.

Multiresolution categorical regression for interpretable cell-type annotation.

In many categorical response regression applications, the response categories admit a multiresolution structure. That is, subsets of the response categories may naturally be combined into coarser response categories. In such applications, practitioners are often interested in estimating the resolution at which a predictor affects the response category probabilities. In this paper, we propose a method for fitting the multinomial logistic regression model in high dimensions that addresses this problem in a unified and data-driven way. Our method allows practitioners to identify which predictors distinguish between coarse categories but not fine categories, which predictors distinguish between fine categories, and which predictors are irrelevant. For model fitting, we propose a scalable algorithm that can be applied when the coarse categories are defined by either overlapping or nonoverlapping sets of fine categories. Statistical properties of our method reveal that it can take advantage of this multiresolution structure in a way existing estimators cannot. We use our method to model cell-type probabilities as a function of a cell's gene expression profile (i.e., cell-type annotation). Our fitted model provides novel biological insights which may be useful for future automated and manual cell-type annotation methodology.

Bayesian Trend Filtering via Proximal Markov Chain Monte Carlo.

Proximal Markov Chain Monte Carlo is a novel construct that lies at the intersection of Bayesian computation and convex optimization, which helped popularize the use of nondifferentiable priors in Bayesian statistics. Existing formulations of proximal MCMC, however, require hyperparameters and regularization parameters to be prespecified. In this work, we extend the paradigm of proximal MCMC through introducing a novel new class of nondifferentiable priors called epigraph priors. As a proof of concept, we place trend filtering, which was originally a nonparametric regression problem, in a parametric setting to provide a posterior median fit along with credible intervals as measures of uncertainty. The key idea is to replace the nonsmooth term in the posterior density with its Moreau-Yosida envelope, which enables the application of the gradient-based MCMC sampler Hamiltonian Monte Carlo. The proposed method identifies the appropriate amount of smoothing in a data-driven way, thereby automating regularization parameter selection. Compared with conventional proximal MCMC methods, our method is mostly tuning free, achieving simultaneous calibration of the mean, scale and regularization parameters in a fully Bayesian framework.

Enhancing neurodynamic approach with physics-informed neural networks for solving non-smooth convex optimization problems.

This paper proposes a deep learning approach for solving non-smooth convex optimization problems (NCOPs), which have broad applications in computer science, engineering, and physics. Our approach combines neurodynamic optimization with physics-informed neural networks (PINNs) to provide an efficient and accurate solution. We first use neurodynamic optimization to formulate an initial value problem (IVP) that involves a system of ordinary differential equations for the NCOP. We then introduce a modified PINN as an approximate state solution to the IVP. Finally, we develop a dedicated algorithm to train the model to solve the IVP and minimize the NCOP objective simultaneously. Unlike existing numerical integration methods, a key advantage of our approach is that it does not require the computation of a series of intermediate states to produce a prediction of the NCOP. Our experimental results show that this computational feature results in fewer iterations being required to produce more accurate prediction solutions. Furthermore, our approach is effective in finding feasible solutions that satisfy the NCOP constraint.

Resource Allocation for Secure MIMO-SWIPT Systems in the Presence of Multi-Antenna Eavesdropper in Vehicular Networks.

In this paper, we optimize the secrecy capacity of the legitimate user under resource allocation and security constraints for a multi-antenna environment for the simultaneous transmission of wireless information and power in a dynamic downlink scenario. We study the relationship between secrecy capacity and harvested energy in a power-splitting configuration for a nonlinear energy-harvesting model under co-located conditions. The capacity maximization problem is formulated for the vehicle-to-vehicle communication scenario. The formulated problem is non-convex NP-hard, so we reformulate it into a convex form using a divide-and-conquer approach. We obtain the optimal transmit power matrix and power-splitting ratio values that guarantee positive values of the secrecy capacity. We analyze different vehicle-to-vehicle communication settings to validate the differentiation of the proposed algorithm in maintaining both reliability and security. We also substantiate the effectiveness of the proposed approach by analyzing the trade-offs between secrecy capacity and harvested energy.

Morphological component analysis under non-convex smoothing penalty framework for gearbox fault diagnosis.

The sparse representation methodology has been identified to be a promising tool for gearbox fault diagnosis. The core is how to precisely reconstruct the fault signal from noisy monitoring signals. The non-convex penalty has the ability to induce sparsity more efficiently than convex penalty. However, the introduction of non-convex penalty usually influences the convexity of the model, resulting in the unstable or sub-optimal solution. In this paper, we propose the non-convex smoothing penalty framework (NSPF) and combine it with morphological component analysis (MCA) for gearbox fault diagnosis. The proposed NSPF is a unify penalty construction framework, which contains many classical penalty while a new set of non-convex smoothing penalty functions can be generated. These non-convex penalty can guarantee the convexity of the objective function while enhancing the sparsity, thus the global optimal solution can be acquired. The simulation and engineering experiments validate that the NSPF enjoys more reconstruction precision compared to the existing penalties.