A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations.

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and to accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. More than three thousand pruning papers have been published from 2020 to 2024. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning (e.g., unstructured/structured, one-shot/iterative, data-free/data-driven, initialized/pre-trained weights, etc.) and explore several emerging topics, including pruning for large language models, vision transformers, diffusion models, and large multimodal models, post-training pruning, and different levels of supervision for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. Finally, we provide some valuable recommendations on selecting pruning methods and prospect several promising research directions for neural network pruning. To facilitate future research on deep neural network pruning, we summarize broad pruning applications (e.g., adversarial robustness, natural language understanding, etc.) and build a curated collection of datasets, networks, and evaluations on different applications. We maintain a repository on https://github.com/hrcheng1066/awesome-pruning that serves as a comprehensive resource for neural network pruning papers and corresponding open-source codes. We will keep updating this repository to include the latest advancements in the field.

Machine Learning Big Data Set Analysis Reveals C-C Electro-Coupling Mechanism.

Carbon-carbon (C-C) coupling is essential in the electrocatalytic reduction of CO2 for the production of green chemicals. However, due to the complexity of the reaction network, there remains controversy regarding the underlying reaction mechanisms and the optimal direction for catalyst material design. Here, we present a global perspective to establish a comprehensive data set encompassing all C-C coupling precursors and catalytic active site compositions to explore the reaction mechanisms and screen catalysts via big data set analysis. The 2D-3D ensemble machine learning strategy, developed to target a variety of adsorption configurations, can quickly and accurately expand quantum chemical calculation data, enabling the rapid acquisition of this extensive big data set. Analyses of the big data set establish that (1) asymmetric coupling mechanisms exhibit greater potential efficiency compared to symmetric coupling, with the optimal path involving the coupling CHO with CH or CH2, and (2) C-C coupling selectivity of Cu-based catalysts can be enhanced through bimetallic doping including CuAgNb sites. Importantly, we experimentally substantiate the CuAgNb catalyst to demonstrate actual boosted performance in C-C coupling. Our finding evidence the practicality of our big data set generated from machine learning-accelerated quantum chemical computations. We conclude that combining big data with complex catalytic reaction mechanisms and catalyst compositions will set a new paradigm for accelerating optimal catalyst design.

Influence Function Based Second-Order Channel Pruning: Evaluating True Loss Changes For Pruning Is Possible Without Retraining.

Channel pruning is attracting increasing attention in the deep model compression community due to its capability of significantly reducing computation and memory footprints without special support from specific software and hardware. A challenge of channel pruning is designing efficient and effective criteria to select channels to prune. A widely used criterion is minimal performance degeneration, e.g., loss changes before and after pruning being the smallest. To accurately evaluate the truth performance degeneration requires retraining the survived weights to convergence, which is prohibitively slow. Hence existing pruning methods settle to use previous weights (without retraining) to evaluate the performance degeneration. However, we observe that the loss changes differ significantly with and without retraining. It motivates us to develop a technique to evaluate true loss changes without retraining, using which to select channels to prune with more reliability and confidence. We first derive a closed-form estimator of the true loss change per mask change, using influence functions without retraining. Influence function is a classic technique from robust statistics that reveals the impacts of a training sample on the model's prediction and is repurposed by us to assess impacts on true loss changes. We then show how to assess the importance of all channels simultaneously and develop a novel global channel pruning algorithm accordingly. We conduct extensive experiments to verify the effectiveness of the proposed algorithm, which significantly outperforms the competing channel pruning methods on both image classification and object detection tasks. One of the attractive properties of our algorithm is that it automatically obtains the prune percentage without the cumbersome yet commonly used sensitivity analysis by local pruning. To the best of our knowledge, we are the first that shows evaluating true loss changes for pruning without retraining is possible. This finding will open up opportunities for a series of new paradigms to emerge that differ from existing pruning methods. The code is available at https://github.com/hrcheng1066/IFSO.

Machine Learning Confirms the Formation Mechanism of a Single-Atom Catalyst via Infrared Spectroscopic Analysis.

Single-atom catalysts (SACs) offer significant potential across various applications, yet our understanding of their formation mechanism remains limited. Notably, the pyrolysis of zeolitic imidazolate frameworks (ZIFs) stands as a pivotal avenue for SAC synthesis, of which the mechanism can be assessed through infrared (IR) spectroscopy. However, the prevailing analysis techniques still rely on manual interpretation. Here, we report a machine learning (ML)-driven analysis of the IR spectroscopy to unravel the pyrolysis process of Pt-doped ZIF-67 to synthesize Pt-Co3O4 SAC. Demonstrating a total Pearson correlation exceeding 0.7 with experimental data, the algorithm provides correlation coefficients for the selected structures, thereby confirming crucial structural changes with time and temperature, including the decomposition of ZIF and formation of Pt-O bonds. These findings reveal and confirm the formation mechanism of SACs. As demonstrated, the integration of ML algorithms, theoretical simulations, and experimental spectral analysis introduces an approach to deciphering experimental characterization data, implying its potential for broader adoption.

Discovery of Graphene Growth Alloy Catalysts Using High-Throughput Machine Learning.

Despite today's commercial-scale graphene production using chemical vapor deposition (CVD), the growth of high-quality single-layer graphene with controlled morphology and crystallinity remains challenging. Considerable effort is still spent on designing improved CVD catalysts for producing high-quality graphene. Conventionally, however, catalyst design has been pursued using empirical intuition or trial-and-error approaches. Here, we combine high-throughput density functional theory and machine learning to identify new prospective transition metal alloy catalysts that exhibit performance comparable to that of established graphene catalysts, such as Ni(111) and Cu(111). The alloys identified through this process generally consist of combinations of early- and late-transition metals, and a majority are alloys of Ni or Cu. Nevertheless, in many cases, these conventional catalyst metals are combined with unconventional partners, such as Zr, Hf, and Nb. The approach presented here therefore highlights an important new approach for identifying novel catalyst materials for the CVD growth of low-dimensional nanomaterials.

Improved genomic prediction using machine learning with Variational Bayesian sparsity.

BACKGROUND: Genomic prediction has become a powerful modelling tool for assessing line performance in plant and livestock breeding programmes. Among the genomic prediction modelling approaches, linear based models have proven to provide accurate predictions even when the number of genetic markers exceeds the number of data samples. However, breeding programmes are now compiling data from large numbers of lines and test environments for analyses, rendering these approaches computationally prohibitive. Machine learning (ML) now offers a solution to this problem through the construction of fully connected deep learning architectures and high parallelisation of the predictive task. However, the fully connected nature of these architectures immediately generates an over-parameterisation of the network that needs addressing for efficient and accurate predictions. RESULTS: In this research we explore the use of an ML architecture governed by variational Bayesian sparsity in its initial layers that we have called VBS-ML. The use of VBS-ML provides a mechanism for feature selection of important markers linked to the trait, immediately reducing the network over-parameterisation. Selected markers then propagate to the remaining fully connected feed-forward components of the ML network to form the final genomic prediction. We illustrated the approach with four large Australian wheat breeding data sets that range from 2665 lines to 10375 lines genotyped across a large set of markers. For all data sets, the use of the VBS-ML architecture improved genomic prediction accuracy over legacy linear based modelling approaches. CONCLUSIONS: An ML architecture governed under a variational Bayesian paradigm was shown to improve genomic prediction accuracy over legacy modelling approaches. This VBS-ML approach can be used to dramatically decrease the parameter burden on the network and provide a computationally feasible approach for improving genomic prediction conducted with large breeding population numbers and genetic markers.

SharpFormer: Learning Local Feature Preserving Global Representations for Image Deblurring.

The goal of dynamic scene deblurring is to remove the motion blur presented in a given image. To recover the details from the severe blurs, conventional convolutional neural networks (CNNs) based methods typically increase the number of convolution layers, kernel-size, or different scale images to enlarge the receptive field. However, these methods neglect the non-uniform nature of blurs, and cannot extract varied local and global information. Unlike the CNNs-based methods, we propose a Transformer-based model for image deblurring, named SharpFormer, that directly learns long-range dependencies via a novel Transformer module to overcome large blur variations. Transformer is good at learning global information but is poor at capturing local information. To overcome this issue, we design a novel Locality preserving Transformer (LTransformer) block to integrate sufficient local information into global features. In addition, to effectively apply LTransformer to the medium-resolution features, a hybrid block is introduced to capture intermediate mixed features. Furthermore, we use a dynamic convolution (DyConv) block, which aggregates multiple parallel convolution kernels to handle the non-uniform blur of inputs. We leverage a powerful two-stage attentive framework composed of the above blocks to learn the global, hybrid, and local features effectively. Extensive experiments on the GoPro and REDS datasets show that the proposed SharpFormer performs favourably against the state-of-the-art methods in blurred image restoration.

Human Interaction Understanding With Consistency-Aware Learning.

Compared with the progress made on human activity classification, much less success has been achieved on human interaction understanding (HIU). Apart from the latter task is much more challenging, the main causation is that recent approaches learn human interactive relations via shallow graphical representations, which are inadequate to model complicated human interactive-relations. This paper proposes a deep consistency-aware framework aiming at tackling the grouping and labelling inconsistencies in HIU. This framework consists of three components, including a backbone CNN to extract image features, a factor graph network to implicitly learn higher-order consistencies among labelling and grouping variables, and a consistency-aware reasoning module to explicitly enforcing consistencies. The last module is inspired by our key observation that the consistency-aware reasoning bias can be embedded into an energy function or a particular loss function, minimizing which delivers consistent predictions. An efficient mean-field inference algorithm is proposed, such that all modules of our network could be trained in an end-to-end fashion. Experimental results demonstrate that the two proposed consistency-learning modules complement each other, and both make considerable contributions in achieving leading performance on three benchmarks of HIU. The effectiveness of the proposed approach is further validated by experiments on detecting human-object interactions.

Learn to Predict Sets Using Feed-Forward Neural Networks.

This paper addresses the task of set prediction using deep feed-forward neural networks. A set is a collection of elements which is invariant under permutation and the size of a set is not fixed in advance. Many real-world problems, such as image tagging and object detection, have outputs that are naturally expressed as sets of entities. This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors. We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks. In our formulation we define a likelihood for a set distribution represented by a) two discrete distributions defining the set cardinally and permutation variables, and b) a joint distribution over set elements with a fixed cardinality. Depending on the problem under consideration, we define different training models for set prediction using deep neural networks. We demonstrate the validity of our set formulations on relevant vision problems such as: 1) multi-label image classification where we outperform the other competing methods on the PASCAL VOC and MS COCO datasets, 2) object detection, for which our formulation outperforms popular state-of-the-art detectors, and 3) a complex CAPTCHA test, where we observe that, surprisingly, our set-based network acquired the ability of mimicking arithmetics without any rules being coded.