Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Quantized convolutional neural networks through the lens of partial differential equations

Quantized convolutional neural networks through the lens of partial differential equations Quantization of convolutional neural networks (CNNs) is a common approach to ease the computational burden involved in the deployment of CNNs, especially on low-resource edge devices. However, fixed-point arithmetic is not natural to the type of computations involved in neural networks. In this work, we explore ways to improve quantized CNNs using PDE-based perspective and analysis. First, we harness the total variation (TV) approach to apply edge-aware smoothing to the feature maps throughout the network. This aims to reduce outliers in the distribution of values and promote piecewise constant maps, which are more suitable for quantization. Secondly, we consider symmetric and stable variants of common CNNs for image classification and graph convolutional networks for graph node classification. We demonstrate through several experiments that the property of forward stability preserves the action of a network under different quantization rates. As a result, stable quantized networks behave similarly to their non-quantized counterparts even though they rely on fewer parameters. We also find that at times, stability even aids in improving accuracy. These properties are of particular interest for sensitive, resource-constrained, low-power or real-time applications like autonomous driving. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Research in the Mathematical Sciences Springer Journals

Quantized convolutional neural networks through the lens of partial differential equations

Download PDF
21 pages

Loading next page...
 
/lp/springer-journals/quantized-convolutional-neural-networks-through-the-lens-of-partial-0SsWMHmYUE
Publisher
Springer Journals
Copyright
Copyright © The Author(s), under exclusive licence to Springer Nature Switzerland AG 2022. Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
eISSN
2197-9847
DOI
10.1007/s40687-022-00354-y
Publisher site
See Article on Publisher Site

Abstract

Quantization of convolutional neural networks (CNNs) is a common approach to ease the computational burden involved in the deployment of CNNs, especially on low-resource edge devices. However, fixed-point arithmetic is not natural to the type of computations involved in neural networks. In this work, we explore ways to improve quantized CNNs using PDE-based perspective and analysis. First, we harness the total variation (TV) approach to apply edge-aware smoothing to the feature maps throughout the network. This aims to reduce outliers in the distribution of values and promote piecewise constant maps, which are more suitable for quantization. Secondly, we consider symmetric and stable variants of common CNNs for image classification and graph convolutional networks for graph node classification. We demonstrate through several experiments that the property of forward stability preserves the action of a network under different quantization rates. As a result, stable quantized networks behave similarly to their non-quantized counterparts even though they rely on fewer parameters. We also find that at times, stability even aids in improving accuracy. These properties are of particular interest for sensitive, resource-constrained, low-power or real-time applications like autonomous driving.

Journal

Research in the Mathematical Sciences – Springer Journals

Published: Dec 1, 2022

Keywords: Quantization; Convolutional neural networks; Total variation; Neural ODEs; Stable symmetric architectures; 68T07; 68T10; 68T45

References

  • Approximation of functional depending on jumps by elliptic functional via t-convergence
    Ambrosio, L; Tortorelli, VM
  • Optimization methods for large-scale machine learning
    Bottou, L; Curtis, FE; Nocedal, J
  • Active contours without edges
    Chan, T; Vese, L
  • Deep relaxation: partial differential equations for optimizing deep neural networks
    Chaudhari, P; Oberman, A; Osher, S; Soatto, S; Carlier, G
  • A survey of accelerator architectures for deep neural networks
    Chen, Y; Xie, Y; Song, L; Chen, F; Tang, T
  • Model compression and acceleration for deep neural networks: the principles, progress, and challenges
    Cheng, Y; Wang, D; Zhou, P; Zhang, T
  • LeanConvNets: low-cost yet effective convolutional neural networks
    Ephrath, J; Eliasof, M; Ruthotto, L; Haber, E; Treister, E
  • Making machine learning robust against adversarial inputs
    Goodfellow, I; McDaniel, P; Papernot, N
  • Layer-parallel training of deep residual neural networks
    Gunther, S; Ruthotto, L; Schroder, JB; Cyr, EC; Gauger, NR
  • Determining optical flow
    Horn, BKP; Schunck, BG
  • Quantized neural networks: training neural networks with low precision weights and activations
    Hubara, I; Courbariaux, M; Soudry, D; El-Yaniv, R; Bengio, Y
  • ImageNet classification with deep convolutional neural networks
    Krizhevsky, A; Sutskever, I; Hinton, G
  • Scale-space and edge detection using anisotropic diffusion
    Perona, P; Malik, J
  • Nonlinear total variation based noise removal algorithms
    Rudin, L; Osher, S; Fatemi, E
  • Collective classification in network data
    Sen, P; Namata, G; Bilgic, M; Getoor, L; Galligher, B; Eliassi-Rad, T
  • A proposal on machine learning via dynamical systems
    Weinan, E
  • Blended coarse gradient descent for full quantization of deep neural networks
    Yin, P; Zhang, S; Lyu, J; Osher, S; Qi, Y; Xin, J
  • Forward stability of ResNet and its variants
    Zhang, L; Schaeffer, H
  • Improving neural network quantization without retraining using outlier channel splitting
    Zhao, R; Hu, Y; Dotzel, J; De Sa, C; Zhang, Z