Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Blended coarse gradient descent for full quantization of deep neural networks

Blended coarse gradient descent for full quantization of deep neural networks Quantized deep neural networks (QDNNs) are attractive due to their much lower memory storage and faster inference speed than their regular full-precision counterparts. To maintain the same performance level especially at low bit-widths, QDNNs must be retrained. Their training involves piecewise constant activation functions and discrete weights; hence, mathematical challenges arise. We introduce the notion of coarse gradient and propose the blended coarse gradient descent (BCGD) algorithm, for training fully quantized neural networks. Coarse gradient is generally not a gradient of any function but an artificial ascent direction. The weight update of BCGD goes by coarse gradient correction of a weighted average of the full-precision weights and their quantization (the so-called blending), which yields sufficient descent in the objective value and thus accelerates the training. Our experiments demonstrate that this simple blending technique is very effective for quantization at extremely low bit-width such as binarization. In full quantization of ResNet-18 for ImageNet classification task, BCGD gives 64.36% top-1 accuracy with binary weights across all layers and 4-bit adaptive activation. If the weights in the first and last layers are kept in full precision, this number increases to 65.46%. As theoretical justification, we show convergence analysis of coarse gradient descent for a two-linear-layer neural network model with Gaussian input data and prove that the expected coarse gradient correlates positively with the underlying true gradient. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Research in the Mathematical Sciences Springer Journals

Blended coarse gradient descent for full quantization of deep neural networks

Download PDF
23 pages

Loading next page...
 
/lp/springer-journals/blended-coarse-gradient-descent-for-full-quantization-of-deep-neural-bzj0w0flLy
Publisher
Springer Journals
Copyright
Copyright © 2019 by Springer Nature Switzerland AG
Subject
Mathematics; Mathematics, general; Applications of Mathematics; Computational Mathematics and Numerical Analysis
eISSN
2197-9847
DOI
10.1007/s40687-018-0177-6
Publisher site
See Article on Publisher Site

Abstract

Quantized deep neural networks (QDNNs) are attractive due to their much lower memory storage and faster inference speed than their regular full-precision counterparts. To maintain the same performance level especially at low bit-widths, QDNNs must be retrained. Their training involves piecewise constant activation functions and discrete weights; hence, mathematical challenges arise. We introduce the notion of coarse gradient and propose the blended coarse gradient descent (BCGD) algorithm, for training fully quantized neural networks. Coarse gradient is generally not a gradient of any function but an artificial ascent direction. The weight update of BCGD goes by coarse gradient correction of a weighted average of the full-precision weights and their quantization (the so-called blending), which yields sufficient descent in the objective value and thus accelerates the training. Our experiments demonstrate that this simple blending technique is very effective for quantization at extremely low bit-width such as binarization. In full quantization of ResNet-18 for ImageNet classification task, BCGD gives 64.36% top-1 accuracy with binary weights across all layers and 4-bit adaptive activation. If the weights in the first and last layers are kept in full precision, this number increases to 65.46%. As theoretical justification, we show convergence analysis of coarse gradient descent for a two-linear-layer neural network model with Gaussian input data and prove that the expected coarse gradient correlates positively with the underlying true gradient.

Journal

Research in the Mathematical Sciences – Springer Journals

Published: Jan 2, 2019

References

  • Nonlinear Programming
    Bertsekas, DP
  • Stochastic approximations and perturbations in forward–backward splitting for monotone operators
    Combettes, PL; Pesquet, JC
  • Large margin classification using the perceptron algorithm
    Freund, Y; Schapire, RE
  • Global convergence properties of conjugate gradient methods for optimization
    Gilbert, JC; Nocedal, J
  • Quantized neural networks: training neural networks with low precision weights and activations
    Hubara, I; Courbariaux, M; Soudry, D; El-Yaniv, R; Bengio, Y
  • Least squares quantization in PCM
    Lloyd, S
  • The Perceptron, a Perceiving and Recognizing Automaton Project Para
    Rosenblatt, F
  • Principles of Neurodynamics
    Rosenblatt, F
  • 30 years of adaptive neural networks: perceptron, madaline, and backpropagation
    Widrow, B; Lehr, MA