Adaptive Divergence for Rapid Adversarial Optimization
Adaptive Divergence for Rapid Adversarial Optimization
Borisyak, Maxim;Gaintseva, Tatiana;Ustyuzhanin, Andrey
2019-12-01 00:00:00
Adaptive Divergence for Rapid Adversarial Optimization 1 1 1, 2, 3 Maxim Borisyak , Tatiana Gaintseva , and Andrey Ustyuzhanin Laboratory of Methods for Big Data Analysis, National Research University Higher School of Economics, 20 Myasnitskaya st, Moscow 101000, Russia Physics department, Imperial College, South Kensington, London SW7 2AZ, United Kingdom Department of multidisciplinary research, National University of Science and Technology MISIS, 4 Leninsky av, Moscow 119049, Russia Corresponding author: Maxim Borisyak Email address: mborisyak@hse.ru ABSTRACT Adversarial Optimization (AO) provides a reliable, practical way to match two implicitly defined distributions, one of which is usually represented by a sample of real data, and the other is defined by a generator. Typically, AO involves training of a high-capacity model on each step of the optimization. In this work, we consider computationally heavy generators, for which training of high-capacity models is associated with substantial computational costs. To address this problem, we introduce a novel family of divergences, which varies the capacity of the underlying model, and allows for a significant acceleration with respect to the number of samples drawn from the generator. We demonstrate the performance of the proposed divergences on several tasks, including tuning parame- ters of a physics simulator, namely, Pythia event generator. 1 INTRODUCTION Adversarial Optimization (AO), introduced in Generative Adversarial Networks (Goodfellow et al., 2014), became popular in many areas of machine learning and beyond with applications ranging from genera- tive (Radford et al., 2015) and inference tasks (Dumoulin et al., 2016), improving image quality (Isola et al., 2017) to tuning stochastic computer simulations (Louppe et al., 2017). AO provides a reliable, practical way to match two implicitly defined distributions, one of which is typically represented by a sample of real data, and the other is represented by a parameterized generator. Matching of the distributions is achieved by minimizing a divergence between these distribution, and estimation of the divergence involves a secondary optimization task, which, typically, requires training a model to discriminate between these distributions. The model is referred to as discriminator or critic (for simplicity, we use term discriminator everywhere below). Training a high-capacity model, however, is computationally expensive (Metz et al., 2016) as each step of divergence minimization is accompanied by fitting the discriminator; therefore, adversarial training often requires significantly more computational resources than, for example, a classification model with a comparable architecture of the networks . Nevertheless, in conventional settings like GAN, this problem is not pronounced for at least two reasons. Firstly, the generator is usually represented by a deep neural network, and sampling is computationally cheap; thus, for properly training the discriminator, a training sample of sufficient size can be quickly drawn. Secondly, GAN training procedures are often regarded not as minimization of a divergence, but as game-like dynamics (Li et al., 2017; Mescheder et al., 2018); such dynamics typically employ gradient optimization with small incremental steps, which involve relatively small sample sizes for adapting previous discriminator to an updated generator configuration. For instance, compare training times, network capacities and computational resources reported by Simonyan and Zisserman (2014) and Choi et al. (2018). arXiv:1912.00520v1 [cs.LG] 1 Dec 2019 Computational costs of AO becomes significant when sampling from the generator is computationally expensive, or optimization procedure does not operate by performing small incremental steps (Metz et al., 2016). One of the practical examples of such settings is fine-tuning parameters of complex computer simulations. Such simulators are usually based on physics laws expressed in computational mathematical forms like differential or stochastic equations. Those equations relate input or initial conditions to the observable quantities under conditions of parameters that define physics laws, geometry or other valuable property of the simulation; these parameters do not depend on inputs or initial conditions. It is not uncommon that such simulations have very high computational complexity. For example, the simulation of a single proton collision event in the CERN ATLAS detector takes several minutes on a single core CPU (Bouhova-Thacker et al., 2010). Due to typically high dimensionality, it takes a considerable amount of samples for fine-tuning, which in turn increases the computational burden. Another essential property of such computer simulations is the lack of gradient information over the simulation parameters. Computations are represented by sophisticated computer programs, which are challenging to differentiate . Thus, global black-box optimization methods are often employed; Bayesian Optimization is one of the most popular approaches. In this work, we introduce a novel family of divergences which enables faster optimization conver- gence measured by the number of samples drawn from the generator. The variation of the underlying discriminator model capacity during optimization leads to a significant speed-up. The proposed diver- gence family suggests using low-capacity models to compare distant distributions (typically, at early optimization steps), and the capacity gradually grows as the distributions become closer to each other. Thus it allows for a significant acceleration of the initial stages of optimization. Additionally, the proposed family of divergences is broad, which offers a wide range of opportunities for further research. We demonstrate the basic idea with some toy examples, and with a real challenge of tuning Pythia event generator (Sjostrand ¨ et al., 2006, 2015) following Louppe et al. (2017) and Ilten et al. (2017). We consider physics-related simulations; nevertheless, all proposed methods are simulation-agnostic. 2 BACKGROUND Adversarial Optimization, initially introduced for Generative Adversarial Networks (GAN) (Goodfellow et al., 2014), offers a general strategy for matching two distributions. Consider feature space X , ground- truth distribution P and parametrized family of distributions Q implicitly defined by a generator with parameters y . Formally, we wish to find such y , that P = Q almost everywhere. AO achieves that by minimizing a divergence or a distance between P and Q with respect to y . One of the most popular divergences is Jensen-Shannon divergence: JSD(P; Q ) = KL(PkM )+ KL(Q kM ) = y y y y 1 1 log 2 min E log f (x) E log(1 f (x)) = xP xQ f2F 2 2 log 2 min L( f ; P; Q ); (1) f2F where: KL — Kullback-Leibler divergence, M (x) = P(x)+ Q (x) , L — cross-entropy loss function, y y and F =f f : X ! [0; 1]g is the set of all possible discriminators. Expression (1) provides a practical way to estimate Jensen-Shannon divergence by training a powerful discriminator to distinguish samples from P against samples from Q . In classical GAN optimization, both generator and discriminator are represented by differentiable neural networks. Hence, a subgradient of JSD(P; Q ) can be easily computed (Goodfellow et al., 2014). The minimization of the divergence can be performed by a gradient method, and the optimization procedure goes iteratively following those steps: using parameters of the discriminator from the previous iteration as an initial guess, adjust f by performing several steps of the gradient descent to minimize L ( f ; P; Q ); There are ways to estimate gradients of such programs, for example, see (Baydin et al., 2018). However, all methods known to the authors require training a surrogate, which encounters the problem of the expensive sampling procedures mentioned above. 2/17 considering f as a constant, compute the gradient of L ( f ; P; Q ) by y , perform one step of the gradient ascent. For computationally heavy generators, gradients are usually practically unfeasible; therefore, we consider black-box optimization methods. One of the most promising methods for black-box AO is Adversarial Variational Optimization (Louppe et al., 2017), which combines AO with Variational Optimization (Wierstra et al., 2014). This method improves upon conventional Variational Optimization (VO) over Jensen-Shannon divergence by training a single discriminator to distinguish samples from ground-truth distribution and samples from a mixture of generators, where the mixture is defined by the search distribution of VO. This eliminates the need to train a classifier for each individual set of parameters drawn from the search distribution. Bayesian Optimization (BO) (Mockus, 2012) is another commonly used black-box optimization method, with applications including tuning of complex simulations (Ilten et al., 2017). As we demonstrate in section 5, BO can be successfully applied for Adversarial Optimization. 3 ADAPTIVE DIVERGENCE Notice, that in Equation (1), minimization is carried over the set of all possible discriminators F = f f : X 7! [0; 1]g. In practice, this is intractable and set F is approximated by a model such as Deep Neural Networks. Everywhere below, we use terms ’low-capacity’ and ’high-capacity’ to describe the set of feasible discriminator functions: low-capacity models are either represent a narrow set of functions (e.g., logistic regression, shallow decision trees) or are heavily regularized (see Section 4 for more examples of capacity regulation); high-capacity models are sufficient for estimating JSD for an Adversarial Optimization problem under consideration. In conventional GAN settings, the generator is represented by a neural network, sampling is computa- tionally cheap, and usage of high-capacity discriminators is satisfactory. In our case, as was discussed above, simulations tend to be computationally heavy, which, combined with a typically slow convergence of black-box optimization algorithms, might make AO with a high-capacity model practically intractable. The choice of the model has its trade-off: high-capacity models provide good estimations of JSD, but, generally, require large sample sizes to be properly trained. In contrast, low-capacity models tend to require fewer samples for training; however, they might provide biased estimations. For example, if the classifier is represented by a narrow set of functions M F , then quantity: D (P; Q) = log 2 min L( f ; P; Q); (2) f2M might no longer be a divergence, so we refer to it as pseudo-divergence. Definition 1. A function D : P(X )P(X )! R is a pseudo-divergence, if: (P1) 8P; Q2 P(X ) : D(P; Q) 0; (P2) 8P; Q2 P(X ) : (P = Q)) D(P; Q) = 0; where P(X ) — set of all probability distributions on space X . It is tempting to use a pseudo-divergence D produced by a low-capacity model M for Adversarial Optimization, however, a pseudo-divergence might not guarantee proper convergence as there might exist such y 2 Y, that JSD(P; Q ) > 0, while D(P; Q ) = 0. For example, naive Bayes classifier is unable y y to distinguish between P and Q that have the same marginal distributions. Nevertheless, if model M is capable of distinguishing between P and some Q , D still provides information about the position of the y M optimal parameters in the configuration space y by narrowing search volume, Ilten et al. (2017) offers a good demonstration of this statement. The core idea of this work is to replace Jensen-Shannon divergence with a so-called adaptive diver- gence that gradually adjusts model capacity depending on the ’difficulty’ of the classification problem with the most ’difficult’ problem being distinguishing between two equal distributions. Formally, this gradual increase of model complexity can be captured by the following definitions. Definition 2. A family of pseudo-divergences D =fD : P(X )P(X )! Rj a 2 [0; 1]g is ordered and complete with respect to Jensen-Shannon divergence if: 3/17 (D0) D is a pseudo-divergence for all a 2 [0; 1]; (D1) 8P; Q2 P(X ) :80 a < a 1 : D (P; Q) D (P; Q); 1 2 a a 1 2 (D2) 8P; Q2 P(X ) : D (P; Q) = JSD(P; Q). There are numerous ways to construct a complete and ordered w.r.t. JSD family of pseudo-divergences. In the context of Adversarial Optimization, we consider the following three methods. The simplest one is to define a nested family of models M =fM F j a 2 [0; 1]g, (e.g., by changing number of hidden units of a neural network), then use pseudo-divergence (2) to form a desired family. Alternatively, for a parameterized model M =f f (q;)j q 2 Qg, one can use a regularization R(q) to control ’capacity’ of the model: D (P; Q) = log 2 L( f (q ;); P; Q); (3) q = arg min L( f (q;); P; Q)+ c(1 a) R(q); q2Q where c : [0; 1]! [0;+¥) is a strictly increasing function and c(0) = 0. The third, boosting-based method is applicable for a discrete approximation: D (P; Q) = log 2 L(F ; P; Q); (4) c(i) i F = F + r arg min L(F + f ; P; Q); i i 1 i 1 f2B F ; where: r — learning rate, B — base estimator, c : Z ! [0; 1] — a strictly increasing function for mapping ensemble size onto a 2 [0; 1]. Although Definition 2 is quite general, in this paper, we focus on families of pseudo-divergence produced in the manner similar to the examples above. All these examples introduce a classification algo- rithm parameterized by a , then define pseudo-divergences D by substituting the optimal discriminator in Equation (1) with the discriminator trained in accordance with this classification algorithm with the parameter a . Of course, one has to make sure that the resulting family of pseudo-divergences is ordered and complete w.r.t. Jensen-Shannon divergence. Appendix B provides formal definitions and proofs for the examples above. With this class of pseudo-divergences in mind, we refer to a as capacity of the pseudo-divergence D 2 D relative to the family D , or simply as capacity if the family D is clear from the context. In the examples above, capacity of pseudo-divergence is directly linked to the capacity of underlying discriminator models: to the size of the model in equation (2), to the strength of the regularization in equation (3) (which, similar to the previous case, effectively restricts the size of the set of feasible models) or to the size of the ensemble for a boosting-based family of divergences in equation (4). Finally, we introduce a function that combines a family of pseudo-divergences into a single divergence. Definition 3. If a family of pseudo-divergences D = fD j a 2 [0; 1]g is ordered and complete with respect to Jensen-Shannon divergence, then adaptive divergence AD produced by D is defined as: AD (P; Q) = inffD (P; Q)j D (P; Q) (1 a) log 2g: (5) D a a We omit index in AD when the family D is clear from the context or is not important. Note, that due to property (D1), D (P; Q) is a non-decreasing function of a , while (1 a) log 2 is a strictly decreasing one. Hence, if family D is such that for any two distributions P and Q D (P; Q) is continuous w.r.t. a , equation (5) can be simplified: AD (P; Q) = D (P; Q); (6) D a where a is the root of the following equation: D (P; Q) = (1 a) log 2: (7) A general procedure for computing AD for this case is outlined in algorithm 1. 4/17 Algorithm 1 General procedure for computing an adaptive divergence by grid search Require: D =fD j a 2 [0; 1]g — ordered and complete w.r.t. Jensen-Shannon divergence family of pseudo-divergences; e — tolerance; P, Q — input distributions a 0; while D (P; Q) < (1 a) log 2 do a a + e end while return D (P; Q) Intuitively, an adaptive divergence AD switches between members of D depending on the ‘difficulty’ of separating P and Q. For example, consider family D produced by equation (3) with a high-capacity neural network as model M and l regularization R on its weights. For a pair of distant P and Q, even a highly regularized network is capable of achieving low cross-entropy loss and, therefore, AD takes values of the pseudo-divergence based on such network. As distribution Q moves close to P, AD lowers the regularization coefficient, effectively increasing the capacity of the underlying model. The idea behind adaptive divergences can be viewed from a different angle. Given two distributions P and Q, it scans producing family of pseudo-divergences, starting from a = 0 (the least powerful pseudo-divergence), and if some pseudo-divergence reports high enough value, it serves as a ‘proof’ of differences between P and Q. If all pseudo-divergences from the family D report 0, then P and Q are equal almost everywhere as the family always includes JSD as a member. Formally, this intuition can be expressed with the following theorem. Theorem 1. If AD is an adaptive divergence produced by a ordered and complete with respect to Jensen-Shannon divergence family of pseudo-divergences D , then for any two distributions P and Q: JSD(P; Q) = 0 if and only if AD(P; Q) = 0. A formal proof of Theorem 1 can be found in Appendix A. Combined with the observation that AD(P; Q) 0 regardless of P and Q, the theorem states that AD is a divergence in the same sense as JSD. This, in turn, allows the use of adaptive divergences as a replacement for Jensen-Shannon divergence in Adversarial Optimization. As can be seen from the definition, adaptive divergences are designed to utilize low-capacity pseudo- divergences (with underlying low-capacity models) whenever it is possible: for a pair of distant P and Q one needs to train only a low-capacity model to estimate AD, using the most powerful model only to prove equality of distributions. As low-capacity models generally require fewer samples for training, AD allows an optimization algorithm to run for more iterations within the same time restrictions. Properties of AD highly depend on the family D , and choice of the latter might either negatively or positively impact convergence of a particular optimization algorithm. Figure 1 demonstrates both cases: here, we evaluate JSD and four variants of AD on two synthetic examples. In each example, the generator produces a rotated version of the ground-truth distribution and is parameterized by the angle of rotation (ground-truth distributions and examples of generator distributions are shown in Fig. 1a and Fig. 1d). In Fig. 1b and Fig. 1c AD shows behaviour similar to that of JSD (both being monotonous and maintaining a significant slope in the respective ranges). In Fig. 1e, both variants of AD introduce an additional local minimum, which is expected to impact convergence of gradient-based algorithms negatively. In contrast, in Fig. 1f neural-network-based AD with l regularization stays monotonous in the range [0;p=2] and keeps a noticeable positive slope, in contrast to saturated JSD. The positive slope is expected to improve convergence of gradient-based algorithms and, possibly, some variants of Bayesian Optimization. In contrast, neural-network-based AD with dropout regularization behaves in a manner similar to adaptive divergences in Fig. 1e. 4 IMPLEMENTATION A general algorithm for computing an adaptive divergence is presented in Algorithm 1. This algorithm might be an expensive procedure as the algorithm probes multiple pseudo-divergences, and for each of these probes, generally, a model needs to be trained from scratch. However, two of the most commonly used machine learning models, boosting-based methods (Friedman, 2001) and Neural Networks, allow for more efficient estimation algorithms due to the iterative nature of training procedures for such models. 5/17 0.200 1.5 JSD 0.6 AD, dropout 0.175 AD, l2 1.0 0.5 0.150 0.5 0.125 0.4 0.100 0.0 0.3 0.075 0.2 0.5 0.050 0.1 JSD 1.0 0.025 generator linear AD ground-truth logarithmic AD 0.0 0.000 1.0 0.5 0.0 0.5 1.0 1.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 rotation angle rotation angle (a) Example configuration. (b) Gradient Boosting. (c) Neural Networks. 0.7 0.7 0.6 0.6 0.5 0.5 0.4 0.4 0.3 0.3 0.2 0.2 JSD JSD 0.1 0.1 generator linear AD AD, dropout ground-truth logarithmic AD AD, l 0.0 0.0 4 3 2 1 0 1 2 3 4 0.25 0.00 0.25 0.50 0.75 1.00 1.25 1.50 0.25 0.00 0.25 0.50 0.75 1.00 1.25 1.50 rotation angle rotation angle (d) Example configuration. (e) Gradient Boosting. (f) Neural Networks. Figure 1. Synthetic examples. (a) and (d): ground-truth distributions and example configurations of generators. Both generators are rotated versions of the corresponding ground-truth distributions. (b) and (e): JSD — Jensen-Shannon divergences estimated by Gradient Boosted Decision Trees with 500 trees of depth 3 (b), 100 trees of depth 3 (e); linear AD and logarithmic AD — adaptive divergences based on the same models as JSD with linear and logarithmic capacity functions, dashed lines represent some pseudo-divergences from the families producing adaptive divergences. (c) and (f): JSD — Jensen-Shannon divergences estimated by fully-connected Neural Networks with one hidden layer with 64 units (c) and 32 units (f); AD, dropout and AD, l — adaptive divergences based on the same architectures as the one for JSD, with dropout and l regularizations; dashed line represent some of pseudo-divergences from the dropout-produced family. See section 4 for the implementation details. 6/17 divergence divergence divergence divergence Algorithm 2 Boosted adaptive divergence Require: X , X — samples from distributions P and Q, B — base estimator training algorithm, N — P Q maximal size of the ensemble, c : Z ! [0; 1] — capacity function; r — learning rate; F 1=2 i 0 L log 2 for i = 1;:::; N do if L > c(i) log 2 then F F + r B(F ; X ; X ) i+1 i i P Q L L(F ; X ; X ) i+1 i+1 P Q i i+ 1 else return log 2 L end if end for return log 2 L 4.1 Gradient Boosted Decision Trees Gradient Boosted Decision Trees (Friedman, 2001) (GBDT) and, generally, boosting-based methods, being ensemble methods, intrinsically produce an ordered and complete with respect to Jensen-Shannon divergence family of pseudo-divergences in the manner similar to equation (4). This allows for an efficient AD estimation procedure shown by algorithm 2. Here, the number of base estimators serves as capacity of pseudo-divergences, and mapping to a 2 [0; 1] is defined through an increasing capacity function c : Z ! [0; 1] . In our experiments, for ensembles of maximal size N, we use the following capacity functions: linear capacity: c(i) = c ; (8) log(i+ 1) logarithmic capacity: c(i) = c : (9) log(N + 1) Notice, however, that Equation (4) defines a discrete variant of AD, which most certainly will result in a discontinuous function . This effect can be seen on Fig. 1e. 4.2 Neural Networks There is a number of ways to regulate the capacity of a neural network, in this work, we use well- established dropout regularization with rescaling of layers’ outputs (similar to weight rescaling suggested by Srivastava et al. (2014)). It is clear that setting dropout probability p to 0 results in an unregularized network, while p = 1 effectively restricts classifier to a constant output and intermediate values of p produce models in between these extreme cases. Additionally, we examine l regularization on network weights. We use a linear capacity function for dropout regularization: c(a) = 1 a , and a logarithmic one for l regularization: c(a) = log(a). In our experiments, we observe that unregularized networks require significantly more samples to be properly trained than regularized ones. We suggest to use additional independent regularization, in this work, following Louppe et al. (2017) we use gradient regularization R suggested by Mescheder et al. (2018). Note, that a discriminator trained with such regularization no longer produces JSD estimations. Nevertheless, the resulting function is a proper divergence (Mescheder et al., 2018), and all results in this work still hold with respect to such divergences. To produce a family of pseudo-divergences, the proposed algorithm varies the strength of the regular- ization depending on the current values of the cross-entropy. The values of the loss function are estimated with exponential moving average over losses on mini-batches during iterations of Stochastic Gradient Descent, with the idea that, for slowly changing loss estimations and small enough learning rate, network Technically, this function should be extended on [0;+¥) to be in agreement with definition 2. Note, that introducing a continuous approximation of the ensemble by, for example, varying learning rate for the last base estimator in the current ensemble from 0 to r , eliminates discontinuity of AD. 7/17 training should converge (Liu et al., 2018). We find that initializing exponential moving average with log 2, which corresponds to the absent regularization, works best. The proposed procedure is outlined in algorithms 3 and 4. Algorithm 3 Adaptive divergence estimation by a dropout-regularized neural network Require: X , X — samples from distributions P and Q; P Q f : X R! R — neural network with parameters q 2 Q, the second argument represents dropout probability and is zero if unspecified; c — capacity function; r — exponential average coefficient; b — coefficient for R regularization; g — learning rate of SGD. L log 2 acc while not converged do x sample(X ) P P x sample(X ) Q Q acc z c 1 log 2 g Ñ L( f (;z); x ; x ) 0 q q P Q g Ñ kÑ f (x )k 1 q q q P L r L +(1 r) L( f ; x ; x ) acc acc q P Q q q g (g + b g ) 0 1 end while return log 2 L( f ; X ; X ) q P Q Algorithm 4 Adaptive divergence estimation by a regularized neural network Require: X , X — samples from distributions P and Q; P Q f : X ! R — neural network with parameters q 2 Q; R : Q! R — regularization function; c — capacity function; r — exponential average coefficient; b — coefficient for R regularization; g — learning rate of SGD. L log 2 acc while not converged do x sample(X ) P P x sample(X ) Q Q acc z c 1 log 2 g Ñ [L( f ; x ; x )+ z R( f )] 0 q q P Q q g Ñ kÑ f (x )k 1 q q q P L r L +(1 r) L( f ; x ; x ) acc acc q P Q q q g (g + b g ) 0 1 end while return log 2 L( f ; X ; X ) q P Q 5 EXPERIMENTS We evaluate the performance of adaptive divergences with two black-box optimization algorithms, namely Bayesian Optimization and Adversarial Variational Optimization . As computational resources spent by simulators are of our primary concern, we measure convergence of Adversarial Optimization with respect to the number of samples generated by the simulation. Each task is presented by a parametrized generator, Code of the experiments is available at https://github.com/HSE-LAMBDA/rapid-ao/ 8/17 ’real-world’ samples are drawn from the same generator with some nominal parameters. Optimization algorithms are expected to converge to these nominal parameters. To measure the number of samples required to estimate a divergence, we search for the minimal number of samples such that the difference between train and validation losses is within 10 for Gradient 2 6 Boosted Decision Trees and 5 10 for Neural Networks . As a significant number of samples is involved in loss estimation, for simplicity, we ignore uncertainties associated with finite sample sizes. For GBDT, we utilize a bisection root-finding routine to reduce time spent on retraining classifiers; however, for more computationally expensive simulators, it is advised to gradually increase the size of the training set until the criterion is met. As the performance of Bayesian Optimization is influenced by choice of the initial points (in our experiments, 5 points uniformly drawn from the search space), each experiment involving Bayesian Optimization is repeated 100 times, and aggregated results are reported. 5.1 XOR-like synthetic data This task repeats one of the synthetic examples presented in Fig. 1d: ground truth distribution is an equal mixture of two Gaussian distributions, the generator produces a rotated version of the ground-truth distribution with the angle of rotation being the single parameter of the generator. The main goal of this example is to demonstrate that, despite significant changes in the shape of the divergence, global optimization algorithms, like Bayesian Optimization, can still benefit from the fast estimation procedures offered by adaptive divergences. For this task, we use an adaptive divergence based on Gradient Boosted Decision Trees (100 trees with the maximal depth of 3) with linear and logarithmic capacity functions given by Equations (8) and (9) and c = 1=4. Gaussian Process Bayesian Optimization with Matern kernel (n = 3=2 and scaling from 3 3 [10 ; 10 ] automatically adjusted by Maximum Likelihood fit) is employed as optimizer. Convergence of the considered divergences is shown in Fig. 2. As can be seen from the results, given the same budget, both variants of adaptive divergence converge on parameters around an order of magnitude closer to the optimum than traditional JSD. Notice, that initial rapid progress slows as optimizer approaches the optimum, and the slope of the curves becomes similar to that of JSD: this can be explained by AD approaching JSD as probed distributions become less distinguishable from the ground-truth one. 5.2 Pythia hyper-parameter tuning This task is introduced by Ilten et al. (2017) and involves tuning hyper-parameters of the Pythia event generator, a high-energy particle collision simulation used at CERN. For this task, electron-positron collision are simulated at a center-of-mass energy 91.2 GeV, the nominal parameters of Pythia generator are set to the values of the Monash tune (Skands et al., 2014). Various physics-motivated statistics of events are used as observables, with a total of more than 400 features. The same statistics were originally used to obtain the Monash tune. For the purposes of this work, we consider one hyper-parameter, namely alphaSValue, with the nominal value of 0:1365 and search range [0:06; 0:25]. We repeat settings of the experiment described by Ilten et al. (2017), with the only difference, that observed statistics are collected on the per-event basis instead of aggregating them over multiple events. We employ Gradient Boosting over Oblivious Decision Trees (CatBoost implementation (Prokhorenkova et al., 2018)) with 100 trees of depth 3 and other parameters set to their default values. We use Gaussian 3 3 Process Bayesian Optimization with Matern kernel (n = 3=2 and scaling from [10 ; 10 ] automatically adjusted by Maximum Likelihood fit) as optimizer. Comparison of unmodified Jensen-Shannon divergence with adaptive divergences with linear and logarithmic capacity functions (defined by Equations (8) and (9) and c = 1=4) presented on Fig. 3. Results indicate that, given the same budget, Bayesian Optimization over adaptive divergences yields solutions about an order of magnitude closer to the nominal value than Jensen-Shannon divergence. Additionally, notice that the slope of the convergence curves for AD gradually approaches that of AD as the proposal distributions become closer to the ground-truth one. This procedure requires generating additional validation set of the size similar to that of the training set, which might be avoided by, e.g., using Bayesian inference, or cross-validation estimates. 9/17 JSD linear AD logarithmic AD 0 10000 20000 30000 40000 50000 60000 examples sampled Figure 2. XOR-like synthetic example, Gradient Boosted Decision Trees. Convergence of Bayesian Optimization on: Jensen-Shannon divergence (marked as JSD), adaptive divergences with a linear capacity function (marked as linear AD), and a logarithmic capacity function (logarithmic AD). Each experiment was repeated 100 times; curves are interpolated, median curves are shown as solid lines, bands indicate first and third quartiles. 5.3 Pythia-alignment In order to test the performance of adaptive divergences with Adversarial Variational Optimization, we repeat the Pythia-alignment experiment suggested by Louppe et al. (2017). The settings of this experiment are similar to the previous one. In this experiment, however, we consider a detector represented by a 32 32 spherical grid with cells uniformly distributed in pseudorapidity n 2 [ 5; 5] and azimuthal angle f 2 [ p;p] space. Each cell of the detector records the energy of particles passing through it. The detector has 3 parameters: x; y; z-offsets of the detector center relative to the collision point, where z-axis is placed along the beam axis, the nominal offsets are zero, and the initial guess is (0:75; 0:75; 0:75). Fig. 4 shows averaged detector responses for the example configurations and samples from each of these configurations. For this task, a 1-hidden-layer Neural Network with 32 hidden units and ReLU activation function is employed. R regularization, proposed by Mescheder et al. (2018), with the coefficient 10, is used for the proposed divergences and the baseline. Adam optimization algorithm (Kingma and Ba, 2014) with learning rate 10 is used to perform updates of the search distribution. We compare the performance of two variants of adaptive divergence (dropout and l regularization) described in Section 4. Results are shown in Fig. 5. Given the same budget, both variants of adaptive divergence show a significant acceleration in contrast to the baseline divergence with only R regularization. Note that the acceleration is even more pronounced in comparison to JSD estimated by an unregularized network: in our experiments, within the budget, we have not been able to consistently achieve the set level of agreement between train and test losses with the unregularized network. 10/17 distance to the solution JSD linear AD logarithmic AD 0 5000 10000 15000 20000 25000 30000 examples sampled Figure 3. Pythia hyper-parameter tuning, CatBoost. Convergence of Bayesian Optimization on: Jensen-Shannon divergence (marked as JSD), adaptive divergences with a linear capacity function (marked as linear AD) and a logarithmic capacity function (logarithmic AD). Each experiment was repeated 100 times, curves are interpolated, median curves are shown as solid lines, bands indicate 25th and 75th percentiles. 6 DISCUSSION To the best knowledge of the authors, this work is the first one that explicitly addresses computational costs of Adversarial Optimization for expensive generators. Interestingly, several recent developments, like Progressive GAN (Karras et al., 2017) and ChainGAN (Hossain et al., 2018), use multiple discriminators of increasing capacity; however, this is done mainly to compensate for the growing capacity of the generators and, probably, not for reducing computational cost. Several recent papers propose improving stability of Adversarial Optimization by employing diver- gences other than Jensen-Shannon (Gulrajani et al., 2017; Arjovsky et al., 2017; Bellemare et al., 2017). Note that all results in this paper also hold for any divergence that can be formulated as an optimization problem, including Wasserstein (Arjovsky et al., 2017) and Cramer (Bellemare et al., 2017) distances. It can be demonstrated by adjusting Definition 2 and repeating the proof of Theorem 1 for a new divergence; presented algorithms also require only minor adjustments. Multiple works introduce regularization (Sønderby et al., 2016; Arjovsky and Bottou, 2017; Roth et al., 2017; Kodali et al., 2017; Mescheder et al., 2018) for improving stability and convergence of Adversarial Optimization. Most of the standard regularization methods can be used to regulate model capacity in an adaptive divergence. Also, one can use these regularization methods in addition to adaptive divergence as any discriminator-based regularization effectively produces a new type of divergence. Pythia-alignment experiment (section 5.3) demonstrates it clearly, where we use R regularization with constant coefficient in addition to varying-strength dropout and l regularization. Finally, we would like to stress that the properties of adaptive divergences highly depend on the under- lying families of pseudo-divergences, and interaction between AD and various proposed regularization schemes is subject to future research. 11/17 distance to the solution Figure 4. Illustration of the Pythia-alignment task. (Top row, from left to right) aggregated events for zero offset (the nominal configuration), 0.25 offset along x-axis, y-axis and z-axis. (Bottom row) single-event examples from the corresponding configurations above. 7 CONCLUSION In this work, we introduce adaptive divergences, a family of divergences meant as an alternative to Jensen- Shannon divergence for Adversarial Optimization. Adaptive divergences generally require smaller sample sizes for estimation, which allows for a significant acceleration of Adversarial Optimization algorithms. These benefits were demonstrated on two fine-tuning problems involving Pythia event generator and two of the most popular black-box optimization algorithms: Bayesian Optimization and Variational Optimization. Experiments show that, given the same budget, adaptive divergences yield results up to an order of magnitude closer to the optimum than Jensen-Shannon divergence. Note, that while we consider physics-related simulations, adaptive divergences can be applied to any stochastic simulation. Theoretical results presented in this work also hold for divergences other than Jensen-Shannon divergence. ACKNOWLEDGMENTS We wish to thank Mikhail Hushchyn, Denis Derkach and Marceline Ivanovna for useful discussions and suggestions on the text. FUNDING STATEMENT The research was carried out with the financial support of the Ministry of Science and Higher Education of Russian Federation within the framework of the Federal Target Program Research and Development in Priority Areas of the Development of the Scientific and Technological Complex of Russia for 2014-2020. Unique identifier RFMEFI58117X0023, agreement 14.581.21.0023 on 03.10.2017. 12/17 0 JSD AD, dropout AD, l2 0 5000 10000 15000 20000 25000 30000 examples sampled Figure 5. Pythia-alignment, Neural Networks. Convergence of Adversarial Variational Optimization on: adaptive divergence produced by l regularization (AD, l ), dropout regularization (AD, dropout), and the 2 2 baseline divergence with constant R regularization (marked as JSD). 13/17 distance to the solution REFERENCES Arjovsky, M. and Bottou, L. (2017). Towards principled methods for training generative adversarial networks. Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein gan. arXiv preprint arXiv:1701.07875. Baydin, A. G., Heinrich, L., Bhimji, W., Gram-Hansen, B., Louppe, G., Shao, L., Cranmer, K., Wood, F., et al. (2018). Efficient probabilistic inference in the quest for physics beyond the standard model. arXiv preprint arXiv:1807.07706. Bellemare, M. G., Danihelka, I., Dabney, W., Mohamed, S., Lakshminarayanan, B., Hoyer, S., and Munos, R. (2017). The cramer distance as a solution to biased wasserstein gradients. arXiv preprint arXiv:1705.10743. Bouhova-Thacker, E., Catmore, J., Cheatham, S., Chilingarov, A., Davidson, R., de Mora, L., Fox, H., Henderson, R., Hughes, G., Jones, R. W. L., et al. (2010). The atlas simulation infrastructure. European Physical Journal C: Particles and Fields, 70(3):823–874. Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., and Choo, J. (2018). Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8789–8797. IEEE. Dumoulin, V., Belghazi, I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., and Courville, A. (2016). Adversarially learned inference. arXiv preprint arXiv:1606.00704. Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189–1232. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. (2017). Improved training of wasserstein gans. In Advances in neural information processing systems, pages 5767–5777. Hossain, S., Jamali, K., Li, Y., and Rudzicz, F. (2018). Chaingan: A sequential approach to gans. arXiv preprint arXiv:1811.08081. Ilten, P., Williams, M., and Yang, Y. (2017). Event generator tuning using bayesian optimization. Journal of Instrumentation, 12(04):P04028. Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. (2017). Image-to-image translation with conditional adver- sarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134. Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2017). Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196. Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. Kodali, N., Abernethy, J., Hays, J., and Kira, Z. (2017). On convergence and stability of gans. arXiv preprint arXiv:1705.07215. Li, J., Madry, A., Peebles, J., and Schmidt, L. (2017). Towards understanding the dynamics of generative adversarial networks. arXiv preprint arXiv:1706.09884. Liu, H., Simonyan, K., and Yang, Y. (2018). Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055. Louppe, G., Hermans, J., and Cranmer, K. (2017). Adversarial variational optimization of non- differentiable simulators. arXiv preprint arXiv:1707.07113. Mescheder, L., Geiger, A., and Nowozin, S. (2018). Which training methods for gans do actually converge? In International Conference on Machine Learning, pages 3478–3487. Metz, L., Poole, B., Pfau, D., and Sohl-Dickstein, J. (2016). Unrolled generative adversarial networks. In ICLR. Mockus, J. (2012). Bayesian approach to global optimization: theory and applications, volume 37. Springer Science & Business Media. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. (2018). Catboost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, pages 6638–6648. Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolu- tional generative adversarial networks. arXiv preprint arXiv:1511.06434. 14/17 Roth, K., Lucchi, A., Nowozin, S., and Hofmann, T. (2017). Stabilizing training of generative adversarial networks through regularization. In Advances in neural information processing systems, pages 2018– Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Sjostrand, ¨ T., Ask, S., Christiansen, J. R., Corke, R., Desai, N., Ilten, P., Mrenna, S., Prestel, S., Rasmussen, C. O., and Skands, P. Z. (2015). An introduction to pythia 8.2. Computer physics communications, 191:159–177. Sjostrand, ¨ T., Mrenna, S., and Skands, P. (2006). Pythia 6.4 physics and manual. Journal of High Energy Physics, 2006(05):026. Skands, P., Carrazza, S., and Rojo, J. (2014). Tuning pythia 8.1: the monash 2013 tune. The European Physical Journal C, 74(8):3024. Sønderby, C. K., Caballero, J., Theis, L., Shi, W., and Huszar ´ , F. (2016). Amortised map inference for image super-resolution. arXiv preprint arXiv:1610.04490. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. Wierstra, D., Schaul, T., Glasmachers, T., Sun, Y., Peters, J., and Schmidhuber, J. (2014). Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949–980. A PROOF OF THEOREM 1 Theorem 1. If AD is an adaptive divergence produced by a complete and ordered with respect to Jensen-Shannon divergence family of pseudo-divergences D , then for any two distributions P and Q: JSD(P; Q) = 0 if and only if AD(P; Q) = 0. Proof. For convenience, we repeat the definition of an adaptive divergence AD here: AD (P; Q) = inffD (P; Q)j D (P; Q) (1 a) log 2g: (10) D a a Firstly, we prove that from JSD(P; Q) = 0 follows AD (P; Q) = 0. Due to Property (D2), D (P; Q) = D 1 JSD(P; Q) = 0, therefore, 8a 2 [0; 1] : D (P; Q) = 0 due to Properties (D2) (pseudo-divergences form a non-decreasing sequence) and (P1) (non-negativity of pseudo-divergences), which, in turn, implies that AD(P; Q) = inff0g = 0. Secondly, we prove that from AD (P; Q) = 0 follows JSD(P; Q) = 0. Let’s assume that, for some P and Q, AD(P; Q) = 0, but JSD(P; Q) = C > 0. Let us define the set of active capacities A (P; Q) as follows: A (P; Q) =fa j D (P; Q) (1 a) log 2g: (11) D a Note, that for every proper family D and for every pair of P and Q: f1g A (P; Q) and, if a 2 A (P; Q) D D then [a; 1] A (P; Q). The latter follows from Property (D1) (pseudo-divergences form a non-decreasing sequence) and the fact, that (1 a) log 2 is a strictly decreasing function. The previous statement implies that there are three possible forms of A (P; Q): 1. a single point: A (P; Q) =f1g; 2. an interval: A (P; Q) = [b; 1]; 3. a half-open interval: A (P; Q) = (b; 1]; for some b 2 [0; 1). The first case would contradict our assumptions, since AD (P; Q) = inffD (P; Q)g = D 1 C > 0. To address the last two cases, note, that 8a 2 A (P; Q) : D (P; Q) (1 b) log 2 > 0 due to D a the definition of A (P; Q). However, this implies that AD (P; Q) = inffD (P; Q)j a 2 A (P; Q)g D D a D (1 b) log 2 > 0, which contradicts our assumptions. From the statements above, we can conclude that if AD (P; Q) = 0, then JSD(P; Q) = 0. Combined with the previouly proven (JSD(P; Q) = 0)) (AD (P; Q) = 0), this finishes the proof. 15/17 B FORMAL DEFINITIONS AND PROOFS Definition 4. A model family M =fM F j a 2 [0; 1]g is complete and nested, if: (N0) (x7! 1=2)2 M ; (N1) M = F ; (N2) 8a;b 2 [0; 1] : (a < b)) (M M ). Theorem 2. If a model family M = fM F j a 2 [0; 1]g is complete and nested, then the family D =fD : P(X )P(X )! Rj a 2 [0; 1]g, where: D (P; Q) = log 2 inf L( f ; P; Q); (12) f2M is a complete and ordered with respect to Jensen-Shannon divergence family of pseudo-divergences. Proof. Let’s introduce function f (x) = 1=2. Now we prove the theorem by proving that the family satisfies all properties from Definition 2. Property (D0) Due to Properties (N0) and (N2), f is a member of each set M . This implies, that 0 a D (P; Q) 0 for all a 2 [0; 1]. For P = Q, cross-entropy loss function L( f ; P; Q) achieves its minimum in f = f , therefore, D (P; Q) = 0 if P = Q for all a 2 [0; 1]. Therefore, for each a 2 [0; 1] D is a 0 a a pseudo-divergence. Property (D1) From Property (N2) follows, that for all 0 a < b 1: D (P; Q) = log 2 inf L( f ; P; Q) log 2 inf L( f ; P; Q) = D (P; Q): f2M f2M Property (D2) This property is directly follows from Property (N1) and Equation (12). Definition 5. If M is a parameterized model family M =f f (q;) : X ! [0; 1]j q 2 Qg, then a function R : Q! R is a proper regularizer for the family M if: (R1) 8q 2 Q : R(q) 0; (R2) 9q 2 Q : f (q;) ^(R(q) = 0). Theorem 3. If M is a parameterized model family: M =f f (q;)j q 2 Qg and M = F , R : Q! R is a proper regularizer for M , and c : [0; 1]! [0;+¥) is a strictly increasing function such, that c(0) = 0, then the family D =fD : P(X )P(X )! Rj a 2 [0; 1]g: D (P; Q) = log 2 min L( f (q;); P; Q); q2Q (P;Q) Q (P; Q) = Arg min L (q; P; Q); a a q2Q L (q; P; Q) = L( f (q;); P; Q)+ c(1 a)R(q); is a complete and ordered with respect to Jensen-Shannon divergence family of pseudo-divergences. Proof. We prove the theorem by showing that the family D satisfies all properties from Definition 2. Property (D0) Due to Property (R2), there exists such q , that f (q ;) 1=2 and R(q ) = 0. Notice, 0 0 0 R R that, for all P and Q, L (q ; P; Q) = log 2 and L (q; P; Q) L( f (q;); P; Q), therefore, D (P; Q) 0 for 0 a a a all P; Q2 P(X ) and for all a 2 [0; 1]. For the case P = Q, q also delivers minimum to L( f (q ;); P; Q)+ 0 0 c(1 a)R(q ), thus, D (P; Q) = 0 if P = Q. This proves D to be a pseudo-divergence for all a 2 [0; 1]. 0 a a 16/17 Property (D1) Let’s assume that 0 a < b 1, yet, for some P and Q, D (P; Q) > D (P; Q). The a b latter implies, that: min L( f (q;); P; Q) < min L( f (q;); P; Q); (13) q2X q2X where: X = Q (P; Q) and X = Q (P; Q). Let us pick some model parameters: a a b b q 2 Arg min L( f (q;); P; Q); q2X q 2 Arg min L( f (q;); P; Q): q2X Since q 2 X , then, by the definition of Q (P; Q): b b b R R L (q ; P; Q) L (q ; P; Q): (14) b a b b From the latter and assumption (13) follows, that R(q ) < R(q ). By the conditions of the theorem, b a C = c(1 a) c(1 b) > 0 and: C R(q ) < C R(q ): (15) b a Adding inequality (14) to inequality (15): R R L (q ; P; Q) < L (q ; P; Q); a b a which contradicts the definition of q . This, in turn, implies that the assumption (13) contradicts conditions of the theorem. Property (D2) Since c(0) = 0 and M = F , D = JSD by the definition. 17/17
http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.pngStatisticsarXiv (Cornell University)http://www.deepdyve.com/lp/arxiv-cornell-university/adaptive-divergence-for-rapid-adversarial-optimization-vm90VNJVXn
Adaptive Divergence for Rapid Adversarial Optimization