joint work with David Kozak, Luis Tenorio, Alireza Doostan
Some optimization settings, such as certain types of PDE-optimization, do not give easy access to the gradient of the objective function. We analyze a simple 0th-order derivative-free stochastic method that computes the directional derivative in a stochastically chosen subspace, and also develop and analyze a variance-reduced version. Under an error bound condition we show convergence, and under strong convexity we show almost-sure convergence of the variables. Numerical experiments on problems from Gaussian Processes and PDE shape-optimization show good results even on non-convex problems.
joint work with Mingrui Liu, Hassan Rafique, Tianbao Yang
In this paper, we consider first-order algorithms for solving a class of non-convex non-concave min-max problems, whose objective function is weakly convex (resp. weakly concave) in the minimization (resp. maximization). It has many important applications in machine learning such as training Generative Adversarial Networks. Our method is an inexact proximal point method which solves the weakly monotone variational inequality corresponding to the min-max problem. Our algorithm finds a nearly $\epsilon$-stationary point of the min-max problem with a provable iteration complexity.
The Boosted Difference of Convex functions Algorithm (BDCA) was recently proposed for minimizing smooth DC functions. BDCA accelerates the convergence of the classical DCA thanks to an additional line search step. The purpose of this paper is twofold: to show that this scheme can be generalized and successfully applied to certain types of nonsmooth DC functions, and to show that there is complete freedom in the choice of the trial step size for the line search. We prove that any limit point of BDCA sequence is a critical point of the problem. The convergent rate is obtained under KL property.