Nonconvex Optimization Bounds.pdf

The authors analyze the convergence of stochastic gradient-based optimization algorithms with early stopping based on a validation function. They derive conditions for the stopping rule to be well-defined and provide bounds on the expected number of iterations and gradient evaluations needed to meet the criterion. The guarantee accounts for the distance between the training and validation sets, measured with the Wasserstein distance.

The authors develop the approach in the general setting of a first-order optimization algorithm, with possibly biased update directions subject to a geometric drift condition. They derive bounds on the expected running time for early stopping variants of several algorithms, including stochastic gradient descent (SGD), decentralized SGD (DSGD), and the stochastic variance reduced gradient (SVRG) algorithm.

The main contributions include:
- A non-asymptotic analysis of SGD with early stopping, leading to a bound on the expected amount of resources needed to find approximate stationary points of the training function.
- Specialization of the results to decentralized SGD, resulting in upper bounds on the number of iterations and gradient evaluations needed by the algorithm.
- Derivation of a run-time bound for a variant of nonconvex SVRG with early stopping, obtaining a bound on the expected number of iterations and IFO calls needed to generate approximate stationary points.
- Demonstration of how Wasserstein concentration bounds can be leveraged to bound the generalization performance of the iterate returned by the algorithms.

The authors also discuss related work, including the study of stochastic optimization, asymptotic performance of biased SGD, and non-asymptotic performance guarantees for SGD applied to nonconvex functions. They highlight the difference between their approach, which uses early stopping based on a validation function, and other works that use randomization to analyze performance in the nonconvex case.