Dynamics, Neural Collapse, And Normalization In Deep Classifiers Trained With Square Loss.pdf

This memo presents research on the dynamics and performance of deep classifiers trained with square loss, challenging the conventional belief that cross-entropy loss is superior. It offers a theoretical framework to understand these classifiers' behavior, particularly in relation to their convergence and generalization capabilities.

Key Findings:

1. Square Loss vs. Cross-Entropy: Recent experiments suggest square loss performs comparably to cross-entropy for classification tasks in deep networks.

2. Dynamics of Training: The study focuses on the dynamics of Gradient Descent (GD) techniques, particularly with Weight Decay (WD) and normalization methods like Batch Normalization (BN). It predicts convergence to minimum norm solutions under specific conditions.

3. Independence from Initial Conditions: Numerical simulations indicate approximate independence from initial conditions for square loss classifiers with WD and BN, mirroring results seen with cross-entropy loss. Without BN+WD, good solutions can be achieved for small initializations.

4. Neural Collapse: The research proves that gradient descent with WD in square loss classifiers exhibits Neural Collapse, a phenomenon where margins become similar, and network representations simplify to form a geometric structure known as an equiangular tight frame (ETF).

5. Normalization Techniques: It compares Lagrange Normalization (LN) with commonly used BN and Weight Normalization (WN), highlighting their roles in stabilizing solutions and the dynamics of deep networks during training.

Future Directions:

The memo outlines plans to extend the analysis to gradient descent, multiclass classification, and further explore the implications of different normalization techniques.