Séminaire Image: Converge to optima without overparameterization – Rayleigh quotients and subnetwork tracking par David A. R. Robin

Nous aurons le plaisir d’écouter David A. R. Robin , Postdoc au LAMSADE (Université Paris Dauphine).
Il donnera un séminaire IMAGE le jeudi 1er octobre 2026 à 14h en salle de séminaire F-200.

Titre : « Converge to optima without overparameterization – Rayleigh quotients and subnetwork tracking »

Résumé : 

The Neural Tangent Kernel (NTK) is a central object used to prove convergence of sufficiently overparameterized neural networks to zero training loss. When the NTK of a network is nearly-constant and positive definite, it decomposes the target signal along its eigenbasis, explains at which speed each component is learned, and even accurately predicts generalization error as a function of the target signal’s NTK-induced complexity. But this powerful tool comes with built-in drawbacks: it immediately breaks without overparameterization (such as with infinite samples by data augmentation), and its formal construction is essentially restricted to architectures resembling MLPs, far from the more complicated transformer and attention-based modern architectures.

In this talk, we will explore two ideas to patch these drawbacks: weakening positive-definiteness to a lower-bounded Rayleigh quotient, and extending to branching architectures by defining and tracking subnetworks across the training trajectory. This will lead us to a more geometric interpretation of neural network dynamics, with intuitions coming from the geometry of function spaces and discrete random graphs. We will see how this yields convergence proofs to (or near) global optima, even for infinite datasets and including transformer-based deep architectures, and discuss the next challenges along the road to quantitative predictions of convergence in modern settings.