On Geometric Dynamics of Deep Linear Networks

Speakers: Florian Kogelbauer (ETH Zürich)\n\nWe discuss the training dynamics of deep linear networks from a geometric and dynamical-systems perspective. Gradient descent induces a Riemannian gradient flow on the end-to-end network map, revealing a slow-fast structure in the evolution of its singular values. This provides a dynamical explanation for the implicit bias toward low-rank solutions and connects learning dynamics with invariant-manifold and model-reduction ideas. We show how this structure can inform initialization strategies for accelerating training.\n\nhttps://indico.math.cnrs.fr/event/16906/