Transformers have become the dominant architecture in modern machine learning, yet the theoretical understanding of their training dynamics remains limited, especially for deep architectures with many layers and Attention heads. I will present a rigorous mathematical framework for analyzing gradient-based training of Transformers in the overparameterized regime, where depth, width and context length tend to infinity. This mean-field model features two types of measure representations: token distributions evolving through layers (measures over the token space), and Attention parameters at each layer (measures over the parameter space). Our contributions are twofold: (i) We prove the well-posedness of gradient flow curves in a conditional Wasserstein metric space, establishing a rigorous foundation for gradient-based Transformer training. (ii) We prove convergence of gradient flow towards global minima when the initial training risk is sufficiently low. A key technical contribution is providing necessary and sufficient conditions for injectivity of the Neural Tangent Kernel (NTK) for Attention mechanisms: we show that NTK injectivity is equivalent to linear independence of log-sum-exp functions modulo affine functions, a condition satisfied by diverse token distributions, including discrete distributions, uniform distributions, and Gaussian mixtures.
These results builds on joint works with M. de Hoop, T. Furuya, G. Peyré and F.-X. Vialard.