ReLU is an archetypical activation function used in a wide range of deep learning models. Its homogeneity induces continuous symmetries in parameter space, which constrains the gradient and influence learning dynamics.

In this talk, I will present two works studying the topology and geometry of the optimization space of ReLU networks. I will begin with the shallow setting (networks with a single hidden layer), before extending to a broader class of architectures. We will investigate geometric properties of these optimization spaces, focusing in particular on their connectivity and singularities. Connectivity properties, fixed at initialization reveal potential topological obstructions to learning, while singularities can be interpreted through the emergence of subnetworks and suggest a framework for differentiable pruning.

References:

[1] Nurisso, Marco, Pierrick Leroy, and Francesco Vaccarino. "Topological obstruction to the training of shallow ReLU neural networks." Advances in Neural Information Processing Systems 37 (2024): 35358-35387.

[2] Nurisso, Marco, Pierrick Leroy, Giovanni Petri, and Francesco Vaccarino. "Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities." arXiv preprint arXiv:2602.00693 (ICLR 2026).