Motivated by the mean-field optimization model of the training of two-layer neural networks, we propose a novel method to approximate the invariant measures of a class of McKean-Vlasov diffusions. We introduce a proxy process that substitutes the mean-field interaction with self-interaction through a weighted occupation measure of the particle's past. If the McKean-Vlasov diffusion is the gradient flow of a convex mean-field potential functional, we show that the self-interacting process exponentially converges towards its unique invariant measure close to that of the McKean-Vlasov diffusion. As an application, we show how to learn the optimal weights of a two-layer neural network by training a single neuron.