I will present a joint work with Borjan Geshkovski, Yury Polyanskiy and Philippe Rigollet in which we used tools from ODEs and PDEs to study an interacting particle system arising in large language models through a mechanism called "self-attention". We demonstrate the formation of clusters of particles as time goes to infinity. We also explain how tools from geometric control theory could help understanding universal approximation properties of these particle systems.