A remarkable byproduct of the work of Viazovska et al. on sphere packing is the notion of a "Fourier uniqueness set": a discrete subset $A$ of $\mathbb{R}^d$ such that any Schwarz function on $\mathbb{R}^d$ is determined by its restriction and the restriction of its Fourier transform to $A$. Once one knows such sets exist, it is natural to ask how dense they must be / how sparse they can be. This was determined in dimension 1 by Kulikov, Nazarov, and Sodin, but the higher dimensional story was much less well understood. We will give an answer in all dimensions using new methods.