We revisit the statistical foundations of nonparametric latent variable models and discuss how even basic statistical properties such as identifiability and consistency are surprisingly subtle. For example, even in simple mixtures, the existence of a uniformly consistent estimator is not guaranteed. Motivated by these difficulties, we develop a uniformly consistent estimator and then characterize the optimal sample complexity for this problem, which turn out to have a nonstandard super-polynomial bound. These results are then be applied to study mixtures of nonparametric regression models. Time permitting, we will also discuss some implications of these results for learning structured representations in machine learning.