Generative models now write working code, produce realistic images, and hold long conversations, yet the mechanisms behind their intelligence still evade our understanding. This thesis explores that puzzle through Associative Memory (AM), an idea that emerged from physics and its concept of energy, in which patterns are stored as energy minima and retrieved through attractor dynamics. Beyond their capacity, AMs form new minima known as spurious states, long regarded as a failure of memory. We argue that this failure is where creativity begins, and that memorization and generalization can coexist as outcomes of one retrieval process. We further argue that AM is broader than the energy functions of Hopfield networks, defined instead by basins of attraction.
First, we show that diffusion models behave as AMs, in which training writes data into memory and generation tries to recall it. With little data, a diffusion model forms a distinct basin of attraction around each training sample and can only reproduce what it has seen. As the data grows, these basins shrink, and spurious states emerge as the first signs of generative capability, appearing alongside memorized samples before the model fully generalizes. We characterize this transition through the volume and curvature of these basins across datasets and architectures, including Stable Diffusion.
We then turn to language and move beyond the energy-based view of AM. We show that uniform-state discrete diffusion models are AMs capable of retrieving unseen data, with basins of attraction that arise from conditional likelihood maximization alone. As the training set grows, basins around training examples contract while basins around unseen examples expand, until corrupted test sentences are recovered as reliably as training ones, and conditional entropy reveals this coexistence within a single sequence. Together, these results suggest that AM is a general principle defined by basins of attraction, and that memorization and generalization can coexist within one model.
