What happens inside the "mind" of artificial intelligence when it learns to understand language? How does it move from simply following the order of words to grasping their meaning? A recently published study offers a theoretical perspective on these internal processes and identifies a transformation that resembles a physical phase transition.
Modern language models—such as ChatGPT or Gemini—are built on so-called transformer architectures, which rely on self-attention layers. These layers help the system detect relationships between words by considering both their positions in a sentence and their meanings. The new research explores the transition between these two strategies—positional and semantic attention—using mathematical and theoretical tools borrowed from physics.
The key finding is that this shift is not gradual but abrupt: up to a certain point, the model primarily depends on word position, but once the training data reaches a critical threshold, it suddenly switches to meaning-based processing. The authors—Hugo Cui and his collaborators—describe this change as a phase transition, similar to how water suddenly becomes steam at its boiling point. The study provides a mathematical characterization of this transition and shows how it can be precisely located within the model’s self-attention mechanism.
To analyze the phenomenon, the researchers used a simplified model in which sentences were composed of randomly generated, uncorrelated words, and the learning process involved only a single attention layer. This design allowed for a high-precision mathematical treatment, including closed-form expressions for the model’s training and test errors. The analysis revealed that with limited training data, the model favors positional cues—but once the data surpasses a certain complexity level, it relies almost entirely on semantic information. This shift also leads to improved performance, assuming enough data is available.
It's important to emphasize that the model studied is a theoretical simplification and does not aim to fully replicate systems like ChatGPT. Rather, the goal was to establish a rigorous framework for interpreting learning behaviors observed in more complex systems. Still, the results are significant: they demonstrate that artificial neural networks can change learning strategies not only gradually or adaptively, but also in discrete, qualitatively distinct ways. In the long run, such insights could support the development of more efficient and interpretable AI systems.
Beyond its relevance for AI theory, the study also forges a link between physics and machine learning. The authors draw an analogy between interacting particles in physics and the units of a neural network: both systems exhibit complex collective behavior that can be described statistically, and both give rise to emergent properties from simple components.
In summary, this research marks an important step toward understanding how language models learn and adapt. It does not provide a final answer, but it lays theoretical groundwork for exploring when and why an AI system shifts its learning strategy—and this understanding may ultimately shape how we design, interpret, and govern such technologies.
