Critical Dynamics Govern Deep Learning: Enhancing Performance, Enabling Continual Learning and Mitigating Model Collapse

Simon Vock

The rapid advances in artificial intelligence (AI) have largely been driven by scaling deep neural networks (DNNs) - increasing model size, data, and computational resources. However, performance is ultimately governed by network dynamics. The lack of a principled understanding of DNN dynamics beyond heuristic-based design has contributed to challenges with their robustness, suboptimal performance, high energy consumption and pathologies in continual and AI-generated content learning. Increasing evidence suggests that the human brain may avoid these problems by operating at a critical phase transition. Inspired by this principle, we here propose that criticality may provide a unifying framework linking structure, dynamics, and function also in DNNs. First, by analyzing more than 80 state-of-the-art DNN models and well established benchmark datasets (ImageNet, MNIST), we report that a decade of AI progress has implicitly driven successful networks towards criticality, as measured by largest Lyapunov exponents λ₀ (Spearman r = -0.5, p < 10⁻⁴), explaining why certain architectures succeeded while others failed. Second, incorporating criticality explicitly into training improves even highly optimized models by more than 0.4 percentage points (p < 0.001) and establishes robustness across weight initializations, preventing key limitations of current models. Third, we show that catastrophic AI pathologies, including continual learning degradation and model collapse from AI-generated training data, constitute a loss of critical dynamics. By keeping networks critical, we provide the first principled solution to these fundamental AI problems by mitigating performance degradation and model collapse. Fourth, we show that this global critical state can be regulated by purely local homeostatic plasticity rules. With no global objective or external supervision required, they point to a biologically plausible mechanism of self-organized criticality in artificial networks. This work confirms criticality as substrate-independent principle of intelligence, connecting AI advancement with core principles of brain function. For systems neuroscience, this work offers a link between optimal function in biological and artificial neural networks. For AI, it provides profound theoretical insights along with immediate practical value solving major challenges to ensure long-term DNN performance and resilience as models grow in scale and complexity.

Weitere Infos zur Veranstaltung