The application of neural networks to solve nonlinear partial differential equations (PDEs) has gained significant attention, with popular methods including Physics-Informed Neural Networks (PINNs) and the deep-Ritz framework. However, these methods face challenges when applied to high-dimensional or instationary problems. A promising alternative is the use of evolutionary deep neural networks (EDNNs), which allow the network parameters to evolve over time. After the network is fitted to the initial conditions, the parameter evolution is then governed by an ordinary differential equation (ODE), solvable with established numerical integration schemes. Building upon this concept, the Neural Galerkin framework and subsequent publications provide further advances. The talk will include the derivation and mathematical interpretation of EDNNs, as well as extensions to parameter-dependent problems. Additionally, we will discuss the JAX-library used for efficient implementation of the proposed methods and present recent progress in goal-oriented error estimation for nonlinear discretizations.