Mathematical Machine Learning Theory
Rigorous properties of learning models, including optimisation, generalisation, asymptotic limits, and scaling behaviour.
University of Warwick · Mathematics
Siyuan Feng
Undergraduate Student in Mathematics
My research focuses on the mathematical theory of machine learning, especially rigorous structures underlying neural-network training. Under the supervision of Fanghui Liu, I currently study the transfer of Warmup–Stable–Decay (WSD) learning-rate schedules under μP across network widths and training steps.
Mathematics · Machine Learning Theory
Profile
About
I am a Mathematics undergraduate at the University of Warwick, with a long-term research focus on machine learning theory. I am primarily interested in rigorous mathematical properties of learning systems, including neural-network theory, optimisation dynamics, infinite-width limits, hyperparameter transfer, and scaling laws.
Fanghui Liu is my research supervisor. Our current research investigates the transfer of three-parameter WSD learning-rate schedules under μP. A central idea is cumulative-learning-rate normalisation, which puts different step counts into a shared dynamical coordinate system and enables prediction of longer-run objectives from shorter reference runs, alongside transfer across model widths.
I am especially interested in problems that can be reduced, through probability, linear algebra, optimisation, and asymptotic analysis, to explicit mathematical structures that support rigorous and verifiable explanations of neural-network training.
Education
University of Warwick, BSc Mathematics
Research
Machine Learning Theory
Current Focus
WSD transfer under μP across width and training steps
Supervisor
Fanghui Liu
Research Interests
Research Interests
Neural Networks, Width Scaling, and μP
Parametrisation, feature learning, infinite-width limits, and the cross-width stability of hyperparameters.
Learning-Rate Schedules and Hyperparameter Transfer
WSD schedules and the transfer of their optimal parameters across network widths and training steps.
Optimisation Dynamics and Asymptotic Methods
Optimisation, probability, spectral analysis, and asymptotic tools for training dynamics, stability, and deterministic limits.
Current Research
Current Research
Warmup–Stable–Decay Schedule Transfer under μP Across Width and Training Steps
The central idea of this project is a cumulative-learning-rate normalisation of WSD schedules. It places different training-step counts in a common dynamical coordinate system, while the final-to-peak ratio and decay fraction retain the schedule shape. This avoids directly comparing raw learning-rate parameters that no longer represent equivalent training across different step counts.
In these coordinates, we derive a finite-step expansion of the complete three-parameter WSD objective with a uniform second-order remainder. We rigorously show that one complete objective at a shorter step count is generally insufficient to determine the objective at a longer target, whereas two shorter-step reference objectives can predict the full target objective with second-order accuracy under suitable regularity conditions. We also connect cross-step prediction to fixed-step width stability under μP.
Research Experience
Research Experience
Warmup–Stable–Decay Schedule Transfer under μP Across Width and Training Steps
Rather than extrapolating only a scalar optimal peak learning rate, this project studies the complete objective over three-parameter WSD schedules: why one short-run reference is generally insufficient, and how two references can predict schedule performance at longer step counts.
- Normalisation and finite-step theory: Introduced cumulative-learning-rate normalisation to express WSD schedules at different step counts in shared coordinates (cumulative rate, final-to-peak ratio, and decay fraction). Derived a finite-step expansion of the complete WSD objective with a common leading term, a first correction, and a uniform second-order remainder.
- One-reference impossibility and two-reference prediction: Proved that even a complete objective at one reference step count cannot generally determine the objective at another. Established that two suitably separated shorter-step reference objectives remove the leading first-order prediction error, recover the full longer-step target objective with second-order accuracy, and control the resulting target-performance error.
- Width transfer and further extensions: Connected cross-step prediction to fixed-target-step width stability under μP and quantified the normalisation restriction gap. Derived conditional power/near-power scaling laws for the optimal peak learning rate from objective geometry, together with deterministic bounds for the effects of dataset mismatch on prediction and target performance.
The framework provides a mathematical explanation of why shorter-run information can predict the full WSD landscape at longer training budgets, offering a theoretical basis for reducing repeated target-scale tuning.
Generic Uniqueness, Exceptional Nonuniqueness, and Perturbative Stability
This project studies the global geometry of the wide-limit loss as a function of the learning rate in a two-step deep linear µP network. The central question is whether width-stable loss curves identify a unique optimal learning rate.
- Derived an exact finite-width state compression and a closed two-step wide limit, yielding an explicit polynomial loss in the learning rate.
- Proved generic uniqueness for positive-definite multisample data and constructed a full-rank exceptional family with exactly three positive nondegenerate global optima.
- Established stability of the global optimiser set under data perturbations and persistence of finite-width local-minimum branches.
The results show that uniqueness is a stable generic phenomenon rather than an unconditional law. In exceptional cases, the correct limiting object is the full optimiser set.
Personal
Personal Interests
4×4×4 Speedcubing Official WCA results and Warwick Winter 2025 champion
I am a 4×4×4 speedcuber and have competed in official WCA competitions. My official results include a 34.22-second single and a 41.46-second average. My cubing profile is available here.
At Warwick Winter 2025, I won the 4×4×4 event with a
36.95-second average, placing first in the university-wide
competition for this event.
A 4×4×4 cube solve
CV & Contact
CV & Contact
Curriculum Vitae
Contact
Email: eric_feng2006@outlook.com