Fan Pu Zeng

Curse of dimensionality and feature learning. Neural feature matrix (NFM). Average gradient outer product (AGOP). Deep neural feature ansatz: empirical evidence and theoretical results. Applications: spurious features, lottery tickets, grokking. Recursive feature machines. Steering LLMs.

SFT and DPO recap. Learning dynamics decomposition, from MNIST to LLMs. Training dynamics of SFT; hallucination explanation. Training dynamics of DPO; the squeezing effect. Off-policy DPO experiments. Remedies. PPO.

Why distillation. KL primer; forward vs. reverse KL. Prior work: DistiLLM and skew KL. Contrastive approach for LLM distillation (CALD). Curriculum for the interpolation coefficient. Results.

Overparameterization, double descent, sensitivity to initialization. Infinite-width networks as Gaussian processes (NNGP). Neural tangent kernel: initialization, training, generalization, limitations. Tensor Programs I‑V. Maximal update parametrization (muP); zero-shot hyperparameter transfer.

Score-based diffusion models are a promising direction for generative models, as they improve on both likelihood-based approaches like variational autoencoders, as well as adversarial methods like Generative Adversarial Networks (GANs). In this blog post, we survey recent developments in the field centered around the line of results developed in (Song & Ermon, 2019), analyze the current strengths and limitations of score-based diffusion models, and discuss possible future directions that can address its drawbacks. Joint work with Owen Wang.

An aperiodic and irreducible Markov chain will eventually converge to a stationary distribution. This is used in many applications in machine learning like Markov Chain Monte Carlo (MCMC) methods, where random walks on Markov chains are used to obtain a good estimate of the log likelihood of the partition function of a model, which is hard to compute directly as it is #P-hard (this is even harder than NP-hard). However, one common problem is that it is unclear how many steps we should take before we are guaranteed that the Markov chain has converged to the its stationary distribution.

In this post, we understand how the spectral gap of the transition matrix of the Markov chain relates to its mixing time.

Deep learning is currently dominated by parametric models, which are models with a fixed number of parameters regardless of the size of the training dataset. Examples include linear regression models and neural networks.

However, it's good to occasionally take a step back and remember that that is not all there is. Non-parametric models like k-NN, decision trees, or kernel density estimation don't rely on a fixed set of weights, but instead grow in complexity based on the size of the data.

In this post we'll talk about Gaussian processes, a conceptually important, but in my opinion under-appreciated non-parametric approach with deep connections with modern-day neural networks. An interesting motivating fact which we will eventually show is that neural networks initialized with Gaussian weights are equivalent to Gaussian processes in the infinite-width limit.

When was the first time you had to use LaTeX? If you are like most people, it was probably suddenly forced upon you during your first math or CS class where you had to start writing proofs, with minimal guidance on how to get started. Unfortunately, this meant that while many people have good operational knowledge of LaTeX, there are still many small mistakes and best practices which are not followed, which are not corrected by TAs as they are either not severe enough to warrant a note, or perhaps even the TAs themselves are not aware of them.

In this post, we cover some common mistakes that are made by LaTeX practitioners (even in heavily cited papers), and how to address them.

Over the past few years, I've received numerous questions about studying abroad and whether it is advisable to take a government scholarship to do so from my juniors. Having witnessed firsthand the early career trajectories of many scholars and non-scholars alike, and having heard their different perspectives, I feel like I have a good understanding of the concerns at hand to answer this question confidently. This post is aimed at Singaporean junior college students debating whether they should take on a scholarship from the government to pursue a fully-paid education abroad. In this post, I will consider both the pros and cons of taking a scholarship. The relevance of some of the points is highly dependent on your current situation, so ultimately take it with a grain of salt and consider your unique circumstances. It is useful to keep in mind that a scholarship is designed to benefit the government's interests, not your own.

This post is addressed to my future self, who (hopefully) would found a startup of his own someday. This is my last week at Saleswhale and I would like to pack as much wisdom into this post from my experiences there and from the wonderful people that I have talked to. It is a very personal post for me and it speaks straight from my heart. To be honest, I teared up a few times while writing this. I hope you will feel the sincerity in my words, and I hope it will inspire you.

15-712 Advanced OS was an excellent seminar-based graduate course that took us on a whirlwind tour through many of the most seminal SigOps Hall of Fame papers across several systems domains. It will prepare you to be a great systems designer and researcher. In this post, I will share my experience in the class, the course structure and content, what I thought were the biggest takeaways, and who this class might be suitable for.

Computer Networks is one of the lesser-known systems classes at Carnegie Mellon that turned out to be surprisingly fun and informative. In this post I'll talk about the projects and content covered, followed by my own thoughts on the usefulness on the class and who should take it.

Last week, I had the wonderful opportunity to participate in Hwa Chong's Undergrad Alumni Forum to share my experiences studying at Carnegie Mellon's School of Computer Science, and to answer any questions that the students had about studying in the US and pursuing Computer Science as a degree. I was a student at Hwa Chong Institution from 2010-2015, during which I made many happy memories and learnt a lot about myself and the world. I had personally found these sharings really helpful back when I was still a student, and am very grateful for the chance to pass it on and hopefully help to inspire and encourage some of the participants to pursue their education overseas. It has personally had brought about incredible personal and intellectual growth, and exposed me to people and ideas that I would otherwise never have had the chance to meet.

Why the gradient norm often goes up near the end of LLM training even when nothing is wrong, built up from how normalization layers and weight decay interact. Then AdamC, a one-line fix to weight decay from Defazio's 2025 paper.

Gradient descent doesn't care which basis your parameters are written in, but Adam does. Why that happens, and how it ties into the outlier features you see in transformers trained with Adam.

An intro to chaos theory (the Lorenz system, Lyapunov exponents, the butterfly effect) starting from basic calculus, and why training a neural net behaves like a chaotic system.

An intro to InfiniBand, the network that connects GPUs in big training clusters, for people with no networking background. How data gets from one GPU to another, why the network never drops packets, and how that differs from Ethernet.

How where you put LayerNorm in a transformer changes the gradients at each layer, and why that means post-LN needs learning rate warmup while pre-LN mostly doesn't.

A simulated fruit fly brain, wired from the real FlyWire connectome, playing blackjack. Only a small readout on top is trained, the wiring is the actual fly's.

Blackjack with the odds showing. Every option is scored against perfect play, so you learn basic strategy and see what each mistake costs you.

Learn xiangqi (Chinese chess) from zero, starting with the board and how each piece moves, then drills and puzzles.

Based on the old literati drinking game where a cup floats down a stream and whoever it stops at has to answer with a line of poetry. A way to pick up classical Chinese phrases.

How to split a big model across GPUs with tensor, pipeline, data and expert parallelism, and what each layout does to memory and communication.