ML Reading List

1. The Emergence of Spectral Universality in Deep Networks

Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

Studied MLP's Jacobian singular value distribution in the limit of large width

2. Free Probability Theory

Roland Speicher

Reference for free probability

3. Haar Measures

Stephan Tornier

Used in RMT

4. Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation

Greg Yang

NTK

5. The Recurrent Neural Tangent Kernel

Sina Alemohammad, Zichao Wang, Randall Balestriero, Richard Baraniuk

NTK for RNNs

6. Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks

Difan Zou, Yuan Cao, Dongruo Zhou, Quanquan Gu

Wide NNs behave as linear models

7. The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks

George Philipp, Jaime G. Carbonell

Signal progation in NN with random weights

8. Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?

Boris Hanin

Signal progation in NN with random weights

9. Gaussian Process Behaviour in Wide Deep Neural Networks

Alexander G. de G. Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, Zoubin Ghahramani

GP behavior of wide nets

10. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice

Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli

Shows that if Jacobian singular value distribution of a wide NN concentrates around 1 even when network gets deper, error signal largely preserved and all layers get signal to improve

11. Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

Wojciech Tarnowski, Piotr Warchoł, Stanisław Jastrzębski, Jacek Tabor, Maciej A. Nowak

Random matrix theory applied to DL

12. Spectrum concentration in deep residual learning: a free probability approach

Zenan Ling, Xing He, Robert C. Qiu

Random matrix theory applied to DL

13. Tensor Programs II: Neural Tangent Kernel for Any Architecture

Greg Yang

Provides a good overview of NTK. Some nice discussions on Gradient Independence Assumption (GIA). Extends TP I language to RNNs

14. Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes

Greg Yang

Framework for analyzing modern NN architectures. Pretty notation heavy though.

15. Analysis of Boolean Functions

Ryan O'Donnell

Contains results on Hermite polynomials that can be useful for DL

16. A Mean Field Theory of Batch Normalization

Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, Samuel S. Schoenholz

17. The dynamics of message passing on dense graphs, with applications to compressed sensing

Mohsen Bayati, Andrea Montanari

Introduces a useful Gaussian conditioning technique

18. Layer Normalization

Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton

Introduced layer norm

19. Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity

Amit Daniely, Roy Frostig, Yoram Singer

20. Message Passing Algorithms for Compressed Sensing

David L. Donoho, Arian Maleki, Andrea Montanari

Approximate message passing

21. Path Integral Approach to Random Neural Networks

A. Crisanti, H. Sompolinsky

Random classic spiking networks

22. Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach

Ryo Karakida, Shotaro Akaho, Shun-ichi Amari

Spectral properties of the empirical Fisher information matrix of random NNs

23. Fisher Information and Natural Gradient Learning of Random Deep Networks

Shun-ichi Amari, Ryo Karakida, Masafumi Oizumi

Spectral properties of the empirical Fisher information matrix of random NNs

24. Mean Field Residual Networks: On the Edge of Chaos

Greg Yang, Samuel S. Schoenholz

Edge of chaos MFT

25. Deep Information Propagation

Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, Jascha Sohl-Dickstein

Mean field theory for backprop

26. Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks

Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S. Schoenholz, Jeffrey Pennington

Training 10k layer CNN w/o batcnorm or skip connections

27. The boundary of neural network trainability is fractal

Jascha Sohl-Dickstein

Boundary between trainable and untrainable NN hyperparameters is fractal, nice visualizations

28. Exponential expressivity in deep neural networks through transient chaos

Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, Surya Ganguli

Signal propagation in NNs, NN expressivitiy

29. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

He initialization

30. Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes

Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abolafia et al.

Showed equivalence betw infinitely wide CNNs to GPs

31. Deep Neural Networks as Gaussian Processes

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein

Scaling limit for infinite width MLPs

32. Steps Toward Deep Kernel Methods from Infinite Neural Networks

Tamir Hazan, Tommi Jaakkola

GP behavior on wide NNs under other conditions

33. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, Christian Szegedy

Batchnorm paper

34. DeepSeek-V3 Technical Report

DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu et al.

First open model to beat SOTA closed models

35. Feature Learning in Infinite-Width Neural Networks

Greg Yang, Edward J. Hu

Shows that NN parameterizations using Standard/Mean Field/NTK either has feature learning or infinite-width training dynamics given by kernel gradient descent

36. DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data

Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan et al.

Uses autoformalization to create training data

37. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao et al.

Introduces Multi-head Latent Attention (MLA)

38. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang et al.

Introduced Group Relative Policy Optimization (GRPO)

39. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel

Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, Surya Ganguli

Benefits of feature learning

40. On Lazy Training in Differentiable Programming

Lenaic Chizat, Edouard Oyallon, Francis Bach

Shows feature learning is usually beneficial in practical large-scale DL settings

41. Meta-Principled Family of Hyperparameter Scaling Strategies

Sho Yaida

42. Disentangling feature and lazy training in deep neural networks

Mario Geiger, Stefano Spigler, Arthur Jacot, Matthieu Wyart

Feature learning limit for NN dynamics

43. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

Song Mei, Theodor Misiakiewicz, Andrea Montanari

Introduces mean-field theory for analyzing NN training

44. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, Jeffrey Pennington

Dynamics of NTK in parameter space

45. Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Arthur Jacot, Franck Gabriel, Clément Hongler

Paper that introduced NTK, nice kernel viewpoint of how NNs evolve during training at the infinite width limit

46. On the distance between two neural networks and the stability of learning

Jeremy Bernstein, Arash Vahdat, Yisong Yue, Ming-Yu Liu

Derived a spectral analysis of feature learning based on perturbation bounds, but obtained wrong scaling relation with network width due to flawed conditioning assumption on gradients

47. A Spectral Condition for Feature Learning

Greg Yang, James B. Simon, Jeremy Bernstein

Shows that using spectral instead of Frobenius norm to analyze how NNs change during training is the right framing. Good summary & generalization of previous Tensor Program series of work.

48. Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach

Grant M. Rotskoff, Eric Vanden-Eijnden

49. Limitations of the NTK for Understanding Generalization in Deep Learning

Nikhil Vyas, Yamini Bansal, Preetum Nakkiran

50. Spectral Normalization for Generative Adversarial Networks

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida

51. Layer rotation: a surprisingly powerful indicator of generalization in deep networks?

Simon Carbonnelle, Christophe De Vleeschouwer

52. Learning by Turning: Neural Architecture Aware Optimisation

Yang Liu, Jeremy Bernstein, Markus Meister, Yisong Yue

53. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen et al.

54. Scaling Data-Constrained Language Models

Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo et al.

55. Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li

56. TIES-Merging: Resolving Interference When Merging Models

Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, Mohit Bansal

57. Editing Models with Task Arithmetic

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi

58. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao

59. AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification

Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, Shanfeng Zhu

60. Jasper and Stella: distillation of SOTA embedding models

Dun Zhang, Jiacheng Li, Ziyang Zeng, Fulong Wang

61. YaRN: Efficient Context Window Extension of Large Language Models

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole

62. Zero Bubble Pipeline Parallelism

Penghui Qi, Xinyi Wan, Guangxing Huang, Min Lin

63. How Does Critical Batch Size Scale in Pre-training?

Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster et al.

64. Better & Faster Large Language Models via Multi-token Prediction

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve

65. Pointer Networks

Oriol Vinyals, Meire Fortunato, Navdeep Jaitly

66. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study

Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang et al.

67. Recursive Introspection: Teaching Language Model Agents How to Self-Improve

Yuxiao Qu, Tianjun Zhang, Naman Garg, Aviral Kumar

68. WARM: On the Benefits of Weight Averaged Reward Models

Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, Johan Ferret

69. Dual Operating Modes of In-Context Learning

Ziqian Lin, Kangwook Lee

70. Combining Induction and Transduction for Abstract Reasoning

Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M. Dunn et al.

71. Phi-4 Technical Report

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison et al.

72. The Impact of Positional Encoding on Length Generalization in Transformers

Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, Siva Reddy

73. RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning

Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, Gabriel Synnaeve

74. Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs

Jonas Hübotter, Sascha Bongni, Ido Hakimi, Andreas Krause

75. HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation

Yuhan Chen, Ang Lv, Jian Luan, Bin Wang, Wei Liu

76. Stream of Search (SoS): Learning to Search in Language

Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, Noah D. Goodman

77. Quadratic models for understanding catapult dynamics of neural networks

Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, Mikhail Belkin

78. Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning

Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, Mikhail Belkin

79. $μ$LO: Compute-Efficient Meta-Generalization of Learned Optimizers

Benjamin Thérien, Charles-Étienne Joseph, Boris Knyazev, Edouard Oyallon, Irina Rish, Eugene Belilovsky

80. VeLO: Training Versatile Learned Optimizers by Scaling Up

Luke Metz, James Harrison, C. Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal et al.

81. Probing the Decision Boundaries of In-context Learning in Large Language Models

Siyan Zhao, Tung Nguyen, Aditya Grover

82. Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers

Gautham Vasan, Mohamed Elsayed, Alireza Azimi, Jiamin He, Fahim Shariar, Colin Bellinger, Martha White et al.

83. VerMCTS: Synthesizing Multi-Step Programs using a Verifier, a Large Language Model, and Tree Search

David Brandfonbrener, Simon Henniger, Sibi Raja, Tarun Prasad, Chloe Loughridge, Federico Cassano, Sabrina Ruixin Hu et al.

84. xLSTM: Extended Long Short-Term Memory

Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer et al.

85. Memory-Efficient LLM Training with Online Subspace Descent

Kaizhao Liang, Bo Liu, Lizhang Chen, Qiang Liu

86. Efficient Model-Free Exploration in Low-Rank MDPs

Zakaria Mhammedi, Adam Block, Dylan J. Foster, Alexander Rakhlin

87. On the Computational Landscape of Replicable Learning

Alkis Kalavasis, Amin Karbasi, Grigoris Velegkas, Felix Zhou

88. TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications

David Salinas, Nick Erickson

89. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second

Noah Hollmann, Samuel Müller, Katharina Eggensperger, Frank Hutter

90. Reinforcement Learning: An Overview

Kevin Murphy

91. Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP

Sriram Balasubramanian, Samyadeep Basu, Soheil Feizi

92. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder et al.

93. Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices

Andres Potapczynski, Shikai Qiu, Marc Finzi, Christopher Ferri, Zixi Chen, Micah Goldblum, Bayan Bruss et al.

94. Compute Better Spent: Replacing Dense Layers with Structured Matrices

Shikai Qiu, Andres Potapczynski, Marc Finzi, Micah Goldblum, Andrew Gordon Wilson

95. CoLA: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebra

Andres Potapczynski, Marc Finzi, Geoff Pleiss, Andrew Gordon Wilson

96. Theoretical Foundations of Conformal Prediction

Anastasios N. Angelopoulos, Rina Foygel Barber, Stephen Bates

97. AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning

Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, Kaidi Cao, Vassilis N. Ioannidis et al.

98. Mechanism of feature learning in convolutional neural networks

Daniel Beaglehole, Adityanarayanan Radhakrishnan, Parthe Pandit, Mikhail Belkin

99. Average gradient outer product as a mechanism for deep neural collapse

Daniel Beaglehole, Peter Súkeník, Marco Mondelli, Mikhail Belkin

100. Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel

Colin Wei, Jason D. Lee, Qiang Liu, Tengyu Ma

101. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation

Mikhail Belkin

102. The duality structure gradient descent algorithm: analysis and applications to neural networks

Thomas Flynn

103. Old Optimizer, New Norm: An Anthology

Jeremy Bernstein, Laker Newhouse

104. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, Yoshua Bengio

105. Investigating the Limitations of Transformers with Simple Arithmetic Tasks

Rodrigo Nogueira, Zhiying Jiang, Jimmy Lin

106. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou et al.

107. Evaluating Language Models for Mathematics through Interactions

Katherine M. Collins, Albert Q. Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz et al.

108. Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks

Tiedong Liu, Bryan Kian Hsiang Low

109. Continual Pre-Training of Large Language Models: How to (re)warm your model?

Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish et al.

110. Functional Data Analysis: An Introduction and Recent Developments

Jan Gertheiss, David Rügamer, Bernard X. W. Liew, Sonja Greven

111. On Calibration of Modern Neural Networks

Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger

112. How Transformers Solve Propositional Logic Problems: A Mechanistic Analysis

Guan Zhe Hong, Nishanth Dikkala, Enming Luo, Cyrus Rashtchian, Xin Wang, Rina Panigrahy

113. LoRA vs Full Fine-tuning: An Illusion of Equivalence

Reece Shuttleworth, Jacob Andreas, Antonio Torralba, Pratyusha Sharma

114. An Empirical Model of Large-Batch Training

Sam McCandlish, Jared Kaplan, Dario Amodei, OpenAI Dota Team

115. The Description Length of Deep Learning Models

Léonard Blier, Yann Ollivier

116. ADOPT: Modified Adam Can Converge with Any $β_2$ with the Optimal Rate

Shohei Taniguchi, Keno Harada, Gouki Minegishi, Yuta Oshima, Seong Cheol Jeong, Go Nagahara, Tomoshi Iiyama et al.

117. Denoising Diffusion Probabilistic Models in Six Simple Steps

Richard E. Turner, Cristiana-Diana Diaconu, Stratis Markou, Aliaksandra Shysheya, Andrew Y. K. Foong, Bruno Mlodozeniec

118. Understanding Optimization in Deep Learning with Central Flows

Jeremy M. Cohen, Alex Damian, Ameet Talwalkar, Zico Kolter, Jason D. Lee

119. Modular Duality in Deep Learning

Jeremy Bernstein, Laker Newhouse

120. Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis

Rachel S. Y. Teo, Tan M. Nguyen

121. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective

Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, Tengyu Ma

122. Impacts of Continued Legal Pre-Training and IFT on LLMs' Latent Representations of Human-Defined Legal Concepts

Shaun Ho

123. A First Course in Monte Carlo Methods

Daniel Sanz-Alonso, Omar Al-Ghattas

124. Mixture of Parrots: Experts improve memorization more than reasoning

Samy Jelassi, Clara Mohri, David Brandfonbrener, Alex Gu, Nikhil Vyas, Nikhil Anand, David Alvarez-Melis et al.

125. The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities

Venkatesh Balavadhani Parthasarathy, Ahtsham Zafar, Aafaq Khan, Arsalan Shahid

126. Agentic Information Retrieval

Weinan Zhang, Junwei Liao, Ning Li, Kounianhua Du

127. nGPT: Normalized Transformer with Representation Learning on the Hypersphere

Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, Boris Ginsburg

128. Why Do We Need Weight Decay in Modern Deep Learning?

Francesco D'Angelo, Maksym Andriushchenko, Aditya Varre, Nicolas Flammarion

129. GLU Variants Improve Transformer

Noam Shazeer

130. Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

Stefan Elfwing, Eiji Uchibe, Kenji Doya

131. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning

Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal et al.

132. Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think

Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, Saining Xie

133. Generative Verifiers: Reward Modeling as Next-Token Prediction

Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal

134. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün et al.

135. Round and Round We Go! What makes Rotary Positional Encodings useful?

Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, Petar Veličković

136. Large Language Models as Markov Chains

Oussama Zekri, Ambroise Odonnat, Abdelhakim Benechehab, Linus Bleistein, Nicolas Boullé, Ievgen Redko

137. Searching for Best Practices in Retrieval-Augmented Generation

Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi et al.

138. Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers

Alicia Curth, Alan Jeffares, Mihaela van der Schaar

139. Reinforced Self-Training (ReST) for Language Modeling

Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant et al.

140. eGAD! double descent is explained by Generalized Aliasing Decomposition

Mark K. Transtrum, Gus L. W. Hart, Tyler J. Jarvis, Jared P. Whitehead

141. A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning

Alicia Curth, Alan Jeffares, Mihaela van der Schaar

142. Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs

Alicia Curth

143. Contextual Document Embeddings

John X. Morris, Alexander M. Rush

144. Were RNNs All We Needed?

Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio, Hossein Hajimirsadeghi

145. Instruction Following without Instruction Tuning

John Hewitt, Nelson F. Liu, Percy Liang, Christopher D. Manning

146. Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, Chuang Gan

147. From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models

Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov et al.

148. Chain of Thought Empowers Transformers to Solve Inherently Serial Problems

Zhiyuan Li, Hong Liu, Denny Zhou, Tengyu Ma

149. Hardware Acceleration of LLMs: A comprehensive survey and comparison

Nikoletta Koilia, Christoforos Kachris

150. Diffusion Models Learn Low-Dimensional Distributions via Subspace Clustering

Peng Wang, Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, Qing Qu

151. White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?

Yaodong Yu, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Hao Bai et al.

152. Just Say the Name: Online Continual Learning with Category Names Only via Data Generation

Minhyuk Seo, Seongwon Cho, Minjae Lee, Diganta Misra, Hyeonbeom Choi, Seon Joo Kim, Jonghyun Choi

153. Magicoder: Empowering Code Generation with OSS-Instruct

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, Lingming Zhang

154. Code Llama: Open Foundation Models for Code

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi et al.

155. #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models

Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou et al.

156. Equivariant neural networks and piecewise linear representation theory

Joel Gibson, Daniel Tubbenhauer, Geordie Williamson

157. Does your data spark joy? Performance gains from domain upsampling at the end of training

Cody Blakeney, Mansheej Paul, Brett W. Larsen, Sean Owen, Jonathan Frankle

158. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar

159. Reconciling modern machine learning practice and the bias-variance trade-off

Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal

160. Moment Matching for Multi-Source Domain Adaptation

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, Bo Wang

161. Invariant Risk Minimization

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, David Lopez-Paz

162. How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks

Divyansh Kaushik, Zachary C. Lipton

163. Do ImageNet Classifiers Generalize to ImageNet?

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar

164. The Effect of Natural Distribution Shift on Question Answering Models

John Miller, Karl Krauth, Benjamin Recht, Ludwig Schmidt

165. Causal inference using invariant prediction: identification and confidence intervals

Jonas Peters, Peter Bühlmann, Nicolai Meinshausen

166. Measuring the Intrinsic Dimension of Objective Landscapes

Chunyuan Li, Heerad Farkhoor, Rosanne Liu, Jason Yosinski

167. Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta

168. No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems

Nimit S. Sohoni, Jared A. Dunnmon, Geoffrey Angus, Albert Gu, Christopher Ré

169. Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization

Elan Rosenfeld, Pradeep Ravikumar, Andrej Risteski

170. In-Context Learning Learns Label Relationships but Is Not Conventional Learning

Jannik Kossen, Yarin Gal, Tom Rainforth

171. Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data

Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak et al.

172. Language Models (Mostly) Know What They Know

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer et al.

173. A Tutorial on Bayesian Optimization

Peter I. Frazier

174. Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson

175. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh

176. The Geometry of Categorical and Hierarchical Concepts in Large Language Models

Kiho Park, Yo Joong Choe, Yibo Jiang, Victor Veitch

177. When Representations Align: Universality in Representation Learning Dynamics

Loek van Rossem, Andrew M. Saxe

178. Scaling Synthetic Data Creation with 1,000,000,000 Personas

Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, Dong Yu

179. A Theory of Interpretable Approximations

Marco Bressan, Nicolò Cesa-Bianchi, Emmanuel Esposito, Yishay Mansour, Shay Moran, Maximilian Thiessen

180. BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment

Wenda Xu, Jiachen Li, William Yang Wang, Lei Li

181. A Tutorial on Thompson Sampling

Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen

182. Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference

Siddhartha Dalal, Vishal Misra

183. Transcendence: Generative Models Can Outperform The Experts That Train Them

Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade et al.

184. Step-by-Step Diffusion: An Elementary Tutorial

Preetum Nakkiran, Arwen Bradley, Hattie Zhou, Madhu Advani

185. Text Embeddings Reveal (Almost) As Much As Text

John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, Alexander M. Rush

186. Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks

Giovanni Luca Marchetti, Christopher Hillar, Danica Kragic, Sophia Sanborn

187. Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models

Sander Land, Max Bartolo

188. Customizing Text-to-Image Models with a Single Image Pair

Maxwell Jones, Sheng-Yu Wang, Nupur Kumari, David Bau, Jun-Yan Zhu

189. Self-Play Preference Optimization for Language Model Alignment

Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, Quanquan Gu

190. Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

Shreya Shankar, J. D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, Ian Arawjo

191. U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Models

Song Mei

192. On the Bottleneck of Graph Neural Networks and its Practical Implications

Uri Alon, Eran Yahav

193. How Can We Know What Language Models Know?

Zhengbao Jiang, Frank F. Xu, Jun Araki, Graham Neubig

194. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam et al.

195. Why do tree-based models still outperform deep learning on tabular data?

Léo Grinsztajn, Edouard Oyallon, Gaël Varoquaux

196. On the Complexity of Best Arm Identification in Multi-Armed Bandit Models

Emilie Kaufmann, Olivier Cappé, Aurélien Garivier

197. Random Utility Theory for Social Choice

Hossein Azari Soufiani, David C. Parkes, Lirong Xia

198. Black-box Dataset Ownership Verification via Backdoor Watermarking

Yiming Li, Mingyan Zhu, Xue Yang, Yong Jiang, Tao Wei, Shu-Tao Xia

199. Decoupled Weight Decay Regularization

Ilya Loshchilov, Frank Hutter

200. Efficient Training of Language Models to Fill in the Middle

Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, Mark Chen

201. From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Rafael Rafailov, Joey Hejna, Ryan Park, Chelsea Finn

202. Large Language Models Can Self-Improve

Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han

203. How to Train Data-Efficient LLMs

Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee et al.

204. Co-training Improves Prompt-based Learning for Large Language Models

Hunter Lang, Monica Agrawal, Yoon Kim, David Sontag

205. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden et al.

206. Learning Transformer Programs

Dan Friedman, Alexander Wettig, Danqi Chen

207. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

Noam Shazeer, Mitchell Stern

208. Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data

Yuanzhi Li, Yingyu Liang

209. A Convergence Theory for Deep Learning via Over-Parameterization

Zeyuan Allen-Zhu, Yuanzhi Li, Zhao Song

210. Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian

Samet Oymak, Zalan Fabian, Mingchen Li, Mahdi Soltanolkotabi

211. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning

Armen Aghajanyan, Luke Zettlemoyer, Sonal Gupta

212. STaR: Bootstrapping Reasoning With Reasoning

Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman

213. Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks

Hao Chen, Jindong Wang, Ankit Shah, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama et al.

214. The Barron Space and the Flow-induced Function Spaces for Neural Network Models

Weinan E, Chao Ma, Lei Wu

215. Deep Equilibrium Based Neural Operators for Steady-State PDEs

Tanya Marwah, Ashwini Pokle, J. Zico Kolter, Zachary C. Lipton, Jianfeng Lu, Andrej Risteski

216. Parametric Complexity Bounds for Approximating PDEs with Neural Networks

Tanya Marwah, Zachary C. Lipton, Andrej Risteski

217. Simple linear attention language models balance the recall-throughput tradeoff

Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou et al.

218. Chain-of-Thought Reasoning Without Prompting

Xuezhi Wang, Denny Zhou

219. Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference

Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski, David Tarjan, Edoardo M. Ponti

220. Evolutionary Optimization of Model Merging Recipes

Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, David Ha

221. An Invitation to Deep Reinforcement Learning

Bernhard Jaeger, Andreas Geiger

222. Deep Neural Networks Tend To Extrapolate Predictably

Katie Kang, Amrith Setlur, Claire Tomlin, Sergey Levine

223. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi et al.

224. Is Cosine-Similarity of Embeddings Really About Similarity?

Harald Steck, Chaitanya Ekanadham, Nathan Kallus

225. Language Modeling with Gated Convolutional Networks

Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier

226. GLU Variants Improve Transformer

Noam Shazeer

227. On the Measure of Intelligence

François Chollet

228. Asymmetry in Low-Rank Adapters of Foundation Models

Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi et al.

229. Scalable Diffusion Models with Transformers

William Peebles, Saining Xie

230. In-Context Learning for Extreme Multi-Label Classification

Karel D'Oosterlinck, Omar Khattab, François Remy, Thomas Demeester, Chris Develder, Christopher Potts

231. Matryoshka Representation Learning

Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder et al.

232. Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization

Elan Rosenfeld, Andrej Risteski

233. Diffusion Models for Generative Artificial Intelligence: An Introduction for Applied Mathematicians

Catherine F. Higham, Desmond J. Higham, Peter Grindrod

234. In-context Learning and Induction Heads

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann et al.

235. Characterizing Implicit Bias in Terms of Optimization Geometry

Suriya Gunasekar, Jason Lee, Daniel Soudry, Nathan Srebro

236. Scaling Instruction-Finetuned Language Models

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li et al.

237. Textbooks Are All You Need II: phi-1.5 technical report

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, Yin Tat Lee

238. Data Selection for Language Models via Importance Resampling

Sang Michael Xie, Shibani Santurkar, Tengyu Ma, Percy Liang

239. Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

240. Data Movement Is All You Need: A Case Study on Optimizing Transformers

Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, Torsten Hoefler

241. Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars

Kaiyue Wen, Yuchen Li, Bingbin Liu, Andrej Risteski

242. Accelerating LLM Inference with Staged Speculative Decoding

Benjamin Spector, Chris Re

243. Accelerating Large Language Model Decoding with Speculative Sampling

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper

244. Fast Inference from Transformers via Speculative Decoding

Yaniv Leviathan, Matan Kalman, Yossi Matias

245. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer

246. Masked Autoencoders Are Scalable Vision Learners

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick

247. One Wide Feedforward is All You Need

Telmo Pessoa Pires, António V. Lopes, Yannick Assogba, Hendra Setiawan

248. A Mean Field View of the Landscape of Two-Layers Neural Networks

Song Mei, Andrea Montanari, Phan-Minh Nguyen

249. Sharpness-Aware Minimization for Efficiently Improving Generalization

Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur

250. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability

Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, Ameet Talwalkar

251. The large learning rate phase of deep learning: the catapult mechanism

Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, Guy Gur-Ari

252. Label Noise SGD Provably Prefers Flat Global Minimizers

Alex Damian, Tengyu Ma, Jason D. Lee

253. Gradient Descent Maximizes the Margin of Homogeneous Neural Networks

Kaifeng Lyu, Jian Li

254. Deep Double Descent: Where Bigger Models and More Data Hurt

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever

255. The generalization error of random features regression: Precise asymptotics and double descent curve

Song Mei, Andrea Montanari

256. Exploring Generalization in Deep Learning

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, Nathan Srebro

257. Understanding deep learning requires rethinking generalization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals

258. On Exact Computation with an Infinitely Wide Neural Net

Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang

259. Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers

Zeyuan Allen-Zhu, Yuanzhi Li, Yingyu Liang

260. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Jonathan Frankle, Michael Carbin

261. A Simple Framework for Contrastive Learning of Visual Representations

Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton

262. Big Transfer (BiT): General Visual Representation Learning

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby

263. A Fourier Perspective on Model Robustness in Computer Vision

Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, Justin Gilmer

264. Certified Adversarial Robustness via Randomized Smoothing

Jeremy M Cohen, Elan Rosenfeld, J. Zico Kolter

265. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani et al.

266. Vision Transformers are Robust Learners

Sayak Paul, Pin-Yu Chen

267. On the Adversarial Robustness of Vision Transformers

Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, Cho-Jui Hsieh

268. Extracting Training Data from Large Language Models

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts et al.

269. Supervised Contrastive Learning

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot et al.

270. Efficiently Modeling Long Sequences with Structured State Spaces

Albert Gu, Karan Goel, Christopher Ré