[Submitted on 13 Feb 2023]
Authors:Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, Quoc V. Le
Abstract: We present a method to formulate algorithm discovery as program search, and
apply it to discover optimization algorithms for deep neural network training.
We leverage efficient search techniques to explore an infinite and sparse
program space. To bridge the large generalization gap between proxy and target
tasks, we also introduce program selection and simplification strategies. Our
method discovers a simple and effective optimization algorithm, $textbf{Lion}$
($textit{Evo$textbf{L}$ved S$textbf{i}$gn M$textbf{o}$me$textbf{n}$tum}$).
It is more memory-efficient than Adam as
