Research

Memory-Efficient Training

  1. galore.png
    GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
    Jiawei Zhao, Zhenyu Zhang , Beidi Chen , and 3 more authors
    In International Conference on Machine Learning (ICML) , 2024
    arXiv Code
  2. inrank.png
    InRank: Incremental Low-Rank Learning
    Jiawei Zhao*, Yifei Zhang* , Beidi Chen , and 2 more authors
    ES-FoMo Workshop at International Conference on Machine Learning (ICML), 2023
    arXiv Code

Low-Precision Training

  1. lns_madam.png
    LNS-Madam: Low-Precision Training in Logarithmic Number System Using Multiplicative Weight Update
    Jiawei Zhao, Steve Dai , Rangharajan Venkatesan , and 6 more authors
    IEEE Transactions on Computers | US Patent 17/346,100, 2022
    arXiv
  2. madam.png
    Learning compositional functions via multiplicative weight updates
    Jeremy Bernstein , Jiawei Zhao, Markus Meister , and 3 more authors
    In Advances in Neural Information Processing Systems (NeurIPS) , 2020
    arXiv Code

Distributed Training

  1. signsgd.jpg
    signSGD with Majority Vote is Communication Efficient and Fault Tolerant
    Jeremy Bernstein* , Jiawei Zhao*, Kamyar Azizzadenesheli , and 1 more author
    In International Conference on Learning Representations (ICLR) , 2019
    arXiv Code

Understanding Training Dynamics

  1. zero_init.png
    ZerO Initialization: Initializing Neural Networks with only Zeros and Ones
    Jiawei Zhao, Florian Tobias Schaefer , and Anima Anandkumar
    Transactions on Machine Learning Research (TMLR), 2022
    arXiv Code

Please refer to my Google Scholar for a complete list of publications.