Adaptive Learning Rate Strategies for Training Large Language Models: Balancing Speed and Stability

EasyChair Preprint 12276

Abstract

The training of large language models (LLMs) demands a delicate equilibrium between speed and stability. Conventional fixed learning rate approaches often encounter challenges in efficiently converging. In this paper, a novel framework is proposed for adaptive learning rate strategies tailored specifically for LLM training. The framework addresses the challenge of dynamically optimizing learning rates throughout the training process to enhance convergence speed and stability. Leveraging insights from adaptive optimization algorithms and recent advancements in large-scale language model training, a comprehensive analysis of various adaptive learning rate techniques and their implications for LLM training is presented.

Keyphrases: adaptive, learning, rate

@booklet{EasyChair:12276,
  author    = {Kurez Oroy and Robert Chris},
  title     = {Adaptive Learning Rate Strategies for Training Large Language Models: Balancing Speed and Stability},
  howpublished = {EasyChair Preprint 12276},
  year      = {EasyChair, 2024}}