摘要：
Large Language Models (LLMs) have revolutionized natural language processing in recent years, demonstrating remarkable abilities in tasks ranging from text generation to reasoning. These models, trained on vast amounts of text data, have become increasingly powerful as they grow in size and complexity.

In this talk, I will introduce how to train large language models (LLMs) from scratch. I will start by introducing the basic building blocks of LLMs, the transformer architecture. Then, I will explain the typical three-stage training pipeline: from tokenizer training to pretraining, instruction tuning and finally reinforcement learning from human feedback (RLHF). Lastly, I'll discuss some practical challenges and solutions when scaling up these models, including scaling laws and various acceleration techniques.

论坛简介：该线上论坛是由张志华教授机器学习实验室组织，每两周主办一次（除了公共假期）。论坛每次邀请一位博士生就某个前沿课题做较为系统深入的介绍，主题包括但不限于机器学习、高维统计学、运筹优化和理论计算机科学。