开发者Samir近期发布了一个名为“Hours of YouTube and 85 handtyped pages of notes on LLM Inference – Start here”的教学项目,旨在通过从零开始的实现来解析大语言模型(LLM)的推理原理1。该项目构建了一个小型LLM推理引擎,核心代码采用C++17和CUDA编写,并依赖ICU库进行Unicode Tokenization处理1。配套资源包括手写笔记、视频讲解及开源代码,详细覆盖了从模型权重加载、Tokenization、Transformer层计算到KV缓存管理及批处理调度的完整技术路径1。
在技术实现上,该引擎针对Llama 3.2 1B Instruct模型进行了优化,支持包含16层Transformer结构的固定布局1。系统直接加载model.safetensors文件,并将BF16权重偏移量验证后存入GPU内存1。架构中实现了RMSNorm、RoPE、分组查询注意力机制(GQA)以及SwiGLU MLP等关键组件1。为了优化显存使用,项目采用了共享BF16 KV页面池策略,默认大小为128 MiB,每页存储16个token的K值和V值1。解码阶段利用带FP32在线Softmax的直接分页注意力机制,避免了非连续前缀收集带来的性能损耗1。
调度与批处理方面,引擎支持静态批处理(Static Batching)和连续批处理(Continuous Batching)两种模式1。预填充(Prefill)步骤每次仅处理一个提示,并与解码步骤交替执行1。请求准入逻辑基于最大处理前缀预算(即prompt_tokens加上allowed_new_tokens减1的结果),并向上取整对齐至16-token页面单位1。需要注意的是,由于BF16计算特性,在不同批量大小下可能导致贪婪输出令牌存在差异,因此不保证位级一致性1。此外,虽然代码部分遵循MIT许可证,但作者的手写笔记和图片资料保留所有权利1。
Samir has released an open-source educational project featuring a small language model inference engine built from scratch using C++17 and CUDA. 1 The initiative aims to teach the principles of large language model (LLM) inference by providing comprehensive resources, including handwritten notes, video explanations, and source code. 1
The engine is specifically designed for the Llama 3.2 1B Instruct model, implementing its fixed 16-layer Transformer structure. 1 It handles technical processes ranging from loading model weights directly from model.safetensors and verifying BF16 weight offsets in GPU memory to Unicode tokenization via ICU dependencies. 1 Key architectural components implemented include RMSNorm, RoPE, Grouped-Query Attention (GQA), and SwiGLU MLPs. 1
Memory management is optimized through a shared BF16 KV page pool, which defaults to 128 MiB, with each page storing key-value pairs for 16 tokens. 1 The system supports both static batching and continuous batching, utilizing direct paged attention with FP32 online Softmax during decoding to avoid non-contiguous prefix collection. 1 Prefill operations process one prompt at a time, alternating with decode steps, while request admission is based on a maximum processed prefix budget calculated as prompt_tokens + allowed_new_tokens - 1 and rounded up to 16-token pages. 1
Notably, the use of BF16 computation may result in different greedy output tokens depending on batch sizes, meaning bit-level consistency is not guaranteed. 1 While portions of the code are licensed under MIT, the handwritten notes and images retain all rights reserved by the author. 1
评论
还没有评论,欢迎留下第一条。