Mini-AGI是一个开源字节级连续学习语言模型，可在单块8GB显存的GPU上从零开始训练1。该项目通过将权重存储在磁盘上并按需调入显存的方式突破了显存限制，使个人PC或笔记本用户能够进行无限继续学习1。
项目采用了自适应深度和动态专家路由等创新架构设计1。目前训练进度已达318.1M字符，包含169个专家，总参数规模540.1M1。其中32个专家常驻显存（约109M参数），整体专家池由磁盘空间而非显存容量决定1。模型使用字节级词汇表（256字节值加9个结构标记），上下文长度为4,0961。
在连续学习能力方面，当干预学习率设置为专家学习率的0.1倍时，模型的连续学习遗忘率从+2.23降至+0.0067nat，知识保留率达99.84%1。项目由Alexey Borsky主导，并在Claude Opus 5模型的协助下完成了大部分代码实现1。
Mini-AGI is an open-source byte-level continual learning language model designed to train from scratch on a single 8GB GPU by storing weights on disk and dynamically loading them into memory as needed 1. This architecture circumvents conventional VRAM constraints, allowing the model pool size to be limited only by available disk space rather than GPU memory 1. The project demonstrates that continual learning without catastrophic forgetting is achievable on consumer-grade hardware, including personal computers and laptops 1.
The model employs adaptive depth and dynamic expert routing as core architectural innovations 1. Currently, the project has trained on 318.1 million characters with 169 experts and a total of 540.1 million parameters 1. The implementation uses a byte-level vocabulary consisting of 256 byte values plus 9 structural tokens, with a context length of 4,096 1. At any given time, 32 experts remain resident in VRAM, approximately 109 million parameters, while the complete expert pool spans all 169 experts 1. Through tuning the intervention learning rate to 0.1 times the expert learning rate, the project reduced continual learning forgetting from +2.23 to +0.0067 nats, achieving 99.84 percent knowledge retention 1.
The project was led by Alexey Borsky and released in September 2026, with Claude Opus 5 assisting in implementing the majority of the codebase 1.
评论
还没有评论，欢迎留下第一条。