OpenArch：现代大语言模型架构的PyTorch实现库

OpenArch是一个开源PyTorch项目，旨在通过清晰易读的代码帮助学习者理解大语言模型的构建原理1。该项目计划实现72个LLM架构1，目前已包含GPT2、Llama3、Qwen3、Grok2.5、DeepSeek V3、Pali-Gemma等多个主流模型的实现1。

项目重点强调代码可读性而非生产性能1，在架构设计上支持多种注意力机制，包括多头注意力（MHA）、分组查询注意力（GQA）、多查询注意力（MQA）、多头潜在注意力（MLA）、滑动窗口、线性和DeltaNet混合体等1。规范化方法方面，OpenArch涵盖了pre-norm、post-norm、QK-Norm、sandwich norm和RMSNorm1；位置编码方面支持RoPE、NoPE、partial RoPE和YaRN等多种方案1。该项目采用Apache License 2.0许可证1，欢迎社区贡献1。

OpenArch is an open-source PyTorch project that provides hand-written implementations of modern large language model architectures designed to help learners understand how LLMs are constructed 1. The project prioritizes code readability over production performance and welcomes community contributions 1.

The project aims to implement 72 different LLM architectures, benchmarking itself against Sebastian Raschka's LLM Architecture Gallery 1. Currently implemented models include GPT2, Llama3, Qwen3, Grok2.5, DeepSeek V3, and Pali-Gemma, among others 1. OpenArch supports multiple attention mechanisms—including Multi-Head Attention (MHA), Grouped Query Attention (GQA), Multi-Query Attention (MQA), Multi-Head Latent Attention (MLA), sliding window attention, and linear/DeltaNet hybrid variants—as well as various normalization techniques such as pre-norm, post-norm, QK-Norm, sandwich norm, and RMSNorm 1. The implementations also cover diverse positional encoding methods, including RoPE, NoPE, partial RoPE, and YaRN 1.

Released under the Apache License 2.0 1, OpenArch explicitly positions itself as a learning resource rather than a production-ready tool, making it accessible for developers and researchers seeking to deepen their understanding of contemporary LLM design principles 1.