开发者Vaibhaav Tiwari在Hacker News社区发布了名为“deepseek-v3-jax”的开源项目,使用190行纯JAX代码复现了DeepSeek-V3的技术架构 1。该实现未依赖Flax、Equinox或PyTorch等框架,而是从头开始预训练模型 1。作者明确指出,这是一个架构演示而非具备强大能力的语言模型 1。
项目运行于配备24GB内存的MacBook Air M5笔记本上 1。数据集采用约207本公共领域的侦探小说(如达希尔·哈米特作品),总计约83.4M字符 1。由此生成的微型模型拥有36.2M参数、6层网络结构及256-token上下文窗口,与原版DeepSeek-V3的671B参数、61层结构及128K上下文形成对比 1。
技术限制方面,该简化版实现省略了FP8训练、DualPipe、节点限制路由、YaRN上下文扩展及后训练步骤 1。验证结果显示,在Step 2000时,开启与关闭负载均衡偏差的验证损失几乎相同(3.921 vs 3.923) 1。
项目代码采用MIT许可证,文本数据在美国为公共领域 1。依赖版本要求为Python 3.12和jax-metal 0.1.1(仅兼容jax==0.4.34) 1。
Developer Vaibhaav Tiwari released an open-source project titled "deepseek-v3-jax" on Hacker News, which reproduces the DeepSeek-V3 technical architecture using approximately 190 lines of pure JAX code 1. The implementation is built from scratch and relies exclusively on JAX, without utilizing frameworks such as Flax, Equinox, or PyTorch 1. It runs on a MacBook Air M5 equipped with 24 GB of memory 1.
The model was pre-trained on a dataset comprising roughly 207 public-domain detective novels, including works by Dashiell Hammett, totaling about 83.4 million characters 1. This simplified version contains 36.2 million parameters across six layers with a 256-token context window, compared to the original DeepSeek-V3's 671 billion parameters, 61 layers, and 128K context capacity 1. The author explicitly states that this project serves as an architectural demonstration rather than a powerful language model 1. Advanced features such as FP8 training, DualPipe, node-limited routing, YaRN context extension, and post-training processes are not included 1.
Technical dependencies for the project include Python 3.12 and jax-metal version 0.1.1, which is compatible only with jax==0.4.34 1. Validation results showed nearly identical loss at step 2000 whether load balancing bias was enabled or disabled (3.921 vs. 3.923) 1. The source code is distributed under the MIT license, while the training texts are in the public domain within the United States 1.
评论
还没有评论,欢迎留下第一条。