开源项目成功在价格仅为8美元的ESP32-S3微控制器上从零开始训练了一个包含31.9万参数的语言模型[1]。该项目采用克林贡语言作为测试语言，实现了完整的训练循环完全在芯片上进行，包括前向传播、反向传播和权重更新[1]。这一成果证明了微控制器可以独立学习，无需依赖预训练模型或外部数据源[1]。
项目采用单块transformer架构，配备单头因果注意力机制和共享权重设计[1]。训练过程中，权重占用内存1.3 MB，训练阶段总共需要数MB PSRAM，耗时数小时至数天，最终模型损失值约为1.87[1]。所有梯度通过手工推导的C代码实现，相对误差控制在1.07e-08以内，词汇表包含约31个符号[1]。该项目采用Apache 2.0开源许可证发布[1]。
An open-source project has successfully trained a 319,000-parameter language model entirely on an $8 ESP32-S3 microcontroller, demonstrating that neural networks can learn independently on resource-constrained hardware.[1] The complete training loop—encompassing forward propagation, backpropagation, and weight updates—runs directly on the chip without reliance on pretrained models or external data sources.[1] The project used Klingon as a test language and achieved a final loss value of approximately 1.87 across a vocabulary of roughly 31 symbols.[1]
The implementation employed a single-block transformer architecture with single-head causal attention and shared weights.[1] All gradients were hand-derived and implemented in C code, achieving a maximum relative error of 1.07e-08.[1] The model's weights require 1.3 MB of memory, with several additional megabytes of PSRAM needed during the training process, which takes anywhere from several hours to several days depending on configuration.[1] The project is released under the Apache 2.0 license.[1]