一名开发者成功在价格约8美元的ESP32-S3微控制器上运行了2890万参数的语言模型1。这一突破相比之前在同类芯片上运行的26万参数模型，参数量增加了约100倍1。该模型在芯片本地运行，无需连接服务器1，文本生成速度约为每秒9tokens1。
开发者通过采用Google Gemma模型中的Per-Layer Embeddings技术实现了这一成果1，该技术将大部分模型参数存储在慢速闪存中，充分利用了芯片512KB的SRAM容量1。所用的语言模型基于TinyStories训练数据集1，展示了边缘设备上本地文本生成的可行性。
A developer has successfully deployed a 28.9 million parameter language model on an ESP32-S3 microcontroller, a device that costs approximately $8.1 The model runs entirely on the chip itself without requiring a connection to external servers, generating text at a rate of approximately 9 tokens per second.1 This represents a roughly 100-fold increase in model size compared to the 260,000 parameter models previously demonstrated on similar hardware.1
The achievement was made possible through the use of Per-Layer Embeddings technology, a technique derived from Google's Gemma models, which allows most of the model parameters to be stored in the chip's slower flash memory rather than its limited 512KB of SRAM.1 The underlying model was trained on the TinyStories dataset.1
评论
还没有评论，欢迎留下第一条。