开发者以998美元训练3.8B参数模型，性能超越GPT-2

Hugo Vergnes利用开源框架little-lm在NVIDIA B200 GPU上训练了一个包含3.8B参数的大语言模型，项目耗资仅998美元，完成时间为43小时1。该模型在CORE基准上获得0.384分，超过了GPT-2 1.5B模型的0.2565分1。

这一项目展示了个人开发者以较低成本训练有意义模型的可行性1。训练过程采用了多项优化技术，包括Muon优化器、FP8混合精度训练和Trapezoidal学习率调度1。模型在65B tokens的数据上进行训练，稳定状态下吞吐量达480,000 tokens/秒，GPU利用率达92% SM活动水平1。值得注意的是，该模型的值嵌入参数占比达19%，共占用721M参数1。

Hugo Vergnes has demonstrated the feasibility of training a capable large language model at minimal cost by successfully training a 3.8 billion parameter model on NVIDIA B200 GPUs for just $998 over 43 hours 1. The model, developed using the open-source little-lm framework, achieved a CORE benchmark score of 0.384 1, surpassing the performance of GPT-2's 1.5B model, which scored 0.2565 on the same benchmark 1.

The training process processed 65 billion tokens and maintained a throughput of 480,000 tokens per second at stable state 1. Vergnes employed several optimization techniques to maximize efficiency, including the Muon optimizer, FP8 mixed-precision training, and a trapezoidal learning rate schedule 1. The setup achieved 92% streaming multiprocessor activity and 25% Model FLOPs Utilization relative to FP8 peak performance 1. The model's architecture allocated 721 million parameters—approximately 19% of total parameters—to value embeddings 1.