开源项目Petals让用户在家以BitTorrent方式运行大型语言模型

Petals是一个开源项目，使用户能够在家中以类似BitTorrent的分布式方式运行大型语言模型 1。该平台通过连接网络中其他用户共享模型部分来完成推理任务 1。

项目支持多款主流大型语言模型，包括Llama 3.1（405B）、Mixtral（8x22B）、Falcon（40B+）和BLOOM（176B） 1。在推理速度方面，Llama 2（70B）可达到6 tokens/sec，Falcon（180B）可达到4 tokens/sec 1。

硬件要求相对易达，用户可使用消费级GPU或Google Colab进行运行 1。该项目提供了包括文本生成、微调、自定义采样方法和查看隐藏状态在内的多项功能特性 1。

Petals是BigScience研究工作坊的一部分 1。

Petals is an open-source project that enables users to run large language models at home through a distributed, peer-to-peer approach similar to BitTorrent 1. The platform allows users to perform inference by connecting to other participants in the network who share different portions of the model 1.

The project supports several state-of-the-art models, including Llama 3.1 (405B), Mixtral (8x22B), Falcon (40B+), and BLOOM (176B) 1. Inference speeds vary depending on the model: Llama 2 (70B) achieves approximately 6 tokens per second, while Falcon (180B) reaches around 4 tokens per second 1.

Hardware requirements are modest, as the system can run on consumer-grade GPUs or Google Colab 1. Beyond text generation, Petals offers additional capabilities including model fine-tuning, custom sampling methods, and the ability to inspect hidden states 1. The project emerged from the BigScience research workshop 1.