开发者发布了一份技术教程,展示了如何在两台通过RDMA连接的NVIDIA DGX Stations上,利用vLLM框架本地部署并运行753亿参数的GLM-5.3大模型 1。该方案针对单卡显存不足以容纳约464 GB模型检查点的问题,采用张量并行(TP=2)技术将模型权重分割至两台设备的GB300 GPU中 1。硬件环境具体配置为两台配备GB300 GPU的NVIDIA DGX Station,并通过ConnectX-8 RDMA进行高速互联 1。
在软件栈方面,教程使用vLLM v0.30.0-aarch64版本,并启用了多令牌预测(MTP)、FP8 KV缓存存储以及前缀缓存(Prefix Caching)等关键优化配置 1。基准测试数据显示,在并发数为16的情况下,整体输出吞吐量峰值达到607.8 tok/s 1。然而,平均单用户吞吐量随并发数增加而显著下降,从并发数为1时的115.3 tok/s降至并发数为32时的13.0 tok/s 1。此外,该部署方案将最大模型上下文长度限制设置为98304 tokens 1。
A technical tutorial details the local deployment and operation of the 75.3-billion-parameter large language model GLM-5.3 using the vLLM framework across two NVIDIA DGX Stations 1. The setup addresses single-GPU memory limitations by employing tensor parallelism (TP=2) to split model weights between the GB300 GPUs in each station, which are interconnected via ConnectX-8 RDMA 1. The specific model variant used is NVIDIA GLM-5.3 NVFP4, featuring approximately 4 billion active parameters and a checkpoint size of about 464 GB 1.
The implementation relies on vLLM version v0.30.0-aarch64 and incorporates key performance optimizations such as multi-token prediction (MTP), FP8 KV cache storage, and prefix caching 1. The maximum context length for the model is configured at 98,304 tokens 1. Benchmark tests demonstrate that the system achieves a peak overall output throughput of 607.8 tokens per second at a concurrency level of 16 1. As user load increases, average single-user throughput decreases from 115.3 tokens per second at concurrency 1 to 13.0 tokens per second at concurrency 32 1.
评论
还没有评论,欢迎留下第一条。