针对大规模AI训练与推理场景，SkyPilot平台提出了一种面向NVIDIA NVL72机架级系统的拓扑感知调度策略 1。传统调度方式常导致多节点作业跨机架通信，从而受限于InfiniBand带宽；数据显示，若使用InfiniBand而非NVLink进行分布式训练可能导致性能下降43% 1。相比之下，NVL72系统在一个机架内连接72个GPU，提供每GPU 1.8 TB/s的带宽和130 TB/s的总内部带宽 1。为最大化利用这一硬件优势，新策略通过保持作业在机架内局部运行、利用备用节点进行原地故障恢复以及自动碎片整理来提升集群效率 1。
在具体配置上，将18节点的机架划分为16节点切片并保留2个备用节点，可显著增加可用GPU容量 1。Meta此前测量显示，其超级集群中4-GPU节点的平均中断时间（MTBI）约为490天，而NVL72机架的MTBI约为27天，表明机架级故障更为频繁 1。实验数据证实，在128机架集群上，采用16节点切片比18节点切片多出27%的可用容量，具体数值从1,602节点提升至2,037节点 1。
在实际应用层面，NVIDIA GR00T团队在512 GB300 GPU上的实验结果显示，实施机架感知放置使训练吞吐量提高了14.4%，迭代时间从1.59秒降至1.39秒 1。这一优化在超大规模集群中具有显著的经济价值，例如在10,000-GPU集群上，相同的速度优化每小时可节省约$16.7K 1。这些高效管理功能由SkyPilot的Topology-Aware Scheduler (TAS)、Frontier Trainer和Automatic Defragmentation模块共同实现 1。
NVIDIA’s NVL72 rack-scale systems interconnect 72 GPUs within a single rack, providing up to 1.8 TB/s of bandwidth per GPU and a total internal bandwidth of 130 TB/s 1. However, traditional scheduling approaches for large-scale AI training often force multi-node jobs to communicate across racks via InfiniBand, which can result in performance degradation of up to 43% compared to utilizing NVLink 1. To address this bottleneck, the SkyPilot platform has proposed a topology-aware scheduling strategy that maximizes cluster efficiency by keeping jobs localized within individual racks 1.
A central component of this approach involves partitioning 18-node racks into smaller slices while reserving capacity for fault tolerance and maintenance. By structuring clusters into 16-node slices with two backup nodes, operators can increase usable GPU capacity by 27%; experiments on a 128-rack cluster showed available nodes rising from 1,602 to 2,037 under this configuration 1. This architectural shift is critical given the reliability challenges of NVL72 hardware; Meta measurements indicate an average Mean Time Between Interruptions (MTBI) of approximately 490 days for standard 4-GPU nodes, whereas the MTBI for NVL72 racks drops significantly to around 27 days 1. Consequently, maintaining spare nodes allows for in-place fault recovery without expanding the overall footprint 1.
The performance benefits of this strategy were validated during pre-training experiments with NVIDIA’s GR00T model on 512 GB300 GPUs, where rack-aware placement improved throughput by 14.4% and reduced iteration time from 1.59 seconds to 1.39 seconds 1. When scaled to larger environments, such as a 10,000-GPU cluster, these speed optimizations translate to cost savings of approximately $16.7K per hour 1. SkyPilot achieves these gains through its Topology-Aware Scheduler (TAS), Frontier Trainer, and automatic defragmentation capabilities, ensuring that resources remain efficiently allocated despite the higher failure rates of dense rack systems 1.
评论
还没有评论，欢迎留下第一条。