研究人员成功在AMD MI355X GPU上部署了参数量达2.8T的大模型Kimi K3 1。该模型需要超过1.5TB的显存 1，在1024-token输入/400-token输出的基准测试中，MI355X实现了952 tok/s/node的吞吐量 1。
性能成本方面，MI355X的运营成本为每小时每GPU 2.50美元，而NVIDIA B300为每小时每GPU 6.00美元 1。相比之下，MI355X在成本效益上优于B300 1.65倍 1。单流延迟方面，MI355X达到118 tok/s，低于B200在TP16部署下249 tok/s/node的表现 1。
研究团队通过修复ROCm框架中采样验证器的NameError缺陷，使单流性能提升了约2.2倍 1。同时，他们优化了prefill算法，利用AITER MLA kernel将172k-token冷prefill的耗时从约51秒降低至每秒约13k token的稳态性能 1。这些优化举措充分展示了AMD GPU在大规模模型推理服务中的竞争潜力 1。
Researchers have successfully deployed the Kimi K3 large language model, which contains 2.8 trillion parameters, on AMD's MI355X GPU, demonstrating significant cost advantages over NVIDIA's B300 accelerator.1 The MI355X achieved a throughput of 952 tokens per second per node on a benchmark test with 1,024-token input and 400-token output.1
The performance economics strongly favor AMD's hardware in this deployment scenario.1 The MI355X operates at a cost of $2.50 per GPU-hour, compared to B300's $6.00 per GPU-hour, giving the MI355X a 2.4-fold price advantage.1 Single-stream latency for the MI355X reached 118 tokens per second, while B200 configurations deployed with tensor parallelism across 16 nodes achieved 249 tokens per second per node.1
The research team optimized performance through targeted infrastructure improvements.1 After resolving a NameError defect in the sampling validator related to top_k_renorm_prob within the ROCm framework, single-stream performance increased approximately 2.2 times.1 Additionally, cold prefill optimization for 172,000-token sequences was accelerated from approximately 51 seconds to a steady-state rate of approximately 13,000 tokens per second through implementation of an AITER MLA kernel.1 These results underscore AMD MI355X GPUs' competitive potential for large language model serving workloads.
评论
还没有评论，欢迎留下第一条。