在 Linux 系统下配置多张支持完整 32GB Resizable BAR(ReBAR)的显卡时,用户遭遇了内核资源分配障碍。硬件环境由 Broadcom PEX88096 PCIe 交换机连接四张 AMD Radeon AI PRO R9700 和两张 NVIDIA RTX 3090 组成,旨在通过启用完整的 32GB 内存映射来激活 AMD ROCm 平台的点对点(P2P)功能 1。然而,Linux 7.0 内核版本存在一个关于桥接窗口大小计算的回归缺陷,具体表现为 pbus_size_mem 函数未能正确计算所需空间,导致错误日志显示 pci 0000:1b:00.0: bridge window [mem size 0x800200000 64bit pref]: can't assign; no space,进而致使多显卡场景下的内存映射失败 1。
为绕过这一内核限制,作者开发了一个名为 gpu-rebar-layout 的内核模块,该模块在 initramfs 阶段运行,通过移除根端口并重新扫描的方式强制调整 BAR 对齐和扫描顺序,从而成功分配正确的内存空间 1。经过调试与优化,最终实现了 GPU 间的高效 P2P 通信,其中 R9700 之间的 P2P 带宽达到 14.0 GB/s,四卡环形复制速度达 54.7 GB/s 1。值得注意的是,尽管维护者 Ilpo Järvinen 已提交包含修复方案的 14-patch 系列补丁(v5 版本截至 11 月 6 日),但该补丁尚未合并进主线内核 1。在整个技术攻关过程中,作者利用 Claude 协助编写内核模块代码、运行测试以及起草相关技术文章 1。
A technical deep dive into configuring multiple high-end graphics cards on Linux has revealed a significant regression in kernel version 7.0 that prevents full 32 GB Resizable BAR mapping when GPUs are connected behind a PCIe switch 1. The issue specifically affects setups utilizing Broadcom PEX88096 switches to connect four AMD Radeon AI PRO R9700 and two NVIDIA RTX 3090 GPUs, where the goal is to enable peer-to-peer (P2P) communication via AMD ROCm by mapping the complete VRAM of each card 1. The failure manifests as an inability to assign memory space due to a bug in the pbus_size_mem function within the kernel's bridge window size calculation logic, generating errors such as pci 0000:1b:00.0: bridge window [mem size 0x800200000 64bit pref]: can't assign; no space 1.
To overcome this limitation, the author developed a custom kernel module named gpu-rebar-layout, which operates during the initramfs stage to manually correct BAR alignment and scanning order by removing root ports and triggering a rescan 1. This workaround successfully established P2P connectivity between the R9700 cards, achieving bandwidth speeds of 14.0 GB/s per link and a combined four-card ring replication throughput of 54.7 GB/s 1. While the fix remains a local solution, progress toward an official resolution is underway, with maintainer Ilpo Järvinen submitting a 14-patch series (version 5 as of November 6) to address the underlying kernel defect, though it has not yet been merged into the mainline 1. The entire debugging process, including coding the kernel module, running tests, and drafting the documentation, was assisted by the AI tool Claude 1.
评论
还没有评论,欢迎留下第一条。