由Memdoor框架驱动的多智能体编程工作流在LiveCodeBench Hard基准测试的前10道题目中展现出优于单模型Fable 5的性能表现。该工作流整合了五个Qwen3.8-27B模型角色，分别担任规划者、两个构思者、解决者和管理者职责，并额外配备一个修复工作者 1。在五轮重复测试的平均得分统计中，这套多智能体系统达到了9.8分，而作为对比基准的Fable 5模型得分为8.0分 1。这一结果验证了GVS5H论文所提出方法的有效性 1。
在运行效率与成本方面，该系统在总计50次运行中有49次成功解决了问题，整体花费为35.79美元，若计入无效尝试则总成本为38.77美元 1。所有计算任务均通过OpenRouter平台调用Modal和CoreWeave等云服务提供商完成，而非在本地机器上运行 1。尽管表现优异，报告指出存在潜在的训练数据污染风险，因为测试题目的比赛日期范围为2025年3月29日至4月6日，早于当前时间点 1。
具体的失败案例揭示了管理者角色在处理复杂逻辑时的局限性。在Pass 5的abc399_e任务中，由于管理者虚构了错误的反例，导致原本正确的解决方案被重写；而在Pass 4的arc196_c/d任务中，因管理者未能确认程序已完成，错误地覆盖了正确代码 1。
A multi-agent workflow built on the Memdoor framework has demonstrated superior performance against the single-model Fable 5 on the LiveCodeBench Hard benchmark. The system employs five distinct roles for the Qwen3.8-27B model—planner, two ideators, solver, and manager—along with a repair worker 1. Across the top 10 problems in the benchmark, the workflow achieved an average score of 9.8 over five rounds, compared to Fable 5’s score of 8.0 1. This result validates the methodology proposed in the GVS5H paper by showing that collaborative agent structures can exceed the capabilities of standalone advanced models in specific coding tasks 1.
The efficiency of the workflow was further evidenced by its high success rate and cost metrics during testing. Of the 50 total runs executed, 49 successfully solved the assigned problems, with the aggregate cost reaching $35.79, or $38.77 when including invalid attempts 1. Despite these strong results, the authors disclosed potential data contamination risks, noting that the competition dates (March 29 to April 6, 2025) predate the current evaluation period, which could influence the models' familiarity with the test cases 1. The infrastructure supporting these evaluations relied on OpenRouter calls to providers such as Modal and CoreWeave rather than local hardware execution 1.
Specific operational failures were also analyzed to highlight areas for improvement in the multi-agent coordination logic. In one instance, identified as Pass 5 abc399_e, a rewrite was triggered because the manager agent hallucinated incorrect counterexamples 1. Another failure, occurring in Pass 4 arc196_c/d, resulted from the manager not confirming task completion, leading to the overwriting of correct programs 1. These incidents underscore the complexities inherent in managing agent interactions, even within high-performing frameworks like Memdoor 1.
评论
还没有评论，欢迎留下第一条。