David Hall于2026年10月1日发表博客文章,指出虽然人工智能的预训练阶段得益于互联网形成的“数据公地”,但随着行业前沿转向以智能体为主的后训练阶段,目前却严重缺乏类似的公共任务数据集1。作者分析认为,人类任务具有情境化且短暂(ephemeral)的特性,难以像文档那样被自然保留和共享,尤其在软件工程领域之外,几乎不存在公开的任务记录1。因此,构建后训练数据需要人为进行大规模的数据“铭刻”工作,若这些私有资产仅由少数大公司掌握,将导致巨大的经济权力失衡,进而影响AI发展的民主性1。
为了论证高质量后训练数据的稀缺与获取难度,文中列举了多项具体项目的统计数据。在预训练阶段,GPT-3的训练数据约有60%来自Common Crawl衍生数据,体现了早期公共数据的丰富性1。然而在后训练基准测试中,原始SWE-bench从12个Python仓库的90,000多个Pull Requests中仅提取出2,294个可用任务1。OpenAI在创建SWE-bench Verified时,聘请93名开发者审查1,699个任务,因提示词不清或测试不公等原因移除了超过三分之二的内容,最终仅保留500个任务;后续审计发现这500个验证任务中仍有约六分之一存在问题1。此外,在TaskTrove项目中,5,000个Java任务转换子集在修复阶段被丢弃的比例超过90%1。Vals审计还发现,小米MiMo v2.6发布中的2,698个编码环境里,有三分之二仍包含可从Git恢复的参考修复代码,暴露了数据清理的不彻底1。文章强调,如果只有少数公司能积累训练前沿模型所需的任务和资源,他们将拥有定义AI能力及其受益者的巨大话语权1。
While pre-training models have benefited significantly from the "data commons" generated by the internet, with approximately 60% of GPT-3’s training data derived from Common Crawl 1, the current shift toward agent-centric post-training lacks a similar public infrastructure for task datasets 1. David Hall and Open Athena argue in a blog post dated October 1, 2026, that human tasks are inherently contextual and ephemeral, making them difficult to preserve and share naturally compared to documents 1. This scarcity is particularly acute outside software engineering, where almost no public records of tasks exist, necessitating large-scale artificial "inscription" efforts to build post-training data 1.
The challenges of curating these datasets are illustrated by extensive filtering processes in existing benchmarks. The original SWE-bench extracted only 2,294 usable tasks from over 90,000 pull requests across 12 Python repositories 1. When OpenAI created SWE-bench Verified, they hired 93 developers to review 1,699 tasks, ultimately retaining just 500 after removing more than two-thirds due to issues like unclear prompts or unfair tests 1. Subsequent audits revealed that even among these 500 verified tasks, roughly one-sixth still contained problems 1. Similarly, in the TaskTrove project, over 90% of a subset of 5,000 Java tasks were discarded during the fixing phase 1. Further highlighting quality control gaps, a Vals audit found that two-thirds of the 2,698 coding environments released in Xiaomi MiMo v2.6 still included reference fix codes recoverable from Git 1.
Without open public task infrastructure, the accumulation of necessary resources risks being concentrated in the hands of a few major corporations, potentially creating significant economic power imbalances 1. The authors warn that if only a small number of companies can amass the tasks required to train frontier models, they will hold immense leverage in defining AI capabilities and determining who benefits from them 1.
评论
还没有评论,欢迎留下第一条。