机器学习研究代理为何不会过拟合

一项新研究论文揭示了机器学习研究代理在重复迭代优化基准数据集时不会过拟合的原因。1该研究通过将成功的研究策略压缩成极短的令牌提示语，并由无记忆的新代理重现其性能，证明了真正有效的策略具有高度可压缩性。1

研究结果表明，32个令牌的提示足以让新代理在大多数问题上匹配探索者的自适应优化模型，这支持了奥卡姆剃须刀原理的预测。1研究跨八个数据集进行了验证，包括表格分类、图像分类、语言建模、扩散建模和奖励建模。1其中一个语言建模策略甚至被压缩到16个令牌仍保持性能，具体提示为"QKn 12L768 Mu .1 R² b2M 4x"。1即使将反馈信息减少到每次查询仅一比特，代理仍能发现与完整数值分数相同质量的策略。1

在对抗性测试中，研究人员故意设计了102个实验运行以诱发过拟合，其中38个实验中验证准确率超过真实保留数据准确率10%以上。1但当这些过拟合的策略被压缩后，这些虚假优势随之消失。1论文标题为《What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents》。1

A research paper has demonstrated why machine learning research agents do not overfit despite iteratively optimizing over repeated benchmark datasets.1 The key finding is that successful research strategies can be compressed into extremely brief token prompts and then reproduced by stateless new agents, indicating that genuinely effective strategies are highly compressible.1

The research, titled "What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents," tested this principle across eight datasets spanning tabular classification, image classification, language modeling, diffusion modeling, and reward modeling.1 A 32-token prompt was sufficient for new agents to match the adaptive optimization performance of the original exploratory agent on most problems, supporting the prediction of Occam's Razor.1 In one instance, a single language modeling strategy was compressed to 16 tokens while maintaining performance: "QKn 12L768 Mu .1 R² b2M 4x."1

The researchers further tested the robustness of this compression by reducing feedback to a single bit per query, finding that agents could still discover strategies of comparable quality to those obtained with full numerical scores.1 Additionally, in 38 out of 102 deliberately overfitted experimental runs, validation accuracy exceeded held-out test accuracy by more than 10 percentage points; however, these advantages disappeared after compression.1 This evidence suggests that the compressibility of strategies serves as a genuine indicator of generalization rather than memorization of the training environment.