研究论文提出"永不放弃"方法改善强化学习中的难题求解

Michael Noukhovitch、Hamish Ivison、Nathan Lambert 和 Aaron Courville 的研究论文指出，强化学习在大语言模型训练中存在"Matthew Effect"现象 1。这一现象表现为强化学习的改进程度与模型初始能力成正比，导致简单任务容易被改进而难题仍然难以解决 1。

为了解决这一问题，研究团队提出了"Never Give Up"（永不放弃）方法 1。该方法采用自适应采样策略，初始对问题采样 k 次，若全部失败则以概率 p 重新采样，期望总采样数为 k/(1-p)，从而为难题增加采样轮数、为简单问题分配较小的 k 值，实现计算资源的重新分配 1。

在多项测试中，该方法取得显著成效。在 GSM8k 数据集上，k=4 配合 NGU p=0.9 的配置优于所有标准 GRPO 配置，特别是在最难子集上表现明显 1。在 Manufactoria 基准测试中，标准 GRPO 在通过 7/12 测试后陷入停滞，而 NGU+GRPO 仍继续改进 1。该方法在数学推理、代码生成等多个任务上都显示出对难题的显著改进 1。论文的 arXiv 编号为 2609.13443 1。

A research paper by Michael Noukhovitch, Hamish Ivison, Nathan Lambert, and Aaron Courville identifies a "Matthew Effect" in reinforcement learning applications for large language models, where RL improvements tend to disproportionately benefit easier tasks while offering limited gains on harder problems 1. The phenomenon describes how the degree of RL improvement correlates with a model's initial capability, causing already-simple tasks to become easier while difficult ones remain challenging 1.

To counteract this bias, the team introduces the "Never Give Up" method, which employs adaptive sampling strategies to reallocate computational resources by using smaller sampling values for simple problems while increasing sampling rounds for harder tasks 1. The approach works by initially sampling k times, and if all attempts fail, resampling with probability p, resulting in an expected total sampling count of k/(1-p) 1.

Testing on the GSM8k benchmark demonstrates that k=4 combined with Never Give Up at p=0.9 outperforms all standard GRPO configurations, with particularly notable improvements on the hardest problem subset 1. In the Manufactoria benchmark, while standard GRPO stalls after passing 7 out of 12 tests, the Never Give Up variant continues to show improvement 1.

The research, posted as arXiv preprint 2609.13443, also addresses the stale completion problem in asynchronous RL settings by implementing a T=4 day threshold filter, while the GRPO baseline retains all samples but rescales negative ones 1. The method shows significant gains across multiple domains including mathematical reasoning and code generation 1.