Wincon研究团队推出了名为TIS-100 Bench的新基准测试,旨在评估16个大语言模型(LLMs)在Zachtronics游戏《TIS-100》中的编程能力 1。该测试涵盖47个游戏内置谜题和8个社区锦标赛谜题,共计55题,并采用pass@1单次尝试机制进行考核 1。结果显示,Opus 5.5和GPT-6 Astra等顶级模型不仅解决了全部55个谜题,还各自找到了11个新的前沿点 1。此外,测试共下载了3,714个模型帕累托点,其中186行被标记为新前沿点,经去重处理后确认为133个新发现 1。
在具体性能指标方面,GPT-6 Astra的记录接近度(Record Proximity)为78.0,略高于Opus 5.5的77.2;然而,Opus 5.5找到了更多的独立前沿点(72个),而GPT-6 Astra为32个 1。成本效益数据显示,GPT-6.1 Sol每谜题成本约0.75美元,接近度为75.6;Opus 5.5成本约5.08美元,接近度为77.2;Fable 5.1则是成本最高的模型,每谜题花费7.19美元,接近度为70.2 1。
这一测试揭示了“AI发现新权衡、人类实现极致优化”的协作模式 1。例如在Undecimal Twin Rounder谜题中,Opus 5.5生成的程序运行周期从社区记录的2,694降至1,918,速度提升了29%,随后玩家 @LongingForRest 利用该AI生成代码进一步优化至1,744周期 1。相关基准快照版本为2026-11-03-v5,而社区排行榜的冻结日期则设定在2026年10月15日 1。
Wincon Research has released a new benchmark, TIS-100 Bench, to evaluate the programming capabilities of large language models (LLMs) within the context of Zachtronics’ puzzle game "TIS-100" 1. The study tested 16 distinct models, including Opus 5.5, GPT-6 Astra, Gemini 3.8 Flash, and Fable 5.1, using a single-attempt mechanism known as pass@1 across a total of 55 puzzles 1. These puzzles comprised 47 built-in challenges from the game itself and eight additional community tournament questions 1.
The top-performing models demonstrated significant coding proficiency, with both Opus 5.5 and GPT-6 Astra successfully solving all 55 puzzles 1. Each of these leading models identified 11 new Pareto frontier points, contributing to a total of 133 unique new frontier points discovered after deduplicating 186 raw data lines from the downloaded model results 1. In terms of overall efficiency, GPT-6 Astra achieved a Record Proximity score of 78.0, slightly edging out Opus 5.5’s score of 77.2, although Opus 5.5 located more independent frontier points (72 compared to 32 for GPT-6 Astra) 1.
Beyond aggregate scores, the models broke specific community records through algorithmic optimization. For instance, in the "Undecimal Twin Rounder" puzzle, a program generated by Opus 5.5 reduced execution cycles from the previous community record of 2,694 down to 1,918, representing a 29% speed improvement 1. Human player @LongingForRest subsequently refined this AI-generated code further, optimizing it to run in just 1,744 cycles, illustrating a collaborative pattern where AI discovers novel trade-offs and humans achieve extreme optimization 1.
The analysis also highlighted the cost-effectiveness disparities among the models. GPT-6.1 Sol offered a relatively low cost of approximately $0.75 per puzzle with a Record Proximity of 75.6, while Opus 5.5 incurred a higher cost of about $5.08 per puzzle but delivered a superior proximity score of 77.2 1. Fable 5.1 proved to be the most expensive option at $7.19 per puzzle, achieving a Record Proximity of 70.2 1. The benchmark snapshot is versioned as 2026-11-03-v5, referencing a community leaderboard frozen date of October 15, 2026 1.
评论
还没有评论,欢迎留下第一条。