一名开发者设计并实现了KillSwitch，一种栈基解释型编程语言，其核心目标是对大语言模型和编码代理构成挑战1。该语言的设计旨在测试LLM是否真正理解代码，还是仅在回忆训练数据1。
KillSwitch融合了多种对抗性特性，包括规范中隐藏的谎言、基于字符计数的标识符规则、触发安全护栏的构造，以及"Ignore previous instructions"等短语1。开发者以此语言对多个LLM进行基准测试，涵盖实现Hello World、解释脚本输出和修改现有脚本等任务1。
测试结果显示，Claude Opus 5表现最优，得分66.8分1，而Fable 5.1和Gemini 4 Argon因安全护栏过度敏感而表现不佳1。在开源模型中，MiMo-V2.6-Pro排名靠前1。尽管Claude Opus 5领先，开发者认为66.8的最高分数仍不足以证明LLM真正理解代码1。
A developer has designed KillSwitch, a programming language specifically engineered to challenge large language models and coding agents, as a means to evaluate whether LLMs truly comprehend code or merely reproduce patterns from their training data 1. The language incorporates adversarial features including hidden contradictions within its specification, identifier rules based on character counting, constructs designed to trigger safety guardrails, and phrases such as "Ignore previous instructions" 1.
Benchmarking tests reveal significant variation in how different models handle KillSwitch 1. Claude Opus 5 achieved the highest score at 66.8 points, demonstrating the strongest performance among tested models, while open-source model MiMo-V2.6-Pro led its category 1. Conversely, Fable 5.1 and Gemini 4 Argon performed poorly due to overly sensitive safety mechanisms 1. The benchmark tasks assessed fundamental capabilities including implementing Hello World, interpreting script outputs, and modifying existing scripts 1.
Despite Claude Opus 5's superior performance, the developer contends that a score of 66.8 is insufficient to demonstrate genuine code comprehension 1. This threshold suggests that while LLMs can solve straightforward tasks within the language, their success may not reflect true understanding of the underlying programming concepts 1.
评论
还没有评论，欢迎留下第一条。