人工智能评估非营利组织METR发布了一份关于AI时间跨度图的更新报告 1。报告指出,虽然部分AI能力呈指数级增长,前沿模型每四个月能成功完成的任务长度翻倍,但在需要判断力的开放式任务上,其表现不如编码等可验证任务 1。新报告纳入了设计AI评估和哲学论证评分等难以量化的任务,结果显示当前最先进模型在这些领域仍未达到人类水平,但已显示出“勉强及格”的进步迹象 1。
具体而言,包括Opus 5.5在内的最新模型在评估设计任务上达到了“勉强及格”的水平,而一年前所有模型均为“明显不及格” 1。OpenAI周三发布的博客称,其模型在员工要求的现实世界任务中表现不如在METR原始套件中的可验证任务好 1。
专家肯定METR扩展评估维度的尝试,但也指出此类任务的定义和评分极其困难 1。研究员Sayash Kapoor认为研究品味是AI研发自动化的主要瓶颈 1。Matt Putz表示由于性质使然,难以像易定义任务那样建立清晰稳健的趋势 1。目前尚难准确预测AI在研发自动化方面的进展速度 1。
METR has released an updated report on the "time horizon" graph, a widely cited but often misunderstood metric in artificial intelligence 1. The new findings confirm that while frontier models continue to double the length of tasks they can successfully complete every four months, their performance remains significantly weaker on open-ended assignments requiring judgment compared to verifiable tasks like coding 1. This distinction underscores a critical gap in current AI capabilities as developers push toward automating complex research and development workflows.
The latest evaluation suite expands beyond traditional benchmarks to include difficult-to-quantify tasks such as designing AI assessments and scoring philosophical arguments 1. Results indicate that state-of-the-art models, including Opus 5.5, have progressed from clearly failing these nuanced evaluations a year ago to achieving merely "passable" performance today 1. Despite this improvement, experts note that top-tier systems still do not reach human-level competence in areas demanding deep reasoning and taste 1.
Independent observations align with METR’s conclusions. OpenAI stated in a Wednesday blog post that its models perform worse on realistic tasks requested by employees than on the verifiable tasks within METR’s original framework 1. Sayash Kapoor identifies "research taste" as a primary bottleneck for AI-driven automation in R&D, suggesting that subjective quality control remains a significant hurdle 1. Matt Putz adds that the inherent nature of these open-ended tasks makes it difficult to establish clear, robust trends comparable to those seen in well-defined domains 1. Consequently, while the expansion of evaluation dimensions is praised, accurately predicting the pace of AI progress in these areas remains challenging 1.
评论
还没有评论,欢迎留下第一条。