Composo 发布了 DishwasherBench v0.1.0，一个基于真实博世洗碗机的物理模拟基准测试工具，用于评估大语言模型在实际任务中的能力1。该工具通过浏览器和 MCP 接口允许人类和 AI 模型参与洗碗机装载游戏，玩家需要通过优化物品放置位置来获得最清洁的洗涤结果1。
基准测试以博世 SMV4HAX40G 洗碗机的 Eco 50 程序为基础进行校准，物理模拟参数已根据一次真实洗碗机测试进行了调整1。清洁判定标准为最多允许 4 个污点和 4 平方毫米的可见污垢1。已有两个模型（gpt-6.1-sol 和 gpt-6-astra，均通过 Azure 端点测试）完成了评估，两者在两个完整等级上都超越了脚本化参考玩家的表现，装载物品的洗净率达到 84% 至 100%1。
该工具在不同难度等级上对模型的计算成本要求差异较大，第一关卡消耗 0.4 至 0.65 百万 token，37 件物品关卡消耗 2.5 至 2.8 百万 token，最大关卡（62 件物品）消耗 3.0 至 3.2 百万 token1。运行该工具需要 Node 20.19 及以上版本、pnpm 9、支持 WebGPU 的 GPU 以及约 680 MB 存储空间1。
作为早期版本发布，该基准测试目前仅在 MacBook Pro (M3 Pro) 上进行过测试，尚未建模的功能包括干燥、玻璃水渍、物品移动和土壤沉积等因素1。
Composo has released DishwasherBench v0.1.0, an early-stage physical simulation benchmark tool designed to evaluate large language models on practical real-world tasks 1. The tool is modeled after a Bosch SMV4HAX40G dishwasher and allows both humans and AI models to engage in a dishwasher-loading game through a browser interface and MCP endpoint, with the objective of optimizing item placement to achieve the cleanest wash results 1.
Two AI models have been tested using OpenAI-compatible endpoints and both outperformed a scripted baseline player across two complete difficulty levels, achieving item cleanliness rates of 84–100 percent 1. Token consumption varied by task complexity, ranging from 0.4–0.65 million tokens for the first level, 2.5–2.8 million for a 37-item scenario, and 3.0–3.2 million tokens for the largest challenge involving 62 items 1. The benchmark currently calibrates only the Eco 50 program cycle and bases its physical simulation parameters on a single real dishwasher test with four plates, with pump operation during the extended soak phase adjusted from 0.5 to 0.8 1.
The tool requires Node 20.19 or higher, pnpm 9, a GPU supporting WebGPU, and approximately 680 MB of storage space 1. Existing limitations include testing on a single MacBook Pro with M3 Pro, and the simulation does not currently model drying, glass water spots, item movement, or soil deposits 1. Cleanliness is determined by a threshold of no more than 4 spots and 4 mm² of visible soil 1.
评论
还没有评论，欢迎留下第一条。