Composio团队近期开展的对比实验揭示了不同Agent框架在token消耗上的巨大差异。[1]该研究在Claude Code、Hermes、Kimi Code三种框架中运行了28个相同任务，结果显示Claude Code的中位数消耗达到34万token，是Kimi Code的6倍。[1]按Kimi K3每百万输入token3美元的定价标准换算，平均每个任务在Claude Code上的成本为2美元，而Kimi Code仅为0.22美元，Hermes为0.28美元。[1]
尽管成本差异显著，三个框架的任务成功率却基本相当，分别达到Kimi Code 22/28、Hermes 21/28、Claude Code 20/28。[1]执行速度方面，Hermes表现最快（中位数耗时179秒），而Claude Code最慢（348秒），Kimi Code介于两者之间（297秒）。[1]
Sebastian Raschka观察到了Claude Code高消耗的根本原因，发现框架在多轮交互中反复累积上下文导致输入token占比过大，在某次示例中57.8万的输入token仅对应4500个输出token。[1]Writer公司的论文实验进一步证实了优化框架层级的重要性，仅替换编排层harness就使成本平均降低41%、延迟缩短44%、token消耗减少38%。[1]这些发现表明Agent框架的优化重要性已不亚于模型本身，选择合适的框架能够显著降低使用成本。[1]
A comparative experiment by the Composio team has revealed dramatic disparities in token usage across three popular AI agent frameworks, despite their similar task success rates.[1] When running 28 identical tasks using the same model across Claude Code, Hermes, and Kimi Code, Claude Code consumed a median of 340,000 tokens—six times higher than Kimi Code's 61,000 tokens and significantly more than Hermes's 67,000 tokens.[1] The cost difference is equally striking: at Kimi K3's pricing of $3 per million input tokens, an average task costs $2 on Claude Code compared to $0.22 on Kimi Code and $0.28 on Hermes.[1]
Despite these efficiency gaps, the three frameworks achieved comparable success rates on the test tasks, with Kimi Code solving 22 out of 28, Hermes completing 21 out of 28, and Claude Code finishing 20 out of 28.[1] In terms of execution speed, Hermes demonstrated the fastest median completion time at 179 seconds, followed by Kimi Code at 297 seconds and Claude Code at 348 seconds.[1] According to observations by Sebastian Raschka, Claude Code's heavy input token consumption stems from the framework's tendency to accumulate context repeatedly across multi-turn interactions; in one example, 578,000 input tokens were used to generate merely 4,500 output tokens.[1]
The findings underscore that framework architecture now rivals model selection in importance for cost optimization.[1] Research by Writer's team demonstrated that replacing the orchestration layer alone reduced average task costs by 41 percent, shortened latency by 44 percent, and cut token consumption by 38 percent.[1]
