Anthropic的Claude Code团队发布文章,阐述如何通过端到端、属性和差分测试来验证AI编程代理的工作成果 1。在90%的代码由Claude编写和审查的背景下,建立可靠的验证循环对于并行运行大量代理至关重要 1。引入独立评分器(grader)检查Claude的工作可将任务成功率提高最多10个百分点 1。
在具体工程实践上,团队推荐使用端到端测试(E2E)、属性测试(Property tests)和差分测试(Differential tests)三种类型 1。团队维护一份包含39个项目的可靠性检查清单,每个项目都有触发器、检查项和相关事件记录 1。使用verify技能进行冒烟测试时,如果检查在三轮后仍然失败,Claude会停止并引入人工介入 1。此外,实施提交门控(commit gate),有off、observe和enforce三种模式;若会话被阻止两次提交,第三次将被允许通过以避免死锁 1。规则提取方法要求问题在至少两个PR中出现且可从diff识别,并由三个代理尝试反驳后经人工批准 1。
Anthropic’s Claude Code team has published a technical article detailing how they verify the work of AI coding agents using end-to-end, property, and differential testing 1. With approximately 90% of the team’s code now written and reviewed by Claude, establishing reliable validation loops is critical to supporting the parallel operation of numerous agents 1. The approach shifts the human engineer’s role from manually reading every diff to defining clear completion standards and reviewing evidence generated through automated checks 1.
The core technical framework relies on three recommended testing types: end-to-end (E2E), property-based, and differential tests 1. To enhance performance, the team introduced independent graders to audit Claude’s work, which resulted in task success rates improving by up to 10 percentage points 1. Further quality assurance is maintained through a reliability checklist comprising 39 items, each associated with specific triggers, check procedures, and historical incident records 1. If a check fails after three consecutive attempts, the system halts and escalates the issue to human intervention 1. The team also utilizes a verify skill to conduct smoke tests as part of their engineering practices 1.
Commit governance is enforced through a commit gate mechanism that operates in "off," "observe," or "enforce" modes 1. To prevent workflow deadlocks, if a session is blocked twice during the commit process, the third attempt is automatically allowed to pass 1. The methodology also includes a systematic rule extraction process, where issues must appear in at least two pull requests and be identifiable via diffs before being proposed as new rules; these proposals are then subjected to refutation attempts by three separate agents followed by manual approval 1.
评论
还没有评论,欢迎留下第一条。