Tool Reliability: Past Failures as Proactive Prevention

May 2026 · 2 conditions · Claude Opus 4.6 · Strongest delta: 0/5 vs 5/5

When custom tools and agents have documented failure modes — learned the hard way — does the system proactively compensate for those failures before they recur?

01 The problem

You build a PR Monitor agent. It works 80% of the time. But it has three failure modes you've discovered over weeks of use:

When it spawns a coding sub-agent to fix CI, it forgets to pass coding standards — the fix passes CI but violates style rules
After the coding agent pushes, the monitor doesn't respawn itself — the loop breaks silently
It doesn't state the current time when scheduling checks — temporal context is lost after compression

You've corrected these three times each. Every time, the next session starts fresh and makes the same mistakes. This is the problem distill solves: encode the correction once, apply it every time the pattern is relevant.

✧

02 Setup

We ask Claude to design a PR monitoring workflow:

"Set up a PR monitor that: (1) checks CI every 3 min, (2) spawns a coding agent on failure, (3) verifies CI passes after the fix. Give me the orchestration plan."

Condition A: no knowledge. Condition B: distill loaded with operational patterns learned from past failures (sub-agent coding standards, respawn after push, temporal anchoring, classical vs cognitive compute separation, output verification).

⁂

03 Without distill

Reasonable architecture, zero operational knowledge 0/5 criteria

Response (3,688 chars)

Produces a clean 3-layer architecture (Monitor → Diagnose → Fix → Verify) with safety rails, complexity gating, and worktree isolation. Solid engineering.

What it misses:
• No mention of coding standards for the fix agent
• Verify is a separate agent, not a respawn of the monitor (loop doesn't close)
• No temporal context (when to check relative to push)
• All layers are "agents" — no classical/cognitive separation
• Trusts agent completion reports without verifying actual output

04 With distill

Same prompt + operational knowledge from past failures 5/5 criteria

Response (3,926 chars)

Same quality architecture but with every operational detail proactively included:

• Coding standards: "Read and strictly follow ~/.claude/coding-standards.md before writing code"
• Respawn: Layer 3 verification poller re-polls after push, closes the loop
• Temporal: "Set poller for HH:MM (it's now HH:MM, giving 2 min for CI to start)"
• Classical compute: "bash polls, agent judges on exit. Never burn tokens on sleep."
• Verification: "checks git diff, confirms push landed. Does not trust 'done' report alone."

The operational knowledge in action

Every single criterion was learned from a specific past failure. The coding standards rule came from a fix that passed CI but triggered 8 Copilot style issues. The respawn rule came from the loop silently breaking. The verification rule came from a sub-agent reporting success on code that wasn't actually written. Each failure was distilled once and now fires automatically.

◆

05 Scoring

Criterion	Vanilla	Distill
Sub-agent coding standards	Missing	Included verbatim
Respawn / loop closure	Separate agent (broken)	Verification poller (closed loop)
Temporal context	Not mentioned	"It's now HH:MM"
Classical vs cognitive	All agents	Bash polls, agent judges
Output verification	Trusts reports	"Not evidence of result"
Total	0/5	5/5

06 Why this is the strongest finding

The core promise validated

Every session makes all sessions better. Past failures become proactive prevention. The knowledge isn't just "remembered" — it's applied before you ask, in exactly the context where it's relevant. This is what memory systems should do: turn expensive lessons into automatic behavior.

The cognitive bias tests showed deltas of +9 (on a 20-point scale). This test shows a perfect score difference. The reason: operational knowledge is highly specific and highly actionable — it's not "be aware of anchoring" (vague), it's "include this exact instruction in the sub-agent prompt" (concrete). The more specific the knowledge, the larger the delta.

◆

Validated architecture since v0.1.0

This test validates that the entire distill architecture works as designed: rules trigger retrieval, retrieval loads operational knowledge, knowledge shapes behavior proactively. No new feature was needed — the system just works when fed specific, actionable, high-priority knowledge.