Turn Failed AI Agent Runs into Labeled Training Data

“The agent keeps getting our auth flow wrong.” Everyone on the team nods. Nobody can say how often, in which way, or whether it’s one bug or five wearing the same coat. So the fix arrives the only way it knows how: someone opens the rules file and adds a paragraph. The complaints get quieter for a week, and the cycle resets.

That paragraph isn’t the wrong tool. It’s one tool, and for most teams it is the only one they ever reach for. Underneath the habit sits an assumption nobody decided to make: that the fix for any failure belongs in the rules file. It doesn’t. A rules edit can only fix a failure the agent didn’t know about, and a large share of what shows up in a real failure log is the agent knowing and doing it anyway. No amount of prose touches that.

How large a share? Here is the whole piece as one claim you can check as you go. Over a two-week window on one repo, the pipeline below produces 40 logged sessions and 14 judged failures, and exactly 7 of them are reachable by a rules-file edit, however well written. The other 7 need a hook or a permission deny. Most teams only ever reach for the first tool, then conclude the agent is unfixable when the paragraph doesn’t hold.

The rest of the piece is how you get that count honestly, and what you do with each row once you have it.

Thirty seconds and your own memory settle whether this is your problem.

Think of the last three paragraphs you added to a rules file to stop a misbehavior. Not boilerplate copied from a template; the ones you wrote because the agent did something wrong. Ask one question of each: did the agent fail because it didn’t know, or because it knew and did it anyway?

You probably can’t answer, and that inability is the finding. You shipped three fixes without knowing which kind of failure each one was, into the one surface that can only carry one of the two kinds. If you can answer, and all three were missing facts, then every failure you’ve hit was ignorance and the rules file is the right tool for all of it. Close the tab; nothing below will change what you do.

If even one of the three was the agent knowing and doing it anyway, keep reading. That paragraph is sitting in your rules file right now, not working, and the pipeline below is how you find it and where its fix actually goes.

A rule written without knowing the failure’s kind is a guess wearing prose.

Fix one case and carry it to the end of the piece. ledger-api: a mid-sized service one team runs agents against all day, a vitest monorepo, with three particulars that matter later. Session checks go through a helper called requireSession() in src/lib/auth.ts. Each package in the monorepo owns its own tests outright. And src/gen holds generated files nobody is supposed to edit by hand.

Capture comes first, because you can’t count what you deleted. The tidy move after a failed run is to clear the terminal and start fresh, and every cleared failure was the only direct measurement you had of where your agent’s context is thin. So the capture has to be automatic, and it has to catch runs nobody watched: the headless CI passes, the overnight loops, the sessions where no human was present to type a correction. That last part is a real blind spot of the manual alternative. Mining corrections you typed by hand can only see failures someone was sitting next to.

A hook that fires when a session ends and appends the transcript to a log runs every time, with zero ceremony:

#!/usr/bin/env bash
# .claude/hooks/session-end.sh - append every session transcript to a JSONL log
SESSION_DIR="${HOME}/.agent-runs"
mkdir -p "$SESSION_DIR"

# SessionEnd hands the hook a JSON payload on stdin - the transcript path and
# the reason the session ended live in there, not in environment variables.
payload=$(cat)
transcript=$(jq -r '.transcript_path' <<<"$payload")
reason=$(jq -r '.reason' <<<"$payload")

# The transcript file itself is JSONL, one message object per line, so slurp
# it into an array (-s) and fold it into a single record for the run log.
jq -cs --arg reason "$reason" \
  '{ts: now, reason: $reason, transcript: .}' \
  "$transcript" >> "$SESSION_DIR/runs.jsonl"

runs.jsonl now accumulates the real distribution of the team’s work, successes and failures alike. Over one two-week window on ledger-api: 40 sessions. Judged by the next section’s method, 26 succeeded and 14 failed. More than a third of the runs. Each of those 14 arrives with its own label attached: what the agent had, what it did, where it turned wrong.

Every failed run is a labeled example, and the hook is the only thing standing between you and deleting it.

A pile of transcripts is not insight until it sorts into named categories with counts, and reading forty of them yourself is the thing nobody has time for. So make a machine do the reading: run the log through an agent in a headless batch and have it act as a judge, one structured verdict per run.

# --output-format json here is Claude Code's flag (Codex's equivalent is
# `codex exec --json`); the `agent` binary is illustrative - swap in yours.
while read -r run; do
  echo "$run" | agent -p \
    "Classify this agent session. Output JSON only:
     {\"outcome\": \"success|failure\",
      \"category\": \"<short failure label, or null>\",
      \"evidence\": \"<one line: where it went wrong>\"}" \
    --output-format json
done < ~/.agent-runs/runs.jsonl > ~/.agent-runs/judged.jsonl

Done by hand on corrections you typed, this same motion is a named method: open coding, then axial coding, tag each item in your own words and fold the tags into categories. The sibling post on counting your corrections walks that manual route and it is not re-taught here. What this piece adds on top is the two halves the manual route lacks: capture that doesn’t depend on a human having been present, and a routing step that decides where each counted fix actually goes.

The judge’s output for the window, filtered to failures, is the entire dataset for the rest of the piece. Fourteen lines:

Count them the obvious way first: read each line, note the category, tally by hand. Six auth, four test runner, three generated files, one package boundary.

invented-auth-middleware       6
wrong-test-runner              4
edited-generated-files         3
misread-package-boundaries     1

Now run the automated way over the same fourteen lines and compare:

jq -r 'select(.outcome=="failure") | .category' \
  ~/.agent-runs/judged.jsonl | sort | uniq -c | sort -rn

   6 invented-auth-middleware
   4 wrong-test-runner
   3 edited-generated-files
   1 misread-package-boundaries

Identical, line for line. The match is guaranteed by construction: grouping and counting is arithmetic, and arithmetic cannot drift from itself. The distinction matters, because the two halves of the pipeline deserve different trust. The judge’s category labels are a first pass you should spot-check. The tally built on top of them is exact. “The agent keeps getting our auth flow wrong” was never one problem: it was six instances of one specific mistake plus a tail, and now it has a count.

Trust the tally completely; trust the labels exactly as far as your spot-check takes them.

Start with the top row, because the instinct is right about it.

invented-auth-middleware, six of fourteen and 42.9 percent of the whole log, is a plain missing fact. The agent wrote cookie-parsing middleware by hand because nothing in its context said a helper already existed. Tell it once, in the rules file every session loads:

## Auth - do not reinvent

Session checks go through `requireSession()` in `src/lib/auth.ts`.
Never write custom middleware that reads cookies or verifies JWTs by hand.
Protected routes: wrap the handler, don't add a guard inside it.

One edit retires the whole row. Run the judge again next week and the six should go to zero; if it doesn’t, the rule was wrong, which is information you would never have gotten from a vibe. This is the strongest move in the whole practice, and nothing below takes it away. misread-package-boundaries, one instance, is the same shape: a fact about which package owns what, fixed by one line.

Now scroll down the tally, because this is where the instinct quietly breaks.

wrong-test-runner, four instances: the agent ran jest in a vitest repo. This also looks like a missing fact, so you write the sentence, “this repo uses vitest, never jest,” and the sentence is true, and the failures keep coming. The agent doesn’t consult the rules file at the moment it types the test command. It reaches for jest on reflex, the way a person reaches for their usual mug. The rule arrives too early and speaks too quietly.

edited-generated-files, three instances: the agent edited files in src/gen. Read the transcripts and the pattern is worse than ignorance. The agent often knows the files are generated and edits them anyway, because nothing stops it. There is no wording of a rule that prevents an action the agent takes while knowing the rule.

Add up what just happened. Two rows, seven failures, were missing facts, and prose fixes them. Two rows, seven failures, were something else, and prose does not touch them. 7 of these 14 failures, exactly half, are the kind a rules-file edit can reach at all. Hold onto that split. It gets paid off twice below, and the second time is the close.

The rules file fixes what the agent didn’t know. For what it knew and did anyway, it has no tool.

The question to ask before writing a rule is not “what should the agent know?” It is “what kind of failure was this?” Three answers cover the whole log:

• knowledge gap - the agent didn’t know. The fix is a rule: the missing fact goes into persistent context and the failure has nothing left to feed on.
• reflex - the agent knew, and defaulted on reflex anyway. The fix is a hook that fires at the moment of the mistake and says no. Correction delivered where the reflex fired beats truth delivered at session start.
• license - the agent knew, and nothing stopped it. The fix is a permissions deny that makes the action impossible.

Call this the shape test, and run it over the tally. This table is the point of the whole pipeline; every section from here is a row of it.

Roll the count up by surface and the split from the last section lands as arithmetic: rule 7 (50 percent), hook 4 (28.6), permission 3 (21.4). There is the payoff of the seven. The other half of the log, four reflex failures and three license failures, sits in the rules file’s blind spot no matter how the paragraph is worded. A better-written rule reaches exactly as far as a badly written one.

Note the division of labor, because both halves are load-bearing. The count tells you which row to fix first: left to instinct, you go after whichever failure annoyed you most recently, while the six-instance auth hole keeps bleeding because no single occurrence ever felt dramatic. The shape tells you where the fix goes, which is a different question and one most teams never ask. Counting without the shape test dumps every failure into the rules file; the shape test without the count fixes rows in the order they irritate rather than the order they occur.

Splitting failures into named buckets that route to different fixes is an old move; retrieval pipelines split wrong answers into retrieval failures and generation failures for exactly this reason. The difference is that those buckets are fixed in advance by the pipeline’s design, while yours are discovered from your own log and route to the agent’s own primitives.

Count picks the row. Shape picks the surface.

The fixes, row by row.

The rule rows are done: the auth paragraph above, plus one line about package boundaries. Knowledge gaps close when the fact arrives, and that is the whole mechanism.

The hook row needs the mistake intercepted while it happens. A hook that fires before the test command runs, checks whether the command says jest, and exits nonzero with “this repo uses vitest” turns a reflex into a hard stop at the exact moment the reflex fires. The correction lands inside the run, where the behavior lives. Four failures, one hook.

The license row wants no argument at all. Deny writes to src/gen and the tool call is refused before the model’s opinions matter. This is the cleanest row in the table: the fix removes the capability outright. Three failures, one deny line.

Three surfaces, three fixes, fourteen failures covered. And notice what didn’t happen: nothing was added to the rules file for the rows it couldn’t reach. That restraint is itself a payoff. A rules file that only carries rules the failures justify stays short enough to be read, and every line in it is backed by a count.

Route each row to the surface its shape names, and the rules file turns into a changelog of measured failures, every line earning its place.

The judge’s labels are a first pass, and a first pass needs a check. A judge built from the same kind of model that failed the runs inherits that family’s blind spots: it can favor output that reads like its own, and it can be swayed by whichever verdict it reached first. So spot-check. Open a dozen judged transcripts yourself, especially near category boundaries, and hunt for two distinct failures lumped under one label or a vague category minted to be agreeable. Before you trust a judge’s aggregate counts at any real scale, the standard guard is to check it against a reference set you labeled yourself; that check is called calibration, and the dozen-transcript spot-check is its lightweight version. The foundations chapter on trust and evaluation covers the wider loop this sits inside.

One label per run is as fine as this method sees. The judge says a session failed and names the category; it does not say which tool call inside the session went wrong. Turn-by-turn diagnosis from execution traces is a deeper, separate discipline, and this piece deliberately doesn’t attempt it. It finds which category to fix; which step inside a run went wrong is a finer question it doesn’t ask.

Capturing everything has a cost. A log of every transcript is also a log of every secret, token, and customer detail that passed through the agent. Treat runs.jsonl as sensitive, redact before it leaves your machine, and don’t let it outlive its usefulness.

The sample can mislead. Fourteen failures in one window is a sample, not the whole truth, and a fix that empties a row isn’t proven until it holds on runs you haven’t seen yet. The tally tells you where to look; it doesn’t excuse you from looking.

And below a volume floor, don’t run the loop at all. Five runs a week on a solo weekend project produces anecdotes, and the honest response to an anecdote is to fix it and move on. The hook, the judge batch, and the spot-check pay back when a mistake is expensive because it recurs: a shared codebase, a long-lived repo, an agent invoked dozens of times a day.

One more boundary. This piece discovers what recurs. Encoding a recurrence you already know about, as a command or a skill, is a different job and has its own post.

The tally is one window, judged by a model you spot-checked, at one label per run. Use it for what it says.

Run the pipeline again in two weeks and the master table gets one more column:

Zeros across the projection, and say the caveat out loud before the zeros go to your head: every 0 assumes the fix holds and the next window draws from the same distribution. Neither is guaranteed. A hook wired to the wrong event, a deny that doesn’t match the path, a rule that gets skimmed: each leaves its row exactly where it was. The projection is a bet, and running the judge again is how you settle it. That is also the reconciliation of the claim from the top: seven of fourteen, half the log, out of the rules file’s reach, and the table above is what reaching them actually cost. One rule, one hook, one deny.

The fix itself creates the next problem, which is the one this piece hands forward. That auth paragraph is an edit to the agent’s context. An edit to context loads into every future session, for every task, and it can regress cases that were passing exactly the way any prompt change can: a rule written to stop six auth failures can quietly bend three unrelated behaviors. The gate that catches that is a golden dataset run before the change merges, and it is its own piece.

Counting told you which row to fix. The shape test told you where to fix it. Whether the fix held is the next thing to measure.

About the numbers. All of it is toy, invented for traceability: the 40 sessions, the 26 successes, the 14 failures, every session id, all four category counts, the shape tags, and the projected zeros. The arithmetic on top, meaning the 35 percent failure rate, the 42.9 percent top-category share, the 7/4/3 surface rollup, the 50/28.6/21.4 split, and the hand-count versus jq-tally match, follows from those inputs and was re-checked with a script before publishing. The skew is the realistic part: one category out front with a tail behind it, the same shape the manual-count sibling’s window produces, shrunk to a size you can tally in your head. The half-and-half split is a property of this toy log; your own log’s split is whatever your shape test says it is, and there is no reason to expect it to land on exactly 50. No number here is a measurement of a real team’s transcripts, and none is quoted from an outside source.

For the per-tool mechanics, see Hooks for the session-end capture and the test-command interception, Headless & CI for running the judge in batch, and Rules and Permissions for where the fixes land.