Reproduce AI Agent Bugs as Synthetic Test Scenarios

Sprint 214. The on-call bot’s ticket-cleanup skill has been running nightly for a month, and a report lands from a support engineer: “It deleted resolved tickets instead of archiving them. I watched it happen on two of the last ten cleanup runs.”

That is the whole report. One sentence of complaint, one sentence of data. There is no stack trace, because nothing crashed. The runs completed, the summaries read fine, and two tickets that should exist in the archive, T-481 and T-490, do not exist anywhere.

You do the normal thing. You read the skill, find a plausible cause, tighten the wording so it says archive and never delete, run one cleanup by hand on a test board, and watch it archive. Green. You type “fixed” into the thread. Three weeks later the same report comes back, slightly reworded, and you are holding a question you cannot answer: did the fix break, or did it never work?

Two things make that loop hard to escape by trying harder. The one artifact that would reproduce the failure exactly, the transcript of the run that deleted T-490, is the one artifact you are not allowed to keep: it is full of real ticket subjects, real customer names, real people having bad days. And the thing you are allowed to keep, a safe fake stand-in you build yourself, runs on a non-deterministic agent, so it will pass on some runs and fail on others.

Here is the number this piece derives and then pays off exactly. Confirm the fix with a single rerun, on a bug that still fires one run in five, and there is an 80% chance you call it fixed when it is not. That 80% falls out of the reporter’s own sentence: “two of the last ten runs” is a flake rate, and a flake rate is the one input the arithmetic needs. By the end you will compute the same number for your own bug, from your own report, and know how many reruns a “fixed” verdict is worth.

Thirty seconds, two questions, on the last bug someone reported against one of your agents.

One. Can you run it right now? Is there a command on your machine that shows that failure today, without asking anyone to remember anything? Not “we saw it in March.” A command, with an exit code.

Two. When you declared that bug fixed, how many times did you rerun the check, and did a number choose that count, or did you stop when it looked right?

If a committed fixture exists that fails on the old code and passes on the new, and you reran it a count you picked in advance, you already practice what this piece builds. Close the tab.

If the failure lives in a chat thread and the all-clear lives in someone’s memory, keep reading. A fix with no check behind it has one test left, and the second bug report is it.

Steel-man the obvious move first: replay the real session. Real traffic is the highest-fidelity test data that exists, and any team that can wire production transcripts into a replay harness should. For most agent failures the path is closed, and for the best reason. The transcript of the cleanup run that went wrong contains ticket subjects, customer names, and internal notes. Privacy rules, compliance, or plain decency say you cannot read it, and you certainly cannot commit it as a test fixture. The evidence and the policy point at the same file.

This is the mirror image of your failed agent runs are labeled data, which harvests every failed session automatically, at volume, to find patterns across many failures. Harvesting works there because those transcripts were safe to keep. The ticket bug is the other case: one reported failure whose transcript is the thing you must not log. Volume tooling cannot help you. The failure has to be rebuilt by hand.

Left alone, the failure stays anecdotal, and an anecdotal failure has one fatal property: you cannot tell when it is gone. The next complaint is the only news you will ever get. Meanwhile the report itself sits there holding the only testable fact in the whole incident, “two of the last ten runs”, and the standard move is to thank the reporter, fix something, and throw that fact away.

You cannot replay the session, so the failure must be manufactured from the report, and the report already carries the one number that manufacturing will need.

The deliverable is a synthetic scenario: the smallest invented world in which the agent provably fails, written as a file with no real user data in it. For the ticket bug it looks like this:

{
  "name": "cleanup archives resolved tickets and deletes nothing",
  "context": {
    "tickets": [
      { "id": "T-481", "status": "resolved" },
      { "id": "T-490", "status": "resolved" },
      { "id": "T-502", "status": "open" }
    ]
  },
  "query": "Clean up the resolved tickets from sprint 214.",
  "expect_tool": "archive_ticket",
  "forbid_tool": "delete_ticket",
  "expect_contains": ["T-481", "T-490"]
}

T-481 and T-490 are the IDs the reporter already put in the tracker, the only thing borrowed from the incident: subjects, names, and notes stay in the transcript you cannot keep. T-502 is open, on purpose: a cleanup skill must leave open tickets alone, so the fixture checks that the agent told the difference, not just that it did something. The assertions sit at the layer where the damage lives.

That layer distinction matters, because agent bugs come in two kinds. Some are in the answer: “it cannot reason about last week’s events” is a complaint about what the agent said, and a substring assertion on the reply, expect_contains, catches it. The reply here reads perfectly: “Archived 2 resolved tickets from sprint 214.” Every word of it true, and two tickets gone anyway. The damage is in the trajectory, the sequence of tool calls the agent made on the way to that sentence. In two of the reporter’s ten runs, the agent reached for delete_ticket instead of archive_ticket. No grep of the prose will ever see that, which is why the assertion moves up a layer: expect_tool and forbid_tool, read out of the structured output that headless mode hands back.

The file is the bug, frozen. It carries no real data, so it can be committed. It is diffable, so it can be reviewed, and anyone on the team can see exactly what “clean up” was supposed to mean. And it fails, today, on the code that is broken, which is the property that makes everything downstream honest.

One report producing one fixture is a fine habit while reports arrive slowly. It becomes a system when authoring stops being your job: a slash command, or a dedicated subagent with a clean window, that reads a report and emits a failing fixture for every reported failure:

Read the failure report pasted below. Produce ONE scenario file
under scenarios/ that reproduces it.

1. Name the capability the report claims is broken.
2. Invent the smallest fake context that exercises it. Nothing copied
   from any transcript. IDs already in the tracker are fine; subjects,
   names, and notes are not.
3. Assert at the layer where the damage lives: expect_tool /
   forbid_tool for tool bugs, expect_contains for answer bugs.
4. Run the scenario once, headlessly. If it does not FAIL on the
   current code, fix the scenario, not the code.

Deliverable: a committed file that fails. You do not fix the bug.

The instruction doing the most work is the last one. The command is forbidden from fixing anything, and the agent whose whole job is proving the bug exists is exactly the agent you do not want getting clever and routing around it. The isolation of a separate window keeps the proof uncontaminated.

If the failure were a feature you had not built yet, you would write the acceptance test forward, before any code, the way bake the acceptance test into the spec does: seed the order, refund it, assert the state. A bug report is the harder prior case: no spec to bake anything into, just a complaint, and the acceptance test has to be reverse-engineered out of it.

The synthetic scenario turns an anecdote into a red test you are allowed to commit, and the command that writes it must be seen failing before anyone is allowed to fix anything.

A scenario on one laptop reproduces the bug once, for one person, until the laptop is wiped. The same scenario wired into CI reproduces it forever, at the exact moment someone reintroduces it. Headless mode makes that cheap: one process, structured output, no human.

#!/usr/bin/env bash
# scripts/confirm.sh <scenario.json> <runs> - the shape, not the shipping version
set -euo pipefail
f="$1"; runs="${2:-14}"
for i in $(seq 1 "$runs"); do
  out=$(agent -p "$(jq -r .query "$f")" \
          --context "$(jq -c .context "$f")" \
          --output-format json --max-turns 20)
  jq -e '.tools | index("archive_ticket")'      <<<"$out" >/dev/null || { echo "run $i: archive_ticket never called"; exit 1; }
  jq -e '.tools | index("delete_ticket") | not' <<<"$out" >/dev/null || { echo "run $i: delete_ticket fired"; exit 1; }
done
echo "confirmed: $runs/$runs green"

Cap every run: --max-turns, a spend cap, a timeout. A regression gate that can burn a CI budget gets disabled, and a disabled guardrail guards nothing.

This is the merge-gate sibling of the runtime guard in make a hook validate the agent’s output. The hook catches a bad output on live traffic, the moment it happens, and retries once. The fixture catches a past failure on every change, before it ships again. Different moment, different question: catch it now, or prove it never comes back.

That handles the first half of the tension. The failure is a file, it runs on every commit, and when it goes red, someone gets a name and a diff instead of a mood. Replayed in CI, the fixture catches the bug coming back; nothing in it yet certifies the bug gone.

You fixed it. The skill says archive, never delete; better, you took delete_ticket out of its toolset entirely. You run the scenario once. Green: archived, nothing deleted, T-502 untouched. You are about to type “fixed” into the thread.

This is where the second half of the tension bites. Agent output is non-deterministic. Even pinned at temperature zero, the same prompt yields different tokens run to run: floating-point addition is not associative, so the matrix math can land on slightly different numbers depending on order, and when two candidate tokens are nearly tied, the order flips. On top of that, whatever requests happen to be batched with yours shifts the numbers. The cleanup skill already proved the point in production: same code, same board, ten runs, two different behaviors. The bug does not fire every run. It fires one run in five, and the reporter counted. Call each run a pass when every assertion holds, and call it a false clear when a still-broken agent hands you green runs and you read “fixed” off them.

Now the smallest version of the question, small enough to check with a pencil. Suppose the fix did nothing at all. You reverted the wrong commit, say, and the skill still misfires on one run in five, p = 0.2, exactly as before. You rerun the scenario twice. What is the chance both runs come back green?

Do it the obvious way first: list everything that can happen. Each run independently passes with probability 0.8 or fails with 0.2:

Four rows, and the probabilities sum to 1.00, which is your proof that the enumeration is complete. The outcome you would have read as “confirmed fixed” is the first row: 0.64. Two reruns of a still-broken one-in-five bug come back green 64% of the time.

Generalize the obvious way. One run passes with probability (1 - p). N independent runs all pass with (1 - p)^N. At p = 0.2 and N = 2 that is 0.8 x 0.8 = 0.64, matching the enumeration exactly, no approximation. The formula earns its keep the moment N outgrows the pencil: at N = 10 there are 1,024 outcome sequences to list, and (1 - p)^N says 0.107 immediately. And because a formula agreeing with its own derivation is not a check, the dice version: 200,000 simulated batches of 10 runs at p = 0.2, counting the all-green batches. The simulation lands on 0.1074 against the formula’s 0.1074. Enumeration, algebra, and simulation agree.

Now the number from the top of the piece, at N = 1: (1 - 0.2) = 0.80. A single confirming rerun on a still-broken one-in-five bug comes back green 80% of the time. That is the spoiler re-derived. When you typed “fixed” into the thread after one green run, the arithmetic says that green run was 80% likely with the fix doing nothing. The bug was in there, waiting for sprint 216. One green run is a sample of one from a distribution you already know is one-in-five hostile, and 80% is what that sample is worth.

The question has flipped. “Did the fix work” has become “how many green runs before a fixed verdict means anything.” That deserves a name, because you will want it in every retro.

The confirmation floor: the smallest number of reruns N for which (1 - p)^N drops under the risk you accept, where p is a fact quoted from the bug report’s own count. It is the floor under the word “fixed.”

You choose the tolerance. Five percent is a defensible one for a destructive bug: you certify fixed while knowingly holding a one-in-twenty chance of being wrong. Move it and only N moves.

Here is the whole relationship for this bug, p = 0.2, in one table. This is the picture to keep; the close replays it.

Row 3 deserves a pause, because a neighbor already gave you a number there. Prompt regression tests scores a whole golden dataset per merge and settles on three runs per case as a sane floor between cost and signal. That floor was sized to keep a fixed-cost suite affordable. Certifying one flaky repro is a different job, and at this bug’s flake rate three green runs still leave a 51.2% chance the bug is in there. Better than even, for the bug. Borrow the neighbor’s N without re-deriving it and you clear this class of failure about half the time, without knowing which half.

The strict floor: 0.8^13 is 5.50%, still over. 0.8^14 is 4.40%, the first integer under. Fourteen reruns for this bug at a 5% tolerance: the same 14 sitting as the default in confirm.sh, now earned. Fourteen headless runs of a two-second scenario is about half a minute of CI, less than the standup you would spend arguing about whether it is really fixed.

For your own bug, the recipe takes three inputs: p from the report’s own count, a tolerance you can defend, and the smallest N that clears it. N is not a feeling; it is the smallest run count whose all-green is harder to explain by luck than by the fix, and the bug report hands you p for free.

Four limits, stated while the win is still warm.

First, every run in this piece scores as one boolean: expect_contains and expect_tool and not forbid_tool, bundled into a single pass or fail. That is honest arithmetic but coarse evaluation. Trajectory evaluation as it ships in 2026, in LangChain’s AgentEvals with its trajectory-match and LLM-judge evaluators, LangSmith’s trajectory evals, Confident AI, and strands-agents/evals, scores a run along separate dimensions: did it pick the right tool, did it extract the right arguments, did the final answer carry the right facts, did it recover from its own error. The separation matters because an aggregate pass can hide a real regression in one layer behind a pass in another. A fix that always calls archive_ticket but starts passing the wrong ticket ID reads green under this piece’s four-line assertion and red under a framework that scores argument extraction on its own. Those tools are the productized version of the hand-rolled check above; the hand-roll exists here because the mechanism is the point. Per-dimension scoring is the next problem, named and handed forward.

Second, one scenario is one query and one response. Agents that hold state across turns fail in ways a single-shot fixture cannot reach: the bug that only appears on turn three, after the user’s phrasing shifted. The frontier practice there is simulation, a synthetic user persona driving the agent across many turns and scoring the whole conversation. It is heavier and different, and it is out of scope here: this piece is about one reported failure, and one reported failure fits in one fixture.

Third, the anti-hype note. The confirmation floor prices the reruns. It says nothing about whether the scenario deserves them. A fixture that does not actually exercise the bug, because its fake context is too easy or its assertion checks the wrong layer, will pass N times at any N and mean nothing at any N. What the arithmetic certifies is the reruns. Whether the scenario deserves them is a judgment no table can make. That is why the authoring command’s “run it once and confirm it fails today” step is load-bearing: the only evidence a scenario offers that it exercises the bug is that it was seen failing on the broken code. No table sizes that risk.

Fourth, when not to reach for any of it. A bug that fires almost every run does not need this math; one or two reruns settle it, and the formula just hands you a small N. A bug that pins to a line of your own code, a crash, a parse error, an off-by-one, is a unit test’s job and always was. The floor earns its cost in the flaky middle, where the bug hides one run in three or five or ten and a single green run feels like proof. That is where this bug lived, and where “I ran it and it worked” does the most damage.

The floor is a tool for the flaky middle; near-certain bugs and deterministic bugs have cheaper answers, and no amount of reruns certifies a scenario that was never exercising the bug.

Replay it with the incident attached. The reporter’s sentence, “two of the last ten runs”, gave you p = 0.2. Nothing else in the whole incident gave you anything testable: no transcript, no log, no repro steps. That one fact, run through (1 - p)^N, becomes a decision that used to be a feeling.

Row 1 is the intro’s 80%, paid off exactly where it was promised. Row 3 is the borrowed floor that does not hold here. Row 14 is 4.40%, the strict floor for this bug at a 5% tolerance, half a minute of CI. Your own bug changes only the inputs. Quote p from your reporter’s own count; if nobody counted, run the scenario on the old code ten times and count yourself. Pick the tolerance you can defend in a retro. Take the smallest N that clears it. If N comes out absurd, that is a bug too flaky to confirm cheaply, and the honest moves are to fix the flakiness itself or to accept a smaller N with your eyes open. Once the table has priced the choice, you sign the verdict knowing what it costs.

One cost is still unpaid, and this piece’s own arithmetic created it. Each run here scores as one boolean, content and trajectory bundled together. That holds while you have one fixture per bug. Against every merge, with dozens of fixtures, the bundling starts to hide things: the tool call correct, the argument wrong, the answer right for the wrong reason, all reading green. Scoring each dimension on its own is the gap named above, the 2026 tooling already does it, and it is the open question this hands you: how many dimensions does a verdict need before you would stake a customer’s tickets on it?

Reproduce the bug before you fix it. Then rerun it as many times as the bug’s own count says a green run is worth.

About the numbers. The incident is invented: sprint 214, the cleanup skill, tickets T-481, T-490 and T-502 are a worked example, sized so every number in the piece can be checked with a pencil. Within the example, the flake rate p = 0.2 is quoted from the reporter’s own count (“two of the last ten runs”), so it is a fact of the scenario rather than a tuning knob. Every later figure, the 0.64 enumeration, the 80%, every table row, and the N = 14 floor, is arithmetic on that p, re-checked with a script and cross-verified two ways: hand enumeration at N = 2, and a 200,000-trial simulation at N = 10 (0.1074 against 0.1074). The 5% tolerance is a choice you are free to move; N = 14 is its consequence at p = 0.2. The half-a-minute CI figure assumes a two-second scenario run; swap in your own runtime and the arithmetic stays the same shape. The temperature-zero non-determinism claims are known properties of inference stacks, not measurements made for this piece. The tooling names are those of products that exist as of 2026; none was tested here.

For the per-tool mechanics, see Headless & CI for replaying scenarios unattended, Slash commands for absorbing the repro workflow into a repeatable command, and Subagents for isolating scenario authoring from the fix.