AI Agent Evals in CI: Run Two Loops, Not One Suite

You wired up evals because you got burned once: an agent shipped a prompt change that quietly broke half your tool calls, and nobody caught it for a week. So you did the responsible thing. You built a suite - deterministic checks and a judge model scoring the hard cases - and bolted it to CI. Every commit now runs the whole thing.

For a few weeks this is glorious. Then a PR sits for four and a half minutes waiting on the eval job. Then the judge flakes on a case that was actually fine, and the build goes red on a one-line docs fix. Then someone adds if: false above the eval step, with a commit message that says “temporarily skip, too slow.” It is never turned back on.

Here’s the tension underneath that story. A suite thorough enough to catch a real regression has to run a judge model over the cases a plain comparison can’t grade - tone, completeness, whether the agent picked the right strategy - and a judge model is slow and occasionally inconsistent in a way a string comparison never is. A suite fast enough that nobody resents running it on every PR has to drop exactly those cases, which means dropping the ones most likely to hide a real regression. No single suite can be both. A team that tries ends up disabling the one it built.

The fix isn’t a faster suite. It’s the same suite, paid for on two different clocks: a fast deterministic gate on every PR, and a slow judge sweep on a schedule. Split it that way and a team merging 8 PRs a day gets back 156 minutes of blocked PR time every week - 2.6 hours, down from just under 3 - without cutting a single case from the suite. Here’s exactly where that number comes from, and you can check every step of it with a calculator.

Before any of the arithmetic below, thirty seconds on your own CI config settles whether this piece is for you.

Open the eval job that runs on every pull request. Find the step that scores an agent’s output as “tone is right,” “answer is complete,” “the agent picked a reasonable strategy,” or anything else a model has to judge rather than a script comparing values. Now ask one more question: has anyone, ever, temporarily skipped that job under deadline pressure?

A judge-scored check on the critical path, and at least one skip in the git history: you have this problem, and the rest of this is worth reading. No judge call in your PR gate, or your whole suite is a dozen exact-match and schema cases: you don’t have this problem yet. Everything below is one worked example of what the split buys and what it costs. Close the tab if your own answer was no.

Fix one concrete case and keep it for the rest of the piece: an invoicing agent that answers billing questions and calls tools like fetch_invoice. Its eval suite has 50 cases, and they split by how they get graded, not by what they cover:

Thirty-two of the fifty cases are deterministic: exact match, a schema check, “did it call fetch_invoice with a numeric id.” Eighteen are judge-scored: the ones where “correct” isn’t a single string, so a model has to read the output and decide. That composition - 32 and 18 - is a stated assumption for this piece, chosen to be a plausible mid-size suite, not a measurement of your repo. Swap in your own counts later; the mechanism doesn’t change.

Compute the obvious way, before naming any fix. One full run traces all 50 cases, then hands the 18 judge-scored ones to the judge model on top of that:

cost = 50 x $0.004  +  18 x $0.03  =  $0.20 + $0.54  =  $0.74
time = 50 x 1s       +  18 x 12s    =   50s + 216s    =   266s

One full run: $0.74, 266 seconds. That’s what a PR sits through today, before either loop exists.

Now carve out only the deterministic third of the suite - no judge, no model call, just comparisons - and run that alone:

cost = 32 x $0.004  =  $0.128
time = 32 x 1s       =   32s

That’s the shape of a gate you’d actually run as a hook, firing before a change lands:

#!/usr/bin/env bash
# .git-hooks/pre-push - the deterministic gate. No judge, no API calls.
set -euo pipefail

npx vitest run evals/deterministic/ --reporter=dot

# 32 cases, ~32s, $0.128. Fails closed: a broken extractor
# or a malformed tool call blocks the push.

test("invoice agent calls fetch_invoice with a numeric id", async () => {
  const trace = await runAgent("show me invoice 4471");
  const call = trace.toolCalls.find(c => c.name === "fetch_invoice");
  expect(call).toBeDefined();
  expect(call!.args.invoice_id).toBe(4471);   // exact, not "looks plausible"
});

Thirty-two seconds. Thirteen cents. No flake, because there’s no model deciding whether the answer is “good enough” - the answer is either 4471 or it isn’t.

Now check the two numbers against each other. Dropping the judge doesn’t just drop 18 of 50 cases (36% of the suite by count). It drops 87.97% of the full run’s time and 82.70% of its cost. Eighteen cases - a little over a third of the suite - are responsible for nearly nine-tenths of what a PR sits through. That’s not a coincidence of this toy; it’s what a 12-second judge call sitting next to a 1-second trace always does to the average. The full run is 8.3125x slower and 5.7812x more expensive than the deterministic third of it alone, and that ratio comes from the same six numbers in the first table above - nothing rounded, nothing estimated.

A third of the suite is carrying nine-tenths of the wait. That’s not a suite problem. That’s a placement problem.

If this shape looks familiar, it should. It’s the same logic behind the test pyramid: run the cheap, fast checks constantly and push the expensive, slow ones to run less often. The one place this diverges from that lineage: the slow tier here isn’t just a bigger scope of the same kind of check, the way a full regression suite is a bigger scope of unit tests. It’s a nondeterministic judge - rerun it on an unchanged case and you are not guaranteed the same number back. That’s a different kind of slow, and it’s why the split has to be by grading method, not by test count.

None of this is a new idea, and it shouldn’t read like one. Search any current writeup on evaluating AI agents in CI and you’ll find the same shape already documented: a fast deterministic-or-classifier check gating the PR, a full judge sweep running on a schedule. What follows is one worked example of that pattern with the arithmetic behind it, not a claim to have invented the split.

Call the two halves what they are. The inner loop is the 32-case deterministic gate above, and it answers one question on every PR: did this change break something we already know about? The outer loop is the 18 judge-scored cases, run on a schedule instead of on a push, and it answers a different question: what’s broken that we don’t know about yet?

# .github/workflows/nightly-eval.yml - the outer loop
on:
  schedule:
    - cron: "0 7 * * *"   # 07:00 UTC, nobody's blocked
jobs:
  full-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npx tsx evals/run-full.ts --judge --all
        # all 50 cases, 18 judge-scored, writes scores.json
      - run: npx tsx evals/diff-baseline.ts
        # compares to a trailing baseline; opens an issue on a real drop

Two things belong to the outer loop and nowhere else: the judge model, and the “compares to a trailing baseline” step. A naive version of that comparison - flag anything more than 5% below yesterday - looks precise and isn’t. A judge that can’t reliably reproduce its own score has a noise floor, and a threshold set below that floor manufactures a phantom regression most nights on cases that never actually changed. The honest version measures the judge’s own spread first (run an unchanged case a handful of times, record how much the score moves on its own) and gates on something like a paired comparison against a trailing baseline rather than a bare percentage - a real statistical test, not a rule of thumb. That measurement is its own piece of work; this one hands it forward rather than building it out.

The inner loop asks about the past. The outer loop asks about the future. Confusing the two is how you end up blocking a PR on a question that can wait until tonight.

Say the outer loop’s nightly run does its job. A case that’s been stable for weeks - the one where the agent explains why it refused a request - comes back lower tonight than the trailing baseline says it should. Nobody’s PR is blocked while that happens. No one gets paged. The finding just sits in scores.json by the time anyone’s at a keyboard.

Hand that transcript to a subagent whose only job is triage, so nobody’s morning starts with reading raw transcripts:

You are the eval triage agent. Input: scores.json + transcripts
for every judge-scored case that dropped vs the trailing baseline.

For each regression:
1. One-line root cause (prompt drift / tool change / model update).
2. Is the failure deterministically checkable - exact match, schema,
   a specific tool-call argument? If yes, write the assertion for
   evals/deterministic/.
3. Rank by blast radius. Output a triage table only.

Now ask the question that actually matters: what happens to that discovery next? Leave it where it is, and the outer loop will find the same drift again tomorrow night, and the night after - a real cost paid on repeat, and one that never once stops it from landing in a PR, because the outer loop isn’t on the PR’s path. Or write the check once, as a deterministic assertion, and add it to the inner loop. Next time, a $0.03 judge call that took 12 seconds becomes a $0.004 comparison that takes 1 second and runs on every push, forever.

Call that the ratchet: every discovery the outer loop makes gets promoted into a permanent case in the inner loop, so the fast gate only ever grows and the guarantee it provides is never re-earned from scratch. The suite that catches a real regression tonight is catching it on every PR by next week, at inner-loop prices.

The outer loop finds what you didn’t know. The ratchet is what stops you having to find it twice.

Back to the running example: 8 PRs merged a day, 5 workdays a week - the team’s own stated cadence, not a universal one. Under the one-loop design, every one of those 8 PRs pays the full $0.74 and sits through the full 266 seconds:

one loop / day:  8 x $0.74 = $5.92     8 x 266s = 2,128s = 35.4667 min blocked

Under the two-loop design, each PR only pays the inner loop’s $0.128 and 32 seconds. The full $0.74 judge sweep still runs, but once, overnight, off the critical path - its cost persists, its 266 seconds don’t block anyone:

two loop / day:  8 x $0.128 + $0.74 = $1.764     8 x 32s = 256s = 4.2667 min blocked

Run that out over a five-day week, 40 PRs:

There’s the number from the top of the piece, reconciled exactly: 156 minutes a week, not an estimate, the same six inputs run forward through a week instead of a run. And it isn’t only wait time - the two-loop design also cuts the week’s judge bill from $29.60 to $8.82, because the judge sweep runs once a night instead of once per PR. Eight PRs paying for eight separate judge passes on cases that hadn’t changed since yesterday was always the expensive part; running the judge once and gating PRs on the cheap third of the suite is what the split actually buys.

Same fifty cases, same two clocks, run five days instead of one: $20.78 and 156 minutes, and neither number moved because anyone worked faster. They moved because the judge stopped riding along on every PR.

Scale the same arithmetic across team size, holding everything else - the suite, the per-case costs, the five-day week - at the running example’s values:

Row 8 is the running example, and it lands exactly on the spoiler from the top: 156 minutes. Look at row 32 next to it. The ratio between one-loop and two-loop wait never moves - it’s 8.3125x at every row, the same number from the per-run ratio two sections back, because doubling the PR count doubles both columns equally. What changes is the size of the number that ratio is multiplying. The teams merging fastest - the ones under the most pressure to disable a slow gate - are exactly the teams the split pays back the most, because they’re the ones paying the full-run price most often today.

“Just run the judge three times and take the majority - doesn’t that fix the noise?” It reduces the noise, and it’s worth doing on cases you do run through the judge. But it doesn’t touch cost or latency, and if it’s still sitting on the PR’s critical path, it multiplies both: three judge calls instead of one is $0.09 and 36 seconds per case instead of $0.03 and 12. Self-consistency sampling is an argument for what to do with the judge once it’s off the critical path. It isn’t an argument for keeping it on.

“Caching and cheaper judges make the full suite affordable - why split at all?” Semantic caching means an unchanged case skips a repeat judge call. A smaller, cheaper judge model drops the per-case dollar cost. Both are real and worth doing regardless. Neither one fixes the reason the split exists: a cached score is still a judge’s score, with the same noise floor, and a cheaper judge is usually a noisier one - it agrees with a human less often and wobbles more between runs, which is precisely the wrong direction for something you’re about to put back on a PR’s critical path. Cost was never the whole problem. Timing and reliability were the other two-thirds, and no amount of caching moves either one.

Both objections assume the judge is the fixed cost and the schedule is negotiable. This piece assumes the reverse.

It relocates the judge’s cost and latency. It does not fix the judge’s reliability. A threshold that isn’t calibrated to the judge’s own noise floor still manufactures false alarms on the nightly run - it just does it without blocking anyone while it happens. This is a documented property of LLM judges, not a guess: “Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks” (Haldar & Hockenmaier, Findings of the Association for Computational Linguistics: EMNLP 2025, arXiv 2510.27106, Oct 2025) finds that the same judge scoring the same output more than once shows low agreement across those runs - the judge disagreeing with itself, not with a human. Moving the judge overnight buys you a window to notice that before it costs anyone a blocked push. It doesn’t make the judge more consistent.

An all-deterministic suite gets nothing from this split. Run the same arithmetic on a 12-case suite with zero judge-scored cases, and the full run and the inner-loop-only run are identical: $0.048, 12 seconds, both ways. There’s nothing to carve out. Building the two-clock machinery before a judge-scored case exists in your suite is pure ceremony - a second CI config, maintained, for the same outcome as the first one.

The nightly clock has a fixed cost whether or not anyone merges. On a day with zero PRs, the one-loop design pays $0 - it only ever runs when a push triggers it. The two-loop design still pays $0.74 that night, for a suite nobody needed yet. A team merging steadily, 8 PRs a day, comes out $20.78 and 156 minutes ahead every week. A bursty or low-cadence team - a few PRs some days, none on others - pays that fixed nightly bill on the quiet days and sees a worse deal than the running example’s numbers suggest. Check your own cadence before assuming the savings transfer.

The ratchet only turns one way, and this piece didn’t say what stops it.

Stop asking one suite to be fast and thorough on the same clock. Split the cases you already have by how they’re graded, not by what they cover: the deterministic third becomes a hook that runs on every push and never gets skipped, because it’s never slow enough to be worth skipping. The judge-scored two-thirds becomes a nightly job that runs while nobody’s waiting on it, and its discoveries get promoted into the fast gate one assertion at a time. Row 8 of the table above is the payoff for an 8-PR-a-day team: $20.78 and 156 minutes back, every single week, off the same 50 cases you already had.

That leaves the ratchet’s own cost as the open question this piece didn’t answer. The inner loop only ever grows - every promoted discovery adds a permanent case, and nothing in this design pages anyone the day the “fast” gate stops being fast. A ratchet that’s been turning for a year is a bigger suite than the one you started with, growing on a schedule nobody set. What prunes it, or budgets it, before the fast loop becomes the next version of the exact problem this piece opened with - just with a slower-growing fuse - is a question for whoever’s still running this suite twelve months from now.

About the numbers. The suite’s composition (32 deterministic cases, 18 judge-scored) and the six per-case cost/time constants are toy values, invented for traceability and sized to be faithful to the real shape of the problem - a judge call costs and takes longer than a string comparison - without claiming to be measured. The team’s cadence (8 PRs a day, 5 workdays a week) is a stated assumption, not an observation of any real team. Every other number in this piece - the full-run and inner-loop-only cost and time, the 8.3125x/5.7812x ratios, the judge cases’ 87.97%/82.70% share of time and cost, the daily and weekly rollups, the master table, the all-deterministic and quiet-day limiting cases - is exact arithmetic derived from those inputs and checked against an independent script before publishing. The Rating Roulette citation is quoted and dated: the paper’s title, authors, venue, and existence were verified directly; its finding is described only in the general terms the paper itself uses (low agreement across repeated judge runs), with no specific score invented to illustrate it.

For the per-tool mechanics, see Headless & CI for running suites unattended, Hooks for the pre-push gate, and Subagents for isolating the triage pass.