Every merged PR is a benchmark.

The question public benchmarks can't answer

SWE-bench and its descendants measure whether a generic agent can patch generic open-source repositories [1]. That's a useful number, but it isn't the question a company deploying a coding agent actually has. That question is narrower: can this agent ship code that meets our bar, on our codebase, under our platform's rules?

This series is about the system we built to answer it. The valuable artifact turned out not to be any particular set of tasks; anyone can accumulate tasks. It's the machinery that can turn an organization's repositories and shipped history into a digital replica of how that organization builds software: its real codebases, the changes it has actually shipped, and the criteria by which it judges them, captured in executable form. Drop any agent into the replica (ours, a vendor's, a frontier lab's) and you get a repeatable answer to "does this work for us." Others have built the measurement half of this: Ramp mined a private benchmark from their own codebase [2], and Harvey did the equivalent for legal work [3]. Both stopped at measurement. Part 2 is about what happens when you don't stop: when you point an optimizer at the replica and let it climb.

One quirk of our setup turns out to matter a lot. InstaLILY builds on a shared platform template that is branched into a family of production codebases, so we don't have a codebase; we have a family of them. That gives the environment something no single-repo benchmark can have: a built-in generalization test. Hold one codebase out of tuning entirely and evaluate on it cold: a gain that carries over means the agent learned the conventions the branches share, not the quirks of a single repository. Part 2 uses this test in chapter 11.

Every merged PR is a graded exam

That insight becomes an environment through what we call the rollback contract. For each case we:

1. Rewind the repository to the commit just before a real, shipped change.
2. Describe the problem in plain English, the way a coworker or end user would: business language, no file paths, no hints about how the original fix worked.
3. Let the agent implement blind. It never sees the reference solution or the grading tests.
4. Score against the real shipped version as the answer key.

Because ground truth is free, the environment is also resettable: rewind any production repository to any point in its shipped history, replay the same real change as many times as you like, and always know the right answer. Replayable tasks with a trusted answer key are what let Part 2 treat this as something an optimizer can climb, not just a one-shot exam.

Most PRs don't make the cut. The tempting move is to mine every merged PR and call it a dataset; curation is the opposite. One invariant governs everything: a case is valid only if its gold behavior fails before the change and passes after it. Before any test enters the suite, we confirm a do-nothing agent scores zero on it and the real shipped fix scores one.

The subtlest filter: the "before" failure has to be a genuine behavioral failure, not an import error. A test that blows up only because a function doesn't exist yet measures existence, not competence; it would reward an agent for creating an empty function with the right name. Beyond that, we reject mega-merges with no single intent, umbrella UI changes too broad to attribute, and low-value hygiene PRs, each dropped with the reason recorded. Eight automated gates enforce these rules on every case.

From a PR to a runnable task

A surviving PR becomes one record with two payloads that pull in opposite directions on purpose. The request the agent sees is deliberately vague about implementation: pure business language. The validation assertions are deliberately precise about behavior, and the agent never sees them. That separation is what makes a task realistic on one side and gradeable on the other.

Each record is compiled into a sandboxed task using Harbor, the task format from the Terminal-Bench team: an instruction file (the only thing the agent sees), an environment built from the codebase's stack at the PR's base commit, a verifier that writes the reward, and the real shipped fix kept aside as the oracle, used only to confirm the task is solvable at all.

A score you can trust

Real product changes aren't graded by unit tests alone, so neither are our tasks. Every check in the suite is one of three kinds, trusted in proportion to how hard it is to fool:

Two design choices keep the subjective parts honest. The judge that writes the grading tests reads the same plain-English request the agent got and writes its tests from the request, never from the agent's code, so the two can't collude. And the judge that scores craft never runs tests at all; it scores only what executable checks can't reach: where the change lives, how it reads, whether it looks like the platform's own code. Everything runs sandboxed, the grading pipeline is fully isolated from the agent, and the deterministic outcome check has the final word. Those guards matter beyond fairness: they're what make the score safe to optimize against, which is Part 2's whole premise.

One run is a random variable, not a fact

One more property belongs in the trust contract, and most eval writeups skip it. The same agent, on the same task, at the same settings, does not reliably produce the same verdict: across our head-to-head runs, 26% of task verdicts flipped between attempts of the same agent: one in four, pure sampling variance, with not a single infrastructure failure involved. That's not a flaw in the environment; it's the true variance of the thing being measured, and we report it rather than averaging it away. It also has a sharp consequence: a gap between two agents means nothing until it beats what this variance produces on its own. Part 2 opens by measuring exactly that.

What it says about our agent: Lily vs. Claude Code on 46 real tasks

We ran the suite as a head-to-head between Lily and Claude Code, both running the same underlying model (claude-opus-4-8 at maximum effort). Three production codebases, 46 tasks, 3 attempts each: 276 trials. Neither agent scored a perfect 1.0 on any single trial.

The two headline lenses genuinely disagree, and the disagreement is information:

Where the lenses conflict, we treat the behavioral bar as the more authoritative one; it's the deterministic, ungameable floor. We publish both, because an eval that only reports the lens its own agent wins isn't an eval.

The anatomy of a failure

Aggregates tell you who's ahead; transcripts tell you why. We read every task that resisted all attempts and sorted the failures into families:

Notice what the taxonomy implies. In most of these failures the model could do the work, and lost on a habit the transcript shows it never applied: check the codebase before inventing a name, run the code before claiming it works. Failures of discipline, unlike failures of capability, are exactly what the machinery around a model can enforce. Part 2 starts from this table.

Why we call it an environment, not a benchmark

A benchmark tells you where you are. An environment lets you climb. The replica qualifies as an environment because of three properties most benchmarks don't have:

Which sets up the move this series is really about. We now have a trustworthy signal of how well any agent does on our work, and a trustworthy signal is an invitation to optimize. The model underneath Lily is frozen; we don't train it. Everything else around it is fair game. But there is a trap in that idea, and we ran into it in the first campaign.

Freeze the model, search everything else

The mismanaged geniuses hypothesis [7] explains why harness search is worth trying: capable models can look weak when the wrapper mismanages them. Automated harness search acts on that idea: stop hand-tuning the scaffold and search over it.

A harness is everything about an agent that isn't the model [8]: what context it assembles, which tools it exposes, how results are fed back, when a run is allowed to end. Part 1's head-to-head was already a two-point sample of this space: the same frozen weights inside two different wrappers produced two measurably different agents. The wrapper is not packaging. It's a variable worth double-digit points, and unlike the weights, it's a variable we fully control. So the move is: freeze the model, and search over the harness.

The search itself is a loop between two roles with opposite temperaments, an idea we borrowed from the meta-harness line of work [5], which validated the same recipe on a public legal benchmark while we were building ours [6]; the same transcript-driven shape drives GEPA's reflective prompt evolution [9] and Self-Harness's weakness-mining loop [10]:

Three rules do most of the work. One change per candidate, so wins stack and causes stay attributable, the same single-mutation discipline AlphaEvolve's evolutionary search is built on [11]. Every proposal must name the failures it attacks, citing the transcripts that motivated it: no vibes, and the held-out codebase is never read. And nothing is believed until it survives both gates: clear the null band, then win again on an independent re-run it was not selected on.

The failure mode researcher is where the trial volume is, so that role runs on a second, cheaper frozen model (gpt-5.5) under the same harness. Every result that follows, including chapter 11's held-out test, is measured on that exploration model. Whether the winning mechanism also lifts the deployment model behind Lily (claude-opus-4-8) is a separate transfer we have not yet measured - promising ground for further exploration, not a conclusion this series draws.

Three candidates, zero survivors: the first campaign's real result

The first campaign targeted the hardest ground: the 14 improvable tasks where the seed agent scores lowest: real headroom, real failure transcripts to learn from. The research agents read those transcripts and proposed three candidates [9], each a carefully written addition to the system prompt: one pushing "reuse what exists" discipline, one a checklist aimed at hidden graders, one enforcing minimal scope. All three came back from the first evaluation looking like winners:

Candidate B looked large: +0.14 against the seed. But on a 14-task slice, the null band is already about ±0.1. On an independent re-run (five fresh trials B had not been selected on) it scored below the baseline, and the three tasks it had "solved" all regressed. Nothing about B changed between the two runs; the first score was sampling noise. This is the left panel of Figure 3.

Campaign 1's honest result was zero survivors out of three. And the pattern in the failures pointed somewhere specific. All three candidates were prompt text: paragraphs added to the system prompt. A prompt requests a behavior; it does not enforce one the way a code change to the harness can. The agent reads the paragraph, agrees with it, and then skips the check anyway. So for the second campaign, we stopped proposing prompt changes and started proposing code.

How big does a gap have to be before it means anything?

Candidate B didn't fall because it was a bad idea. It fell because +0.14 on a 14-task slice was never outside the margin of error to begin with. Due to the non-determinism of LLMs, the environment contains swings that size inherently. This begs the question every result in Part 2 depends on: how big does a gap have to be before it means anything at all? An optimizer is a machine for finding differences; if the environment manufactures differences out of thin air, that is exactly what it will ship: the "p-hacking and eureka-ing" pattern already documented when frontier models were pointed at open science problems [12].

The ±0.1 that swallowed candidate B was not a guess. It came from an experiment in which we ran two identical agents through the same environment, independently, and measured the gap. Same agent, same tasks; only the luck changed, so whatever difference shows up must be manufactured. It is the picture to keep in mind for the rest of Part 2:

The rule is blunt: if the margin of error is about 0.1, a +0.1 gain is not evidence yet; it may be the whole effect, or it may be nothing, and only fresh trials the candidate was never selected on can tell the two apart. That is the test candidate B failed. The same test applies to our own results.

Put proper error bars on Part 1's scoreboard and Lily's 13-point pass@3 lead sits right at the edge of what 46 tasks can establish, and Claude Code's craft lead doesn't clear the bar either. So we downgrade our own headline: both are leans, not leads. That's not a loss. An eval that audits its own headline is the only kind whose headline you should believe, and this discipline is as much a part of the product as the tasks are.

What survived: mechanisms, not messages

The next candidate came from the same automated researcher as Figure 2, the language model that reads failure transcripts and proposes one change per round. Handed campaign 1's rejected prompt advice, the pivot was to stop suggesting behavior and start enforcing it. Because Lily's harness exposes lifecycle hooks, the candidate could be working machinery rather than another paragraph: a ~30-line extension that intercepts the moment the agent believes it's finished. It went through the same two gates as every other candidate. The mechanism, the forced review, does one thing: when the agent tries to end its run, it doesn't get to. The harness injects one structured review turn: check every name you introduced against the codebase, re-check every requirement, confirm the change is in the right file, exercise the non-obvious branch. Fix what's broken; if nothing is, stop.

On the same 14 tasks that killed the prompt candidates, the forced review lifted the strict pass rate by 0.08 points, from 0.11 to 0.19, and this time the gates agreed: the gain held on two independent re-runs, with one task moving from never-solved to solved in both. The second mechanism pushed the idea further. A review the agent merely thinks through can fail exactly the way the original work failed; it reasons about the edge case instead of running it. So the execution-forced review requires the agent to show evidence:

Set that against Part 1's failure taxonomy and you can see why it works: each line targets a family. "Grep, don't recall" attacks contract precision, half of all failures on its own. "Execute and paste the output" makes verification theater impossible. The result was the campaign's headline: strict pass rose from 0.11 to 0.24, a little more than doubling the seed rate, with +0.13 over the seed held across ten evaluation passes (p = 0.007), launched as two independent batches. Figure 3's right panel: ten fresh draws, all high, because the distribution itself moved.

The full tally across both campaigns: fifteen candidates evaluated, two promoted. Most candidates failed a gate, which is the expected outcome; the gates exist to reject changes that don't replicate. Rejected candidates aren't deleted; they stay in the researcher's reading pile, and the winning mechanism reuses an earlier reject's idea with a stronger requirement.

Generalization or memorization

A validated win on the tasks you tuned against is still only half a result. Maybe the mechanism learned the platform's engineering contract, or maybe it memorized the dev slice's quirks. The environment gives us a clean way to ask which: an entire codebase the optimization never touched. Codebase C is twenty sealed tasks, re-authored from real merged-PR history with fresh grading suites; no candidate was ever screened against any of it, and we wrote down the decision rule before running a single trial.

The test: seed harness versus optimized harness, cold, five attempts per task. Both arms run the frozen exploration model, the same substrate the search optimized on; this chapter measures whether the mechanism crosses codebases, not whether it lifts the deployment model. If the campaign only memorized its dev slice, this is where that shows up as nothing. Here's what happened instead, task by task:

Seven of the twenty tasks improved. Two declined. Eleven didn't move. Four tasks the seed never solved in any of its five attempts got solved: the review mechanism's signature move, rescuing work that was almost right but never checked. Both declines are a single attempt's worth, the size of ordinary luck. On the simple count: the seed solved 8 of 20 held-out tasks; the optimized harness solved 11.

That answers the chapter's question. The mechanism was tuned against two codebases and it moved a third it never touched, on freshly authored graders, keeping about half of its internal margin. Effects usually shrink outside the set they were tuned on; a memorization collapse would have shown up as no transfer at all. What it couldn't do is just as informative: eleven tasks never moved, pinned at the same score attempt after attempt.

Where the wrapper stops, and what we're publishing

The search does not climb forever. The review mechanism attacks contract precision and verification theater, and those two families are where its gains demonstrably come from. What's left, the tasks still failing every attempt under the best harness, increasingly looks like Part 1's genuinely hard family: missing domain knowledge, multi-system reasoning. A wrapper can force a model to check its work; it cannot give the model knowledge it doesn't have. The model chooses your ceiling; the harness decides how close to it you operate.

That split is the practical takeaway. A public leaderboard score is the product of model × wrapper, entangled. This environment disentangles them for our work specifically: the wrapper is worth double-digit points, it costs thousands of dollars rather than a training run, and its gains belong to whoever owns the harness; they compound on your environment and transfer to the next codebase your platform takes on. Buy the strongest base model you can. Then measure the tuning, because the same search that found us a real +0.13 will, run without a null band, return a +0.14 that is pure noise.

In summary: mine an organization's merged PRs into a sealed, resettable replica of how it builds software; measure how much improvement that replica can manufacture out of nothing, so its scores can be trusted; run a gated, trace-grounded search over everything around a frozen model, promoting only what survives independent replication; and prove the survivors on a codebase the search never touched. Every piece compounds: each new codebase adds a held-out fold, each incident adds a gate, each failure family is the next mechanism's hypothesis.

The tasks themselves stay private, partly because the codebases behind the tasks are proprietary, partly to avoid contamination: the moment task data is public, every future model may have trained on it [2]. The machinery is a different story. We will soon release a package that builds this environment for any organization: point it at your repositories and it mines your merged PRs, applies the curation gates, and compiles a sealed, resettable suite that can test any agent on your work, with the null-band audit and the promotion gates built in, so the first number it shows you comes with the error bars that make it believable. The tasks it produces from your history are yours, and they stay as private as ours do. If you want to run it against your own codebase, or point the optimizer at your own agent, tell us.

References

1. Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2024). arxiv.org/abs/2310.06770
2. Ramp Labs, Ramp SWE-Bench: Evaluating background coding agents on financial SWE work (June 2026). labs.ramp.com/swebench
3. Harvey, Introducing Harvey's Legal Agent Benchmark (May 2026). harvey.ai/blog/introducing-harveys-legal-agent-benchmark
4. Lee et al., Meta-Harness: Optimizing the Scaffold Around a Frozen Model (2026). arxiv.org/abs/2603.28052
5. J. Niklaus, harness-optimization run of the Meta-Harness loop on Harvey's LAB with a frozen open-weights model (2026). github.com/JoelNiklaus/harness-optimization
6. Zhang, Li & Khattab, The Mismanaged Geniuses Hypothesis (April 2026). alexzhang13.github.io/blog/2026/mgh
7. L. Weng, Harness Engineering for Self-Improvement (July 2026). lilianweng.github.io/posts/2026-07-04-harness
8. Agrawal et al., GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning (2025). arxiv.org/abs/2507.19457
9. Zhang et al., Self-Harness: Harnesses That Improve Themselves (2026). arxiv.org/abs/2606.09498
10. Novikov et al., AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery (2025). arxiv.org/abs/2506.13131
11. Bubeck et al., Early Science Acceleration Experiments with GPT-5 (2025). arxiv.org/abs/2511.16072
12. Madaan et al., Self-Refine: Iterative Refinement with Self-Feedback (2023). arxiv.org/abs/2303.17651