When agents write the code, the pipeline is the product

AI agents let our small team ship at a pace no traditional team could: one engineer directing agents shipped 1,000 story points in a single sprint. That speed moved our bottleneck from writing code to trusting it and landing it. This is how our delivery pipeline had to grow up, what we chose and why, and what it means for teams adopting agentic workflows with DeepWind.

Where we started

We built DeepWind agent-first from day one. A handful of engineers direct a fleet of AI coding agents: builders take a brief and write the change, separate reviewer agents check it, and coordinator agents sequence the work.

Early on, the pipeline behind that was simple. An agent opened a PR, tests ran, the PR merged, and every merge auto-deployed to staging, with frequent production releases behind it. With a small app and a few agents, that loop was fast and good enough.

Then three things grew at once: the product (hundreds of routes, a multi-tenant database with row-level security, real-time agent features), the test suite that protected it, and the number of agents opening PRs at the same moment.

The bars aren't like for like, and that is the point. Before May 23, agents committed straight to main, so each bar counts raw commits. After it, every change is a reviewed, squash-merged PR carrying far more work per landing.

The wall

By September, writing code was no longer the constraint. Getting it safely onto main was.

• Merges slowed down. Median time from PR to merge reached 8.1 hours, and the slowest tenth took nearly a week.
• Checks ran over and over. In the week of September 14 we averaged 6.5 full gate runs per merged PR, and 65% of runs failed.
• Collisions multiplied. With a dozen agents landing at once, one PR's merge often invalidated another's green result, sending it back through the gate.
• Our hosted CI couldn't carry the load. Our plan's metered CI minutes ran out, so we had to run the quality gate ourselves.

The failures were rarely flaky tests. On one measured day, all 20 failed runs were deterministic. The cost was work that was never going to count: runs on stale code, runs judged by an outdated copy of the checks, runs that raced a newer merge.

The choices we had

We chose external sandboxes for scale, a dedicated machine for the few changes that alter the gate itself, and an agent-specific trust layer on top of both.

In hindsight, we built some plumbing that mature products already offer. The part worth building was the layer that makes agent-written code trustworthy at speed.

Why external sandboxes

External sandboxes turned gating from a queue into a fan-out: every PR gets its own disposable machine with a fresh database, and a dozen can run at once for about $0.23 each.

• Scale: capacity grows with demand instead of with the hardware on our desks.
• Isolation: each run starts clean, so one PR's leftovers can't poison another's result.
• Honesty: the sandbox always runs the checks from main, so a PR can't bring its own lenient copy.
• Cost: you pay per run, not for idle capacity. Our cost per merged PR fell from $0.68 to $0.48.

The key tools we built

Eight pieces carry most of the weight. Each exists because a specific failure cost us real time.

Around them sit a ratchet that lets known failures stay known without letting new ones in, and a disposable real database per run so security tests exercise real row-level rules.

What we learned

The lesson was not "test more". It was: make every check trustworthy, and never spend an hour on a run that could never count.

• Treat the author as a party who can grade its own homework. Agents optimize for green. Pin the judge.
• Bind evidence to exactly what it tested. A result for a different commit, base or version of the checks proves nothing.
• One source of truth per decision. Two components deciding the same thing drift apart. Ours did, five times in two days.
• Check before you spend. Up to a third of our failed runs over three days were predictable before they started.
• Batch the risky changes. Changes to the gate itself now land once a night as one batch, instead of freezing everyone during the day.
• Coordinate agents through records, not chatter. Holds are announced before long runs, and a shared ledger replaces message chains.

The results so far: median time-to-merge fell from 8.1 hours to 2.3 hours, and gate runs per merged PR from 6.5 to 2.9. The preflight doctor and affected-step re-runs are landing now, and we measure a full week from September 28.

Flaky tests were never the main problem. Most failed runs were predictable: stale code, an outdated copy of the checks, or real defects that a quick preflight or a local test run would have caught.

Gating more work took more runs. In four weeks, merged PRs went from about 2 sandbox gate runs each to about 5. Most of that growth is new revisions of the same PR, not reruns of the same code: the number of distinct commits gated per merged PR rose from 1.7 to 3.4, as agents refreshed branches onto a moving main and fixed what the gate found.

The waiting around the gate shrank even as gated merges grew ninefold, from 14 a week to 131. A new PR now reaches its first gate run in a median of 34 minutes, down from about 9 hours, and nine in ten green PRs merge within about an hour of going green, down from 12 hours.

The cost moved into the gate itself. A single run now takes a median of 28 minutes, up from 8, because the test suite and the list of checks grew with the product. Cutting that is our next target: rerun only the checks a change can affect, instead of everything.

The week of September 21 got faster

The week of September 21 felt faster, and the numbers agree. Compared with the week before:

Most of the gain came from removing reasons to fail, not from adding checks:

• False failures fixed. A tool missing from the sandbox image failed one check on every run; two self-tests broke on how shell pipes close; a file-date command behaved differently on Linux than on macOS; and one self-test leaked the host's settings into its fixtures. Each looked like a broken PR and cost a rerun.
• Batches instead of queues. About 35 PRs landed through merge trains between September 24 and 26, gated once per batch instead of once each.
• Fewer duplicate runs. A green sandbox run now also counts as the self-test for changes that touch gate configuration but not the gate's own code, and the four heaviest checks run in parallel.
• No more rate limits in the dependency audit. The sandbox reads a security advisory report fetched ahead of time instead of calling a rate-limited service from every run.

Two numbers got worse: each merged PR still takes about five gate runs, and each run is a little longer. That is where the next round of work goes. September 28 is not in the comparison: we paused merges for most of that day to land a batch of gate changes, so its numbers reflect the pause, not the pipeline.

More gate runs, each a little longer: is that fine?

Mostly yes. The two numbers that went the wrong way measure effort, not waiting, and most of the extra effort is the gate doing its job.

Why each PR takes more runs. A gate result counts only for the exact code it checked, on top of the exact main it will land on. With about 20 merges a day, main moves every hour or so. Any PR open for a few hours has to catch up with main before it lands, and that new version is checked again. Agents also push fixes for what the gate finds. Both create a new version of the PR, and every version gets gated. In the week of September 21:

• New versions drove the growth. Versions gated per merged PR rose from 2.7 to 3.4.
• Reruns of identical code fell. Runs per version dropped from 1.8 to 1.5, so less of the extra work was pure repetition.
• Nearly every merged PR now goes through the gate: 92%, up from 75% the week before.

Why each run is longer. Most of the growth happened between late August and mid-September, when a run went from 8 to 25 minutes. That is when the sandbox took over checks that used to run only on a developer machine: the tests that need a real database, and the gate's own self-tests. Those local runs could take close to an hour and tied up the machine while they ran. Since then the length has held roughly steady, 25 to 28 minutes, while the product and its test suite kept growing.

Why that trade is acceptable. Sandbox runs happen in parallel and nobody sits waiting for them: agents move on to other work while the gate runs. Compute is cheap next to engineering time. What matters for delivery is how long a finished change waits to land, and that fell sharply (see the table above).

Where it is not fine. Two costs are real:

• A rerun of unchanged code is waste. Most such reruns follow a failure that had nothing to do with the change.
• Run length sets a floor. Even a one-line fix can't land faster than about half an hour plus review, and each false failure adds another half hour.

September 28 gave a clear example. A billing change we needed for a launch failed the gate three times, and none of the failures came from the change itself:

1. A database safety check refused to start on an overloaded developer machine.
2. On the same busy machine, an unrelated test's setup timed out. Run on its own, the same test passed in under a second.
3. In the sandbox, a timing-sensitive test in an unrelated area failed, then passed on a rerun of the same code: a flaky test. That rerun still went red, on a gate rule that refuses skipped tests, because the change pulled in older files that skip tests on purpose. That gap in the gate, not the change, is what kept it from landing that day; the fix went into that night's batch.

Chasing these also turned up a test that was quietly broken on main and would have failed some other unlucky PR next.

What we are doing about it:

• Rerun only the checks a change can affect: the first part has landed.
• Give database-migration changes a lighter path through the gate.
• Fix hidden broken tests as soon as they surface, rather than letting them fail someone else's PR.

The goal is fewer runs per change, not fewer checks.

What a merged PR costs

We priced every model call our agents made, from their session logs at list prices, and every sandbox gate run, and divided by the PRs merged each week:

This is a fully loaded figure: it includes every agent session that week, including planning, reviews, marketing and the work of running the pipeline itself, not just the work of writing each change. Before mid-August our session logs are incomplete, so we leave those weeks out.

Two things stand out. First, the sandbox gate is about 1% of the bill: an extra gate run costs cents. The expensive part of a false failure is not the rerun but the agent time spent investigating it, which is why removing false failures matters more than making runs cheaper. Second, cost per PR has come down from its September 7 peak while volume doubled, which is the same trend as the wait times above.

Why it matters for you

If your team is moving to agentic workflows, you will hit the same wall, usually sooner than you expect. More agents means more PRs at once, more collisions, and more chances for an agent to pass a check without meeting its intent.

We run DeepWind the way we ask you to run your delivery, so what we learned goes straight into the product:

• Briefs that agents can execute. A DeepWind brief carries the objective, the scope and the acceptance checks, so an agent knows what done means before it starts.
• Agents on your own machines. The DeepWind Agent Bridge runs brief work with your local Claude, Codex or Kimi CLI, and you watch it live from the web app.
• Free workflows for your own repo. The DeepWind Harness installs the same agent roles, brief-to-PR workflows and merge gate we use, for Claude Code and Codex.
• The loop, closed at delivery. Merged work links back to the brief and the epic it serves, so each cycle aims at what actually moved your number, not just what shipped.

The practical effect for customers: fixes and features reach you faster, and they have been through the same exact-commit review and pinned checks described above before they ship.

Free tools, and what a subscription adds

You can start without paying: the free tools give an agent-driven team structure on day one, and a subscription closes the loop across your whole portfolio.

The harness is useful on its own because the hard part of agentic delivery is structure, not tooling: clear briefs, independent review, and checks an agent can't bend. We are adding this gating work to it, including the preflight check that stops doomed runs, so every team gets it for free. A subscription closes the loop: it shows which of that work is moving your objectives, and aims the next cycle there.