Headless AI Agents in CI

A headless agent is not a cut-down agent. It is the same tool-calling loop you drive interactively, started with a flag that says "do not wait for me." Claude Code documents this plainly: "Add the -p (or --print) flag to any claude command to run it non-interactively." OpenAI documents codex exec for the same purpose, explicitly for teams that want to "Run as part of a pipeline (CI, pre-merge checks, scheduled jobs)." AWS makes the same claim for its CLI: "Headless mode lets you run Kiro CLI as part of your CI/CD pipeline to automate code reviews, generate tests, or troubleshoot build failures - no interactive terminal required."

What changes is what happens at a decision point. Interactively, a permission prompt stops the loop and a person answers it. In a pipeline nobody is there, so every one of those decisions has to be pre-committed as configuration: which tools are allowed, how many turns are permitted, how much money may be spent, and what the process should do when it runs out of any of those.

The most common sizing mistake is treating an agent like a compiler. Between tool calls, a headless agent is doing almost nothing: it has issued an HTTPS request to a model provider and is waiting for the response. That wait dominates wall-clock time on most jobs. Buying a 16 vCPU runner so the agent "thinks faster" buys you a more expensive idle process, because the thinking happens somewhere else.

What does consume real resources is everything the agent triggers. It clones the repo, installs dependencies, runs your build, runs your test suite, maybe spins up a database container. Those are the workloads to size for, and they are the same workloads your existing CI already sizes for. If your test suite needs 8 GB of RAM, the agent job needs 8 GB of RAM plus a little headroom for the CLI process and its file cache. If your dependency install is 3 GB on disk, size the disk for that and cache it, because an agent that re-downloads the world on every turn wastes both time and your provider's patience.

Concurrency is where this gets expensive. Several agent jobs on one host will share CPU happily while they are all waiting on the network, then all wake up and run the test suite at the same moment. Size for the concurrent peak of the triggered work, not the average of the agent process.

Simon Willison named the pattern in The lethal trifecta for AI agents (June 16, 2025). The three legs are "Access to your private data", "Exposure to untrusted content", and "The ability to externally communicate in a way that could be used to steal your data". His conclusion is blunt: "If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker." Note that he does not call the third leg exfiltration - in the post he demotes that word to a parenthetical, saying he is "not confident that term is widely understood."

Now map that onto a pull-request review job. Private data: the entire repository, plus whatever is in the runner's environment. Untrusted content: the PR title, the issue body, the branch name, commit messages, dependency metadata, and the README of any transitive dependency the agent happens to open. External communication: the runner has outbound internet by default, and the agent can run curl if you let it use a shell. A stock CI agent has all three legs before you have written a line of prompt.

The controls belong on their own pages rather than duplicated here: see Prompt Injection Defense for content-boundary and egress patterns, and Agent Identity for giving the pipeline agent its own scoped principal instead of borrowing a human's. The one thing to do in the workflow file itself is narrow the tool allowlist. Claude Code exposes --allowed-tools (also spelled --allowedTools, with the docs now steering restriction toward --tools) and a --permission-mode that accepts default, acceptEdits, plan, auto, dontAsk, bypassPermissions and manual. A review job that only needs to read should be given read tools only.

An agent with no budget will keep going as long as its loop keeps producing tool calls. The nearest control is --max-budget-usd, which stops the run at a spend ceiling you choose per invocation. Set it to something a little above your observed p95 for that job rather than a round number you guessed, and revisit it when the prompt changes, because a prompt that starts asking the agent to read more files will move the distribution.

Above that sits the workflow ceiling. A per-invocation cap does nothing about a matrix job that runs 40 times, so the ceiling that matters organizationally is per workflow per day. And above both sits the control most teams skip: give the pipeline its own billing key, separate from the one your engineers use interactively. That is not bookkeeping. It means a runaway scheduled job cannot drain the budget developers rely on to get work done, and the spend graph stays legible about which of the two is growing.

Capture three things separately, because they answer different questions. The transcript - --output-format stream-json streamed to a file - answers what the agent did and in what order. The diff, saved on its own, answers what changed. The commentary, the human-readable summary, is what goes on the pull request. Merging them into one blob means that reviewing the change requires reading the reasoning, and auditing the reasoning requires scrolling past the diff.

Upload all of it with an explicit retention window long enough to cover the gap between a change shipping and someone asking about it. Then treat those artifacts as repository content, because that is what they are: a transcript from a private repo quotes private source at length, and it inherits that repository's confidentiality classification. If the repo is not world-readable, its transcripts should not be either.

The last operational trap is fan-out. A matrix job running an agent across 30 repositories will hit your model provider's rate limits long before it hits any runner limit, and the failure looks like a flaky agent rather than a quota problem. Use a concurrency group to cap how many agent jobs run at once, stagger the start times rather than firing them all on the same cron minute, and handle HTTP 429 with exponential backoff and jitter instead of an immediate retry that arrives inside the same window. A pipeline that runs 5 agents reliably is worth more than one that launches 30 and completes 11.