What is a hermetic build?

Bazel defines it as a system where, given the same input source code and product configuration, the build always returns the same output by isolating the build from changes to the host system. It has two aspects: isolation, where hermetic build systems treat tools as source code and manage their storage inside managed file trees, and source identity, ensuring the sameness of inputs by hashing them. Bazel lists four benefits: speed through caching, parallel execution, multiple builds coexisting on one machine, and reproducibility. Buck2 states the complementary requirement more bluntly, that a build rule is required to correctly declare all of its inputs.

Why do hermetic builds matter for AI coding agents?

A human recognizes an environmental build failure and works around it using knowledge that was never written down. An agent cannot. The failure arrives as a compiler message, so the agent edits source code to make the message go away, burns a full run, and produces a diff that must be reviewed and rejected. Because runs are billed per attempt whether or not the attempt was doomed by the environment, a fleet running attempts in parallel multiplies every environmental defect by the fleet size. Hermeticity also makes aggressive caching safe, which is what keeps an agent's edit-build-test loop fast.

What is the Remote Execution API and why does hermeticity have to come first?

The Remote Execution API is an open gRPC specification maintained at bazelbuild/remote-apis, described by its authors as APIs that work together to enable large scale distributed execution and caching. Clients include Bazel, Buck2, Pants, Please and BuildStream; servers include Buildbarn, Buildfarm, BuildGrid and NativeLink alongside commercial offerings. Caching is keyed on an action digest computed from the declared inputs, so if an action secretly reads an undeclared file then two different situations hash to the same key and the cache returns the wrong answer. A shared cache over a non-hermetic build does not merely fail to help, it poisons results across the team, which is why sandboxing must be enforced before a remote cache is enabled.

Hermetic Builds and Reproducible Development Setups

A build that works for you and fails for an agent is not a flaky build, it is an undeclared dependency. Hermeticity, remote execution and content-addressed caching are how you find it.

The Build as a Pure Function

Same inputs, same outputs, on any machine, at any time

Bazel's documentation states the idea in one sentence: "When given the same input source code and product configuration, a hermetic build system always returns the same output by isolating the build from changes to the host system." That is a stronger claim than "the build usually works". It says the build is a function, its arguments are the declared inputs, and nothing outside that argument list is allowed to influence the result.

Bazel splits the property into two aspects. Isolation means hermetic build systems "treat tools as source code", downloading copies of them and managing "their storage and use inside managed file trees", which isolates the build from the host including its installed language versions. Source identity means ensuring "the sameness of inputs" by identifying each set of code mutations with a unique hash. Together the build becomes "self-contained as it doesn't rely on services external to the build environment", which is the honest way to state the no-network principle: a step that fetches something while it runs has an input the build system cannot see, cannot hash and cannot reason about.

The complementary requirement, that every input be declared rather than merely present, is stated most bluntly by Buck2, whose documentation says that under remote execution "it is required for a build rule to correctly declare all of its inputs" and whose landing page puts it as "Missing dependencies are errors." Sandboxing turns that requirement into enforcement; Bazel's own guidance is to "Enable strict sandboxing at the per-action level, since actions in a build can be stateful and affect the build or the output."

The payoff is not purity for its own sake. Once a build is a function of its inputs, you can hash those inputs, look up the result, and skip the work. Bazel lists four benefits, each downstream of that one guarantee: speed, because "the output of an action can be cached, and the action need not be run again unless inputs change"; parallel execution, because the system can graph every action and hash its inputs; multiple builds with different tools coexisting on one machine; and reproducibility, because "you know the exact conditions that produced the build".

Declared Inputs

Every file an action may read is listed in the build graph, and anything undeclared must be invisible to it. A build that compiles correctly can still be non-hermetic because it happens to find a header nobody declared, and it keeps compiling right up until it meets a machine without one.

Sandboxing

The sandbox turns a convention into an enforcement mechanism. Actions see only their declared inputs, so an undeclared read fails on the machine of the person who introduced it rather than quietly on someone else's laptop three weeks later.

No Network at Build Time

A step that runs a package manager is not building, it is resolving. The registry can change between runs, so one commit produces two binaries. Fetching belongs in a separate, lock-pinned phase; the compile phase runs with the network closed.

Why Agents Change the Economics

A human works around a flaky build. An agent tries to fix the code instead.

Non-hermetic builds have always been expensive, but the cost used to be absorbed quietly by people. A developer who sees a strange linker error knows to try a clean build, knows the CI machine has a different glibc, knows a colleague's checkout works. That knowledge is not written down anywhere, and it is invisible to anything that was not in the room.

A coding agent has none of it. When a build fails for an environmental reason, the failure arrives as a compiler message, and the agent does the only thing the evidence supports: it edits source code to make the message go away. The agent is not being stupid; it is being handed a signal that genuinely looks like a code defect. It will burn a full run and produce a plausible diff that must now be reviewed and rejected. Worse, runs are billed per attempt whether or not the attempt was doomed by the environment, so a fleet running attempts in parallel multiplies every environmental defect by the fleet size. A one-in-twenty flake is an annoyance for one developer and a standing tax across hundreds of concurrent sessions.

There is a second reason, specific to how agents work. An agent's edit-build-test loop is tight and repetitive, so build latency is felt on every iteration rather than once a morning. Hermeticity is the precondition for aggressive caching, and caching is what makes that loop fast. The reproducibility argument and the speed argument are the same argument.

The fleet-level cost of failed attempts is covered on agent fleets, and the slower version of the same problem, where environments diverge gradually rather than failing outright, is covered on environment drift.

What Actually Breaks Hermeticity

Almost none of these look like bugs when you introduce them

Bazel names four common causes of non-hermeticity, worth reading as a checklist: arbitrary processing in build-script fragments; "actions or tooling that create files non-deterministically, usually involving build IDs or timestamps"; "system binaries that differ across hosts (such as /usr/bin binaries, absolute paths, system C++ compilers for native C++ rules autoconfiguration)"; and "writing to the source tree during the build". The table expands that with the failures that appear once a codebase has real third-party dependencies.

Leak	How it shows up	Fix
Network fetch during a build action	Same commit, different binary, depending on the day	Move fetching to a declared, lock-pinned phase; disable network in the sandbox
Floating version ranges and mutable tags	A transitive upgrade lands with no commit to blame	Lock files with integrity hashes; pin base images by digest, not tag
Host tools found on PATH	Works on the machine that happens to have the right compiler	Registered toolchains that are themselves build inputs; strict action environment
Undeclared header or data file	Compiles locally, fails in CI or in a fresh workspace	Sandboxing, so the undeclared read fails immediately
Timestamps, hostnames and build IDs baked into artifacts	Byte-different outputs from identical inputs, so nothing caches	Stamp only where required, and keep stamped targets off the cached path
Absolute paths, locale and timezone	Cache misses between machines, and sorting differences in generated code	Relative paths, consistent execution roots, an explicitly fixed action environment
Non-deterministic code generation	Map iteration order or parallelism changes the emitted file	Sort before emit; treat generator determinism as a tested property

Reproducible Is Not the Same as Hermetic

A hermetic build is isolated from the host. A reproducible build produces byte-identical output from identical input. Hermeticity makes reproducibility achievable, but you can be hermetic and still non-reproducible if your actions embed timestamps or iterate a hash map. The distinction matters because caching depends on the second property, not just the first: a build that produces a different byte stream every time will never get a cache hit no matter how well isolated it is.

Pinning the Toolchain

The compiler is an input. Treat it like one.

The most common gap in an otherwise careful setup is that application dependencies are pinned and the tools that consume them are not. The Python packages are locked, but the interpreter comes from whatever is on the image. The Go modules are in go.sum, but the Go version is whatever the runner installed. Under a hermetic model the toolchain is a build input like any other: downloaded by the build system, verified against a hash, and reached through a registered toolchain rather than through PATH. Bazel drives that from the module file, and it also lets you pin the version of Bazel itself, which is easy to skip and worth doing.

The ground has moved here recently. The Bazel 9 release announcement states that "In Bazel 8.0, we disabled the legacy WORKSPACE system by default; and in 9.0, we've completely removed the code supporting WORKSPACE." Bzlmod is not the default dependency system on Bazel 9, it is the only one, so any tutorial telling you to pass an enable flag predates the release you are probably running.

Pinning Bazel, the Toolchains, and the Environment

Module and rule versions below are illustrative. Resolve them against the Bazel Central Registry for the release you are on, and check flag names against your Bazel version rather than copying them from any article, including this one.

# .bazelversion - pins the build tool itself, read by Bazelisk
9.2.0

# MODULE.bazel - dependencies and toolchains, resolved by Bzlmod
module(
    name = "example_service",
    version = "0.1.0",
)

bazel_dep(name = "rules_python", version = "1.0.0")
bazel_dep(name = "rules_go", version = "0.53.0")

# The interpreter is a build input, fetched and hashed by the build system.
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(python_version = "3.12.8")
use_repo(python, "python_3_12")

# Third-party Python comes from a lock file with integrity hashes,
# resolved at fetch time and never at action execution time.
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pypi",
    python_version = "3.12.8",
    requirements_lock = "//:requirements_lock.txt",
)
use_repo(pip, "pypi")

# .bazelrc - stop the host environment leaking into actions
build --incompatible_strict_action_env
build --sandbox_default_allow_network=false

# Hermeticity is the precondition; this is the payoff.
build --remote_cache=grpcs://cache.internal:443
build --experimental_guard_against_concurrent_changes

The two sandbox flags turn a stated intention into an enforced one. A strict action environment stops actions inheriting the developer's shell, and disabling network in the sandbox means a rule that quietly ran a package manager fails immediately, which is exactly the moment you want to find it.

The Container Image Is Not the Boundary

Teams often assume a devcontainer already provides hermeticity. It does not. A container image fixes the starting filesystem; it does not constrain what an action reads once it is running. A Dockerfile that installs packages at image build time produces a different image next month from the same source, and a build inside that container is still free to read anything in it, declared or not. Containers give a consistent starting point, which is genuinely valuable and covered on devcontainers and advanced devcontainers, with digest pinning on container registry. Hermeticity is the layer above: it constrains what happens after the container starts.

Content-Addressed Caching and Remote Execution

What hermeticity buys you, and the standard that makes it portable

Content-addressed caching works by hashing an action rather than naming it. The build system combines the command line, the environment, the platform requirements and the digests of every declared input into an action digest, and uses that as a key. If the key is in the cache, the outputs are fetched instead of computed. Blobs live in a content-addressable store keyed by their own hashes, so identical files are stored once no matter how many targets produce them.

This is where the two halves of the argument meet. The cache key is only correct if the declared inputs are the complete set of things that can affect the output. If an action secretly reads a file outside its declaration, two different situations hash to the same key and the cache hands back the wrong answer with total confidence. A shared cache over a non-hermetic build does not merely fail to help, it poisons results across the team. That is the strongest practical reason to enforce sandboxing before turning on a remote cache.

Remote execution goes one step further: instead of only fetching cached outputs, the build system ships the action and its inputs to a pool of workers and gets the outputs back. The protocol is the Remote Execution API, an open gRPC specification at bazelbuild/remote-apis, described by its authors as APIs that "work together to enable large scale distributed execution and caching on source code and other inputs". Because it is a shared standard rather than a vendor protocol, clients include Bazel, Buck2, Pants, Please and BuildStream, and servers span open-source implementations such as Buildbarn, Buildfarm, BuildGrid and NativeLink alongside commercial offerings. The backend is replaceable without rewriting your build.

Action Cache

Maps an action digest to the outputs that action produced. This is the layer that turns "someone already built this" into "we do not build it again", across machines and across people.

Content-Addressable Store

Stores blobs under the hash of their contents. Deduplication is automatic, integrity checking is free, and a transfer is skipped when the remote side already holds the digest.

Execution Service

Runs actions on a worker pool with a declared platform. Fan-out is bounded by the build graph rather than by the laptop, which is what lets a thin client drive a wide build.

A Note on Published Speedup Figures

Vendors here publish dramatic speedup multiples, usually undated and always measured on someone else's repository. Cache effectiveness depends on properties of your build that a case study cannot capture: graph depth, how often shared low-level targets change, whether generated code is deterministic, and whether developers and CI share an execution platform. Run your own build twice, against an empty cache and then a warm one, and measure the difference.

One figure is worth quoting because its methodology is published. BuildBuddy described adding content-defined chunking to its remote cache and reported "40% less data uploaded" and a "40% smaller disk cache", benchmarked across 50 commits of its own repository, plus roughly 300 TiB of duplicate chunk data skipped on the write path over a two-week production window (BuildBuddy, "Remote Cache CDC: Reusing Bytes", May 2026). Two qualifications: this is a vendor measuring its own product, and it is transfer deduplication, not a cache hit rate.

Bazel, Buck2 and Nix

Two build systems and a package manager, solving adjacent halves of the same problem

Bazel

The reference implementation and the one with the largest rule ecosystem. Bazel 9 is the current long-term-support line, and Bzlmod resolves modules from a central registry, removing a large class of hand-rolled repository rules that used to be a common source of non-hermeticity.

Buck2

Meta's build system, written in Rust and dual-licensed Apache-2.0 and MIT, with the same commitments and a design that pushes language logic out of the core into Starlark rules. It speaks the same Remote Execution API as Bazel. Two caveats from its own documentation: it "currently does not have a stable release tag", and it "does not sandbox local-only build steps; in contrast, Buck2 using Remote Execution is always hermetic by design."

Nix

Not a competitor. Nix is good at producing a pinned, content-addressed set of tools and system libraries; Bazel and Buck2 are good at building source against them. The natural division is Nix for the outer environment, a build system for the code.

Composing Nix With a Build System

Tweag's rules_nixpkgs exists to "use Nix and the Nixpkgs package set to import external dependencies (like system packages) into Bazel hermetically", exposing Nix-provided packages as Bazel toolchain targets. The compiler Bazel registers is then the one Nix pinned, which is appealing when the hard part of your reproducibility problem is system libraries rather than your own source graph.

On the developer-environment side, Cachix's devenv is open source and actively maintained, and gives a project a declarative, Nix-backed shell without asking every developer to learn the Nix language. Its 2.0 release, "A Fresh Interface to Nix", was published by Cachix on March 5, 2026 and replaced repeated Nix CLI invocations with an in-process backend plus an incremental evaluation cache, shipped a native process manager, and added a background shell reload. It is a local reproducibility tool that pairs with a cloud development environment rather than replacing one, and the comparison with Devbox and raw flakes lives on Nix environments. The practical caution is that adopting both at once is two learning curves and two debugging surfaces; most teams get further by making one of them the source of truth first.

The Costs, Stated Plainly

Hermeticity is not free, and pretending otherwise is how migrations stall

It Fights Dynamic Package Managers

Most language ecosystems resolve dependencies at build time by design. Making that hermetic means lock files with integrity hashes, a mirrored package source, and rules that map the ecosystem's layout into the build graph. Native extensions and install-time scripts are where it gets genuinely hard, and every ecosystem has a few.

The Build Files Become Real Code

A hermetic build graph must be maintained, reviewed and often generated. A half-migrated repository, where some targets are hermetic and some are not, gives you the costs of both models and the cache hit rate of neither.

Escape Hatches Erode It

Every build system provides a way to run an arbitrary command, and under deadline that is how the awkward step gets solved. Those exceptions stay invisible until a cache returns a stale artifact. Make them explicit, greppable and reviewed rather than ambient.

Remote Backends Are Infrastructure

A remote cache and executor pool is a production system with storage growth, eviction policy, authentication and an availability requirement, because when it is down every build is slow. Sizing guidance is on capacity planning.

An Adoption Path That Does Not Stall

Full hermeticity is a destination, not an entry requirement, and the intermediate states are worth real money on their own. A reasonable order, each step of which pays for itself before the next one starts:

Pin what you already have: lock files with hashes, base images by digest, the build tool version in the repository.
Split fetch from build, then close the network for the compile phase. What breaks is the list of your real undeclared inputs.
Turn on sandboxing before turning on a shared cache. Order matters: caching an unsandboxed build distributes wrong answers instead of catching them.
Make the toolchain an input, so the compiler version is a property of the commit rather than of the runner.
Build the same commit twice on different machines and diff the outputs. It is the only test that asserts the property you claim to have.
Only then add remote execution, the largest infrastructure commitment and the one that benefits most from the rest being true.

The rest of the reproducibility problem, and where the results get stored