Database Branching and Test Data for Ephemeral Envs
Spinning up an application per pull request is a solved problem. Giving that application a database with the right data in it, without copying production, is not.
The Unsolved Half of Preview Environments
Stateless services are cheap to duplicate. State is where the whole idea gets expensive.
The compute half of ephemeral environments has become routine. A container image, a namespace, a hostname, a teardown hook, and a pull request gets its own copy of the application. None of that is where teams get stuck. They get stuck the moment somebody asks what is in the database.
There are only three honest answers, and each has a real cost. An empty schema is instant and free, and it fails to exercise anything: no query is slow against zero rows, no migration is dangerous, and no reviewer can click through the feature. A restored production dump is realistic and is the option that creates a legal problem, a storage bill and a wait. Generated data sits between them, and is only useful to the extent that it is deterministic, because data that differs between runs turns a test suite into a random number generator.
The pressure on this has increased for a structural reason. When environments were created by people, a handful existed at a time. When they are created per pull request, and then per agent attempt, the count is set by automation rather than by headcount. A data provisioning step that takes twenty minutes and 200 GB was tolerable at five environments. At two hundred it is the constraint that decides whether the whole pattern is affordable, which is the same arithmetic described on agent fleets.
Empty Schema
Instant, free, and exercises nothing. Every query plan is trivially fast, every migration looks safe, and every UI renders an empty state. Fine for unit tests, actively misleading as a review environment.
Production Dump
The most realistic and the most expensive in every dimension that matters: restore time, storage per environment, and the fact that you have just copied personal data into an environment with weaker controls.
Generated Seed
Cheap, safe, and only as good as its determinism and its coverage. A seed that produces different rows each run makes failures unreproducible, which is the specific thing an agent cannot work around.
Why "Just Restore a Production Dump" Fails Twice
Once on the clock, once on the audit
The Operational Failure
Restore time scales with data volume, and full-copy storage scales with the number of environments times the size of the database. Both are linear in exactly the direction you do not want when environment count is driven by automation. The instinct to fix this by dumping a subset runs into a documented limitation rather than a tuning problem. PostgreSQL's own manual is explicit about what table selection does not do: "When -t is specified, pg_dump makes no attempt to dump any other database objects that the selected table(s) might depend upon. Therefore, there is no guarantee that the results of a specific-table dump can be successfully restored by themselves into a clean database."
That sentence is the reason ad-hoc subsetting scripts fail. Taking ten thousand orders out of a production database is easy; taking the customers, addresses, products and payment records those orders reference, and nothing else, is a graph traversal problem that has to respect every foreign key. This is a genuine tooling need, not a scripting exercise.
The Compliance Failure
Under the GDPR, the sharper objection is usually not data minimization but purpose limitation. Article 5(1)(b) requires that personal data be "collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes." Data collected to provide a service to a customer, then copied into a disposable preview environment to make a reviewer's screenshot look realistic, is a different purpose. Article 5(1)(c) adds that data must be "limited to what is necessary in relation to the purposes for which they are processed."
The common mitigation is misunderstood just as often. Article 4(5) defines pseudonymization as processing such that data can no longer be attributed to a subject "without the use of additional information", kept separately. Recital 26 then states plainly that pseudonymized data which could be re-attributed using that additional information "should be considered to be information on an identifiable natural person." Pseudonymized data is still personal data. Only genuinely anonymized data falls outside the regulation, and reversibility is what separates the two.
The Environment Is Where the Control Gap Lives
Production databases sit behind network policy, audit logging, break-glass access and retention rules. A preview environment usually has none of that by design, because its entire value proposition is that it is disposable and easy to reach. Copying production data into it does not just move rows; it moves them into a place where the controls that justified holding them do not apply. If your organization is in a regulated sector, the sector-specific consequences are covered on healthcare and financial services, and the general framing on compliance.
Copy-on-Write Branching
Share the pages, store only the delta, and decouple creation time from database size
The mechanism is the same one filesystems have used for decades. Neon's documentation describes a branch as "a copy-on-write clone of your data", where creating a branch does not copy anything up front but "references the same data pages as the parent database", and subsequent "writes to a branch are saved as a delta". Databricks describes its Lakebase branching the same way, as copy-on-write over a "versioned storage engine that records every change as a new version rather than overwriting existing data". Xata states it identically: "Branches use copy-on-write technology, so the data is shared with the parent instead of being fully copied."
The property this buys is the one that matters for automation: creation time stops being a function of database size. Databricks puts the claim as "A 10GB database and a 2TB database branch in the same amount of time", and states the storage consequence as "A branch that modifies 50MB of data in a 500GB database uses roughly 50MB of additional storage". These are the vendors' own claims about their own products, not independent benchmarks, and they should be treated that way. What is verifiable without trusting anybody is the architecture: if the branch shares pages with its parent, then creation cannot be proportional to size, because nothing was copied.
One consolidation is worth knowing before you evaluate these as independent options. Databricks announced its agreement to acquire Neon in May 2025, and Neon's documentation now states that Neon and Lakebase are built on the same architectural foundation. Treat them as one technology with two commercial front doors rather than two data points.
What Each Vendor Means by "Branch"
| Platform | Engine | Copy-on-write? | Does the branch carry data? | Documented limit |
|---|---|---|---|---|
| Neon | Postgres | Yes | Yes, shared pages plus a delta | Plan-dependent branch quotas, then a per-branch monthly charge |
| Databricks Lakebase | Postgres | Yes, same foundation as Neon | Yes, inherits schema and data by pointer | Idle branches scale compute to zero |
| Xata | Postgres | Yes | Yes, and it markets anonymization on the branch | Vendor claims instant creation regardless of size |
| Amazon Aurora cloning | MySQL and Postgres | Yes | Yes | Up to 15 copy-on-write clones; the 16th is a full copy |
| Supabase branching | Postgres | No | No. "New branches do not start with any data from your main project" | Schema and migrations only; data comes from a seed file |
| PlanetScale | MySQL and Postgres | No, restore-based | MySQL dev branches copy only the schema | Postgres branches are isolated databases, capped at 100 development branches |
Quoted limits and behaviors are from each vendor's own documentation. Verify them against current docs before designing around one, because this is a fast-moving category and pricing in particular changes without notice.
Supabase's Default Is a Position, Not a Gap
It is worth reading Supabase's stated reason carefully rather than filing it as a missing feature. Its documentation says new branches carry no data from the main project, and gives the rationale directly: it "is meant to better protect your sensitive production data." That is the compliance argument from the previous section expressed as a product default. A team that adopts copy-on-write branching from production is choosing convenience over that default, which is a legitimate choice, but it should be a deliberate one with an anonymization step attached rather than an accident of which vendor was picked.
The Same Trick, Self-Hosted
You do not need a managed vendor to get copy-on-write. You need a filesystem that has it.
OpenZFS has provided the underlying primitive for a long time, and its manual pages describe it in terms that map directly onto database branching: "A snapshot is a read-only copy of a file system or volume. Snapshots can be created extremely quickly, and initially consume no additional space within the pool", and a clone is "a writable volume or file system whose initial contents are the same as another dataset", where "creating a clone is nearly instantaneous, and initially consumes no additional space." Put a Postgres data directory on a ZFS dataset and a branch becomes a clone plus a new postmaster on a new port.
The open-source project that packages this specifically for Postgres is the Database Lab Engine from postgres.ai, Apache-2.0 licensed. Its documentation states that "thin cloning is fast because it is based on Copy-on-Write (CoW)", that it "employs two technologies for enabling thin cloning: ZFS (default) and LVM", and gives the illustrative figure that "cloning a 1 TiB PostgreSQL database takes just about 10 seconds." That last number is the project's own claim on its own hardware, and your storage will decide whether you see anything like it.
Two operational cautions apply to the self-hosted route regardless of which filesystem you use. First, a snapshot of a running database captures whatever was on disk at that instant, so the restored copy performs crash recovery on startup; that is normal and safe for a single dataset, but it is not automatically consistent across several datasets, which matters if WAL and data live on different pools. Second, thin clones share physical blocks, so a single origin snapshot cannot be released while any clone still depends on it. Storage that looked free at creation time is retained by the clones, and a forgotten branch pins it indefinitely.
Deterministic Seeding
Random test data is a test suite that fails differently every time
Generated seed data is the option that keeps production data out entirely, and it lives or dies on determinism. Most fixture libraries default to a fresh random source, which means the same commit produces different rows on every run, which means a failure cannot be reproduced by re-running it. That is a mild annoyance for a person who can eyeball the difference. For an agent, it is the same failure mode described on hermetic builds: an environmental non-determinism arrives disguised as a code defect, and the agent edits code to chase it.
Four properties make a seed script safe to run in an automated environment. It must be deterministic, driven by an explicit seed value rather than an ambient random source. It must be idempotent, so re-running it converges rather than duplicating. It must be separate from schema migrations, because mixing them means you cannot re-seed without re-migrating. And it must record what produced it, so a failing preview environment can be recreated exactly.
A Branch-per-PR Seed Script
The shape below is deliberately boring. Schema from migrations, reference data from the repository, volume data generated from a seed derived from the pull request number, and a row recording what was run.
#!/usr/bin/env bash
# seed-branch.sh - deterministic, idempotent seeding for one preview branch
set -euo pipefail
: "${DATABASE_URL:?DATABASE_URL is required}"
: "${PR_NUMBER:?PR_NUMBER is required}"
: "${GIT_SHA:?GIT_SHA is required}"
# 1. Schema comes from migrations. Never from a dump, so the preview
# environment exercises the same migration path production will.
migrate -database "$DATABASE_URL" -path ./migrations up
# 2. Reference data is committed alongside the schema and versioned with it.
# Written as upserts so re-running converges instead of duplicating.
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f ./seeds/reference.sql
# 3. Volume data is generated from an explicit seed. The same PR always
# produces the same rows; two PRs never produce colliding ones.
SEED="pr-${PR_NUMBER}" node ./scripts/generate-fixtures.mjs \
| psql "$DATABASE_URL" -v ON_ERROR_STOP=1
# 4. Record the provenance so a broken preview can be rebuilt exactly.
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -c \
"INSERT INTO seed_metadata (git_sha, seed_value, seeded_at)
VALUES ('${GIT_SHA}', 'pr-${PR_NUMBER}', now())
ON CONFLICT (git_sha, seed_value) DO NOTHING;"The generator is where determinism is actually won or lost. Three rules cover most of it: draw every value from a pseudo-random generator explicitly constructed from the seed, never from the language's global random source; derive primary keys from the seed too, for example as version-5 UUIDs over a fixed namespace, so foreign keys line up across runs; and take every timestamp from a fixed base instant rather than from the clock, otherwise the data drifts every night and a date-boundary bug appears to be flaky.
Seed Data Needs Its Own Tests
Seed scripts rot silently. A column becomes NOT NULL, a constraint is added, an enum gains a value, and the seed keeps working against the old shape until someone tries to create an environment on a Friday. Running the seed against a fresh schema in CI, on every migration change, converts that into a normal failing build.
Volume Is a Separate Problem From Shape
A seed with fifty rows validates behavior and tells you nothing about the query plan you will get at fifty million. Keep the two concerns apart: a small deterministic seed for correctness and review, and a separate, larger generated dataset for anything performance-related. Conflating them makes every preview environment slow for no benefit.
Subsetting and Anonymization
If you must start from real data, the transformation has to happen before it leaves the boundary
Some domains genuinely cannot be represented by generated data. Legacy schemas with undocumented invariants, decades of accumulated edge cases, and query plans that only misbehave against real distributions are all reasonable arguments for starting from production. The design rule in that case is simple and non-negotiable: the transformation runs inside the production trust boundary, and only transformed data crosses it. An anonymization step that runs after the dump has landed in the preview environment has already failed, because the untransformed copy existed.
Note also that pseudonymization is not an exit. Article 32(1)(a) of the GDPR names "the pseudonymisation and encryption of personal data" as a security measure, which is exactly what it is, and Recital 26 keeps re-identifiable data inside the regulation's scope. Consistent hashing of an email address is a reasonable control and it is not anonymization, because the mapping survives. If your goal is to take the data out of scope entirely, the transformation has to be irreversible and it has to defeat re-identification from combinations of quasi-identifiers, not just from the obvious columns.
Tooling Status
| Tool | What it does | License | Status |
|---|---|---|---|
| Greenmask | Logical dump, anonymization, synthetic generation, subsetting with referential integrity | Apache-2.0 | Actively released |
| postgresql_anonymizer | Postgres extension: static, dynamic and replica masking declared in DDL | Open source, from Dalibo | Active |
| Database Lab Engine | Thin clones over ZFS or LVM, plus branching for CI | Apache-2.0 | Active |
| Replibyte | Seeds databases from production data with subsetting and masking | GPL-3.0 | Active. Note the license differs from the rest |
| Neosync | Anonymization and synthetic data | MIT | Repository archived in August 2025 after the team was acquired. Do not adopt |
| Snaplet | Seed, Copycat and Snapshot | MIT | Company closed in 2024; tools open-sourced and now under the supabase-community organization |
Two of the six entries here are dead or acquired, which is the most useful thing this table says. This is a category with a high project mortality rate, so check the repository's last release date before you build a pipeline on any of it. Commercial options exist in the same space and are worth evaluating if the maintenance risk of an open-source dependency is the deciding factor.
Branch-per-PR Lifecycle
Creation is the easy half. Deletion is where the bill comes from.
Every team that adopts this wires up creation first, because creation is what the demo shows. Deletion gets attached to the pull-request-closed webhook, which is correct and insufficient, because webhooks are missed, pull requests are abandoned, branches are created by retries that then fail, and forks behave differently. The result is a slow accumulation of database branches that nobody owns, each holding storage its parent snapshot cannot release.
The fix is to stop treating the webhook as the mechanism and treat it as an optimization. Give every branch an expiry at creation time and run a reaper on a schedule that deletes anything past it, regardless of whether the corresponding pull request still exists. Then the webhook simply reclaims capacity sooner. Two supporting habits make the reaper safe: label each branch with its pull request, its creator and its cost center at creation, because attribution added later never gets added; and alert on the count of live branches rather than on individual leaks, since the signal you want is a trend, not an event.
Where the Cost Actually Sits
Parent database
|
+-- branch pr-4821 delta: 40 MB compute: idle, scaled to zero
+-- branch pr-4822 delta: 12 MB compute: idle, scaled to zero
\-- branch pr-4790 delta: 3.1 GB compute: idle [LEAKED]
PR closed six weeks ago. Delta grew with every migration run
against it, and it still pins the origin snapshot it descends from.Copy-on-write makes the marginal branch cheap, which is exactly why leaked branches go unnoticed. The cost is not the branch that was created yesterday; it is the one created six weeks ago whose delta has grown with every migration run against it, and which prevents the parent snapshot it descends from being released. Per-branch charges and idle compute add to that, and the general approach to attributing and controlling this kind of spend is on FinOps.
Do
- Set an expiry at creation, always
- Label with PR, owner and cost center
- Run migrations on the branch, not the parent
- Test the seed script in CI on every migration
Do Not
- Rely on a webhook as the only teardown
- Anonymize after the copy has left production
- Give preview credentials write access to production
- Assume a branch of production is exempt from policy
Watch
- Live branch count over time
- Total delta storage held by branches
- Oldest branch age
- Seed script duration, which only grows
Related
The environments this data goes into, and what they cost to run
Ephemeral Environments
The compute half of the same pattern, and the lifecycle it needs.
Hermetic Builds
The same determinism argument, applied to the code rather than the data.
Workspace Snapshots
Copy-on-write applied to the whole workspace, not just its database.
FinOps
Attributing and controlling spend that automation creates faster than people can.
CI/CD Integration
Where branch creation, seeding and teardown actually get wired up.
Agent Fleets
What happens to environment count when creation stops being a human decision.
Sandbox Environments
Isolated execution for generated code, and the data it should not reach.
Capacity Planning
Sizing the storage and compute that branch sprawl quietly consumes.
Sources and Citations
Every claim on this site with a publisher, a year and a link.