Skip to content
PtahDocs
latest

Graphic preview

100%
Page type: how-to

Roll a release out in stages

Apply one versioned migration release to many databases with ptah migrations rollout, a canary first and each stage gated on the one before, then read every target's outcome and resume after a failure.

ptah migrations rollout applies one versioned migration release to many databases in stages: a canary first, then the stages that depend on it, each with a limit on how many targets run at once, a response to a failure, and a number of targets that must succeed before the next stage may start. Every outcome is written to a state file, and a run that failed or was interrupted continues from it.

Each target runs ptah migrations up in a process of its own. The rollout decides which target runs when; it does not change what happens inside a target’s run. The integrity check, the migration lock, the dirty-revision guard, pre-migration checks and the approved sequence apply to each target as they do when you run ptah migrations up against it by hand.

Shape Command Targets Order and failure
Single target ptah migrations up one database one run, which stops at the first failed migration
Sequential multi-target ptah-compat migrate apply --env <name> the instances of one atlas.hcl env one target at a time, stopping at the first failure
Staged rollout ptah migrations rollout run the inventory of a rollout definition stages with dependencies, concurrency, failure policy and success requirement; resumable
  • A migration release: a migration directory, ideally sealed with ptah migrations hash, or an OCI artifact published with ptah migrations push.
  • Every target database reachable from the machine that runs the rollout, with the same access ptah migrations up needs.
  • The ptah binary. Nothing else runs the rollout: no service, no account and no external orchestrator. A CI job or a scheduled task that runs the commands below is enough.

The staged rollout example runs the same workflow against five SQLite files.

A definition lists the targets, then the stages that group them:

targets:
- name: canary
url_env: CANARY_DATABASE_URL
- name: tenant-a
url_env: TENANT_A_DATABASE_URL
- name: tenant-b
url_env: TENANT_B_DATABASE_URL
expect_sequence: approvals/tenant-b.json
- name: tenant-c
url_env: TENANT_C_DATABASE_URL
- name: tenant-d
url_env: TENANT_D_DATABASE_URL
stages:
- name: canary
canary: true
targets: [canary]
- name: wave-1
depends_on: [canary]
targets: [tenant-a, tenant-b, tenant-c]
concurrency: 2
on_failure: continue
require: 2
- name: wave-2
depends_on: [wave-1]
targets: [tenant-d]

url_env names the variable that holds a target’s URL, so the file carries no credentials and can be reviewed and committed. The canary stage runs first, and every other stage has to depend on it, directly or through another stage. wave-1 runs two targets at a time, keeps going when one fails, and lets wave-2 start when at least two of its three targets succeed.

expect_sequence is an approval for one target: the migrations its run must select under its migration lock, in the format that ptah migrations up --expect-sequence reads. Approvals are per target on purpose, and so is allow_destructive. There is no setting that approves a migration for the whole rollout. Rollout definition and state lists every key.

plan validates the definition and prints the stages in the order they run, with their members in the order they start. It contacts no database and no registry:

Terminal window
ptah migrations rollout plan --definition rollout.yaml
Rollout plan sha256:05e9aaed…
Stage 1: canary (canary)
concurrency: 1, on failure: stop, requires: all (1 of 1)
- canary postgres://app:***@db-canary.internal/app (url from $CANARY_DATABASE_URL)
Stage 2: wave-1
after: canary
concurrency: 2, on failure: continue, requires: 2 (2 of 3)
- tenant-a postgres://app:***@db-a.internal/app (url from $TENANT_A_DATABASE_URL)
...

Every rule a definition breaks is reported in one run: a repeated target, a stage naming a target that is not in the inventory, a target in two stages or in none, a dependency cycle with its path, and a stage that could start before the canary succeeds. plan reads the variables url_env names, so run it where they are set, such as the job that runs the rollout, before the rollout starts.

Terminal window
ptah migrations rollout run \
--definition rollout.yaml \
--migrations-dir ./migrations \
--state-file rollout-state.json

The run binds the release before any target starts. A local directory is captured once and every target reads a private copy of that capture, so an edit to the directory during the rollout changes nothing it runs. An oci:// reference is resolved to a digest once and every target pulls that digest, so a tag moved during the rollout changes nothing either. The state file records the digest, and resume refuses content that does not match it.

Each line a target’s run writes is relayed to standard error with the target’s name in front of it, and its credentials taken out. The rollout’s own progress goes to standard output:

Stage canary: running, at most 1 at once
canary: running against postgres://app:***@db-canary.internal/app
canary: succeeded: applied 2, at version 2
Stage canary: succeeded: 1 of 1 targets succeeded, 1 required
Stage wave-1: running, at most 2 at once

The command exits 0 when every stage succeeded, and 1 when the rollout ran and a stage failed or was blocked. Exit 2 means it refused before running any target, for an invalid definition or a state file that already exists, for example.

The state file is the record. It names the plan digest, the release, the Ptah version that ran, and the outcome of every stage and target:

Terminal window
jq '.stages[] | {stage: .name, status, targets: [.targets[] | {name, status}]}' rollout-state.json

A target’s status is one of not-started, running, succeeded, no-op, failed, partial, unknown, skipped and blocked. Its outcome comes from the run document of ptah migrations up --json, which reads the target’s revision table after the run, not from the process exit status. --json prints the same state document on standard output when the run ends. Rollout definition and state says what each status means.

With on_failure: stop, the default, a failure starts no further target of the stage. Targets already running finish, and their outcomes are kept. The rest of the stage is skipped. With on_failure: continue, every target of the stage runs.

Either way, the stage succeeds only when its successes meet its require, and every stage that depends on a stage that did not succeed is blocked, with each of its targets left untouched. A stage that does not depend on the failed one still runs.

The target’s error is in the state file. Decide what it needs before resuming:

  • A refusal before anything ran, such as a stale approval, a pre-migration check, a destructive migration the lint gate stopped, or a lock held by another runner, leaves a clean revision table. Fix the cause and resume with --retry-failed.
  • A migration that failed leaves a dirty revision row in that target’s revision table. The rollout never retries it. Recover that database by hand, with ptah migrations up --allow-dirty for a verified retry or ptah migrations repair, and then resume.

The rollout does not roll anything back. Rolling a target back is ptah migrations down against that target, and it stays your decision.

Pass the same flags to resume. Add --retry-failed only to run a failed target again:

Terminal window
ptah migrations rollout resume \
--definition rollout.yaml \
--migrations-dir ./migrations \
--state-file rollout-state.json \
--retry-failed

The definition must resolve to the plan the run started with; a changed member, stage setting or database is refused. A rotated password is not a change. The release must hold the recorded content. To roll out a changed definition or a new release, start a new run with a new state file. Targets already at the release’s head come out no-op there.

Before anything runs, every target the run started is read back from its own revision table with ptah migrations status:

What the revision table holds What the resume does
Every migration of the release applied Counts the target as succeeded and does not run it, whatever was recorded
A dirty revision row Records failed, partial or unknown as the row says, and does not run it
Nothing readable Records unknown, and does not run it
Pending migrations after a recorded failed or partial Runs it again only with --retry-failed
Pending migrations after any other status, unknown included Runs it again

A run that is started again selects only the migrations its revision table does not record, under its migration lock, so no statement runs twice. A target the run never started is read by its own run when its turn comes. Skipped and blocked targets start once their stage may run. --retry-failed takes no environment variable: a resume retries failures only when whoever starts it asks.

Ctrl-C, or a SIGTERM from a CI runner, stops the rollout from starting further targets. On Linux and macOS each target run in progress is asked to stop once, as Ctrl-C asks ptah migrations up, and stops at a safe point. On Windows the rollout sends a run nothing: a Ctrl-C in the console reaches it directly, and otherwise it finishes. The outcomes of the runs in progress are recorded, the state file says interrupted, and the command exits with the signal status. A run that fails once the interrupt has arrived is recorded as unknown, not failed, because the interrupt reaches the run too: a pre-migration hook it stops reports a failure. resume continues from there, and reads each such target back from its revision table.

A second Ctrl-C, or a rollout killed outright, does not stop the target runs in progress: on Linux and macOS each runs in a process group of its own and finishes or stops at its own safe point unobserved. resume reads back what they left.

Two rollouts on one state file at a time are refused: the first holds a lock file beside the state, rollout-state.json.lock. A process killed outright leaves the lock behind, and the refusal names the file to remove once you know no rollout is running.

Each target run takes the migration lock ptah migrations up takes, named ptah_migrate unless the target chooses another name, on the server its URL names. How far stage concurrency gets depends on where that lock lives:

Engine Lock scope Targets on one server
PostgreSQL family one database Targets in different databases run in parallel. Targets in one database, such as one schema per tenant, take the same lock and run one after another, unless each target sets its own lock name.
MySQL and MariaDB the whole server GET_LOCK names are server-wide, so every target on one server runs one after another, whatever the stage concurrency, unless each target sets its own lock name.
SQL Server one database Targets in different databases run in parallel.
SQLite none Each file is its own target.

A target waiting for the lock waits as ptah migrations up does: without limit, or as long as --migration-lock-timeout allows, after which its run fails with a lock timeout and leaves a clean revision table. Set the timeout the way you set it for a single run, in the target’s project config or with PTAH_MIGRATION_LOCK_TIMEOUT, and size concurrency to the number of simultaneous schema changes the server can carry. A name set with migration.migration_lock_name in the target’s project config moves that target onto its own lock. That is safe only for targets that are independent histories; see Choose the lock name. Every target’s run inherits the rollout’s environment, so PTAH_MIGRATION_LOCK_NAME gives all of them one name and separates none of them.

Two targets whose URLs name one database with the same connection options and config are refused as duplicates. The comparison reads the URL as written after cleaning its path and the case of its host, so two spellings of one database, such as postgres:// and postgresql:// or a default port written out, are not caught; their runs then take one migration lock in turn. A different search_path, or a config that names another revision table, makes them two targets.

A target’s run starts with the rollout’s environment and working directory, so ./ptah.yaml, PTAH_LOCK_TIMEOUT and the other settings of ptah migrations up reach every target. The rollout sets the database URL, --migrations-dir, --json and text logs itself, and adds a target’s config, expect_sequence and allow_destructive from the definition. The variables the other targets’ url_env name are taken out of a run’s environment, so a run, and any hook it starts, holds only its own target’s URL. PTAH_SERVER_DETAILS=1 reaches every run the same way, so each failed target’s server details, which may contain application data, are relayed to standard error with the rest of its output.

A variable that would waive a target’s guard, or change what its run does, for every target at once refuses the rollout instead: PTAH_ALLOW_DIRTY, PTAH_SKIP_CHECKS, PTAH_ALLOW_DESTRUCTIVE, PTAH_EXPECT_SEQUENCE, PTAH_DRY_RUN, PTAH_LIMIT, PTAH_TO_VERSION and PTAH_EXEC_ORDER. Unset it, and set the per-target key where one target needs it.

Area Covered
Engines Every engine ptah migrations up applies to, since each target runs it.
Releases A local migration directory in any format ptah migrations up reads, or an oci:// artifact.
Component versions One ptah binary runs the rollout and every target of an invocation; the state file records its version in ptah_version, and a resume records the binary that ran it.
Kubernetes Ptah Operator reconciles each database resource on its own and is not involved; run the rollout from a Job or a CI step.
  • Only versioned migrations are rolled out. Direct schema changes are not: a schema plan is computed for one database, and an approval of it authorizes nothing on another.
  • A rollout never rolls a fleet back and never retries a dirty revision row.
  • A local release holds at most 1000 files, the limit of the content digest the run binds to. Publish a larger directory with ptah migrations push and roll out the artifact.
  • Stages run one at a time. Two stages that depend on the same canary run in the order the definition declares them, not side by side.
  • The rollout passes no --provider-config or --provider-engine to a target’s run, so a target that needs either is not covered. A provider installed with ptah providers install serves a target from its URL, as it serves ptah migrations up.