Migrate a live table

The table keeps changing while the backfill runs. This page is about not losing those changes.

The mechanism

prepare does two things that matter here. It installs a companion table and two triggers on your source, and it records a boundary — the point in the database’s transaction ordering that the backfill will embed the source as of.

From then on, every committed change to a source row writes a row into the companion table in the same transaction as the change. That is the guarantee: a change that committed has an event, because one transaction committed both.

consistency:
  mode: outbox
source:
  mutable: true

Both lines are required. mutable: true says the source changes; mode: outbox says how that is accounted for.

Run it

ptah inference prepare  --spec spec.yaml --db-url "$DB" --run-id "$RUN"
ptah inference backfill --spec spec.yaml --db-url "$DB" --run-id "$RUN"

The backfill embeds the source as it was at the boundary. Rows inserted after it are not in that set, and rows updated after it have vectors computed from the old text. Both are the catch-up’s work.

ptah inference catchup --spec spec.yaml --db-url "$DB" --run-id "$RUN"

Run it until it reports nothing left:

caught up to transaction 8842: 0 changed rows, 0 tombstoned

The numbers are this pass’s work, not the run’s, which is what makes the line a stop condition at all: a count that included the backfill could never reach zero.

On a busy table this takes several passes, because rows keep changing while catch-up is running. That is expected. Each pass has less to do than the last as long as your write rate is below the rate Ptah can embed.

What happens to deletions

A row deleted from the source becomes a tombstone rather than a deleted target row. The tombstone is what stops a late-arriving embedding — one already in flight when the delete happened — from recreating a row your source no longer has.

Tombstones are counted separately in the catch-up output, and coverage verification treats them as accounted for.

What happens to a row you skip

preprocessing.empty_policy: skip means a row whose model input comes out empty is recorded as a deliberate skip rather than embedded. Coverage counts it as accounted for; a rollback to the generation does not treat it as a gap.

A row that nothing ever embedded is a real gap and is reported as one. The two are one finding apart in the same layer, and Ptah keeps them distinct.

Verify with the writes still running

ptah inference verify --spec spec.yaml --db-url "$DB" --run-id "$RUN"

Verification measures a moving target, and the consistency layer is where that shows up: it refuses if the backfill has not finished or catch-up has not reached the barrier. A freshness finding on a busy table is often the last few seconds of writes — run catchup again and re-verify.

Cut over

The cutover checks the same things again at the moment it runs, so a table that moved between your verification and your approval is caught there rather than after.

ptah inference cutover --spec spec.yaml --db-url "$DB" --run-id "$RUN" \
  --approve <digest> --approver "your name" --stabilize-for 24h

Keep catching up afterwards

The generation you switched away from stops receiving changes the moment your queries stop reading it. If you want a rollback to be possible, it has to keep being caught up:

ptah inference catchup --spec previous-spec.yaml --db-url "$DB" \
  --run-id previous-run --maintain-for 1h

Put that on a schedule for the length of the window. Without it, the window elapses over a generation that drifted, and rollback refuses it.

When the triggers go away

retire removes the generation and its bookkeeping. Until then the outbox table and its triggers stay on your source table — that is the cost of the guarantee, and it is worth knowing it is not free.

The outbox belongs to the source table, not to one generation, so retiring a generation while another still reads that table leaves it in place. The question is asked about the source rather than the target, and about both halves of its name: a second generation reading the same table but writing into a different one still needs the capture, and a same-named table in another schema does not count as a reader. retire says which it did, and names the source:

generation 3e0df4e18980 is gone, with 3 vectors
  - the outbox is gone: its triggers, capture function and event table were the
    last thing Ptah had on public.articles

or, where something still needs it:

  - the outbox stays: 1 other generation(s) still read public.articles

Which mode the generation was built with is read from the specification the registry recorded for it, not from the file you pass to retire. Retiring an outbox-built generation while holding a specification that declares immutable still removes the triggers, and an immutable generation over the same source is not a reader of an outbox it was never fed by.

The same fact decides how big the table gets. catchup removes the events every usable live feeder reading that table has passed, so the table holds the backlog and not the history — but a run that is behind holds its events for everyone. If that attempt is over, abandon releases its floor position without deleting its generation or vectors. retire remains the destructive generation-level lever, and removes the shared outbox only after the last generation that owns it is gone.