Run status and findings

Reading status

$ ptah inference status --spec spec.yaml --db-url "$DB" --run-id 2026-08-articles
run 2026-08-articles: caught_up, running
  - generation: 31122cc8322d44317514ca2b54f29853f1c43d19ecd3b2a1b183320ef8f5bb37
  - scanned 48231, embedded 48200, skipped 31, deleted 0
  - 965 batches committed, 0 retries since the last one
  - 6104382 prompt tokens, 6104382 total, as the provider reported them
  - snapshot boundary: 8842
  - catch-up watermark: 9017
  - lease: worker-1, fencing token 1

Line	Means
caught_up, running	The phase reached, and the run’s state: running, paused, failed, abandoned, or complete after its generation was retired
generation	The identity this run is building
scanned / embedded / skipped / deleted	Rows read, rows given a vector, rows deliberately skipped, rows tombstoned
batches committed	How many provider round trips landed
prompt tokens	What the provider said it charged for, which is the number to compare against an invoice. Ptah counts none of its own
the provider reported no token usage	Stands in place of the line above when no answer carried a usage object. A provider that charged zero and one that said nothing both leave the counts at zero, and this is which
snapshot boundary	The point the backfill embeds the source as of
catch-up watermark	How far catch-up has read past it
snapshot_done	Whether the backfill’s walk ran off the end of the source. In the JSON only; the text form says it through the consistency finding
lease	Who holds the run, and which token may still commit

A catch-up watermark is usually a transaction identity, as above: every transaction below it is processed in full. A run stopped partway through a transaction — a page that filled before that transaction ended — records the sequence it reached as well, and reads 9017:412. Both are ordinary; the second says the next catchup resumes inside transaction 9017 rather than after it.

The phase

The phase is a high-water mark: it records the furthest point the run reached, not where it is now. Running catchup again after a verification leaves it at verified.

The order is boundary_captured, backfilling, backfilled, caught_up, indexed, verified, cut_over, and then either rolled_back or retired.

Those last two are not the same kind of end. Retirement destroys the corpus, makes every run for that generation complete, releases their leases, and fences their workers. A run that had reached cut_over or rolled_back also advances to retired; an earlier run keeps the furthest phase it truthfully reached. rolled_back is reversible before retirement, because the rollback it records is — cutting the generation over again returns the run to cut_over. A generation merely replaced by a newer cutover is neither: it keeps cut_over, because that is the furthest point its run reached, and which generation queries read now is what the pointer says.

abandoned is a terminal run status, not another phase. It means an operator ended that attempt with ptah inference abandon: the checkpoint, progress, generation, and vectors remain, but the run cannot resume and no longer holds shared outbox events. The text status adds the recorded reason:

  - abandoned: superseded by the multilingual model run

Retiring the preserved generation later destroys its vectors and changes the run status from abandoned to complete. A run that had already reached cut_over or rolled_back also advances to retired; an earlier run keeps the furthest phase it actually reached. Abandonment itself never changes the phase or destroys vectors.

backfilling and backfilled are two facts, not one worded twice. A run is at backfilling while it walks the snapshot and at backfilled once the walk reached the end.

Neither of them decides whether the snapshot is complete, and the phase cannot: a high-water mark records the furthest point reached, so a run whose backfill finished and was then given more to do — a resumed pass that failed partway — is still at backfilled with work left. status reports the answer separately, as snapshot_done, and the backfill is what writes it: the flag follows the last page its walk saw, and only the page that ran off the end of the source sets it.

Rows written after a walk has finished do not reopen it. Under outbox those belong to catch-up, which is what the barrier finding is about; under immutable they are a source that changed when the specification said it would not, and coverage reports them as rows with no vector.

Out of scope, and the keys a finding names

A target row this generation stamped, for a key the specification no longer asks for, is reported as outside the scope — it holds a vector queries can still return. A tombstone is exempt, because that is Ptah’s own record of a row leaving scope and catch-up writes one for exactly this. A tombstone that still holds a vector is not exempt: Ptah assigns the state and the vector in one statement, so that shape came from somewhere else, and it is searchable.

A composite key is printed as the tuple it is — (acme, 2) — in the specification’s key order. Internally each component is length-prefixed, so no value a column can hold decides where one component ends and the next begins; that internal form reads 6:acme1:2 and is never what you see.

A delimiter was used for this and a rare one is not a safe one: with key_fields: [tenant, id], a tenant holding the delimiter byte could produce the same identity as a different row’s, and coverage then compared one source row against the other’s target row.

skipped is not missing

A skipped row is one the specification declined to embed — an empty input under empty_policy: skip. Coverage counts it as accounted for.

A row nothing ever embedded is a gap, and it is reported as one. The two are one finding apart in the same layer, and Ptah keeps them distinct because a rollback to a corpus that is exactly what was asked for should not be refused.

The lease and the fencing token

A lease says who should be working. A fencing token says who may still commit. They are different questions, and the second is the one that matters after a worker was paused long enough for its lease to lapse and then resumed: it still believes it holds the run, and the token is what stops it.

A worker whose token is behind the run’s is refused before it touches your table. The token moves when a worker takes the run: prepare takes it, and so does every verb that does work. So a second backfill started against a run another process is already embedding fences the first at its next commit, rather than both writing.

Reading verify

$ ptah inference verify --spec spec.yaml --db-url "$DB" --run-id 2026-08-articles
generation 31122cc8...: 48231 source rows, 48200 target rows
  - [freshness/blocking] 12 target rows were computed from a source state that has since changed
      keys: 4471, 4472, 4480, ...
  - [structural/advisory] the index uses ivfflat and the generation expects hnsw
  - not measured: the stored vectors were not read back, so their dimension was
    checked and their values were not
error: verification found 1 blocking findings

The header’s two numbers are the shape the report was taken on: in-scope source rows, and positions the walk stood opposite on the target side.

A target row is not always a vector. A tombstone, a skip, and a row nothing ever wrote each occupy a position and hold none. Where the two counts differ the header names the deliberate absences:

generation 4d42572104c3...: 2 source rows, 3 target rows (2 with a vector, 1 tombstoned)

The total is what the verification record stores and what its digest binds, so it is kept rather than replaced by the vector count. The breakdown is silent on a generation where every target row holds a vector, because there is nothing to explain.

It is not a partition. A row nothing ever wrote is neither tombstoned nor skipped, so it is part of the difference and not part of the breakdown — the coverage layer reports it as a finding, and that is where to read about it. A tombstone that still holds a vector is counted in both, and is also a finding.

Each finding names its layer and its severity.

The five layers

Layer	Asks	Typical finding
structural	Does the column exist with the right type and dimension? Is the index there and valid?	the generation's vector column does not exist
coverage	Does every in-scope row have a vector, a skip, or a tombstone, and does anything else carry one?	rows with no vector and not marked skipped or deleted; N target rows are outside the generation's source scope
freshness	Was each vector computed from the source as it is now?	rows computed from a source state that has since changed
vector_validity	Are the stored vectors the shape the generation declares?	the column holds N dimensions and the generation expects M
consistency	Has the backfill finished, has catch-up reached the barrier, does the run have a consistency mode?	catch-up has not reached the barrier, so changes after the snapshot are unprocessed

A held lease is reported, not judged. Where one is live, verify says so among its unmeasured entries and names the holder — it does not raise a finding. Whether that worker could still write is not answerable from what a run records: every command claims under the one name ptah-cli, and no verb releases its lease, so a live lease is the ordinary state immediately after a backfill. A finding there would refuse the sequence these guides publish, on every healthy run.

Severity

Blocking refuses the cutover and exits non-zero. Advisory is reported and does not.

Accepting a blocking finding takes both halves. The specification says whether accepting is permitted at all:

policy:
  allow_accepted_findings: true

and the cutover names which findings, by their exact summary:

ptah inference cutover --spec spec.yaml --db-url "$DB" --run-id r1 \
  --accept-finding "3 in-scope source rows have no vector in this generation"

The specification is where the permission lives because it is reviewed; the summary cannot be, because it carries counts and keys that only exist once a report has run.

Three refusals follow from that, and each is deliberate. Every blocking finding has to be named — accepting one does not carry the others. A summary matching no blocking finding is refused rather than ignored, because an acceptance copied into a runbook outlives the finding it was written for. And the consistency decision is not reachable this way at all: accepting “changes after the snapshot are unprocessed” would be accepting a cutover onto a generation nobody claims covers the source.

What was accepted, and what was left blocking, are both in the plan digest. So an approval given for a plan that accepted one finding does not authorize a plan that accepted another.

A generation that covers nothing

A report over an empty corpus passes every layer, because there is nothing for any of them to disagree about. It says so:

generation 04e7dbf7...: 0 source rows, 0 target rows
  - [coverage/advisory] this generation covers no source rows, so every layer
    below passed over nothing: check the specification's source filter and
    schema before cutting over to it

Advisory rather than blocking, because an empty generation is not wrong. A table backfilled before its first rows arrive, and a source.filter that legitimately selects nothing yet, are both a specification doing what it says.

What the finding separates is those from the reachable mistake: a filter with a typo in it. The backfill embeds nothing, the verification reads the same nothing through the same predicate, and the two agree perfectly — so the counts in the header are the only thing that would have told you, and a pipeline reads the findings.

An environment that knows its corpus is never empty can refuse rather than read. policy.min_source_rows is the floor, and the cutover names what it found:

cutover refused:
  - this generation covers 3 source rows and this policy requires at least 5

The count is in the plan digest and on the line an approver reads, so an approval given for a plan over the whole corpus does not carry to one built over none of it. See the specification’s policy block.

What was not measured

verify reports what it did not check, not only what it did. A report saying only what it checked reads as though it checked everything.

An unmeasured check is not a blocker. It is a sentence, and the difference matters: the question six months later is exactly which of these numbers was measured.

Reading evaluate

generation b115a08fbd46e92857378a5eea6855091e76d428e522c6c180134541a1e1af4c against corpus 479e9ad227ce
  - recall 1.000, MRR 1.000, NDCG 1.000 over 3 cases
  - the index agrees with an exhaustive search on 1.000 of results, over 3 cases
  - measured under hnsw.ef_search=40,ivfflat.probes=8
  - not measured: retrieval quality was not compared against the generation being replaced, because no baseline was measured for it

That corpus holds three cases and every score is a 1.000, which is what a small corpus over a small table looks like. The shape is what to read.

In the report	Means
recall	The share of expected answers found, at the depth this case looked to
MRR	Mean reciprocal rank: how high the first correct answer came
NDCG	How well the whole ordering matches the expected one
the index agrees with an exhaustive search	How often the index returned what a scan of every row would have

The depth is the case’s own k where it names one, otherwise the corpus’s default_k, otherwise --k. The corpus wins over the flag on purpose: a run at a different depth is a different measurement, and the file is where that is written down.

The agreement figure measures the index, not the model. A low number means the index or its query parameters need attention, not that the embeddings are wrong.

measured under names the query-time settings every number above it was taken at, because a recall figure without them is not comparable to any other. A setting the session does not carry is reported as (absent) rather than filled in with a default Ptah did not read.

Where the records go

verify --publish-evidence and cutover --publish-evidence write these to an OCI registry. The verification record carries the findings whole rather than a verdict, and what was not measured; the cutover record carries the plan digest the approval bound to, the approver, and the verification it cited.

A verification is attached to its release as a referrer rather than tagged, because evidence accumulates: a generation gets one release and several verifications, and finding them by remembering a tag each is how a record goes missing.