Specification reference

The specification is a YAML file. It describes the source rows, how they become text, which model turns that text into vectors, where the vectors go, how source changes are accounted for, and what a cutover requires.

Fields marked identity are hashed into the generation identity: changing one addresses a different corpus.

Fields marked required are refused when they are absent, before any verb does work – describe included, which opens no database. A field not marked required is not thereby harmless to leave out: source.table is accepted by describe and produces a specification nothing else can run. What the mark says is narrower and worth having exactly: without these, every verb that reads the specification exits 2 and names the field. version is required in the same way, at the top of the file.

Top level

Field	Meaning
version	The specification format version. 1.
name	A human-readable name for the migration. Not the generation identity.
description	Free text.
source	Which rows are read.
preprocessing	How a row becomes text.
model	What turns text into vectors.
target	Where the vectors go.
consistency	How source changes during the run are accounted for.
policy	What a cutover requires.

source

Field	Identity	Required	Meaning
schema	yes		The schema the source table is in.
table	yes		The source table.
filter	yes		A SQL condition narrowing the rows in scope.
key_fields	yes		The columns identifying a row, in order.
input_fields	yes		The columns whose text is sent, in order.
version_strategy	yes		How a stale vector is recognized.
version_field	yes		The column the strategy reads, where it needs one.
mutable	no	yes	Whether the source changes during the run.

filter narrows both halves of the run, and the second half is worth stating. The backfill scans only rows the condition matches, and catch-up rereads a changed row through the same condition — so a row that stops matching is tombstoned on the next catch-up rather than kept, and a row that never matched is never sent to the provider even when a write to it produced an outbox event.

There is a case the condition does not reach. A row leaves scope through a column the outbox does not watch — UPDATE articles SET published = false, where published is neither a key, an input field, nor the version — and that write produces no event at all, so the vector the backfill gave it stays until something else about the row changes. Put the filter’s columns among the input fields, or expect to re-run the backfill, if a row leaving scope has to lose its vector promptly.

version_strategy accepts:

Value	Means
updated_at	A timestamp column that moves on every write. Needs version_field.
monotonic	A counter or sequence that only increases. Needs version_field.
outbox_sequence	The outbox’s own ordering is the version.
input_hash	A vector is stale when the text that produced it changed.

The strategy also decides how two versions of one row are put in order, which is what stops a late answer overwriting a newer one. updated_at compares instants, monotonic and outbox_sequence compare numbers, and input_hash records no version and orders nothing — under it a repeated answer is recognized by its input hash rather than by its age.

version_field has to hold a value the strategy can read. A monotonic column holding a timestamp, or an updated_at column holding a counter, gives versions that order nothing: a late retry then replaces a newer vector, because nothing establishes which is newer. Ptah does not guess an order for a value it cannot read — guessing is what made a shorter rendering of a later instant look stale — so this is a configuration error with a silent cost, and the pairing is worth checking when you write the specification.

preprocessing

Every field here is identity: each one changes the text that is sent. Four of them are also required, including truncate even where max_input_bytes names no cap for it to act at.

Field	Required	Meaning
separator		What joins the input fields.
prefix		Text prepended to every input. Some models expect one.
null_policy	yes	What a NULL field becomes: empty, skip, or refuse.
empty_policy	yes	What an empty input means: skip or refuse.
unicode_normalization	yes	none, nfc, nfd, nfkc, or nfkd.
collapse_whitespace		Whether runs of whitespace become one space.
max_input_bytes		The cap on one input’s size, and the size of one chunk.
truncate	yes	What happens at the cap: refuse, bytes, or chunk.
overlap_bytes		How much of the previous chunk each chunk repeats. Read only under chunk.

null_policy: skip leaves the field out of the joined text. refuse stops the run on that row. empty_policy: skip records the row as a deliberate skip, which coverage verification counts as accounted for.

truncate: chunk

The other two policies decide what to lose when an input is over the cap. chunk decides not to lose any of it: the input is split into pieces of at most max_input_bytes, every piece is embedded, and the source row holds a set of vectors instead of one.

max_input_bytes is the chunk size — there is no second key for it, because a chunk is what a provider is asked to embed and that is what the cap already names. It must be at least 16 bytes. overlap_bytes repeats the tail of each chunk at the head of the next, so a passage that straddles a boundary is in one chunk’s meaning rather than half of two; it must be smaller than the cap, and an overlap at or above it is refused rather than clamped, because it describes a split that never reaches the end of the input.

Every piece begins and ends on a rune boundary, so a chunk is never handed to a provider as invalid UTF-8.

A set of vectors cannot live in the row it was split from, so a chunking specification needs target.layout: own_table and is refused without it. See Choose a target layout, and ADR 0017 for the identity and cardinality model the storage follows.

model

Field	Identity	Required	Meaning
provider	yes		The API shape. openai-compatible is what is implemented.
endpoint_class	yes	yes	local, hosted, or gateway. Your declaration, not a measurement.
endpoint	no		The base URL. A credential in its userinfo is refused.
identifier	yes		The model name sent to the provider.
revision	yes		The provider’s immutable revision, where it has one.
requested_dimension	yes		The dimension asked for, where the provider supports asking.
reported_dimension	yes	yes	The dimension the model produces.
normalization	yes	yes	none or l2.
pooling	yes		The pooling strategy, where the provider exposes one.
credential	no		Where the credential is, never what it is.

endpoint is excluded from the identity because moving the same model behind a different address does not change the vectors. endpoint_class is included because a change of trust boundary is a change worth being a different corpus.

credential is a reference: env:PTAH_EMBED_TOKEN reads that environment variable at run time. The value is never written to the run state or to published evidence.

endpoint is held to the same rule, and it is refused rather than accepted with a warning:

spec.yaml: model.endpoint carries a credential in its userinfo, before the
"api.example.com" host; a key must not appear in project configuration, so put
it in model.credential as env:NAME or file:/path

A URL written https://user:secret@api.example.com/v1 becomes an Authorization: Basic header on every provider request, so it is a credential that reached the wire through the field with no check on it. It is refused at the document, which is before any verb reads a row or opens a connection, and the message names the host rather than the URL so that reporting the problem is not another copy of it.

target

Field	Identity	Required	Meaning
schema	yes		The schema the target table is in.
table	yes	yes	The table the vectors go on.
column	yes	yes	The vector column. Two generations need two columns.
representation	yes	yes	vector, halfvec, or sparsevec.
metric	yes	yes	cosine, l2, or inner_product.
index_method	yes		hnsw or ivfflat. Omit for no index.
index_options	no		Build options such as m and ef_construction.
layout	no		source_columns, or own_table to have Ptah create the target table. Omit for source_columns.

index_options is excluded deliberately: retuning an index trades build cost against recall over the same vectors, so it does not make a different corpus.

layout is excluded for a different reason: schema, table and column already say where a vector went, and the layout says how that relation came to exist. Two specifications naming one relation under two layouts are refused when the generation is prepared rather than admitted as two generations, so the layout discriminates nothing those three fields do not.

Omitting layout and writing layout: source_columns are the same specification. Writing layout: with nothing after it is not: it is refused, because a template that filled in nothing is the one spelling of this key that means an author tried to choose and did not. Which layout to pick is Choose a target layout.

Option values must be whole numbers, and option names lower-case identifiers. PostgreSQL takes no parameter in a WITH clause, so anything else is refused by name rather than escaped.

consistency

Field	Meaning
mode	outbox, immutable, or omitted.
paused	Whether writes are declared stopped.

Not part of the identity: how changes were captured does not change what the vectors are. See Consistency modes.

policy

Field	Meaning
require_exact_approval	A cutover needs an approval bound to the plan digest.
require_signed_approval	That approval has to be a verified signature rather than a name typed beside the digest. Needs require_exact_approval, because a signature is given over one exact plan.
require_consistency_mode	A cutover is refused when no mode is selected.
allow_accepted_findings	Whether cutover --accept-finding may name a blocking finding to proceed over.
max_plan_age	How old a plan may be when it is approved.
min_source_rows	The smallest corpus this environment cuts over to. Omitted or zero is no requirement.

min_source_rows is the one refusal here that is about the data rather than about authority. A verification over an empty corpus passes every layer, because there is nothing for any of them to disagree about — and the reachable cause is a source.filter with a typo in it rather than an empty table. The report says so as an advisory whether or not this is set, because an empty generation is not wrong: a table backfilled before its first rows arrive is a specification doing what it says. This is how an environment that knows its corpus is never empty says so.

The count the floor is compared against is in the plan, so it is in the plan digest and on the line an approver reads:

source rows: 48231

An approval given for a plan over the whole corpus therefore stops matching one built over none of it.

A complete example

version: 1
name: articles
source:
  schema: public
  table: articles
  filter: "published = true"
  key_fields: [id]
  input_fields: [title, body]
  version_strategy: updated_at
  version_field: updated_at
  mutable: true
preprocessing:
  separator: "\n"
  null_policy: empty
  empty_policy: skip
  unicode_normalization: nfc
  collapse_whitespace: true
  max_input_bytes: 8000
  truncate: refuse
model:
  provider: openai-compatible
  endpoint_class: hosted
  endpoint: https://api.example.com/v1
  identifier: text-embedding-3-small
  revision: "2024-02"
  reported_dimension: 1536
  normalization: none
  credential: env:PTAH_EMBED_TOKEN
target:
  schema: public
  table: articles
  column: embedding_v2
  representation: vector
  metric: cosine
  index_method: hnsw
  index_options:
    m: "16"
    ef_construction: "64"
consistency:
  mode: outbox
policy:
  require_exact_approval: true
  require_consistency_mode: true

Checking one

ptah inference plan resolves the specification against a live database and reports what it could not establish. It writes nothing, so it is the safe way to find out whether a specification says what you meant.