tutorial

Build a minimal provider

Build an executable provider for a new engine with the provider SDK, check it with Ptah, and apply a schema through it.

Edge documents the current development build and may describe unreleased behavior. To use those commands, build Ptah from master. For an installed release, select its version in the header.

Ptah reaches a database engine it does not ship through a provider: a separate executable that Ptah starts, describes, and calls over a local connection. You build one here from an empty Go module. It serves an engine named tutorial, and at the end ptah schema apply creates a table through it.

A tutorial database is a SQLite file, so you need no database server. Where the provider calls database/sql with a SQLite driver, a provider for your engine calls that engine’s client. The finished module is examples/minimal-provider in the latest development source, and every Go block below is an excerpt of one of its files.

Prerequisites

A Go toolchain at least as new as the go line in the go.mod of step 1.
ptah on your PATH. See Install Ptah.
shasum, or another program that prints the SHA-256 digest of a file.

1. Create the module

Create an empty module and add the provider SDK and the SQLite driver:

mkdir ptah-tutorial-provider && cd ptah-tutorial-provider
go mod init example.com/ptah-tutorial-provider
go get ptah.run/provider-sdk@v0.40.0 modernc.org/sqlite@v1.61.1

The SDK is a Go module of its own, ptah.run/provider-sdk, released under tags named provider-sdk/vX.Y.Z. It imports nothing from Ptah, and your provider imports nothing else from Ptah either: the two meet only at the process boundary. The provider owns its database driver.

Keep the module path while you follow along, because the excerpts import example.com/ptah-tutorial-provider/backend. After go mod tidy in step 6, the direct requirements match the example’s:

module example.com/ptah-tutorial-provider

go 1.26.5

require (
  github.com/frankban/quicktest v1.14.6
  google.golang.org/grpc v1.86.0-dev
  google.golang.org/protobuf v1.36.12
  modernc.org/sqlite v1.61.1
  ptah.run/provider-sdk v0.40.0
)

The example’s tests add quicktest; a module without tests does not need it.

2. Describe the provider

Each of the next steps adds one file to a backend package and shows its central declarations. The complete file, with its package clause, imports and helpers, is in the example: copy it rather than assembling it from the excerpts.

Ptah’s first call to a provider is Describe. The reply says who the provider is, which engines it serves, and which services it offers. Create backend/description.go, which also declares version and identifierRules:

const (
  // ProviderID identifies the provider. A manifest entry names the same
  // value, and Ptah refuses an executable that describes another one.
  ProviderID = "example.com/ptah-tutorial-provider"
  // Engine is the engine the provider serves. It is also the scheme of the
  // addresses Open accepts: tutorial:///absolute/path/to/file.db.
  Engine = "tutorial"
  // defaultSchema is the one schema a tutorial database has.
  defaultSchema = "main"
)

// Description is what the provider tells Ptah when Ptah starts it: who it
// is, the engine it serves and the services it offers. Ptah starts a
// provider only when it offers inspection, rendering, execution and SQL
// analysis.
func Description() *protocol.Description {
  return &protocol.Description{
    Revision:         provider.Revision,
    ProviderId:       ProviderID,
    Version:          version,
    SemanticIdentity: "tutorial-1",
    Engines: []*protocol.Engine{{
      Id:            Engine,
      DefaultSchema: defaultSchema,
      Identifiers:   identifierRules(),
      // Ptah plans the tables and columns of this engine with its
      // common planner and asks the provider to render each change.
      Features: []string{provider.CommonTableIdentityFeature, provider.LocksFeature},
    }},
    Features: []string{
      provider.InspectFeature,
      provider.RenderFeature,
      provider.ExecuteFeature,
      provider.AnalyzeSQLFeature,
      // Ptah takes a lock before it applies a plan.
      provider.LocksFeature,
      // A failed call carries a reason the provider wrote.
      provider.ErrorReasonsFeature,
    },
  }
}

The fields carry the provider’s identity and its promises:

Revision is the contract’s major version. Ptah refuses a provider of another major before it opens a session.
ProviderId is a namespaced name. A manifest entry names the same value, and Ptah refuses an executable that reports another one.
SemanticIdentity names what the provider’s values and plans mean. A saved plan binds to it, so change it when the same input would mean something else.
Engines[].Id is the engine name. It is also the URL scheme a command uses to reach the engine, as in tutorial:///path/app.db, and Aliases adds more schemes.
Identifiers says how the engine compares names. Ptah never infers it.

Feature names select services. Ptah starts a provider only when it lists schema.inspect, schema.render, units.execute and sql.analyze. The engine’s schema.common-table-identity.v1 asks Ptah to plan the engine’s tables and columns with its common planner, so the provider renders each change rather than planning it. sessions.locks offers the lock that ptah schema apply takes, on the provider and on the engine. errors.reasons lets a failed call carry a reason the provider wrote.

The description sets no minimum contract, which means any host of contract 25 can start the provider. How a provider works explains contract minors and features.

3. Open sessions and read the database

Ptah opens a session for each database a command connects to. The address travels in the Open request, never on a command line, because it can carry a credential. Create backend/sessions.go. Its core is the Provider type and Open:

// Provider serves the tutorial engine. The zero value is ready to use, and
// one value serves every session of the process.
type Provider struct {
  mu       sync.Mutex
  sessions map[string]*session
}

// session is one open database. Its mutex serializes the calls that use it.
type session struct {
  mu    sync.Mutex
  db    *sql.DB
  path  string
  locks map[string]*lock
  // lost is set once a lock the session held is gone. It never clears:
  // the session runs nothing more.
  lost bool
}

// Open connects to the database file the address names and returns a session
// bound to it. SQLite creates a missing file on the first connection.
func (p *Provider) Open(ctx context.Context, request *protocol.OpenRequest) (*protocol.Session, error) {
  if request.GetEngine() != Engine {
    return nil, runtime.ErrorWithReason(codes.InvalidArgument, "the provider serves the tutorial engine only")
  }
  path, err := databasePath(request.GetAddress())
  if err != nil {
    return nil, err
  }
  db, err := sql.Open("sqlite", path)
  if err != nil {
    return nil, runtime.ErrorWithReason(codes.InvalidArgument, "the database file cannot be opened")
  }
  db.SetMaxOpenConns(1)
  var serverVersion string
  if err := db.QueryRowContext(ctx, "SELECT sqlite_version()").Scan(&serverVersion); err != nil {
    _ = db.Close()
    return nil, runtime.ErrorWithReason(codes.Unavailable, "the database file cannot be read")
  }
  id := rand.Text()
  p.mu.Lock()
  defer p.mu.Unlock()
  if p.sessions == nil {
    p.sessions = make(map[string]*session)
  }
  p.sessions[id] = &session{db: db, path: path}
  return &protocol.Session{
    Id:            id,
    Engine:        Engine,
    Database:      path,
    DefaultSchema: defaultSchema,
    ServerVersion: serverVersion,
    Identifiers:   identifierRules(),
  }, nil
}

A reason the provider writes never quotes the address. The file also holds Close, which releases a session and its locks, lookup, and databasePath, which turns tutorial:///path into a file path.

A read of the database is Inspect. The snapshot it returns is complete, or the call fails. Ptah compares what a read reports with the desired schema, so an object a read leaves out is an object Ptah plans to create. Create backend/inspect.go:

// Inspect reads every table of the database with its columns. The snapshot is
// complete or the call fails: an object the engine cannot describe, such as a
// view or a table with a CHECK constraint, fails the read rather than
// disappearing from it, because Ptah would plan to create what a read leaves
// out.
func (p *Provider) Inspect(ctx context.Context, request *protocol.InspectRequest) (*protocol.Schema, error) {
  current, err := p.lookup(request.GetSessionId())
  if err != nil {
    return nil, err
  }
  for _, schema := range request.GetSchemas() {
    if schema != defaultSchema {
      return nil, runtime.ErrorWithReason(codes.InvalidArgument, "a tutorial database has the schema main only")
    }
  }
  current.mu.Lock()
  defer current.mu.Unlock()
  if err := refuseOtherObjects(ctx, current.db); err != nil {
    return nil, err
  }
  names, err := tableNames(ctx, current.db)
  if err != nil {
    return nil, err
  }
  snapshot := &protocol.Schema{
    Complete: true,
    Engine:   Engine,
    Coverage: &protocol.ModelCoverage{Representation: protocol.ValueInput_OBSERVED},
  }
  for _, name := range names {
    table, err := readTable(ctx, current, name)
    if err != nil {
      return nil, err
    }
    snapshot.Tables = append(snapshot.Tables, table)
  }
  // A read narrowed to schemas says which namespace each selector names.
  for _, selector := range request.GetSchemas() {
    snapshot.ScopeBindings = append(snapshot.ScopeBindings, &protocol.InspectionScopeBinding{
      Selector: selector,
      Namespaces: []*protocol.ModelNamespace{{
        Database: &protocol.NamespaceComponent{Source: current.path, Normalized: current.path},
        Catalog:  &protocol.NamespaceComponent{},
        Schema:   &protocol.NamespaceComponent{},
      }},
    })
  }
  return snapshot, nil
}

The helpers in the same file read the catalog. readTable proves that a read is complete: it renders the columns it read back into a CREATE TABLE statement and compares it with the definition SQLite stored, so a CHECK constraint or a default the engine does not model fails the read. A catalog query that fails reports a reason of the provider’s own and never the driver’s text, which can quote a value.

4. Render, run and analyze changes

Ptah plans a change and then asks the provider to write it. Render connects to nothing. It returns one unit for each change, in the order the changes arrive, or refuses the whole batch. Create backend/render.go:

// Render writes one SQL statement for each change Ptah planned, in the order
// the changes arrive. It connects to nothing. A change it cannot write
// refuses the whole batch, so Ptah never receives part of a plan.
func (*Provider) Render(_ context.Context, request *protocol.RenderRequest) (*protocol.RenderResult, error) {
  result := &protocol.RenderResult{Complete: true}
  for _, change := range request.GetChanges() {
    statement, err := renderChange(change)
    if err != nil {
      return nil, err
    }
    result.Units = append(result.Units, &protocol.Unit{
      ChangeId: change.GetId(),
      Body:     &protocol.Unit_Sql{Sql: statement},
    })
  }
  return result, nil
}

// renderChange writes the statement of one change.
func renderChange(change *protocol.Change) (string, error) {
  object := change.GetObject()
  if object.GetName() == "" || object.GetSchema() != "" {
    return "", refuse("a tutorial table has a name and no schema qualifier")
  }
  table := quote(object.GetName())
  switch operation := change.GetOperation().(type) {
  case *protocol.Change_CreateTable:
    return createTable(table, operation.CreateTable)
  case *protocol.Change_DropTable:
    return "DROP TABLE " + table + ";", nil
  case *protocol.Change_AddColumn:
    if operation.AddColumn.GetPrimaryKey() {
      return "", refuse("the tutorial engine cannot add a primary key column to an existing table")
    }
    column, err := columnDefinition(operation.AddColumn)
    if err != nil {
      return "", err
    }
    return "ALTER TABLE " + table + " ADD COLUMN " + column + ";", nil
  case *protocol.Change_DropColumn:
    return "ALTER TABLE " + table + " DROP COLUMN " + quote(operation.DropColumn) + ";", nil
  default:
    return "", refuse("the tutorial engine creates and drops tables and adds and drops columns, nothing else")
  }
}

// refuse fails a call with a reason Ptah shows to the operator. A reason
// names what failed and why. It never quotes a statement, a value, an address
// or a credential.
func refuse(reason string) error {
  return runtime.ErrorWithReason(codes.InvalidArgument, reason)
}

The same file writes the column definitions and quotes names. Inspect reuses createTable for its completeness check.

Execute runs the units and reports how many the database applied. Create backend/execute.go:

// Execute runs the units of one request in order and reports how many of them
// the database applied. It never runs a request again because of its ID: when
// the reply is lost, Ptah reads the database to learn what happened.
//
// Each unit is one SQLite statement in autocommit mode, so a statement the
// database refuses applied nothing, and the units before it stay applied. A
// canceled call may have stopped inside a statement, so its outcome is
// UNKNOWN.
func (p *Provider) Execute(ctx context.Context, request *protocol.ExecuteRequest) (*protocol.Execution, error) {
  current, err := p.lookup(request.GetSessionId())
  if err != nil {
    return nil, err
  }
  if request.GetTransactionId() != "" {
    return nil, runtime.ErrorWithReason(codes.InvalidArgument, "the tutorial engine offers no transactions")
  }
  current.mu.Lock()
  defer current.mu.Unlock()
  units := request.GetUnits()
  var applied uint32
  for _, unit := range units {
    statement, ok := unit.GetBody().(*protocol.Unit_Sql)
    if !ok {
      return nil, runtime.ErrorWithReason(codes.InvalidArgument, "the tutorial engine runs SQL units only")
    }
    if !current.guard(ctx) {
      return stoppedAt(applied, fmt.Sprintf("the session lost its lock before unit %d of %d", applied+1, len(units))), nil
    }
    if _, err := current.db.ExecContext(ctx, statement.Sql); err != nil {
      return failedAt(ctx, applied, len(units)), nil
    }
    applied++
  }
  return &protocol.Execution{Outcome: protocol.Execution_APPLIED, AppliedCount: applied}, nil
}

The outcome is a contract. APPLIED confirms the whole batch, NOT_APPLIED confirms that no unit applied, PARTIAL confirms a prefix, and UNKNOWN says nobody knows. The helpers failedAt and stoppedAt in the same file choose between them, and each result that is not APPLIED carries a reason that names the unit and never its text.

Ptah reads SQL of the engine through AnalyzeSQL wherever it judges a statement, such as in the data-safety check before a plan runs. Create backend/analyze.go:

// AnalyzeSQL splits each source into statements and says what each one is.
// Ptah reads SQL through it wherever it judges a statement of the engine,
// such as the data-safety check before a plan runs. Every byte of a source is
// part of a statement or white space between statements, and a statement the
// engine does not know refuses the whole request.
func (*Provider) AnalyzeSQL(ctx context.Context, request *protocol.AnalyzeSQLRequest) (*protocol.SQLAnalysis, error) {
  if request.GetEngine() != Engine {
    return nil, refuse("the provider analyzes the SQL of the tutorial engine only")
  }
  result := &protocol.SQLAnalysis{Complete: true}
  for _, source := range request.GetSources() {
    if err := ctx.Err(); err != nil {
      return nil, err
    }
    file, err := analyzeSource(source)
    if err != nil {
      return nil, err
    }
    result.Files = append(result.Files, file)
  }
  return result, nil
}

// analyzedStatement describes the statement that spans source[start:end].
// The engine knows the four statements Render writes. Each is common DDL:
// Ptah judges what a common statement does to the schema, and a DROP is
// destructive in its judgment, not in the provider's.
func analyzedStatement(source string, start, end int, words []string) (*protocol.SQLStatement, error) {
  canonical := make([]string, len(words))
  for i, word := range words {
    canonical[i] = word
    if !strings.HasPrefix(word, `"`) {
      canonical[i] = strings.ToUpper(word)
    }
  }
  if !knownStatement(canonical) {
    return nil, refuse("the tutorial engine runs CREATE TABLE, DROP TABLE and ALTER TABLE ADD COLUMN or DROP COLUMN only")
  }
  line := 1 + strings.Count(source[:start], "\n")
  return &protocol.SQLStatement{
    Start:       uint32(start), // #nosec G115 -- analyzeSource refuses a source longer than math.MaxUint32
    End:         uint32(end),   // #nosec G115 -- analyzeSource refuses a source longer than math.MaxUint32
    Line:        uint32(line),  // #nosec G115 -- a line number is at most the source length
    Words:       canonical,
    SourceWords: words,
    Canonical:   strings.Join(canonical, " "),
    Safety:      protocol.SQLStatement_COMMON_DDL,
  }, nil
}

Every byte of a source belongs to a statement or to the white space between statements. COMMON_DDL tells Ptah to judge the statement with its own rules for common DDL; it does not mean that the statement is safe. The tokenizer analyzeSource and tokenEnd complete the file.

5. Hold the apply lock

ptah schema apply takes a lock before it reads the target, so two runs never change one database at once. The provider owns how the engine excludes a second session, and Ptah owns how long it waits. Create backend/locks.go:

// lock is one lock ID of a session. Ptah takes a lock before it applies a
// plan, so two runs never change one database at once.
//
// The tutorial engine holds a lock as a write transaction on a sidecar file
// beside the database, one file per lock name. SQLite lets one connection at
// a time hold a write transaction on a file, in any process, so a second
// attempt is BUSY. The transaction never commits: rolling it back releases
// the lock, and so does the end of the provider process.
type lock struct {
  name    string
  outcome protocol.LockResult_Outcome
  db      *sql.DB
  tx      *sql.Tx
}

// TryLock makes one attempt to take a lock and does not wait. A lock ID is
// used once: a second request with the same ID gets the answer of the first.
func (p *Provider) TryLock(ctx context.Context, request *protocol.LockRequest) (*protocol.LockResult, error) {
  current, err := p.lookup(request.GetScope().GetSessionId())
  if err != nil {
    return nil, err
  }
  current.mu.Lock()
  defer current.mu.Unlock()
  record, fresh, err := current.reserve(request.GetScope())
  if err != nil {
    return nil, err
  }
  switch {
  case current.lost:
    return receipt(request, protocol.LockResult_LOST), nil
  case !fresh:
    current.verify(ctx, record)
    return receipt(request, record.outcome), nil
  }
  if err := record.acquire(ctx, current.path); err != nil {
    return nil, err
  }
  return receipt(request, record.outcome), nil
}

// guard verifies every lock the session holds. Execute calls it before each
// unit, because a statement must not run after the lock that protects it is
// gone.
func (s *session) guard(ctx context.Context) bool {
  for _, record := range s.locks {
    s.verify(ctx, record)
  }
  return !s.lost
}

A lock ID is used once, and each receipt echoes the request’s scope and request ID. The file also holds CheckLock, which proves ownership now rather than repeating an earlier answer, and ReleaseLock, which fences an ID that never took the lock. In acquire, the write transaction starts under context.WithoutCancel, because the lock outlives the call that took it. Execute calls guard before each unit, so a statement never runs after the lock that protects it is gone.

6. Serve the provider and check it

Join the description with its services in backend/server.go:

// Server joins the description with the services that implement it. Each
// service the description offers needs an implementation, and each
// implementation needs its feature in the description: runtime.Serve refuses
// a server where the two differ.
func Server() runtime.Server {
  tutorial := &Provider{}
  return runtime.Server{
    Description: Description(),
    Sessions:    tutorial,
    Inspector:   tutorial,
    Renderer:    tutorial,
    Executor:    tutorial,
    SQLAnalyzer: tutorial,
    Locks:       tutorial,
  }
}

Serve it from main.go:

func main() {
  if err := runtime.Serve(backend.Server()); err != nil {
    fmt.Fprintln(os.Stderr, err)
    os.Exit(1)
  }
}

runtime.Serve refuses a server that offers a service it does not implement, or implements one it does not offer. It reserves standard output for the transport, and Ptah discards what the provider writes to standard error. The executable runs only under Ptah: started by hand, it exits with an error.

Tidy the module and build the provider:

go mod tidy
go build -o ptah-tutorial-provider .

Ptah starts an executable provider only when a manifest names its absolute path and the SHA-256 digest of its bytes. Write providers.json for the build:

digest=$(shasum -a 256 ptah-tutorial-provider | cut -d ' ' -f 1)
cat > providers.json <<EOF
{
  "format_version": 2,
  "providers": [
    {
      "executable": "$PWD/ptah-tutorial-provider",
      "sha256": "$digest",
      "provider_id": "example.com/ptah-tutorial-provider",
      "engines": [{"id": "tutorial", "aliases": []}],
      "environment": []
    }
  ]
}
EOF

Every build has a new digest, so write the manifest again after each one. Start the provider without opening a database:

ptah providers check --provider-config providers.json --provider-engine tutorial

Expected output on standard output:

example.com/ptah-tutorial-provider 0.0.0-dev: compatible protocol revision 25 (/path/to/ptah-tutorial-provider/ptah-tutorial-provider)

Ptah verified the digest, started the process, negotiated the contract and compared the provider ID and the engine with the manifest entry. Add --json to see the features the two sides agreed on.

7. Apply a schema

Create schema.yaml:

tables:
  notes:
    columns:
      id:
        type: INTEGER
        primary: true
        not_null: true
      body:
        type: TEXT

Read the empty database. SQLite creates the file on the first connection:

export DATABASE_URL="tutorial://$PWD/app.db"
ptah schema inspect --provider-config providers.json --db-url "$DATABASE_URL"

Expected output on standard output:

// Code generated by ptah; DO NOT EDIT.
// ptah:not-described object_owner reason=not-inspected provenance=observed
// ptah:not-described role_membership reason=not-inspected provenance=observed

schema "main" {
}

The not-described lines say what the read did not inspect, so a later comparison does not treat those objects as absent. Plan the schema and save the plan:

ptah schema plan --provider-config providers.json \
  --db-url "$DATABASE_URL" --schema-file schema.yaml --output change.plan.json

Expected output on standard output:

Planned schema changes:
CREATE TABLE "notes" ("id" INTEGER NOT NULL PRIMARY KEY, "body" TEXT);
Plan saved to file://change.plan.json

Ptah compared the read with the YAML schema, planned one table creation, and asked the provider to render it. Apply the saved plan:

ptah schema apply --provider-config providers.json \
  --db-url "$DATABASE_URL" --plan change.plan.json --tx-mode none --auto-approve

Expected output on standard output:

Planned schema changes:
CREATE TABLE "notes" ("id" INTEGER NOT NULL PRIMARY KEY, "body" TEXT);
Auto-approval enabled; applying schema changes.
Schema apply completed successfully.

Ptah took the lock through TryLock, ran the statement through Execute, and released the lock. The provider offers no transactions, so the apply runs with --tx-mode none; the default mode is refused before any statement runs.

Plan the same schema again to verify that the database matches it:

ptah schema plan --provider-config providers.json \
  --db-url "$DATABASE_URL" --schema-file schema.yaml --dry-run --json

Expected output on standard output:

{
  "contract_version": 3,
  "command": "schema plan",
  "outcome": "no-changes"
}

A declaration the engine cannot hold is refused with the provider’s reason. Change the type of body in schema.yaml to VARCHAR(80) and plan again:

error: compare database schema: provider render failed (InvalidArgument): the tutorial engine stores the column types INTEGER, TEXT, REAL and BLOB only

The text after the code is the reason refuse wrote. Without errors.reasons, Ptah prints the code alone.

Clean up

Leave the module and remove it:

cd ..
rm -rf ptah-tutorial-provider

Next steps

To decide which services your engine needs next, read How a provider works.
To prove each service before Ptah calls it, follow Test a provider.
To publish builds that ptah providers install and a manifest can trust, follow Package and release a provider.