Document workflows: uploads that trigger processing

Decided 12 Sep 2026 (DWF‑1…8, see decisions.md). All five steps of the plan below are built (slices 57–59): the four tools in domain/repo/doc-tools.rkt, the validator in domain/tools/jsonschema.rkt, the doc-pipeline plugin, "Run workflow…" on a document, upload triggers on the seam in repo-put! (domain/repo/triggers.rkt, /api/doc-triggers), the Automations card and the "Processed by" panel in the console, DOCX rendering, and the invoice scenario in the e2e gate — with test/doc-pipeline-tests.rkt, test/doc-triggers-tests.rkt and test/doc-pipeline-smoke.sh (which ends with an aws s3 cp firing a run). This is the first-user scenario the platform is for: a team uploads files, and a workflow processes them — extracts the data, runs inference over it, generates filled forms, translates the result. Sharing of what the pipeline produces is in document-sharing.md.*

What exists, and the one thing missing

The missing thing is small and precise: nothing runs a workflow because a document arrived. Runs start when a person posts one. And there are three places a document arrives — the console upload, the documents shim, and the S3 PUT — with no shared seam after the write.

Everything else in this design is composition.

The seam: inside repo-put!, after the version is committed

Triggers are evaluated in exactly one place: at the end of repo-put!, once the new version row exists. That covers all three entry points by construction — including S3, which is how a team uploads files at any scale (DWF‑1). The alternative, hooking each endpoint, is the same bug as the three copies of the periwinkle palette: it drifts.

A match enqueues a run. It never executes inline: the upload returns as fast as it does today, and the run is a scheduler job, so it inherits quota admission, cancellation, the org gate and durability across a restart. A team that uploads five hundred scans gets five hundred queued runs and the governor's concurrency cap, not a five-hundred-way fan-out against the local model.

Triggers

A trigger is a team-scoped subscription: when a document matching this lands, run that workflow with it.

doc_triggers(id pk, team_id, workflow_slug, enabled bool,
             match_prefix text,           -- key prefix, e.g. "inbox/invoices/"
             match_types text,            -- comma-separated content types, "" = any
             input jsonb,                 -- extra input merged into the run, e.g. a schema ref
             fire_on_derived bool default false,
             created_by, created_at)

doc_trigger_fires(trigger_id, version_id, run_id, fired_at,  -- exactly-once per version
                  pk(trigger_id, version_id))

• Fires once per version. doc_trigger_fires is the idempotency key: a re-upload of the same bytes to the same key is a new version and fires again; a retried request for the same version does not.
• Derived documents do not re-trigger unless the trigger opts in. Without this a pipeline that writes …/extracted.json back into inbox/ runs itself forever (DWF‑3).
• The run executes as the uploader. Its principal is whoever repo-put! ran as — a person in the console, or the S3 key's user with the key's scopes. So a run can only read what its uploader could read, and its outputs are owned by someone real. A trigger whose creator has broader rights than the uploader does not lend them (DWF‑2).
• Creating a trigger needs workflows:write on the team; the picked workflow must already be published there.

The run's input is the engine's typed input block:

{ "object_id": "…", "version_id": "…", "key": "inbox/invoices/2026-09-acme.pdf",
  "content_type": "application/pdf", ...trigger.input }

The step tools

Four define-tools. Being tools, they are also available to the agent and to any hand-written workflow; the pipeline below is just the shipped composition.

doc_text — already exists as repo_extract_text; returns the extracted text of (object_id, version_id). Kept as the first step so a pipeline never depends on the search index having run.

doc_extract_fields — text + a JSON schema → an object that validates against it. The model is asked for JSON; the reply is parsed, validated, and refused on mismatch — a missing required field, a string where a number was asked for, an invented key. This is the Localization Manager's lesson applied to data: an extraction that looks finished and is wrong is worse than a failed step (DWF‑5). Output is written as a derived document, <key>.extracted.json.

doc_render — a template document from the repository plus data → a new document. v1 renders Markdown and HTML templates with {{field}} substitution and a {{#each}} for line items; DOCX via a template whose paragraphs carry the same placeholders (the repository already unzips DOCX to extract text; writing one back is the same file/unzip in reverse). PDF came in slice 68 as format: "pdf" on doc_render: the Markdown or HTML template is filled exactly as before and the finished text goes through pandoc → tectonic, the two tools the developer e-book had already pinned in flake.nix and that the packaged server carries on its PATH. The toolchain is looked up at call time, so a box without it boots and renders every other format; the one PDF call fails by name. A DOCX template is refused rather than converted — LibreOffice is not a dependency this platform takes on (DWF‑6).

doc_translate — wraps translate!: a document's text (or a rendered form) into a target locale, glossary applied, written as <key>.<locale>.<ext>. Fanned out with map over a list of locales.

Every output goes through inherit (document-sharing.md): it takes the source's visibility and live grants at creation, is owned by the run's principal, and gets a repo_derivations row naming the source version, the run and the step. That row is what makes "where did this come from" a click, and what lets a trigger tell a derived document from an original.

The reference pipeline

Shipped as a plugin, doc-pipeline, in the pattern of doc-indexer:

(define-workflow process-upload
  #:name "Process an uploaded document"
  #:description "Extract text, pull structured fields against a schema, fill a form
                 template, and translate the result into the team's languages."
  (step text      (tool doc_text #:object (input object_id) #:version (input version_id)))
  (step fields    (tool doc_extract_fields #:text (out text result)
                                           #:schema (input schema)))
  (step form      (tool doc_render #:template (input template)
                                   #:data (out fields result)))
  (step translate (map #:over (input locales)
                       (tool doc_translate #:object (out form object_id) #:locale (item)))
        #:end))

schema, template and locales come from the trigger's input block, so the same workflow serves invoices and intake forms with different configuration and no new code. Each step is optional in the obvious way: a trigger with no template gets fields and translations of the source; one with no locales gets a form and stops.

The first-user proof, end to end: upload an invoice PDF to inbox/invoices/ → extracted.json with the vendor, date, line items and total → a filled purchase approval form as DOCX → the form in Japanese, Dutch and Spanish — all four sitting beside the original in the repository, each with a provenance link, each shared exactly as the original was.

Running one by hand

A "Run workflow…" action on any document runs the same pipeline for one object through the same run endpoint. It is how a pipeline is tested before a trigger automates it, and it is the same code path, so what worked by hand works on upload (DWF‑7). A per-document "Processed by" panel lists the runs and the derived documents, with status.

Failure, retry, re-run

A failing step fails the run, as the engine already does. Outputs already written stay — they are documents, and a partially processed invoice with its extracted fields is worth more than nothing. Re-running on the same version writes new versions of the derived documents (the repository versions by key), so a fixed template or a better model supersedes rather than duplicates. An unreadable document records nothing and never wedges the trigger — the indexing rule, kept.

AI steps meter ai.tokens.total through the ordinary quota, and the same over-budget stall the Localization Manager hit applies: a queue that stops is a team out of budget, not a hang. The trigger list shows the team's remaining budget for that reason.

Data shapes (backend-neutral)

doc_triggers and doc_trigger_fires above; repo_derivations from the sharing design. Nothing else. Runs, steps, jobs, quota and audit are the existing tables.

Contract (any backend implements)

DocumentPipeline:
  on_put(object, version, principal) -> [run_id…]       # the seam; enqueue only
  triggers(team) / trigger_create / trigger_update / trigger_delete
  fired(trigger, version) -> bool                        # exactly-once
  run_on(principal, object, workflow_slug, input) -> run_id     # the manual path
  derivations(object) -> [{object, version, run, step}]

Tools:
  doc_text(object, version) -> text
  doc_extract_fields(text, schema) -> object | refused
  doc_render(template_object, data) -> object_id
  doc_translate(object, locale) -> object_id

Bootstrapping plan

1. ✅ repo_derivations + inherit (slice 56); doc_extract_fields with schema validation and the refuse-on-mismatch tests, run against a scripted model AND a live one, as the Manager's drafting was.
2. ✅ doc_render for Markdown/HTML; doc_translate; the doc-pipeline plugin with process-upload; "Run workflow…" on a document. Smoke: upload → run by hand → four derived documents with provenance.
3. ✅ Triggers: the tables, the seam in repo-put!, exactly-once, the derived-document guard. Smoke: an S3 PUT fires the run — the path a team will actually use.
4. ✅ The Automations card and the "Processed by" panel; the e2e gate gains the invoice scenario end to end.
5. ✅ DOCX rendering. ✅ PDF (slice 68): doc_render {format: "pdf"} → <key>.form.pdf via pandoc + tectonic; format is a tool argument, so a workflow binds it as a literal ("with": {"format": "pdf", …}) — the four process-upload inputs are unchanged.

As built (steps 1–2)

• The model is a seam, not a dependency. current-doc-chat (a parameter, default run-chat at temperature 0) is what the tools call; the unit tests script it, test/mock-llm.rkt gained a CHAT mode (MOCK_REPLY_FILE) so the HTTP smoke is deterministic, and the same smoke runs against a live model when TELEMACHUS_MODEL_URL is set. Without a model the default refuses ("no model configured") rather than letting the uppercase-echo fallback fail every schema with a misleading message.
• The validator is a subset of JSON Schema with one deliberate difference: an object that says nothing about additionalProperties is closed — an invented key is refused — because the point of an extraction schema is to say what the data is. "additionalProperties": true opens it. Supported: type (incl. a list), properties, required, items, enum, min/max, minLength/maxLength, minItems/maxItems, pattern. Unsupported keywords are ignored, but type is always enforced, so nothing is ever accepted on the strength of a keyword the validator does not know.
• The renderer is strict too. {{field}}, {{a.b}}, {{#each items}}…{{/each}} with {{this}} / {{@index}} and the outer scope visible inside a block; HTML templates escape values. A placeholder the data cannot satisfy is an error, never a blank — a form with an empty Total that looks finished is the failure DWF‑5 exists to prevent.
• Keys read as a family: inbox/acme.pdf → inbox/acme.pdf.extracted.json, inbox/acme.pdf.form.md (extension = the template's), inbox/acme.pdf.form.ja.md (locale before the extension); a translation of the source is inbox/acme.ja.txt. A re-run writes the next version of the same key.
• A derived document is private from its first byte: repo-put! is called with the source's visibility and then repo-inherit! copies the grants and writes the derivation row — never a window where a private invoice's fields are team-visible.
• All four inputs are declared (object_id, schema, template, locales) and therefore required by the engine; "" and [] are the "skip" values (no template → the source is translated; no locales → stop after the form). The console's run form sends prefilled empties rather than dropping them.
• The fields step retries once: a schema mismatch is the model's mistake and a second reading at temperature 0 often conforms; a third would not. The refusal message names the path and the mismatch ($.total: expected number, got string).
• The AI tools meter themselves (ai.requests, ai.tokens.total, both tiers): the scheduler bills a job's tokens_used, and a workflow step's tool result is nested where it does not look. Raise the team's daily token budget before a batch, exactly as for the Localization Manager, or the queue stalls.
• A viewer can run the pipeline on a shared document, and the outputs are theirs (owned by the run's principal, DWF‑2) — but if the owner already ran it, the derived keys exist as the owner's private objects and the viewer's write is refused: the run fails cleanly instead of overwriting someone else's document.

As built (step 3 — triggers)

• The seam is a box on repo-put! (set-put-hook!), installed when domain/repo/triggers.rkt is required; the engine never depends on the repository, so there is no cycle. The hook receives the written object and #:derived? — #t, or the run id the output came from. The run id matters because the derivation row is written after repo-put! returns; at the seam it does not exist yet.
• A fire never fails an upload. doc_trigger_fires is claimed before the run starts (exactly once per version); if the run cannot start, the row keeps the reason and an audit event doc.trigger.error is written. The two reasons seen in practice: the workflow was unpublished, and the uploader's credential cannot run workflows — an S3 key issued with the default files:* scopes. Issue the key with workflows:read and workflows:run for a prefix meant to fire.
• The arrival wins on a clash with the trigger's input: a trigger can carry a schema or a template, never redirect the run at a different document.
• fire_on_derived means "other pipelines' outputs", never its own. A trigger never fires on the output of a run it started (recognized by the run id at the seam, or by the derivation rows on a re-upload). Without this an opted-in trigger whose output matches itself ran 50 times in the test before the drain limit stopped it; "the operator opted in" is no consolation at 3 a.m.
• match_types takes exact types or a family (image/*); "" is any. Fires are ordered by an epoch-millisecond column, because CURRENT_TIMESTAMP is seconds and "newest first" is a promise the history has to keep.
• The list endpoint carries the team's remaining AI budget: a trigger whose runs sit queued is a team out of tokens, not a broken trigger.

As built (steps 4–5)

• "Processed by" is GET /api/repo-obj/<id>/processing: what was derived from the document (key, step, run), what it was derived from, and every run that touched it — started by hand with it as input, started by a trigger on it, or the run that wrote it. Each row is authorized on its own; a derived document the caller cannot read is simply absent. The console panel sits under the document's versions with "View" links into the run.
• Automations is a card on the Workflows tab: the team's triggers (workflow, prefix, types, enabled), each with a history (which version, which run, or why not), enable/disable/delete for workflows:write, a create form whose input is prefilled with the example schema, and the team's remaining AI budget.
• DOCX rendering (DWF‑6): a .docx template is a zip whose word/document.xml carries the same {{placeholders}}. Two things make that harder than it sounds and docx-prepare handles both: Word splits a placeholder across runs the moment the author pauses or the spell-checker looks at it (every tag between a {{ and its }} is dropped), and line items want a table row per item (a row whose only text is {{#each items}} opens a block, a row that is only {{/each}} closes it; the rows between are repeated). Values are XML-escaped; the output is <key>.form.docx with the DOCX content type, and its translation is its text as .txt. Newlines in values, images and native tables of contents are v2.
• The e2e gate boots the scripted mock model when it boots its own server and runs the whole scenario through the console: upload a template and an invoice, "Run workflow…", wait for the run, find the derived documents, open the panel, create a trigger from the Automations card, upload into its prefix, watch it fire. On a live box the scenario runs only with VALIDATE_PIPELINE=1.

Decisions (confirmed 12 Sep 2026)