Workflow Engine — setup & test runbook

Operator-facing. How to stand the workflow engine up, prove it works, and read the failures. Design rationale lives in ../design/workflow-engine.md; this file is the runbook.

Everything below runs from refimpl/racketmaximus/.

1. What you are deploying

A workflow is a validated JSON spec, published per team, executed one step at a time. Each step becomes a row in the existing jobs table, so the engine adds no new runtime, no new daemon and no new datastore — if the server runs, workflows run.

POST /api/workflows/<slug>/run
        │
        ▼
  workflow_runs  ──advance──▶  workflow_steps  ──▶  jobs  ──▶  worker pool
   (cursor_json)                (one per step)      (existing scheduler)

Operational consequences, all inherited:

2. Prerequisites

Option A — Nix (recommended)

Nothing to install but Nix itself. The toolchain, its version, and every test dependency are pinned in flake.lock.

nix run   github:IoTone/Telemachus/dev       # run a server, no checkout needed
nix build github:IoTone/Telemachus/dev       # build + run the full unit suite
nix develop                                  # dev shell, from a checkout
nix flake check                              # unit suite + HTTP smoke, sandboxed

The /dev ref is not optional yet. A bare github:IoTone/Telemachus resolves to the repository's default branch, which is main — and main has neither the flake nor the workflow engine. It fails with path '«github:IoTone/Telemachus/<sha>»/flake.nix' does not exist. Drop the /dev only once this work is merged to main.

nix develop gives you Racket 9.2, PLTCOLLECTS already exported, plus PostgreSQL, SQLite, OpenSSL, Node and jq. It replaces the entire ritual in Option B — no brew, no PATH surgery, no exported collection paths.

The packaged server writes nothing to its own install prefix: state goes to $TELEMACHUS_DATA_DIR, defaulting to ${XDG_STATE_HOME:-$HOME/.local/state}/telemachus.

nix profile install github:IoTone/Telemachus/dev
TELEMACHUS_DATA_DIR=/var/lib/telemachus PORT=8835 telemachus-server

From a checkout, nix run . and nix build . use the working tree instead — but Nix only reads git-tracked files, so git add a new source file before building or it will be missing from the sandbox.

nix build runs raco test test/*-tests.rkt and the localization gate inside the sandbox, so a successful build is a passing test run. nix flake check adds the HTTP smoke, which binds a port and therefore cannot live in the build.

Option B — manual toolchain

# Linux (linuxbrew) / macOS (homebrew) — same formula
brew install minimal-racket
export PATH="$(brew --prefix minimal-racket)/bin:$PATH"

cd refimpl/racketmaximus
export PLTCOLLECTS="$PWD/pkgs:"     # REQUIRED, every shell — see §8
raco make server/main.rkt           # precompile; startup is slow otherwise

3. Configuration

Only the first three matter for a workflow deployment. Everything else has a working default.

The one that will burn you. With TELEMACHUS_MODEL_URL unset, the server does not error — it answers every model call with a simulated uppercase echo. A workflow with translation steps will complete successfully and return garbage. If your workflows call a model, treat a missing TELEMACHUS_MODEL_URL as a failed deployment. test/translate-chat-demo.sh refuses to start without it for exactly this reason.

Second one that will burn you. The server binds loopback by default. In a container or on a remote host it will come up healthy, log nothing unusual, and be unreachable from outside. Bind it explicitly — and put a TLS terminator in front, or set TELEMACHUS_TLS.

Reach for the narrowest address that works. TELEMACHUS_BIND=<tailnet-ip> exposes the instance to your tailnet and nothing else; 0.0.0.0 also publishes it to every other machine on the local network, including the unauthenticated POST /api/bootstrap route on a fresh database.

Third one, and it is the one that actually bites during a trial. The server reads static/index.html once, at startup, and serves it from memory. A long-lived process therefore keeps serving the console it started with — no error, no warning, just a UI missing whatever you shipped since. Any deploy that touches the UI is a restart, and "Is the console you are serving the one you built?" below is how you prove the restart took.

Migrations run automatically at startup. Workflows add 0017-workflows (three tables) and 0018-user-locale (one column). No manual step.

4. Health & readiness

curl -s localhost:8835/health
# {"kdf":"pbkdf2_sha1","multitenant":false,"ok":true,"service":"telemachus","tls":false,"version":"0.1.0"}

/health is liveness. For workflow readiness, probe the contract endpoint — it is unauthenticated by design so a probe needs no credentials:

curl -s localhost:8835/api/workflows/schema \
  | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["spec"], d["step_kinds"], d["unknown_fields"])'
# 1 ['tool:<name>', 'choice', 'map'] rejected

The full document also carries predicates, references, step_fields, limits and planned_step_kinds — everything an author or a second implementation needs in order to write a spec this build will accept.

A non-200, or a spec that is not the version your workflow documents declare, is the signal to stop the rollout.

Plugins that shipped workflows are listed on startup and at /api/plugins:

  plugin: translate-chat v0.1.0 — 2 tool(s), 1 workflow(s)

Is the console you are serving the one you built?

/health reports a static version string, so it cannot answer this. The served asset can — the console is one file, so grep the thing the process is actually handing out:

B=http://127.0.0.1:8835
diff <(curl -s $B/ | grep -o "const tabs=\[[^]]*\]") \
     <(grep -o "const tabs=\[[^]]*\]" static/index.html) \
  && echo "console is current" || echo "STALE — restart the server"

A mismatch means the running process predates your checkout. Nothing is broken; it is holding a startup-time copy. Restart it.

5. Standing it up for a trial

The engine adds no daemon and no datastore, so "deploying workflows" is just deploying the server. What follows is the whole sequence, in order.

cd refimpl/racketmaximus
export PATH="$HOME/.linuxbrew/opt/minimal-racket/bin:$PATH"   # skip under `nix develop`
export PLTCOLLECTS="$PWD/pkgs:"

export DATABASE_URL="sqlite:///$PWD/data/telemachus.db"       # absolute — see §8
export TELEMACHUS_MODEL_URL=http://127.0.0.1:11434/v1/chat/completions
export TELEMACHUS_MODEL=qwen2.5:7b
export TELEMACHUS_BIND=127.0.0.1        # or a tailnet/VPC IP to share it
export PORT=8835

raco make server/main.rkt                # precompile — startup is slow otherwise
racket server/main.rkt

Under Nix the first three lines are unnecessary and the launch is nix run github:IoTone/Telemachus/dev; the same environment variables apply.

Redeploying over a running instance

There is no reload. Stop the old process, then start the new one — and stop it by PID. pkill -f 'server/main' also matches the shell line running it, so it kills your own command (exit 144); a bare pgrep -f has the same problem and lists the wrapper shells alongside the server:

pgrep -a -x racket | grep server/main   # 920634 racket server/main.rkt
kill 920634                             # SIGTERM; in-flight runs resume from the DB
raco make server/main.rkt && racket server/main.rkt

-x matches the executable rather than the command line, which is what keeps the shell that ran the pgrep out of the answer. The plugin subprocesses (mock-mcp, the sandboxed OOP plugins) are racket too, hence the grep; they exit with the parent.

Killing mid-run is safe: run state is rows, not memory. A step whose job was running at the moment of death is the one exception — see §10.

After the restart, prove it took, on the server and in the browser:

curl -s $B/ | grep -c workflowsView     # 0 = stale console, ≥1 = current

Then hard-refresh the client (⇧⌘R / ctrl-shift-R). A cached page looks exactly like a stale server, and you will chase the wrong one.

Exposing it to a trial audience

Bind to the narrowest address that reaches your testers:

A tailnet IP is the right default for a trial: no port forwarding, no certificate, and the instance is invisible to the local network. For HTTPS at a real hostname, tailscale serve --bg 8835 fronts it with a Let's Encrypt certificate and lets you keep TELEMACHUS_BIND=127.0.0.1.

Whatever you choose, remember POST /api/bootstrap is unauthenticated on a fresh database — it creates the first operator. Bootstrap immediately after first start, before anyone else can reach the port.

Pre-flight checklist

Run through this once per trial deployment. B is the base URL — B=http://127.0.0.1:8835, or whatever you bound to; A is the Authorization: Bearer … header from §7.

curl -s $B/health                                    # "ok":true
curl -s $B/api/workflows/schema \
  | python3 -c 'import sys,json;d=json.load(sys.stdin);print(d["spec"],d["step_kinds"])'
curl -s $B/ | grep -c workflowsView                  # ≥1 → the console is current
curl -s -m 5 "$TELEMACHUS_MODEL_URL" -H 'Content-Type: application/json' \
  -d "{\"model\":\"$TELEMACHUS_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}]}" \
  | head -c 120                                      # the model actually answers

• /health is "ok":true
• /api/workflows/schema reports 1 ['tool:<name>', 'choice', 'map'] — the spec version and step kinds your documents rely on
• TELEMACHUS_MODEL_URL is set and the endpoint answers — otherwise every model step returns a simulated echo and the trial silently proves nothing
• Startup log lists the plugins you expect, with their workflow counts
• The console is current (grep -c workflowsView ≥ 1), and you hard-refreshed
• Sign in and open Workflows: the definitions you expect are listed (a plugin's workflow materializes into the team on that first listing)
• Run one end to end and watch it reach done
• Back up data/ — every in-flight run lives in that database

Turning it off

The workflows feature is per team, in the database, and defaults to on:

curl -s -X POST $B/api/features/workflows -H "$A" -d '{"enabled":false}'

The tab disappears from the console and every workflow endpoint returns 403 — except /api/workflows/schema, which describes the build rather than the instance. Existing runs are unaffected; they are already scheduler jobs. This is the kill switch for a trial that goes wrong: it needs no restart and no deploy.

6. The test tiers

Four tiers, cheapest first. Tiers 1–3 need no model and run in CI today.

Tier 1 — unit (~15 s, no server, no network)

export PLTCOLLECTS="$PWD/pkgs:"
raco test test/flow-tests.rkt        # the engine: 20 cases
raco test test/*-tests.rkt           # everything: 137 cases

Covers the validator (including that unknown fields and newer spec versions are rejected), the binding sublanguage, fan-out ordering and failure, max_steps, retries, cross-org refusal, and a run resumed by a second database connection after the first is closed — the durability claim, tested rather than asserted.

Never run raco test test/*.rkt. That glob pulls in test/mock-*.rkt, which are mock servers that block forever. Always test/*-tests.rkt.

Tier 2 — HTTP smoke (~30 s, boots a server on a temp DB)

bash test/server-smoke.sh            # or PORT=8890 bash test/server-smoke.sh
# → server-smoke: PASS

Includes the workflow endpoints end to end: publish, list, run, poll, and assert the notes the steps actually created — plus that a spec with an unknown field is refused with 400, that a member without workflows:write is refused with 403, and that a workflow honors per-team tool activation.

Tier 3 — multi-tenancy (~40 s, needs PORT and PORT+1)

bash test/multitenant-demo.sh        # 56 assertions

Not workflow-specific, but it is what proves the org gate the engine relies on.

Tier 4 — the real thing (~2 min, needs a live model)

# warm the model first; a cold pull inside the demo looks like a hang
ollama serve &
ollama run qwen2.5:7b </dev/null

export TELEMACHUS_MODEL_URL=http://127.0.0.1:11434/v1/chat/completions
export TELEMACHUS_MODEL=qwen2.5:7b
bash test/translate-chat-demo.sh

Drives the translate-chat plugin's workflow: one chat turn, a fan-out to Spanish, Dutch and Icelandic, then a second fan-out bringing each back to the user's own language — 3 steps, 9 step rows, 7 model calls. Then it switches the translation tool off mid-demo and asserts the run stops at the failing fan-out and hands the reason back without running anything downstream.

Not in CI (no model on the runner), and it exits 2 rather than run against the echo fallback if TELEMACHUS_MODEL_URL is unset.

Use qwen2.5:7b. Avoid qwen3.5 "reasoning" models — their answer lands in a reasoning field the OpenAI-compatible path does not read, so steps come back empty.

Tier 5 — the console (~2 min, browser, needs Node ≥ 18)

Drives the Workflows tab the way an operator would: lists what the team can run, runs one, watches it advance, and forces a failure. Proof that the GUI and the API agree, since the GUI is only a client of the API.

export PATH=~/.nvm/versions/node/v24.18.0/bin:$PATH   # box default node may be too old
cd test/e2e && npm ci                                  # once
BASE_URL=http://127.0.0.1:8835 node workflow-tour.mjs  # → catalog/workflow/*.png

Point BASE_URL at any running instance; it signs in as alice/s3cret and bootstraps that operator if the instance is fresh. WF_SLUG picks a different workflow. It disables translate_text for the last shot and re-enables it after.

Use plain Playwright, as the script does — the @playwright/test runner hangs in a headless sandbox with no output.

Exit codes

All four scripts run on a mktemp -d database and clean up after themselves. They touch no deployed data.

7. Verifying a deployment by hand

B=localhost:8835
TOK=$(curl -s -X POST $B/api/bootstrap -d '{"username":"ops","password":"CHANGE-ME"}' \
      | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
A="Authorization: Bearer $TOK"

# 1. the contract this build speaks
curl -s $B/api/workflows/schema

# 2. what is installed (plugin workflows appear on first listing)
curl -s $B/api/workflows -H "$A"

# 3. publish a trivial one
curl -s -X POST $B/api/workflows -H "$A" -d '{
  "spec":1,"slug":"ping","input":{"title":"string"},
  "steps":[{"id":"note","uses":"tool:create_note",
            "with":{"title":"${input.title}","body":"from a workflow"},"end":true}]}'

# 4. run it, then read the run
RID=$(curl -s -X POST $B/api/workflows/ping/run -H "$A" \
        -d '{"input":{"title":"deploy check"}}' \
      | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])')
curl -s $B/api/runs/$RID -H "$A"

A healthy run reaches "status":"done" within a second or two for tool-only steps.

In the console

The same thing without curl — sign in and open Workflows:

1. Definitions lists everything the team can run, with its version, its source (plugin:<id> for a workflow a plugin shipped, api for one that was POSTed) and its step count. A plugin's workflow appears here the first time the tab is opened — that listing is what materializes it into the team.
2. Run generates a form from the spec's own input declaration — the same declaration the server enforces — and POSTs to /api/workflows/<slug>/run.
3. The run view polls /api/runs/<id> every 1.5 s while the run is live. Each step shows its status, its retry count, and its output; a map step's children are the indented rows beneath it, one per item. Cancel is offered while the run is still live.
4. A failed run shows the failing step in red and, above the step table, the run's own error naming which step failed and why — siblings that never started stay queued rather than running on.
5. Runs on the tab's landing page is the team's history, newest first.

The tab is hidden and its endpoints return 403 when the workflows feature is switched off for the team (Admin › Features). GET /api/workflows/schema stays open either way — it describes the build, not the instance.

Endpoints

Org admins hold workflows:read and not workflows:run: running someone's workflow is reading their data by proxy, which TEN‑2a forbids.

8. Failure modes

Ordered by how often they will actually happen.

Reading a failed run

GET /api/runs/<id> is the whole diagnosis. error says which step and why; the steps array carries per-step status, attempt, error and the job_id that executed it. A fan-out child is named <map>#<i>, so to_all#0 is the first item of the to_all fan-out.

{"status":"error",
 "error":"step 'to_all#0' failed: flow: tool 'translate_text' is disabled for this team",
 "steps":[{"step_id":"chat","status":"done", …},
          {"step_id":"to_all","status":"error", …},
          {"step_id":"to_all#0","status":"error","attempt":2, …}]}

attempt: 2 means the step's retry.max was honored before it gave up. Steps after the failure are absent because they never ran — a failed step stops the run.

9. CI

The existing job already covers tiers 1–3; the unit step is a glob, so flow-tests.rkt was picked up with no CI change:

- name: Unit tests
  run: |
    export PLTCOLLECTS="$(pwd)/pkgs:"
    raco test test/*-tests.rkt      # glob — a named list silently skips new suites
- name: Server smoke
  run: bash test/server-smoke.sh
- name: Multi-tenancy demo
  run: bash test/multitenant-demo.sh

Tier 4 is deliberately absent: a runner with no model would run the echo fallback and report success. To cover it, add a job with a model service and set TELEMACHUS_MODEL_URL; without one, leave it out rather than let it pass falsely.

10. Backup & upgrade

• State is entirely in the database: workflow_defs, workflow_runs, workflow_steps, jobs. Back up the database and you have backed up every in-flight run.
• Migrations apply at startup, in order, once. Roll forward only.
• In-flight runs across an upgrade: a run references the workflow_defs row and version it started on, so republishing a workflow never changes a running one. Steps mid-flight resume from cursor_json after a restart.
• A step whose job was running when the process died stays running — the scheduler does not reap orphans. Rare, but if a run is stuck on a step whose job is running with no worker, cancel the run and start it again.
• The console is not state. It is one file read at startup, so upgrading the UI is a restart and nothing more — there is no build step, no bundle, no cache to invalidate server-side. The browser's cache is the only one you have to think about.