Platform runbook
This runbook is for the operator of a Provenience Site: the person who opens its hosting, deploys it, and answers its alerts. It is written for any Site. Every identifying value is a placeholder, and each Site's real values stay in its GitHub Environment variables and each platform's own settings.
How to read it
Each procedure is a numbered list. A step says what to do, then Check says what proves it worked, then If not says what to do when it did not. Do not skip a check to save time: most of these steps change something that is hard to see from the outside.
These placeholders stand for values you look up on the day. Never write the real value into a page, a commit, a pull request, or a ticket.
| Placeholder | Meaning |
|---|---|
<site> | The Site's short name. The GitHub Environment is named site-<site>. |
<project-ref> | The Site's Supabase project reference. |
<ingress domain> | The public domain Railway generates for the ingress service. |
<site url> | The address the Site web app is served from. |
<full sha> | A full 40-character commit SHA. |
The pages
- Open a Site: Supabase, Twilio, Railway, and GitHub, in order, ending with the Deposit identities.
- Set up the Site web app: Vercel, sign-in, administrators, and the first checks after a first deploy.
- Deploy and roll back: the deploy workflows, applying and rolling back migrations, and the caveats each reverse script carries.
- Turn Extract on: the typology seed and the go-live order.
- Turn MCP on: the OAuth server, token hook, and MCP service, in order, with revoking and turning it off.
- Rotate credentials: each credential, and what changing it requires.
- Monitoring and alerts: what is watched and what each alert means.
- Incident runbooks: symptoms, checks, and actions for each failure a Site can have.
Which incident runbooks exist
Seven incident runbooks are written, all on the incident runbooks page.
| Runbook |
|---|
| A Site's database down |
| LLM provider down or throttling |
| Anthropic account refusal |
| Repeated lease expiry |
| Poison payload |
| Ingestion channel silent |
| Cost runaway |
What this runbook does not cover
- Backup and restore. Supabase provides point-in-time recovery for the Site database, and restoring is an action against the vendor. The Deposit has no backup, so this runbook gives no restore procedure.
- Writing new tooling. Everything here uses scripts, workflows, and dashboards that exist.
- Customer content. The operator has no Site account and no standing access to a customer's messages or files. Nothing here asks you to read them; diagnosis uses counts, states, and timestamps.
Where the procedures come from
The scripts and workflows are the source of truth, and each page links to the ones it relies on. If a page and a script disagree, the script is right and the page is stale: fix the page.
site/db/scripts/holds the operator scripts.site/db/rollback/holds one reverse script per migration, with the same filename..github/workflows/holdsci.yml,deploy.yml, anddeploy-web.yml.- Run the local Site database covers running everything on a laptop, including the typology seed. This runbook does not repeat it.