Deploy and roll back
Every green commit on main deploys. The order of a migration and the code that calls it is the part that goes wrong, so it comes first.
Migrations and code, in order
Apply a migration to the Site's hosted database before you deploy any code that calls it. The Site web app's access gate calls api.current_administrator() on every route and answers 503 without it. The Review Queue, the project page, and the worker's roster backfill each call functions their own migrations add. Migrations apply in filename order.
The MCP service calls api.current_principal() on every request. Apply 20261112120000_current_principal.sql to the hosted Site before deploying the MCP service that uses it; authenticated requests fail without it. Applying the migration before the code is safe: it adds a read-only function and changes no stored data.
- Push the migration first. From
site/db, runsupabase db push, thensupabase migration list.- Check: the new file shows as applied remotely.
- If not: read the push error. Do not edit an applied migration; write a new one.
- Then merge the code. A green
CIrun onmainstarts the deploys.
Do this for every Site, because each Site has its own database.
Deploy the backend
The Deploy workflow builds the backend image once per commit, pushes it to GitHub's container registry tagged with the full commit SHA, then points each Site's ingress and worker services at that tag and its mcp service when that Site configures one.
- Let it run. A push to
mainrunsCI, and a greenCIstartsDeploy. Runs go one at a time, oldest first, so an older commit never deploys over a newer one.- Check: the run summary lists the commit, the image, and its digest, and each Site's deploy job finishes. The deploy waits for Railway to report success and then checks ingress's
/healthand, when configured, MCP's/health. - If not: the job's exit code says where it stopped: 2 a missing setting or bad tag, 3 updating the service, 4 deploying, 5 waiting, 6 the health check.
- Check: the run summary lists the commit, the image, and its digest, and each Site's deploy job finishes. The deploy waits for Railway to report success and then checks ingress's
- Check the worker yourself. The worker has no healthcheck, so a green deploy proves only that Railway started it. Open the worker's logs a few minutes later.
- Check: it is still running and has logged no error. A worker that exits with a Deposit sign-in error has a stale credential; see Rotate credentials.
Service settings (start command, healthcheck path, replicas, restart policy) come from site/backend/deploy/services.json, not the dashboard. A dashboard edit is overwritten on the next deploy. A rollback applies today's manifest to an older image.
Workflow: deploy.yml. Script: railway_deploy.py.
Deploy the web app
Deploy web builds the Site web app once per Site for each green main commit, because the build inlines that Site's public settings, and uploads it to the Site's Vercel project.
- Check: the run finishes, and the Site loads at
<site url>with a working sign-in. - If not: its exit codes are 2 for a missing or wrong setting, 3 pull, 4 build, 5 deploy.
Workflow: deploy-web.yml.
Roll back the backend
- Find the last good full SHA. Use a commit whose
Deployrun succeeded and whose image exists in the registry. - Dispatch
Deployfrommainwith that SHA asimage_tag. It must be the full 40-character lowercase SHA. No image is rebuilt.- Check: the run summary shows
build: falseand the SHA you gave, and the health check passes. - If not: exit 2 means the tag is not a full SHA. A dispatch from any branch other than
mainis refused by the Site's GitHub Environment.
- Check: the run summary shows
- Check the worker as above.
Roll back the web app
There is no rollback tag for the web app. Vercel keeps earlier deployments: promote the last good one from the Vercel dashboard, or dispatch Deploy web from main after reverting the bad commit.
- Check: the Site loads and you can sign in.
Roll back a migration
Roll back in reverse: code first, then the migration.
- Roll the code back. Web: promote the last good deployment. Backend: dispatch
Deploywith the last good SHA.- Check: neither is running code that calls the migration's functions.
- Run the migration's reverse script from
site/db/rollback/withpsql, against the Site's operator connection string. It has the same filename as the migration it undoes. Nothing runs reverse scripts automatically, and the Supabase CLI's ownmigration downis not a substitute: it resets and replays the database, discarding every row written since.- Check: the objects the migration added are gone from the database itself. Do not rely on
supabase migration listhere, because it reads the CLI's history and not the reverse script's effect. Then confirm the Site's health endpoint still reportsdb: ok.
- Check: the objects the migration added are gone from the database itself. Do not rely on
- For anything beyond the latest migration, use point-in-time recovery instead. It is the ordinary path, and it is an action you take against Supabase, not a script.
Read the reverse script's header first
Before you run any reverse script, read the comment at the top of that exact file. It says whether the script is safe against data written since, what it loses, and what must be rolled back first. The tables below summarize the headers as they stand; the file is authoritative, and a new migration's script may say more than this page does.
Read-only migrations roll back safely. They add reads and write no data. Rolling one back breaks the screen, gate, or service that calls it, so roll that code back first.
current_administrator(the access gate stops working for everyone)current_principal(20261112120000_current_principal.sql; drops onlyapi.current_principal(). Roll the MCP deploy back first, or authenticated requests fail for every caller.)review_readsreview_unknown_number_sourcesadmin_artifact_reads
Other safe ones, with a condition.
| Migration | Condition |
|---|---|
ingress_role | Safe for data, but ingress loses its access at once. Point ingress's SITE_DATABASE_URL back at a privileged role first. |
deposit_process_identities | Safe, touches no Deposit object or Auth user. A Site rolled back here also needs the single earlier Deposit Auth user and the earlier backend release; the script creates and deletes no Auth users. |
deposit_backend_auth_identity | Safe; it only recreates a role, grants, and policies. |
Reverse scripts that lose data or refuse. Each is a deliberate choice, not a routine undo.
| Migration | What reversing it does |
|---|---|
staff_role | Refuses while any active or revoked principal holds staff; reassign those principals first. Otherwise removes the role and project-people capability, helpers and policies, and restores administrator-only sender names without deleting other stored records. |
deposit_storage_identity | Refuses unless the Deposit bucket is empty, and never deletes a Deposit object. Once a Site has captured content, use point-in-time recovery. Run it inside a transaction and end it at once: it locks Storage tables, which pauses every bucket's Storage traffic until the transaction ends. |
site_level_channels | Drops the company roster, and turns every company-level artifact back into an unresolved one with no project-undetermined item behind it. |
admin_writes | Drops category descriptions. Refuses if a retired category's name was added again. |
contextualize | Drops each text's recorded parties and who filed each artifact, so a later backfill could overwrite an administrator's filing. |
queue_and_artifact_writes | Refuses while any captured artifact has a level with no ceiling, which is one set by hand or by an exclusion. Clear those first. |
roster_backfill | Drops every backfill request, pending or done. Roll the worker back first. Texts from a pending one stay unattributed and held until the person is added again. |
category_source | Drops who set each category. Corrections written meanwhile then read as model errors, so do not calibrate over them. Roll the project page back first. |
resolve_uncategorized_partial | Restores the prior whole-card category resolve. Items already split remain a closed answered item and an ordinary open continuing item; past Daybook split details remain unread. Roll the web app back first, or partial answers are refused. |
reprocess_review | Drops the record of carried work and pending category cards, while carried values, inherited corrections, and reprocessing Daybook events remain. It also leaves awaiting-review runs with proposed generations that cannot be decided. Decide every run first, and take the reprocessing worker and review page out of service before reversing it. |
Scratch databases only. The earlier schema migrations (foundation, identity, projects_roster_typology, intake_deposit, artifacts, reference_data, reconciliation, review_queue, governance_and_monitors) drop tables that hold a Site's real records: the Daybook, principals, projects and rosters, Catalog rows that address the Deposit, human corrections, confirmed reconciliations, the Review Queue, break-glass records. Never run them against a live Site.
Two more are dangerous in the other direction. lock_down_data_api reverses to the vendor's default exposure, which re-opens every table to the public key over HTTP. authorization removes every access policy and the exposed read surface, leaving a database nobody can read through the Data API. Neither has a legitimate use against a live Site.
A reverse script that refuses says so with an error. Do the clearing step it names, then run it again. Reverting Govern's code does not undo what it did: rows already governed stay governed and need deliberate reprocessing.
After any deploy
GET https://<ingress domain>/healthreturns 200 with both checksok.- When the Site has MCP configured,
GET https://<mcp domain>/healthreturns 200. - The worker is still up.
- Send a test text and see it reach
captured, if the change touched ingress or Capture.