Rotate credentials
Rotate a credential when it may have been exposed, when someone who held it leaves, or before it expires. Each one below says where it lives, what to do, and what a service that keeps the old value will do. Always set the new value everywhere it lives before you retire the old one, unless a step says the old one must stop working first.
Never paste a credential into a page, a ticket, a commit, or a pull request. Put it only in the setting this page names.
| Credential | Lives in | Needs a redeploy |
|---|---|---|
| Deposit identity passwords | Railway SITE_DEPOSIT_AUTH_PASSWORD, one per service | That service |
site_ingress database password | Railway ingress SITE_DATABASE_URL | ingress |
| Worker database password | Railway worker SITE_DATABASE_URL | worker |
| Twilio auth token | Railway ingress TWILIO_AUTH_TOKENS | ingress |
| Twilio media API key | Railway worker TWILIO_API_KEY_SID and TWILIO_API_KEY_SECRET | worker |
| Anthropic API key | Railway worker ANTHROPIC_API_KEY | worker |
| Container registry credential | Registry setting of every Railway service the Site runs (ingress, worker, and mcp when the Site runs it) | Each, on its next deploy |
| Railway project token | GitHub Environment secret RAILWAY_PROJECT_TOKEN | Nothing; used by the next deploy |
| Vercel token | GitHub Environment secret VERCEL_TOKEN | Nothing; used by the next deploy |
| Service-role key | Operator's shell, for one run | Nothing; no service holds it |
A Site that runs the mcp service holds no secret there: it forwards each caller's own token, so rotating a secret never touches it. Only the registry credential and the Supabase publishable key do.
Deposit identities
The ingress and worker services each sign in to Supabase Auth as their own Deposit user. Ingress may only add to the Deposit; the worker may add and read.
-
Rotate one identity from
site/db, with the service-role key set for this run only:$env:SUPABASE_URL = "https://<project-ref>.supabase.co" $env:SUPABASE_SERVICE_ROLE_KEY = "<service-role key>" uv run python scripts/provision_deposit_identities.py --rotate ingressUse
--rotate workerfor the worker. The script sets a new password, prints it once, and signs that user out everywhere by revoking every refresh token it has.- Check: it prints one
<email> <password>line and exits 0.
- Check: it prints one
-
Set the new password in that service's
SITE_DEPOSIT_AUTH_PASSWORDon Railway, and redeploy that service.- Check: for ingress,
/healthreportsdeposit_auth: ok. For the worker, it stays up and a test text reachescaptured.
- Check: for ingress,
What happens if you skip step 2. The old password is refused the next time the service has to sign in. A worker that meets the refused credential during Capture or Extract logs one error and exits, releasing the row it held without processing it and without failing it. Railway shows the worker as crashed once its restart limit is used up, and rows stay received or captured until you update the variable and redeploy. An ingress that is refused when it signs in reports deposit_auth: error on /health and returns 503, so Twilio retries rather than losing the message.
What stays valid. An access token already issued stays valid until it expires, up to an hour by default, even after the sign-out. Rotation is not an instant cutoff for a token already in someone's hands.
Script: provision_deposit_identities.py.
Database passwords
site_ingress. Set a new password for the role in the Supabase SQL editor, then update the password in ingress'sSITE_DATABASE_URL, percent-encoded, through the session pooler, and redeployingress.- Check:
/healthreportsdb: ok. - What happens in the gap. Between the change and the redeploy, ingress cannot insert and returns an error that Twilio retries. Do the two steps back to back.
- Check:
- The worker's role. Change the password in the Supabase project's database settings, then update the worker's
SITE_DATABASE_URL, and redeployworker. Any operator connection string that uses the same role needs the same update.- Check: the worker stays up and processes a test text.
Twilio
- Auth token (ingress uses it to check webhook signatures).
TWILIO_AUTH_TOKENStakes a comma-separated list so the old and new tokens overlap. Add the new token beside the old one, redeployingress, change the token in Twilio, then remove the old one and redeploy again.- Check: after each redeploy a test text produces
intake_inserted. Asignature_rejectedline means ingress does not hold the token Twilio is signing with.
- Check: after each redeploy a test text produces
- Media API key (the worker uses it to fetch attachments). Create a new media-read-only key, set
TWILIO_API_KEY_SIDandTWILIO_API_KEY_SECRETon the worker, redeployworker, then delete the old key in Twilio.- Check: a test text with a photo reaches
capturedwith its attachment.
- Check: a test text with a photo reaches
Anthropic API key
Create a new key with its own monthly spend limit, set ANTHROPIC_API_KEY on the worker, redeploy worker, then revoke the old key.
- Check: a waiting row moves from
capturedtoextracted. - If not: see the LLM provider runbook in Incident runbooks.
Container registry credential
The registry credential is a classic personal access token with only read:packages, set on every Railway service the Site runs (ingress, worker, and mcp when the Site runs it) so they can pull the private image. It expires.
- Renew it before it expires. Create the new token, set it on each of those services, then revoke the old one.
- Check: the next
Deployrun passes its wait step. An expired credential shows up as a failed wait step on the next deploy, not before it, so a calendar reminder is the only early warning.
- Check: the next
Deploy tokens
The Railway project token and the Vercel token live only as secrets in the Site's GitHub Environment site-<site>.
- Create the new token on the platform, scoped as before: the Railway token to the one environment.
- Replace the secret in the GitHub Environment, then delete the old token on the platform.
- Check: dispatch the matching workflow from
mainand see it authenticate.
- Check: dispatch the matching workflow from
Credentials no service holds
The Supabase service-role key is an operator credential. The provisioning scripts read it from the shell for one run, and neither backend process nor the web app receives it. If it may have been exposed, rotate it in Supabase; no deployed service needs updating. The Supabase publishable key is designed to appear in browser code. If it changes, update SITE_SUPABASE_PUBLISHABLE_KEY on each Railway service the Site runs (ingress, worker, and mcp when the Site runs it) and NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY in the GitHub Environment, then redeploy each of those services and the web app, since the web build inlines it.