Incident runbooks
Each runbook has the same shape: symptoms (what you see), checks (what to look at, in order), and actions (what to do). Diagnose with counts, states, timestamps, and log lines. Do not read a customer's messages or files to find a fault; the operator has no standing access to them, and none of these need it.
Where a check reads the database, use the Site's operator connection string and select only metadata columns of intake: state, attempt_count, lease_holder, lease_expires_at, and ingested_at.
The intake states are received, captured, extracted, contextualized, governed, complete, and failed. A healthy row moves forward through them, ending at complete, or at failed with one extraction-failed item for an administrator to see.
A Site's database is down
Symptoms. Ingress's /health returns 503 with db: error. Twilio emails error 11200. Ingress logs intake_insert_failed with an error_type such as OperationalError. The web app shows errors or a 503, because its access gate cannot reach the database.
Checks.
- Is the Supabase project up? Look at the vendor's status page and the project's own dashboard (a paused or restarting project looks like this).
- Did something change? A rotated password, a changed pooler string, or a new
site_ingresspassword not yet inSITE_DATABASE_URL. See Rotate credentials. - Read the
error_typein the ingress log. A permission class points at the role; a connection class points at reachability.
Actions.
- Fix the cause above. If it is the vendor's outage, wait; Twilio keeps retrying a failed delivery, and ingress answers an error rather than dropping the message.
- If data is damaged rather than unreachable, use Supabase's point-in-time recovery, an action against the vendor. This runbook gives no restore procedure.
- Confirm:
/healthreturns 200, then a test text producesintake_inserted.
LLM provider down or throttling
Symptoms. With Extract on, captured rows stop moving to extracted. Rows show a rising attempt_count. Some rows reach failed, and administrators see more extraction-failed items in the Review Queue. An account refusal is different: see Anthropic account refusal below.
Checks.
- Is the model provider having an incident, or is your account rate-limited or out of spend? Check the provider's status and the key's usage and limit.
- Count the rows by state. A growing
capturedcount with fewextractedis the shape of this. - Was the key rotated or revoked?
What the worker does on its own. Each failed Extract attempt is retried with a backoff that doubles from 30 seconds up to 15 minutes, for at most five attempts. A call that runs past 60 seconds counts as a failure. A text that was held once for requested context receives one additional attempt after that hold. After its allowed attempts fail, the row becomes failed with one extraction-failed item. So a long outage can turn ordinary texts into failed items.
Actions.
- For an outage expected to outlast the retries, turn Extract off so rows wait instead of burning attempts: unset
WORKER_EXTRACT_ENABLEDand redeploy the worker. Capture continues, andcapturedrows wait. See Turn Extract on. - If you hit your own spend or rate limit, raise it or wait, then turn Extract back on.
- For rows that already failed, an administrator selects Retry on each failed-extraction item in the Review Queue once the provider is healthy.
- Confirm: a waiting row reaches
governed, and the failed-item count stops growing.
Anthropic account refusal
Symptoms. Railway sends a worker crash email. The worker log has an ERROR line beginning Anthropic refused the account, with an HTTP status, Anthropic error type, and request id; the stage then logs its halt. The held captured row is released untouched: it has no new failure item and does not consume a retry.
What it is. This is an account fact, not a bad text. The worker halts for an Anthropic 401, 402, or 403; a 429 tier spend cap (enforced_spend_limit_reached); or the documented user-set spend-limit 400 message beginning You have reached your specified.
Actions.
- For a spend limit, raise the Anthropic limit or wait for the first of the month at 00:00 UTC.
- For a refused key, fix
ANTHROPIC_API_KEYon Railway. See Rotate credentials. - Then restart or redeploy the Railway
worker. It resumes the released row. - Confirm: the worker stays up and a waiting row advances. Do not retry or dispose of the released row by hand.
Repeated lease expiry
Symptoms. The same rows keep being picked up again. Their attempt_count climbs, and lease_expires_at keeps passing while the row stays in the same state. Rows are not reaching complete.
What it is. The worker leases a row for five minutes and moves it on when the stage completes. If the worker crashes or stalls mid-stage, the lease expires and the next claim takes the row, raising its attempt_count. That is by design, and the stages are built so a repeated attempt converges. Repeated expiry on the same rows means something stops the stage finishing.
Checks.
- Is the worker running? A crash email, or a log that ends abruptly, means it is restarting in a loop or has stopped.
- Read the worker's log for the rows' stage. A refused Deposit credential or a halt each explain an expiring lease.
- Are the same few rows repeating, or all of them? The same few rows point at a poison payload; all of them point at the worker, the database, or a provider.
Actions.
- If the worker is stopped, find its reason (see Monitoring and alerts), fix it, and redeploy.
- If only a few rows repeat, follow Poison payload.
- Confirm: the repeated rows'
attempt_countstops rising and their state advances.
Poison payload
Symptoms. One or a few rows fail every attempt while others pass, or a row stays in one state with a high attempt_count. An administrator sees an extraction-failed item, including one the model refused.
What the worker does on its own. Capture gets five attempts with backoff. Extract gets five, or six when the text was held once for requested context. A row that exhausts its allowed attempts becomes failed and raises one extraction-failed item, so a poison row cannot hide: it reaches a person. A Twilio media item the worker is not allowed to fetch, or one over the size cap (20 MiB by default), is refused at once without retries. A model refusal also goes straight to extraction-failed, with no retry and no fallback model.
Checks.
- Is the failure one row, or many? Many rows point at a provider or credential problem, not a payload.
- Is the row's state
failed? If so it already has a Review Queue item. - Read the worker's log for that row's id: ids and error classes appear without content.
Actions.
- Leave the item for the administrator. They can Retry it, or exclude it so it does not enter the Catalog.
- Do not edit the row by hand to move it along, and never delete an
intakerow, a raw source, or a Daybook entry. - If a legitimate attachment keeps being refused for size, raising
CAPTURE_MEDIA_MAX_BYTESon the worker is a deliberate choice; weigh its cost, then redeploy. - Confirm: the row is
completeorfailedwith one item, never stuck.
Ingestion channel silent
Symptoms. Nothing is wrong that an alert can see. /health is 200, there is no 11200 email, the worker is up, the queue is empty, and no new intake rows are arriving. A quiet customer looks the same, so this is a judgment call: a remodeler's project does go quiet for days.
Why it matters. A channel that fails silently loses messages before they ever become an item, so the Review Queue cannot catch it. Nothing alerts on absence, so this is a manual check.
Checks.
- When did the last
intakerow arrive (max(ingested_at))? Compare it with the Site's usual rhythm. - Send a test text to the Site's number. Does ingress log
intake_insertedfor it? - If not, check what Twilio shows: the number's inbound webhook must still be
https://<ingress domain>/webhooks/twilio/sms#rc=3&rp=5xx,ct,rt. A recreated ingress, an edited number, or a lapsed registration for the number each break it silently. - If ingress logs
signature_rejected,TWILIO_SMS_WEBHOOK_URL(no fragment) orTWILIO_AUTH_TOKENSis out of step with Twilio.
Actions.
- Restore the webhook URL, the setting, or the registration that broke.
- Confirm: a new test text produces
intake_insertedand reachescaptured. - Tell the customer's administrator if the gap was long. Texts sent during it may not have reached the Site, and only they can say whether to ask people to resend.
Cost runaway
Symptoms. The Anthropic usage page shows spend well above your estimate, or the worker stops itself and Railway emails a crash. The worker's log names a budget halt.
What the worker does on its own. Each Extract call has an estimated budget of $0.25. After three consecutive over-budget calls, the worker stops calling the model. The halt is in the worker process only: the row it was holding is released untouched, the process exits, and restarting the worker lifts the halt. It is a stop for a human to look, not a cutoff the platform enforces on its own.
Checks.
- Read the log lines before the halt. Are the overruns from large attachments (many photos, a long PDF) or from something else?
- Compare the day's spend with your estimate: a text about $0.02, three photos about $0.04, a 20-page PDF about $0.20.
- Was a backlog of
capturedrows released at once when Extract was turned on? Every waiting row is processed with no throttle.
Actions.
- Stop the spend first. Unset
WORKER_EXTRACT_ENABLEDand redeploy the worker, or revoke the Anthropic key. Rows staycaptured. - Decide with the Site's owner whether the spend is expected. If so, raise the key's monthly limit deliberately. If not, find the cause before restarting.
- A change to the model, prompt, schema, or a threshold is a calibration change that adds a new record entry; see Turn Extract on.
- Confirm: spend is flat after the change, and the worker stays up once Extract is back on.