Browse this guideIncident runbooks
DevelopmentUpdated 2026-10-17

Incident runbooks

Each runbook has the same shape: symptoms (what you see), checks (what to look at, in order), and actions (what to do). Diagnose with counts, states, timestamps, and log lines. Do not read a customer's messages or files to find a fault; the operator has no standing access to them, and none of these need it.

Where a check reads the database, use the Site's operator connection string and select only metadata columns of intake: state, attempt_count, lease_holder, lease_expires_at, and ingested_at.

The intake states are received, captured, extracted, contextualized, governed, complete, and failed. A healthy row moves forward through them, ending at complete, or at failed with one extraction-failed item for an administrator to see.

A Site's database is down

Symptoms. Ingress's /health returns 503 with db: error. Twilio emails error 11200. Ingress logs intake_insert_failed with an error_type such as OperationalError. The web app shows errors or a 503, because its access gate cannot reach the database.

Checks.

  1. Is the Supabase project up? Look at the vendor's status page and the project's own dashboard (a paused or restarting project looks like this).
  2. Did something change? A rotated password, a changed pooler string, or a new site_ingress password not yet in SITE_DATABASE_URL. See Rotate credentials.
  3. Read the error_type in the ingress log. A permission class points at the role; a connection class points at reachability.

Actions.

LLM provider down or throttling

Symptoms. With Extract on, captured rows stop moving to extracted. Rows show a rising attempt_count. Some rows reach failed, and administrators see more extraction-failed items in the Review Queue. An account refusal is different: see Anthropic account refusal below.

Checks.

  1. Is the model provider having an incident, or is your account rate-limited or out of spend? Check the provider's status and the key's usage and limit.
  2. Count the rows by state. A growing captured count with few extracted is the shape of this.
  3. Was the key rotated or revoked?

What the worker does on its own. Each failed Extract attempt is retried with a backoff that doubles from 30 seconds up to 15 minutes, for at most five attempts. A call that runs past 60 seconds counts as a failure. A text that was held once for requested context receives one additional attempt after that hold. After its allowed attempts fail, the row becomes failed with one extraction-failed item. So a long outage can turn ordinary texts into failed items.

Actions.

Anthropic account refusal

Symptoms. Railway sends a worker crash email. The worker log has an ERROR line beginning Anthropic refused the account, with an HTTP status, Anthropic error type, and request id; the stage then logs its halt. The held captured row is released untouched: it has no new failure item and does not consume a retry.

What it is. This is an account fact, not a bad text. The worker halts for an Anthropic 401, 402, or 403; a 429 tier spend cap (enforced_spend_limit_reached); or the documented user-set spend-limit 400 message beginning You have reached your specified.

Actions.

Repeated lease expiry

Symptoms. The same rows keep being picked up again. Their attempt_count climbs, and lease_expires_at keeps passing while the row stays in the same state. Rows are not reaching complete.

What it is. The worker leases a row for five minutes and moves it on when the stage completes. If the worker crashes or stalls mid-stage, the lease expires and the next claim takes the row, raising its attempt_count. That is by design, and the stages are built so a repeated attempt converges. Repeated expiry on the same rows means something stops the stage finishing.

Checks.

  1. Is the worker running? A crash email, or a log that ends abruptly, means it is restarting in a loop or has stopped.
  2. Read the worker's log for the rows' stage. A refused Deposit credential or a halt each explain an expiring lease.
  3. Are the same few rows repeating, or all of them? The same few rows point at a poison payload; all of them point at the worker, the database, or a provider.

Actions.

Poison payload

Symptoms. One or a few rows fail every attempt while others pass, or a row stays in one state with a high attempt_count. An administrator sees an extraction-failed item, including one the model refused.

What the worker does on its own. Capture gets five attempts with backoff. Extract gets five, or six when the text was held once for requested context. A row that exhausts its allowed attempts becomes failed and raises one extraction-failed item, so a poison row cannot hide: it reaches a person. A Twilio media item the worker is not allowed to fetch, or one over the size cap (20 MiB by default), is refused at once without retries. A model refusal also goes straight to extraction-failed, with no retry and no fallback model.

Checks.

  1. Is the failure one row, or many? Many rows point at a provider or credential problem, not a payload.
  2. Is the row's state failed? If so it already has a Review Queue item.
  3. Read the worker's log for that row's id: ids and error classes appear without content.

Actions.

Ingestion channel silent

Symptoms. Nothing is wrong that an alert can see. /health is 200, there is no 11200 email, the worker is up, the queue is empty, and no new intake rows are arriving. A quiet customer looks the same, so this is a judgment call: a remodeler's project does go quiet for days.

Why it matters. A channel that fails silently loses messages before they ever become an item, so the Review Queue cannot catch it. Nothing alerts on absence, so this is a manual check.

Checks.

  1. When did the last intake row arrive (max(ingested_at))? Compare it with the Site's usual rhythm.
  2. Send a test text to the Site's number. Does ingress log intake_inserted for it?
  3. If not, check what Twilio shows: the number's inbound webhook must still be https://<ingress domain>/webhooks/twilio/sms#rc=3&rp=5xx,ct,rt. A recreated ingress, an edited number, or a lapsed registration for the number each break it silently.
  4. If ingress logs signature_rejected, TWILIO_SMS_WEBHOOK_URL (no fragment) or TWILIO_AUTH_TOKENS is out of step with Twilio.

Actions.

Cost runaway

Symptoms. The Anthropic usage page shows spend well above your estimate, or the worker stops itself and Railway emails a crash. The worker's log names a budget halt.

What the worker does on its own. Each Extract call has an estimated budget of $0.25. After three consecutive over-budget calls, the worker stops calling the model. The halt is in the worker process only: the row it was holding is released untouched, the process exits, and restarting the worker lifts the halt. It is a stop for a human to look, not a cutoff the platform enforces on its own.

Checks.

  1. Read the log lines before the halt. Are the overruns from large attachments (many photos, a long PDF) or from something else?
  2. Compare the day's spend with your estimate: a text about $0.02, three photos about $0.04, a 20-page PDF about $0.20.
  3. Was a backlog of captured rows released at once when Extract was turned on? Every waiting row is processed with no throttle.

Actions.