Browse this guideMonitoring and alerts
DevelopmentUpdated 2026-10-17

Monitoring and alerts

A Site has three alerts and one log. All of them reach one person, the operator: there is no on-call rotation, so each alert should be answerable from this page alone.

There is also no outside uptime monitor. A Site that is quiet looks the same as a Site that is healthy until someone checks, which is why the silent-channel runbook exists.

The alerts

AlertWhere it comes fromWhat it meansFirst check
Ingress healthcheck failingRailway, on GET /healthIngress returned 503: its database check or its Deposit sign-in check is error.Read /health and see which check failed.
Twilio error 11200Twilio Debugger emailTwilio could not deliver a webhook to ingress.Is ingress up? Is the webhook URL still <ingress domain>?
Worker crash emailRailwayThe worker exited or crashed past its restart limit. A halt shows up this way.Read the worker's last log lines.

Ingress healthcheck

Railway checks ingress's /health as set by site/backend/deploy/services.json. The endpoint returns 200 only when both checks are ok. It returns 503 when either db or deposit_auth is error. db is database connectivity. deposit_auth is the outcome of the Deposit sign-in's last attempt: a success stays ok without a new request even after the token idles out, and after a failure each ingress process tries to recover at most once a minute. The response shape is in the health endpoint reference.

Twilio error 11200

Twilio retries a failed delivery (the webhook URL asks it to retry on server errors and connection problems), so a single alert is not necessarily a lost message. Ingress answers an error rather than silently accepting, so Twilio's retry covers a short outage.

  1. Check /health. If it is failing, follow the health runbook above.
  2. If /health is fine, check the URL. The webhook URL Twilio calls must still be the Site's current <ingress domain>. A recreated ingress has a new domain.
  3. Check for signature_rejected. Rejected signatures mean TWILIO_SMS_WEBHOOK_URL or TWILIO_AUTH_TOKENS no longer matches what Twilio signs; see Rotate credentials.

Worker crash

The worker has no health endpoint. Its liveness is the intake lease table itself: rows leave received and captured. A crash email means the process stopped and Railway gave up restarting it.

  1. Read its last log lines. A refused Deposit credential or a model halt explains the exit. Anthropic refused the account includes an HTTP status, error type, and request id; it means an account refusal, not a bad text. These are deliberate halts, not bugs: the worker releases the row it held untouched and stops until a human acts.
  2. Fix the cause, then redeploy or restart the worker. A refused Deposit credential means updating its variable (Rotate credentials). For an Anthropic account refusal, raise the limit or wait for the first of the month at 00:00 UTC, or fix the key; then restart the worker. For the in-process budget halt, read the cost runaway runbook before restarting, because restarting lifts the halt.
  3. Check the worker stays up and rows move.

The ingress log

Ingress writes one JSON line to standard output per delivery, with ids and counts only: never message content, sender or recipient numbers, media URLs, exception text, or stack traces. Each line has ts, level, event, MessageSid, NumMedia, latency_ms, and outcome.

EventLevelOutcomeMeaning
intake_insertedinfoacceptedA new message was stored.
intake_duplicateinfoacceptedTwilio redelivered a message already stored.
signature_rejectedwarningrejectedThe request's signature did not match.
config_missingerrorretryA required setting was missing when the request arrived.
deposit_unavailableerrorretryIngress could not sign in to the Deposit.
intake_insert_failederrorretryThe insert failed. Its line also carries error_type, the database exception class name.

LOG_LEVEL sets the verbosity and defaults to INFO. Reading error_type is the fastest way to tell a permission problem (such as InsufficientPrivilege) from a connection problem (such as OperationalError).

What is not watched