Monitoring and alerts
A Site has three alerts and one log. All of them reach one person, the operator: there is no on-call rotation, so each alert should be answerable from this page alone.
There is also no outside uptime monitor. A Site that is quiet looks the same as a Site that is healthy until someone checks, which is why the silent-channel runbook exists.
The alerts
| Alert | Where it comes from | What it means | First check |
|---|---|---|---|
| Ingress healthcheck failing | Railway, on GET /health | Ingress returned 503: its database check or its Deposit sign-in check is error. | Read /health and see which check failed. |
| Twilio error 11200 | Twilio Debugger email | Twilio could not deliver a webhook to ingress. | Is ingress up? Is the webhook URL still <ingress domain>? |
| Worker crash email | Railway | The worker exited or crashed past its restart limit. A halt shows up this way. | Read the worker's last log lines. |
Ingress healthcheck
Railway checks ingress's /health as set by site/backend/deploy/services.json. The endpoint returns 200 only when both checks are ok. It returns 503 when either db or deposit_auth is error. db is database connectivity. deposit_auth is the outcome of the Deposit sign-in's last attempt: a success stays ok without a new request even after the token idles out, and after a failure each ingress process tries to recover at most once a minute. The response shape is in the health endpoint reference.
db: error: go to A Site's database is down in Incident runbooks.deposit_auth: error: the ingress Deposit identity is being refused or Supabase Auth is unreachable. If you just rotated it, finish the rotation. If not, check Supabase's status and Auth settings (sign-ups off, Email provider on).
Twilio error 11200
Twilio retries a failed delivery (the webhook URL asks it to retry on server errors and connection problems), so a single alert is not necessarily a lost message. Ingress answers an error rather than silently accepting, so Twilio's retry covers a short outage.
- Check
/health. If it is failing, follow the health runbook above. - If
/healthis fine, check the URL. The webhook URL Twilio calls must still be the Site's current<ingress domain>. A recreated ingress has a new domain. - Check for
signature_rejected. Rejected signatures meanTWILIO_SMS_WEBHOOK_URLorTWILIO_AUTH_TOKENSno longer matches what Twilio signs; see Rotate credentials.
Worker crash
The worker has no health endpoint. Its liveness is the intake lease table itself: rows leave received and captured. A crash email means the process stopped and Railway gave up restarting it.
- Read its last log lines. A refused Deposit credential or a model halt explains the exit.
Anthropic refused the accountincludes an HTTP status, error type, and request id; it means an account refusal, not a bad text. These are deliberate halts, not bugs: the worker releases the row it held untouched and stops until a human acts. - Fix the cause, then redeploy or restart the worker. A refused Deposit credential means updating its variable (Rotate credentials). For an Anthropic account refusal, raise the limit or wait for the first of the month at 00:00 UTC, or fix the key; then restart the worker. For the in-process budget halt, read the cost runaway runbook before restarting, because restarting lifts the halt.
- Check the worker stays up and rows move.
The ingress log
Ingress writes one JSON line to standard output per delivery, with ids and counts only: never message content, sender or recipient numbers, media URLs, exception text, or stack traces. Each line has ts, level, event, MessageSid, NumMedia, latency_ms, and outcome.
| Event | Level | Outcome | Meaning |
|---|---|---|---|
intake_inserted | info | accepted | A new message was stored. |
intake_duplicate | info | accepted | Twilio redelivered a message already stored. |
signature_rejected | warning | rejected | The request's signature did not match. |
config_missing | error | retry | A required setting was missing when the request arrived. |
deposit_unavailable | error | retry | Ingress could not sign in to the Deposit. |
intake_insert_failed | error | retry | The insert failed. Its line also carries error_type, the database exception class name. |
LOG_LEVEL sets the verbosity and defaults to INFO. Reading error_type is the fastest way to tell a permission problem (such as InsufficientPrivilege) from a connection problem (such as OperationalError).
What is not watched
- A healthy Site that receives nothing. No alert fires on absence. Check by hand; see Incident runbooks.
- Spend. The only automatic limit on model cost is the worker's in-process halt and the spend limit you set on the Anthropic key.
- Worker liveness other than the crash email. A worker that is running but stuck produces no alert.