Run the local Site database
This guide is for developers working on Provenience, not businesses setting up projects in the hosted product. Blue Data Works operates the hosted infrastructure.
The local development substrate provides a Postgres 16 service for a Site. It binds to loopback on port 5432, stores data in a named Docker volume, and keeps that data when stopped. Docker labels identify it as local-development.
The Site backend has one container image for both long-running processes. Its default command runs
ingress; overriding the command with python -m worker.main runs the worker. It is available for
local builds and smoke testing, and the Deploy workflow publishes it to GHCR by full commit SHA
after a green CI run on main.
Run the Site web app locally
The Site web app runs from site/web. Install the repository's one pnpm workspace dependency set
from anywhere in the repository, then run the app and its checks from that directory:
pnpm install
cd site/web
pnpm dev
pnpm lint
pnpm typecheck
pnpm test
pnpm build
The app listens at http://127.0.0.1:3000; use that address rather than localhost. Before
running pnpm dev or pnpm build, create site/web/.env.local from .env.example and supply its
four public settings. Against the local stack, supabase status -o env from site/db prints the
Supabase URL and PUBLISHABLE_KEY. Set NEXT_PUBLIC_SITE_URL to http://127.0.0.1:3000, which
must match the local Auth Site URL. A missing or malformed setting reports its key's name, not its
value.
Local Auth sends at most two sign-in links an hour. Read a delivered message in Mailpit at
http://127.0.0.1:54324. The app does not create users: its magic-link request disables sign-up,
and public sign-ups are disabled for the local project. An administrator exists only after an
operator has provisioned that principal; the administrator-provisioning procedure is not part of
this guide.
CI checks on main
GitHub Actions runs the CI workflow after every push to main and when manually dispatched; it
does not run for pull requests. It runs the Site backend and database unit suites, checks ingress
OpenAPI drift, runs the Museum verification, and runs both integration suites plus the container
image smoke against local Supabase and Compose services. The workflow uses no secrets and does not
contact a real vendor or model. A green CI run from a push to main gates Deploy: it builds
the backend image once, pushes its full-SHA tag to GHCR, then updates and deploys ingress and the
worker for each configured Site. A manual Deploy run with a full SHA can redeploy or roll back
to an existing image tag.
Ingress and worker integration tests share a separate Postgres 16 service, site-test-db, labeled
local-test. It uses a dynamically assigned loopback port and its own named volume, so it cannot
read or overwrite development data. A test session starts the service when it is down and leaves a
running service alone; sessions do not stop it when they finish. Do not use site-test-db for
local development.
Run the ingress integration test from site/backend with:
uv run pytest ingress/tests/test_health_integration.py -v
No manual Compose startup or cleanup is needed. Unit tests do not start Docker. The lock around
service startup prevents competing starts, then each ingress session uses its own database. Worker
tests and Site database tests that use isolated_dsn copy scratch databases from a migrated
template. Test sessions remove
their databases when they finish; on startup, each scope also removes only its own timestamped
scratch databases that are more than six hours old. Rows do not leak between tests or sessions.
Start Postgres
From the repository root:
docker compose up -d site-db
The development-only connection values are:
| Setting | Value |
|---|---|
| Host | 127.0.0.1 |
| Port | 5432 |
| Database | site |
| User | postgres |
| Password | postgres |
These values are local defaults, not production credentials.
Seed a starter typology
Before enabling Extract on a new Site, a platform operator can load a starter category vocabulary. The seed is for an empty typology only: the script refuses to run if the Site already has a live category. It validates the file before connecting, then writes the typology version, categories, operator service principal, and Daybook event in one transaction.
From site/db, set an operator connection string for the intended Site and run:
$env:SITE_DATABASE_URL = "<operator connection string>"
uv run python scripts/seed_typology.py --file seeds/typology/remodeler.yaml
The bundled seed is a starter vocabulary for a residential design-build remodeler. It creates twelve company-wide categories. Afterward, an administrator reviews and maintains those categories in Typology; do not run the seed again to change an existing Site's vocabulary.
Run the Site ingress service
The ingress and worker use different Deposit identities. From site/db, restart an existing local
Supabase stack so it reloads supabase/config.toml, then create both identities once with the
local stack's SECRET_KEY from supabase status -o env:
cd site/db
supabase stop
supabase start -x studio,edge-runtime,realtime,logflare,vector,imgproxy,supavisor
supabase migration up --local
$env:SUPABASE_URL = "http://127.0.0.1:54321"
$env:SUPABASE_SERVICE_ROLE_KEY = "<the local SECRET_KEY>"
uv run python scripts/provision_deposit_identities.py
On the first start of a fresh stack, omit supabase stop. For an already-running stack, the stop
and start are required: supabase start alone does not reload the setting that disables public
sign-ups, and the provisioning script refuses to run while sign-ups are enabled. The excluded
services are not required by the local Site stack; start without studio in the exclusion list
when you need Studio. Apply later migrations with supabase migration up --local without
restarting the stack.
The script prints one generated password for deposit-ingress@site.internal and one for
deposit-worker@site.internal. Keep each value only in its process environment. The service-role
key is an operator credential for provisioning; neither backend process receives it.
In a second terminal, use the ingress identity and its generated password:
cd site/backend
$env:SITE_DATABASE_URL = "postgresql://postgres:postgres@127.0.0.1:5432/site"
$env:SITE_SUPABASE_URL = "http://127.0.0.1:54321"
$env:SITE_SUPABASE_PUBLISHABLE_KEY = "<the local PUBLISHABLE_KEY>"
$env:SITE_DEPOSIT_AUTH_EMAIL = "deposit-ingress@site.internal"
$env:SITE_DEPOSIT_AUTH_PASSWORD = "<the generated ingress password>"
# Optional; defaults to INFO.
$env:LOG_LEVEL = "INFO"
uv run uvicorn ingress.app:app --reload
Open http://127.0.0.1:8000/health. The endpoint reports both database connectivity and the
outcome of the Deposit token provider's last Auth attempt. It returns HTTP 200 only when both are
ok; either error returns HTTP 503. A successful last attempt stays ok without making another
Auth request, even after an idle token expires. After a failed attempt, health probes try recovery
at most once a minute per ingress process. See the health endpoint reference
for the response shape.
LOG_LEVEL is optional and defaults to INFO. Ingress writes one structured JSON line to stdout
for each of six delivery outcomes: signature_rejected, config_missing,
deposit_unavailable, intake_inserted, intake_duplicate, and intake_insert_failed. Each line
contains ts, level, event, MessageSid, NumMedia, latency_ms, and outcome.
intake_insert_failed alone also contains error_type, the psycopg exception class name, such as
InsufficientPrivilege or OperationalError. Ingress never renders message content, the From,
To, or media-URL fields, exception text, or tracebacks.
signature_rejected is warning/rejected; config_missing, deposit_unavailable, and
intake_insert_failed are error/retry; and intake_inserted and intake_duplicate are
info/accepted.
For a Twilio callback, set TWILIO_SMS_WEBHOOK_URL to the public callback URL used for signature
verification, without a connection-override fragment. Configure Twilio's webhook URL with
#rc=3&rp=5xx,ct,rt, which retries 5xx responses and connection problems. Twilio documents that
URL-fragment override syntax and omits the fragment from the signed URL. Do not copy it into
TWILIO_SMS_WEBHOOK_URL, because doing so makes signature verification fail.
Run the worker
The worker is a separate long-running process from ingress. It first leases received rows for
Capture, records their raw source and original, copies any MMS attachments from Twilio into the
Deposit, and moves complete rows to captured. Extract is off by default, so this path does not
need an Anthropic key. In a third terminal:
cd site/backend
$env:SITE_DATABASE_URL = "postgresql://postgres:postgres@127.0.0.1:5432/site"
$env:SITE_SUPABASE_URL = "http://127.0.0.1:54321"
$env:SITE_SUPABASE_PUBLISHABLE_KEY = "<the local PUBLISHABLE_KEY>"
$env:SITE_DEPOSIT_AUTH_EMAIL = "deposit-worker@site.internal"
$env:SITE_DEPOSIT_AUTH_PASSWORD = "<the generated worker password>"
$env:TWILIO_ACCOUNT_SID = "<the Site's Twilio account SID>"
$env:TWILIO_API_KEY_SID = "<a Twilio API key SID>"
$env:TWILIO_API_KEY_SECRET = "<the Twilio API key secret>"
uv run python -m worker.main
The Twilio subaccount must have Enforce HTTP Auth on Media URLs enabled. A restricted API key
with twilio/messaging/messages.media/read is sufficient for Capture. Keep the key read-only until
go-live, then add the MMS media delete permission (confirm its exact name in the Twilio console).
The worker deletes Twilio's media copies after a new text is captured; the message record and older
captured texts stay at Twilio. Until the key can delete, cleanup retries every five minutes without
failing Capture. The worker accepts media URLs only for
that account, follows redirects only to Twilio's secured media CDN without forwarding the API key,
and rejects an item larger than 20 MiB by default. Set CAPTURE_MEDIA_MAX_BYTES to a positive
number of bytes to choose a different per-item cap.
To run Extract after Capture, add both settings before starting the worker:
$env:WORKER_EXTRACT_ENABLED = "true"
$env:ANTHROPIC_API_KEY = "<your Anthropic API key>"
Leaving WORKER_EXTRACT_ENABLED unset, empty, false, or 0 keeps Extract off. The worker then
leaves captured rows ready for a later Extract-enabled run. Any other value stops startup rather
than guessing whether model calls should be enabled.
The current configuration gives one Extract call a 60-second timeout and an estimated $0.25 budget. For an MMS, it can send up to 10 qualifying images or PDFs of up to 20 pages; images and the whole attachment set also have configured byte limits. Files outside those limits are still cataloged but are named in the model message rather than sent.
Apply the Site database migrations to the local Supabase stack (supabase migration up --local
from site/db) before running a worker. It polls backfill_request only while Extract is enabled;
pending requests wait until then.
Both processes sign in before they begin serving or polling and refuse to start when their Deposit
credentials are missing or refused. Each keeps its own short-lived access token and rotating
refresh token in memory, and signs in again after a restart or failed refresh. Neither uses a
static Storage token, signs JWTs, or holds a service-role key. SITE_DEPOSIT_BUCKET defaults to
deposit and rarely needs setting. WORKER_ID defaults to <hostname>-<pid> and only needs
setting to run more than one worker process against the same lease table with identifiable ids.
When an operator rotates the worker Deposit identity, update the worker credential variable and
redeploy it together. If the running worker encounters the now-refused credential during Capture
or Extract, it logs one error and exits rather than failing the leased row. Railway shows the
worker as crashed after its restart limit until the variable is updated and the worker is
redeployed. The row remains received or captured as applicable, with no failure item.
When Extract is enabled, the worker resolves the current typology version after leasing each
captured text and before its model call. A category added through the admin path is therefore used
for the next extraction without restarting the worker. There is no health endpoint for the worker
— its liveness is the intake lease table itself.
site/backend/.env.example is the keys-only reference for every setting read by the image. It is
grouped by ingress and worker, contains no values, and is not copied into the image.
Smoke-test the container image
With the local Supabase stack running from site/db, run this from the repository root in a Bash
shell:
site/backend/scripts/smoke_image.sh
The script builds the shared image, confirms it runs as a non-root user and carries its source and
revision labels, starts ingress with the image's default command and probes /health, then starts
the worker with python -m worker.main and verifies that it Captures a seeded received row. It
uses local-only credentials and fake Twilio values; the seeded message has no media, so the smoke
run makes no Twilio request.
The smoke run leaves the two local Deposit identities, one content-addressed Deposit object, and a
captured intake row with its raw source in the local Supabase stack. Rerunning it reuses the fixed
object. Run supabase db reset from site/db when you intentionally want to clear those rows.
Stop the service
From the repository root:
docker compose stop site-db
Stopping the container preserves its named volume. Removing local data is a separate, deliberate Docker operation and is not part of this guide.
The development container and volume carry the Docker labels com.provenience.environment=local-development and com.provenience.scope=site. The integration-test container and volume use com.provenience.environment=local-test and com.provenience.scope=site-backend.