Browse this guideRun the local Site database
DevelopmentUpdated 2026-11-10

Run the local Site database

This guide is for developers working on Provenience, not businesses setting up projects in the hosted product. Blue Data Works operates the hosted infrastructure.

The local development substrate provides a Postgres 16 service for a Site. It binds to loopback on port 5432, stores data in a named Docker volume, and keeps that data when stopped. Docker labels identify it as local-development.

The Site backend has one container image for both long-running processes. Its default command runs ingress; overriding the command with python -m worker.main runs the worker. It is available for local builds and smoke testing, and the Deploy workflow publishes it to GHCR by full commit SHA after a green CI run on main.

Run the Site web app locally

The Site web app runs from site/web. Install the repository's one pnpm workspace dependency set from anywhere in the repository, then run the app and its checks from that directory:

pnpm install
cd site/web
pnpm dev
pnpm lint
pnpm typecheck
pnpm test
pnpm build

The app listens at http://127.0.0.1:3000; use that address rather than localhost. Before running pnpm dev or pnpm build, create site/web/.env.local from .env.example and supply its four public settings. Against the local stack, supabase status -o env from site/db prints the Supabase URL and PUBLISHABLE_KEY. Set NEXT_PUBLIC_SITE_URL to http://127.0.0.1:3000, which must match the local Auth Site URL. A missing or malformed setting reports its key's name, not its value.

Local Auth sends at most two sign-in links an hour. Read a delivered message in Mailpit at http://127.0.0.1:54324. The app does not create users: its magic-link request disables sign-up, and public sign-ups are disabled for the local project. An administrator exists only after an operator has provisioned that principal; the administrator-provisioning procedure is not part of this guide.

CI checks on main

GitHub Actions runs the CI workflow after every push to main and when manually dispatched; it does not run for pull requests. It runs the Site backend and database unit suites, checks ingress OpenAPI drift, runs the Museum verification, and runs both integration suites plus the container image smoke against local Supabase and Compose services. The workflow uses no secrets and does not contact a real vendor or model. A green CI run from a push to main gates Deploy: it builds the backend image once, pushes its full-SHA tag to GHCR, then updates and deploys ingress and the worker for each configured Site. A manual Deploy run with a full SHA can redeploy or roll back to an existing image tag.

Ingress and worker integration tests share a separate Postgres 16 service, site-test-db, labeled local-test. It uses a dynamically assigned loopback port and its own named volume, so it cannot read or overwrite development data. A test session starts the service when it is down and leaves a running service alone; sessions do not stop it when they finish. Do not use site-test-db for local development.

Run the ingress integration test from site/backend with:

uv run pytest ingress/tests/test_health_integration.py -v

No manual Compose startup or cleanup is needed. Unit tests do not start Docker. The lock around service startup prevents competing starts, then each ingress session uses its own database. Worker tests and Site database tests that use isolated_dsn copy scratch databases from a migrated template. Test sessions remove their databases when they finish; on startup, each scope also removes only its own timestamped scratch databases that are more than six hours old. Rows do not leak between tests or sessions.

Start Postgres

From the repository root:

docker compose up -d site-db

The development-only connection values are:

SettingValue
Host127.0.0.1
Port5432
Databasesite
Userpostgres
Passwordpostgres

These values are local defaults, not production credentials.

Seed a starter typology

Before enabling Extract on a new Site, a platform operator can load a starter category vocabulary. The seed is for an empty typology only: the script refuses to run if the Site already has a live category. It validates the file before connecting, then writes the typology version, categories, operator service principal, and Daybook event in one transaction.

From site/db, set an operator connection string for the intended Site and run:

$env:SITE_DATABASE_URL = "<operator connection string>"
uv run python scripts/seed_typology.py --file seeds/typology/remodeler.yaml

The bundled seed is a starter vocabulary for a residential design-build remodeler. It creates twelve company-wide categories. Afterward, an administrator reviews and maintains those categories in Typology; do not run the seed again to change an existing Site's vocabulary.

Run the Site ingress service

The ingress and worker use different Deposit identities. From site/db, restart an existing local Supabase stack so it reloads supabase/config.toml, then create both identities once with the local stack's SECRET_KEY from supabase status -o env:

cd site/db
supabase stop
supabase start -x studio,edge-runtime,realtime,logflare,vector,imgproxy,supavisor
supabase migration up --local
$env:SUPABASE_URL = "http://127.0.0.1:54321"
$env:SUPABASE_SERVICE_ROLE_KEY = "<the local SECRET_KEY>"
uv run python scripts/provision_deposit_identities.py

On the first start of a fresh stack, omit supabase stop. For an already-running stack, the stop and start are required: supabase start alone does not reload the setting that disables public sign-ups, and the provisioning script refuses to run while sign-ups are enabled. The excluded services are not required by the local Site stack; start without studio in the exclusion list when you need Studio. Apply later migrations with supabase migration up --local without restarting the stack.

The script prints one generated password for deposit-ingress@site.internal and one for deposit-worker@site.internal. Keep each value only in its process environment. The service-role key is an operator credential for provisioning; neither backend process receives it.

In a second terminal, use the ingress identity and its generated password:

cd site/backend
$env:SITE_DATABASE_URL = "postgresql://postgres:postgres@127.0.0.1:5432/site"
$env:SITE_SUPABASE_URL = "http://127.0.0.1:54321"
$env:SITE_SUPABASE_PUBLISHABLE_KEY = "<the local PUBLISHABLE_KEY>"
$env:SITE_DEPOSIT_AUTH_EMAIL = "deposit-ingress@site.internal"
$env:SITE_DEPOSIT_AUTH_PASSWORD = "<the generated ingress password>"
# Optional; defaults to INFO.
$env:LOG_LEVEL = "INFO"
uv run uvicorn ingress.app:app --reload

Open http://127.0.0.1:8000/health. The endpoint reports both database connectivity and the outcome of the Deposit token provider's last Auth attempt. It returns HTTP 200 only when both are ok; either error returns HTTP 503. A successful last attempt stays ok without making another Auth request, even after an idle token expires. After a failed attempt, health probes try recovery at most once a minute per ingress process. See the health endpoint reference for the response shape.

LOG_LEVEL is optional and defaults to INFO. Ingress writes one structured JSON line to stdout for each of six delivery outcomes: signature_rejected, config_missing, deposit_unavailable, intake_inserted, intake_duplicate, and intake_insert_failed. Each line contains ts, level, event, MessageSid, NumMedia, latency_ms, and outcome. intake_insert_failed alone also contains error_type, the psycopg exception class name, such as InsufficientPrivilege or OperationalError. Ingress never renders message content, the From, To, or media-URL fields, exception text, or tracebacks. signature_rejected is warning/rejected; config_missing, deposit_unavailable, and intake_insert_failed are error/retry; and intake_inserted and intake_duplicate are info/accepted.

For a Twilio callback, set TWILIO_SMS_WEBHOOK_URL to the public callback URL used for signature verification, without a connection-override fragment. Configure Twilio's webhook URL with #rc=3&rp=5xx,ct,rt, which retries 5xx responses and connection problems. Twilio documents that URL-fragment override syntax and omits the fragment from the signed URL. Do not copy it into TWILIO_SMS_WEBHOOK_URL, because doing so makes signature verification fail.

Run the worker

The worker is a separate long-running process from ingress. It first leases received rows for Capture, records their raw source and original, copies any MMS attachments from Twilio into the Deposit, and moves complete rows to captured. Extract is off by default, so this path does not need an Anthropic key. In a third terminal:

cd site/backend
$env:SITE_DATABASE_URL = "postgresql://postgres:postgres@127.0.0.1:5432/site"
$env:SITE_SUPABASE_URL = "http://127.0.0.1:54321"
$env:SITE_SUPABASE_PUBLISHABLE_KEY = "<the local PUBLISHABLE_KEY>"
$env:SITE_DEPOSIT_AUTH_EMAIL = "deposit-worker@site.internal"
$env:SITE_DEPOSIT_AUTH_PASSWORD = "<the generated worker password>"
$env:TWILIO_ACCOUNT_SID = "<the Site's Twilio account SID>"
$env:TWILIO_API_KEY_SID = "<a Twilio API key SID>"
$env:TWILIO_API_KEY_SECRET = "<the Twilio API key secret>"
uv run python -m worker.main

The Twilio subaccount must have Enforce HTTP Auth on Media URLs enabled. A restricted API key with twilio/messaging/messages.media/read is sufficient for Capture. Keep the key read-only until go-live, then add the MMS media delete permission (confirm its exact name in the Twilio console). The worker deletes Twilio's media copies after a new text is captured; the message record and older captured texts stay at Twilio. Until the key can delete, cleanup retries every five minutes without failing Capture. The worker accepts media URLs only for that account, follows redirects only to Twilio's secured media CDN without forwarding the API key, and rejects an item larger than 20 MiB by default. Set CAPTURE_MEDIA_MAX_BYTES to a positive number of bytes to choose a different per-item cap.

To run Extract after Capture, add both settings before starting the worker:

$env:WORKER_EXTRACT_ENABLED = "true"
$env:ANTHROPIC_API_KEY = "<your Anthropic API key>"

Leaving WORKER_EXTRACT_ENABLED unset, empty, false, or 0 keeps Extract off. The worker then leaves captured rows ready for a later Extract-enabled run. Any other value stops startup rather than guessing whether model calls should be enabled.

The current configuration gives one Extract call a 60-second timeout and an estimated $0.25 budget. For an MMS, it can send up to 10 qualifying images or PDFs of up to 20 pages; images and the whole attachment set also have configured byte limits. Files outside those limits are still cataloged but are named in the model message rather than sent.

Apply the Site database migrations to the local Supabase stack (supabase migration up --local from site/db) before running a worker. It polls backfill_request only while Extract is enabled; pending requests wait until then.

Both processes sign in before they begin serving or polling and refuse to start when their Deposit credentials are missing or refused. Each keeps its own short-lived access token and rotating refresh token in memory, and signs in again after a restart or failed refresh. Neither uses a static Storage token, signs JWTs, or holds a service-role key. SITE_DEPOSIT_BUCKET defaults to deposit and rarely needs setting. WORKER_ID defaults to <hostname>-<pid> and only needs setting to run more than one worker process against the same lease table with identifiable ids.

When an operator rotates the worker Deposit identity, update the worker credential variable and redeploy it together. If the running worker encounters the now-refused credential during Capture or Extract, it logs one error and exits rather than failing the leased row. Railway shows the worker as crashed after its restart limit until the variable is updated and the worker is redeployed. The row remains received or captured as applicable, with no failure item.

When Extract is enabled, the worker resolves the current typology version after leasing each captured text and before its model call. A category added through the admin path is therefore used for the next extraction without restarting the worker. There is no health endpoint for the worker — its liveness is the intake lease table itself.

site/backend/.env.example is the keys-only reference for every setting read by the image. It is grouped by ingress and worker, contains no values, and is not copied into the image.

Smoke-test the container image

With the local Supabase stack running from site/db, run this from the repository root in a Bash shell:

site/backend/scripts/smoke_image.sh

The script builds the shared image, confirms it runs as a non-root user and carries its source and revision labels, starts ingress with the image's default command and probes /health, then starts the worker with python -m worker.main and verifies that it Captures a seeded received row. It uses local-only credentials and fake Twilio values; the seeded message has no media, so the smoke run makes no Twilio request.

The smoke run leaves the two local Deposit identities, one content-addressed Deposit object, and a captured intake row with its raw source in the local Supabase stack. Rerunning it reuses the fixed object. Run supabase db reset from site/db when you intentionally want to clear those rows.

Stop the service

From the repository root:

docker compose stop site-db

Stopping the container preserves its named volume. Removing local data is a separate, deliberate Docker operation and is not part of this guide.

The development container and volume carry the Docker labels com.provenience.environment=local-development and com.provenience.scope=site. The integration-test container and volume use com.provenience.environment=local-test and com.provenience.scope=site-backend.