Self-hosting

Secrets and key rotation

Authenticated Redis, NATS and ClickHouse, what production mode refuses, and how to rotate the key that seals stored credentials without downtime.

Metrixed stores third-party credentials (payment provider keys, e-mail and SMS provider settings, BI connector credentials, SSO client secrets, webhook and SDK signing secrets, MFA seeds, tax IDs) sealed with AES-256-GCM under DT_SECRETS_KEY. The data stores it talks to (Redis, NATS, ClickHouse) require credentials in production.

Authenticated infrastructure

With DT_ENV=production every service refuses to start when a connection it will use carries no credentials, the same way it refuses dev passwords:

The production compose profile (deploy/production) wires all of it: gen-secrets.sh generates DT_REDIS_PASSWORD, DT_NATS_PASSWORD and DT_CH_PASSWORD, Redis runs with --requirepass, NATS with a user and password, ClickHouse with a named user (the default user is removed). make deploy-verify checks that each store turns away an anonymous client and that each service image refuses unauthenticated settings. The development stack stays passwordless: none of this applies unless DT_ENV=production.

Rotating DT_SECRETS_KEY

Every sealed value is tagged with the id of the key that sealed it (a short hash of the key, not the key). Services read with a keyring: DT_SECRETS_KEY plus the retired keys in DT_SECRETS_KEY_PREVIOUS (comma-separated, newest first). They always seal with DT_SECRETS_KEY. Values written before tagging existed are opened by trying each key in turn.

1. Generate a new key and keep the old one readable:

   shCopy

   # .env
   DT_SECRETS_KEY_PREVIOUS=<the current DT_SECRETS_KEY>   # prepend if the list already has keys
   DT_SECRETS_KEY=<new 64 hex characters: od -An -tx1 -N32 /dev/urandom | tr -d ' \n'>
   

2. Restart the services (docker compose ... up -d). New writes use the new key; old ciphertexts keep opening.

3. Re-seal what is stored, first as a dry run:

   shCopy

   docker compose -f deploy/production/docker-compose.yml --env-file deploy/production/.env -p dtrackier \
     run --rm workers rotate-secrets -dry-run
   docker compose -f deploy/production/docker-compose.yml --env-file deploy/production/.env -p dtrackier \
     run --rm workers rotate-secrets
   

   The command walks every sealed column in primary-key batches. Each batch commits on its own, so an interrupted run is resumed by running it again; values already on the new key are skipped, and every update is compare-and-swap, so a secret edited during the run is never overwritten (it is reported as raced and picked up by the next run). It prints per-column counts and writes one operator_audit entry (secrets.rotate) with the new key id, the retired key ids and the counts. Exit status 0 means nothing is left on a retired key; 1 means some values could not be opened with any key in the ring (listed by primary key) or changed mid-run; 2 is an error. -json prints the report as JSON, -batch sets the batch size, -actor the e-mail recorded in the audit entry.

4. Keep the retired key in DT_SECRETS_KEY_PREVIOUS while links signed with it should keep working: e-mail click and one-click unsubscribe links, and MCP client ids issued before the rotation. Then remove it and restart.

A value whose key is not in the ring is refused (it never decrypts to garbage). Keep each retired key with the backups taken while it was current: restoring such a backup needs it in DT_SECRETS_KEY_PREVIOUS, then a rotation run.