Self-hosting

Observability

Health endpoints, Prometheus metrics, traces, logs, dashboards, alerts and the synthetic canary.

Every service is observable the same way, and the repository includes a ready-made stack.

Endpoints and logs

/healthz: the process is alive. /readyz: its dependencies answer. /metrics: Prometheus text format.
Logs are structured JSON (log/slog). Every request carries an X-Request-Id, which also appears in API error bodies and logs; the NATS and HTTP hops between services propagate the same id and trace context where the sender has a context.
Settings, System health in the admin portal shows live readiness of each service, consumer lag, postback outcomes and the latest synthetic run (GET /status/health). DT_HEALTH_TARGETS tells the API which services to probe.

Metrics

The edge keeps its own lean metrics (the hot path is deliberately not wrapped); the other services use a shared RED middleware.

Metric	Meaning
`dt_edge_redirect_seconds`	Handler latency histogram of the edge.
`dt_edge_clicks_total`	Clicks by link style, redirect mode and invalid reason.
`dt_edge_events_published_total`, `dt_edge_events_dropped_total`, `dt_edge_events_failed_total`	Click-event publication to NATS.
`dt_edge_snapshot_age_seconds`, `dt_edge_snapshot_version`	Freshness of the edge config snapshot.
`dt_ingest_latency_seconds`, `dt_ingest_outcomes_total`	Conversion ingest latency and outcomes (`created`, `duplicate`, `rejected`, ...).
`dt_http_requests_total`, `dt_http_request_duration_seconds`, `dt_http_in_flight`	RED metrics of the API, ingest, reporting and workers HTTP servers.
`dt_consumer_pending`, `dt_consumer_ack_pending`, `dt_consumer_redelivered`, `dt_consumer_max_deliveries_total`	JetStream consumer lag per durable.
`dt_postback_results_total`, `dt_postback_latency_seconds`, `dt_postback_dlq_total`	Outbound postback outcomes and dead letters.
`dt_business_clicks_window`, `dt_business_conversions_window`, `dt_business_revenue_window`	Trailing-window business gauges (`DT_OBS_BIZ_WINDOW_SECONDS`).
`dt_dep_call_seconds`, `dt_dep_errors_total`	Calls to dependencies (ClickHouse, Redis, Postgres).
`dt_synth_*`	Synthetic canary results, consecutive failures and certificate expiry.

The tenant label is bounded (DT_OBS_TENANT_LIMIT) so metric cardinality stays safe on large installs.

Traces

Set DT_OTEL_ENDPOINT (OTLP over HTTP, host:port or a URL) on any service and it exports traces with the service name dtrackier-<service>. Unset, tracing is off and costs nothing.

The bundled stack

bash

docker compose -f infra/observability/docker-compose.observability.yml up -d

This starts Prometheus (scraping the five services, 15 days retention), Grafana, Tempo, Loki and an OpenTelemetry collector, all with pinned versions. Grafana provisioning includes six dashboards: edge, ingest, api, reporting, workers and business, generated by infra/observability/gen_dashboards.py. Point services at the collector with DT_OTEL_ENDPOINT=127.0.0.1:24318.

Alerts

infra/observability/prometheus/alerts.yml ships rules for: redirect p99 above 8 ms, ingest p99 above 25 ms, report API p95 above 800 ms, any service down, API 5xx ratio, consumer lag high or growing, consumer dead letters, postback failure rate, edge event drops, ClickHouse errors, stale config snapshot, synthetic canary failing or silent, slow visibility, and certificates expiring soon or critical.

Business alerts for tenants (cap nearly reached, postback failure rate by partner, conversion-rate drop, fraud spike) are different: they are automation presets inside the product, see the automation reference.

Synthetic canary

The synthetic-monitor worker proves the whole path works from the outside, every DT_SYNTH_INTERVAL_SECONDS (30 by default) and per region: it requests the canary link (DT_SYNTH_LINK on the host DT_SYNTH_HOST), fires a postback for the click it got, and waits for the conversion to become visible. It exports dt_synth_* metrics and certificate expiry for DT_SYNTH_TLS_HOSTS. After DT_SYNTH_FAIL_AFTER consecutive failures (2 by default) it opens an incident on the tenant status page and resolves it after two consecutive successful probes. It is off unless DT_SYNTH_TENANT is set.