Self-hosting
Sizing and capacity
How much traffic a node takes, what each component costs per click and postback, Redis memory and NATS disk per click, node shapes for 10 M to 1 B clicks a month, and the alerts that signal saturation.
Every component of Metrixed is measured the same way: a full stack (edge, ingest, API, reporting, workers, Redis, NATS JetStream, ClickHouse and the load generator) on two vCPUs, driven with clicks and postbacks while the accuracy checks confirm every click, conversion, postback, webhook and ledger entry landed exactly once. The full method, the before and after numbers and the formulas are in docs/ops/SIZING.md in the repository.
What a node takes
A node's capacity is the highest rate at which the redirect and the postback answer stay under 50 ms at the 99th percentile and at most 5 % of postbacks are answered queued.
| Rate (clicks/s + postbacks/s), 2 vCPU | Redirect p99 | Postback p99 | Answered queued |
|---|---|---|---|
| 300 + 50 | 11 ms | 26 ms | 0 % |
| 450 + 75 | 19 ms | 51 ms | 0 % |
| 600 + 100 | 26 ms | 66 ms | 0 % |
| 750 + 125 | 39 ms | 133 ms | 0 % |
| 900 + 150 | 108 ms | 515 ms | 0 % |
That is roughly 250 events per second per vCPU with everything on the node, Postgres included. Past about 90 % CPU the latency tail grows quickly. Answers stay exact: an over-capacity node answers queued and replays every queued postback exactly once. A shared 2-vCPU CI runner holds about a quarter of this.
Cost per component
At 600 clicks/s and 116 postbacks/s on two vCPUs:
| Component | vCPU used | Notes |
|---|---|---|
| edge | 0.17 | about 1.25 Redis round trips per click, about 3 300 clicks/s per vCPU |
| ingest | 0.19 | three Redis round trips per postback (one more per cap pass on a capped offer); the click stream it consumes is kept in memory |
| workers | 0.43 | ClickHouse writers, ledger, postbacks, webhooks |
| ClickHouse | 0.37 | one insert per table per flush window (DT_CHW_FLUSH_MS) |
| NATS JetStream | 0.18 | |
| Redis | 0.16 | about 15 commands per event |
| Postgres | 0.35 | ledger, postbacks, webhooks, fraud queue |
Memory and disk
Redis memory sets the shape before CPU does. Each click is kept once, as a compact record of about 0.4 KB (it was about 1 KB of JSON, plus a second 1 KB copy for the mobile index), for the network's hot window: its longest offer conversion window plus one day, between 1 and 45 days (DT_CLICK_CACHE_TTL sets a fixed window). Older clicks are looked up in ClickHouse when a postback names them, without changing the answer. Each conversion keeps about 0.9 KB of state (a compact record, its de-duplication claims and its transaction index) for 45 days (DT_CONV_HOT_TTL; it was 1.4 KB for 400 days); older conversions are read from ClickHouse and Postgres when a status change or a resend names them.
| Clicks a month (one conversion per 20 clicks) | Redis in October 2026 | Redis now (30-day offer windows) |
|---|---|---|
| 10 M | about 36 GB | about 6 GB |
| 100 M | about 350 GB | about 55 GB |
| 1 B | about 3.5 TB | about 0.6 TB |
The click part fell about five times (from about 77 KB to 14.5 KB per click per day of traffic) and the conversion part about fourteen times. Postgres keeps every conversion's de-duplication keys for 400 days (conversion_keys, about 0.65 KB per conversion on disk: 43 GB at 100 M clicks a month). Leave 30 % headroom: Redis runs with noeviction, and a full Redis refuses writes.
Postbacks for clicks past the hot window cost about 2 ms of ClickHouse CPU each. With the default window they are rare (they arrive after their offer's window). A node that runs everything resolves about 40 of them a second on 2 vCPU, 100 on 4 and 200 on 8 (next to two to three times as many clicks) with the answer under 50 ms; if ClickHouse cannot answer in 60 ms the postback is answered queued and replayed, never matched to no click.
NATS JetStream keeps about 1.23 KB per message, roughly one message per click and three per conversion. Every stream has a byte cap (DT_JS_MAX_BYTES_<STREAM>): a full stream refuses new events, which the edge and ingest keep in their spools until there is room, and nothing is dropped. Near a cap, messages every consumer has already processed are purged first. The production profile caps the click stream at 16 GiB (about four days at 100 M clicks a month) and all streams at 32 GiB together, which must fit the NATS volume.
Node shapes
Assuming a peak of four times the month's average, one postback per 20 clicks and at most 60 % CPU at peak:
| Volume | Peak clicks/s | Shape |
|---|---|---|
| 10 M clicks a month | 15 | one node, 2 vCPU / 16 to 32 GB, everything on it |
| 100 M clicks a month | 155 | app node 4 vCPU / 16 GB (NATS volume at least 48 GB), Redis 2 vCPU / 96 to 128 GB, Postgres managed or 2 vCPU / 8 GB |
| 1 B clicks a month | 1 550 | split by role: edge 2 × 2 vCPU, ingest 2 × 2 vCPU, workers 2 × 4 vCPU, ClickHouse 8 vCPU / 32 GB, Postgres 4 vCPU / 16 GB with a replica, NATS 3 × 2 vCPU; Redis about 0.8 TB (a Redis Cluster) |
Edge and ingest scale out freely; workers scale by role (DT_WORKERS_ONLY); ClickHouse and Redis scale up first.
Signs of saturation
The dtrackier-saturation alert group in infra/observability/prometheus/alerts.yml:
- IngestDeferralRateHigh / Severe: more than 5 % (25 %) of postbacks answered
queued.dt_ingest_deferred_by_reason_totalnames the cause. - IngestRedisSlow: ingest's Redis p99 above 50 ms. Redis or the node's CPU is the bottleneck.
- EdgeRedisSlow: the edge's Redis p99 above 25 ms, half of its 50 ms budget; past the budget repeat clicks count as unique.
- RedisCircuitOpening: edge or ingest stopped calling Redis for a cooldown.
- Existing: EdgeStateDegraded and ConsumerLagHigh / Growing.
The dtrackier-jetstream group watches the event streams:
- JetStreamStreamFilling: a stream above 90 % of its cap for 10 minutes. A consumer is behind.
- JetStreamStreamFull: a stream at its cap. New events wait in the edge and ingest spools.
- JetStreamPublishesRefused: more than 1 000 events waiting in the spools while publishing fails.
- JetStreamStreamUncapped: a stream without a cap (the caps did not fit the NATS storage).
Profiling a node
Set DT_PPROF_ADDR=127.0.0.1:6060 (loopback only) on edge, ingest or workers to serve Go CPU and allocation profiles, then take a 30-second CPU profile while the node is busy: go tool pprof -top -cum http://127.0.0.1:6060/debug/pprof/profile?seconds=30.