A CPU profile during a traffic spike (4,850 sockets, 74 uploads/s) showed
the backend spending ~28% of its busy time in two loops over every
connected socket: notify_clients on each upload, and addSystemClients on
each /systems request.
- Add client_registry.js: sockets in a Map, with active clients indexed
by system plus a firehose set. notify_clients only visits the
uploaded call's system, with the same matching rules as before.
- /systems: counts come from the index, the serialized body is reused
for 5s, and counts now reset to 0 when a system's listeners leave.
Firehose clients no longer count toward a system.
- nginx: 10s microcache for /systems on api.*.
- Test: node --test test/client_registry.test.js checks recipients and
counts against the old full-scan implementation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A traffic surge exhausted nginx's 3 x 4096 worker_connections (each
proxied websocket uses two), so Cloudflare got empty responses (520)
and the backend's listen queue filled.
- worker_processes auto (8 cores), worker_connections 16384,
worker_rlimit_nofile 65535.
- Cache public /calls endpoints on api.*: 5s for live lists, 60s for
historical ones, with cache lock and stale-while-updating so bursts
collapse to one backend query. Origin is in the cache key because
the backend's CORS header varies by Origin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Node main thread was saturated at about 39 uploads/s, overflowing the
listen queue so uploads timed out connecting from nginx.
- Enable keepAlive on the S3 agent; every upload did a full TLS handshake
- Remove OpenTelemetry tracing
- Cache the system key lookup for 60s
- Use lean queries when listing calls
- Serialize each call once in notify_clients, not per recipient
- Add an nginx upstream keepalive pool, and hold Node's keepAliveTimeout
above it so nginx never reuses a connection Node has closed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>