A traffic surge exhausted nginx's 3 x 4096 worker_connections (each
proxied websocket uses two), so Cloudflare got empty responses (520)
and the backend's listen queue filled.
- worker_processes auto (8 cores), worker_connections 16384,
worker_rlimit_nofile 65535.
- Cache public /calls endpoints on api.*: 5s for live lists, 60s for
historical ones, with cache lock and stale-while-updating so bursts
collapse to one backend query. Origin is in the cache key because
the backend's CORS header varies by Origin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Node main thread was saturated at about 39 uploads/s, overflowing the
listen queue so uploads timed out connecting from nginx.
- Enable keepAlive on the S3 agent; every upload did a full TLS handshake
- Remove OpenTelemetry tracing
- Cache the system key lookup for 60s
- Use lean queries when listing calls
- Serialize each call once in notify_clients, not per recipient
- Add an nginx upstream keepalive pool, and hold Node's keepAliveTimeout
above it so nginx never reuses a connection Node has closed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>