A CPU profile during a traffic spike (4,850 sockets, 74 uploads/s) showed
the backend spending ~28% of its busy time in two loops over every
connected socket: notify_clients on each upload, and addSystemClients on
each /systems request.
- Add client_registry.js: sockets in a Map, with active clients indexed
by system plus a firehose set. notify_clients only visits the
uploaded call's system, with the same matching rules as before.
- /systems: counts come from the index, the serialized body is reused
for 5s, and counts now reset to 0 when a system's listeners leave.
Firehose clients no longer count toward a system.
- nginx: 10s microcache for /systems on api.*.
- Test: node --test test/client_registry.test.js checks recipients and
counts against the old full-scan implementation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A traffic surge exhausted nginx's 3 x 4096 worker_connections (each
proxied websocket uses two), so Cloudflare got empty responses (520)
and the backend's listen queue filled.
- worker_processes auto (8 cores), worker_connections 16384,
worker_rlimit_nofile 65535.
- Cache public /calls endpoints on api.*: 5s for live lists, 60s for
historical ones, with cache lock and stale-while-updating so bursts
collapse to one backend query. Origin is in the cache key because
the backend's CORS header varies by Origin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
patches was an implicit global set before an await, so concurrent uploads
could save each other's patch lists.
Replace the stale 2024 lockfile with the one from the running production
image (436 packages, verified identical) and install with npm ci, so
uncached rebuilds stop pulling newer versions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rejected uploads (unknown system, bad key, invalid file or source list) and
uploads that threw returned without deleting multer's temp file, leaking
751k files (39 GB) in the backend container since April.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Node main thread was saturated at about 39 uploads/s, overflowing the
listen queue so uploads timed out connecting from nginx.
- Enable keepAlive on the S3 agent; every upload did a full TLS handshake
- Remove OpenTelemetry tracing
- Cache the system key lookup for 60s
- Use lean queries when listing calls
- Serialize each call once in notify_clients, not per recipient
- Add an nginx upstream keepalive pool, and hold Node's keepAliveTimeout
above it so nginx never reuses a connection Node has closed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instructions to get Trunk-Server dev running entirely Local using MIN.IO for storage. Please take note that this does NOT cover SSL and should not be used as an edge facing service as it is not secure.