A CPU profile during a traffic spike (4,850 sockets, 74 uploads/s) showed
the backend spending ~28% of its busy time in two loops over every
connected socket: notify_clients on each upload, and addSystemClients on
each /systems request.
- Add client_registry.js: sockets in a Map, with active clients indexed
by system plus a firehose set. notify_clients only visits the
uploaded call's system, with the same matching rules as before.
- /systems: counts come from the index, the serialized body is reused
for 5s, and counts now reset to 0 when a system's listeners leave.
Firehose clients no longer count toward a system.
- nginx: 10s microcache for /systems on api.*.
- Test: node --test test/client_registry.test.js checks recipients and
counts against the old full-scan implementation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
patches was an implicit global set before an await, so concurrent uploads
could save each other's patch lists.
Replace the stale 2024 lockfile with the one from the running production
image (436 packages, verified identical) and install with npm ci, so
uncached rebuilds stop pulling newer versions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rejected uploads (unknown system, bad key, invalid file or source list) and
uploads that threw returned without deleting multer's temp file, leaking
751k files (39 GB) in the backend container since April.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Node main thread was saturated at about 39 uploads/s, overflowing the
listen queue so uploads timed out connecting from nginx.
- Enable keepAlive on the S3 agent; every upload did a full TLS handshake
- Remove OpenTelemetry tracing
- Cache the system key lookup for 60s
- Use lean queries when listing calls
- Serialize each call once in notify_clients, not per recipient
- Add an nginx upstream keepalive pool, and hold Node's keepAliveTimeout
above it so nginx never reuses a connection Node has closed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>