huge rewrite

stream-api/reference/operations.md

@@ -1,37 +1,107 @@
-# Health, testing, and safe operations
+# Operations, Health & Diagnostics
 
-<p class="page-description">Observe SDK providers and diagnose a Stream API deployment without making debug data public.</p>
+<p class="page-description">Monitoring, cluster diagnostics, live health probing, and testing endpoints in Stream API.</p>
 
-## Health
+## The Health Check Endpoint: `GET /health`
 
-`GET /health` (or `/api/health`) probes active providers and returns a JSON summary. It includes a status, timestamp, TMDB-key presence, cache size, and a map of provider keys to `{ ok, ms }` results.
+Stream API provides an automated health probe endpoint that executes live responsiveness checks across all registered scraper modules.
 
-Health is a signal, not a promise of playback. Provider behavior can differ by title, client network, and the moment of the request. Use it for observability and source selection-not as a guarantee that a viewer will receive a stream.
-
-## Test one provider
-
-`GET /test/:id?source=:providerKey` resolves a single provider for a movie. Add `season` and `episode` (or `s` and `e`) for TV.
-
-```bash
-curl "YOUR_API_URL/test/155?source=provider-key"
-curl "YOUR_API_URL/test/1396?season=1&episode=1&source=provider-key"
+```http
+GET /health HTTP/1.1
+Host: your-api.example.com
 ```
 
-Use source keys reported by the running deployment. A test result reports whether resolution and verification succeeded; clients should inspect the result body rather than relying only on a transport success.
+### Response Status Codes
+- **`200 OK`**: All active provider sources passed their health probes.
+- **`207 Multi-Status`**: One or more providers failed their probes (degraded state), but the API gateway remains functional.
 
-## Debug route
+### Example Response Payload
+```json
+{
+  "note": "This is a health check endpoint. This is extremely unreliable, be sure to test sources directly.",
+  "status": "ok",
+  "timestamp": "2026-11-14T20:30:00.000Z",
+  "tmdb": true,
+  "cache": 142,
+  "probe_id": 155,
+  "sources": {
+    "vidlink": { "ok": true, "ms": 420 },
+    "vixsrc": { "ok": true, "ms": 610 },
+    "cinecat": { "ok": true, "ms": 890 },
+    "movy": { "ok": false, "ms": null }
+  }
+}
+```
 
-`GET /debug/:id?source=:providerKey` offers detailed candidate and request traces, but it is only available when `ENABLE_DEBUG_ROUTE=true`.
+#### Response Fields
+| Field | Type | Description |
+| :--- | :--- | :--- |
+| `status` | `string` | Overall system state (`"ok"` or `"degraded"`). |
+| `timestamp` | `string` | ISO 8601 timestamp of probe completion. |
+| `tmdb` | `boolean` | `true` if `TMDB_API_KEY` is configured and valid in the environment. |
+| `cache` | `integer` | Current number of active cached titles in the primary LRU cache. |
+| `probe_id` | `integer` | The standard TMDB ID used to probe source responsiveness. |
+| `sources` | `object` | Dictionary of per-source availability (`ok`) and roundtrip latency in milliseconds (`ms`). |
+
+---
+
+## Direct Source Testing: `GET /test/:source`
+
+To test an individual scraper directly without querying all 60 providers:
+
+```http
+GET /test/vidlink?id=155 HTTP/1.1
+```
+
+For television episodes, pass season and episode query parameters:
+```http
+GET /test/vixsrc?id=1396&s=1&e=1 HTTP/1.1
+```
+
+### Response
+Returns the raw resolved candidate object directly from the provider scraper:
+
+```json
+{
+  "source": "vidlink",
+  "url": "https://upstream.example.com/playlist.m3u8",
+  "quality": "1080p",
+  "type": "hls",
+  "server": "vidlink-cf"
+}
+```
+
+---
+
+## Deep Debug Route: `GET /debug/:source`
+
+When `ENABLE_DEBUG_ROUTE=true` is set in your environment variables, Stream API exposes a comprehensive diagnostic route:
+
+```http
+GET /debug/vidlink?id=155 HTTP/1.1
+```
+
+### Diagnostic Output Includes:
+1. **Raw Scraper Payload**: Unmodified JSON or HTML received from the upstream host.
+2. **Extraction Timings**: Milliseconds spent parsing DOM, solving tokens, and resolving URLs.
+3. **Phase 1 Verification**: Status code, response headers, and latency of the `HEAD` request.
+4. **Phase 2 Verification**: First 500 bytes of the master M3U8 playlist and test chunk fetch status.
+5. **Rewritten URLs**: Preview of how chunk URIs will be rewritten by the proxy.
 
 :::callout-warning
-Do not enable the debug route on a public deployment. It can expose upstream request data, raw candidates, and headers. Enable it temporarily in a controlled environment, then disable it again.
+**Security Note**: Keep `ENABLE_DEBUG_ROUTE=false` in public production environments to avoid leaking internal scraper tokens or upstream endpoints.
 :::
 
-## Production checklist
+---
 
-- Set `TMDB_API_KEY` when metadata-dependent sources are required.
-- Keep `ENABLE_DEBUG_ROUTE=false` outside controlled troubleshooting.
-- Use TLS and enforce your own access policy before making the service remotely reachable.
-- Monitor health trends and playback failures separately.
-- Run updates and provider changes through your normal deployment process.
-- Build player fallbacks instead of assuming a specific provider is always available.
+## Cluster & Cache Architecture
+
+### Primary Supervisor & Worker Pool Architecture
+- **Central In-Memory Cache**: Holds up to 1,500 verified title entries in a shared LRU table with a 5-minute TTL.
+- **Periodic Pruner**: Runs every 30 seconds to clean expired keys and enforce memory ceilings.
+- **Worker Lifecycle Supervisor**: Spawns and supervises $N$ worker processes matching CPU core count, automatically restarting terminated workers.
+- **IPC Message Dispatcher**: Routes `cache:get` and `cache:set` messages between worker threads and the primary process.
+
+1. **IPC Cache Synchronization**: When a worker resolves a title, it sends a `cache:set` message to the primary process. When another worker receives a request for the same title, it sends a `cache:get` message and receives an instant `cache:hit` without re-scraping upstream hosts.
+2. **Cache Pruning**: The primary process evicts stale items older than 300,000ms (5 minutes). If total cache size exceeds 1,500 items, the oldest entries are evicted immediately.
+3. **Worker Auto-Restart**: If a worker process exits unexpectedly or exceeds memory limits, the primary cluster supervisor automatically spawns a replacement worker.