# Provider Architecture & Mechanics

<p class="page-description">How Vyla SDK sources are structured, scheduled, verified, proxied, and cached across the ecosystem.</p>

## Overview

Rather than relying on a single upstream host, the Vyla SDK aggregates over 60 independent provider scrapers. Each scraper handles its own domain logic, extraction mechanics, and header requirements.

---

## Provider Configuration

Scrapers are configured in `src/config.js`:

```typescript
export interface SourceConfig {
    key: string;
    label: string;
    sourceFile: string;
    proxyParam: string;
    timeout: number;
    sourcesTimeout?: number;
    jitter: number;
    retries: number;
    multiUrl: boolean;
    skipProxy?: boolean;
    skipVerify?: boolean;
    verifyHeaders?: boolean;
    disabled?: boolean;
    cdnHeaders?: {
        pattern: RegExp;
        headers: { Referer: string; Origin: string; };
    }[];
}
```

---

## Query Pipeline

1. **Source Filtering**: Excludes disabled scrapers and loads active configurations.
2. **Jitter Delay**: Sleeps for a randomized duration between `0` and `cfg.jitter` ms to avoid burst requests.
3. **Concurrent Scraping**: Runs scrapers in parallel with provider-specific `User-Agent`, `Referer`, and `Origin` headers.
4. **Two-Phase Verification**: Validates candidate URLs via `verifyStream()` (HTTP status < 400) and `verifyPlayable()` (HLS playlist and segment checks).
5. **Proxy URL Rewriting**: Encodes verified streams into `/api?url=...` with required proxy flags.
6. **Progressive Emission**: Pushes candidate streams over Server-Sent Events (SSE) immediately.

---

## Jitter Staggering

To prevent triggering upstream rate limits when querying dozens of scrapers simultaneously, each scraper applies random jitter:

```javascript
const jitterDelay = Math.floor(Math.random() * (cfg.jitter || 400));
await new Promise(resolve => setTimeout(resolve, jitterDelay));
```

This spreads outbound requests across a 100ms–800ms window, reducing HTTP 429 errors.

---

## Two-Phase Verification

1. **Phase 1 (`verifyStream`)**: Sends a lightweight HTTP `HEAD` request with source-specific headers. Discards any response >= 400.
2. **Phase 2 (`verifyPlayable`)**: Fetches the initial playlist bytes, verifies the `#EXTM3U` header, confirms playable segment paths, and tests the first chunk or key URI.

Only streams that pass both checks are returned to the client.

---

## Proxy & Header Stripping

- **M3U8 Rewriting**: Rewrites AES-128 encryption key URIs and `.ts` segment paths to route through the proxy with CORS headers (`Access-Control-Allow-Origin: *`).
- **120-Byte Mask Stripping**: Slices off fake 120-byte image headers prepended by certain CDNs:

```javascript
const needsStrip = searchParams.has('tt') || STRIP_REGEX.test(cleanUrl);
if (needsStrip && (chunk[0] === 0x89 || chunk[0] === 0xFF || chunk[0] === 0x00)) {
    return chunk.subarray(120);
}
```

---

## In-Memory Caching

Verified stream candidates are cached in-memory for 5 minutes (300,000 ms):
- Cache keys format: `source_key + tmdb_id + season + episode`.
- In cluster mode, workers coordinate entries via IPC messages (`cache:get`, `cache:hit`, `cache:set`).
- Expired entries are pruned every 30 seconds, maintaining a 1,500-entry ceiling.