How Vyla SDK sources are structured, scheduled, verified, proxied, and cached across the ecosystem.
## Overview At the heart of the Vyla ecosystem is the **Provider Layer**. Rather than relying on a single upstream video host, Vyla SDK aggregates over 60 independent, community-maintained scraping resolvers. Each provider operates with distinct upstream mechanics, domain rotations, rate-limiting policies, and security defenses. This document explains how Vyla orchestrates these disparate sources into a unified, high-reliability streaming pipeline. --- ## Provider Configuration Interface In `@vyla-entertainment/sdk`, each provider is declared in `src/config.js` via the `SourceConfig` interface: ```typescript export interface SourceConfig { key: string; // Unique source identifier (e.g. "vidlink", "vixsrc") label: string; // Human-readable display label sourceFile: string; // Scraper module filename in src/sources/ proxyParam: string; // Query parameter used for HLS proxy routing timeout: number; // Total milliseconds allowed before source abort sourcesTimeout?: number; // Secondary timeout for multi-server extraction jitter: number; // Maximum randomized startup delay in ms retries: number; // Number of automatic retry attempts on error multiUrl: boolean; // Returns multiple stream candidate URLs if true skipProxy?: boolean; // If true, returns direct upstream URLs without proxy skipVerify?: boolean; // If true, bypasses verification checks verifyHeaders?: boolean; // Injects custom headers into verification probes disabled?: boolean; // If true, provider is completely excluded cdnHeaders?: { // Custom headers injected for specific CDN patterns pattern: RegExp; headers: { Referer: string; Origin: string; }; }[]; } ``` --- ## The Query Pipeline When a client requests a title (`getStream()` in SDK or `/movie` / `/tv` in Stream API), the engine executes the following pipeline: | Pipeline Stage | Operation | Purpose & Verification | | :--- | :--- | :--- | | **1. Source Filtering** | `SOURCES.filter(!disabled)` | Excludes disabled scrapers and selects active providers based on configuration. | | **2. Jitter Staggering** | Sleep between `0` and `cfg.jitter` ms | Disperses outbound connection spikes across time to prevent DDoS protection triggers. | | **3. Concurrent Scraping** | Parallel async scraper execution | Scrapers parse HTML/JSON endpoints using customized `User-Agent`, `Referer`, and `Origin` headers. | | **4. Two-Phase Verification** | `verifyStream()` then `verifyPlayable()` | Phase 1 validates HTTP status ($< 400$ via HEAD). Phase 2 parses HLS manifests and verifies segment availability. Dead streams are discarded. | | **5. Proxy URL Rewriting** | Encodes target URL into `/api?url=...` | Adds proxy flags, sets required playback headers, and prepares segment de-obfuscation. | | **6. Progressive Emission** | Pushes SSE `source` event | Delivers playable stream candidates to the client player immediately without waiting for slower providers. | --- ## Jitter Staggering & Rate-Limit Defense If a client triggers 60 scrapers instantaneously, your client IP address could send hundreds of simultaneous HTTP requests in the same millisecond, triggering upstream Cloudflare or DDoS rate limits. To mitigate this, Vyla introduces **per-source jitter**: ```javascript const jitterDelay = Math.floor(Math.random() * (cfg.jitter || 400)); await new Promise(resolve => setTimeout(resolve, jitterDelay)); ``` By staggering outbound requests across a 100ms–800ms window, upstream hosts perceive natural traffic patterns, dramatically reducing HTTP 429 (Too Many Requests) errors. --- ## Two-Phase Playback Verification Many streaming extractors successfully return a URL, only for that URL to return an HTTP 403 Forbidden, an empty manifest, or a dead video chunk. To protect players from stalling, Vyla validates every candidate through two independent verification stages before emitting it: ### Phase 1: `verifyStream()` - Dispatches a lightweight HTTP `HEAD` request to the upstream target URL. - Enforces custom `User-Agent`, `Referer`, and `Origin` headers. - Requires HTTP status code `< 400` within 4,000 milliseconds. ### Phase 2: `verifyPlayable()` - Fetches the initial playlist bytes through the local proxy. - Validates the presence of the `#EXTM3U` header tag. - Checks for stream definitions: - If a Master Playlist: ensures valid `#EXT-X-STREAM-INF` entries exist. - If a Media Playlist: ensures valid `#EXTINF` duration tags and playable segment URIs exist. - Attempts a test read on the first media segment or key URI. - Only candidates that pass both phases are deemed **verified**. --- ## The Universal HLS Proxy & Byte-Mask Stripper Modern streaming hosts often protect their video segments by adding CORS restrictions, token signatures, or binary header masking. Stream API handles this transparently: ### 1. M3U8 Playlist Rewriting When an HLS stream is proxied, Stream API fetches the upstream playlist and rewrites every internal URI: - **AES-128 Key URIs**: `#EXT-X-KEY:METHOD=AES-128,URI="/api?url=