mirror of
https://gitlab.com/vyla-entertainment/documentation.git
synced 2026-11-14 22:47:26 -05:00
118 lines
6.5 KiB
Markdown
118 lines
6.5 KiB
Markdown
# Provider Architecture & Mechanics
|
||
|
||
<p class="page-description">How Vyla SDK sources are structured, scheduled, verified, proxied, and cached across the ecosystem.</p>
|
||
|
||
## Overview
|
||
|
||
At the heart of the Vyla ecosystem is the **Provider Layer**. Rather than relying on a single upstream video host, Vyla SDK aggregates over 60 independent, community-maintained scraping resolvers.
|
||
|
||
Each provider operates with distinct upstream mechanics, domain rotations, rate-limiting policies, and security defenses. This document explains how Vyla orchestrates these disparate sources into a unified, high-reliability streaming pipeline.
|
||
|
||
---
|
||
|
||
## Provider Configuration Interface
|
||
|
||
In `@vyla-entertainment/sdk`, each provider is declared in `src/config.js` via the `SourceConfig` interface:
|
||
|
||
```typescript
|
||
export interface SourceConfig {
|
||
key: string; // Unique source identifier (e.g. "vidlink", "vixsrc")
|
||
label: string; // Human-readable display label
|
||
sourceFile: string; // Scraper module filename in src/sources/
|
||
proxyParam: string; // Query parameter used for HLS proxy routing
|
||
timeout: number; // Total milliseconds allowed before source abort
|
||
sourcesTimeout?: number; // Secondary timeout for multi-server extraction
|
||
jitter: number; // Maximum randomized startup delay in ms
|
||
retries: number; // Number of automatic retry attempts on error
|
||
multiUrl: boolean; // Returns multiple stream candidate URLs if true
|
||
skipProxy?: boolean; // If true, returns direct upstream URLs without proxy
|
||
skipVerify?: boolean; // If true, bypasses verification checks
|
||
verifyHeaders?: boolean; // Injects custom headers into verification probes
|
||
disabled?: boolean; // If true, provider is completely excluded
|
||
cdnHeaders?: { // Custom headers injected for specific CDN patterns
|
||
pattern: RegExp;
|
||
headers: { Referer: string; Origin: string; };
|
||
}[];
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## The Query Pipeline
|
||
|
||
When a client requests a title (`getStream()` in SDK or `/movie` / `/tv` in Stream API), the engine executes the following pipeline:
|
||
|
||
| Pipeline Stage | Operation | Purpose & Verification |
|
||
| :--- | :--- | :--- |
|
||
| **1. Source Filtering** | `SOURCES.filter(!disabled)` | Excludes disabled scrapers and selects active providers based on configuration. |
|
||
| **2. Jitter Staggering** | Sleep between `0` and `cfg.jitter` ms | Disperses outbound connection spikes across time to prevent DDoS protection triggers. |
|
||
| **3. Concurrent Scraping** | Parallel async scraper execution | Scrapers parse HTML/JSON endpoints using customized `User-Agent`, `Referer`, and `Origin` headers. |
|
||
| **4. Two-Phase Verification** | `verifyStream()` then `verifyPlayable()` | Phase 1 validates HTTP status ($< 400$ via HEAD). Phase 2 parses HLS manifests and verifies segment availability. Dead streams are discarded. |
|
||
| **5. Proxy URL Rewriting** | Encodes target URL into `/api?url=...` | Adds proxy flags, sets required playback headers, and prepares segment de-obfuscation. |
|
||
| **6. Progressive Emission** | Pushes SSE `source` event | Delivers playable stream candidates to the client player immediately without waiting for slower providers. |
|
||
|
||
---
|
||
|
||
## Jitter Staggering & Rate-Limit Defense
|
||
|
||
If a client triggers 60 scrapers instantaneously, your client IP address could send hundreds of simultaneous HTTP requests in the same millisecond, triggering upstream Cloudflare or DDoS rate limits.
|
||
|
||
To mitigate this, Vyla introduces **per-source jitter**:
|
||
```javascript
|
||
const jitterDelay = Math.floor(Math.random() * (cfg.jitter || 400));
|
||
await new Promise(resolve => setTimeout(resolve, jitterDelay));
|
||
```
|
||
By staggering outbound requests across a 100ms–800ms window, upstream hosts perceive natural traffic patterns, dramatically reducing HTTP 429 (Too Many Requests) errors.
|
||
|
||
---
|
||
|
||
## Two-Phase Playback Verification
|
||
|
||
Many streaming extractors successfully return a URL, only for that URL to return an HTTP 403 Forbidden, an empty manifest, or a dead video chunk. To protect players from stalling, Vyla validates every candidate through two independent verification stages before emitting it:
|
||
|
||
### Phase 1: `verifyStream()`
|
||
- Dispatches a lightweight HTTP `HEAD` request to the upstream target URL.
|
||
- Enforces custom `User-Agent`, `Referer`, and `Origin` headers.
|
||
- Requires HTTP status code `< 400` within 4,000 milliseconds.
|
||
|
||
### Phase 2: `verifyPlayable()`
|
||
- Fetches the initial playlist bytes through the local proxy.
|
||
- Validates the presence of the `#EXTM3U` header tag.
|
||
- Checks for stream definitions:
|
||
- If a Master Playlist: ensures valid `#EXT-X-STREAM-INF` entries exist.
|
||
- If a Media Playlist: ensures valid `#EXTINF` duration tags and playable segment URIs exist.
|
||
- Attempts a test read on the first media segment or key URI.
|
||
- Only candidates that pass both phases are deemed **verified**.
|
||
|
||
---
|
||
|
||
## The Universal HLS Proxy & Byte-Mask Stripper
|
||
|
||
Modern streaming hosts often protect their video segments by adding CORS restrictions, token signatures, or binary header masking. Stream API handles this transparently:
|
||
|
||
### 1. M3U8 Playlist Rewriting
|
||
When an HLS stream is proxied, Stream API fetches the upstream playlist and rewrites every internal URI:
|
||
- **AES-128 Key URIs**: `#EXT-X-KEY:METHOD=AES-128,URI="/api?url=<key>&<param>=1"`
|
||
- **Video Chunks**: Segment URIs are rewritten to route back through the `/api` proxy endpoint.
|
||
- **CORS Headers**: Returned with `Access-Control-Allow-Origin: *` and `Access-Control-Allow-Headers: *`.
|
||
|
||
### 2. Segment Byte-Mask Stripping
|
||
Certain major CDNs mask MPEG-TS chunks by prepending a 120-byte fake image header (`image/png`, `image/jpeg`, TikTok CDN, or `letsgocdn` signatures) so firewalls treat them as images.
|
||
|
||
When Stream API proxies these segments, it inspects the leading bytes and automatically strips the 120-byte image wrapper:
|
||
```javascript
|
||
const needsStrip = searchParams.has('tt') || STRIP_REGEX.test(cleanUrl);
|
||
if (needsStrip && (chunk[0] === 0x89 || chunk[0] === 0xFF || chunk[0] === 0x00)) {
|
||
return chunk.subarray(120); // Strips fake header, leaving clean MPEG-TS
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Inter-Process Caching
|
||
|
||
Verified stream results are cached in-memory with a **5-minute Time-To-Live (TTL)**:
|
||
- **Cache Key**: `source_key + tmdb_id + season + episode`
|
||
- In multi-core cluster deployments, cache entries are coordinated across workers via inter-process communication (IPC) messages (`cache:get`, `cache:hit`, `cache:set`).
|
||
- Automatic prune routines run every 30 seconds to evict expired items and enforce maximum cache limits (default: 1,500 entries).
|