mirror of
https://gitlab.com/vyla-entertainment/documentation.git
synced 2026-11-14 22:47:26 -05:00
fixed syntax issues
This commit is contained in:
@@ -4,32 +4,30 @@
|
||||
|
||||
## Overview
|
||||
|
||||
At the heart of the Vyla ecosystem is the **Provider Layer**. Rather than relying on a single upstream video host, Vyla SDK aggregates over 60 independent, community-maintained scraping resolvers.
|
||||
|
||||
Each provider operates with distinct upstream mechanics, domain rotations, rate-limiting policies, and security defenses. This document explains how Vyla orchestrates these disparate sources into a unified, high-reliability streaming pipeline.
|
||||
Rather than relying on a single upstream host, the Vyla SDK aggregates over 60 independent provider scrapers. Each scraper handles its own domain logic, extraction mechanics, and header requirements.
|
||||
|
||||
---
|
||||
|
||||
## Provider Configuration Interface
|
||||
## Provider Configuration
|
||||
|
||||
In `@vyla-entertainment/sdk`, each provider is declared in `src/config.js` via the `SourceConfig` interface:
|
||||
Scrapers are configured in `src/config.js`:
|
||||
|
||||
```typescript
|
||||
export interface SourceConfig {
|
||||
key: string; // Unique source identifier (e.g. "vidlink", "vixsrc")
|
||||
label: string; // Human-readable display label
|
||||
sourceFile: string; // Scraper module filename in src/sources/
|
||||
proxyParam: string; // Query parameter used for HLS proxy routing
|
||||
timeout: number; // Total milliseconds allowed before source abort
|
||||
sourcesTimeout?: number; // Secondary timeout for multi-server extraction
|
||||
jitter: number; // Maximum randomized startup delay in ms
|
||||
retries: number; // Number of automatic retry attempts on error
|
||||
multiUrl: boolean; // Returns multiple stream candidate URLs if true
|
||||
skipProxy?: boolean; // If true, returns direct upstream URLs without proxy
|
||||
skipVerify?: boolean; // If true, bypasses verification checks
|
||||
verifyHeaders?: boolean; // Injects custom headers into verification probes
|
||||
disabled?: boolean; // If true, provider is completely excluded
|
||||
cdnHeaders?: { // Custom headers injected for specific CDN patterns
|
||||
key: string;
|
||||
label: string;
|
||||
sourceFile: string;
|
||||
proxyParam: string;
|
||||
timeout: number;
|
||||
sourcesTimeout?: number;
|
||||
jitter: number;
|
||||
retries: number;
|
||||
multiUrl: boolean;
|
||||
skipProxy?: boolean;
|
||||
skipVerify?: boolean;
|
||||
verifyHeaders?: boolean;
|
||||
disabled?: boolean;
|
||||
cdnHeaders?: {
|
||||
pattern: RegExp;
|
||||
headers: { Referer: string; Origin: string; };
|
||||
}[];
|
||||
@@ -38,80 +36,56 @@ export interface SourceConfig {
|
||||
|
||||
---
|
||||
|
||||
## The Query Pipeline
|
||||
## Query Pipeline
|
||||
|
||||
When a client requests a title (`getStream()` in SDK or `/movie` / `/tv` in Stream API), the engine executes the following pipeline:
|
||||
|
||||
| Pipeline Stage | Operation | Purpose & Verification |
|
||||
| :--- | :--- | :--- |
|
||||
| **1. Source Filtering** | `SOURCES.filter(!disabled)` | Excludes disabled scrapers and selects active providers based on configuration. |
|
||||
| **2. Jitter Staggering** | Sleep between `0` and `cfg.jitter` ms | Disperses outbound connection spikes across time to prevent DDoS protection triggers. |
|
||||
| **3. Concurrent Scraping** | Parallel async scraper execution | Scrapers parse HTML/JSON endpoints using customized `User-Agent`, `Referer`, and `Origin` headers. |
|
||||
| **4. Two-Phase Verification** | `verifyStream()` then `verifyPlayable()` | Phase 1 validates HTTP status ($< 400$ via HEAD). Phase 2 parses HLS manifests and verifies segment availability. Dead streams are discarded. |
|
||||
| **5. Proxy URL Rewriting** | Encodes target URL into `/api?url=...` | Adds proxy flags, sets required playback headers, and prepares segment de-obfuscation. |
|
||||
| **6. Progressive Emission** | Pushes SSE `source` event | Delivers playable stream candidates to the client player immediately without waiting for slower providers. |
|
||||
1. **Source Filtering**: Excludes disabled scrapers and loads active configurations.
|
||||
2. **Jitter Delay**: Sleeps for a randomized duration between `0` and `cfg.jitter` ms to avoid burst requests.
|
||||
3. **Concurrent Scraping**: Runs scrapers in parallel with provider-specific `User-Agent`, `Referer`, and `Origin` headers.
|
||||
4. **Two-Phase Verification**: Validates candidate URLs via `verifyStream()` (HTTP status < 400) and `verifyPlayable()` (HLS playlist and segment checks).
|
||||
5. **Proxy URL Rewriting**: Encodes verified streams into `/api?url=...` with required proxy flags.
|
||||
6. **Progressive Emission**: Pushes candidate streams over Server-Sent Events (SSE) immediately.
|
||||
|
||||
---
|
||||
|
||||
## Jitter Staggering & Rate-Limit Defense
|
||||
## Jitter Staggering
|
||||
|
||||
If a client triggers 60 scrapers instantaneously, your client IP address could send hundreds of simultaneous HTTP requests in the same millisecond, triggering upstream Cloudflare or DDoS rate limits.
|
||||
To prevent triggering upstream rate limits when querying dozens of scrapers simultaneously, each scraper applies random jitter:
|
||||
|
||||
To mitigate this, Vyla introduces **per-source jitter**:
|
||||
```javascript
|
||||
const jitterDelay = Math.floor(Math.random() * (cfg.jitter || 400));
|
||||
await new Promise(resolve => setTimeout(resolve, jitterDelay));
|
||||
```
|
||||
By staggering outbound requests across a 100ms–800ms window, upstream hosts perceive natural traffic patterns, dramatically reducing HTTP 429 (Too Many Requests) errors.
|
||||
|
||||
This spreads outbound requests across a 100ms–800ms window, reducing HTTP 429 errors.
|
||||
|
||||
---
|
||||
|
||||
## Two-Phase Playback Verification
|
||||
## Two-Phase Verification
|
||||
|
||||
Many streaming extractors successfully return a URL, only for that URL to return an HTTP 403 Forbidden, an empty manifest, or a dead video chunk. To protect players from stalling, Vyla validates every candidate through two independent verification stages before emitting it:
|
||||
1. **Phase 1 (`verifyStream`)**: Sends a lightweight HTTP `HEAD` request with source-specific headers. Discards any response >= 400.
|
||||
2. **Phase 2 (`verifyPlayable`)**: Fetches the initial playlist bytes, verifies the `#EXTM3U` header, confirms playable segment paths, and tests the first chunk or key URI.
|
||||
|
||||
### Phase 1: `verifyStream()`
|
||||
- Dispatches a lightweight HTTP `HEAD` request to the upstream target URL.
|
||||
- Enforces custom `User-Agent`, `Referer`, and `Origin` headers.
|
||||
- Requires HTTP status code `< 400` within 4,000 milliseconds.
|
||||
|
||||
### Phase 2: `verifyPlayable()`
|
||||
- Fetches the initial playlist bytes through the local proxy.
|
||||
- Validates the presence of the `#EXTM3U` header tag.
|
||||
- Checks for stream definitions:
|
||||
- If a Master Playlist: ensures valid `#EXT-X-STREAM-INF` entries exist.
|
||||
- If a Media Playlist: ensures valid `#EXTINF` duration tags and playable segment URIs exist.
|
||||
- Attempts a test read on the first media segment or key URI.
|
||||
- Only candidates that pass both phases are deemed **verified**.
|
||||
Only streams that pass both checks are returned to the client.
|
||||
|
||||
---
|
||||
|
||||
## The Universal HLS Proxy & Byte-Mask Stripper
|
||||
## Proxy & Header Stripping
|
||||
|
||||
Modern streaming hosts often protect their video segments by adding CORS restrictions, token signatures, or binary header masking. Stream API handles this transparently:
|
||||
- **M3U8 Rewriting**: Rewrites AES-128 encryption key URIs and `.ts` segment paths to route through the proxy with CORS headers (`Access-Control-Allow-Origin: *`).
|
||||
- **120-Byte Mask Stripping**: Slices off fake 120-byte image headers prepended by certain CDNs:
|
||||
|
||||
### 1. M3U8 Playlist Rewriting
|
||||
When an HLS stream is proxied, Stream API fetches the upstream playlist and rewrites every internal URI:
|
||||
- **AES-128 Key URIs**: `#EXT-X-KEY:METHOD=AES-128,URI="/api?url=<key>&<param>=1"`
|
||||
- **Video Chunks**: Segment URIs are rewritten to route back through the `/api` proxy endpoint.
|
||||
- **CORS Headers**: Returned with `Access-Control-Allow-Origin: *` and `Access-Control-Allow-Headers: *`.
|
||||
|
||||
### 2. Segment Byte-Mask Stripping
|
||||
Certain major CDNs mask MPEG-TS chunks by prepending a 120-byte fake image header (`image/png`, `image/jpeg`, TikTok CDN, or `letsgocdn` signatures) so firewalls treat them as images.
|
||||
|
||||
When Stream API proxies these segments, it inspects the leading bytes and automatically strips the 120-byte image wrapper:
|
||||
```javascript
|
||||
const needsStrip = searchParams.has('tt') || STRIP_REGEX.test(cleanUrl);
|
||||
if (needsStrip && (chunk[0] === 0x89 || chunk[0] === 0xFF || chunk[0] === 0x00)) {
|
||||
return chunk.subarray(120); // Strips fake header, leaving clean MPEG-TS
|
||||
return chunk.subarray(120);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Inter-Process Caching
|
||||
## In-Memory Caching
|
||||
|
||||
Verified stream results are cached in-memory with a **5-minute Time-To-Live (TTL)**:
|
||||
- **Cache Key**: `source_key + tmdb_id + season + episode`
|
||||
- In multi-core cluster deployments, cache entries are coordinated across workers via inter-process communication (IPC) messages (`cache:get`, `cache:hit`, `cache:set`).
|
||||
- Automatic prune routines run every 30 seconds to evict expired items and enforce maximum cache limits (default: 1,500 entries).
|
||||
Verified stream candidates are cached in-memory for 5 minutes (300,000 ms):
|
||||
- Cache keys format: `source_key + tmdb_id + season + episode`.
|
||||
- In cluster mode, workers coordinate entries via IPC messages (`cache:get`, `cache:hit`, `cache:set`).
|
||||
- Expired entries are pruned every 30 seconds, maintaining a 1,500-entry ceiling.
|
||||
|
||||
Reference in New Issue
Block a user