mirror of
https://gitlab.com/vyla-entertainment/documentation.git
synced 2026-11-14 22:47:26 -05:00
huge rewrite
This commit is contained in:
@@ -1,87 +1,117 @@
|
||||
# Providers
|
||||
# Provider Architecture & Mechanics
|
||||
|
||||
<p class="page-description">How Vyla SDK sources are configured, selected, and verified.</p>
|
||||
<p class="page-description">How Vyla SDK sources are structured, scheduled, verified, proxied, and cached across the ecosystem.</p>
|
||||
|
||||
Vyla SDK aggregates multiple independent providers. Each has different internal mechanisms, reliability profiles, and timeout characteristics. This page explains the source layer that powers both direct SDK use and the optional Stream API.
|
||||
## Overview
|
||||
|
||||
:::callout-note
|
||||
Provider availability and performance can vary based on your network environment. When self-hosting, providers may work differently than on residential connections due to IP-based blocking from upstream sources.
|
||||
:::
|
||||
At the heart of the Vyla ecosystem is the **Provider Layer**. Rather than relying on a single upstream video host, Vyla SDK aggregates over 60 independent, community-maintained scraping resolvers.
|
||||
|
||||
Each provider operates with distinct upstream mechanics, domain rotations, rate-limiting policies, and security defenses. This document explains how Vyla orchestrates these disparate sources into a unified, high-reliability streaming pipeline.
|
||||
|
||||
---
|
||||
|
||||
## How Providers Are Used
|
||||
## Provider Configuration Interface
|
||||
|
||||
When your app calls `getStream()`-directly or through Stream API:
|
||||
In `@vyla-entertainment/sdk`, each provider is declared in `src/config.js` via the `SourceConfig` interface:
|
||||
|
||||
1. All active (non-disabled) providers are queried **simultaneously** with per-source random jitter to stagger upstream requests
|
||||
2. Each provider has its own **timeout**, **retry count**, and **backoff delay**
|
||||
3. Every returned URL is **verified** - first via a HEAD request to the raw upstream URL, then via a proxied HLS playability check
|
||||
4. Only URLs that pass both verification steps appear in the response as `source` events
|
||||
|
||||
Because the endpoint is SSE, **verified sources are emitted immediately as each provider resolves** - your player receives them progressively. The stream closes once every provider has either resolved or timed out.
|
||||
|
||||
:::callout-info
|
||||
Call `sdk.getSources(true)` to inspect enabled providers and `sdk.probeAllSources()` for a live availability signal. If you use Stream API, its `/health` endpoint exposes probes over HTTP.
|
||||
:::
|
||||
|
||||
---
|
||||
|
||||
## Provider Configuration
|
||||
|
||||
Each provider is defined with the following properties:
|
||||
|
||||
| Field | Description |
|
||||
| --- | --- |
|
||||
| `key` | Internal identifier used in API responses and the `source` field |
|
||||
| `label` | Human-readable name shown in the `label` field |
|
||||
| `proxyParam` | Short query parameter used to route proxied requests |
|
||||
| `timeout` | Maximum time (ms) to wait for a stream URL |
|
||||
| `sourcesTimeout` | Optional secondary timeout (ms) for an internal source-listing step |
|
||||
| `jitter` | Max random delay (ms) before the request fires |
|
||||
| `retries` | Number of retry attempts on failure |
|
||||
| `skipProxy` | If `true`, stream URLs are returned as-is without proxy wrapping |
|
||||
| `disabled` | If `true`, the provider is excluded from all requests |
|
||||
|
||||
---
|
||||
|
||||
## Response Time
|
||||
|
||||
The first `source` event typically arrives within **3–8 seconds**. Subsequent sources stream in as each provider resolves. The stream closes once the **slowest configured provider** times out.
|
||||
|
||||
**Best practice:** always show a loading state, begin playback on the first `source` event, and implement a client-side timeout for the overall stream. See the [Error Handling](/guides/error-handling) guide.
|
||||
|
||||
---
|
||||
|
||||
## Caching
|
||||
|
||||
Results are cached in-memory with a **5-minute TTL**. The per-source stream cache is keyed by `source_key + tmdb_id + season + episode`.
|
||||
|
||||
```
|
||||
CACHE_TTL = 5 minutes
|
||||
```typescript
|
||||
export interface SourceConfig {
|
||||
key: string; // Unique source identifier (e.g. "vidlink", "vixsrc")
|
||||
label: string; // Human-readable display label
|
||||
sourceFile: string; // Scraper module filename in src/sources/
|
||||
proxyParam: string; // Query parameter used for HLS proxy routing
|
||||
timeout: number; // Total milliseconds allowed before source abort
|
||||
sourcesTimeout?: number; // Secondary timeout for multi-server extraction
|
||||
jitter: number; // Maximum randomized startup delay in ms
|
||||
retries: number; // Number of automatic retry attempts on error
|
||||
multiUrl: boolean; // Returns multiple stream candidate URLs if true
|
||||
skipProxy?: boolean; // If true, returns direct upstream URLs without proxy
|
||||
skipVerify?: boolean; // If true, bypasses verification checks
|
||||
verifyHeaders?: boolean; // Injects custom headers into verification probes
|
||||
disabled?: boolean; // If true, provider is completely excluded
|
||||
cdnHeaders?: { // Custom headers injected for specific CDN patterns
|
||||
pattern: RegExp;
|
||||
headers: { Referer: string; Origin: string; };
|
||||
}[];
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## The HLS Proxy
|
||||
## The Query Pipeline
|
||||
|
||||
All provider stream URLs are wrapped in the server proxy (unless `skipProxy: true`):
|
||||
When a client requests a title (`getStream()` in SDK or `/movie` / `/tv` in Stream API), the engine executes the following pipeline:
|
||||
|
||||
```
|
||||
http://YOUR_API_URL/api?url=<encoded_stream_url>&<proxyParam>=1
|
||||
```
|
||||
|
||||
When your player fetches this path from your self-hosted instance, the server:
|
||||
|
||||
1. Fetches the upstream M3U8 with the correct `Referer`, `Origin`, and any provider-specific headers
|
||||
2. Rewrites all segment URLs and encryption key URIs in the playlist to route through the same proxy
|
||||
3. Returns the rewritten playlist with `Access-Control-Allow-Origin: *`
|
||||
| Pipeline Stage | Operation | Purpose & Verification |
|
||||
| :--- | :--- | :--- |
|
||||
| **1. Source Filtering** | `SOURCES.filter(!disabled)` | Excludes disabled scrapers and selects active providers based on configuration. |
|
||||
| **2. Jitter Staggering** | Sleep between `0` and `cfg.jitter` ms | Disperses outbound connection spikes across time to prevent DDoS protection triggers. |
|
||||
| **3. Concurrent Scraping** | Parallel async scraper execution | Scrapers parse HTML/JSON endpoints using customized `User-Agent`, `Referer`, and `Origin` headers. |
|
||||
| **4. Two-Phase Verification** | `verifyStream()` then `verifyPlayable()` | Phase 1 validates HTTP status ($< 400$ via HEAD). Phase 2 parses HLS manifests and verifies segment availability. Dead streams are discarded. |
|
||||
| **5. Proxy URL Rewriting** | Encodes target URL into `/api?url=...` | Adds proxy flags, sets required playback headers, and prepares segment de-obfuscation. |
|
||||
| **6. Progressive Emission** | Pushes SSE `source` event | Delivers playable stream candidates to the client player immediately without waiting for slower providers. |
|
||||
|
||||
---
|
||||
|
||||
## Stream Verification
|
||||
## Jitter Staggering & Rate-Limit Defense
|
||||
|
||||
Before a URL is emitted as a `source` event, it passes two verification steps:
|
||||
If a client triggers 60 scrapers instantaneously, your client IP address could send hundreds of simultaneous HTTP requests in the same millisecond, triggering upstream Cloudflare or DDoS rate limits.
|
||||
|
||||
1. **`verifyStream`** - a HEAD request to the raw upstream URL. Must return HTTP `< 400`.
|
||||
2. **`verifyPlayable`** - fetches the proxied URL, validates `#EXTM3U`, checks for `#EXTINF` or `#EXT-X-STREAM-INF`, then fetches the first segment.
|
||||
To mitigate this, Vyla introduces **per-source jitter**:
|
||||
```javascript
|
||||
const jitterDelay = Math.floor(Math.random() * (cfg.jitter || 400));
|
||||
await new Promise(resolve => setTimeout(resolve, jitterDelay));
|
||||
```
|
||||
By staggering outbound requests across a 100ms–800ms window, upstream hosts perceive natural traffic patterns, dramatically reducing HTTP 429 (Too Many Requests) errors.
|
||||
|
||||
---
|
||||
|
||||
## Two-Phase Playback Verification
|
||||
|
||||
Many streaming extractors successfully return a URL, only for that URL to return an HTTP 403 Forbidden, an empty manifest, or a dead video chunk. To protect players from stalling, Vyla validates every candidate through two independent verification stages before emitting it:
|
||||
|
||||
### Phase 1: `verifyStream()`
|
||||
- Dispatches a lightweight HTTP `HEAD` request to the upstream target URL.
|
||||
- Enforces custom `User-Agent`, `Referer`, and `Origin` headers.
|
||||
- Requires HTTP status code `< 400` within 4,000 milliseconds.
|
||||
|
||||
### Phase 2: `verifyPlayable()`
|
||||
- Fetches the initial playlist bytes through the local proxy.
|
||||
- Validates the presence of the `#EXTM3U` header tag.
|
||||
- Checks for stream definitions:
|
||||
- If a Master Playlist: ensures valid `#EXT-X-STREAM-INF` entries exist.
|
||||
- If a Media Playlist: ensures valid `#EXTINF` duration tags and playable segment URIs exist.
|
||||
- Attempts a test read on the first media segment or key URI.
|
||||
- Only candidates that pass both phases are deemed **verified**.
|
||||
|
||||
---
|
||||
|
||||
## The Universal HLS Proxy & Byte-Mask Stripper
|
||||
|
||||
Modern streaming hosts often protect their video segments by adding CORS restrictions, token signatures, or binary header masking. Stream API handles this transparently:
|
||||
|
||||
### 1. M3U8 Playlist Rewriting
|
||||
When an HLS stream is proxied, Stream API fetches the upstream playlist and rewrites every internal URI:
|
||||
- **AES-128 Key URIs**: `#EXT-X-KEY:METHOD=AES-128,URI="/api?url=<key>&<param>=1"`
|
||||
- **Video Chunks**: Segment URIs are rewritten to route back through the `/api` proxy endpoint.
|
||||
- **CORS Headers**: Returned with `Access-Control-Allow-Origin: *` and `Access-Control-Allow-Headers: *`.
|
||||
|
||||
### 2. Segment Byte-Mask Stripping
|
||||
Certain major CDNs mask MPEG-TS chunks by prepending a 120-byte fake image header (`image/png`, `image/jpeg`, TikTok CDN, or `letsgocdn` signatures) so firewalls treat them as images.
|
||||
|
||||
When Stream API proxies these segments, it inspects the leading bytes and automatically strips the 120-byte image wrapper:
|
||||
```javascript
|
||||
const needsStrip = searchParams.has('tt') || STRIP_REGEX.test(cleanUrl);
|
||||
if (needsStrip && (chunk[0] === 0x89 || chunk[0] === 0xFF || chunk[0] === 0x00)) {
|
||||
return chunk.subarray(120); // Strips fake header, leaving clean MPEG-TS
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Inter-Process Caching
|
||||
|
||||
Verified stream results are cached in-memory with a **5-minute Time-To-Live (TTL)**:
|
||||
- **Cache Key**: `source_key + tmdb_id + season + episode`
|
||||
- In multi-core cluster deployments, cache entries are coordinated across workers via inter-process communication (IPC) messages (`cache:get`, `cache:hit`, `cache:set`).
|
||||
- Automatic prune routines run every 30 seconds to evict expired items and enforce maximum cache limits (default: 1,500 entries).
|
||||
|
||||
Reference in New Issue
Block a user