mirror of
https://gitlab.com/vyla-entertainment/documentation.git
synced 2026-11-14 22:47:26 -05:00
92 lines
3.2 KiB
Markdown
92 lines
3.2 KiB
Markdown
# Provider Architecture & Mechanics
|
||
|
||
<p class="page-description">How Vyla SDK sources are structured, scheduled, verified, proxied, and cached across the ecosystem.</p>
|
||
|
||
## Overview
|
||
|
||
Rather than relying on a single upstream host, the Vyla SDK aggregates over 60 independent provider scrapers. Each scraper handles its own domain logic, extraction mechanics, and header requirements.
|
||
|
||
---
|
||
|
||
## Provider Configuration
|
||
|
||
Scrapers are configured in `src/config.js`:
|
||
|
||
```typescript
|
||
export interface SourceConfig {
|
||
key: string;
|
||
label: string;
|
||
sourceFile: string;
|
||
proxyParam: string;
|
||
timeout: number;
|
||
sourcesTimeout?: number;
|
||
jitter: number;
|
||
retries: number;
|
||
multiUrl: boolean;
|
||
skipProxy?: boolean;
|
||
skipVerify?: boolean;
|
||
verifyHeaders?: boolean;
|
||
disabled?: boolean;
|
||
cdnHeaders?: {
|
||
pattern: RegExp;
|
||
headers: { Referer: string; Origin: string; };
|
||
}[];
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Query Pipeline
|
||
|
||
1. **Source Filtering**: Excludes disabled scrapers and loads active configurations.
|
||
2. **Jitter Delay**: Sleeps for a randomized duration between `0` and `cfg.jitter` ms to avoid burst requests.
|
||
3. **Concurrent Scraping**: Runs scrapers in parallel with provider-specific `User-Agent`, `Referer`, and `Origin` headers.
|
||
4. **Two-Phase Verification**: Validates candidate URLs via `verifyStream()` (HTTP status < 400) and `verifyPlayable()` (HLS playlist and segment checks).
|
||
5. **Proxy URL Rewriting**: Encodes verified streams into `/api?url=...` with required proxy flags.
|
||
6. **Progressive Emission**: Pushes candidate streams over Server-Sent Events (SSE) immediately.
|
||
|
||
---
|
||
|
||
## Jitter Staggering
|
||
|
||
To prevent triggering upstream rate limits when querying dozens of scrapers simultaneously, each scraper applies random jitter:
|
||
|
||
```javascript
|
||
const jitterDelay = Math.floor(Math.random() * (cfg.jitter || 400));
|
||
await new Promise(resolve => setTimeout(resolve, jitterDelay));
|
||
```
|
||
|
||
This spreads outbound requests across a 100ms–800ms window, reducing HTTP 429 errors.
|
||
|
||
---
|
||
|
||
## Two-Phase Verification
|
||
|
||
1. **Phase 1 (`verifyStream`)**: Sends a lightweight HTTP `HEAD` request with source-specific headers. Discards any response >= 400.
|
||
2. **Phase 2 (`verifyPlayable`)**: Fetches the initial playlist bytes, verifies the `#EXTM3U` header, confirms playable segment paths, and tests the first chunk or key URI.
|
||
|
||
Only streams that pass both checks are returned to the client.
|
||
|
||
---
|
||
|
||
## Proxy & Header Stripping
|
||
|
||
- **M3U8 Rewriting**: Rewrites AES-128 encryption key URIs and `.ts` segment paths to route through the proxy with CORS headers (`Access-Control-Allow-Origin: *`).
|
||
- **120-Byte Mask Stripping**: Slices off fake 120-byte image headers prepended by certain CDNs:
|
||
|
||
```javascript
|
||
const needsStrip = searchParams.has('tt') || STRIP_REGEX.test(cleanUrl);
|
||
if (needsStrip && (chunk[0] === 0x89 || chunk[0] === 0xFF || chunk[0] === 0x00)) {
|
||
return chunk.subarray(120);
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## In-Memory Caching
|
||
|
||
Verified stream candidates are cached in-memory for 5 minutes (300,000 ms):
|
||
- Cache keys format: `source_key + tmdb_id + season + episode`.
|
||
- In cluster mode, workers coordinate entries via IPC messages (`cache:get`, `cache:hit`, `cache:set`).
|
||
- Expired entries are pruned every 30 seconds, maintaining a 1,500-entry ceiling.
|