huge rewrite

This commit is contained in:
2026-11-14 18:21:12 -08:00
parent f465d1ec0d
commit 2cec85a8f3
69 changed files with 11479 additions and 1576 deletions

View File

@@ -1,179 +1,139 @@
# Contributing to Sources
# Contributing a Source Provider
<p class="page-description">Help maintain, fix broken scrapers, and add new video or download providers to the Vyla SDK.</p>
<p class="page-description">Complete technical guide for authoring, testing, and contributing a new video scraping resolver to the Vyla SDK.</p>
Third-party streaming and scraping endpoints frequently change their DOM layout, API endpoints, tokens, and CDN protections. Community contributions are vital to keeping providers operational.
## Overview
The Vyla SDK maintains its high reliability through a distributed collection of independent provider scrapers located in `src/sources/`. We welcome contributions of new sources that expand coverage for movies, television, and international content.
---
### Thank You
## Scraper Architecture
If you’re a developer or just want to help test and fix sources, feel free to contribute! If you get a source working, send me the updated version.
Every provider module is a self-contained ES Module adhering to a single exported contract:
As a thank-you, contributors will receive the Contributor role and credit directly in the code for their contribution, along with a custom message of your choice and your Discord username.
Any help is appreciated! ❤️
## Getting Started
### 1. Clone the repository
Clone the SDK repository to your local machine:
```bash
git clone https://gitlab.com/vyla-entertainment/sdk.git
cd sdk
```typescript
export async function getStream(args: StreamArgs): Promise<StreamResult | { allUrls: StreamResult[] }>
```
### 2. Install dependencies
### The `args` Object
When your scraper's `getStream()` is invoked, the SDK passes a normalized `args` context containing:
Ensure you are running Node.js 18 or newer:
```bash
npm install
```
### 3. Environment configuration
Copy the example environment file and add your TMDB API key:
```bash
cp .env.example .env
```
Open `.env` and set your key:
```ini
TMDB_API_KEY=your_actual_tmdb_api_key_here
```
| Field | Type | Description |
| :--- | :--- | :--- |
| `id` | `string \| number` | The TMDB numeric media ID or string IMDb ID. |
| `s` | `number \| null` | Season number for television episodes (or `null` for movies). |
| `e` | `number \| null` | Episode number for television episodes (or `null` for movies). |
| `clientIP` | `string \| null` | The viewer's client IP address (if passed from an upstream proxy). |
| `fetch` | `Function` | Node.js fetch wrapper pre-configured with timeout controls. |
| `imdbId` | `string \| null` | Automatically resolved IMDb ID (if TMDB API key is configured). |
| `title` | `string \| null` | Verified media title string. |
| `year` | `number \| null` | Media release year. |
---
## Project Structure
## Step-by-Step Implementation Guide
- `src/sources/`: Scraper implementations for each provider.
- `src/manga/`: Manga catalog search and reader page resolvers.
- `src/config.js`: Source registry containing timeouts, retries, headers, and proxy flags.
- `src/downloads/downloads.js`: Downloader resolver extracting direct MP4 streams.
- `src/subtitles.js`: Subtitle mirror aggregation endpoints.
- `test/index.js`: Test runner for testing streams, subtitles, downloads, and manga.
---
## Testing Sources
Testing is driven by `test/index.js`. Open the file and update `CONFIG` to target the source you are fixing or adding:
### Step 1: Create the Scraper Module
Create a new file in `src/sources/<source_name>.js`:
```javascript
const CONFIG = {
DEBUG: true,
const BASE_URL = 'https://provider-domain.example.com';
MOVIE_ID: "936075",
TV_ID: "37854",
TV_SEASON: "1",
TV_EPISODE: "1",
STREAM_SOURCE: "vidrock",
TEST_SUBTITLE: false,
TEST_DOWNLOAD: false,
TEST_MOVIE: true,
TEST_TV: true,
EXTRA_SDK_PARAMS: {}
const DEFAULT_HEADERS = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
'Referer': `${BASE_URL}/`,
'Origin': BASE_URL,
};
```
Run the test suite:
export async function getStream({ id, s, e, imdbId }) {
const isTv = s != null && e != null;
const endpoint = isTv
? `${BASE_URL}/api/tv/${id}/${s}/${e}`
: `${BASE_URL}/api/movie/${id}`;
```bash
npm test
```
const response = await fetch(endpoint, {
headers: DEFAULT_HEADERS
});
When `DEBUG: true` is enabled, all outbound fetch calls, headers, request payloads, and raw responses will be logged to your terminal and saved to `test/debug-output.json`.
if (!response.ok) {
throw new Error(`Upstream returned HTTP ${response.status}`);
}
---
const data = await response.json();
if (!data.streamUrl) {
throw new Error('No stream found in upstream response');
}
## Fixing or Adding a Source
### 1. Source Resolver
All source files are located in `src/sources/`. Each module must export an asynchronous `getStream` function:
```javascript
export async function getStream({ id, s, e, clientIP, audio, sdk, config }) {
const isTv = Boolean(s && e);
const mediaId = id;
const streamUrl = "https://example-cdn.com/stream/playlist.m3u8";
return {
url: streamUrl,
quality: "auto",
isM3U8: true,
headers: {
Referer: "https://example-provider.to/",
Origin: "https://example-provider.to"
},
allUrls: [
{
url: streamUrl,
quality: "1080p",
isM3U8: true
}
]
};
return {
url: data.streamUrl,
headers: {
...DEFAULT_HEADERS,
'Referer': data.referer || DEFAULT_HEADERS.Referer
},
server: 'MySource-Primary',
quality: data.quality || '1080p',
type: data.streamUrl.includes('.m3u8') ? 'hls' : 'mp4'
};
}
```
### 2. Registering in `src/config.js`
---
Add or update the source entry inside the `SOURCES` array in `src/config.js`:
### Step 2: Register in `src/config.js`
Add your provider configuration to the `SOURCES` array in `src/config.js`:
```javascript
{
key: 'mysource',
label: 'MySource',
label: 'My Source',
sourceFile: 'mysource',
proxyParam: 'ms',
timeout: 25000,
jitter: 500,
retries: 2,
multiUrl: true,
skipProxy: true,
cdnHeaders: [{
pattern: /cdn\.mysource\.to/i,
headers: {
Referer: 'https://mysource.to/',
Origin: 'https://mysource.to',
},
}],
timeout: 12000,
jitter: 400,
retries: 1,
multiUrl: false,
skipProxy: false,
skipVerify: false
}
```
- `skipProxy`: Set to `true` if the stream plays directly in client players without server-side proxying.
- `multiUrl`: Set to `true` if the scraper provides multiple resolutions or fallback mirrors.
- `cdnHeaders`: Include custom headers required by the upstream CDN to avoid HTTP 403 Forbidden responses.
#### Configuration Field Guidelines:
- `key`: Lowercase alphanumeric identifier without spaces (e.g. `vidlink`, `cinecat`).
- `label`: Clean display title shown in UI menus.
- `proxyParam`: Unique 2-to-4 character query flag used by the Stream API HLS proxy (must not collide with existing flags).
- `timeout`: Maximum execution budget before aborting (typically 8,000ms–15,000ms).
- `jitter`: Random delay ceiling (typically 300ms–500ms) to avoid server burst locks.
---
## Download Providers
### Step 3: Local Testing
Downloads are handled automatically by `src/downloads/downloads.js`. The downloader queries active sources that have `skipProxy: true` enabled and inspects returned streams:
Run your scraper against known TMDB titles using the test runner:
- URLs ending in `.mp4` or containing `.mp4` queries are automatically extracted.
- `.m3u8` streams are filtered out because they are HLS segment playlists rather than direct file downloads.
- If you are implementing a provider that supplies raw MP4 video files, return them in `allUrls` or `url` with `skipProxy: true` in `src/config.js`.
```bash
# Test a known movie (The Dark Knight - TMDB 155)
node test/index.js mysource 155
To verify download resolution in `test/index.js`, toggle:
```javascript
TEST_DOWNLOAD: true
# Test a known TV episode (Breaking Bad S01E01 - TMDB 1396)
node test/index.js mysource 1396 1 1
```
Verify that:
1. A valid URL is printed.
2. The `type` is correctly identified as `'hls'` or `'mp4'`.
3. Required upstream headers (`Referer`, `Origin`, `User-Agent`) are included.
4. Testing an invalid ID throws a clean error without crashing the process.
---
## Submitting Pull Requests
## Contribution Rules & Checklist
1. Verify that your updated source passes tests for both movies (`TEST_MOVIE: true`) and TV shows (`TEST_TV: true`).
2. Ensure you have not committed your `.env` file or personal TMDB API keys.
3. Push your branch and open a Merge Request on GitLab.
Before submitting a Pull Request to the GitLab repository:
- [ ] Code is formatted with clean ES Module syntax.
- [ ] No hardcoded personal API keys or credentials.
- [ ] Provider does not execute external untrusted `eval()` or un-sanitized script execution.
- [ ] Upstream stream candidate URLs must be verified for playback.
- [ ] Errors are thrown with informative messages (`throw new Error(...)`).