mirror of
https://gitlab.com/vyla-entertainment/documentation.git
synced 2026-11-19 17:07:22 -05:00
huge rewrite
This commit is contained in:
@@ -1,179 +1,139 @@
|
||||
# Contributing to Sources
|
||||
# Contributing a Source Provider
|
||||
|
||||
<p class="page-description">Help maintain, fix broken scrapers, and add new video or download providers to the Vyla SDK.</p>
|
||||
<p class="page-description">Complete technical guide for authoring, testing, and contributing a new video scraping resolver to the Vyla SDK.</p>
|
||||
|
||||
Third-party streaming and scraping endpoints frequently change their DOM layout, API endpoints, tokens, and CDN protections. Community contributions are vital to keeping providers operational.
|
||||
## Overview
|
||||
|
||||
The Vyla SDK maintains its high reliability through a distributed collection of independent provider scrapers located in `src/sources/`. We welcome contributions of new sources that expand coverage for movies, television, and international content.
|
||||
|
||||
---
|
||||
|
||||
### Thank You
|
||||
## Scraper Architecture
|
||||
|
||||
If you’re a developer or just want to help test and fix sources, feel free to contribute! If you get a source working, send me the updated version.
|
||||
Every provider module is a self-contained ES Module adhering to a single exported contract:
|
||||
|
||||
As a thank-you, contributors will receive the Contributor role and credit directly in the code for their contribution, along with a custom message of your choice and your Discord username.
|
||||
|
||||
Any help is appreciated! ❤️
|
||||
|
||||
## Getting Started
|
||||
|
||||
### 1. Clone the repository
|
||||
|
||||
Clone the SDK repository to your local machine:
|
||||
|
||||
```bash
|
||||
git clone https://gitlab.com/vyla-entertainment/sdk.git
|
||||
cd sdk
|
||||
```typescript
|
||||
export async function getStream(args: StreamArgs): Promise<StreamResult | { allUrls: StreamResult[] }>
|
||||
```
|
||||
|
||||
### 2. Install dependencies
|
||||
### The `args` Object
|
||||
When your scraper's `getStream()` is invoked, the SDK passes a normalized `args` context containing:
|
||||
|
||||
Ensure you are running Node.js 18 or newer:
|
||||
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
### 3. Environment configuration
|
||||
|
||||
Copy the example environment file and add your TMDB API key:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Open `.env` and set your key:
|
||||
|
||||
```ini
|
||||
TMDB_API_KEY=your_actual_tmdb_api_key_here
|
||||
```
|
||||
| Field | Type | Description |
|
||||
| :--- | :--- | :--- |
|
||||
| `id` | `string \| number` | The TMDB numeric media ID or string IMDb ID. |
|
||||
| `s` | `number \| null` | Season number for television episodes (or `null` for movies). |
|
||||
| `e` | `number \| null` | Episode number for television episodes (or `null` for movies). |
|
||||
| `clientIP` | `string \| null` | The viewer's client IP address (if passed from an upstream proxy). |
|
||||
| `fetch` | `Function` | Node.js fetch wrapper pre-configured with timeout controls. |
|
||||
| `imdbId` | `string \| null` | Automatically resolved IMDb ID (if TMDB API key is configured). |
|
||||
| `title` | `string \| null` | Verified media title string. |
|
||||
| `year` | `number \| null` | Media release year. |
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
## Step-by-Step Implementation Guide
|
||||
|
||||
- `src/sources/`: Scraper implementations for each provider.
|
||||
- `src/manga/`: Manga catalog search and reader page resolvers.
|
||||
- `src/config.js`: Source registry containing timeouts, retries, headers, and proxy flags.
|
||||
- `src/downloads/downloads.js`: Downloader resolver extracting direct MP4 streams.
|
||||
- `src/subtitles.js`: Subtitle mirror aggregation endpoints.
|
||||
- `test/index.js`: Test runner for testing streams, subtitles, downloads, and manga.
|
||||
|
||||
---
|
||||
|
||||
## Testing Sources
|
||||
|
||||
Testing is driven by `test/index.js`. Open the file and update `CONFIG` to target the source you are fixing or adding:
|
||||
### Step 1: Create the Scraper Module
|
||||
Create a new file in `src/sources/<source_name>.js`:
|
||||
|
||||
```javascript
|
||||
const CONFIG = {
|
||||
DEBUG: true,
|
||||
const BASE_URL = 'https://provider-domain.example.com';
|
||||
|
||||
MOVIE_ID: "936075",
|
||||
TV_ID: "37854",
|
||||
TV_SEASON: "1",
|
||||
TV_EPISODE: "1",
|
||||
STREAM_SOURCE: "vidrock",
|
||||
|
||||
TEST_SUBTITLE: false,
|
||||
TEST_DOWNLOAD: false,
|
||||
TEST_MOVIE: true,
|
||||
TEST_TV: true,
|
||||
|
||||
EXTRA_SDK_PARAMS: {}
|
||||
const DEFAULT_HEADERS = {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36',
|
||||
'Referer': `${BASE_URL}/`,
|
||||
'Origin': BASE_URL,
|
||||
};
|
||||
```
|
||||
|
||||
Run the test suite:
|
||||
export async function getStream({ id, s, e, imdbId }) {
|
||||
const isTv = s != null && e != null;
|
||||
const endpoint = isTv
|
||||
? `${BASE_URL}/api/tv/${id}/${s}/${e}`
|
||||
: `${BASE_URL}/api/movie/${id}`;
|
||||
|
||||
```bash
|
||||
npm test
|
||||
```
|
||||
const response = await fetch(endpoint, {
|
||||
headers: DEFAULT_HEADERS
|
||||
});
|
||||
|
||||
When `DEBUG: true` is enabled, all outbound fetch calls, headers, request payloads, and raw responses will be logged to your terminal and saved to `test/debug-output.json`.
|
||||
if (!response.ok) {
|
||||
throw new Error(`Upstream returned HTTP ${response.status}`);
|
||||
}
|
||||
|
||||
---
|
||||
const data = await response.json();
|
||||
if (!data.streamUrl) {
|
||||
throw new Error('No stream found in upstream response');
|
||||
}
|
||||
|
||||
## Fixing or Adding a Source
|
||||
|
||||
### 1. Source Resolver
|
||||
|
||||
All source files are located in `src/sources/`. Each module must export an asynchronous `getStream` function:
|
||||
|
||||
```javascript
|
||||
export async function getStream({ id, s, e, clientIP, audio, sdk, config }) {
|
||||
const isTv = Boolean(s && e);
|
||||
const mediaId = id;
|
||||
|
||||
const streamUrl = "https://example-cdn.com/stream/playlist.m3u8";
|
||||
|
||||
return {
|
||||
url: streamUrl,
|
||||
quality: "auto",
|
||||
isM3U8: true,
|
||||
headers: {
|
||||
Referer: "https://example-provider.to/",
|
||||
Origin: "https://example-provider.to"
|
||||
},
|
||||
allUrls: [
|
||||
{
|
||||
url: streamUrl,
|
||||
quality: "1080p",
|
||||
isM3U8: true
|
||||
}
|
||||
]
|
||||
};
|
||||
return {
|
||||
url: data.streamUrl,
|
||||
headers: {
|
||||
...DEFAULT_HEADERS,
|
||||
'Referer': data.referer || DEFAULT_HEADERS.Referer
|
||||
},
|
||||
server: 'MySource-Primary',
|
||||
quality: data.quality || '1080p',
|
||||
type: data.streamUrl.includes('.m3u8') ? 'hls' : 'mp4'
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Registering in `src/config.js`
|
||||
---
|
||||
|
||||
Add or update the source entry inside the `SOURCES` array in `src/config.js`:
|
||||
### Step 2: Register in `src/config.js`
|
||||
|
||||
Add your provider configuration to the `SOURCES` array in `src/config.js`:
|
||||
|
||||
```javascript
|
||||
{
|
||||
key: 'mysource',
|
||||
label: 'MySource',
|
||||
label: 'My Source',
|
||||
sourceFile: 'mysource',
|
||||
proxyParam: 'ms',
|
||||
timeout: 25000,
|
||||
jitter: 500,
|
||||
retries: 2,
|
||||
multiUrl: true,
|
||||
skipProxy: true,
|
||||
cdnHeaders: [{
|
||||
pattern: /cdn\.mysource\.to/i,
|
||||
headers: {
|
||||
Referer: 'https://mysource.to/',
|
||||
Origin: 'https://mysource.to',
|
||||
},
|
||||
}],
|
||||
timeout: 12000,
|
||||
jitter: 400,
|
||||
retries: 1,
|
||||
multiUrl: false,
|
||||
skipProxy: false,
|
||||
skipVerify: false
|
||||
}
|
||||
```
|
||||
|
||||
- `skipProxy`: Set to `true` if the stream plays directly in client players without server-side proxying.
|
||||
- `multiUrl`: Set to `true` if the scraper provides multiple resolutions or fallback mirrors.
|
||||
- `cdnHeaders`: Include custom headers required by the upstream CDN to avoid HTTP 403 Forbidden responses.
|
||||
#### Configuration Field Guidelines:
|
||||
- `key`: Lowercase alphanumeric identifier without spaces (e.g. `vidlink`, `cinecat`).
|
||||
- `label`: Clean display title shown in UI menus.
|
||||
- `proxyParam`: Unique 2-to-4 character query flag used by the Stream API HLS proxy (must not collide with existing flags).
|
||||
- `timeout`: Maximum execution budget before aborting (typically 8,000ms–15,000ms).
|
||||
- `jitter`: Random delay ceiling (typically 300ms–500ms) to avoid server burst locks.
|
||||
|
||||
---
|
||||
|
||||
## Download Providers
|
||||
### Step 3: Local Testing
|
||||
|
||||
Downloads are handled automatically by `src/downloads/downloads.js`. The downloader queries active sources that have `skipProxy: true` enabled and inspects returned streams:
|
||||
Run your scraper against known TMDB titles using the test runner:
|
||||
|
||||
- URLs ending in `.mp4` or containing `.mp4` queries are automatically extracted.
|
||||
- `.m3u8` streams are filtered out because they are HLS segment playlists rather than direct file downloads.
|
||||
- If you are implementing a provider that supplies raw MP4 video files, return them in `allUrls` or `url` with `skipProxy: true` in `src/config.js`.
|
||||
```bash
|
||||
# Test a known movie (The Dark Knight - TMDB 155)
|
||||
node test/index.js mysource 155
|
||||
|
||||
To verify download resolution in `test/index.js`, toggle:
|
||||
|
||||
```javascript
|
||||
TEST_DOWNLOAD: true
|
||||
# Test a known TV episode (Breaking Bad S01E01 - TMDB 1396)
|
||||
node test/index.js mysource 1396 1 1
|
||||
```
|
||||
|
||||
Verify that:
|
||||
1. A valid URL is printed.
|
||||
2. The `type` is correctly identified as `'hls'` or `'mp4'`.
|
||||
3. Required upstream headers (`Referer`, `Origin`, `User-Agent`) are included.
|
||||
4. Testing an invalid ID throws a clean error without crashing the process.
|
||||
|
||||
---
|
||||
|
||||
## Submitting Pull Requests
|
||||
## Contribution Rules & Checklist
|
||||
|
||||
1. Verify that your updated source passes tests for both movies (`TEST_MOVIE: true`) and TV shows (`TEST_TV: true`).
|
||||
2. Ensure you have not committed your `.env` file or personal TMDB API keys.
|
||||
3. Push your branch and open a Merge Request on GitLab.
|
||||
Before submitting a Pull Request to the GitLab repository:
|
||||
|
||||
- [ ] Code is formatted with clean ES Module syntax.
|
||||
- [ ] No hardcoded personal API keys or credentials.
|
||||
- [ ] Provider does not execute external untrusted `eval()` or un-sanitized script execution.
|
||||
- [ ] Upstream stream candidate URLs must be verified for playback.
|
||||
- [ ] Errors are thrown with informative messages (`throw new Error(...)`).
|
||||
|
||||
Reference in New Issue
Block a user