Compare commits

...

23 Commits

Author SHA1 Message Date
pukkandan
4524baf056 Release 2021.02.19 2021-02-20 03:14:01 +05:30
pukkandan
bc2ca1bb75 Update to ytdl-commit-cf2dbec
cf2dbec630

Except: [kakao] improve info extraction and detect geo restriction
d8085580f6
2021-02-20 02:32:22 +05:30
pukkandan
5e41dca334 [viki] Fix extractor (Closes #91) 2021-02-19 18:21:29 +05:30
pukkandan
2a86f3da07 [build] Publish on PyPi only if token is set
This allows forks to easily build releases
:ci skip all
2021-02-19 17:04:25 +05:30
pukkandan
a40258a259 [documentation] Remove --flat-videos
It does not work as documented
It was an experimental option that I forgot to remove when making the fork public

:ci skip all
2021-02-19 04:52:05 +05:30
pukkandan
ba7bf12d89 [youtube] Fix for empty comment text (Closes #97) 2021-02-19 04:15:25 +05:30
pukkandan
f983b87567 [formatsort] Remove misuse of 'preference'
'preference' is to be used only when the format is better that ALL qualities of a lower preference irrespective of ANY sorting order the user requests. See deezer.py for correct use of this

In the older sorting method, `preference`, `quality` and `language_preference` were functionally almost equivalent. So these disparities doesn't really matter there

Also, despite what the documentation says, the default for `preference` was actually 0 and not -1. I have tried to correct this and also account for it when converting `preference` to `quality`
2021-02-19 03:33:45 +05:30
pukkandan
dca3ff4a5e [formatsort] Remove forced priority of quality
When making `FormatSort`, I misinterpreted the purpose `quality`
2021-02-19 00:12:21 +05:30
pukkandan
da9be05edf [documentation] Better document --prefer-free-formats
Also added `--no-prefer-free-formats`
2021-02-18 23:52:32 +05:30
pukkandan
155d2b48c5 [formatsort] Prefer vp9.2 over other vp9 codecs
vp9.2 may contain HDR while vp9.0 doesn't
2021-02-18 23:52:31 +05:30
pukkandan
54f37eeabd [formatsort] Remove unnecessary field_preference from extractors
These were written with the old format sorting in mind and is no longer needed
2021-02-18 23:52:28 +05:30
pukkandan
9ba5705ac0 [youtube] Fix hashtag continuation
Eg: https://www.youtube.com/hashtag/youtube

:ci skip dl
2021-02-18 13:54:06 +05:30
pukkandan
c2934512c2 Option --windows-filenames to force use of windows compatible filenames
* Also changed `--trim-file-name` to `--trim-filenames` to be similar to related options

Related: https://web.archive.org/web/20210217190806/https://old.reddit.com/r/youtubedl/comments/llc4o5/do_you_guys_also_have_this_error

:ci skip dl
2021-02-18 01:06:40 +05:30
shirt-dev
55e36f035c #93 Build improvements
* Lock all python package versions to the last officially supported releases for x86
* Bugfix for UNIX hash output
* Use wheels to avoid compilation of python packages
* Hash calculation on Windows now uses PowerShell rather than the legacy certutil

Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com>
2021-02-17 11:40:39 +05:30
pukkandan
c86d5023d0 [youtube] Add more Invidious instances (Closes #92)
:ci skip dl
2021-02-17 04:40:55 +05:30
pukkandan
42bb0c59f8 [MoveFiles] Fix when merger can't run
:ci skip dl
2021-02-17 00:42:27 +05:30
pukkandan
c3e1f0c4f2 [contributors] update
Forgot to do it when making release

:ci skip dl
2021-02-16 17:09:54 +05:30
pukkandan
6b027907ce Don't raise parser.error when exiting for update 2021-02-16 17:04:53 +05:30
pukkandan
f3b7c69377 [version] Set version number based on UTC time, not local time 2021-02-16 17:04:53 +05:30
Jody Bruchon
46261325be #89 [pyinst.py] Exclude vcruntime140.dll from UPX (#89)
Related: https://github.com/blackjack4494/yt-dlc/pull/182 (7b400ac40b)

Authored by: jbruchon
2021-02-16 16:41:47 +05:30
kurumigi
78b9a616cc #90 [niconico] Extract channel and channel_id (Closes #77)
Authored by kurumigi
2021-02-16 16:19:37 +05:30
pukkandan
55b53b338b [ExtractAudio] Bugfix for 1de75fa129
Fixes: #58
:ci skip dl
2021-02-16 15:00:54 +05:30
pukkandan
d16ab6ef1c [version] update
:ci skip dl
2021-02-16 04:17:55 +05:30
115 changed files with 1341 additions and 613 deletions

View File

@@ -21,7 +21,7 @@ assignees: ''
<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
-->
- [ ] I'm reporting a broken site support
- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
- [ ] I've checked that all provided URLs are alive and playable in a browser
- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] yt-dlp version 2021.02.09
[debug] yt-dlp version 2021.02.16
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}

View File

@@ -21,7 +21,7 @@ assignees: ''
<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
- Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
-->
- [ ] I'm reporting a new site support request
- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
- [ ] I've checked that all provided URLs are alive and playable in a browser
- [ ] I've checked that none of provided URLs violate any copyrights
- [ ] I've searched the bugtracker for similar site support requests including closed ones

View File

@@ -21,13 +21,13 @@ assignees: ''
<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->
- [ ] I'm reporting a site feature request
- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
- [ ] I've searched the bugtracker for similar site feature requests including closed ones

View File

@@ -21,7 +21,7 @@ assignees: ''
<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
- Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
-->
- [ ] I'm reporting a broken site support issue
- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
- [ ] I've checked that all provided URLs are alive and playable in a browser
- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] yt-dlp version 2021.02.09
[debug] yt-dlp version 2021.02.16
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}

View File

@@ -21,13 +21,13 @@ assignees: ''
<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.09. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
- Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
-->
- [ ] I'm reporting a feature request
- [ ] I've verified that I'm running yt-dlp version **2021.02.09**
- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
- [ ] I've searched the bugtracker for similar feature requests including closed ones

View File

@@ -55,8 +55,11 @@ jobs:
asset_content_type: application/octet-stream
- name: Get SHA2-256SUMS for youtube-dlc
id: sha2_file
run: echo "::set-output name=sha2_unix::$(sha256sum youtube-dlc)"
run: echo "::set-output name=sha2_unix::$(sha256sum youtube-dlc | awk '{print $1}')"
- name: Install dependencies for pypi
env:
PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
if: "env.PYPI_TOKEN != ''"
run: |
python -m pip install --upgrade pip
pip install setuptools wheel twine
@@ -64,6 +67,7 @@ jobs:
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
if: "env.TWINE_PASSWORD != ''"
run: |
rm -rf dist/*
python setup.py sdist bdist_wheel
@@ -84,6 +88,8 @@ jobs:
uses: actions/setup-python@v2
with:
python-version: '3.8'
- name: Upgrade pip and enable wheel support
run: python -m pip install --upgrade pip setuptools wheel
- name: Install Requirements
run: pip install pyinstaller mutagen pycryptodome
- name: Bump version
@@ -105,7 +111,7 @@ jobs:
asset_content_type: application/vnd.microsoft.portable-executable
- name: Get SHA2-256SUMS for youtube-dlc.exe
id: sha2_file_win
run: echo "::set-output name=sha2_windows::$(certUtil -hashfile dist\youtube-dlc.exe SHA256 | findstr -v :)"
run: echo "::set-output name=sha2_windows::$((Get-FileHash dist\youtube-dlc.exe -Algorithm SHA256).Hash.ToLower())"
build_windows32:
@@ -123,10 +129,10 @@ jobs:
with:
python-version: '3.4.4'
architecture: 'x86'
- name: Install VS libs
run: choco install vcexpress2010
- name: Upgrade pip and enable wheel support
run: python -m pip install pip==19.1.1 setuptools==43.0.0 wheel==0.33.6
- name: Install Requirements for 32 Bit
run: pip install pyinstaller==3.5 mutagen pycryptodome
run: pip install pyinstaller==3.5 mutagen==1.42.0 pycryptodome==3.9.4
- name: Bump version
id: bump_version
run: python devscripts/update-version.py
@@ -146,7 +152,7 @@ jobs:
asset_content_type: application/vnd.microsoft.portable-executable
- name: Get SHA2-256SUMS for youtube-dlc_x86.exe
id: sha2_file_win32
run: echo "::set-output name=sha2_windows32::$(certUtil -hashfile dist\youtube-dlc_x86.exe SHA256 | findstr -v :)"
run: echo "::set-output name=sha2_windows32::$((Get-FileHash dist\youtube-dlc_x86.exe -Algorithm SHA256).Hash.ToLower())"
- name: Make SHA2-256SUMS file
env:
SHA2_WINDOWS: ${{ needs.build_windows.outputs.sha2_windows }}
@@ -169,16 +175,3 @@ jobs:
asset_path: ./SHA2-256SUMS
asset_name: SHA2-256SUMS
asset_content_type: text/plain
# update_version_badge:
# runs-on: ubuntu-latest
# needs: build_unix
# steps:
# - name: Create Version Badge
# uses: schneegans/dynamic-badges-action@v1.0.0
# with:
# auth: ${{ secrets.GIST_TOKEN }}
# gistID: c69cb23c3c5b3316248e52022790aa57
# filename: version.json
# label: Version
# message: ${{ needs.build_unix.outputs.ytdlc_version }}

1
.gitignore vendored
View File

@@ -36,6 +36,7 @@ README.txt
# Binary
youtube-dl
youtube-dlc
youtube-dlc.zip
*.exe
# Downloaded

View File

@@ -17,4 +17,7 @@ alxnull
FelixFrog
Zocker1999NET
nao20010128nao
shirt-dev
shirt-dev
kurumigi
tsukumi
bbepis

View File

@@ -8,7 +8,7 @@
* Change "Merged with youtube-dl" version in Readme.md if needed
* Commit to master as `Release <version>`
* Push to origin/release - build task will now run
* Update version.py using devscripts\update-version.py (be wary of timezones)
* Update version.py using devscripts\update-version.py
* Run `make issuetemplates`
* Commit to master as `[version] update :ci skip all`
* Push to origin/master
@@ -17,25 +17,50 @@
-->
### 2021.02.19
* **Merge youtube-dl:** Upto [commit/cf2dbec](https://github.com/ytdl-org/youtube-dl/commit/cf2dbec6301177a1fddf72862de05fa912d9869d) (except kakao)
* [viki] Fix extractor
* [niconico] Extract `channel` and `channel_id` by [kurumigi](https://github.com/kurumigi)
* [youtube] Multiple page support for hashtag URLs
* [youtube] Add more invidious instances
* [youtube] Fix comment extraction when comment text is empty
* Option `--windows-filenames` to force use of windows compatible filenames
* [ExtractAudio] Bugfix
* Don't raise `parser.error` when exiting for update
* [MoveFiles] Fix for when merger can't run
* Changed `--trim-file-name` to `--trim-filenames` to be similar to related options
* Format Sort improvements:
* Prefer `vp9.2` more than other `vp9` codecs
* Remove forced priority of `quality`
* Remove unnecessary `field_preference` and misuse of `preference` from extractors
* Build improvements:
* Fix hash output by [shirt](https://github.com/shirt-dev)
* Lock python package versions for x86 and use `wheels` by [shirt](https://github.com/shirt-dev)
* Exclude `vcruntime140.dll` from UPX by [jbruchon](https://github.com/jbruchon)
* Set version number based on UTC time, not local time
* Publish on PyPi only if token is set
* [documentation] Better document `--prefer-free-formats` and add `--no-prefer-free-format`
### 2021.02.15
* **Merge youtube-dl:** Upto [2021.02.10](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.10) (except archive.org)
* [niconico] Improved extraction and support encrypted/SMILE movies
* Fix HLS AES-128 with multiple keys in external downloaders
* [youtube_live_chat] Fix by using POST API
* [niconico] Improved extraction and support encrypted/SMILE movies by [kurumigi](https://github.com/kurumigi), [tsukumi](https://github.com/tsukumi), [bbepis](https://github.com/bbepis), [pukkandan](https://github.com/pukkandan)
* Fix HLS AES-128 with multiple keys in external downloaders by [shirt](https://github.com/shirt-dev)
* [youtube_live_chat] Fix by using POST API by [siikamiika](https://github.com/siikamiika)
* [rumble] Add support for video page
* Option to allow downloading unplayable video formats (`--allow-unplayable-formats`)
* Option `--allow-unplayable-formats` to allow downloading unplayable video formats
* [ExtractAudio] Don't re-encode when file is already in a common audio format
* Change optional dependency to `pycryptodome`
* [youtube] Fix search continuations
* [youtube] Fix for new accounts
* Improve build/updater:
* Improve build/updater: by [pukkandan](https://github.com/pukkandan) and [shirt](https://github.com/shirt-dev)
* Fix SHA256 calculation in build and implement hash checking for updater
* Exit immediately in windows once the update process starts
* Fix updater for `x86.exe`
* Updater looks for both `yt-dlp` and `youtube-dlc` in releases for future-proofing
* Fix issue with unicode filenames in aria2c
* Change optional dependency to `pycryptodome`
* Fix issue with unicode filenames in aria2c by [shirt](https://github.com/shirt-dev)
* Fix `allow_playlist_files` not being correctly passed through
* Fix for empty HTTP head requests
* Fix for empty HTTP head requests by [shirt](https://github.com/shirt-dev)
* Fix `get_executable_path` in UNIX
* [sponskrub] Print ffmpeg output and errors to terminal
* `__real_download` should be false when ffmpeg unavailable and no download
@@ -82,7 +107,7 @@
### 2021.01.29
* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by [animelover1984](https://github.com/animelover1984) and [bbepis](https://github.com/bbepis)
* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: by [animelover1984](https://github.com/animelover1984) and [bbepis](https://github.com/bbepis)
* Add `--get-comments`
* [youtube] Extract comments
* [billibilli] Added BiliBiliSearchIE, BilibiliChannelIE

View File

@@ -177,7 +177,6 @@ Then simply type this
containing directory
--flat-playlist Do not extract the videos of a playlist,
only list them
--flat-videos Do not resolve the video urls
--no-flat-playlist Extract the videos of a playlist
--mark-watched Mark videos watched (YouTube only)
--no-mark-watched Do not mark videos watched
@@ -361,6 +360,12 @@ Then simply type this
filenames
--no-restrict-filenames Allow Unicode characters, "&" and spaces in
filenames (default)
--windows-filenames Force filenames to be windows compatible
--no-windows-filenames Make filenames windows compatible only if
using windows (default)
--trim-filenames LENGTH Limit the filename length (excluding
extension) to the specified number of
characters
-w, --no-overwrites Do not overwrite any files
--force-overwrites Overwrite all video and metadata files.
This option includes --no-continue
@@ -411,8 +416,6 @@ Then simply type this
may change
--no-cache-dir Disable filesystem caching
--rm-cache-dir Delete all filesystem cache files
--trim-file-name LENGTH Limit the filename length (extension
excluded)
## Thumbnail Images:
--write-thumbnail Write thumbnail image to disk
@@ -516,8 +519,12 @@ Then simply type this
--no-audio-multistreams Only one audio stream is downloaded for
each output file (default)
--all-formats Download all available video formats
--prefer-free-formats Prefer free video formats over non-free
formats of same quality
--prefer-free-formats Prefer video formats with free containers
over non-free ones of same quality. Use
with "-S ext" to strictly prefer free
containers irrespective of quality
--no-prefer-free-formats Don't give any special preference to free
containers (default)
-F, --list-formats List all available formats of requested
videos
--list-formats-as-table Present the output of -F in tabular form
@@ -978,10 +985,10 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
- `hasaud`: Gives priority to formats that has a audio stream
- `ie_pref`: The format preference as given by the extractor
- `lang`: Language preference as given by the extractor
- `quality`: The quality of the format. This is a metadata field available in some websites
- `quality`: The quality of the format as given by the extractor
- `source`: Preference of the source as given by the extractor
- `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
- `vcodec`: Video Codec (`av01` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
- `vcodec`: Video Codec (`av01` > `vp9.2` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
- `acodec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
- `codec`: Equivalent to `vcodec,acodec`
- `vext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
@@ -1000,9 +1007,9 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
- `br`: Equivalent to using `tbr,vbr,abr`
- `asr`: Audio sample rate in Hz
Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
The fields `hasvid`, `ie_pref`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec:vp9,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
The fields `hasvid`, `ie_pref`, `lang` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `quality,res,fps,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all repects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.

View File

@@ -13,7 +13,7 @@ old_version_list = old_version.split(".", 4)
old_ver = '.'.join(old_version_list[:3])
old_rev = old_version_list[3] if len(old_version_list) > 3 else ''
ver = datetime.now().strftime("%Y.%m.%d")
ver = datetime.utcnow().strftime("%Y.%m.%d")
rev = str(int(old_rev or 0) + 1) if old_ver == ver else ''
VERSION = '.'.join((ver, rev)) if rev else ver

View File

@@ -5,6 +5,7 @@
- **23video**
- **24video**
- **3qsdn**: 3Q SDN
- **3sat**
- **4tube**
- **56.com**
- **5min**
@@ -222,6 +223,7 @@
- **curiositystream**
- **curiositystream:collection**
- **CWTV**
- **DagelijkseKost**: dagelijksekost.een.be
- **DailyMail**
- **dailymotion**
- **dailymotion:playlist**
@@ -244,6 +246,7 @@
- **DiscoveryGo**
- **DiscoveryGoPlaylist**
- **DiscoveryNetworksDe**
- **DiscoveryPlus**
- **DiscoveryVR**
- **Disney**
- **dlive:stream**
@@ -370,6 +373,7 @@
- **HentaiStigma**
- **hetklokhuis**
- **hgtv.com:show**
- **HGTVDe**
- **HiDive**
- **HistoricFilms**
- **history:player**
@@ -854,6 +858,9 @@
- **ShahidShow**
- **Shared**: shared.sx
- **ShowRoomLive**
- **simplecast**
- **simplecast:episode**
- **simplecast:podcast**
- **Sina**
- **sky.it**
- **sky:news**
@@ -1231,6 +1238,7 @@
- **ZattooLive**
- **ZDF-3sat**
- **ZDFChannel**
- **Zhihu**
- **zingmp3**: mp3.zing.vn
- **zoom**
- **Zype**

View File

@@ -73,6 +73,7 @@ PyInstaller.__main__.run([
'--exclude-module=ytdlp_plugins',
'--hidden-import=mutagen',
'--hidden-import=Crypto',
'--upx-exclude=vcruntime140.dll',
'youtube_dlc/__main__.py',
])
SetVersion('dist/youtube-dlc%s.exe' % _x86, VERSION_FILE)

View File

@@ -12,6 +12,7 @@ from test.helper import FakeYDL
from youtube_dlc.extractor import (
YoutubePlaylistIE,
YoutubeTabIE,
YoutubeIE,
)
@@ -57,14 +58,22 @@ class TestYoutubeLists(unittest.TestCase):
entries = result['entries']
self.assertEqual(len(entries), 100)
def test_youtube_flat_playlist_titles(self):
def test_youtube_flat_playlist_extraction(self):
dl = FakeYDL()
dl.params['extract_flat'] = True
ie = YoutubePlaylistIE(dl)
result = ie.extract('https://www.youtube.com/playlist?list=PL-KKIb8rvtMSrAO9YFbeM6UQrAqoFTUWv')
ie = YoutubeTabIE(dl)
result = ie.extract('https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc')
self.assertIsPlaylist(result)
for entry in result['entries']:
self.assertTrue(entry.get('title'))
entries = list(result['entries'])
self.assertTrue(len(entries) == 1)
video = entries[0]
self.assertEqual(video['_type'], 'url_transparent')
self.assertEqual(video['ie_key'], 'Youtube')
self.assertEqual(video['id'], 'BaW_jenozKc')
self.assertEqual(video['url'], 'BaW_jenozKc')
self.assertEqual(video['title'], 'youtube-dl test video "\'/\\ä↭𝕐')
self.assertEqual(video['duration'], 10)
self.assertEqual(video['uploader'], 'Philipp Hagemeister')
if __name__ == '__main__':

View File

@@ -189,11 +189,15 @@ class YoutubeDL(object):
into a single file
allow_multiple_audio_streams: Allow multiple audio streams to be merged
into a single file
paths: Dictionary of output paths. The allowed keys are 'home'
'temp' and the keys of OUTTMPL_TYPES (in utils.py)
outtmpl: Dictionary of templates for output names. Allowed keys
are 'default' and the keys of OUTTMPL_TYPES (in utils.py)
are 'default' and the keys of OUTTMPL_TYPES (in utils.py).
A string a also accepted for backward compatibility
outtmpl_na_placeholder: Placeholder for unavailable meta fields.
restrictfilenames: Do not allow "&" and spaces in file names
trim_file_name: Limit length of filename (extension excluded)
windowsfilenames: Force the filenames to be windows compatible
ignoreerrors: Do not stop on download errors
(Default True when running youtube-dlc,
but False when directly accessing YoutubeDL class)
@@ -868,13 +872,6 @@ class YoutubeDL(object):
sub_ext = fn_groups[-2]
filename = '.'.join(filter(None, [fn_groups[0][:trim_file_name], sub_ext, ext]))
# Temporary fix for #4787
# 'Treat' all problem characters by passing filename through preferredencoding
# to workaround encoding issues with subprocess on python2 @ Windows
if sys.version_info < (3, 0) and sys.platform == 'win32':
filename = encodeFilename(filename, True).decode(preferredencoding())
filename = sanitize_path(filename)
return filename
except ValueError as err:
self.report_error('Error in output template: ' + str(err) + ' (encoding: ' + repr(preferredencoding()) + ')')
@@ -901,7 +898,14 @@ class YoutubeDL(object):
assert isinstance(homepath, compat_str)
subdir = expand_path(paths.get(dir_type, '').strip()) if dir_type else ''
assert isinstance(subdir, compat_str)
return sanitize_path(os.path.join(homepath, subdir, filename))
path = os.path.join(homepath, subdir, filename)
# Temporary fix for #4787
# 'Treat' all problem characters by passing filename through preferredencoding
# to workaround encoding issues with subprocess on python2 @ Windows
if sys.version_info < (3, 0) and sys.platform == 'win32':
path = encodeFilename(path, True).decode(preferredencoding())
return sanitize_path(path, force=self.params.get('windowsfilenames'))
def _match_entry(self, info_dict, incomplete):
""" Returns None if the file should be downloaded """
@@ -2360,6 +2364,9 @@ class YoutubeDL(object):
info_dict['__files_to_merge'] = downloaded
# Even if there were no downloads, it is being merged only now
info_dict['__real_download'] = True
else:
for file in downloaded:
files_to_move[file] = None
else:
# Just a single file
dl_filename = existing_file(full_filename, temp_filename)
@@ -2713,8 +2720,6 @@ class YoutubeDL(object):
if f.get('preference') is None or f['preference'] >= -1000]
header_line = ['format code', 'extension', 'resolution', 'note']
# if len(formats) > 1:
# table[-1][-1] += (' ' if table[-1][-1] else '') + '(best)'
self.to_screen(
'[info] Available formats for %s:\n%s' % (info_dict['id'], render_table(
header_line,

View File

@@ -440,6 +440,7 @@ def _real_main(argv=None):
'autonumber_size': opts.autonumber_size,
'autonumber_start': opts.autonumber_start,
'restrictfilenames': opts.restrictfilenames,
'windowsfilenames': opts.windowsfilenames,
'ignoreerrors': opts.ignoreerrors,
'force_generic_extractor': opts.force_generic_extractor,
'ratelimit': opts.ratelimit,
@@ -560,7 +561,7 @@ def _real_main(argv=None):
# If updater returns True, exit. Required for windows
if update_self(ydl.to_screen, opts.verbose, ydl._opener):
if actual_use:
parser.error('The program must exit for the update to complete')
sys.exit('ERROR: The program must exit for the update to complete')
sys.exit()
# Maybe do nothing

View File

@@ -66,7 +66,7 @@ class AdobeTVBaseIE(InfoExtractor):
if original_filename.startswith('s3://') and not s3_extracted:
formats.append({
'format_id': 'original',
'preference': 1,
'quality': 1,
'url': original_filename.replace('s3://', 'https://s3.amazonaws.com/'),
})
s3_extracted = True

View File

@@ -67,7 +67,7 @@ class AluraIE(InfoExtractor):
f['height'] = int('720' if m.group('res') == 'hd' else '480')
formats.extend(video_format)
self._sort_formats(formats, field_preference=('height', 'width', 'tbr', 'format_id'))
self._sort_formats(formats)
return {
'id': video_id,

View File

@@ -125,7 +125,7 @@ class AolIE(YahooIE):
'height': int_or_none(qs.get('h', [None])[0]),
})
formats.append(f)
self._sort_formats(formats, ('width', 'height', 'tbr', 'format_id'))
self._sort_formats(formats)
return {
'id': video_id,

View File

@@ -72,8 +72,7 @@ class AparatIE(InfoExtractor):
r'(\d+)[pP]', label or '', 'height',
default=None)),
})
self._sort_formats(
formats, field_preference=('height', 'width', 'tbr', 'format_id'))
self._sort_formats(formats)
info = self._search_json_ld(webpage, video_id, default={})

View File

@@ -129,10 +129,6 @@ class ArcPublishingIE(InfoExtractor):
if all([f.get('acodec') == 'none' for f in m3u8_formats]):
continue
for f in m3u8_formats:
if f.get('acodec') == 'none':
f['preference'] = -40
elif f.get('vcodec') == 'none':
f['preference'] = -50
height = f.get('height')
if not height:
continue
@@ -150,10 +146,9 @@ class ArcPublishingIE(InfoExtractor):
'height': int_or_none(s.get('height')),
'filesize': int_or_none(s.get('filesize')),
'url': s_url,
'preference': -1,
'quality': -10,
})
self._sort_formats(
formats, ('preference', 'width', 'height', 'vbr', 'filesize', 'tbr', 'ext', 'format_id'))
self._sort_formats(formats)
subtitles = {}
for subtitle in (try_get(video, lambda x: x['subtitles']['urls'], list) or []):

View File

@@ -324,20 +324,42 @@ class ARDIE(InfoExtractor):
formats = []
for a in video_node.findall('.//asset'):
file_name = xpath_text(a, './fileName', default=None)
if not file_name:
continue
format_type = a.attrib.get('type')
format_url = url_or_none(file_name)
if format_url:
ext = determine_ext(file_name)
if ext == 'm3u8':
formats.extend(self._extract_m3u8_formats(
format_url, display_id, 'mp4', entry_protocol='m3u8_native',
m3u8_id=format_type or 'hls', fatal=False))
continue
elif ext == 'f4m':
formats.extend(self._extract_f4m_formats(
update_url_query(format_url, {'hdcore': '3.7.0'}),
display_id, f4m_id=format_type or 'hds', fatal=False))
continue
f = {
'format_id': a.attrib['type'],
'width': int_or_none(a.find('./frameWidth').text),
'height': int_or_none(a.find('./frameHeight').text),
'vbr': int_or_none(a.find('./bitrateVideo').text),
'abr': int_or_none(a.find('./bitrateAudio').text),
'vcodec': a.find('./codecVideo').text,
'tbr': int_or_none(a.find('./totalBitrate').text),
'format_id': format_type,
'width': int_or_none(xpath_text(a, './frameWidth')),
'height': int_or_none(xpath_text(a, './frameHeight')),
'vbr': int_or_none(xpath_text(a, './bitrateVideo')),
'abr': int_or_none(xpath_text(a, './bitrateAudio')),
'vcodec': xpath_text(a, './codecVideo'),
'tbr': int_or_none(xpath_text(a, './totalBitrate')),
}
if a.find('./serverPrefix').text:
f['url'] = a.find('./serverPrefix').text
f['playpath'] = a.find('./fileName').text
server_prefix = xpath_text(a, './serverPrefix', default=None)
if server_prefix:
f.update({
'url': server_prefix,
'playpath': file_name,
})
else:
f['url'] = a.find('./fileName').text
if not format_url:
continue
f['url'] = format_url
formats.append(f)
self._sort_formats(formats)

View File

@@ -150,7 +150,6 @@ class ArteTVIE(ArteTVBaseIE):
format = {
'format_id': format_id,
'preference': -10 if f.get('videoFormat') == 'M3U8' else None,
'language_preference': lang_pref,
'format_note': '%s, %s' % (f.get('versionCode'), f.get('versionLibelle')),
'width': int_or_none(f.get('width')),
@@ -168,7 +167,9 @@ class ArteTVIE(ArteTVBaseIE):
formats.append(format)
self._sort_formats(formats)
# For this extractor, quality only represents the relative quality
# with respect to other formats with the same resolution
self._sort_formats(formats, ('res', 'quality'))
return {
'id': player_info.get('VID') or video_id,

View File

@@ -69,12 +69,10 @@ class BeatportIE(InfoExtractor):
'vcodec': 'none',
}
if ext == 'mp3':
fmt['preference'] = 0
fmt['acodec'] = 'mp3'
fmt['abr'] = 96
fmt['asr'] = 44100
elif ext == 'mp4':
fmt['preference'] = 1
fmt['acodec'] = 'aac'
fmt['abr'] = 96
fmt['asr'] = 44100

View File

@@ -203,7 +203,7 @@ class BiliBiliIE(InfoExtractor):
formats.append({
'url': backup_url,
# backup URLs have lower priorities
'preference': -2 if 'hd.mp4' in backup_url else -3,
'quality': -2 if 'hd.mp4' in backup_url else -3,
})
for a_format in formats:

View File

@@ -23,7 +23,7 @@ class BokeCCBaseIE(InfoExtractor):
formats = [{
'format_id': format_id,
'url': quality.find('./copy').attrib['playurl'],
'preference': int(quality.attrib['value']),
'quality': int(quality.attrib['value']),
} for quality in info_xml.findall('./video/quality')]
self._sort_formats(formats)

View File

@@ -47,7 +47,7 @@ class BpbIE(InfoExtractor):
quality = 'high' if '_high' in video_url else 'low'
formats.append({
'url': video_url,
'preference': 10 if quality == 'high' else 0,
'quality': 10 if quality == 'high' else 0,
'format_note': quality,
'format_id': '%s-%s' % (quality, determine_ext(video_url)),
})

View File

@@ -82,7 +82,7 @@ class CamModelsIE(InfoExtractor):
f.update({
'ext': 'mp4',
# hls skips fragments, preferring rtmp
'preference': -1,
'quality': -10,
})
else:
continue

View File

@@ -89,7 +89,7 @@ class CanalplusIE(InfoExtractor):
# the secret extracted from ya function in http://player.canalplus.fr/common/js/canalPlayer.js
'url': format_url + '?secret=pqzerjlsmdkjfoiuerhsdlfknaes',
'format_id': format_id,
'preference': preference(format_id),
'quality': preference(format_id),
})
self._sort_formats(formats)

View File

@@ -7,19 +7,21 @@ from .common import InfoExtractor
from .gigya import GigyaBaseIE
from ..compat import compat_HTTPError
from ..utils import (
extract_attributes,
ExtractorError,
strip_or_none,
clean_html,
extract_attributes,
float_or_none,
get_element_by_class,
int_or_none,
merge_dicts,
str_or_none,
strip_or_none,
url_or_none,
)
class CanvasIE(InfoExtractor):
_VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrt(?:video|nieuws)|sporza)/assets/(?P<id>[^/?#&]+)'
_VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrt(?:video|nieuws)|sporza|dako)/assets/(?P<id>[^/?#&]+)'
_TESTS = [{
'url': 'https://mediazone.vrt.be/api/v1/ketnet/assets/md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
'md5': '68993eda72ef62386a15ea2cf3c93107',
@@ -332,3 +334,51 @@ class VrtNUIE(GigyaBaseIE):
'display_id': display_id,
'season_number': int_or_none(page.get('episode_season')),
})
class DagelijkseKostIE(InfoExtractor):
IE_DESC = 'dagelijksekost.een.be'
_VALID_URL = r'https?://dagelijksekost\.een\.be/gerechten/(?P<id>[^/?#&]+)'
_TEST = {
'url': 'https://dagelijksekost.een.be/gerechten/hachis-parmentier-met-witloof',
'md5': '30bfffc323009a3e5f689bef6efa2365',
'info_dict': {
'id': 'md-ast-27a4d1ff-7d7b-425e-b84f-a4d227f592fa',
'display_id': 'hachis-parmentier-met-witloof',
'ext': 'mp4',
'title': 'Hachis parmentier met witloof',
'description': 'md5:9960478392d87f63567b5b117688cdc5',
'thumbnail': r're:^https?://.*\.jpg$',
'duration': 283.02,
},
'expected_warnings': ['is not a supported codec'],
}
def _real_extract(self, url):
display_id = self._match_id(url)
webpage = self._download_webpage(url, display_id)
title = strip_or_none(get_element_by_class(
'dish-metadata__title', webpage
) or self._html_search_meta(
'twitter:title', webpage))
description = clean_html(get_element_by_class(
'dish-description', webpage)
) or self._html_search_meta(
('description', 'twitter:description', 'og:description'),
webpage)
video_id = self._html_search_regex(
r'data-url=(["\'])(?P<id>(?:(?!\1).)+)\1', webpage, 'video id',
group='id')
return {
'_type': 'url_transparent',
'url': 'https://mediazone.vrt.be/api/v1/dako/assets/%s' % video_id,
'ie_key': CanvasIE.ie_key(),
'id': video_id,
'display_id': display_id,
'title': title,
'description': description,
}

View File

@@ -1,12 +1,14 @@
# coding: utf-8
from __future__ import unicode_literals
import calendar
import datetime
import re
from .common import InfoExtractor
from ..utils import (
clean_html,
extract_timezone,
int_or_none,
parse_duration,
parse_resolution,
@@ -97,8 +99,9 @@ class CCMAIE(InfoExtractor):
timestamp = None
data_utc = try_get(informacio, lambda x: x['data_emissio']['utc'])
try:
timestamp = datetime.datetime.strptime(
data_utc, '%Y-%d-%mT%H:%M:%S%z').timestamp()
timezone, data_utc = extract_timezone(data_utc)
timestamp = calendar.timegm((datetime.datetime.strptime(
data_utc, '%Y-%d-%mT%H:%M:%S') - timezone).timetuple())
except TypeError:
pass

View File

@@ -162,7 +162,7 @@ class CCTVIE(InfoExtractor):
'url': video_url,
'format_id': 'http',
'quality': quality,
'preference': -1,
'source_preference': -10
})
hls_url = try_get(data, lambda x: x['hls_url'], compat_str)

View File

@@ -1367,12 +1367,12 @@ class InfoExtractor(object):
regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<seperator>[~:])(?P<limit>.*?))?)? *$'
default = ('hidden', 'hasvid', 'ie_pref', 'lang', 'quality',
'res', 'fps', 'codec:vp9', 'size', 'br', 'asr',
'res', 'fps', 'codec:vp9.2', 'size', 'br', 'asr',
'proto', 'ext', 'has_audio', 'source', 'format_id') # These must not be aliases
settings = {
'vcodec': {'type': 'ordered', 'regex': True,
'order': ['av0?1', 'vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
'order': ['av0?1', 'vp0?9.2', 'vp0?9', '[hx]265|he?vc?', '[hx]264|avc', 'vp0?8', 'mp4v|h263', 'theora', '', None, 'none']},
'acodec': {'type': 'ordered', 'regex': True,
'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
'proto': {'type': 'ordered', 'regex': True, 'field': 'protocol',
@@ -1384,11 +1384,11 @@ class InfoExtractor(object):
'order': ('m4a', 'aac', 'mp3', 'ogg', 'opus', 'webm', '', 'none'),
'order_free': ('opus', 'ogg', 'webm', 'm4a', 'mp3', 'aac', '', 'none')},
'hidden': {'visible': False, 'forced': True, 'type': 'extractor', 'max': -1000},
'ie_pref': {'priority': True, 'type': 'extractor', 'field': 'extractor_preference'},
'ie_pref': {'priority': True, 'type': 'extractor'},
'hasvid': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
'hasaud': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
'lang': {'priority': True, 'convert': 'ignore', 'field': 'language_preference'},
'quality': {'priority': True, 'convert': 'float_none'},
'lang': {'priority': True, 'convert': 'ignore', 'type': 'extractor', 'field': 'language_preference'},
'quality': {'convert': 'float_none', 'type': 'extractor'},
'filesize': {'convert': 'bytes'},
'fs_approx': {'convert': 'bytes', 'field': 'filesize_approx'},
'id': {'convert': 'string', 'field': 'format_id'},
@@ -1399,7 +1399,7 @@ class InfoExtractor(object):
'vbr': {'convert': 'float_none'},
'abr': {'convert': 'float_none'},
'asr': {'convert': 'float_none'},
'source': {'convert': 'ignore', 'field': 'source_preference'},
'source': {'convert': 'ignore', 'type': 'extractor', 'field': 'source_preference'},
'codec': {'type': 'combined', 'field': ('vcodec', 'acodec')},
'br': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
@@ -1469,13 +1469,12 @@ class InfoExtractor(object):
elif conversion == 'bytes':
return FileDownloader.parse_bytes(value)
elif conversion == 'order':
order_free = self._get_field_setting(field, 'order_free')
order_list = order_free if order_free and self._use_free_order else self._get_field_setting(field, 'order')
order_list = (self._use_free_order and self._get_field_setting(field, 'order_free')) or self._get_field_setting(field, 'order')
use_regex = self._get_field_setting(field, 'regex')
list_length = len(order_list)
empty_pos = order_list.index('') if '' in order_list else list_length + 1
if use_regex and value is not None:
for (i, regex) in enumerate(order_list):
for i, regex in enumerate(order_list):
if regex and re.match(regex, value):
return list_length - i
return list_length - empty_pos # not in list
@@ -1544,7 +1543,7 @@ class InfoExtractor(object):
def print_verbose_info(self, to_screen):
to_screen('[debug] Sort order given by user: %s' % ','.join(self._sort_user))
if self._sort_extractor:
to_screen('[debug] Sort order given by extractor: %s' % ','.join(self._sort_extractor))
to_screen('[debug] Sort order given by extractor: %s' % ', '.join(self._sort_extractor))
to_screen('[debug] Formats sorted by: %s' % ', '.join(['%s%s%s' % (
'+' if self._get_field_setting(field, 'reverse') else '', field,
'%s%s(%s)' % ('~' if self._get_field_setting(field, 'closest') else ':',
@@ -1561,7 +1560,7 @@ class InfoExtractor(object):
if type == 'extractor':
maximum = self._get_field_setting(field, 'max')
if value is None or (maximum is not None and value >= maximum):
value = 0
value = -1
elif type == 'boolean':
in_list = self._get_field_setting(field, 'in_list')
not_in_list = self._get_field_setting(field, 'not_in_list')
@@ -1694,7 +1693,7 @@ class InfoExtractor(object):
self.to_screen(msg)
time.sleep(timeout)
def _extract_f4m_formats(self, manifest_url, video_id, preference=None, f4m_id=None,
def _extract_f4m_formats(self, manifest_url, video_id, preference=None, quality=None, f4m_id=None,
transform_source=lambda s: fix_xml_ampersands(s).strip(),
fatal=True, m3u8_id=None, data=None, headers={}, query={}):
manifest = self._download_xml(
@@ -1709,10 +1708,10 @@ class InfoExtractor(object):
return []
return self._parse_f4m_formats(
manifest, manifest_url, video_id, preference=preference, f4m_id=f4m_id,
manifest, manifest_url, video_id, preference=preference, quality=quality, f4m_id=f4m_id,
transform_source=transform_source, fatal=fatal, m3u8_id=m3u8_id)
def _parse_f4m_formats(self, manifest, manifest_url, video_id, preference=None, f4m_id=None,
def _parse_f4m_formats(self, manifest, manifest_url, video_id, preference=None, quality=None, f4m_id=None,
transform_source=lambda s: fix_xml_ampersands(s).strip(),
fatal=True, m3u8_id=None):
if not isinstance(manifest, compat_etree_Element) and not fatal:
@@ -1777,7 +1776,7 @@ class InfoExtractor(object):
ext = determine_ext(manifest_url)
if ext == 'f4m':
f4m_formats = self._extract_f4m_formats(
manifest_url, video_id, preference=preference, f4m_id=f4m_id,
manifest_url, video_id, preference=preference, quality=quality, f4m_id=f4m_id,
transform_source=transform_source, fatal=fatal)
# Sometimes stream-level manifest contains single media entry that
# does not contain any quality metadata (e.g. http://matchtv.ru/#live-player).
@@ -1797,7 +1796,7 @@ class InfoExtractor(object):
elif ext == 'm3u8':
formats.extend(self._extract_m3u8_formats(
manifest_url, video_id, 'mp4', preference=preference,
m3u8_id=m3u8_id, fatal=fatal))
quality=quality, m3u8_id=m3u8_id, fatal=fatal))
continue
formats.append({
'format_id': format_id,
@@ -1810,22 +1809,24 @@ class InfoExtractor(object):
'height': height,
'vcodec': vcodec,
'preference': preference,
'quality': quality,
})
return formats
def _m3u8_meta_format(self, m3u8_url, ext=None, preference=None, m3u8_id=None):
def _m3u8_meta_format(self, m3u8_url, ext=None, preference=None, quality=None, m3u8_id=None):
return {
'format_id': '-'.join(filter(None, [m3u8_id, 'meta'])),
'url': m3u8_url,
'ext': ext,
'protocol': 'm3u8',
'preference': preference - 100 if preference else -100,
'quality': quality,
'resolution': 'multiple',
'format_note': 'Quality selection URL',
}
def _extract_m3u8_formats(self, m3u8_url, video_id, ext=None,
entry_protocol='m3u8', preference=None,
entry_protocol='m3u8', preference=None, quality=None,
m3u8_id=None, note=None, errnote=None,
fatal=True, live=False, data=None, headers={},
query={}):
@@ -1843,10 +1844,10 @@ class InfoExtractor(object):
return self._parse_m3u8_formats(
m3u8_doc, m3u8_url, ext=ext, entry_protocol=entry_protocol,
preference=preference, m3u8_id=m3u8_id, live=live)
preference=preference, quality=quality, m3u8_id=m3u8_id, live=live)
def _parse_m3u8_formats(self, m3u8_doc, m3u8_url, ext=None,
entry_protocol='m3u8', preference=None,
entry_protocol='m3u8', preference=None, quality=None,
m3u8_id=None, live=False):
if '#EXT-X-FAXS-CM:' in m3u8_doc: # Adobe Flash Access
return []
@@ -1884,6 +1885,7 @@ class InfoExtractor(object):
'ext': ext,
'protocol': entry_protocol,
'preference': preference,
'quality': quality,
}]
groups = {}
@@ -1912,6 +1914,7 @@ class InfoExtractor(object):
'ext': ext,
'protocol': entry_protocol,
'preference': preference,
'quality': quality,
}
if media_type == 'AUDIO':
f['vcodec'] = 'none'
@@ -1971,6 +1974,7 @@ class InfoExtractor(object):
'fps': float_or_none(last_stream_inf.get('FRAME-RATE')),
'protocol': entry_protocol,
'preference': preference,
'quality': quality,
}
resolution = last_stream_inf.get('RESOLUTION')
if resolution:
@@ -2678,7 +2682,7 @@ class InfoExtractor(object):
})
return formats
def _parse_html5_media_entries(self, base_url, webpage, video_id, m3u8_id=None, m3u8_entry_protocol='m3u8', mpd_id=None, preference=None):
def _parse_html5_media_entries(self, base_url, webpage, video_id, m3u8_id=None, m3u8_entry_protocol='m3u8', mpd_id=None, preference=None, quality=None):
def absolute_url(item_url):
return urljoin(base_url, item_url)
@@ -2701,7 +2705,7 @@ class InfoExtractor(object):
formats = self._extract_m3u8_formats(
full_url, video_id, ext='mp4',
entry_protocol=m3u8_entry_protocol, m3u8_id=m3u8_id,
preference=preference, fatal=False)
preference=preference, quality=quality, fatal=False)
elif ext == 'mpd':
is_plain_url = False
formats = self._extract_mpd_formats(

View File

@@ -87,7 +87,7 @@ class CoubIE(InfoExtractor):
'filesize': int_or_none(item.get('size')),
'vcodec': 'none' if kind == 'audio' else None,
'quality': quality_key(quality),
'preference': preference_key(HTML5),
'source_preference': preference_key(HTML5),
})
iphone_url = file_versions.get(IPHONE, {}).get('url')
@@ -95,7 +95,7 @@ class CoubIE(InfoExtractor):
formats.append({
'url': iphone_url,
'format_id': IPHONE,
'preference': preference_key(IPHONE),
'source_preference': preference_key(IPHONE),
})
mobile_url = file_versions.get(MOBILE, {}).get('audio_url')
@@ -103,7 +103,7 @@ class CoubIE(InfoExtractor):
formats.append({
'url': mobile_url,
'format_id': '%s-audio' % MOBILE,
'preference': preference_key(MOBILE),
'source_preference': preference_key(MOBILE),
})
self._sort_formats(formats)

View File

@@ -473,15 +473,11 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
stream.get('url'), video_id, stream.get('format'),
audio_lang, hardsub_lang)
for f in vrv_formats:
if not hardsub_lang:
f['preference'] = 1
language_preference = 0
if audio_lang == language:
language_preference += 1
if hardsub_lang == language:
language_preference += 1
if language_preference:
f['language_preference'] = language_preference
f['language_preference'] = 1 if audio_lang == language else 0
f['quality'] = (
1 if not hardsub_lang
else 0 if hardsub_lang == language
else -1)
formats.extend(vrv_formats)
if not formats:
available_fmts = []
@@ -571,7 +567,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
'ext': 'flv',
})
formats.append(format_info)
self._sort_formats(formats, ('preference', 'language_preference', 'height', 'width', 'tbr', 'fps'))
self._sort_formats(formats)
metadata = self._call_rpc_api(
'VideoPlayer_GetMediaMetadata', video_id,

View File

@@ -82,6 +82,7 @@ class DigitallySpeakingIE(InfoExtractor):
'play_path': remove_end(audio.get('url'), '.flv'),
'ext': 'flv',
'vcodec': 'none',
'quality': 1,
'format_id': audio.get('code'),
})
slide_video_path = xpath_text(metadata, './slideVideo', fatal=True)
@@ -91,7 +92,6 @@ class DigitallySpeakingIE(InfoExtractor):
'ext': 'flv',
'format_note': 'slide deck video',
'quality': -2,
'preference': -2,
'format_id': 'slides',
'acodec': 'none',
})
@@ -102,7 +102,6 @@ class DigitallySpeakingIE(InfoExtractor):
'ext': 'flv',
'format_note': 'speaker video',
'quality': -1,
'preference': -1,
'format_id': 'speaker',
})
return formats

View File

@@ -1,6 +1,7 @@
# coding: utf-8
from __future__ import unicode_literals
import json
import re
from .common import InfoExtractor
@@ -10,11 +11,13 @@ from ..utils import (
ExtractorError,
float_or_none,
int_or_none,
strip_or_none,
unified_timestamp,
)
class DPlayIE(InfoExtractor):
_PATH_REGEX = r'/(?P<id>[^/]+/[^/?#]+)'
_VALID_URL = r'''(?x)https?://
(?P<domain>
(?:www\.)?(?P<host>d
@@ -24,7 +27,7 @@ class DPlayIE(InfoExtractor):
)
)|
(?P<subdomain_country>es|it)\.dplay\.com
)/[^/]+/(?P<id>[^/]+/[^/?#]+)'''
)/[^/]+''' + _PATH_REGEX
_TESTS = [{
# non geo restricted, via secure api, unsigned download hls URL
@@ -151,56 +154,79 @@ class DPlayIE(InfoExtractor):
'only_matching': True,
}]
def _process_errors(self, e, geo_countries):
info = self._parse_json(e.cause.read().decode('utf-8'), None)
error = info['errors'][0]
error_code = error.get('code')
if error_code == 'access.denied.geoblocked':
self.raise_geo_restricted(countries=geo_countries)
elif error_code in ('access.denied.missingpackage', 'invalid.token'):
raise ExtractorError(
'This video is only available for registered users. You may want to use --cookies.', expected=True)
raise ExtractorError(info['errors'][0]['detail'], expected=True)
def _update_disco_api_headers(self, headers, disco_base, display_id, realm):
headers['Authorization'] = 'Bearer ' + self._download_json(
disco_base + 'token', display_id, 'Downloading token',
query={
'realm': realm,
})['data']['attributes']['token']
def _download_video_playback_info(self, disco_base, video_id, headers):
streaming = self._download_json(
disco_base + 'playback/videoPlaybackInfo/' + video_id,
video_id, headers=headers)['data']['attributes']['streaming']
streaming_list = []
for format_id, format_dict in streaming.items():
streaming_list.append({
'type': format_id,
'url': format_dict.get('url'),
})
return streaming_list
def _get_disco_api_info(self, url, display_id, disco_host, realm, country):
geo_countries = [country.upper()]
self._initialize_geo_bypass({
'countries': geo_countries,
})
disco_base = 'https://%s/' % disco_host
token = self._download_json(
disco_base + 'token', display_id, 'Downloading token',
query={
'realm': realm,
})['data']['attributes']['token']
headers = {
'Referer': url,
'Authorization': 'Bearer ' + token,
}
video = self._download_json(
disco_base + 'content/videos/' + display_id, display_id,
headers=headers, query={
'fields[channel]': 'name',
'fields[image]': 'height,src,width',
'fields[show]': 'name',
'fields[tag]': 'name',
'fields[video]': 'description,episodeNumber,name,publishStart,seasonNumber,videoDuration',
'include': 'images,primaryChannel,show,tags'
})
self._update_disco_api_headers(headers, disco_base, display_id, realm)
try:
video = self._download_json(
disco_base + 'content/videos/' + display_id, display_id,
headers=headers, query={
'fields[channel]': 'name',
'fields[image]': 'height,src,width',
'fields[show]': 'name',
'fields[tag]': 'name',
'fields[video]': 'description,episodeNumber,name,publishStart,seasonNumber,videoDuration',
'include': 'images,primaryChannel,show,tags'
})
except ExtractorError as e:
if isinstance(e.cause, compat_HTTPError) and e.cause.code == 400:
self._process_errors(e, geo_countries)
raise
video_id = video['data']['id']
info = video['data']['attributes']
title = info['name'].strip()
formats = []
try:
streaming = self._download_json(
disco_base + 'playback/videoPlaybackInfo/' + video_id,
display_id, headers=headers)['data']['attributes']['streaming']
streaming = self._download_video_playback_info(
disco_base, video_id, headers)
except ExtractorError as e:
if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
info = self._parse_json(e.cause.read().decode('utf-8'), display_id)
error = info['errors'][0]
error_code = error.get('code')
if error_code == 'access.denied.geoblocked':
self.raise_geo_restricted(countries=geo_countries)
elif error_code == 'access.denied.missingpackage':
self.raise_login_required()
raise ExtractorError(info['errors'][0]['detail'], expected=True)
self._process_errors(e, geo_countries)
raise
for format_id, format_dict in streaming.items():
for format_dict in streaming:
if not isinstance(format_dict, dict):
continue
format_url = format_dict.get('url')
if not format_url:
continue
format_id = format_dict.get('type')
ext = determine_ext(format_url)
if format_id == 'dash' or ext == 'mpd':
formats.extend(self._extract_mpd_formats(
@@ -248,7 +274,7 @@ class DPlayIE(InfoExtractor):
'id': video_id,
'display_id': display_id,
'title': title,
'description': info.get('description'),
'description': strip_or_none(info.get('description')),
'duration': float_or_none(info.get('videoDuration'), 1000),
'timestamp': unified_timestamp(info.get('publishStart')),
'series': series,
@@ -268,3 +294,75 @@ class DPlayIE(InfoExtractor):
host = 'disco-api.' + domain if domain[0] == 'd' else 'eu2-prod.disco-api.com'
return self._get_disco_api_info(
url, display_id, host, 'dplay' + country, country)
class DiscoveryPlusIE(DPlayIE):
_VALID_URL = r'https?://(?:www\.)?discoveryplus\.com/video' + DPlayIE._PATH_REGEX
_TESTS = [{
'url': 'https://www.discoveryplus.com/video/property-brothers-forever-home/food-and-family',
'info_dict': {
'id': '1140794',
'display_id': 'property-brothers-forever-home/food-and-family',
'ext': 'mp4',
'title': 'Food and Family',
'description': 'The brothers help a Richmond family expand their single-level home.',
'duration': 2583.113,
'timestamp': 1609304400,
'upload_date': '20201230',
'creator': 'HGTV',
'series': 'Property Brothers: Forever Home',
'season_number': 1,
'episode_number': 1,
},
'skip': 'Available for Premium users',
}]
def _update_disco_api_headers(self, headers, disco_base, display_id, realm):
headers['x-disco-client'] = 'WEB:UNKNOWN:dplus_us:15.0.0'
def _download_video_playback_info(self, disco_base, video_id, headers):
return self._download_json(
disco_base + 'playback/v3/videoPlaybackInfo',
video_id, headers=headers, data=json.dumps({
'deviceInfo': {
'adBlocker': False,
},
'videoId': video_id,
'wisteriaProperties': {
'platform': 'desktop',
},
}).encode('utf-8'))['data']['attributes']['streaming']
def _real_extract(self, url):
display_id = self._match_id(url)
return self._get_disco_api_info(
url, display_id, 'us1-prod-direct.discoveryplus.com', 'go', 'us')
class HGTVDeIE(DPlayIE):
_VALID_URL = r'https?://de\.hgtv\.com/sendungen' + DPlayIE._PATH_REGEX
_TESTS = [{
'url': 'https://de.hgtv.com/sendungen/tiny-house-klein-aber-oho/wer-braucht-schon-eine-toilette/',
'info_dict': {
'id': '151205',
'display_id': 'tiny-house-klein-aber-oho/wer-braucht-schon-eine-toilette',
'ext': 'mp4',
'title': 'Wer braucht schon eine Toilette',
'description': 'md5:05b40a27e7aed2c9172de34d459134e2',
'duration': 1177.024,
'timestamp': 1595705400,
'upload_date': '20200725',
'creator': 'HGTV',
'series': 'Tiny House - klein, aber oho',
'season_number': 3,
'episode_number': 3,
},
'params': {
'format': 'bestvideo',
},
}]
def _real_extract(self, url):
display_id = self._match_id(url)
return self._get_disco_api_info(
url, display_id, 'eu1-prod.disco-api.com', 'hgtv', 'de')

View File

@@ -0,0 +1,193 @@
from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..utils import (
int_or_none,
unified_strdate,
xpath_text,
determine_ext,
float_or_none,
ExtractorError,
)
class DreiSatIE(InfoExtractor):
IE_NAME = '3sat'
_GEO_COUNTRIES = ['DE']
_VALID_URL = r'https?://(?:www\.)?3sat\.de/mediathek/(?:(?:index|mediathek)\.php)?\?(?:(?:mode|display)=[^&]+&)*obj=(?P<id>[0-9]+)'
_TESTS = [
{
'url': 'http://www.3sat.de/mediathek/index.php?mode=play&obj=45918',
'md5': 'be37228896d30a88f315b638900a026e',
'info_dict': {
'id': '45918',
'ext': 'mp4',
'title': 'Waidmannsheil',
'description': 'md5:cce00ca1d70e21425e72c86a98a56817',
'uploader': 'SCHWEIZWEIT',
'uploader_id': '100000210',
'upload_date': '20140913'
},
'params': {
'skip_download': True, # m3u8 downloads
}
},
{
'url': 'http://www.3sat.de/mediathek/mediathek.php?mode=play&obj=51066',
'only_matching': True,
},
]
def _parse_smil_formats(self, smil, smil_url, video_id, namespace=None, f4m_params=None, transform_rtmp_url=None):
param_groups = {}
for param_group in smil.findall(self._xpath_ns('./head/paramGroup', namespace)):
group_id = param_group.get(self._xpath_ns(
'id', 'http://www.w3.org/XML/1998/namespace'))
params = {}
for param in param_group:
params[param.get('name')] = param.get('value')
param_groups[group_id] = params
formats = []
for video in smil.findall(self._xpath_ns('.//video', namespace)):
src = video.get('src')
if not src:
continue
bitrate = int_or_none(self._search_regex(r'_(\d+)k', src, 'bitrate', None)) or float_or_none(video.get('system-bitrate') or video.get('systemBitrate'), 1000)
group_id = video.get('paramGroup')
param_group = param_groups[group_id]
for proto in param_group['protocols'].split(','):
formats.append({
'url': '%s://%s' % (proto, param_group['host']),
'app': param_group['app'],
'play_path': src,
'ext': 'flv',
'format_id': '%s-%d' % (proto, bitrate),
'tbr': bitrate,
})
self._sort_formats(formats)
return formats
def extract_from_xml_url(self, video_id, xml_url):
doc = self._download_xml(
xml_url, video_id,
note='Downloading video info',
errnote='Failed to download video info')
status_code = xpath_text(doc, './status/statuscode')
if status_code and status_code != 'ok':
if status_code == 'notVisibleAnymore':
message = 'Video %s is not available' % video_id
else:
message = '%s returned error: %s' % (self.IE_NAME, status_code)
raise ExtractorError(message, expected=True)
title = xpath_text(doc, './/information/title', 'title', True)
urls = []
formats = []
for fnode in doc.findall('.//formitaeten/formitaet'):
video_url = xpath_text(fnode, 'url')
if not video_url or video_url in urls:
continue
urls.append(video_url)
is_available = 'http://www.metafilegenerator' not in video_url
geoloced = 'static_geoloced_online' in video_url
if not is_available or geoloced:
continue
format_id = fnode.attrib['basetype']
format_m = re.match(r'''(?x)
(?P<vcodec>[^_]+)_(?P<acodec>[^_]+)_(?P<container>[^_]+)_
(?P<proto>[^_]+)_(?P<index>[^_]+)_(?P<indexproto>[^_]+)
''', format_id)
ext = determine_ext(video_url, None) or format_m.group('container')
if ext == 'meta':
continue
elif ext == 'smil':
formats.extend(self._extract_smil_formats(
video_url, video_id, fatal=False))
elif ext == 'm3u8':
# the certificates are misconfigured (see
# https://github.com/ytdl-org/youtube-dl/issues/8665)
if video_url.startswith('https://'):
continue
formats.extend(self._extract_m3u8_formats(
video_url, video_id, 'mp4', 'm3u8_native',
m3u8_id=format_id, fatal=False))
elif ext == 'f4m':
formats.extend(self._extract_f4m_formats(
video_url, video_id, f4m_id=format_id, fatal=False))
else:
quality = xpath_text(fnode, './quality')
if quality:
format_id += '-' + quality
abr = int_or_none(xpath_text(fnode, './audioBitrate'), 1000)
vbr = int_or_none(xpath_text(fnode, './videoBitrate'), 1000)
tbr = int_or_none(self._search_regex(
r'_(\d+)k', video_url, 'bitrate', None))
if tbr and vbr and not abr:
abr = tbr - vbr
formats.append({
'format_id': format_id,
'url': video_url,
'ext': ext,
'acodec': format_m.group('acodec'),
'vcodec': format_m.group('vcodec'),
'abr': abr,
'vbr': vbr,
'tbr': tbr,
'width': int_or_none(xpath_text(fnode, './width')),
'height': int_or_none(xpath_text(fnode, './height')),
'filesize': int_or_none(xpath_text(fnode, './filesize')),
'protocol': format_m.group('proto').lower(),
})
geolocation = xpath_text(doc, './/details/geolocation')
if not formats and geolocation and geolocation != 'none':
self.raise_geo_restricted(countries=self._GEO_COUNTRIES)
self._sort_formats(formats)
thumbnails = []
for node in doc.findall('.//teaserimages/teaserimage'):
thumbnail_url = node.text
if not thumbnail_url:
continue
thumbnail = {
'url': thumbnail_url,
}
thumbnail_key = node.get('key')
if thumbnail_key:
m = re.match('^([0-9]+)x([0-9]+)$', thumbnail_key)
if m:
thumbnail['width'] = int(m.group(1))
thumbnail['height'] = int(m.group(2))
thumbnails.append(thumbnail)
upload_date = unified_strdate(xpath_text(doc, './/details/airtime'))
return {
'id': video_id,
'title': title,
'description': xpath_text(doc, './/information/detail'),
'duration': int_or_none(xpath_text(doc, './/details/lengthSec')),
'thumbnails': thumbnails,
'uploader': xpath_text(doc, './/details/originChannelTitle'),
'uploader_id': xpath_text(doc, './/details/originChannelId'),
'upload_date': upload_date,
'formats': formats,
}
def _real_extract(self, url):
video_id = self._match_id(url)
details_url = 'http://www.3sat.de/mediathek/xmlservice/web/beitragsDetails?id=%s' % video_id
return self.extract_from_xml_url(video_id, details_url)

View File

@@ -242,7 +242,7 @@ class DRTVIE(InfoExtractor):
elif target == 'HLS':
formats.extend(self._extract_m3u8_formats(
uri, video_id, 'mp4', entry_protocol='m3u8_native',
preference=preference, m3u8_id=format_id,
quality=preference, m3u8_id=format_id,
fatal=False))
else:
bitrate = link.get('Bitrate')
@@ -254,7 +254,7 @@ class DRTVIE(InfoExtractor):
'tbr': int_or_none(bitrate),
'ext': link.get('FileFormat'),
'vcodec': 'none' if kind == 'AudioResource' else None,
'preference': preference,
'quality': preference,
})
subtitles_list = asset.get('SubtitlesList') or asset.get('Subtitleslist')
if isinstance(subtitles_list, list):

View File

@@ -154,7 +154,7 @@ class ESPNIE(OnceIE):
'tbr': int(mobj.group(3)),
})
if source_id == 'mezzanine':
f['preference'] = 1
f['quality'] = 1
formats.append(f)
links = clip.get('links', {})

View File

@@ -182,6 +182,7 @@ from .canvas import (
CanvasIE,
CanvasEenIE,
VrtNUIE,
DagelijkseKostIE,
)
from .carambatv import (
CarambaTVIE,
@@ -309,7 +310,12 @@ from .douyutv import (
DouyuShowIE,
DouyuTVIE,
)
from .dplay import DPlayIE
from .dplay import (
DPlayIE,
DiscoveryPlusIE,
HGTVDeIE,
)
from .dreisat import DreiSatIE
from .drbonanza import DRBonanzaIE
from .drtuber import DrTuberIE
from .drtv import (
@@ -1107,6 +1113,11 @@ from .shared import (
VivoIE,
)
from .showroomlive import ShowRoomLiveIE
from .simplecast import (
SimplecastIE,
SimplecastEpisodeIE,
SimplecastPodcastIE,
)
from .sina import SinaIE
from .sixplay import SixPlayIE
from .skyit import (
@@ -1165,11 +1176,6 @@ from .spike import (
BellatorIE,
ParamountNetworkIE,
)
from .storyfire import (
StoryFireIE,
StoryFireUserIE,
StoryFireSeriesIE,
)
from .stitcher import StitcherIE
from .sport5 import Sport5IE
from .sportbox import SportBoxIE
@@ -1193,6 +1199,11 @@ from .srgssr import (
from .srmediathek import SRMediathekIE
from .stanfordoc import StanfordOpenClassroomIE
from .steam import SteamIE
from .storyfire import (
StoryFireIE,
StoryFireUserIE,
StoryFireSeriesIE,
)
from .streamable import StreamableIE
from .streamcloud import StreamcloudIE
from .streamcz import StreamCZIE
@@ -1652,6 +1663,7 @@ from .zattoo import (
ZattooLiveIE,
)
from .zdf import ZDFIE, ZDFChannelIE
from .zhihu import ZhihuIE
from .zingmp3 import ZingMp3IE
from .zoom import ZoomIE
from .zype import ZypeIE

View File

@@ -619,7 +619,7 @@ class FacebookIE(InfoExtractor):
formats.append({
'format_id': '%s_%s_%s' % (format_id, quality, src_type),
'url': src,
'preference': preference,
'quality': preference,
})
extract_dash_manifest(f[0], formats)
subtitles_src = f[0].get('subtitles_src')

View File

@@ -104,7 +104,7 @@ class FirstTVIE(InfoExtractor):
'tbr': tbr,
'source_preference': quality(f.get('name')),
# quality metadata of http formats may be incorrect
'preference': -1,
'preference': -10,
})
# m3u8 URL format is reverse engineered from [1] (search for
# master.m3u8). dashEdges (that is currently balancer-vod.1tv.ru)

View File

@@ -88,7 +88,7 @@ class FlickrIE(InfoExtractor):
formats.append({
'format_id': stream_type,
'url': stream['_content'],
'preference': preference(stream_type),
'quality': preference(stream_type),
})
self._sort_formats(formats)

View File

@@ -133,6 +133,7 @@ from .bitchute import BitChuteIE
from .rumble import RumbleEmbedIE
from .arcpublishing import ArcPublishingIE
from .medialaan import MedialaanIE
from .simplecast import SimplecastIE
class GenericIE(InfoExtractor):
@@ -2240,6 +2241,15 @@ class GenericIE(InfoExtractor):
'duration': 159,
},
},
{
# Simplecast player embed
'url': 'https://www.bio.org/podcast',
'info_dict': {
'id': 'podcast',
'title': 'I AM BIO Podcast | BIO',
},
'playlist_mincount': 52,
},
]
def report_following_redirect(self, new_url):
@@ -2794,6 +2804,12 @@ class GenericIE(InfoExtractor):
return self.playlist_from_matches(
matches, video_id, video_title, getter=unescapeHTML, ie='FunnyOrDie')
# Look for Simplecast embeds
simplecast_urls = SimplecastIE._extract_urls(webpage)
if simplecast_urls:
return self.playlist_from_matches(
simplecast_urls, video_id, video_title)
# Look for BBC iPlayer embed
matches = re.findall(r'setPlaylist\("(https?://www\.bbc\.co\.uk/iplayer/[^/]+/[\da-z]{8})"\)', webpage)
if matches:

View File

@@ -236,7 +236,7 @@ class GoIE(AdobePassIE):
if re.search(r'(?:/mp4/source/|_source\.mp4)', asset_url):
f.update({
'format_id': ('%s-' % format_id if format_id else '') + 'SOURCE',
'preference': 1,
'quality': 1,
})
else:
mobj = re.search(r'/(\d+)x(\d+)/', asset_url)

View File

@@ -115,7 +115,7 @@ class HearThisAtIE(InfoExtractor):
'vcodec': 'none',
'ext': ext,
'url': download_url,
'preference': 2, # Usually better quality
'quality': 2, # Usually better quality
})
self._sort_formats(formats)

View File

@@ -100,7 +100,7 @@ class IGNIE(IGNBaseIE):
formats.append({
'ext': determine_ext(mezzanine_url, 'mp4'),
'format_id': 'mezzanine',
'preference': 1,
'quality': 1,
'url': mezzanine_url,
})

View File

@@ -72,7 +72,7 @@ class ImgurIE(InfoExtractor):
gif_json, video_id, transform_source=js_to_json)
formats.append({
'format_id': 'gif',
'preference': -10,
'preference': -10, # gifs are worse than videos
'width': width,
'height': height,
'ext': 'gif',

View File

@@ -373,7 +373,7 @@ class IqiyiIE(InfoExtractor):
'url': stream['m3utx'],
'format_id': vd,
'ext': 'mp4',
'preference': self._FORMATS_MAP.get(vd, -1),
'quality': self._FORMATS_MAP.get(vd, -1),
'protocol': 'm3u8_native',
})

View File

@@ -49,7 +49,7 @@ class KuwoBaseIE(InfoExtractor):
'url': song_url,
'format_id': file_format['format'],
'format': file_format['format'],
'preference': file_format['preference'],
'quality': file_format['preference'],
'abr': file_format.get('abr'),
})

View File

@@ -185,7 +185,7 @@ class LeIE(InfoExtractor):
f['height'] = int_or_none(format_id[:-1])
formats.append(f)
self._sort_formats(formats, ('height', 'quality', 'format_id'))
self._sort_formats(formats, ('res', 'quality'))
publish_time = parse_iso8601(self._html_search_regex(
r'发布时间&nbsp;([^<>]+) ', page, 'publish time', default=None),

View File

@@ -201,7 +201,7 @@ class LifeEmbedIE(InfoExtractor):
formats.append({
'url': original_url,
'format_id': determine_ext(original_url, None),
'preference': 1,
'quality': 1,
})
playlist = self._parse_json(

View File

@@ -175,7 +175,7 @@ class LimelightBaseIE(InfoExtractor):
formats.append({
'url': media_url,
'format_id': format_id,
'preference': -1,
'quality': -10,
'ext': ext,
})

View File

@@ -124,7 +124,10 @@ class LinkedInLearningIE(LinkedInLearningBaseIE):
streaming_url, video_slug, 'mp4',
'm3u8_native', m3u8_id='hls', fatal=False))
self._sort_formats(formats, ('width', 'height', 'source_preference', 'tbr', 'abr'))
# It seems like this would be correctly handled by default
# However, unless someone can confirm this, the old
# behaviour is being kept as-is
self._sort_formats(formats, ('res', 'source_preference'))
return {
'id': self._get_video_id(video_data, course_slug, video_slug),

View File

@@ -143,7 +143,7 @@ class LiveLeakIE(InfoExtractor):
formats.append({
'format_id': format_id,
'url': orig_url,
'preference': 1,
'quality': 1,
})
self._sort_formats(formats)
info_dict['formats'] = formats

View File

@@ -84,7 +84,7 @@ class LivestreamIE(InfoExtractor):
'format_id': 'smil_%d' % tbr,
'ext': 'flv',
'tbr': tbr,
'preference': -1000,
'preference': -1000, # Strictly inferior than all other formats?
})
return formats

View File

@@ -67,7 +67,7 @@ class MassengeschmackTVIE(InfoExtractor):
'vcodec': 'none' if format_id.startswith('Audio') else None,
})
self._sort_formats(formats, ('width', 'height', 'filesize', 'tbr'))
self._sort_formats(formats)
return {
'id': episode,

View File

@@ -137,11 +137,11 @@ class MDRIE(InfoExtractor):
if ext == 'm3u8':
formats.extend(self._extract_m3u8_formats(
video_url, video_id, 'mp4', entry_protocol='m3u8_native',
preference=0, m3u8_id='HLS', fatal=False))
quality=1, m3u8_id='HLS', fatal=False))
elif ext == 'f4m':
formats.extend(self._extract_f4m_formats(
video_url + '?hdcore=3.7.0&plugin=aasp-3.7.0.39.44', video_id,
preference=0, f4m_id='HDS', fatal=False))
quality=1, f4m_id='HDS', fatal=False))
else:
media_type = xpath_text(asset, './mediaType', 'media type', default='MP4')
vbr = int_or_none(xpath_text(asset, './bitrateVideo', 'vbr'), 1000)

View File

@@ -206,7 +206,7 @@ class MediasiteIE(InfoExtractor):
# disprefer 'secondary' streams
if stream_type != 0:
for fmt in stream_formats:
fmt['preference'] = -1
fmt['quality'] = -10
thumbnail_url = Stream.get('ThumbnailUrl')
if thumbnail_url:

View File

@@ -132,7 +132,7 @@ class MSNIE(InfoExtractor):
'width': int_or_none(file_.get('width')),
'height': int_or_none(file_.get('height')),
'vbr': int_or_none(self._search_regex(r'_(\d+)\.mp4', format_url, 'vbr', default=None)),
'preference': 1 if format_id == '1001' else None,
'quality': 1 if format_id == '1001' else None,
})
self._sort_formats(formats)

View File

@@ -61,7 +61,7 @@ class MuenchenTVIE(InfoExtractor):
'tbr': int_or_none(s.get('label')),
'ext': 'mp4',
'format_id': format_id,
'preference': -100 if '.smil' in s['file'] else 0,
'preference': -100 if '.smil' in s['file'] else 0, # Strictly inferior than all other formats?
})
self._sort_formats(formats)

View File

@@ -303,7 +303,7 @@ class NBABaseIE(NBACVPBaseIE):
formats.append({
'format_id': 'source',
'url': source_url,
'preference': 1,
'quality': 1,
})
m3u8_url = video.get('m3u8')

View File

@@ -546,11 +546,29 @@ class NiconicoIE(InfoExtractor):
webpage_url = get_video_info_web('watch_url') or url
# for channel movie and community movie
channel_id = try_get(
api_data,
(lambda x: x['channel']['globalId'],
lambda x: x['community']['globalId']))
channel = try_get(
api_data,
(lambda x: x['channel']['name'],
lambda x: x['community']['name']))
# Note: cannot use api_data.get('owner', {}) because owner may be set to "null"
# in the JSON, which will cause None to be returned instead of {}.
owner = try_get(api_data, lambda x: x.get('owner'), dict) or {}
uploader_id = get_video_info_web(['ch_id', 'user_id']) or owner.get('id')
uploader = get_video_info_web(['ch_name', 'user_nickname']) or owner.get('nickname')
uploader_id = (
get_video_info_web(['ch_id', 'user_id'])
or owner.get('id')
or channel_id
)
uploader = (
get_video_info_web(['ch_name', 'user_nickname'])
or owner.get('nickname')
or channel
)
return {
'id': video_id,
@@ -561,6 +579,8 @@ class NiconicoIE(InfoExtractor):
'uploader': uploader,
'timestamp': timestamp,
'uploader_id': uploader_id,
'channel': channel,
'channel_id': channel_id,
'view_count': view_count,
'comment_count': comment_count,
'duration': duration,

View File

@@ -2,10 +2,11 @@ from __future__ import unicode_literals
from .common import InfoExtractor
from ..utils import (
determine_ext,
ExtractorError,
determine_ext,
int_or_none,
try_get,
unescapeHTML,
url_or_none,
)
@@ -14,7 +15,7 @@ class NineGagIE(InfoExtractor):
IE_NAME = '9gag'
_VALID_URL = r'https?://(?:www\.)?9gag\.com/gag/(?P<id>[^/?&#]+)'
_TEST = {
_TESTS = [{
'url': 'https://9gag.com/gag/ae5Ag7B',
'info_dict': {
'id': 'ae5Ag7B',
@@ -29,7 +30,11 @@ class NineGagIE(InfoExtractor):
'dislike_count': int,
'comment_count': int,
}
}
}, {
# HTML escaped title
'url': 'https://9gag.com/gag/av5nvyb',
'only_matching': True,
}]
def _real_extract(self, url):
post_id = self._match_id(url)
@@ -43,7 +48,7 @@ class NineGagIE(InfoExtractor):
'The given url does not contain a video',
expected=True)
title = post['title']
title = unescapeHTML(post['title'])
duration = None
formats = []

View File

@@ -183,7 +183,7 @@ class NocoIE(InfoExtractor):
'filesize': int_or_none(fmt.get('filesize')),
'format_note': qualities[format_id].get('quality_name'),
'quality': qualities[format_id].get('priority'),
'preference': preference,
'language_preference': preference,
})
self._sort_formats(formats)

View File

@@ -425,7 +425,7 @@ class NPOIE(NPOBaseIE):
stream_url, video_id, fatal=False)
# f4m downloader downloads only piece of live stream
for f4m_format in f4m_formats:
f4m_format['preference'] = -1
f4m_format['preference'] = -5
formats.extend(f4m_formats)
elif stream_type == 'hls':
formats.extend(self._extract_m3u8_formats(

View File

@@ -62,7 +62,7 @@ class NTVDeIE(InfoExtractor):
m3u8_url = compat_urlparse.urljoin('http://video.n-tv.de', vdata['videoM3u8'])
formats.extend(self._extract_m3u8_formats(
m3u8_url, video_id, ext='mp4', entry_protocol='m3u8_native',
preference=0, m3u8_id='hls', fatal=False))
quality=1, m3u8_id='hls', fatal=False))
self._sort_formats(formats)
return {

View File

@@ -72,7 +72,7 @@ class NYTimesBaseIE(InfoExtractor):
'tbr': int_or_none(video.get('bitrate'), 1000) or None,
'ext': ext,
})
self._sort_formats(formats, ('height', 'width', 'filesize', 'tbr', 'fps', 'format_id'))
self._sort_formats(formats)
thumbnails = []
for image in video_data.get('images', []):

View File

@@ -55,7 +55,7 @@ class OraTVIE(InfoExtractor):
formats.append({
'url': http_template % q,
'format_id': q,
'preference': preference(q),
'quality': preference(q),
})
self._sort_formats(formats)
else:

View File

@@ -78,7 +78,7 @@ class PicartoIE(InfoExtractor):
update_url_query(
'https://%s/hls/%s/index.m3u8'
% (edge_ep, channel_id), params),
channel_id, 'mp4', preference=preference,
channel_id, 'mp4', quality=preference,
m3u8_id='-'.join(format_id), fatal=False))
continue
elif tech_type == 'video/mp4' or tech_label == 'MP4':
@@ -88,7 +88,7 @@ class PicartoIE(InfoExtractor):
'https://%s/mp4/%s.mp4' % (edge_ep, channel_id),
params),
'format_id': '-'.join(format_id),
'preference': preference,
'quality': preference,
})
else:
# rtmp format does not seem to work

View File

@@ -54,8 +54,7 @@ class PinterestBaseIE(InfoExtractor):
'height': int_or_none(format_dict.get('height')),
'duration': duration,
})
self._sort_formats(
formats, field_preference=('height', 'width', 'tbr', 'format_id'))
self._sort_formats(formats)
description = data.get('description') or data.get('description_html') or data.get('seo_description')
timestamp = unified_timestamp(data.get('created_at'))

View File

@@ -150,7 +150,7 @@ class PlaytvakIE(InfoExtractor):
ext = 'mp4'
# Some streams have mp3 audio which does not play
# well with ffmpeg filter aac_adtstoasc
preference = -1
preference = -10
elif format_ == 'adobe': # f4m manifest fails with 404 in 80% of requests
continue
else: # Other formats not supported yet

View File

@@ -121,7 +121,7 @@ class QQMusicIE(InfoExtractor):
% (details['prefix'], mid, details['ext'], vkey, guid),
'format': format_id,
'format_id': format_id,
'preference': details['preference'],
'quality': details['preference'],
'abr': details.get('abr'),
})
self._check_formats(formats, mid)

View File

@@ -43,7 +43,7 @@ class RadioFranceIE(InfoExtractor):
'format_id': fm[0],
'url': fm[1],
'vcodec': 'none',
'preference': i,
'quality': i,
}
for i, fm in
enumerate(re.findall(r"([a-z0-9]+)\s*:\s*'([^']+)'", formats_str))

View File

@@ -40,7 +40,7 @@ class RedditIE(InfoExtractor):
'https://v.redd.it/%s/DASHPlaylist.mpd' % video_id, video_id,
mpd_id='dash', fatal=False))
self._sort_formats(formats, ('height', 'width'))
self._sort_formats(formats)
return {
'id': video_id,

View File

@@ -93,7 +93,7 @@ class RTL2IE(InfoExtractor):
'flash_version': 'LNX 11,2,202,429',
'rtmp_conn': rtmp_conn,
'no_resume': True,
'preference': 1,
'quality': 1,
})
m3u8_url = video_info.get('streamurl_hls')

View File

@@ -45,7 +45,7 @@ class RTPIE(InfoExtractor):
if file_key:
formats.append({
'url': 'https://cdn-ondemand.rtp.pt' + file_key,
'preference': 1,
'quality': 1,
})
self._sort_formats(formats)
else:

View File

@@ -180,11 +180,11 @@ class RUTVIE(InfoExtractor):
'rtmp_live': True,
'ext': 'flv',
'vbr': int(quality),
'preference': preference,
'quality': preference,
}
elif transport == 'm3u8':
formats.extend(self._extract_m3u8_formats(
url, video_id, 'mp4', preference=preference, m3u8_id='hls'))
url, video_id, 'mp4', quality=preference, m3u8_id='hls'))
continue
else:
fmt = {

View File

@@ -80,7 +80,9 @@ class SendtoNewsIE(InfoExtractor):
'format_id': '%s-%d' % (determine_protocol(f), tbr),
'tbr': tbr,
})
self._sort_formats(info_dict['formats'], ('tbr', 'height', 'width', 'format_id'))
# 'tbr' was explicitly set to be prefered over 'height' originally,
# So this is being kept unless someone can confirm this is unnecessary
self._sort_formats(info_dict['formats'], ('tbr', 'res'))
thumbnails = []
if video.get('thumbnailUrl'):

View File

@@ -0,0 +1,160 @@
# coding: utf-8
from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..utils import (
clean_podcast_url,
int_or_none,
parse_iso8601,
strip_or_none,
try_get,
urlencode_postdata,
)
class SimplecastBaseIE(InfoExtractor):
_UUID_REGEX = r'[\da-f]{8}-(?:[\da-f]{4}-){3}[\da-f]{12}'
_API_BASE = 'https://api.simplecast.com/'
def _call_api(self, path_tmpl, video_id):
return self._download_json(
self._API_BASE + path_tmpl % video_id, video_id)
def _call_search_api(self, resource, resource_id, resource_url):
return self._download_json(
'https://api.simplecast.com/%ss/search' % resource, resource_id,
data=urlencode_postdata({'url': resource_url}))
def _parse_episode(self, episode):
episode_id = episode['id']
title = episode['title'].strip()
audio_file = episode.get('audio_file') or {}
audio_file_url = audio_file.get('url') or episode.get('audio_file_url') or episode['enclosure_url']
season = episode.get('season') or {}
season_href = season.get('href')
season_id = None
if season_href:
season_id = self._search_regex(
r'https?://api.simplecast.com/seasons/(%s)' % self._UUID_REGEX,
season_href, 'season id', default=None)
webpage_url = episode.get('episode_url')
channel_url = None
if webpage_url:
channel_url = self._search_regex(
r'(https?://[^/]+\.simplecast\.com)',
webpage_url, 'channel url', default=None)
return {
'id': episode_id,
'display_id': episode.get('slug'),
'title': title,
'url': clean_podcast_url(audio_file_url),
'webpage_url': webpage_url,
'channel_url': channel_url,
'series': try_get(episode, lambda x: x['podcast']['title']),
'season_number': int_or_none(season.get('number')),
'season_id': season_id,
'thumbnail': episode.get('image_url'),
'episode_id': episode_id,
'episode_number': int_or_none(episode.get('number')),
'description': strip_or_none(episode.get('description')),
'timestamp': parse_iso8601(episode.get('published_at')),
'duration': int_or_none(episode.get('duration')),
'filesize': int_or_none(audio_file.get('size') or episode.get('audio_file_size')),
}
class SimplecastIE(SimplecastBaseIE):
IE_NAME = 'simplecast'
_VALID_URL = r'https?://(?:api\.simplecast\.com/episodes|player\.simplecast\.com)/(?P<id>%s)' % SimplecastBaseIE._UUID_REGEX
_COMMON_TEST_INFO = {
'display_id': 'errant-signal-chris-franklin-new-wave-video-essays',
'id': 'b6dc49a2-9404-4853-9aa9-9cfc097be876',
'ext': 'mp3',
'title': 'Errant Signal - Chris Franklin & New Wave Video Essays',
'episode_number': 1,
'episode_id': 'b6dc49a2-9404-4853-9aa9-9cfc097be876',
'description': 'md5:34752789d3d2702e2d2c975fbd14f357',
'season_number': 1,
'season_id': 'e23df0da-bae4-4531-8bbf-71364a88dc13',
'series': 'The RE:BIND.io Podcast',
'duration': 5343,
'timestamp': 1580979475,
'upload_date': '20200206',
'webpage_url': r're:^https?://the-re-bind-io-podcast\.simplecast\.com/episodes/errant-signal-chris-franklin-new-wave-video-essays',
'channel_url': r're:^https?://the-re-bind-io-podcast\.simplecast\.com$',
}
_TESTS = [{
'url': 'https://api.simplecast.com/episodes/b6dc49a2-9404-4853-9aa9-9cfc097be876',
'md5': '8c93be7be54251bf29ee97464eabe61c',
'info_dict': _COMMON_TEST_INFO,
}, {
'url': 'https://player.simplecast.com/b6dc49a2-9404-4853-9aa9-9cfc097be876',
'only_matching': True,
}]
@staticmethod
def _extract_urls(webpage):
return re.findall(
r'''(?x)<iframe[^>]+src=["\']
(
https?://(?:embed\.simplecast\.com/[0-9a-f]{8}|
player\.simplecast\.com/%s
))''' % SimplecastBaseIE._UUID_REGEX, webpage)
def _real_extract(self, url):
episode_id = self._match_id(url)
episode = self._call_api('episodes/%s', episode_id)
return self._parse_episode(episode)
class SimplecastEpisodeIE(SimplecastBaseIE):
IE_NAME = 'simplecast:episode'
_VALID_URL = r'https?://(?!api\.)[^/]+\.simplecast\.com/episodes/(?P<id>[^/?&#]+)'
_TEST = {
'url': 'https://the-re-bind-io-podcast.simplecast.com/episodes/errant-signal-chris-franklin-new-wave-video-essays',
'md5': '8c93be7be54251bf29ee97464eabe61c',
'info_dict': SimplecastIE._COMMON_TEST_INFO,
}
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
episode = self._call_search_api(
'episode', mobj.group(1), mobj.group(0))
return self._parse_episode(episode)
class SimplecastPodcastIE(SimplecastBaseIE):
IE_NAME = 'simplecast:podcast'
_VALID_URL = r'https?://(?!(?:api|cdn|embed|feeds|player)\.)(?P<id>[^/]+)\.simplecast\.com(?!/episodes/[^/?&#]+)'
_TESTS = [{
'url': 'https://the-re-bind-io-podcast.simplecast.com',
'playlist_mincount': 33,
'info_dict': {
'id': '07d28d26-7522-42eb-8c53-2bdcfc81c43c',
'title': 'The RE:BIND.io Podcast',
},
}, {
'url': 'https://the-re-bind-io-podcast.simplecast.com/episodes',
'only_matching': True,
}]
def _real_extract(self, url):
subdomain = self._match_id(url)
site = self._call_search_api('site', subdomain, url)
podcast = site['podcast']
podcast_id = podcast['id']
podcast_title = podcast.get('title')
def entries():
episodes = self._call_api('podcasts/%s/episodes', podcast_id)
for episode in (episodes.get('collection') or []):
info = self._parse_episode(episode)
info['series'] = podcast_title
yield info
return self.playlist_result(entries(), podcast_id, podcast_title)

View File

@@ -99,7 +99,7 @@ class SinaIE(InfoExtractor):
formats.append({
'format_id': quality_id,
'url': update_url_query(file_api, {'vid': file_id}),
'preference': preference(quality_id),
'quality': preference(quality_id),
'ext': 'mp4',
})
self._sort_formats(formats)

View File

@@ -422,7 +422,7 @@ class SoundcloudIE(InfoExtractor):
'ext': urlhandle_detect_ext(urlh) or 'mp3',
'filesize': int_or_none(urlh.headers.get('Content-Length')),
'url': format_url,
'preference': 10,
'quality': 10,
})
def invalid_url(url):

View File

@@ -129,7 +129,7 @@ class SpankBangIE(InfoExtractor):
format_url = format_url[0]
extract_format(format_id, format_url)
self._sort_formats(formats, field_preference=('preference', 'height', 'width', 'fps', 'tbr', 'format_id'))
self._sort_formats(formats)
info = self._search_json_ld(webpage, video_id, default={})

View File

@@ -108,7 +108,7 @@ class SpankwireIE(InfoExtractor):
formats.extend(self._extract_m3u8_formats(
m3u8_url, video_id, 'mp4', entry_protocol='m3u8_native',
m3u8_id='hls', fatal=False))
self._sort_formats(formats, ('height', 'tbr', 'width', 'format_id'))
self._sort_formats(formats)
view_count = str_to_int(video.get('viewed'))

View File

@@ -89,7 +89,7 @@ class SRGSSRIE(InfoExtractor):
formats.append({
'format_id': format_id,
'url': asset_url,
'preference': preference(quality),
'quality': preference(quality),
'ext': 'flv' if protocol == 'RTMP' else None,
})
self._sort_formats(formats)

View File

@@ -1,255 +1,151 @@
# coding: utf-8
from __future__ import unicode_literals
import itertools
import functools
from .common import InfoExtractor
from ..utils import (
# HEADRequest,
int_or_none,
OnDemandPagedList,
smuggle_url,
)
class StoryFireIE(InfoExtractor):
_VALID_URL = r'(?:(?:https?://(?:www\.)?storyfire\.com/video-details)|(?:https://storyfire.app.link))/(?P<id>[^/\s]+)'
_TESTS = [{
class StoryFireBaseIE(InfoExtractor):
_VALID_URL_BASE = r'https?://(?:www\.)?storyfire\.com/'
def _call_api(self, path, video_id, resource, query=None):
return self._download_json(
'https://storyfire.com/app/%s/%s' % (path, video_id), video_id,
'Downloading %s JSON metadata' % resource, query=query)
def _parse_video(self, video):
title = video['title']
vimeo_id = self._search_regex(
r'https?://player\.vimeo\.com/external/(\d+)',
video['vimeoVideoURL'], 'vimeo id')
# video_url = self._request_webpage(
# HEADRequest(video['vimeoVideoURL']), video_id).geturl()
# formats = []
# for v_url, suffix in [(video_url, '_sep'), (video_url.replace('/sep/video/', '/video/'), '')]:
# formats.extend(self._extract_m3u8_formats(
# v_url, video_id, 'mp4', 'm3u8_native',
# m3u8_id='hls' + suffix, fatal=False))
# formats.extend(self._extract_mpd_formats(
# v_url.replace('.m3u8', '.mpd'), video_id,
# mpd_id='dash' + suffix, fatal=False))
# self._sort_formats(formats)
uploader_id = video.get('hostID')
return {
'_type': 'url_transparent',
'id': vimeo_id,
'title': title,
'description': video.get('description'),
'url': smuggle_url(
'https://player.vimeo.com/video/' + vimeo_id, {
'http_headers': {
'Referer': 'https://storyfire.com/',
}
}),
# 'formats': formats,
'thumbnail': video.get('storyImage'),
'view_count': int_or_none(video.get('views')),
'like_count': int_or_none(video.get('likesCount')),
'comment_count': int_or_none(video.get('commentsCount')),
'duration': int_or_none(video.get('videoDuration')),
'timestamp': int_or_none(video.get('publishDate')),
'uploader': video.get('username'),
'uploader_id': uploader_id,
'uploader_url': 'https://storyfire.com/user/%s/video' % uploader_id if uploader_id else None,
'episode_number': int_or_none(video.get('episodeNumber') or video.get('episode_number')),
}
class StoryFireIE(StoryFireBaseIE):
_VALID_URL = StoryFireBaseIE._VALID_URL_BASE + r'video-details/(?P<id>[0-9a-f]{24})'
_TEST = {
'url': 'https://storyfire.com/video-details/5df1d132b6378700117f9181',
'md5': '560953bfca81a69003cfa5e53ac8a920',
'md5': 'caec54b9e4621186d6079c7ec100c1eb',
'info_dict': {
'id': '5df1d132b6378700117f9181',
'id': '378954662',
'ext': 'mp4',
'title': 'Buzzfeed Teaches You About Memes',
'uploader_id': 'ntZAJFECERSgqHSxzonV5K2E89s1',
'timestamp': 1576129028,
'description': 'Mocking Buzzfeed\'s meme lesson. Reuploaded from YouTube because of their new policies',
'description': 'md5:0b4e28021548e144bed69bb7539e62ea',
'uploader': 'whang!',
'upload_date': '20191212',
'duration': 418,
'view_count': int,
'like_count': int,
'comment_count': int,
},
'params': {'format': 'bestvideo'} # There are no merged formats in the playlist.
}, {
'url': 'https://storyfire.app.link/5GxAvWOQr8', # Alternate URL format, with unrelated short ID
'md5': '7a2dc6d60c4889edfed459c620fe690d',
'info_dict': {
'id': '5f1e11ecd78a57b6c702001d',
'ext': 'm4a',
'title': 'Weird Nintendo Prototype Leaks',
'description': 'A stream taking a look at some weird Nintendo Prototypes with Luigi in Mario 64 and weird Yoshis',
'timestamp': 1595808576,
'upload_date': '20200727',
'uploader': 'whang!',
'uploader_id': 'ntZAJFECERSgqHSxzonV5K2E89s1',
'params': {
'skip_download': True,
},
'params': {'format': 'bestaudio'} # Verifying audio extraction
}]
_aformats = {
'audio-medium-audio': {'acodec': 'aac', 'abr': 125, 'preference': -10},
'audio-high-audio': {'acodec': 'aac', 'abr': 254, 'preference': -1},
'expected_warnings': ['Unable to download JSON metadata']
}
def _real_extract(self, url):
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
# Extracting the json blob is mandatory to proceed with extraction.
jsontext = self._html_search_regex(
r'<script id="__NEXT_DATA__" type="application/json">(.+?)</script>',
webpage, 'json_data')
json = self._parse_json(jsontext, video_id)
# The currentVideo field in the json is mandatory
# because it contains the only link to the m3u playlist
video = json['props']['initialState']['video']['currentVideo']
videourl = video['vimeoVideoURL'] # Video URL is mandatory
# Extract other fields from the json in an error tolerant fashion
# ID may be incorrect (on short URL format), correct it.
parsed_id = video.get('_id')
if parsed_id:
video_id = parsed_id
title = video.get('title')
description = video.get('description')
thumbnail = video.get('storyImage')
views = video.get('views')
likes = video.get('likesCount')
comments = video.get('commentsCount')
duration = video.get('videoDuration')
publishdate = video.get('publishDate') # Apparently epoch time, day only
uploader = video.get('username')
uploader_id = video.get('hostID')
# Construct an uploader URL
uploader_url = None
if uploader_id:
uploader_url = "https://storyfire.com/user/%s/video" % uploader_id
# Collect root playlist to determine formats
formats = self._extract_m3u8_formats(
videourl, video_id, 'mp4', 'm3u8_native')
# Modify formats to fill in missing information about audio codecs
for format in formats:
aformat = self._aformats.get(format['format_id'])
if aformat:
format['acodec'] = aformat['acodec']
format['abr'] = aformat['abr']
format['preference'] = aformat['preference']
format['ext'] = 'm4a'
self._sort_formats(formats)
return {
'id': video_id,
'title': title,
'description': description,
'ext': "mp4",
'url': videourl,
'formats': formats,
'thumbnail': thumbnail,
'view_count': views,
'like_count': likes,
'comment_count': comments,
'duration': duration,
'timestamp': publishdate,
'uploader': uploader,
'uploader_id': uploader_id,
'uploader_url': uploader_url,
}
video = self._call_api(
'generic/video-detail', video_id, 'video')['video']
return self._parse_video(video)
class StoryFireUserIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?storyfire\.com/user/(?P<id>[^/\s]+)/video'
_TESTS = [{
'url': 'https://storyfire.com/user/ntZAJFECERSgqHSxzonV5K2E89s1/video',
'info_dict': {
'id': 'ntZAJFECERSgqHSxzonV5K2E89s1',
'title': 'whang!',
},
'playlist_mincount': 18
}, {
class StoryFireUserIE(StoryFireBaseIE):
_VALID_URL = StoryFireBaseIE._VALID_URL_BASE + r'user/(?P<id>[^/]+)/video'
_TEST = {
'url': 'https://storyfire.com/user/UQ986nFxmAWIgnkZQ0ftVhq4nOk2/video',
'info_dict': {
'id': 'UQ986nFxmAWIgnkZQ0ftVhq4nOk2',
'title': 'McJuggerNuggets',
},
'playlist_mincount': 143
'playlist_mincount': 151,
}
_PAGE_SIZE = 20
}]
# Generator for fetching playlist items
def _enum_videos(self, baseurl, user_id, firstjson):
totalVideos = int(firstjson['videosCount'])
haveVideos = 0
json = firstjson
for page in itertools.count(1):
for video in json['videos']:
id = video['_id']
url = "https://storyfire.com/video-details/%s" % id
haveVideos += 1
yield {
'_type': 'url',
'id': id,
'url': url,
'ie_key': 'StoryFire',
'title': video.get('title'),
'description': video.get('description'),
'view_count': video.get('views'),
'comment_count': video.get('commentsCount'),
'duration': video.get('videoDuration'),
'timestamp': video.get('publishDate'),
}
# Are there more pages we could fetch?
if haveVideos < totalVideos:
pageurl = baseurl + ("%i" % haveVideos)
json = self._download_json(pageurl, user_id,
note='Downloading page %s' % page)
# Are there any videos in the new json?
videos = json.get('videos')
if not videos or len(videos) == 0:
break # no videos
else:
break # We have fetched all the videos, stop
def _fetch_page(self, user_id, page):
videos = self._call_api(
'publicVideos', user_id, 'page %d' % (page + 1), {
'skip': page * self._PAGE_SIZE,
})['videos']
for video in videos:
yield self._parse_video(video)
def _real_extract(self, url):
user_id = self._match_id(url)
baseurl = "https://storyfire.com/app/publicVideos/%s?skip=" % user_id
# Download first page to ensure it can be downloaded, and get user information if available.
firstpage = baseurl + "0"
firstjson = self._download_json(firstpage, user_id)
title = None
videos = firstjson.get('videos')
if videos and len(videos):
title = videos[1].get('username')
return {
'_type': 'playlist',
'entries': self._enum_videos(baseurl, user_id, firstjson),
'id': user_id,
'title': title,
}
entries = OnDemandPagedList(functools.partial(
self._fetch_page, user_id), self._PAGE_SIZE)
return self.playlist_result(entries, user_id)
class StoryFireSeriesIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?storyfire\.com/write/series/stories/(?P<id>[^/\s]+)'
class StoryFireSeriesIE(StoryFireBaseIE):
_VALID_URL = StoryFireBaseIE._VALID_URL_BASE + r'write/series/stories/(?P<id>[^/?&#]+)'
_TESTS = [{
'url': 'https://storyfire.com/write/series/stories/-Lq6MsuIHLODO6d2dDkr/',
'info_dict': {
'id': '-Lq6MsuIHLODO6d2dDkr',
},
'playlist_mincount': 13
'playlist_mincount': 13,
}, {
'url': 'https://storyfire.com/write/series/stories/the_mortal_one/',
'info_dict': {
'id': 'the_mortal_one',
},
'playlist_count': 0 # This playlist has entries, but no videos.
}, {
'url': 'https://storyfire.com/write/series/stories/story_time',
'info_dict': {
'id': 'story_time',
},
'playlist_mincount': 10
'playlist_count': 0,
}]
# Generator for returning playlist items
# This object is substantially different than the one in the user videos page above
def _enum_videos(self, jsonlist):
for video in jsonlist:
id = video['_id']
if video.get('hasVideo'): # Boolean element
url = "https://storyfire.com/video-details/%s" % id
yield {
'_type': 'url',
'id': id,
'url': url,
'ie_key': 'StoryFire',
'title': video.get('title'),
'description': video.get('description'),
'view_count': video.get('views'),
'likes_count': video.get('likesCount'),
'comment_count': video.get('commentsCount'),
'duration': video.get('videoDuration'),
'timestamp': video.get('publishDate'),
}
def _extract_videos(self, stories):
for story in stories.values():
if story.get('hasVideo'):
yield self._parse_video(story)
def _real_extract(self, url):
list_id = self._match_id(url)
listurl = "https://storyfire.com/app/seriesStories/%s/list" % list_id
json = self._download_json(listurl, list_id)
return {
'_type': 'playlist',
'entries': self._enum_videos(json),
'id': list_id
}
series_id = self._match_id(url)
stories = self._call_api(
'seriesStories', series_id, 'series stories')
return self.playlist_result(self._extract_videos(stories), series_id)

View File

@@ -70,7 +70,7 @@ class Tele13IE(InfoExtractor):
formats.append({
'url': format_url,
'format_id': f.get('label'),
'preference': preference(f.get('label')),
'quality': preference(f.get('label')),
'ext': ext,
})
urls.append(format_url)

View File

@@ -57,7 +57,7 @@ class TeleMBIE(InfoExtractor):
'app': rtmp.group('app'),
'player_url': 'http://p.jwpcdn.com/6/10/jwplayer.flash.swf',
'page_url': 'http://www.telemb.be',
'preference': -1,
'preference': -10,
})
formats.append(fmt)
self._sort_formats(formats)

View File

@@ -133,12 +133,10 @@ class ThreeQSDNIE(InfoExtractor):
'vcodec': 'none' if height == 0 else None,
'width': width,
})
for f in formats:
if f.get('acodec') == 'none':
f['preference'] = -40
elif f.get('vcodec') == 'none':
f['preference'] = -50
self._sort_formats(formats, ('preference', 'width', 'height', 'source_preference', 'tbr', 'vbr', 'abr', 'ext', 'format_id'))
# It seems like this would be correctly handled by default
# However, unless someone can confirm this, the old
# behaviour is being kept as-is
self._sort_formats(formats, ('res', 'source_preference'))
subtitles = {}
for subtitle in (config.get('subtitles') or []):

View File

@@ -405,7 +405,7 @@ class UdemyIE(InfoExtractor):
if f.get('url'):
formats.append(f)
self._sort_formats(formats, field_preference=('height', 'width', 'tbr', 'format_id'))
self._sort_formats(formats)
return {
'id': video_id,

View File

@@ -91,7 +91,7 @@ class UMGDeIE(InfoExtractor):
if not formats:
for format_id in (867, 836, 940):
add_m3u8_format(format_id)
self._sort_formats(formats, ('width', 'height', 'filesize', 'tbr'))
self._sort_formats(formats)
return {
'id': video_id,

View File

@@ -110,7 +110,6 @@ class UOLIE(InfoExtractor):
'format_id': format_id,
'url': f_url,
'quality': quality(format_id),
'preference': -1,
})
self._sort_formats(formats)

View File

@@ -44,7 +44,7 @@ class UrortIE(InfoExtractor):
'ext': f['FileType'],
'format_id': '%s-%s' % (f['FileType'], f.get('Quality', '')),
'url': 'http://p3urort.blob.core.windows.net/tracks/%s' % f['FileRef'],
'preference': 3 if f['FileType'] == 'mp3' else 2,
'quality': 3 if f['FileType'] == 'mp3' else 2,
} for f in s['Files']]
self._sort_formats(formats)
e = {

View File

@@ -4,21 +4,22 @@ from __future__ import unicode_literals
import re
from .common import InfoExtractor
from ..compat import compat_str
from ..utils import (
determine_ext,
float_or_none,
int_or_none,
parse_age_limit,
qualities,
random_birthday,
try_get,
unified_timestamp,
urljoin,
)
class VideoPressIE(InfoExtractor):
_VALID_URL = r'https?://videopress\.com/embed/(?P<id>[\da-zA-Z]+)'
_ID_REGEX = r'[\da-zA-Z]{8}'
_PATH_REGEX = r'video(?:\.word)?press\.com/embed/'
_VALID_URL = r'https?://%s(?P<id>%s)' % (_PATH_REGEX, _ID_REGEX)
_TESTS = [{
'url': 'https://videopress.com/embed/kUJmAcSf',
'md5': '706956a6c875873d51010921310e4bc6',
@@ -36,35 +37,36 @@ class VideoPressIE(InfoExtractor):
# 17+, requires birth_* params
'url': 'https://videopress.com/embed/iH3gstfZ',
'only_matching': True,
}, {
'url': 'https://video.wordpress.com/embed/kUJmAcSf',
'only_matching': True,
}]
@staticmethod
def _extract_urls(webpage):
return re.findall(
r'<iframe[^>]+src=["\']((?:https?://)?videopress\.com/embed/[\da-zA-Z]+)',
r'<iframe[^>]+src=["\']((?:https?://)?%s%s)' % (VideoPressIE._PATH_REGEX, VideoPressIE._ID_REGEX),
webpage)
def _real_extract(self, url):
video_id = self._match_id(url)
query = random_birthday('birth_year', 'birth_month', 'birth_day')
query['fields'] = 'description,duration,file_url_base,files,height,original,poster,rating,title,upload_date,width'
video = self._download_json(
'https://public-api.wordpress.com/rest/v1.1/videos/%s' % video_id,
video_id, query=query)
title = video['title']
def base_url(scheme):
return try_get(
video, lambda x: x['file_url_base'][scheme], compat_str)
base_url = base_url('https') or base_url('http')
file_url_base = video.get('file_url_base') or {}
base_url = file_url_base.get('https') or file_url_base.get('http')
QUALITIES = ('std', 'dvd', 'hd')
quality = qualities(QUALITIES)
formats = []
for format_id, f in video['files'].items():
for format_id, f in (video.get('files') or {}).items():
if not isinstance(f, dict):
continue
for ext, path in f.items():
@@ -75,12 +77,14 @@ class VideoPressIE(InfoExtractor):
'ext': determine_ext(path, ext),
'quality': quality(format_id),
})
original_url = try_get(video, lambda x: x['original'], compat_str)
original_url = video.get('original')
if original_url:
formats.append({
'url': original_url,
'format_id': 'original',
'quality': len(QUALITIES),
'width': int_or_none(video.get('width')),
'height': int_or_none(video.get('height')),
})
self._sort_formats(formats)

View File

@@ -181,6 +181,7 @@ class VidmeIE(InfoExtractor):
'url': format_url,
'width': int_or_none(f.get('width')),
'height': int_or_none(f.get('height')),
# Clips should never be prefered over full video
'preference': 0 if f.get('type', '').endswith(
'clip') else 1,
})

View File

@@ -134,7 +134,7 @@ class ViewLiftEmbedIE(ViewLiftBaseIE):
if hls_url:
formats.extend(self._extract_m3u8_formats(
hls_url, film_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
self._sort_formats(formats, ('height', 'tbr', 'format_id'))
self._sort_formats(formats)
info = {
'id': film_id,

View File

@@ -22,6 +22,7 @@ from ..utils import (
parse_iso8601,
sanitized_Request,
std_headers,
try_get,
)
@@ -31,7 +32,7 @@ class VikiBaseIE(InfoExtractor):
_API_URL_TEMPLATE = 'https://api.viki.io%s&sig=%s'
_APP = '100005a'
_APP_VERSION = '2.2.5.1428709186'
_APP_VERSION = '6.0.0'
_APP_SECRET = 'MM_d*yP@`&1@]@!AVrXf_o-HVEnoTnm$O-ti4[G~$JDI/Dc-&piU&z&5.;:}95=Iad'
_GEO_BYPASS = False
@@ -42,7 +43,7 @@ class VikiBaseIE(InfoExtractor):
_ERRORS = {
'geo': 'Sorry, this content is not available in your region.',
'upcoming': 'Sorry, this content is not yet available.',
# 'paywall': 'paywall',
'paywall': 'Sorry, this content is only available to Viki Pass Plus subscribers',
}
def _prepare_call(self, path, timestamp=None, post_data=None):
@@ -63,14 +64,25 @@ class VikiBaseIE(InfoExtractor):
def _call_api(self, path, video_id, note, timestamp=None, post_data=None):
resp = self._download_json(
self._prepare_call(path, timestamp, post_data), video_id, note)
self._prepare_call(path, timestamp, post_data),
video_id, note,
headers={
'x-client-user-agent': std_headers['User-Agent'],
'x-viki-as-id': self._APP,
'x-viki-app-ver': self._APP_VERSION,
})
error = resp.get('error')
if error:
if error == 'invalid timestamp':
resp = self._download_json(
self._prepare_call(path, int(resp['current_timestamp']), post_data),
video_id, '%s (retry)' % note)
video_id, '%s (retry)' % note,
headers={
'x-client-user-agent': std_headers['User-Agent'],
'x-viki-as-id': self._APP,
'x-viki-app-ver': self._APP_VERSION,
})
error = resp.get('error')
if error:
self._raise_error(resp['error'])
@@ -83,11 +95,13 @@ class VikiBaseIE(InfoExtractor):
expected=True)
def _check_errors(self, data):
for reason, status in data.get('blocking', {}).items():
for reason, status in (data.get('blocking') or {}).items():
if status and reason in self._ERRORS:
message = self._ERRORS[reason]
if reason == 'geo':
self.raise_geo_restricted(msg=message)
elif reason == 'paywall':
self.raise_login_required(message)
raise ExtractorError('%s said: %s' % (
self.IE_NAME, message), expected=True)
@@ -132,13 +146,19 @@ class VikiIE(VikiBaseIE):
'info_dict': {
'id': '1023585v',
'ext': 'mp4',
'title': 'Heirs Episode 14',
'uploader': 'SBS',
'description': 'md5:c4b17b9626dd4b143dcc4d855ba3474e',
'title': 'Heirs - Episode 14',
'uploader': 'SBS Contents Hub',
'timestamp': 1385047627,
'upload_date': '20131121',
'age_limit': 13,
'duration': 3570,
'episode_number': 14,
},
'params': {
'format': 'bestvideo',
},
'skip': 'Blocked in the US',
'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
}, {
# clip
'url': 'http://www.viki.com/videos/1067139v-the-avengers-age-of-ultron-press-conference',
@@ -154,7 +174,8 @@ class VikiIE(VikiBaseIE):
'uploader': 'Arirang TV',
'like_count': int,
'age_limit': 0,
}
},
'skip': 'Sorry. There was an error loading this video',
}, {
'url': 'http://www.viki.com/videos/1048879v-ankhon-dekhi',
'info_dict': {
@@ -172,7 +193,7 @@ class VikiIE(VikiBaseIE):
}, {
# episode
'url': 'http://www.viki.com/videos/44699v-boys-over-flowers-episode-1',
'md5': '94e0e34fd58f169f40c184f232356cfe',
'md5': '0a53dc252e6e690feccd756861495a8c',
'info_dict': {
'id': '44699v',
'ext': 'mp4',
@@ -184,6 +205,10 @@ class VikiIE(VikiBaseIE):
'uploader': 'group8',
'like_count': int,
'age_limit': 13,
'episode_number': 1,
},
'params': {
'format': 'bestvideo',
},
'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
}, {
@@ -210,7 +235,7 @@ class VikiIE(VikiBaseIE):
}, {
# non-English description
'url': 'http://www.viki.com/videos/158036v-love-in-magic',
'md5': 'adf9e321a0ae5d0aace349efaaff7691',
'md5': '41faaba0de90483fb4848952af7c7d0d',
'info_dict': {
'id': '158036v',
'ext': 'mp4',
@@ -221,6 +246,10 @@ class VikiIE(VikiBaseIE):
'title': 'Love In Magic',
'age_limit': 13,
},
'params': {
'format': 'bestvideo',
},
'expected_warnings': ['Unknown MIME type image/jpeg in DASH manifest'],
}]
def _real_extract(self, url):
@@ -230,29 +259,27 @@ class VikiIE(VikiBaseIE):
'https://www.viki.com/api/videos/' + video_id,
video_id, 'Downloading video JSON', headers={
'x-client-user-agent': std_headers['User-Agent'],
'x-viki-app-ver': '4.0.57',
'x-viki-as-id': self._APP,
'x-viki-app-ver': self._APP_VERSION,
})
video = resp['video']
self._check_errors(video)
title = self.dict_selection(video.get('titles', {}), 'en', allow_fallback=False)
episode_number = int_or_none(video.get('number'))
if not title:
title = 'Episode %d' % video.get('number') if video.get('type') == 'episode' else video.get('id') or video_id
container_titles = video.get('container', {}).get('titles', {})
title = 'Episode %d' % episode_number if video.get('type') == 'episode' else video.get('id') or video_id
container_titles = try_get(video, lambda x: x['container']['titles'], dict) or {}
container_title = self.dict_selection(container_titles, 'en')
title = '%s - %s' % (container_title, title)
description = self.dict_selection(video.get('descriptions', {}), 'en')
duration = int_or_none(video.get('duration'))
timestamp = parse_iso8601(video.get('created_at'))
uploader = video.get('author')
like_count = int_or_none(video.get('likes', {}).get('count'))
age_limit = parse_age_limit(video.get('rating'))
like_count = int_or_none(try_get(video, lambda x: x['likes']['count']))
thumbnails = []
for thumbnail_id, thumbnail in video.get('images', {}).items():
for thumbnail_id, thumbnail in (video.get('images') or {}).items():
thumbnails.append({
'id': thumbnail_id,
'url': thumbnail.get('url'),
@@ -263,7 +290,12 @@ class VikiIE(VikiBaseIE):
# New way to fetch subtitles
new_video = self._download_json(
'https://www.viki.com/api/videos/%s' % video_id, video_id,
'Downloading new video JSON to get subtitles', fatal=False)
'Downloading new video JSON to get subtitles', fatal=False,
headers={
'x-client-user-agent': std_headers['User-Agent'],
'x-viki-as-id': self._APP,
'x-viki-app-ver': self._APP_VERSION,
})
for sub in new_video.get('streamSubtitles').get('dash'):
subtitles[sub.get('srclang')] = [{
'ext': 'vtt',
@@ -272,7 +304,7 @@ class VikiIE(VikiBaseIE):
}]
except AttributeError:
# fall-back to the old way if there isn't a streamSubtitles attribute
for subtitle_lang, _ in video.get('subtitle_completions', {}).items():
for subtitle_lang, _ in (video.get('subtitle_completions') or {}).items():
subtitles[subtitle_lang] = [{
'ext': subtitles_format,
'url': self._prepare_call(
@@ -283,13 +315,15 @@ class VikiIE(VikiBaseIE):
'id': video_id,
'title': title,
'description': description,
'duration': duration,
'timestamp': timestamp,
'uploader': uploader,
'duration': int_or_none(video.get('duration')),
'timestamp': parse_iso8601(video.get('created_at')),
'uploader': video.get('author'),
'uploader_url': video.get('author_url'),
'like_count': like_count,
'age_limit': age_limit,
'age_limit': parse_age_limit(video.get('rating')),
'thumbnails': thumbnails,
'subtitles': subtitles,
'episode_number': episode_number,
}
formats = []
@@ -383,7 +417,7 @@ class VikiChannelIE(VikiBaseIE):
'info_dict': {
'id': '50c',
'title': 'Boys Over Flowers',
'description': 'md5:ecd3cff47967fe193cff37c0bec52790',
'description': 'md5:804ce6e7837e1fd527ad2f25420f4d59',
},
'playlist_mincount': 71,
}, {
@@ -394,6 +428,7 @@ class VikiChannelIE(VikiBaseIE):
'description': 'md5:05bf5471385aa8b21c18ad450e350525',
},
'playlist_count': 127,
'skip': 'Page not found',
}, {
'url': 'http://www.viki.com/news/24569c-showbiz-korea',
'only_matching': True,

View File

@@ -116,7 +116,8 @@ class VimeoBaseInfoExtractor(InfoExtractor):
def _vimeo_sort_formats(self, formats):
# Bitrates are completely broken. Single m3u8 may contain entries in kbps and bps
# at the same time without actual units specified. This lead to wrong sorting.
self._sort_formats(formats, field_preference=('preference', 'height', 'width', 'fps', 'tbr', 'format_id'))
# But since yt-dlp prefers 'res,fps' anyway, 'field_preference' is not needed
self._sort_formats(formats)
def _parse_config(self, config, video_id):
video_data = config['video']
@@ -178,16 +179,9 @@ class VimeoBaseInfoExtractor(InfoExtractor):
formats.append({
'format_id': 'live-archive-source',
'url': live_archive_source_url,
'preference': 1,
'quality': 10,
})
# Reduntant code! This is already done in common.py
# for f in formats:
# if f.get('vcodec') == 'none':
# f['preference'] = -50
# elif f.get('acodec') == 'none':
# f['preference'] = -40
subtitles = {}
text_tracks = config['request'].get('text_tracks')
if text_tracks:
@@ -227,10 +221,12 @@ class VimeoBaseInfoExtractor(InfoExtractor):
'is_live': is_live,
}
def _extract_original_format(self, url, video_id):
def _extract_original_format(self, url, video_id, unlisted_hash=None):
query = {'action': 'load_download_config'}
if unlisted_hash:
query['unlisted_hash'] = unlisted_hash
download_data = self._download_json(
url, video_id, fatal=False,
query={'action': 'load_download_config'},
url, video_id, fatal=False, query=query,
headers={'X-Requested-With': 'XMLHttpRequest'})
if download_data:
source_file = download_data.get('source_file')
@@ -250,7 +246,7 @@ class VimeoBaseInfoExtractor(InfoExtractor):
'height': int_or_none(source_file.get('height')),
'filesize': parse_filesize(source_file.get('size')),
'format_id': source_name,
'preference': 1,
'quality': 1,
}
@@ -510,6 +506,11 @@ class VimeoIE(VimeoBaseInfoExtractor):
{
'url': 'https://vimeo.com/160743502/abd0e13fb4',
'only_matching': True,
},
{
# requires passing unlisted_hash(a52724358e) to load_download_config request
'url': 'https://vimeo.com/392479337/a52724358e',
'only_matching': True,
}
# https://gettingthingsdone.com/workflowmap/
# vimeo embed with check-password page protected by Referer header
@@ -674,7 +675,8 @@ class VimeoIE(VimeoBaseInfoExtractor):
if config.get('view') == 4:
config = self._verify_player_video_password(redirect_url, video_id, headers)
vod = config.get('video', {}).get('vod', {})
video = config.get('video') or {}
vod = video.get('vod') or {}
def is_rented():
if '>You rented this title.<' in webpage:
@@ -734,7 +736,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
formats = []
source_format = self._extract_original_format(
'https://vimeo.com/' + video_id, video_id)
'https://vimeo.com/' + video_id, video_id, video.get('unlisted_hash'))
if source_format:
formats.append(source_format)

View File

@@ -70,7 +70,7 @@ class VzaarIE(InfoExtractor):
f = {
'url': source_url,
'format_id': 'http',
'preference': 1,
'quality': 1,
}
if 'audio' in source_url:
f.update({

Some files were not shown because too many files have changed in this diff Show More