Release 2021.02.19

Update to ytdl-commit-cf2dbec
cf2dbec630 Except: [kakao] improve info extraction and detect geo restriction d8085580f6
2025-12-17 03:12:24 +01:00 · 2021-02-20 03:14:01 +05:30 · 2021-02-20 02:32:22 +05:30 · 2021-02-19 18:21:29 +05:30 · 2021-02-19 17:04:25 +05:30 · 2021-02-19 04:52:05 +05:30
150 changed files with 2428 additions and 1191 deletions
--- a/.github/ISSUE_TEMPLATE/1_broken_site.md
+++ b/.github/ISSUE_TEMPLATE/1_broken_site.md
@@ -21,7 +21,7 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->

 - [ ] I'm reporting a broken site support
- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.02.04
+ [debug] yt-dlp version 2021.02.16
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/.github/ISSUE_TEMPLATE/2_site_support_request.md
+++ b/.github/ISSUE_TEMPLATE/2_site_support_request.md
@@ -21,7 +21,7 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->

 - [ ] I'm reporting a new site support request
- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones
--- a/.github/ISSUE_TEMPLATE/3_site_feature_request.md
+++ b/.github/ISSUE_TEMPLATE/3_site_feature_request.md
@@ -21,13 +21,13 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->

 - [ ] I'm reporting a site feature request
- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones


--- a/.github/ISSUE_TEMPLATE/4_bug_report.md
+++ b/.github/ISSUE_TEMPLATE/4_bug_report.md
@@ -21,7 +21,7 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->

 - [ ] I'm reporting a broken site support issue
- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.02.04
+ [debug] yt-dlp version 2021.02.16
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/.github/ISSUE_TEMPLATE/5_feature_request.md
+++ b/.github/ISSUE_TEMPLATE/5_feature_request.md
@@ -21,13 +21,13 @@ assignees: ''

 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.04. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.02.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->

 - [ ] I'm reporting a feature request
- [ ] I've verified that I'm running yt-dlp version **2021.02.04**
+- [ ] I've verified that I'm running yt-dlp version **2021.02.16**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones


--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -55,10 +55,11 @@ jobs:
        asset_content_type: application/octet-stream
    - name: Get SHA2-256SUMS for youtube-dlc
      id: sha2_file
-      env:
-        SHA2: ${{ hashFiles('youtube-dlc') }}
-      run: echo "::set-output name=sha2_unix::$SHA2"
+      run: echo "::set-output name=sha2_unix::$(sha256sum youtube-dlc | awk '{print $1}')"
    - name: Install dependencies for pypi
+      env:
+        PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
+      if: "env.PYPI_TOKEN != ''"
      run: |
        python -m pip install --upgrade pip
        pip install setuptools wheel twine
@@ -66,6 +67,7 @@ jobs:
      env:
        TWINE_USERNAME: __token__
        TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
+      if: "env.TWINE_PASSWORD != ''"
      run: |
        rm -rf dist/*
        python setup.py sdist bdist_wheel
@@ -75,6 +77,9 @@ jobs:

    runs-on: windows-latest

+    outputs:
+      sha2_windows: ${{ steps.sha2_file_win.outputs.sha2_windows }}
+
    needs: build_unix

    steps:
@@ -83,8 +88,10 @@ jobs:
      uses: actions/setup-python@v2
      with:
          python-version: '3.8'
+    - name: Upgrade pip and enable wheel support
+      run: python -m pip install --upgrade pip setuptools wheel
    - name: Install Requirements
-      run: pip install pyinstaller mutagen Crypto
+      run: pip install pyinstaller mutagen pycryptodome
    - name: Bump version
      id: bump_version
      run: python devscripts/update-version.py
@@ -104,14 +111,15 @@ jobs:
        asset_content_type: application/vnd.microsoft.portable-executable
    - name: Get SHA2-256SUMS for youtube-dlc.exe
      id: sha2_file_win
-      env:
-        SHA2_win: ${{ hashFiles('dist/youtube-dlc.exe') }}
-      run: echo "::set-output name=sha2_windows::$SHA2_win"
+      run: echo "::set-output name=sha2_windows::$((Get-FileHash dist\youtube-dlc.exe -Algorithm SHA256).Hash.ToLower())"

  build_windows32:

    runs-on: windows-latest

+    outputs:
+      sha2_windows32: ${{ steps.sha2_file_win32.outputs.sha2_windows32 }}
+
    needs: [build_unix, build_windows]

    steps:
@@ -121,8 +129,10 @@ jobs:
      with:
          python-version: '3.4.4'
          architecture: 'x86'
+    - name: Upgrade pip and enable wheel support
+      run: python -m pip install pip==19.1.1 setuptools==43.0.0 wheel==0.33.6
    - name: Install Requirements for 32 Bit
-      run: pip install pyinstaller==3.5 mutagen Crypto
+      run: pip install pyinstaller==3.5 mutagen==1.42.0 pycryptodome==3.9.4
    - name: Bump version
      id: bump_version
      run: python devscripts/update-version.py
@@ -142,9 +152,7 @@ jobs:
        asset_content_type: application/vnd.microsoft.portable-executable
    - name: Get SHA2-256SUMS for youtube-dlc_x86.exe
      id: sha2_file_win32
-      env:
-        SHA2_win32: ${{ hashFiles('dist/youtube-dlc_x86.exe') }}
-      run: echo "::set-output name=sha2_windows32::$SHA2_win32"
+      run: echo "::set-output name=sha2_windows32::$((Get-FileHash dist\youtube-dlc_x86.exe -Algorithm SHA256).Hash.ToLower())"
    - name: Make SHA2-256SUMS file
      env:
        SHA2_WINDOWS: ${{ needs.build_windows.outputs.sha2_windows }}
@@ -167,16 +175,3 @@ jobs:
        asset_path: ./SHA2-256SUMS
        asset_name: SHA2-256SUMS
        asset_content_type: text/plain
-
-#  update_version_badge:
-#    runs-on: ubuntu-latest
-#    needs: build_unix
-#    steps:
-#    - name: Create Version Badge
-#      uses: schneegans/dynamic-badges-action@v1.0.0
-#      with:
-#        auth: ${{ secrets.GIST_TOKEN }}
-#        gistID: c69cb23c3c5b3316248e52022790aa57
-#        filename: version.json
-#        label: Version
-#        message: ${{ needs.build_unix.outputs.ytdlc_version }}
--- a/.gitignore
+++ b/.gitignore
@@ -36,6 +36,7 @@ README.txt
 # Binary
 youtube-dl
 youtube-dlc
+youtube-dlc.zip
 *.exe

 # Downloaded
@@ -55,6 +56,9 @@ youtube-dlc
 *.swf
 *.part
 *.ytdl
+*.frag
+*.frag.urls
+*.aria2
 *.swp
 *.ogg
 *.opus
--- a/5
+++ b/5
@@ -17,4 +17,7 @@ alxnull
 FelixFrog
 Zocker1999NET
 nao20010128nao
-shirt-dev
+shirt-dev
+kurumigi
+tsukumi
+bbepis
--- a/Changelog.md
+++ b/Changelog.md
@@ -8,7 +8,7 @@
 * Change "Merged with youtube-dl" version in Readme.md if needed
 * Commit to master as `Release <version>`
 * Push to origin/release - build task will now run
-* Update version.py using devscripts\update-version.py (be wary of timezones)
+* Update version.py using devscripts\update-version.py
 * Run `make issuetemplates`
 * Commit to master as `[version] update :ci skip all`
 * Push to origin/master
@@ -17,6 +17,56 @@
 -->


+### 2021.02.19
+* **Merge youtube-dl:** Upto [commit/cf2dbec](https://github.com/ytdl-org/youtube-dl/commit/cf2dbec6301177a1fddf72862de05fa912d9869d) (except kakao)
+* [viki] Fix extractor
+* [niconico] Extract `channel` and `channel_id` by [kurumigi](https://github.com/kurumigi)
+* [youtube] Multiple page support for hashtag URLs
+* [youtube] Add more invidious instances
+* [youtube] Fix comment extraction when comment text is empty
+* Option `--windows-filenames` to force use of windows compatible filenames
+* [ExtractAudio] Bugfix
+* Don't raise `parser.error` when exiting for update
+* [MoveFiles] Fix for when merger can't run
+* Changed `--trim-file-name` to `--trim-filenames` to be similar to related options
+* Format Sort improvements:
+    * Prefer `vp9.2` more than other `vp9` codecs
+    * Remove forced priority of `quality`
+    * Remove unnecessary `field_preference` and misuse of `preference` from extractors
+* Build improvements:
+    * Fix hash output by [shirt](https://github.com/shirt-dev)
+    * Lock python package versions for x86 and use `wheels` by [shirt](https://github.com/shirt-dev)
+    * Exclude `vcruntime140.dll` from UPX by [jbruchon](https://github.com/jbruchon)
+    * Set version number based on UTC time, not local time
+    * Publish on PyPi only if token is set
+* [documentation] Better document `--prefer-free-formats` and add `--no-prefer-free-format`
+
+
+### 2021.02.15
+* **Merge youtube-dl:** Upto [2021.02.10](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.10) (except archive.org)
+* [niconico] Improved extraction and support encrypted/SMILE movies by [kurumigi](https://github.com/kurumigi), [tsukumi](https://github.com/tsukumi), [bbepis](https://github.com/bbepis), [pukkandan](https://github.com/pukkandan)
+* Fix HLS AES-128 with multiple keys in external downloaders by [shirt](https://github.com/shirt-dev)
+* [youtube_live_chat] Fix by using POST API by [siikamiika](https://github.com/siikamiika)
+* [rumble] Add support for video page
+* Option `--allow-unplayable-formats` to allow downloading unplayable video formats
+* [ExtractAudio] Don't re-encode when file is already in a common audio format
+* [youtube] Fix search continuations
+* [youtube] Fix for new accounts
+* Improve build/updater: by [pukkandan](https://github.com/pukkandan) and [shirt](https://github.com/shirt-dev)
+    * Fix SHA256 calculation in build and implement hash checking for updater
+    * Exit immediately in windows once the update process starts
+    * Fix updater for `x86.exe`
+    * Updater looks for both `yt-dlp` and `youtube-dlc` in releases for future-proofing
+    * Change optional dependency to `pycryptodome`
+* Fix issue with unicode filenames in aria2c by [shirt](https://github.com/shirt-dev)
+* Fix `allow_playlist_files` not being correctly passed through
+* Fix for empty HTTP head requests by [shirt](https://github.com/shirt-dev)
+* Fix `get_executable_path` in UNIX
+* [sponskrub] Print ffmpeg output and errors to terminal
+* `__real_download` should be false when ffmpeg unavailable and no download
+* Show `exe`/`zip`/`source` and 32/64bit in verbose message
+
+
 ### 2021.02.09
 * **aria2c support for DASH/HLS**: by [shirt](https://github.com/shirt-dev)
 * **Implement Updater** (`-U`) by [shirt](https://github.com/shirt-dev)
@@ -57,7 +107,7 @@


 ### 2021.01.29
-* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: Co-authored by [animelover1984](https://github.com/animelover1984) and [bbepis](https://github.com/bbepis)
+* **Features from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl)**: by [animelover1984](https://github.com/animelover1984) and [bbepis](https://github.com/bbepis)
    * Add `--get-comments`
    * [youtube] Extract comments
    * [billibilli] Added BiliBiliSearchIE, BilibiliChannelIE
@@ -205,7 +255,7 @@
    * Changed video format sorting to show video only files and video+audio files together.
    * Added `--video-multistreams`, `--no-video-multistreams`, `--audio-multistreams`, `--no-audio-multistreams`
    * Added `b`,`w`,`v`,`a` as alias for `best`, `worst`, `video` and `audio` respectively
-* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by [h-h-h-h](https://github.com/h-h-h-h) - See [Internet Shortcut Options]README.md(#internet-shortcut-options) for details
+* **Shortcut Options:** Added `--write-link`, `--write-url-link`, `--write-webloc-link`, `--write-desktop-link` by [h-h-h-h](https://github.com/h-h-h-h) - See [Internet Shortcut Options](README.md#internet-shortcut-options) for details
 * **Sponskrub integration:** Added `--sponskrub`, `--sponskrub-cut`, `--sponskrub-force`, `--sponskrub-location`, `--sponskrub-args` - See [SponSkrub Options](README.md#sponskrub-options-sponsorblock) for details
 * Added `--force-download-archive` (`--force-write-archive`) by [h-h-h-h](https://github.com/h-h-h-h)
 * Added `--list-formats-as-table`,  `--list-formats-old`
--- a/2
+++ b/2
@@ -4,7 +4,7 @@ man: README.txt youtube-dlc.1 youtube-dlc.bash-completion youtube-dlc.zsh youtub


 clean:
-	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
+	rm -rf youtube-dlc.1.temp.md youtube-dlc.1 youtube-dlc.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dlc.tar.gz youtube-dlc.zsh youtube-dlc.fish youtube_dlc/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.spec *.frag *.frag.urls *.frag.aria2 CONTRIBUTING.md.tmp youtube-dlc youtube-dlc.exe
 	find . -name "*.pyc" -delete
 	find . -name "*.class" -delete

--- a/README.md
+++ b/README.md
@@ -56,18 +56,20 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/

 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))

-* **Merged with youtube-dl v2021.02.04.1**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl v2021.02.10**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)

 * **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--get-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, Playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/pukkandan/yt-dlp/pull/31) for details.

 * **Youtube improvements**:
-    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and support downloading multiple pages of content
-    * Youtube search works correctly (`ytsearch:`, `ytsearchdate:`) along with Search URLs
+    * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and supports downloading multiple pages of content
+    * Youtube search (`ytsearch:`, `ytsearchdate:`) along with Search URLs works correctly
    * Redirect channel's home URL automatically to `/video` to preserve the old behaviour

+* **Aria2c with HLS/DASH**: You can use aria2c as the external downloader for DASH(mpd) and HLS(m3u8) formats. No more slow ffmpeg/native downloads
+
 * **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius

-* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina
+* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina, rumble

 * **Plugin support**: Extractors can be loaded from an external file. See [plugins](#plugins) for details

@@ -81,6 +83,8 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/

 * **Improvements**: Multiple `--postprocessor-args` and `--external-downloader-args`, Date/time formatting in `-o`, faster archive checking, more [format selection options](#format-selection) etc

+* **Self-updater**: The releases can be updated using `youtube-dlc -U`
+

 See [changelog](Changelog.md) or [commits](https://github.com/pukkandan/yt-dlp/commits) for the full list of changes

@@ -93,20 +97,21 @@ If you are coming from [youtube-dl](https://github.com/ytdl-org/youtube-dl), the
 # INSTALLATION

 You can install yt-dlp using one of the following methods:
+* Download the binary from the [latest release](https://github.com/pukkandan/yt-dlp/releases/latest) (recommended method)
 * Use [PyPI package](https://pypi.org/project/yt-dlp): `python -m pip install --upgrade yt-dlp`
-* Download the binary from the [latest release](https://github.com/pukkandan/yt-dlp/releases/latest)
 * Use pip+git: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp.git@release`
 * Install master branch: `python -m pip install --upgrade git+https://github.com/pukkandan/yt-dlp`

 ### UPDATE
-`-U` does not work. Simply repeat the install process to update.
+Starting from version `2021.02.09`, you can use `youtube-dlc -U` to update if you are using the provided release.
+If you are using `pip`, simply re-run the same command that was used to install the program.

 ### COMPILE

 **For Windows**:
-To build the Windows executable, you must have pyinstaller (and optionally mutagen and Crypto)
+To build the Windows executable, you must have pyinstaller (and optionally mutagen and pycryptodome)

-    python -m pip install --upgrade pyinstaller mutagen Crypto
+    python -m pip install --upgrade pyinstaller mutagen pycryptodome

 Once you have all the necessary dependancies installed, just run `py pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it. It is strongly reccomended to use python3 although python2.6+ is supported.

@@ -136,9 +141,9 @@ Then simply type this
 ## General Options:
    -h, --help                       Print this help text and exit
    --version                        Print program version and exit
-    -U, --update                     [BROKEN] Update this program to latest
-                                     version. Make sure that you have sufficient
-                                     permissions (run with sudo if needed)
+    -U, --update                     Update this program to latest version. Make
+                                     sure that you have sufficient permissions
+                                     (run with sudo if needed)
    -i, --ignore-errors              Continue on download errors, for example to
                                     skip unavailable videos in a playlist
                                     (default) (Alias: --no-abort-on-error)
@@ -172,7 +177,6 @@ Then simply type this
                                     containing directory
    --flat-playlist                  Do not extract the videos of a playlist,
                                     only list them
-    --flat-videos                    Do not resolve the video urls
    --no-flat-playlist               Extract the videos of a playlist
    --mark-watched                   Mark videos watched (YouTube only)
    --no-mark-watched                Do not mark videos watched
@@ -356,6 +360,12 @@ Then simply type this
                                     filenames
    --no-restrict-filenames          Allow Unicode characters, "&" and spaces in
                                     filenames (default)
+    --windows-filenames              Force filenames to be windows compatible
+    --no-windows-filenames           Make filenames windows compatible only if
+                                     using windows (default)
+    --trim-filenames LENGTH          Limit the filename length (excluding
+                                     extension) to the specified number of
+                                     characters
    -w, --no-overwrites              Do not overwrite any files
    --force-overwrites               Overwrite all video and metadata files.
                                     This option includes --no-continue
@@ -406,8 +416,6 @@ Then simply type this
                                     may change
    --no-cache-dir                   Disable filesystem caching
    --rm-cache-dir                   Delete all filesystem cache files
-    --trim-file-name LENGTH          Limit the filename length (extension
-                                     excluded)

 ## Thumbnail Images:
    --write-thumbnail                Write thumbnail image to disk
@@ -463,10 +471,6 @@ Then simply type this
                                     files in the current directory to debug
                                     problems
    --print-traffic                  Display sent and read HTTP traffic
-    -C, --call-home                  [Broken] Contact the youtube-dlc server for
-                                     debugging
-    --no-call-home                   Do not contact the youtube-dlc server for
-                                     debugging (default)

 ## Workarounds:
    --encoding ENCODING              Force the specified encoding (experimental)
@@ -515,8 +519,12 @@ Then simply type this
    --no-audio-multistreams          Only one audio stream is downloaded for
                                     each output file (default)
    --all-formats                    Download all available video formats
-    --prefer-free-formats            Prefer free video formats over non-free
-                                     formats of same quality
+    --prefer-free-formats            Prefer video formats with free containers
+                                     over non-free ones of same quality. Use
+                                     with "-S ext" to strictly prefer free
+                                     containers irrespective of quality
+    --no-prefer-free-formats         Don't give any special preference to free
+                                     containers (default)
    -F, --list-formats               List all available formats of requested
                                     videos
    --list-formats-as-table          Present the output of -F in tabular form
@@ -539,6 +547,11 @@ Then simply type this
                                     bestvideo+bestaudio), output to given
                                     container format. One of mkv, mp4, ogg,
                                     webm, flv. Ignored if no merge is required
+    --allow-unplayable-formats       Allow unplayable formats to be listed and
+                                     downloaded. All video postprocessing will
+                                     also be turned off
+    --no-allow-unplayable-formats    Do not allow unplayable formats to be
+                                     listed or downloaded (default)

 ## Subtitle Options:
    --write-subs                     Write subtitle file
@@ -972,10 +985,10 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
 - `hasaud`: Gives priority to formats that has a audio stream
 - `ie_pref`: The format preference as given by the extractor
 - `lang`: Language preference as given by the extractor
- - `quality`: The quality of the format. This is a metadata field available in some websites
+ - `quality`: The quality of the format as given by the extractor
 - `source`: Preference of the source as given by the extractor
 - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
- - `vcodec`: Video Codec (`av01` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
+ - `vcodec`: Video Codec (`av01` > `vp9.2` > `vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
 - `acodec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
 - `codec`: Equivalent to `vcodec,acodec`
 - `vext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
@@ -994,9 +1007,9 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
 - `br`: Equivalent to using `tbr,vbr,abr`
 - `asr`: Audio sample rate in Hz

-Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
+Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.

-The fields `hasvid`, `ie_pref`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec:vp9,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
+The fields `hasvid`, `ie_pref`, `lang` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `quality,res,fps,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.

 If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all repects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.

--- a/devscripts/update-version.py
+++ b/devscripts/update-version.py
@@ -13,7 +13,7 @@ old_version_list = old_version.split(".", 4)
 old_ver = '.'.join(old_version_list[:3])
 old_rev = old_version_list[3] if len(old_version_list) > 3 else ''

-ver = datetime.now().strftime("%Y.%m.%d")
+ver = datetime.utcnow().strftime("%Y.%m.%d")
 rev = str(int(old_rev or 0) + 1) if old_ver == ver else ''

 VERSION = '.'.join((ver, rev)) if rev else ver
--- a/docs/supportedsites.md
+++ b/docs/supportedsites.md
@@ -1,11 +1,11 @@
 # Supported sites
 - **1tv**: Первый канал
- - **1up.com**
 - **20min**
 - **220.ro**
 - **23video**
 - **24video**
 - **3qsdn**: 3Q SDN
+ - **3sat**
 - **4tube**
 - **56.com**
 - **5min**
@@ -223,6 +223,7 @@
 - **curiositystream**
 - **curiositystream:collection**
 - **CWTV**
+ - **DagelijkseKost**: dagelijksekost.een.be
 - **DailyMail**
 - **dailymotion**
 - **dailymotion:playlist**
@@ -245,6 +246,7 @@
 - **DiscoveryGo**
 - **DiscoveryGoPlaylist**
 - **DiscoveryNetworksDe**
+ - **DiscoveryPlus**
 - **DiscoveryVR**
 - **Disney**
 - **dlive:stream**
@@ -371,6 +373,7 @@
 - **HentaiStigma**
 - **hetklokhuis**
 - **hgtv.com:show**
+ - **HGTVDe**
 - **HiDive**
 - **HistoricFilms**
 - **history:player**
@@ -394,6 +397,8 @@
 - **HungamaSong**
 - **Hypem**
 - **ign.com**
+ - **IGNArticle**
+ - **IGNVideo**
 - **IHeartRadio**
 - **iheartradio:podcast**
 - **imdb**: Internet Movie Database trailers
@@ -701,7 +706,6 @@
 - **parliamentlive.tv**: UK parliament videos
 - **Patreon**
 - **pbs**: Public Broadcasting Service (PBS) and member stations: PBS: Public Broadcasting Service, APT - Alabama Public Television (WBIQ), GPB/Georgia Public Broadcasting (WGTV), Mississippi Public Broadcasting (WMPN), Nashville Public Television (WNPT), WFSU-TV (WFSU), WSRE (WSRE), WTCI (WTCI), WPBA/Channel 30 (WPBA), Alaska Public Media (KAKM), Arizona PBS (KAET), KNME-TV/Channel 5 (KNME), Vegas PBS (KLVX), AETN/ARKANSAS ETV NETWORK (KETS), KET (WKLE), WKNO/Channel 10 (WKNO), LPB/LOUISIANA PUBLIC BROADCASTING (WLPB), OETA (KETA), Ozarks Public Television (KOZK), WSIU Public Broadcasting (WSIU), KEET TV (KEET), KIXE/Channel 9 (KIXE), KPBS San Diego (KPBS), KQED (KQED), KVIE Public Television (KVIE), PBS SoCal/KOCE (KOCE), ValleyPBS (KVPT), CONNECTICUT PUBLIC TELEVISION (WEDH), KNPB Channel 5 (KNPB), SOPTV (KSYS), Rocky Mountain PBS (KRMA), KENW-TV3 (KENW), KUED Channel 7 (KUED), Wyoming PBS (KCWC), Colorado Public Television / KBDI 12 (KBDI), KBYU-TV (KBYU), Thirteen/WNET New York (WNET), WGBH/Channel 2 (WGBH), WGBY (WGBY), NJTV Public Media NJ (WNJT), WLIW21 (WLIW), mpt/Maryland Public Television (WMPB), WETA Television and Radio (WETA), WHYY (WHYY), PBS 39 (WLVT), WVPT - Your Source for PBS and More! (WVPT), Howard University Television (WHUT), WEDU PBS (WEDU), WGCU Public Media (WGCU), WPBT2 (WPBT), WUCF TV (WUCF), WUFT/Channel 5 (WUFT), WXEL/Channel 42 (WXEL), WLRN/Channel 17 (WLRN), WUSF Public Broadcasting (WUSF), ETV (WRLK), UNC-TV (WUNC), PBS Hawaii - Oceanic Cable Channel 10 (KHET), Idaho Public Television (KAID), KSPS (KSPS), OPB (KOPB), KWSU/Channel 10 & KTNW/Channel 31 (KWSU), WILL-TV (WILL), Network Knowledge - WSEC/Springfield (WSEC), WTTW11 (WTTW), Iowa Public Television/IPTV (KDIN), Nine Network (KETC), PBS39 Fort Wayne (WFWA), WFYI Indianapolis (WFYI), Milwaukee Public Television (WMVS), WNIN (WNIN), WNIT Public Television (WNIT), WPT (WPNE), WVUT/Channel 22 (WVUT), WEIU/Channel 51 (WEIU), WQPT-TV (WQPT), WYCC PBS Chicago (WYCC), WIPB-TV (WIPB), WTIU (WTIU), CET  (WCET), ThinkTVNetwork (WPTD), WBGU-TV (WBGU), WGVU TV (WGVU), NET1 (KUON), Pioneer Public Television (KWCM), SDPB Television (KUSD), TPT (KTCA), KSMQ (KSMQ), KPTS/Channel 8 (KPTS), KTWU/Channel 11 (KTWU), East Tennessee PBS (WSJK), WCTE-TV (WCTE), WLJT, Channel 11 (WLJT), WOSU TV (WOSU), WOUB/WOUC (WOUB), WVPB (WVPB), WKYU-PBS (WKYU), KERA 13 (KERA), MPBN (WCBB), Mountain Lake PBS (WCFE), NHPTV (WENH), Vermont PBS (WETK), witf (WITF), WQED Multimedia (WQED), WMHT Educational Telecommunications (WMHT), Q-TV (WDCQ), WTVS Detroit Public TV (WTVS), CMU Public Television (WCMU), WKAR-TV (WKAR), WNMU-TV Public TV 13 (WNMU), WDSE - WRPT (WDSE), WGTE TV (WGTE), Lakeland Public Television (KAWE), KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS), MontanaPBS (KUSM), KRWG/Channel 22 (KRWG), KACV (KACV), KCOS/Channel 13 (KCOS), WCNY/Channel 24 (WCNY), WNED (WNED), WPBS (WPBS), WSKG Public TV (WSKG), WXXI (WXXI), WPSU (WPSU), WVIA Public Media Studios (WVIA), WTVI (WTVI), Western Reserve PBS (WNEO), WVIZ/PBS ideastream (WVIZ), KCTS 9 (KCTS), Basin PBS (KPBT), KUHT / Channel 8 (KUHT), KLRN (KLRN), KLRU (KLRU), WTJX Channel 12 (WTJX), WCVE PBS (WCVE), KBTC Public Television (KBTC)
- - **pcmag**
 - **PearVideo**
 - **PeerTube**
 - **People**
@@ -854,6 +858,9 @@
 - **ShahidShow**
 - **Shared**: shared.sx
 - **ShowRoomLive**
+ - **simplecast**
+ - **simplecast:episode**
+ - **simplecast:podcast**
 - **Sina**
 - **sky.it**
 - **sky:news**
@@ -1231,6 +1238,7 @@
 - **ZattooLive**
 - **ZDF-3sat**
 - **ZDFChannel**
+ - **Zhihu**
 - **zingmp3**: mp3.zing.vn
 - **zoom**
 - **Zype**
--- a/pyinst.py
+++ b/pyinst.py
@@ -73,6 +73,7 @@ PyInstaller.__main__.run([
    '--exclude-module=ytdlp_plugins',
    '--hidden-import=mutagen',
    '--hidden-import=Crypto',
+    '--upx-exclude=vcruntime140.dll',
    'youtube_dlc/__main__.py',
 ])
 SetVersion('dist/youtube-dlc%s.exe' % _x86, VERSION_FILE)
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,2 +1,2 @@
 mutagen
-Crypto
+pycryptodome
--- a/setup.py
+++ b/setup.py
@@ -20,7 +20,7 @@ LONG_DESCRIPTION = '\n\n'.join((
    '**PS**: Many links in this document will not work since this is a copy of the README.md from Github',
    open("README.md", "r", encoding="utf-8").read()))

-REQUIREMENTS = ['mutagen', 'Crypto']
+REQUIREMENTS = ['mutagen', 'pycryptodome']


 if len(sys.argv) >= 2 and sys.argv[1] == 'py2exe':
--- a/test/test_youtube_lists.py
+++ b/test/test_youtube_lists.py
@@ -12,6 +12,7 @@ from test.helper import FakeYDL

 from youtube_dlc.extractor import (
    YoutubePlaylistIE,
+    YoutubeTabIE,
    YoutubeIE,
 )

@@ -57,14 +58,22 @@ class TestYoutubeLists(unittest.TestCase):
        entries = result['entries']
        self.assertEqual(len(entries), 100)

-    def test_youtube_flat_playlist_titles(self):
+    def test_youtube_flat_playlist_extraction(self):
        dl = FakeYDL()
        dl.params['extract_flat'] = True
-        ie = YoutubePlaylistIE(dl)
-        result = ie.extract('https://www.youtube.com/playlist?list=PL-KKIb8rvtMSrAO9YFbeM6UQrAqoFTUWv')
+        ie = YoutubeTabIE(dl)
+        result = ie.extract('https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc')
        self.assertIsPlaylist(result)
-        for entry in result['entries']:
-            self.assertTrue(entry.get('title'))
+        entries = list(result['entries'])
+        self.assertTrue(len(entries) == 1)
+        video = entries[0]
+        self.assertEqual(video['_type'], 'url_transparent')
+        self.assertEqual(video['ie_key'], 'Youtube')
+        self.assertEqual(video['id'], 'BaW_jenozKc')
+        self.assertEqual(video['url'], 'BaW_jenozKc')
+        self.assertEqual(video['title'], 'youtube-dl test video "\'/\\ä↭𝕐')
+        self.assertEqual(video['duration'], 10)
+        self.assertEqual(video['uploader'], 'Philipp Hagemeister')


 if __name__ == '__main__':
--- a/test/test_youtube_signature.py
+++ b/test/test_youtube_signature.py
@@ -19,55 +19,46 @@ from youtube_dlc.compat import compat_str, compat_urlretrieve
 _TESTS = [
    (
        'https://s.ytimg.com/yts/jsbin/html5player-vflHOr_nV.js',
-        'js',
        86,
        '>=<;:/.-[+*)(\'&%$#"!ZYX0VUTSRQPONMLKJIHGFEDCBA\\yxwvutsrqponmlkjihgfedcba987654321',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-vfldJ8xgI.js',
-        'js',
        85,
        '3456789a0cdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRS[UVWXYZ!"#$%&\'()*+,-./:;<=>?@',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-vfle-mVwz.js',
-        'js',
        90,
        ']\\[@?>=<;:/.-,+*)(\'&%$#"hZYXWVUTSRQPONMLKJIHGFEDCBAzyxwvutsrqponmlkjiagfedcb39876',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vfl0Cbn9e.js',
-        'js',
        84,
        'O1I3456789abcde0ghijklmnopqrstuvwxyzABCDEFGHfJKLMN2PQRSTUVW@YZ!"#$%&\'()*+,-./:;<=',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflXGBaUN.js',
-        'js',
        '2ACFC7A61CA478CD21425E5A57EBD73DDC78E22A.2094302436B2D377D14A3BBA23022D023B8BC25AA',
        'A52CB8B320D22032ABB3A41D773D2B6342034902.A22E87CDD37DBE75A5E52412DC874AC16A7CFCA2',
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflBb0OQx.js',
-        'js',
        84,
        '123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQ0STUVWXYZ!"#$%&\'()*+,@./:;<=>'
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vfl9FYC6l.js',
-        'js',
        83,
        '123456789abcdefghijklmnopqr0tuvwxyzABCDETGHIJKLMNOPQRS>UVWXYZ!"#$%&\'()*+,-./:;<=F'
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflCGk6yw/html5player.js',
-        'js',
        '4646B5181C6C3020DF1D9C7FCFEA.AD80ABF70C39BD369CCCAE780AFBB98FA6B6CB42766249D9488C288',
        '82C8849D94266724DC6B6AF89BBFA087EACCD963.B93C07FBA084ACAEFCF7C9D1FD0203C6C1815B6B'
    ),
    (
        'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflKjOTVq/html5player.js',
-        'js',
        '312AA52209E3623129A412D56A40F11CB0AF14AE.3EE09501CB14E3BCDC3B2AE808BF3F1D14E7FBF12',
        '112AA5220913623229A412D56A40F11CB0AF14AE.3EE0950FCB14EEBCDC3B2AE808BF331D14E7FBF3',
    )
@@ -78,6 +69,10 @@ class TestPlayerInfo(unittest.TestCase):
    def test_youtube_extract_player_info(self):
        PLAYER_URLS = (
            ('https://www.youtube.com/s/player/64dddad9/player_ias.vflset/en_US/base.js', '64dddad9'),
+            ('https://www.youtube.com/s/player/64dddad9/player_ias.vflset/fr_FR/base.js', '64dddad9'),
+            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-phone-en_US.vflset/base.js', '64dddad9'),
+            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-phone-de_DE.vflset/base.js', '64dddad9'),
+            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-tablet-en_US.vflset/base.js', '64dddad9'),
            # obsolete
            ('https://www.youtube.com/yts/jsbin/player_ias-vfle4-e03/en_US/base.js', 'vfle4-e03'),
            ('https://www.youtube.com/yts/jsbin/player_ias-vfl49f_g4/en_US/base.js', 'vfl49f_g4'),
@@ -100,13 +95,13 @@ class TestSignature(unittest.TestCase):
            os.mkdir(self.TESTDATA_DIR)


-def make_tfunc(url, stype, sig_input, expected_sig):
+def make_tfunc(url, sig_input, expected_sig):
    m = re.match(r'.*-([a-zA-Z0-9_-]+)(?:/watch_as3|/html5player)?\.[a-z]+$', url)
    assert m, '%r should follow URL format' % url
    test_id = m.group(1)

    def test_func(self):
-        basename = 'player-%s.%s' % (test_id, stype)
+        basename = 'player-%s.js' % test_id
        fn = os.path.join(self.TESTDATA_DIR, basename)

        if not os.path.exists(fn):
@@ -114,22 +109,16 @@ def make_tfunc(url, stype, sig_input, expected_sig):

        ydl = FakeYDL()
        ie = YoutubeIE(ydl)
-        if stype == 'js':
-            with io.open(fn, encoding='utf-8') as testf:
-                jscode = testf.read()
-            func = ie._parse_sig_js(jscode)
-        else:
-            assert stype == 'swf'
-            with open(fn, 'rb') as testf:
-                swfcode = testf.read()
-            func = ie._parse_sig_swf(swfcode)
+        with io.open(fn, encoding='utf-8') as testf:
+            jscode = testf.read()
+        func = ie._parse_sig_js(jscode)
        src_sig = (
            compat_str(string.printable[:sig_input])
            if isinstance(sig_input, int) else sig_input)
        got_sig = func(src_sig)
        self.assertEqual(got_sig, expected_sig)

-    test_func.__name__ = str('test_signature_' + stype + '_' + test_id)
+    test_func.__name__ = str('test_signature_js_' + test_id)
    setattr(TestSignature, test_func.__name__, test_func)


--- a/youtube_dlc/YoutubeDL.py
+++ b/youtube_dlc/YoutubeDL.py
@@ -27,6 +27,7 @@ import traceback
 import random

 from string import ascii_letters
+from zipimport import zipimporter

 from .compat import (
    compat_basestring,
@@ -179,6 +180,7 @@ class YoutubeDL(object):
                       of 'skip_download' or 'simulate'.
    simulate:          Do not download the video files.
    format:            Video format code. see "FORMAT SELECTION" for more details.
+    allow_unplayable_formats:   Allow unplayable formats to be extracted and downloaded.
    format_sort:       How to sort the video formats. see "Sorting Formats"
                       for more details.
    format_sort_force: Force the given format_sort. see "Sorting Formats"
@@ -187,11 +189,15 @@ class YoutubeDL(object):
                       into a single file
    allow_multiple_audio_streams:   Allow multiple audio streams to be merged
                       into a single file
+    paths:             Dictionary of output paths. The allowed keys are 'home'
+                       'temp' and the keys of OUTTMPL_TYPES (in utils.py)
    outtmpl:           Dictionary of templates for output names. Allowed keys
-                       are 'default' and the keys of OUTTMPL_TYPES (in utils.py)
+                       are 'default' and the keys of OUTTMPL_TYPES (in utils.py).
+                       A string a also accepted for backward compatibility
    outtmpl_na_placeholder: Placeholder for unavailable meta fields.
    restrictfilenames: Do not allow "&" and spaces in file names
    trim_file_name:    Limit length of filename (extension excluded)
+    windowsfilenames:  Force the filenames to be windows compatible
    ignoreerrors:      Do not stop on download errors
                       (Default True when running youtube-dlc,
                       but False when directly accessing YoutubeDL class)
@@ -866,13 +872,6 @@ class YoutubeDL(object):
                    sub_ext = fn_groups[-2]
                filename = '.'.join(filter(None, [fn_groups[0][:trim_file_name], sub_ext, ext]))

-            # Temporary fix for #4787
-            # 'Treat' all problem characters by passing filename through preferredencoding
-            # to workaround encoding issues with subprocess on python2 @ Windows
-            if sys.version_info < (3, 0) and sys.platform == 'win32':
-                filename = encodeFilename(filename, True).decode(preferredencoding())
-            filename = sanitize_path(filename)
-
            return filename
        except ValueError as err:
            self.report_error('Error in output template: ' + str(err) + ' (encoding: ' + repr(preferredencoding()) + ')')
@@ -899,7 +898,14 @@ class YoutubeDL(object):
        assert isinstance(homepath, compat_str)
        subdir = expand_path(paths.get(dir_type, '').strip()) if dir_type else ''
        assert isinstance(subdir, compat_str)
-        return sanitize_path(os.path.join(homepath, subdir, filename))
+        path = os.path.join(homepath, subdir, filename)
+
+        # Temporary fix for #4787
+        # 'Treat' all problem characters by passing filename through preferredencoding
+        # to workaround encoding issues with subprocess on python2 @ Windows
+        if sys.version_info < (3, 0) and sys.platform == 'win32':
+            path = encodeFilename(path, True).decode(preferredencoding())
+        return sanitize_path(path, force=self.params.get('windowsfilenames'))

    def _match_entry(self, info_dict, incomplete):
        """ Returns None if the file should be downloaded """
@@ -2291,10 +2297,15 @@ class YoutubeDL(object):
                if info_dict.get('requested_formats') is not None:
                    downloaded = []
                    merger = FFmpegMergerPP(self)
-                    if not merger.available:
-                        self.report_warning('You have requested multiple '
-                                            'formats but ffmpeg is not installed.'
-                                            ' The formats won\'t be merged.')
+                    if self.params.get('allow_unplayable_formats'):
+                        self.report_warning(
+                            'You have requested merging of multiple formats '
+                            'while also allowing unplayable formats to be downloaded. '
+                            'The formats won\'t be merged to prevent data corruption.')
+                    elif not merger.available:
+                        self.report_warning(
+                            'You have requested merging of multiple formats but ffmpeg is not installed. '
+                            'The formats won\'t be merged.')

                    def compatible_formats(formats):
                        # TODO: some formats actually allow this (mkv, webm, ogg, mp4), but not all of them.
@@ -2334,6 +2345,7 @@ class YoutubeDL(object):
                    full_filename = correct_ext(full_filename)
                    temp_filename = correct_ext(temp_filename)
                    dl_filename = existing_file(full_filename, temp_filename)
+                    info_dict['__real_download'] = False
                    if dl_filename is None:
                        for f in requested_formats:
                            new_info = dict(info_dict)
@@ -2345,12 +2357,16 @@ class YoutubeDL(object):
                                return
                            downloaded.append(fname)
                            partial_success, real_download = dl(fname, new_info)
+                            info_dict['__real_download'] = info_dict['__real_download'] or real_download
                            success = success and partial_success
-                        if merger.available:
+                        if merger.available and not self.params.get('allow_unplayable_formats'):
                            info_dict['__postprocessors'].append(merger)
-                        info_dict['__files_to_merge'] = downloaded
-                        # Even if there were no downloads, it is being merged only now
-                        info_dict['__real_download'] = True
+                            info_dict['__files_to_merge'] = downloaded
+                            # Even if there were no downloads, it is being merged only now
+                            info_dict['__real_download'] = True
+                        else:
+                            for file in downloaded:
+                                files_to_move[file] = None
                else:
                    # Just a single file
                    dl_filename = existing_file(full_filename, temp_filename)
@@ -2681,7 +2697,7 @@ class YoutubeDL(object):
                    '|',
                    format_field(f, 'filesize', ' %s', func=format_bytes) + format_field(f, 'filesize_approx', '~%s', func=format_bytes),
                    format_field(f, 'tbr', '%4dk'),
-                    f.get('protocol').replace('http_dash_segments', 'dash').replace("native", "n"),
+                    f.get('protocol').replace('http_dash_segments', 'dash').replace("native", "n").replace('niconico_', ''),
                    '|',
                    format_field(f, 'vcodec', default='unknown').replace('none', ''),
                    format_field(f, 'vbr', '%4dk'),
@@ -2704,8 +2720,6 @@ class YoutubeDL(object):
                if f.get('preference') is None or f['preference'] >= -1000]
            header_line = ['format code', 'extension', 'resolution', 'note']

-        # if len(formats) > 1:
-        #     table[-1][-1] += (' ' if table[-1][-1] else '') + '(best)'
        self.to_screen(
            '[info] Available formats for %s:\n%s' % (info_dict['id'], render_table(
                header_line,
@@ -2762,7 +2776,12 @@ class YoutubeDL(object):
                self.get_encoding()))
        write_string(encoding_str, encoding=None)

-        self._write_string('[debug] yt-dlp version %s\n' % __version__)
+        source = (
+            '(exe)' if hasattr(sys, 'frozen')
+            else '(zip)' if isinstance(globals().get('__loader__'), zipimporter)
+            else '(source)' if os.path.basename(sys.argv[0]) == '__main__.py'
+            else '')
+        self._write_string('[debug] yt-dlp version %s %s\n' % (__version__, source))
        if _LAZY_LOADER:
            self._write_string('[debug] Lazy loading extractors enabled\n')
        if _PLUGIN_CLASSES:
@@ -2789,8 +2808,10 @@ class YoutubeDL(object):
                return impl_name + ' version %d.%d.%d' % sys.pypy_version_info[:3]
            return impl_name

-        self._write_string('[debug] Python version %s (%s) - %s\n' % (
-            platform.python_version(), python_implementation(),
+        self._write_string('[debug] Python version %s (%s %s) - %s\n' % (
+            platform.python_version(),
+            python_implementation(),
+            platform.architecture()[0],
            platform_name()))

        exe_versions = FFmpegPostProcessor.get_versions(self)
--- a/youtube_dlc/init.py
+++ b/youtube_dlc/init.py
@@ -212,9 +212,6 @@ def _real_main(argv=None):
    if opts.recodevideo is not None:
        if opts.recodevideo not in REMUX_EXTENSIONS:
            parser.error('invalid video recode format specified')
-    if opts.remuxvideo and opts.recodevideo:
-        opts.remuxvideo = None
-        write_string('WARNING: --remux-video is ignored since --recode-video was given\n', out=sys.stderr)
    if opts.remuxvideo is not None:
        opts.remuxvideo = opts.remuxvideo.replace(' ', '')
        remux_regex = r'{0}(?:/{0})*$'.format(r'(?:\w+>)?(?:%s)' % '|'.join(REMUX_EXTENSIONS))
@@ -265,6 +262,40 @@ def _real_main(argv=None):
    any_printing = opts.print_json
    download_archive_fn = expand_path(opts.download_archive) if opts.download_archive is not None else opts.download_archive

+    def report_conflict(arg1, arg2):
+        write_string('WARNING: %s is ignored since %s was given\n' % (arg2, arg1), out=sys.stderr)
+    if opts.remuxvideo and opts.recodevideo:
+        report_conflict('--recode-video', '--remux-video')
+        opts.remuxvideo = False
+    if opts.allow_unplayable_formats:
+        if opts.extractaudio:
+            report_conflict('--allow-unplayable-formats', '--extract-audio')
+            opts.extractaudio = False
+        if opts.remuxvideo:
+            report_conflict('--allow-unplayable-formats', '--remux-video')
+            opts.remuxvideo = False
+        if opts.recodevideo:
+            report_conflict('--allow-unplayable-formats', '--recode-video')
+            opts.recodevideo = False
+        if opts.addmetadata:
+            report_conflict('--allow-unplayable-formats', '--add-metadata')
+            opts.addmetadata = False
+        if opts.embedsubtitles:
+            report_conflict('--allow-unplayable-formats', '--embed-subs')
+            opts.embedsubtitles = False
+        if opts.embedthumbnail:
+            report_conflict('--allow-unplayable-formats', '--embed-thumbnail')
+            opts.embedthumbnail = False
+        if opts.xattrs:
+            report_conflict('--allow-unplayable-formats', '--xattrs')
+            opts.xattrs = False
+        if opts.fixup and opts.fixup.lower() not in ('never', 'ignore'):
+            report_conflict('--allow-unplayable-formats', '--fixup')
+        opts.fixup = 'never'
+        if opts.sponskrub:
+            report_conflict('--allow-unplayable-formats', '--sponskrub')
+        opts.sponskrub = False
+
    # PostProcessors
    postprocessors = []
    if opts.metafromfield:
@@ -348,12 +379,15 @@ def _real_main(argv=None):
            'when': 'aftermove'
        })

-    _args_compat_warning = 'WARNING: %s given without specifying name. The arguments will be given to all %s\n'
+    def report_args_compat(arg, name):
+        write_string(
+            'WARNING: %s given without specifying name. The arguments will be given to all %s\n' % (arg, name),
+            out=sys.stderr)
    if 'default' in opts.external_downloader_args:
-        write_string(_args_compat_warning % ('--external-downloader-args', 'external downloaders'), out=sys.stderr),
+        report_args_compat('--external-downloader-args', 'external downloaders')

    if 'default-compat' in opts.postprocessor_args and 'default' not in opts.postprocessor_args:
-        write_string(_args_compat_warning % ('--post-processor-args', 'post-processors'), out=sys.stderr),
+        report_args_compat('--post-processor-args', 'post-processors')
        opts.postprocessor_args.setdefault('sponskrub', [])
        opts.postprocessor_args['default'] = opts.postprocessor_args['default-compat']

@@ -393,6 +427,7 @@ def _real_main(argv=None):
        'simulate': opts.simulate or any_getting,
        'skip_download': opts.skip_download,
        'format': opts.format,
+        'allow_unplayable_formats': opts.allow_unplayable_formats,
        'format_sort': opts.format_sort,
        'format_sort_force': opts.format_sort_force,
        'allow_multiple_video_streams': opts.allow_multiple_video_streams,
@@ -405,6 +440,7 @@ def _real_main(argv=None):
        'autonumber_size': opts.autonumber_size,
        'autonumber_start': opts.autonumber_start,
        'restrictfilenames': opts.restrictfilenames,
+        'windowsfilenames': opts.windowsfilenames,
        'ignoreerrors': opts.ignoreerrors,
        'force_generic_extractor': opts.force_generic_extractor,
        'ratelimit': opts.ratelimit,
@@ -431,6 +467,7 @@ def _real_main(argv=None):
        'writedescription': opts.writedescription,
        'writeannotations': opts.writeannotations,
        'writeinfojson': opts.writeinfojson or opts.getcomments,
+        'allow_playlist_files': opts.allow_playlist_files,
        'getcomments': opts.getcomments,
        'writethumbnail': opts.writethumbnail,
        'write_all_thumbnails': opts.write_all_thumbnails,
@@ -513,16 +550,22 @@ def _real_main(argv=None):
    }

    with YoutubeDL(ydl_opts) as ydl:
-        # Update version
-        if opts.update_self:
-            update_self(ydl.to_screen, opts.verbose, ydl._opener)
+        actual_use = len(all_urls) or opts.load_info_filename

        # Remove cache dir
        if opts.rm_cachedir:
            ydl.cache.remove()

+        # Update version
+        if opts.update_self:
+            # If updater returns True, exit. Required for windows
+            if update_self(ydl.to_screen, opts.verbose, ydl._opener):
+                if actual_use:
+                    sys.exit('ERROR: The program must exit for the update to complete')
+                sys.exit()
+
        # Maybe do nothing
-        if (len(all_urls) < 1) and (opts.load_info_filename is None):
+        if not actual_use:
            if opts.update_self or opts.rm_cachedir:
                sys.exit()

--- a/youtube_dlc/downloader/init.py
+++ b/youtube_dlc/downloader/init.py
@@ -21,6 +21,7 @@ from .http import HttpFD
 from .rtmp import RtmpFD
 from .rtsp import RtspFD
 from .ism import IsmFD
+from .niconico import NiconicoDmcFD
 from .youtube_live_chat import YoutubeLiveChatReplayFD
 from .external import (
    get_external_downloader,
@@ -36,6 +37,7 @@ PROTOCOL_MAP = {
    'f4m': F4mFD,
    'http_dash_segments': DashSegmentsFD,
    'ism': IsmFD,
+    'niconico_dmc': NiconicoDmcFD,
    'youtube_live_chat_replay': YoutubeLiveChatReplayFD,
 }

--- a/youtube_dlc/downloader/external.py
+++ b/youtube_dlc/downloader/external.py
@@ -5,7 +5,6 @@ import re
 import subprocess
 import sys
 import time
-import shutil

 try:
    from Crypto.Cipher import AES
@@ -32,6 +31,7 @@ from ..utils import (
    is_outdated_version,
    process_communicate_or_kill,
    sanitized_Request,
+    sanitize_open,
 )


@@ -126,23 +126,29 @@ class ExternalFD(FileDownloader):
            for [i, url] in enumerate(info_dict['url_list']):
                tmpsegmentname = '%s_%s.frag' % (tmpfilename, i)
                file_list.append(tmpsegmentname)
-            with open(tmpfilename, 'wb') as dest:
-                for i in file_list:
-                    if 'decrypt_info' in info_dict:
-                        decrypt_info = info_dict['decrypt_info']
-                        with open(i, 'rb') as src:
-                            if decrypt_info['METHOD'] == 'AES-128':
-                                iv = decrypt_info.get('IV')
-                                decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
-                                    self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
-                                encrypted_data = src.read()
-                                decrypted_data = AES.new(
-                                    decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(encrypted_data)
-                                dest.write(decrypted_data)
-                            else:
-                                shutil.copyfileobj(open(i, 'rb'), dest)
+            key_list = info_dict.get('key_list')
+            decrypt_info = None
+            dest, _ = sanitize_open(tmpfilename, 'wb')
+            for i, file in enumerate(file_list):
+                src, _ = sanitize_open(file, 'rb')
+                if key_list:
+                    decrypt_info = next((x for x in key_list if x['INDEX'] == i), decrypt_info)
+                    if decrypt_info['METHOD'] == 'AES-128':
+                        iv = decrypt_info.get('IV')
+                        decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
+                            self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
+                        encrypted_data = src.read()
+                        decrypted_data = AES.new(
+                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(encrypted_data)
+                        dest.write(decrypted_data)
                    else:
-                        shutil.copyfileobj(open(i, 'rb'), dest)
+                        fragment_data = src.read()
+                        dest.write(fragment_data)
+                else:
+                    fragment_data = src.read()
+                    dest.write(fragment_data)
+                src.close()
+            dest.close()
            if not self.params.get('keep_fragments', False):
                for file_path in file_list:
                    try:
@@ -263,8 +269,9 @@ class Aria2cFD(ExternalFD):
            for [i, url] in enumerate(info_dict['url_list']):
                tmpsegmentname = '%s_%s.frag' % (os.path.basename(tmpfilename), i)
                url_list.append('%s\n\tout=%s' % (url, tmpsegmentname))
-            with open(url_list_file, 'w') as f:
-                f.write('\n'.join(url_list))
+            stream, _ = sanitize_open(url_list_file, 'wb')
+            stream.write('\n'.join(url_list).encode('utf-8'))
+            stream.close()

            cmd += ['-i', url_list_file]
        else:
--- a/youtube_dlc/downloader/f4m.py
+++ b/youtube_dlc/downloader/f4m.py
@@ -267,13 +267,14 @@ class F4mFD(FragmentFD):
        media = doc.findall(_add_ns('media'))
        if not media:
            self.report_error('No media found')
-        for e in (doc.findall(_add_ns('drmAdditionalHeader'))
-                  + doc.findall(_add_ns('drmAdditionalHeaderSet'))):
-            # If id attribute is missing it's valid for all media nodes
-            # without drmAdditionalHeaderId or drmAdditionalHeaderSetId attribute
-            if 'id' not in e.attrib:
-                self.report_error('Missing ID in f4m DRM')
-        media = remove_encrypted_media(media)
+        if not self.params.get('allow_unplayable_formats'):
+            for e in (doc.findall(_add_ns('drmAdditionalHeader'))
+                      + doc.findall(_add_ns('drmAdditionalHeaderSet'))):
+                # If id attribute is missing it's valid for all media nodes
+                # without drmAdditionalHeaderId or drmAdditionalHeaderSetId attribute
+                if 'id' not in e.attrib:
+                    self.report_error('Missing ID in f4m DRM')
+            media = remove_encrypted_media(media)
        if not media:
            self.report_error('Unsupported DRM')
        return media
--- a/youtube_dlc/downloader/fragment.py
+++ b/youtube_dlc/downloader/fragment.py
@@ -95,11 +95,12 @@ class FragmentFD(FileDownloader):
        frag_index_stream.write(json.dumps({'downloader': downloader}))
        frag_index_stream.close()

-    def _download_fragment(self, ctx, frag_url, info_dict, headers=None):
+    def _download_fragment(self, ctx, frag_url, info_dict, headers=None, request_data=None):
        fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], ctx['fragment_index'])
        fragment_info_dict = {
            'url': frag_url,
            'http_headers': headers or info_dict.get('http_headers'),
+            'request_data': request_data,
        }
        success = ctx['dl'].download(fragment_filename, fragment_info_dict)
        if not success:
--- a/youtube_dlc/downloader/hls.py
+++ b/youtube_dlc/downloader/hls.py
@@ -29,9 +29,8 @@ class HlsFD(FragmentFD):
    FD_NAME = 'hlsnative'

    @staticmethod
-    def can_download(manifest, info_dict):
-        UNSUPPORTED_FEATURES = (
-            r'#EXT-X-KEY:METHOD=(?!NONE|AES-128)',  # encrypted streams [1]
+    def can_download(manifest, info_dict, allow_unplayable_formats=False):
+        UNSUPPORTED_FEATURES = [
            # r'#EXT-X-BYTERANGE',  # playlists composed of byte ranges of media files [2]

            # Live streams heuristic does not always work (e.g. geo restricted to Germany
@@ -50,7 +49,11 @@ class HlsFD(FragmentFD):
            # 3. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.2
            # 4. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.5
            # 5. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.5
-        )
+        ]
+        if not allow_unplayable_formats:
+            UNSUPPORTED_FEATURES += [
+                r'#EXT-X-KEY:METHOD=(?!NONE|AES-128)',  # encrypted streams [1]
+            ]
        check_results = [not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES]
        is_aes128_enc = '#EXT-X-KEY:METHOD=AES-128' in manifest
        check_results.append(can_decrypt_frag or not is_aes128_enc)
@@ -66,7 +69,7 @@ class HlsFD(FragmentFD):
        man_url = urlh.geturl()
        s = urlh.read().decode('utf-8', 'ignore')

-        if not self.can_download(s, info_dict):
+        if not self.can_download(s, info_dict, self.params.get('allow_unplayable_formats')):
            if info_dict.get('extra_param_to_segment_url') or info_dict.get('_decryption_key_url'):
                self.report_error('pycrypto not found. Please install it.')
                return False
@@ -131,6 +134,7 @@ class HlsFD(FragmentFD):
        i = 0
        media_sequence = 0
        decrypt_info = {'METHOD': 'NONE'}
+        key_list = []
        byte_range = {}
        frag_index = 0
        ad_frag_next = False
@@ -212,6 +216,10 @@ class HlsFD(FragmentFD):
                            decrypt_info['URI'] = update_url_query(decrypt_info['URI'], extra_query)
                        if decrypt_url != decrypt_info['URI']:
                            decrypt_info['KEY'] = None
+                    key_data = decrypt_info.copy()
+                    key_data['INDEX'] = frag_index
+                    key_list.append(key_data)
+
                elif line.startswith('#EXT-X-MEDIA-SEQUENCE'):
                    media_sequence = int(line[22:])
                elif line.startswith('#EXT-X-BYTERANGE'):
@@ -229,7 +237,7 @@ class HlsFD(FragmentFD):
        if real_downloader:
            info_copy = info_dict.copy()
            info_copy['url_list'] = fragment_urls
-            info_copy['decrypt_info'] = decrypt_info
+            info_copy['key_list'] = key_list
            fd = real_downloader(self.ydl, self.params)
            # TODO: Make progress updates work without hooking twice
            # for ph in self._progress_hooks:
--- a/youtube_dlc/downloader/http.py
+++ b/youtube_dlc/downloader/http.py
@@ -27,6 +27,7 @@ from ..utils import (
 class HttpFD(FileDownloader):
    def real_download(self, filename, info_dict):
        url = info_dict['url']
+        request_data = info_dict.get('request_data', None)

        class DownloadContext(dict):
            __getattr__ = dict.get
@@ -101,7 +102,7 @@ class HttpFD(FileDownloader):
                range_end = ctx.data_len - 1
            has_range = range_start is not None
            ctx.has_range = has_range
-            request = sanitized_Request(url, None, headers)
+            request = sanitized_Request(url, request_data, headers)
            if has_range:
                set_range(request, range_start, range_end)
            # Establish connection
@@ -152,7 +153,7 @@ class HttpFD(FileDownloader):
                    try:
                        # Open the connection again without the range header
                        ctx.data = self.ydl.urlopen(
-                            sanitized_Request(url, None, headers))
+                            sanitized_Request(url, request_data, headers))
                        content_length = ctx.data.info()['Content-Length']
                    except (compat_urllib_error.HTTPError, ) as err:
                        if err.code < 500 or err.code >= 600:
--- a/youtube_dlc/downloader/niconico.py
+++ b/youtube_dlc/downloader/niconico.py
@@ -0,0 +1,54 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import threading
+
+from .common import FileDownloader
+from ..downloader import _get_real_downloader
+from ..extractor.niconico import NiconicoIE
+from ..compat import compat_urllib_request
+
+
+class NiconicoDmcFD(FileDownloader):
+    """ Downloading niconico douga from DMC with heartbeat """
+
+    FD_NAME = 'niconico_dmc'
+
+    def real_download(self, filename, info_dict):
+        self.to_screen('[%s] Downloading from DMC' % self.FD_NAME)
+
+        ie = NiconicoIE(self.ydl)
+        info_dict, heartbeat_info_dict = ie._get_heartbeat_info(info_dict)
+
+        fd = _get_real_downloader(info_dict, params=self.params)(self.ydl, self.params)
+
+        success = download_complete = False
+        timer = [None]
+
+        heartbeat_lock = threading.Lock()
+        heartbeat_url = heartbeat_info_dict['url']
+        heartbeat_data = heartbeat_info_dict['data']
+        heartbeat_interval = heartbeat_info_dict.get('interval', 30)
+        self.to_screen('[%s] Heartbeat with %s second interval...' % (self.FD_NAME, heartbeat_interval))
+
+        def heartbeat():
+            try:
+                compat_urllib_request.urlopen(url=heartbeat_url, data=heartbeat_data.encode())
+            except Exception:
+                self.to_screen('[%s] Heartbeat failed' % self.FD_NAME)
+
+            with heartbeat_lock:
+                if not download_complete:
+                    timer[0] = threading.Timer(heartbeat_interval, heartbeat)
+                    timer[0].start()
+
+        try:
+            heartbeat()
+            success = fd.real_download(filename, info_dict)
+        finally:
+            if heartbeat_lock:
+                with heartbeat_lock:
+                    timer[0].cancel()
+                    download_complete = True
+
+            return success
--- a/youtube_dlc/downloader/youtube_live_chat.py
+++ b/youtube_dlc/downloader/youtube_live_chat.py
@@ -1,11 +1,13 @@
 from __future__ import division, unicode_literals

-import re
 import json

 from .fragment import FragmentFD
 from ..compat import compat_urllib_error
-from ..utils import try_get
+from ..utils import (
+    try_get,
+    RegexNotFoundError,
+)
 from ..extractor.youtube import YoutubeBaseInfoExtractor as YT_BaseIE


@@ -27,40 +29,28 @@ class YoutubeLiveChatReplayFD(FragmentFD):
            'total_frags': None,
        }

-        def dl_fragment(url):
-            headers = info_dict.get('http_headers', {})
-            return self._download_fragment(ctx, url, info_dict, headers)
+        ie = YT_BaseIE(self.ydl)

-        def parse_yt_initial_data(data):
-            patterns = (
-                r'%s\\s*%s' % (YT_BaseIE._YT_INITIAL_DATA_RE, YT_BaseIE._YT_INITIAL_BOUNDARY_RE),
-                r'%s' % YT_BaseIE._YT_INITIAL_DATA_RE)
-            data = data.decode('utf-8', 'replace')
-            for patt in patterns:
-                try:
-                    raw_json = re.search(patt, data).group(1)
-                    return json.loads(raw_json)
-                except AttributeError:
-                    continue
+        def dl_fragment(url, data=None, headers=None):
+            http_headers = info_dict.get('http_headers', {})
+            if headers:
+                http_headers = http_headers.copy()
+                http_headers.update(headers)
+            return self._download_fragment(ctx, url, info_dict, http_headers, data)

-        def download_and_parse_fragment(url, frag_index):
+        def download_and_parse_fragment(url, frag_index, request_data):
            count = 0
            while count <= fragment_retries:
                try:
-                    success, raw_fragment = dl_fragment(url)
+                    success, raw_fragment = dl_fragment(url, request_data, {'content-type': 'application/json'})
                    if not success:
                        return False, None, None
-                    data = parse_yt_initial_data(raw_fragment)
+                    try:
+                        data = ie._extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
+                    except RegexNotFoundError:
+                        data = None
                    if not data:
-                        raw_data = json.loads(raw_fragment)
-                        # sometimes youtube replies with a list
-                        if not isinstance(raw_data, list):
-                            raw_data = [raw_data]
-                        try:
-                            data = next(item['response'] for item in raw_data if 'response' in item)
-                        except StopIteration:
-                            data = {}
-
+                        data = json.loads(raw_fragment)
                    live_chat_continuation = try_get(
                        data,
                        lambda x: x['continuationContents']['liveChatContinuation'], dict) or {}
@@ -93,22 +83,37 @@ class YoutubeLiveChatReplayFD(FragmentFD):
            'https://www.youtube.com/watch?v={}'.format(video_id))
        if not success:
            return False
-        data = parse_yt_initial_data(raw_fragment)
+        try:
+            data = ie._extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
+        except RegexNotFoundError:
+            return False
        continuation_id = try_get(
            data,
            lambda x: x['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation'])
        # no data yet but required to call _append_fragment
        self._append_fragment(ctx, b'')

+        ytcfg = ie._extract_ytcfg(video_id, raw_fragment.decode('utf-8', 'replace'))
+
+        if not ytcfg:
+            return False
+        api_key = try_get(ytcfg, lambda x: x['INNERTUBE_API_KEY'])
+        innertube_context = try_get(ytcfg, lambda x: x['INNERTUBE_CONTEXT'])
+        if not api_key or not innertube_context:
+            return False
+        url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat_replay?key=' + api_key
+
        frag_index = offset = 0
        while continuation_id is not None:
            frag_index += 1
-            url = ''.join((
-                'https://www.youtube.com/live_chat_replay',
-                '/get_live_chat_replay' if frag_index > 1 else '',
-                '?continuation=%s' % continuation_id,
-                '&playerOffsetMs=%d&hidden=false&pbj=1' % max(offset - 5000, 0) if frag_index > 1 else ''))
-            success, continuation_id, offset = download_and_parse_fragment(url, frag_index)
+            request_data = {
+                'context': innertube_context,
+                'continuation': continuation_id,
+            }
+            if frag_index > 1:
+                request_data['currentPlayerState'] = {'playerOffsetMs': str(max(offset - 5000, 0))}
+            success, continuation_id, offset = download_and_parse_fragment(
+                url, frag_index, json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n')
            if not success:
                return False
            if test:
--- a/youtube_dlc/extractor/adobetv.py
+++ b/youtube_dlc/extractor/adobetv.py
@@ -66,7 +66,7 @@ class AdobeTVBaseIE(InfoExtractor):
                if original_filename.startswith('s3://') and not s3_extracted:
                    formats.append({
                        'format_id': 'original',
-                        'preference': 1,
+                        'quality': 1,
                        'url': original_filename.replace('s3://', 'https://s3.amazonaws.com/'),
                    })
                    s3_extracted = True
--- a/youtube_dlc/extractor/alura.py
+++ b/youtube_dlc/extractor/alura.py
@@ -67,7 +67,7 @@ class AluraIE(InfoExtractor):
                            f['height'] = int('720' if m.group('res') == 'hd' else '480')
                formats.extend(video_format)

-            self._sort_formats(formats, field_preference=('height', 'width', 'tbr', 'format_id'))
+            self._sort_formats(formats)

            return {
                'id': video_id,
--- a/youtube_dlc/extractor/aol.py
+++ b/youtube_dlc/extractor/aol.py
@@ -125,7 +125,7 @@ class AolIE(YahooIE):
                        'height': int_or_none(qs.get('h', [None])[0]),
                    })
                formats.append(f)
-        self._sort_formats(formats, ('width', 'height', 'tbr', 'format_id'))
+        self._sort_formats(formats)

        return {
            'id': video_id,
--- a/youtube_dlc/extractor/aparat.py
+++ b/youtube_dlc/extractor/aparat.py
@@ -72,8 +72,7 @@ class AparatIE(InfoExtractor):
                            r'(\d+)[pP]', label or '', 'height',
                            default=None)),
                    })
-        self._sort_formats(
-            formats, field_preference=('height', 'width', 'tbr', 'format_id'))
+        self._sort_formats(formats)

        info = self._search_json_ld(webpage, video_id, default={})

--- a/youtube_dlc/extractor/arcpublishing.py
+++ b/youtube_dlc/extractor/arcpublishing.py
@@ -129,10 +129,6 @@ class ArcPublishingIE(InfoExtractor):
                if all([f.get('acodec') == 'none' for f in m3u8_formats]):
                    continue
                for f in m3u8_formats:
-                    if f.get('acodec') == 'none':
-                        f['preference'] = -40
-                    elif f.get('vcodec') == 'none':
-                        f['preference'] = -50
                    height = f.get('height')
                    if not height:
                        continue
@@ -150,10 +146,9 @@ class ArcPublishingIE(InfoExtractor):
                    'height': int_or_none(s.get('height')),
                    'filesize': int_or_none(s.get('filesize')),
                    'url': s_url,
-                    'preference': -1,
+                    'quality': -10,
                })
-        self._sort_formats(
-            formats, ('preference', 'width', 'height', 'vbr', 'filesize', 'tbr', 'ext', 'format_id'))
+        self._sort_formats(formats)

        subtitles = {}
        for subtitle in (try_get(video, lambda x: x['subtitles']['urls'], list) or []):
--- a/youtube_dlc/extractor/ard.py
+++ b/youtube_dlc/extractor/ard.py
@@ -324,20 +324,42 @@ class ARDIE(InfoExtractor):

        formats = []
        for a in video_node.findall('.//asset'):
+            file_name = xpath_text(a, './fileName', default=None)
+            if not file_name:
+                continue
+            format_type = a.attrib.get('type')
+            format_url = url_or_none(file_name)
+            if format_url:
+                ext = determine_ext(file_name)
+                if ext == 'm3u8':
+                    formats.extend(self._extract_m3u8_formats(
+                        format_url, display_id, 'mp4', entry_protocol='m3u8_native',
+                        m3u8_id=format_type or 'hls', fatal=False))
+                    continue
+                elif ext == 'f4m':
+                    formats.extend(self._extract_f4m_formats(
+                        update_url_query(format_url, {'hdcore': '3.7.0'}),
+                        display_id, f4m_id=format_type or 'hds', fatal=False))
+                    continue
            f = {
-                'format_id': a.attrib['type'],
-                'width': int_or_none(a.find('./frameWidth').text),
-                'height': int_or_none(a.find('./frameHeight').text),
-                'vbr': int_or_none(a.find('./bitrateVideo').text),
-                'abr': int_or_none(a.find('./bitrateAudio').text),
-                'vcodec': a.find('./codecVideo').text,
-                'tbr': int_or_none(a.find('./totalBitrate').text),
+                'format_id': format_type,
+                'width': int_or_none(xpath_text(a, './frameWidth')),
+                'height': int_or_none(xpath_text(a, './frameHeight')),
+                'vbr': int_or_none(xpath_text(a, './bitrateVideo')),
+                'abr': int_or_none(xpath_text(a, './bitrateAudio')),
+                'vcodec': xpath_text(a, './codecVideo'),
+                'tbr': int_or_none(xpath_text(a, './totalBitrate')),
            }
-            if a.find('./serverPrefix').text:
-                f['url'] = a.find('./serverPrefix').text
-                f['playpath'] = a.find('./fileName').text
+            server_prefix = xpath_text(a, './serverPrefix', default=None)
+            if server_prefix:
+                f.update({
+                    'url': server_prefix,
+                    'playpath': file_name,
+                })
            else:
-                f['url'] = a.find('./fileName').text
+                if not format_url:
+                    continue
+                f['url'] = format_url
            formats.append(f)
        self._sort_formats(formats)

--- a/youtube_dlc/extractor/arte.py
+++ b/youtube_dlc/extractor/arte.py
@@ -150,7 +150,6 @@ class ArteTVIE(ArteTVBaseIE):

            format = {
                'format_id': format_id,
-                'preference': -10 if f.get('videoFormat') == 'M3U8' else None,
                'language_preference': lang_pref,
                'format_note': '%s, %s' % (f.get('versionCode'), f.get('versionLibelle')),
                'width': int_or_none(f.get('width')),
@@ -168,7 +167,9 @@ class ArteTVIE(ArteTVBaseIE):

            formats.append(format)

-        self._sort_formats(formats)
+        # For this extractor, quality only represents the relative quality
+        # with respect to other formats with the same resolution
+        self._sort_formats(formats, ('res', 'quality'))

        return {
            'id': player_info.get('VID') or video_id,
--- a/youtube_dlc/extractor/beatport.py
+++ b/youtube_dlc/extractor/beatport.py
@@ -69,12 +69,10 @@ class BeatportIE(InfoExtractor):
                'vcodec': 'none',
            }
            if ext == 'mp3':
-                fmt['preference'] = 0
                fmt['acodec'] = 'mp3'
                fmt['abr'] = 96
                fmt['asr'] = 44100
            elif ext == 'mp4':
-                fmt['preference'] = 1
                fmt['acodec'] = 'aac'
                fmt['abr'] = 96
                fmt['asr'] = 44100
--- a/youtube_dlc/extractor/bilibili.py
+++ b/youtube_dlc/extractor/bilibili.py
@@ -203,7 +203,7 @@ class BiliBiliIE(InfoExtractor):
                    formats.append({
                        'url': backup_url,
                        # backup URLs have lower priorities
-                        'preference': -2 if 'hd.mp4' in backup_url else -3,
+                        'quality': -2 if 'hd.mp4' in backup_url else -3,
                    })

                for a_format in formats:
--- a/youtube_dlc/extractor/bokecc.py
+++ b/youtube_dlc/extractor/bokecc.py
@@ -23,7 +23,7 @@ class BokeCCBaseIE(InfoExtractor):
        formats = [{
            'format_id': format_id,
            'url': quality.find('./copy').attrib['playurl'],
-            'preference': int(quality.attrib['value']),
+            'quality': int(quality.attrib['value']),
        } for quality in info_xml.findall('./video/quality')]

        self._sort_formats(formats)
--- a/youtube_dlc/extractor/bpb.py
+++ b/youtube_dlc/extractor/bpb.py
@@ -47,7 +47,7 @@ class BpbIE(InfoExtractor):
            quality = 'high' if '_high' in video_url else 'low'
            formats.append({
                'url': video_url,
-                'preference': 10 if quality == 'high' else 0,
+                'quality': 10 if quality == 'high' else 0,
                'format_note': quality,
                'format_id': '%s-%s' % (quality, determine_ext(video_url)),
            })
--- a/youtube_dlc/extractor/brightcove.py
+++ b/youtube_dlc/extractor/brightcove.py
@@ -478,11 +478,12 @@ class BrightcoveNewIE(AdobePassIE):
            container = source.get('container')
            ext = mimetype2ext(source.get('type'))
            src = source.get('src')
+            skip_unplayable = not self._downloader.params.get('allow_unplayable_formats')
            # https://support.brightcove.com/playback-api-video-fields-reference#key_systems_object
-            if container == 'WVM' or source.get('key_systems'):
+            if skip_unplayable and (container == 'WVM' or source.get('key_systems')):
                num_drm_sources += 1
                continue
-            elif ext == 'ism':
+            elif ext == 'ism' and skip_unplayable:
                continue
            elif ext == 'm3u8' or container == 'M2TS':
                if not src:
@@ -546,7 +547,8 @@ class BrightcoveNewIE(AdobePassIE):
                error = errors[0]
                raise ExtractorError(
                    error.get('message') or error.get('error_subcode') or error['error_code'], expected=True)
-            if sources and num_drm_sources == len(sources):
+            if (not self._downloader.params.get('allow_unplayable_formats')
+                    and sources and num_drm_sources == len(sources)):
                raise ExtractorError('This video is DRM protected.', expected=True)

        self._sort_formats(formats)
--- a/youtube_dlc/extractor/cammodels.py
+++ b/youtube_dlc/extractor/cammodels.py
@@ -82,7 +82,7 @@ class CamModelsIE(InfoExtractor):
                    f.update({
                        'ext': 'mp4',
                        # hls skips fragments, preferring rtmp
-                        'preference': -1,
+                        'quality': -10,
                    })
                else:
                    continue
--- a/youtube_dlc/extractor/canalplus.py
+++ b/youtube_dlc/extractor/canalplus.py
@@ -89,7 +89,7 @@ class CanalplusIE(InfoExtractor):
                    # the secret extracted from ya function in http://player.canalplus.fr/common/js/canalPlayer.js
                    'url': format_url + '?secret=pqzerjlsmdkjfoiuerhsdlfknaes',
                    'format_id': format_id,
-                    'preference': preference(format_id),
+                    'quality': preference(format_id),
                })
        self._sort_formats(formats)

--- a/youtube_dlc/extractor/canvas.py
+++ b/youtube_dlc/extractor/canvas.py
@@ -7,19 +7,21 @@ from .common import InfoExtractor
 from .gigya import GigyaBaseIE
 from ..compat import compat_HTTPError
 from ..utils import (
-    extract_attributes,
    ExtractorError,
-    strip_or_none,
+    clean_html,
+    extract_attributes,
    float_or_none,
+    get_element_by_class,
    int_or_none,
    merge_dicts,
    str_or_none,
+    strip_or_none,
    url_or_none,
 )


 class CanvasIE(InfoExtractor):
-    _VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrt(?:video|nieuws)|sporza)/assets/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://mediazone\.vrt\.be/api/v1/(?P<site_id>canvas|een|ketnet|vrt(?:video|nieuws)|sporza|dako)/assets/(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://mediazone.vrt.be/api/v1/ketnet/assets/md-ast-4ac54990-ce66-4d00-a8ca-9eac86f4c475',
        'md5': '68993eda72ef62386a15ea2cf3c93107',
@@ -332,3 +334,51 @@ class VrtNUIE(GigyaBaseIE):
            'display_id': display_id,
            'season_number': int_or_none(page.get('episode_season')),
        })
+
+
+class DagelijkseKostIE(InfoExtractor):
+    IE_DESC = 'dagelijksekost.een.be'
+    _VALID_URL = r'https?://dagelijksekost\.een\.be/gerechten/(?P<id>[^/?#&]+)'
+    _TEST = {
+        'url': 'https://dagelijksekost.een.be/gerechten/hachis-parmentier-met-witloof',
+        'md5': '30bfffc323009a3e5f689bef6efa2365',
+        'info_dict': {
+            'id': 'md-ast-27a4d1ff-7d7b-425e-b84f-a4d227f592fa',
+            'display_id': 'hachis-parmentier-met-witloof',
+            'ext': 'mp4',
+            'title': 'Hachis parmentier met witloof',
+            'description': 'md5:9960478392d87f63567b5b117688cdc5',
+            'thumbnail': r're:^https?://.*\.jpg$',
+            'duration': 283.02,
+        },
+        'expected_warnings': ['is not a supported codec'],
+    }
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        webpage = self._download_webpage(url, display_id)
+
+        title = strip_or_none(get_element_by_class(
+            'dish-metadata__title', webpage
+        ) or self._html_search_meta(
+            'twitter:title', webpage))
+
+        description = clean_html(get_element_by_class(
+            'dish-description', webpage)
+        ) or self._html_search_meta(
+            ('description', 'twitter:description', 'og:description'),
+            webpage)
+
+        video_id = self._html_search_regex(
+            r'data-url=(["\'])(?P<id>(?:(?!\1).)+)\1', webpage, 'video id',
+            group='id')
+
+        return {
+            '_type': 'url_transparent',
+            'url': 'https://mediazone.vrt.be/api/v1/dako/assets/%s' % video_id,
+            'ie_key': CanvasIE.ie_key(),
+            'id': video_id,
+            'display_id': display_id,
+            'title': title,
+            'description': description,
+        }
--- a/youtube_dlc/extractor/ccma.py
+++ b/youtube_dlc/extractor/ccma.py
@@ -1,12 +1,14 @@
 # coding: utf-8
 from __future__ import unicode_literals

+import calendar
 import datetime
 import re

 from .common import InfoExtractor
 from ..utils import (
    clean_html,
+    extract_timezone,
    int_or_none,
    parse_duration,
    parse_resolution,
@@ -97,8 +99,9 @@ class CCMAIE(InfoExtractor):
        timestamp = None
        data_utc = try_get(informacio, lambda x: x['data_emissio']['utc'])
        try:
-            timestamp = datetime.datetime.strptime(
-                data_utc, '%Y-%d-%mT%H:%M:%S%z').timestamp()
+            timezone, data_utc = extract_timezone(data_utc)
+            timestamp = calendar.timegm((datetime.datetime.strptime(
+                data_utc, '%Y-%d-%mT%H:%M:%S') - timezone).timetuple())
        except TypeError:
            pass

--- a/youtube_dlc/extractor/cctv.py
+++ b/youtube_dlc/extractor/cctv.py
@@ -162,7 +162,7 @@ class CCTVIE(InfoExtractor):
                        'url': video_url,
                        'format_id': 'http',
                        'quality': quality,
-                        'preference': -1,
+                        'source_preference': -10
                    })

        hls_url = try_get(data, lambda x: x['hls_url'], compat_str)
--- a/youtube_dlc/extractor/cda.py
+++ b/youtube_dlc/extractor/cda.py
@@ -95,6 +95,9 @@ class CDAIE(InfoExtractor):
        if 'Ten film jest dostępny dla użytkowników premium' in webpage:
            raise ExtractorError('This video is only available for premium users.', expected=True)

+        if re.search(r'niedostępn[ey] w(?:&nbsp;|\s+)Twoim kraju\s*<', webpage):
+            self.raise_geo_restricted()
+
        need_confirm_age = False
        if self._html_search_regex(r'(<form[^>]+action="[^"]*/a/validatebirth[^"]*")',
                                   webpage, 'birthday validate form', default=None):
--- a/youtube_dlc/extractor/ceskatelevize.py
+++ b/youtube_dlc/extractor/ceskatelevize.py
@@ -147,7 +147,8 @@ class CeskaTelevizeIE(InfoExtractor):
                is_live = item.get('type') == 'LIVE'
                formats = []
                for format_id, stream_url in item.get('streamUrls', {}).items():
-                    if 'drmOnly=true' in stream_url:
+                    if (not self._downloader.params.get('allow_unplayable_formats')
+                            and 'drmOnly=true' in stream_url):
                        continue
                    if 'playerType=flash' in stream_url:
                        stream_formats = self._extract_m3u8_formats(
--- a/youtube_dlc/extractor/common.py
+++ b/youtube_dlc/extractor/common.py
@@ -1367,12 +1367,12 @@ class InfoExtractor(object):
        regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<seperator>[~:])(?P<limit>.*?))?)? *$'

        default = ('hidden', 'hasvid', 'ie_pref', 'lang', 'quality',
-                   'res', 'fps', 'codec:vp9', 'size', 'br', 'asr',
+                   'res', 'fps', 'codec:vp9.2', 'size', 'br', 'asr',
                   'proto', 'ext', 'has_audio', 'source', 'format_id')  # These must not be aliases

        settings = {
            'vcodec': {'type': 'ordered', 'regex': True,
-                       'order': ['av0?1', 'vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
+                       'order': ['av0?1', 'vp0?9.2', 'vp0?9', '[hx]265|he?vc?', '[hx]264|avc', 'vp0?8', 'mp4v|h263', 'theora', '', None, 'none']},
            'acodec': {'type': 'ordered', 'regex': True,
                       'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
            'proto': {'type': 'ordered', 'regex': True, 'field': 'protocol',
@@ -1384,11 +1384,11 @@ class InfoExtractor(object):
                     'order': ('m4a', 'aac', 'mp3', 'ogg', 'opus', 'webm', '', 'none'),
                     'order_free': ('opus', 'ogg', 'webm', 'm4a', 'mp3', 'aac', '', 'none')},
            'hidden': {'visible': False, 'forced': True, 'type': 'extractor', 'max': -1000},
-            'ie_pref': {'priority': True, 'type': 'extractor', 'field': 'extractor_preference'},
+            'ie_pref': {'priority': True, 'type': 'extractor'},
            'hasvid': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
            'hasaud': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
-            'lang': {'priority': True, 'convert': 'ignore', 'field': 'language_preference'},
-            'quality': {'priority': True, 'convert': 'float_none'},
+            'lang': {'priority': True, 'convert': 'ignore', 'type': 'extractor', 'field': 'language_preference'},
+            'quality': {'convert': 'float_none', 'type': 'extractor'},
            'filesize': {'convert': 'bytes'},
            'fs_approx': {'convert': 'bytes', 'field': 'filesize_approx'},
            'id': {'convert': 'string', 'field': 'format_id'},
@@ -1399,7 +1399,7 @@ class InfoExtractor(object):
            'vbr': {'convert': 'float_none'},
            'abr': {'convert': 'float_none'},
            'asr': {'convert': 'float_none'},
-            'source': {'convert': 'ignore', 'field': 'source_preference'},
+            'source': {'convert': 'ignore', 'type': 'extractor', 'field': 'source_preference'},

            'codec': {'type': 'combined', 'field': ('vcodec', 'acodec')},
            'br': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
@@ -1469,13 +1469,12 @@ class InfoExtractor(object):
            elif conversion == 'bytes':
                return FileDownloader.parse_bytes(value)
            elif conversion == 'order':
-                order_free = self._get_field_setting(field, 'order_free')
-                order_list = order_free if order_free and self._use_free_order else self._get_field_setting(field, 'order')
+                order_list = (self._use_free_order and self._get_field_setting(field, 'order_free')) or self._get_field_setting(field, 'order')
                use_regex = self._get_field_setting(field, 'regex')
                list_length = len(order_list)
                empty_pos = order_list.index('') if '' in order_list else list_length + 1
                if use_regex and value is not None:
-                    for (i, regex) in enumerate(order_list):
+                    for i, regex in enumerate(order_list):
                        if regex and re.match(regex, value):
                            return list_length - i
                    return list_length - empty_pos  # not in list
@@ -1544,7 +1543,7 @@ class InfoExtractor(object):
        def print_verbose_info(self, to_screen):
            to_screen('[debug] Sort order given by user: %s' % ','.join(self._sort_user))
            if self._sort_extractor:
-                to_screen('[debug] Sort order given by extractor: %s' % ','.join(self._sort_extractor))
+                to_screen('[debug] Sort order given by extractor: %s' % ', '.join(self._sort_extractor))
            to_screen('[debug] Formats sorted by: %s' % ', '.join(['%s%s%s' % (
                '+' if self._get_field_setting(field, 'reverse') else '', field,
                '%s%s(%s)' % ('~' if self._get_field_setting(field, 'closest') else ':',
@@ -1561,7 +1560,7 @@ class InfoExtractor(object):
            if type == 'extractor':
                maximum = self._get_field_setting(field, 'max')
                if value is None or (maximum is not None and value >= maximum):
-                    value = 0
+                    value = -1
            elif type == 'boolean':
                in_list = self._get_field_setting(field, 'in_list')
                not_in_list = self._get_field_setting(field, 'not_in_list')
@@ -1694,7 +1693,7 @@ class InfoExtractor(object):
        self.to_screen(msg)
        time.sleep(timeout)

-    def _extract_f4m_formats(self, manifest_url, video_id, preference=None, f4m_id=None,
+    def _extract_f4m_formats(self, manifest_url, video_id, preference=None, quality=None, f4m_id=None,
                             transform_source=lambda s: fix_xml_ampersands(s).strip(),
                             fatal=True, m3u8_id=None, data=None, headers={}, query={}):
        manifest = self._download_xml(
@@ -1709,10 +1708,10 @@ class InfoExtractor(object):
            return []

        return self._parse_f4m_formats(
-            manifest, manifest_url, video_id, preference=preference, f4m_id=f4m_id,
+            manifest, manifest_url, video_id, preference=preference, quality=quality, f4m_id=f4m_id,
            transform_source=transform_source, fatal=fatal, m3u8_id=m3u8_id)

-    def _parse_f4m_formats(self, manifest, manifest_url, video_id, preference=None, f4m_id=None,
+    def _parse_f4m_formats(self, manifest, manifest_url, video_id, preference=None, quality=None, f4m_id=None,
                           transform_source=lambda s: fix_xml_ampersands(s).strip(),
                           fatal=True, m3u8_id=None):
        if not isinstance(manifest, compat_etree_Element) and not fatal:
@@ -1777,7 +1776,7 @@ class InfoExtractor(object):
                ext = determine_ext(manifest_url)
                if ext == 'f4m':
                    f4m_formats = self._extract_f4m_formats(
-                        manifest_url, video_id, preference=preference, f4m_id=f4m_id,
+                        manifest_url, video_id, preference=preference, quality=quality, f4m_id=f4m_id,
                        transform_source=transform_source, fatal=fatal)
                    # Sometimes stream-level manifest contains single media entry that
                    # does not contain any quality metadata (e.g. http://matchtv.ru/#live-player).
@@ -1797,7 +1796,7 @@ class InfoExtractor(object):
                elif ext == 'm3u8':
                    formats.extend(self._extract_m3u8_formats(
                        manifest_url, video_id, 'mp4', preference=preference,
-                        m3u8_id=m3u8_id, fatal=fatal))
+                        quality=quality, m3u8_id=m3u8_id, fatal=fatal))
                    continue
            formats.append({
                'format_id': format_id,
@@ -1810,22 +1809,24 @@ class InfoExtractor(object):
                'height': height,
                'vcodec': vcodec,
                'preference': preference,
+                'quality': quality,
            })
        return formats

-    def _m3u8_meta_format(self, m3u8_url, ext=None, preference=None, m3u8_id=None):
+    def _m3u8_meta_format(self, m3u8_url, ext=None, preference=None, quality=None, m3u8_id=None):
        return {
            'format_id': '-'.join(filter(None, [m3u8_id, 'meta'])),
            'url': m3u8_url,
            'ext': ext,
            'protocol': 'm3u8',
            'preference': preference - 100 if preference else -100,
+            'quality': quality,
            'resolution': 'multiple',
            'format_note': 'Quality selection URL',
        }

    def _extract_m3u8_formats(self, m3u8_url, video_id, ext=None,
-                              entry_protocol='m3u8', preference=None,
+                              entry_protocol='m3u8', preference=None, quality=None,
                              m3u8_id=None, note=None, errnote=None,
                              fatal=True, live=False, data=None, headers={},
                              query={}):
@@ -1843,10 +1844,10 @@ class InfoExtractor(object):

        return self._parse_m3u8_formats(
            m3u8_doc, m3u8_url, ext=ext, entry_protocol=entry_protocol,
-            preference=preference, m3u8_id=m3u8_id, live=live)
+            preference=preference, quality=quality, m3u8_id=m3u8_id, live=live)

    def _parse_m3u8_formats(self, m3u8_doc, m3u8_url, ext=None,
-                            entry_protocol='m3u8', preference=None,
+                            entry_protocol='m3u8', preference=None, quality=None,
                            m3u8_id=None, live=False):
        if '#EXT-X-FAXS-CM:' in m3u8_doc:  # Adobe Flash Access
            return []
@@ -1884,6 +1885,7 @@ class InfoExtractor(object):
                'ext': ext,
                'protocol': entry_protocol,
                'preference': preference,
+                'quality': quality,
            }]

        groups = {}
@@ -1912,6 +1914,7 @@ class InfoExtractor(object):
                    'ext': ext,
                    'protocol': entry_protocol,
                    'preference': preference,
+                    'quality': quality,
                }
                if media_type == 'AUDIO':
                    f['vcodec'] = 'none'
@@ -1971,6 +1974,7 @@ class InfoExtractor(object):
                    'fps': float_or_none(last_stream_inf.get('FRAME-RATE')),
                    'protocol': entry_protocol,
                    'preference': preference,
+                    'quality': quality,
                }
                resolution = last_stream_inf.get('RESOLUTION')
                if resolution:
@@ -2358,6 +2362,8 @@ class InfoExtractor(object):
                        extract_Initialization(segment_template)
            return ms_info

+        skip_unplayable = not self._downloader.params.get('allow_unplayable_formats')
+
        mpd_duration = parse_duration(mpd_doc.get('mediaPresentationDuration'))
        formats = []
        for period in mpd_doc.findall(_add_ns('Period')):
@@ -2367,11 +2373,11 @@ class InfoExtractor(object):
                'timescale': 1,
            })
            for adaptation_set in period.findall(_add_ns('AdaptationSet')):
-                if is_drm_protected(adaptation_set):
+                if skip_unplayable and is_drm_protected(adaptation_set):
                    continue
                adaption_set_ms_info = extract_multisegment_info(adaptation_set, period_ms_info)
                for representation in adaptation_set.findall(_add_ns('Representation')):
-                    if is_drm_protected(representation):
+                    if skip_unplayable and is_drm_protected(representation):
                        continue
                    representation_attrib = adaptation_set.attrib.copy()
                    representation_attrib.update(representation.attrib)
@@ -2585,7 +2591,10 @@ class InfoExtractor(object):
         1. [MS-SSTR]: Smooth Streaming Protocol,
            https://msdn.microsoft.com/en-us/library/ff469518.aspx
        """
-        if ism_doc.get('IsLive') == 'TRUE' or ism_doc.find('Protection') is not None:
+        if ism_doc.get('IsLive') == 'TRUE':
+            return []
+        if (not self._downloader.params.get('allow_unplayable_formats')
+                and ism_doc.find('Protection') is not None):
            return []

        duration = int(ism_doc.attrib['Duration'])
@@ -2673,7 +2682,7 @@ class InfoExtractor(object):
                })
        return formats

-    def _parse_html5_media_entries(self, base_url, webpage, video_id, m3u8_id=None, m3u8_entry_protocol='m3u8', mpd_id=None, preference=None):
+    def _parse_html5_media_entries(self, base_url, webpage, video_id, m3u8_id=None, m3u8_entry_protocol='m3u8', mpd_id=None, preference=None, quality=None):
        def absolute_url(item_url):
            return urljoin(base_url, item_url)

@@ -2696,7 +2705,7 @@ class InfoExtractor(object):
                formats = self._extract_m3u8_formats(
                    full_url, video_id, ext='mp4',
                    entry_protocol=m3u8_entry_protocol, m3u8_id=m3u8_id,
-                    preference=preference, fatal=False)
+                    preference=preference, quality=quality, fatal=False)
            elif ext == 'mpd':
                is_plain_url = False
                formats = self._extract_mpd_formats(
--- a/youtube_dlc/extractor/coub.py
+++ b/youtube_dlc/extractor/coub.py
@@ -87,7 +87,7 @@ class CoubIE(InfoExtractor):
                    'filesize': int_or_none(item.get('size')),
                    'vcodec': 'none' if kind == 'audio' else None,
                    'quality': quality_key(quality),
-                    'preference': preference_key(HTML5),
+                    'source_preference': preference_key(HTML5),
                })

        iphone_url = file_versions.get(IPHONE, {}).get('url')
@@ -95,7 +95,7 @@ class CoubIE(InfoExtractor):
            formats.append({
                'url': iphone_url,
                'format_id': IPHONE,
-                'preference': preference_key(IPHONE),
+                'source_preference': preference_key(IPHONE),
            })

        mobile_url = file_versions.get(MOBILE, {}).get('audio_url')
@@ -103,7 +103,7 @@ class CoubIE(InfoExtractor):
            formats.append({
                'url': mobile_url,
                'format_id': '%s-audio' % MOBILE,
-                'preference': preference_key(MOBILE),
+                'source_preference': preference_key(MOBILE),
            })

        self._sort_formats(formats)
--- a/youtube_dlc/extractor/crackle.py
+++ b/youtube_dlc/extractor/crackle.py
@@ -103,7 +103,7 @@ class CrackleIE(InfoExtractor):

            formats = []
            for e in media['MediaURLs']:
-                if e.get('UseDRM') is True:
+                if not self._downloader.params.get('allow_unplayable_formats') and e.get('UseDRM') is True:
                    continue
                format_url = url_or_none(e.get('Path'))
                if not format_url:
--- a/youtube_dlc/extractor/crunchyroll.py
+++ b/youtube_dlc/extractor/crunchyroll.py
@@ -473,15 +473,11 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
                stream.get('url'), video_id, stream.get('format'),
                audio_lang, hardsub_lang)
            for f in vrv_formats:
-                if not hardsub_lang:
-                    f['preference'] = 1
-                language_preference = 0
-                if audio_lang == language:
-                    language_preference += 1
-                if hardsub_lang == language:
-                    language_preference += 1
-                if language_preference:
-                    f['language_preference'] = language_preference
+                f['language_preference'] = 1 if audio_lang == language else 0
+                f['quality'] = (
+                    1 if not hardsub_lang
+                    else 0 if hardsub_lang == language
+                    else -1)
            formats.extend(vrv_formats)
        if not formats:
            available_fmts = []
@@ -571,7 +567,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
                        'ext': 'flv',
                    })
                    formats.append(format_info)
-        self._sort_formats(formats, ('preference', 'language_preference', 'height', 'width', 'tbr', 'fps'))
+        self._sort_formats(formats)

        metadata = self._call_rpc_api(
            'VideoPlayer_GetMediaMetadata', video_id,
--- a/youtube_dlc/extractor/dispeak.py
+++ b/youtube_dlc/extractor/dispeak.py
@@ -82,6 +82,7 @@ class DigitallySpeakingIE(InfoExtractor):
                'play_path': remove_end(audio.get('url'), '.flv'),
                'ext': 'flv',
                'vcodec': 'none',
+                'quality': 1,
                'format_id': audio.get('code'),
            })
        slide_video_path = xpath_text(metadata, './slideVideo', fatal=True)
@@ -91,7 +92,6 @@ class DigitallySpeakingIE(InfoExtractor):
            'ext': 'flv',
            'format_note': 'slide deck video',
            'quality': -2,
-            'preference': -2,
            'format_id': 'slides',
            'acodec': 'none',
        })
@@ -102,7 +102,6 @@ class DigitallySpeakingIE(InfoExtractor):
            'ext': 'flv',
            'format_note': 'speaker video',
            'quality': -1,
-            'preference': -1,
            'format_id': 'speaker',
        })
        return formats
--- a/youtube_dlc/extractor/dplay.py
+++ b/youtube_dlc/extractor/dplay.py
@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals

+import json
 import re

 from .common import InfoExtractor
@@ -10,11 +11,13 @@ from ..utils import (
    ExtractorError,
    float_or_none,
    int_or_none,
+    strip_or_none,
    unified_timestamp,
 )


 class DPlayIE(InfoExtractor):
+    _PATH_REGEX = r'/(?P<id>[^/]+/[^/?#]+)'
    _VALID_URL = r'''(?x)https?://
        (?P<domain>
            (?:www\.)?(?P<host>d
@@ -24,7 +27,7 @@ class DPlayIE(InfoExtractor):
                )
            )|
            (?P<subdomain_country>es|it)\.dplay\.com
-        )/[^/]+/(?P<id>[^/]+/[^/?#]+)'''
+        )/[^/]+''' + _PATH_REGEX

    _TESTS = [{
        # non geo restricted, via secure api, unsigned download hls URL
@@ -151,56 +154,79 @@ class DPlayIE(InfoExtractor):
        'only_matching': True,
    }]

+    def _process_errors(self, e, geo_countries):
+        info = self._parse_json(e.cause.read().decode('utf-8'), None)
+        error = info['errors'][0]
+        error_code = error.get('code')
+        if error_code == 'access.denied.geoblocked':
+            self.raise_geo_restricted(countries=geo_countries)
+        elif error_code in ('access.denied.missingpackage', 'invalid.token'):
+            raise ExtractorError(
+                'This video is only available for registered users. You may want to use --cookies.', expected=True)
+        raise ExtractorError(info['errors'][0]['detail'], expected=True)
+
+    def _update_disco_api_headers(self, headers, disco_base, display_id, realm):
+        headers['Authorization'] = 'Bearer ' + self._download_json(
+            disco_base + 'token', display_id, 'Downloading token',
+            query={
+                'realm': realm,
+            })['data']['attributes']['token']
+
+    def _download_video_playback_info(self, disco_base, video_id, headers):
+        streaming = self._download_json(
+            disco_base + 'playback/videoPlaybackInfo/' + video_id,
+            video_id, headers=headers)['data']['attributes']['streaming']
+        streaming_list = []
+        for format_id, format_dict in streaming.items():
+            streaming_list.append({
+                'type': format_id,
+                'url': format_dict.get('url'),
+            })
+        return streaming_list
+
    def _get_disco_api_info(self, url, display_id, disco_host, realm, country):
        geo_countries = [country.upper()]
        self._initialize_geo_bypass({
            'countries': geo_countries,
        })
        disco_base = 'https://%s/' % disco_host
-        token = self._download_json(
-            disco_base + 'token', display_id, 'Downloading token',
-            query={
-                'realm': realm,
-            })['data']['attributes']['token']
        headers = {
            'Referer': url,
-            'Authorization': 'Bearer ' + token,
        }
-        video = self._download_json(
-            disco_base + 'content/videos/' + display_id, display_id,
-            headers=headers, query={
-                'fields[channel]': 'name',
-                'fields[image]': 'height,src,width',
-                'fields[show]': 'name',
-                'fields[tag]': 'name',
-                'fields[video]': 'description,episodeNumber,name,publishStart,seasonNumber,videoDuration',
-                'include': 'images,primaryChannel,show,tags'
-            })
+        self._update_disco_api_headers(headers, disco_base, display_id, realm)
+        try:
+            video = self._download_json(
+                disco_base + 'content/videos/' + display_id, display_id,
+                headers=headers, query={
+                    'fields[channel]': 'name',
+                    'fields[image]': 'height,src,width',
+                    'fields[show]': 'name',
+                    'fields[tag]': 'name',
+                    'fields[video]': 'description,episodeNumber,name,publishStart,seasonNumber,videoDuration',
+                    'include': 'images,primaryChannel,show,tags'
+                })
+        except ExtractorError as e:
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 400:
+                self._process_errors(e, geo_countries)
+            raise
        video_id = video['data']['id']
        info = video['data']['attributes']
        title = info['name'].strip()
        formats = []
        try:
-            streaming = self._download_json(
-                disco_base + 'playback/videoPlaybackInfo/' + video_id,
-                display_id, headers=headers)['data']['attributes']['streaming']
+            streaming = self._download_video_playback_info(
+                disco_base, video_id, headers)
        except ExtractorError as e:
            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
-                info = self._parse_json(e.cause.read().decode('utf-8'), display_id)
-                error = info['errors'][0]
-                error_code = error.get('code')
-                if error_code == 'access.denied.geoblocked':
-                    self.raise_geo_restricted(countries=geo_countries)
-                elif error_code == 'access.denied.missingpackage':
-                    self.raise_login_required()
-                raise ExtractorError(info['errors'][0]['detail'], expected=True)
+                self._process_errors(e, geo_countries)
            raise
-        for format_id, format_dict in streaming.items():
+        for format_dict in streaming:
            if not isinstance(format_dict, dict):
                continue
            format_url = format_dict.get('url')
            if not format_url:
                continue
+            format_id = format_dict.get('type')
            ext = determine_ext(format_url)
            if format_id == 'dash' or ext == 'mpd':
                formats.extend(self._extract_mpd_formats(
@@ -248,7 +274,7 @@ class DPlayIE(InfoExtractor):
            'id': video_id,
            'display_id': display_id,
            'title': title,
-            'description': info.get('description'),
+            'description': strip_or_none(info.get('description')),
            'duration': float_or_none(info.get('videoDuration'), 1000),
            'timestamp': unified_timestamp(info.get('publishStart')),
            'series': series,
@@ -268,3 +294,75 @@ class DPlayIE(InfoExtractor):
        host = 'disco-api.' + domain if domain[0] == 'd' else 'eu2-prod.disco-api.com'
        return self._get_disco_api_info(
            url, display_id, host, 'dplay' + country, country)
+
+
+class DiscoveryPlusIE(DPlayIE):
+    _VALID_URL = r'https?://(?:www\.)?discoveryplus\.com/video' + DPlayIE._PATH_REGEX
+    _TESTS = [{
+        'url': 'https://www.discoveryplus.com/video/property-brothers-forever-home/food-and-family',
+        'info_dict': {
+            'id': '1140794',
+            'display_id': 'property-brothers-forever-home/food-and-family',
+            'ext': 'mp4',
+            'title': 'Food and Family',
+            'description': 'The brothers help a Richmond family expand their single-level home.',
+            'duration': 2583.113,
+            'timestamp': 1609304400,
+            'upload_date': '20201230',
+            'creator': 'HGTV',
+            'series': 'Property Brothers: Forever Home',
+            'season_number': 1,
+            'episode_number': 1,
+        },
+        'skip': 'Available for Premium users',
+    }]
+
+    def _update_disco_api_headers(self, headers, disco_base, display_id, realm):
+        headers['x-disco-client'] = 'WEB:UNKNOWN:dplus_us:15.0.0'
+
+    def _download_video_playback_info(self, disco_base, video_id, headers):
+        return self._download_json(
+            disco_base + 'playback/v3/videoPlaybackInfo',
+            video_id, headers=headers, data=json.dumps({
+                'deviceInfo': {
+                    'adBlocker': False,
+                },
+                'videoId': video_id,
+                'wisteriaProperties': {
+                    'platform': 'desktop',
+                },
+            }).encode('utf-8'))['data']['attributes']['streaming']
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        return self._get_disco_api_info(
+            url, display_id, 'us1-prod-direct.discoveryplus.com', 'go', 'us')
+
+
+class HGTVDeIE(DPlayIE):
+    _VALID_URL = r'https?://de\.hgtv\.com/sendungen' + DPlayIE._PATH_REGEX
+    _TESTS = [{
+        'url': 'https://de.hgtv.com/sendungen/tiny-house-klein-aber-oho/wer-braucht-schon-eine-toilette/',
+        'info_dict': {
+            'id': '151205',
+            'display_id': 'tiny-house-klein-aber-oho/wer-braucht-schon-eine-toilette',
+            'ext': 'mp4',
+            'title': 'Wer braucht schon eine Toilette',
+            'description': 'md5:05b40a27e7aed2c9172de34d459134e2',
+            'duration': 1177.024,
+            'timestamp': 1595705400,
+            'upload_date': '20200725',
+            'creator': 'HGTV',
+            'series': 'Tiny House - klein, aber oho',
+            'season_number': 3,
+            'episode_number': 3,
+        },
+        'params': {
+            'format': 'bestvideo',
+        },
+    }]
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        return self._get_disco_api_info(
+            url, display_id, 'eu1-prod.disco-api.com', 'hgtv', 'de')
--- a/youtube_dlc/extractor/dreisat.py
+++ b/youtube_dlc/extractor/dreisat.py
@@ -0,0 +1,193 @@
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    int_or_none,
+    unified_strdate,
+    xpath_text,
+    determine_ext,
+    float_or_none,
+    ExtractorError,
+)
+
+
+class DreiSatIE(InfoExtractor):
+    IE_NAME = '3sat'
+    _GEO_COUNTRIES = ['DE']
+    _VALID_URL = r'https?://(?:www\.)?3sat\.de/mediathek/(?:(?:index|mediathek)\.php)?\?(?:(?:mode|display)=[^&]+&)*obj=(?P<id>[0-9]+)'
+    _TESTS = [
+        {
+            'url': 'http://www.3sat.de/mediathek/index.php?mode=play&obj=45918',
+            'md5': 'be37228896d30a88f315b638900a026e',
+            'info_dict': {
+                'id': '45918',
+                'ext': 'mp4',
+                'title': 'Waidmannsheil',
+                'description': 'md5:cce00ca1d70e21425e72c86a98a56817',
+                'uploader': 'SCHWEIZWEIT',
+                'uploader_id': '100000210',
+                'upload_date': '20140913'
+            },
+            'params': {
+                'skip_download': True,  # m3u8 downloads
+            }
+        },
+        {
+            'url': 'http://www.3sat.de/mediathek/mediathek.php?mode=play&obj=51066',
+            'only_matching': True,
+        },
+    ]
+
+    def _parse_smil_formats(self, smil, smil_url, video_id, namespace=None, f4m_params=None, transform_rtmp_url=None):
+        param_groups = {}
+        for param_group in smil.findall(self._xpath_ns('./head/paramGroup', namespace)):
+            group_id = param_group.get(self._xpath_ns(
+                'id', 'http://www.w3.org/XML/1998/namespace'))
+            params = {}
+            for param in param_group:
+                params[param.get('name')] = param.get('value')
+            param_groups[group_id] = params
+
+        formats = []
+        for video in smil.findall(self._xpath_ns('.//video', namespace)):
+            src = video.get('src')
+            if not src:
+                continue
+            bitrate = int_or_none(self._search_regex(r'_(\d+)k', src, 'bitrate', None)) or float_or_none(video.get('system-bitrate') or video.get('systemBitrate'), 1000)
+            group_id = video.get('paramGroup')
+            param_group = param_groups[group_id]
+            for proto in param_group['protocols'].split(','):
+                formats.append({
+                    'url': '%s://%s' % (proto, param_group['host']),
+                    'app': param_group['app'],
+                    'play_path': src,
+                    'ext': 'flv',
+                    'format_id': '%s-%d' % (proto, bitrate),
+                    'tbr': bitrate,
+                })
+        self._sort_formats(formats)
+        return formats
+
+    def extract_from_xml_url(self, video_id, xml_url):
+        doc = self._download_xml(
+            xml_url, video_id,
+            note='Downloading video info',
+            errnote='Failed to download video info')
+
+        status_code = xpath_text(doc, './status/statuscode')
+        if status_code and status_code != 'ok':
+            if status_code == 'notVisibleAnymore':
+                message = 'Video %s is not available' % video_id
+            else:
+                message = '%s returned error: %s' % (self.IE_NAME, status_code)
+            raise ExtractorError(message, expected=True)
+
+        title = xpath_text(doc, './/information/title', 'title', True)
+
+        urls = []
+        formats = []
+        for fnode in doc.findall('.//formitaeten/formitaet'):
+            video_url = xpath_text(fnode, 'url')
+            if not video_url or video_url in urls:
+                continue
+            urls.append(video_url)
+
+            is_available = 'http://www.metafilegenerator' not in video_url
+            geoloced = 'static_geoloced_online' in video_url
+            if not is_available or geoloced:
+                continue
+
+            format_id = fnode.attrib['basetype']
+            format_m = re.match(r'''(?x)
+                (?P<vcodec>[^_]+)_(?P<acodec>[^_]+)_(?P<container>[^_]+)_
+                (?P<proto>[^_]+)_(?P<index>[^_]+)_(?P<indexproto>[^_]+)
+            ''', format_id)
+
+            ext = determine_ext(video_url, None) or format_m.group('container')
+
+            if ext == 'meta':
+                continue
+            elif ext == 'smil':
+                formats.extend(self._extract_smil_formats(
+                    video_url, video_id, fatal=False))
+            elif ext == 'm3u8':
+                # the certificates are misconfigured (see
+                # https://github.com/ytdl-org/youtube-dl/issues/8665)
+                if video_url.startswith('https://'):
+                    continue
+                formats.extend(self._extract_m3u8_formats(
+                    video_url, video_id, 'mp4', 'm3u8_native',
+                    m3u8_id=format_id, fatal=False))
+            elif ext == 'f4m':
+                formats.extend(self._extract_f4m_formats(
+                    video_url, video_id, f4m_id=format_id, fatal=False))
+            else:
+                quality = xpath_text(fnode, './quality')
+                if quality:
+                    format_id += '-' + quality
+
+                abr = int_or_none(xpath_text(fnode, './audioBitrate'), 1000)
+                vbr = int_or_none(xpath_text(fnode, './videoBitrate'), 1000)
+
+                tbr = int_or_none(self._search_regex(
+                    r'_(\d+)k', video_url, 'bitrate', None))
+                if tbr and vbr and not abr:
+                    abr = tbr - vbr
+
+                formats.append({
+                    'format_id': format_id,
+                    'url': video_url,
+                    'ext': ext,
+                    'acodec': format_m.group('acodec'),
+                    'vcodec': format_m.group('vcodec'),
+                    'abr': abr,
+                    'vbr': vbr,
+                    'tbr': tbr,
+                    'width': int_or_none(xpath_text(fnode, './width')),
+                    'height': int_or_none(xpath_text(fnode, './height')),
+                    'filesize': int_or_none(xpath_text(fnode, './filesize')),
+                    'protocol': format_m.group('proto').lower(),
+                })
+
+        geolocation = xpath_text(doc, './/details/geolocation')
+        if not formats and geolocation and geolocation != 'none':
+            self.raise_geo_restricted(countries=self._GEO_COUNTRIES)
+
+        self._sort_formats(formats)
+
+        thumbnails = []
+        for node in doc.findall('.//teaserimages/teaserimage'):
+            thumbnail_url = node.text
+            if not thumbnail_url:
+                continue
+            thumbnail = {
+                'url': thumbnail_url,
+            }
+            thumbnail_key = node.get('key')
+            if thumbnail_key:
+                m = re.match('^([0-9]+)x([0-9]+)$', thumbnail_key)
+                if m:
+                    thumbnail['width'] = int(m.group(1))
+                    thumbnail['height'] = int(m.group(2))
+            thumbnails.append(thumbnail)
+
+        upload_date = unified_strdate(xpath_text(doc, './/details/airtime'))
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': xpath_text(doc, './/information/detail'),
+            'duration': int_or_none(xpath_text(doc, './/details/lengthSec')),
+            'thumbnails': thumbnails,
+            'uploader': xpath_text(doc, './/details/originChannelTitle'),
+            'uploader_id': xpath_text(doc, './/details/originChannelId'),
+            'upload_date': upload_date,
+            'formats': formats,
+        }
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        details_url = 'http://www.3sat.de/mediathek/xmlservice/web/beitragsDetails?id=%s' % video_id
+        return self.extract_from_xml_url(video_id, details_url)
--- a/youtube_dlc/extractor/drtv.py
+++ b/youtube_dlc/extractor/drtv.py
@@ -242,7 +242,7 @@ class DRTVIE(InfoExtractor):
                    elif target == 'HLS':
                        formats.extend(self._extract_m3u8_formats(
                            uri, video_id, 'mp4', entry_protocol='m3u8_native',
-                            preference=preference, m3u8_id=format_id,
+                            quality=preference, m3u8_id=format_id,
                            fatal=False))
                    else:
                        bitrate = link.get('Bitrate')
@@ -254,7 +254,7 @@ class DRTVIE(InfoExtractor):
                            'tbr': int_or_none(bitrate),
                            'ext': link.get('FileFormat'),
                            'vcodec': 'none' if kind == 'AudioResource' else None,
-                            'preference': preference,
+                            'quality': preference,
                        })
            subtitles_list = asset.get('SubtitlesList') or asset.get('Subtitleslist')
            if isinstance(subtitles_list, list):
--- a/youtube_dlc/extractor/espn.py
+++ b/youtube_dlc/extractor/espn.py
@@ -154,7 +154,7 @@ class ESPNIE(OnceIE):
                        'tbr': int(mobj.group(3)),
                    })
                if source_id == 'mezzanine':
-                    f['preference'] = 1
+                    f['quality'] = 1
                formats.append(f)

        links = clip.get('links', {})
--- a/youtube_dlc/extractor/extractors.py
+++ b/youtube_dlc/extractor/extractors.py
@@ -182,6 +182,7 @@ from .canvas import (
    CanvasIE,
    CanvasEenIE,
    VrtNUIE,
+    DagelijkseKostIE,
 )
 from .carambatv import (
    CarambaTVIE,
@@ -309,7 +310,12 @@ from .douyutv import (
    DouyuShowIE,
    DouyuTVIE,
 )
-from .dplay import DPlayIE
+from .dplay import (
+    DPlayIE,
+    DiscoveryPlusIE,
+    HGTVDeIE,
+)
+from .dreisat import DreiSatIE
 from .drbonanza import DRBonanzaIE
 from .drtuber import DrTuberIE
 from .drtv import (
@@ -502,8 +508,8 @@ from .hungama import (
 from .hypem import HypemIE
 from .ign import (
    IGNIE,
-    OneUPIE,
-    PCMagIE,
+    IGNVideoIE,
+    IGNArticleIE,
 )
 from .iheart import (
    IHeartRadioIE,
@@ -1107,6 +1113,11 @@ from .shared import (
    VivoIE,
 )
 from .showroomlive import ShowRoomLiveIE
+from .simplecast import (
+    SimplecastIE,
+    SimplecastEpisodeIE,
+    SimplecastPodcastIE,
+)
 from .sina import SinaIE
 from .sixplay import SixPlayIE
 from .skyit import (
@@ -1165,11 +1176,6 @@ from .spike import (
    BellatorIE,
    ParamountNetworkIE,
 )
-from .storyfire import (
-    StoryFireIE,
-    StoryFireUserIE,
-    StoryFireSeriesIE,
-)
 from .stitcher import StitcherIE
 from .sport5 import Sport5IE
 from .sportbox import SportBoxIE
@@ -1193,6 +1199,11 @@ from .srgssr import (
 from .srmediathek import SRMediathekIE
 from .stanfordoc import StanfordOpenClassroomIE
 from .steam import SteamIE
+from .storyfire import (
+    StoryFireIE,
+    StoryFireUserIE,
+    StoryFireSeriesIE,
+)
 from .streamable import StreamableIE
 from .streamcloud import StreamcloudIE
 from .streamcz import StreamCZIE
@@ -1652,6 +1663,7 @@ from .zattoo import (
    ZattooLiveIE,
 )
 from .zdf import ZDFIE, ZDFChannelIE
+from .zhihu import ZhihuIE
 from .zingmp3 import ZingMp3IE
 from .zoom import ZoomIE
 from .zype import ZypeIE
--- a/youtube_dlc/extractor/facebook.py
+++ b/youtube_dlc/extractor/facebook.py
@@ -619,7 +619,7 @@ class FacebookIE(InfoExtractor):
                        formats.append({
                            'format_id': '%s_%s_%s' % (format_id, quality, src_type),
                            'url': src,
-                            'preference': preference,
+                            'quality': preference,
                        })
            extract_dash_manifest(f[0], formats)
            subtitles_src = f[0].get('subtitles_src')
--- a/youtube_dlc/extractor/firsttv.py
+++ b/youtube_dlc/extractor/firsttv.py
@@ -104,7 +104,7 @@ class FirstTVIE(InfoExtractor):
                    'tbr': tbr,
                    'source_preference': quality(f.get('name')),
                    # quality metadata of http formats may be incorrect
-                    'preference': -1,
+                    'preference': -10,
                })
            # m3u8 URL format is reverse engineered from [1] (search for
            # master.m3u8). dashEdges (that is currently balancer-vod.1tv.ru)
--- a/youtube_dlc/extractor/flickr.py
+++ b/youtube_dlc/extractor/flickr.py
@@ -88,7 +88,7 @@ class FlickrIE(InfoExtractor):
                formats.append({
                    'format_id': stream_type,
                    'url': stream['_content'],
-                    'preference': preference(stream_type),
+                    'quality': preference(stream_type),
                })
            self._sort_formats(formats)

--- a/youtube_dlc/extractor/generic.py
+++ b/youtube_dlc/extractor/generic.py
@@ -130,8 +130,10 @@ from .kinja import KinjaEmbedIE
 from .gedi import GediEmbedsIE
 from .rcs import RCSEmbedsIE
 from .bitchute import BitChuteIE
+from .rumble import RumbleEmbedIE
 from .arcpublishing import ArcPublishingIE
 from .medialaan import MedialaanIE
+from .simplecast import SimplecastIE


 class GenericIE(InfoExtractor):
@@ -2239,6 +2241,15 @@ class GenericIE(InfoExtractor):
                'duration': 159,
            },
        },
+        {
+            # Simplecast player embed
+            'url': 'https://www.bio.org/podcast',
+            'info_dict': {
+                'id': 'podcast',
+                'title': 'I AM BIO Podcast | BIO',
+            },
+            'playlist_mincount': 52,
+        },
    ]

    def report_following_redirect(self, new_url):
@@ -2793,6 +2804,12 @@ class GenericIE(InfoExtractor):
            return self.playlist_from_matches(
                matches, video_id, video_title, getter=unescapeHTML, ie='FunnyOrDie')

+        # Look for Simplecast embeds
+        simplecast_urls = SimplecastIE._extract_urls(webpage)
+        if simplecast_urls:
+            return self.playlist_from_matches(
+                simplecast_urls, video_id, video_title)
+
        # Look for BBC iPlayer embed
        matches = re.findall(r'setPlaylist\("(https?://www\.bbc\.co\.uk/iplayer/[^/]+/[\da-z]{8})"\)', webpage)
        if matches:
@@ -3338,6 +3355,13 @@ class GenericIE(InfoExtractor):
            return self.playlist_from_matches(
                bitchute_urls, video_id, video_title, ie=BitChuteIE.ie_key())

+        rumble_urls = RumbleEmbedIE._extract_urls(webpage)
+        if len(rumble_urls) == 1:
+            return self.url_result(rumble_urls[0], RumbleEmbedIE.ie_key())
+        if rumble_urls:
+            return self.playlist_from_matches(
+                rumble_urls, video_id, video_title, ie=RumbleEmbedIE.ie_key())
+
        # Look for HTML5 media
        entries = self._parse_html5_media_entries(url, webpage, video_id, m3u8_id='hls')
        if entries:
--- a/youtube_dlc/extractor/globo.py
+++ b/youtube_dlc/extractor/globo.py
@@ -96,7 +96,7 @@ class GloboIE(InfoExtractor):
        video = self._download_json(
            'http://api.globovideos.com/videos/%s/playlist' % video_id,
            video_id)['videos'][0]
-        if video.get('encrypted') is True:
+        if not self._downloader.params.get('allow_unplayable_formats') and video.get('encrypted') is True:
            raise ExtractorError('This video is DRM protected.', expected=True)

        title = video['title']
--- a/youtube_dlc/extractor/go.py
+++ b/youtube_dlc/extractor/go.py
@@ -236,7 +236,7 @@ class GoIE(AdobePassIE):
                if re.search(r'(?:/mp4/source/|_source\.mp4)', asset_url):
                    f.update({
                        'format_id': ('%s-' % format_id if format_id else '') + 'SOURCE',
-                        'preference': 1,
+                        'quality': 1,
                    })
                else:
                    mobj = re.search(r'/(\d+)x(\d+)/', asset_url)
--- a/youtube_dlc/extractor/hearthisat.py
+++ b/youtube_dlc/extractor/hearthisat.py
@@ -115,7 +115,7 @@ class HearThisAtIE(InfoExtractor):
                    'vcodec': 'none',
                    'ext': ext,
                    'url': download_url,
-                    'preference': 2,  # Usually better quality
+                    'quality': 2,  # Usually better quality
                })
        self._sort_formats(formats)

--- a/youtube_dlc/extractor/hotstar.py
+++ b/youtube_dlc/extractor/hotstar.py
@@ -141,7 +141,7 @@ class HotStarIE(HotStarBaseIE):

        title = video_data['title']

-        if video_data.get('drmProtected'):
+        if not self._downloader.params.get('allow_unplayable_formats') and video_data.get('drmProtected'):
            raise ExtractorError('This video is DRM protected.', expected=True)

        headers = {'Referer': url}
--- a/youtube_dlc/extractor/ign.py
+++ b/youtube_dlc/extractor/ign.py
@@ -3,230 +3,255 @@ from __future__ import unicode_literals
 import re

 from .common import InfoExtractor
+from ..compat import (
+    compat_parse_qs,
+    compat_urllib_parse_urlparse,
+)
 from ..utils import (
+    HEADRequest,
+    determine_ext,
    int_or_none,
    parse_iso8601,
+    strip_or_none,
+    try_get,
 )


-class IGNIE(InfoExtractor):
+class IGNBaseIE(InfoExtractor):
+    def _call_api(self, slug):
+        return self._download_json(
+            'http://apis.ign.com/{0}/v3/{0}s/slug/{1}'.format(self._PAGE_TYPE, slug), slug)
+
+
+class IGNIE(IGNBaseIE):
    """
    Extractor for some of the IGN sites, like www.ign.com, es.ign.com de.ign.com.
    Some videos of it.ign.com are also supported
    """

-    _VALID_URL = r'https?://.+?\.ign\.com/(?:[^/]+/)?(?P<type>videos|show_videos|articles|feature|(?:[^/]+/\d+/video))(/.+)?/(?P<name_or_id>.+)'
+    _VALID_URL = r'https?://(?:.+?\.ign|www\.pcmag)\.com/videos/(?:\d{4}/\d{2}/\d{2}/)?(?P<id>[^/?&#]+)'
    IE_NAME = 'ign.com'
+    _PAGE_TYPE = 'video'

-    _API_URL_TEMPLATE = 'http://apis.ign.com/video/v3/videos/%s'
-    _EMBED_RE = r'<iframe[^>]+?["\']((?:https?:)?//.+?\.ign\.com.+?/embed.+?)["\']'
-
-    _TESTS = [
-        {
-            'url': 'http://www.ign.com/videos/2013/06/05/the-last-of-us-review',
-            'md5': 'febda82c4bafecd2d44b6e1a18a595f8',
-            'info_dict': {
-                'id': '8f862beef863986b2785559b9e1aa599',
-                'ext': 'mp4',
-                'title': 'The Last of Us Review',
-                'description': 'md5:c8946d4260a4d43a00d5ae8ed998870c',
-                'timestamp': 1370440800,
-                'upload_date': '20130605',
-                'uploader_id': 'cberidon@ign.com',
-            }
-        },
-        {
-            'url': 'http://me.ign.com/en/feature/15775/100-little-things-in-gta-5-that-will-blow-your-mind',
-            'info_dict': {
-                'id': '100-little-things-in-gta-5-that-will-blow-your-mind',
-            },
-            'playlist': [
-                {
-                    'info_dict': {
-                        'id': '5ebbd138523268b93c9141af17bec937',
-                        'ext': 'mp4',
-                        'title': 'GTA 5 Video Review',
-                        'description': 'Rockstar drops the mic on this generation of games. Watch our review of the masterly Grand Theft Auto V.',
-                        'timestamp': 1379339880,
-                        'upload_date': '20130916',
-                        'uploader_id': 'danieljkrupa@gmail.com',
-                    },
-                },
-                {
-                    'info_dict': {
-                        'id': '638672ee848ae4ff108df2a296418ee2',
-                        'ext': 'mp4',
-                        'title': '26 Twisted Moments from GTA 5 in Slow Motion',
-                        'description': 'The twisted beauty of GTA 5 in stunning slow motion.',
-                        'timestamp': 1386878820,
-                        'upload_date': '20131212',
-                        'uploader_id': 'togilvie@ign.com',
-                    },
-                },
-            ],
-            'params': {
-                'skip_download': True,
-            },
-        },
-        {
-            'url': 'http://www.ign.com/articles/2014/08/15/rewind-theater-wild-trailer-gamescom-2014?watch',
-            'md5': '618fedb9c901fd086f6f093564ef8558',
-            'info_dict': {
-                'id': '078fdd005f6d3c02f63d795faa1b984f',
-                'ext': 'mp4',
-                'title': 'Rewind Theater - Wild Trailer Gamescom 2014',
-                'description': 'Brian and Jared explore Michel Ancel\'s captivating new preview.',
-                'timestamp': 1408047180,
-                'upload_date': '20140814',
-                'uploader_id': 'jamesduggan1990@gmail.com',
-            },
-        },
-        {
-            'url': 'http://me.ign.com/en/videos/112203/video/how-hitman-aims-to-be-different-than-every-other-s',
-            'only_matching': True,
-        },
-        {
-            'url': 'http://me.ign.com/ar/angry-birds-2/106533/video/lrd-ldyy-lwl-lfylm-angry-birds',
-            'only_matching': True,
-        },
-        {
-            # videoId pattern
-            'url': 'http://www.ign.com/articles/2017/06/08/new-ducktales-short-donalds-birthday-doesnt-go-as-planned',
-            'only_matching': True,
-        },
-    ]
-
-    def _find_video_id(self, webpage):
-        res_id = [
-            r'"video_id"\s*:\s*"(.*?)"',
-            r'class="hero-poster[^"]*?"[^>]*id="(.+?)"',
-            r'data-video-id="(.+?)"',
-            r'<object id="vid_(.+?)"',
-            r'<meta name="og:image" content=".*/(.+?)-(.+?)/.+.jpg"',
-            r'videoId&quot;\s*:\s*&quot;(.+?)&quot;',
-            r'videoId["\']\s*:\s*["\']([^"\']+?)["\']',
-        ]
-        return self._search_regex(res_id, webpage, 'video id', default=None)
+    _TESTS = [{
+        'url': 'http://www.ign.com/videos/2013/06/05/the-last-of-us-review',
+        'md5': 'd2e1586d9987d40fad7867bf96a018ea',
+        'info_dict': {
+            'id': '8f862beef863986b2785559b9e1aa599',
+            'ext': 'mp4',
+            'title': 'The Last of Us Review',
+            'description': 'md5:c8946d4260a4d43a00d5ae8ed998870c',
+            'timestamp': 1370440800,
+            'upload_date': '20130605',
+            'tags': 'count:9',
+        }
+    }, {
+        'url': 'http://www.pcmag.com/videos/2015/01/06/010615-whats-new-now-is-gogo-snooping-on-your-data',
+        'md5': 'f1581a6fe8c5121be5b807684aeac3f6',
+        'info_dict': {
+            'id': 'ee10d774b508c9b8ec07e763b9125b91',
+            'ext': 'mp4',
+            'title': 'What\'s New Now: Is GoGo Snooping on Your Data?',
+            'description': 'md5:817a20299de610bd56f13175386da6fa',
+            'timestamp': 1420571160,
+            'upload_date': '20150106',
+            'tags': 'count:4',
+        }
+    }, {
+        'url': 'https://www.ign.com/videos/is-a-resident-evil-4-remake-on-the-way-ign-daily-fix',
+        'only_matching': True,
+    }]

    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        name_or_id = mobj.group('name_or_id')
-        page_type = mobj.group('type')
-        webpage = self._download_webpage(url, name_or_id)
-        if page_type != 'video':
-            multiple_urls = re.findall(
-                r'<param name="flashvars"[^>]*value="[^"]*?url=(https?://www\.ign\.com/videos/.*?)["&]',
-                webpage)
-            if multiple_urls:
-                entries = [self.url_result(u, ie='IGN') for u in multiple_urls]
-                return {
-                    '_type': 'playlist',
-                    'id': name_or_id,
-                    'entries': entries,
-                }
-
-        video_id = self._find_video_id(webpage)
-        if not video_id:
-            return self.url_result(self._search_regex(
-                self._EMBED_RE, webpage, 'embed url'))
-        return self._get_video_info(video_id)
-
-    def _get_video_info(self, video_id):
-        api_data = self._download_json(
-            self._API_URL_TEMPLATE % video_id, video_id)
+        display_id = self._match_id(url)
+        video = self._call_api(display_id)
+        video_id = video['videoId']
+        metadata = video['metadata']
+        title = metadata.get('longTitle') or metadata.get('title') or metadata['name']

        formats = []
-        m3u8_url = api_data['refs'].get('m3uUrl')
+        refs = video.get('refs') or {}
+
+        m3u8_url = refs.get('m3uUrl')
        if m3u8_url:
            formats.extend(self._extract_m3u8_formats(
                m3u8_url, video_id, 'mp4', 'm3u8_native',
                m3u8_id='hls', fatal=False))
-        f4m_url = api_data['refs'].get('f4mUrl')
+
+        f4m_url = refs.get('f4mUrl')
        if f4m_url:
            formats.extend(self._extract_f4m_formats(
                f4m_url, video_id, f4m_id='hds', fatal=False))
-        for asset in api_data['assets']:
+
+        for asset in (video.get('assets') or []):
+            asset_url = asset.get('url')
+            if not asset_url:
+                continue
            formats.append({
-                'url': asset['url'],
-                'tbr': asset.get('actual_bitrate_kbps'),
-                'fps': asset.get('frame_rate'),
+                'url': asset_url,
+                'tbr': int_or_none(asset.get('bitrate'), 1000),
+                'fps': int_or_none(asset.get('frame_rate')),
                'height': int_or_none(asset.get('height')),
                'width': int_or_none(asset.get('width')),
            })
+
+        mezzanine_url = try_get(video, lambda x: x['system']['mezzanineUrl'])
+        if mezzanine_url:
+            formats.append({
+                'ext': determine_ext(mezzanine_url, 'mp4'),
+                'format_id': 'mezzanine',
+                'quality': 1,
+                'url': mezzanine_url,
+            })
+
        self._sort_formats(formats)

-        thumbnails = [{
-            'url': thumbnail['url']
-        } for thumbnail in api_data.get('thumbnails', [])]
+        thumbnails = []
+        for thumbnail in (video.get('thumbnails') or []):
+            thumbnail_url = thumbnail.get('url')
+            if not thumbnail_url:
+                continue
+            thumbnails.append({
+                'url': thumbnail_url,
+            })

-        metadata = api_data['metadata']
+        tags = []
+        for tag in (video.get('tags') or []):
+            display_name = tag.get('displayName')
+            if not display_name:
+                continue
+            tags.append(display_name)

        return {
-            'id': api_data.get('videoId') or video_id,
-            'title': metadata.get('longTitle') or metadata.get('name') or metadata.get['title'],
-            'description': metadata.get('description'),
+            'id': video_id,
+            'title': title,
+            'description': strip_or_none(metadata.get('description')),
            'timestamp': parse_iso8601(metadata.get('publishDate')),
            'duration': int_or_none(metadata.get('duration')),
-            'display_id': metadata.get('slug') or video_id,
-            'uploader_id': metadata.get('creator'),
+            'display_id': display_id,
            'thumbnails': thumbnails,
            'formats': formats,
+            'tags': tags,
        }


-class OneUPIE(IGNIE):
-    _VALID_URL = r'https?://gamevideos\.1up\.com/(?P<type>video)/id/(?P<name_or_id>.+)\.html'
-    IE_NAME = '1up.com'
-
+class IGNVideoIE(InfoExtractor):
+    _VALID_URL = r'https?://.+?\.ign\.com/(?:[a-z]{2}/)?[^/]+/(?P<id>\d+)/(?:video|trailer)/'
    _TESTS = [{
-        'url': 'http://gamevideos.1up.com/video/id/34976.html',
-        'md5': 'c9cc69e07acb675c31a16719f909e347',
+        'url': 'http://me.ign.com/en/videos/112203/video/how-hitman-aims-to-be-different-than-every-other-s',
+        'md5': 'dd9aca7ed2657c4e118d8b261e5e9de1',
        'info_dict': {
-            'id': '34976',
+            'id': 'e9be7ea899a9bbfc0674accc22a36cc8',
            'ext': 'mp4',
-            'title': 'Sniper Elite V2 - Trailer',
-            'description': 'md5:bf0516c5ee32a3217aa703e9b1bc7826',
-            'timestamp': 1313099220,
-            'upload_date': '20110811',
-            'uploader_id': 'IGN',
+            'title': 'How Hitman Aims to Be Different Than Every Other Stealth Game - NYCC 2015',
+            'description': 'Taking out assassination targets in Hitman has never been more stylish.',
+            'timestamp': 1444665600,
+            'upload_date': '20151012',
        }
+    }, {
+        'url': 'http://me.ign.com/ar/angry-birds-2/106533/video/lrd-ldyy-lwl-lfylm-angry-birds',
+        'only_matching': True,
+    }, {
+        # Youtube embed
+        'url': 'https://me.ign.com/ar/ratchet-clank-rift-apart/144327/trailer/embed',
+        'only_matching': True,
+    }, {
+        # Twitter embed
+        'url': 'http://adria.ign.com/sherlock-season-4/9687/trailer/embed',
+        'only_matching': True,
+    }, {
+        # Vimeo embed
+        'url': 'https://kr.ign.com/bic-2018/3307/trailer/embed',
+        'only_matching': True,
    }]

    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
-        result = super(OneUPIE, self)._real_extract(url)
-        result['id'] = mobj.group('name_or_id')
-        return result
+        video_id = self._match_id(url)
+        req = HEADRequest(url.rsplit('/', 1)[0] + '/embed')
+        url = self._request_webpage(req, video_id).geturl()
+        ign_url = compat_parse_qs(
+            compat_urllib_parse_urlparse(url).query).get('url', [None])[0]
+        if ign_url:
+            return self.url_result(ign_url, IGNIE.ie_key())
+        return self.url_result(url)


-class PCMagIE(IGNIE):
-    _VALID_URL = r'https?://(?:www\.)?pcmag\.com/(?P<type>videos|article2)(/.+)?/(?P<name_or_id>.+)'
-    IE_NAME = 'pcmag'
-
-    _EMBED_RE = r'iframe\.setAttribute\("src",\s*__util.objToUrlString\("http://widgets\.ign\.com/video/embed/content\.html?[^"]*url=([^"]+)["&]'
-
+class IGNArticleIE(IGNBaseIE):
+    _VALID_URL = r'https?://.+?\.ign\.com/(?:articles(?:/\d{4}/\d{2}/\d{2})?|(?:[a-z]{2}/)?feature/\d+)/(?P<id>[^/?&#]+)'
+    _PAGE_TYPE = 'article'
    _TESTS = [{
-        'url': 'http://www.pcmag.com/videos/2015/01/06/010615-whats-new-now-is-gogo-snooping-on-your-data',
-        'md5': '212d6154fd0361a2781075f1febbe9ad',
+        'url': 'http://me.ign.com/en/feature/15775/100-little-things-in-gta-5-that-will-blow-your-mind',
        'info_dict': {
-            'id': 'ee10d774b508c9b8ec07e763b9125b91',
-            'ext': 'mp4',
-            'title': '010615_What\'s New Now: Is GoGo Snooping on Your Data?',
-            'description': 'md5:a7071ae64d2f68cc821c729d4ded6bb3',
-            'timestamp': 1420571160,
-            'upload_date': '20150106',
-            'uploader_id': 'cozzipix@gmail.com',
-        }
+            'id': '524497489e4e8ff5848ece34',
+            'title': '100 Little Things in GTA 5 That Will Blow Your Mind',
+        },
+        'playlist': [
+            {
+                'info_dict': {
+                    'id': '5ebbd138523268b93c9141af17bec937',
+                    'ext': 'mp4',
+                    'title': 'GTA 5 Video Review',
+                    'description': 'Rockstar drops the mic on this generation of games. Watch our review of the masterly Grand Theft Auto V.',
+                    'timestamp': 1379339880,
+                    'upload_date': '20130916',
+                },
+            },
+            {
+                'info_dict': {
+                    'id': '638672ee848ae4ff108df2a296418ee2',
+                    'ext': 'mp4',
+                    'title': '26 Twisted Moments from GTA 5 in Slow Motion',
+                    'description': 'The twisted beauty of GTA 5 in stunning slow motion.',
+                    'timestamp': 1386878820,
+                    'upload_date': '20131212',
+                },
+            },
+        ],
+        'params': {
+            'playlist_items': '2-3',
+            'skip_download': True,
+        },
    }, {
-        'url': 'http://www.pcmag.com/article2/0,2817,2470156,00.asp',
-        'md5': '94130c1ca07ba0adb6088350681f16c1',
+        'url': 'http://www.ign.com/articles/2014/08/15/rewind-theater-wild-trailer-gamescom-2014?watch',
        'info_dict': {
-            'id': '042e560ba94823d43afcb12ddf7142ca',
-            'ext': 'mp4',
-            'title': 'HTC\'s Weird New Re Camera - What\'s New Now',
-            'description': 'md5:53433c45df96d2ea5d0fda18be2ca908',
-            'timestamp': 1412953920,
-            'upload_date': '20141010',
-            'uploader_id': 'chris_snyder@pcmag.com',
-        }
+            'id': '53ee806780a81ec46e0790f8',
+            'title': 'Rewind Theater - Wild Trailer Gamescom 2014',
+        },
+        'playlist_count': 2,
+    }, {
+        # videoId pattern
+        'url': 'http://www.ign.com/articles/2017/06/08/new-ducktales-short-donalds-birthday-doesnt-go-as-planned',
+        'only_matching': True,
+    }, {
+        # Youtube embed
+        'url': 'https://www.ign.com/articles/2021-mvp-named-in-puppy-bowl-xvii',
+        'only_matching': True,
+    }, {
+        # IMDB embed
+        'url': 'https://www.ign.com/articles/2014/08/07/sons-of-anarchy-final-season-trailer',
+        'only_matching': True,
+    }, {
+        # Facebook embed
+        'url': 'https://www.ign.com/articles/2017/09/20/marvels-the-punisher-watch-the-new-trailer-for-the-netflix-series',
+        'only_matching': True,
+    }, {
+        # Brightcove embed
+        'url': 'https://www.ign.com/articles/2016/01/16/supergirl-goes-flying-with-martian-manhunter-in-new-clip',
+        'only_matching': True,
    }]
+
+    def _real_extract(self, url):
+        display_id = self._match_id(url)
+        article = self._call_api(display_id)
+
+        def entries():
+            media_url = try_get(article, lambda x: x['mediaRelations'][0]['media']['metadata']['url'])
+            if media_url:
+                yield self.url_result(media_url, IGNIE.ie_key())
+            for content in (article.get('content') or []):
+                for video_url in re.findall(r'(?:\[(?:ignvideo\s+url|youtube\s+clip_id)|<iframe[^>]+src)="([^"]+)"', content):
+                    yield self.url_result(video_url)
+
+        return self.playlist_result(
+            entries(), article.get('articleId'),
+            strip_or_none(try_get(article, lambda x: x['metadata']['headline'])))
--- a/youtube_dlc/extractor/imgur.py
+++ b/youtube_dlc/extractor/imgur.py
@@ -72,7 +72,7 @@ class ImgurIE(InfoExtractor):
                gif_json, video_id, transform_source=js_to_json)
            formats.append({
                'format_id': 'gif',
-                'preference': -10,
+                'preference': -10,  # gifs are worse than videos
                'width': width,
                'height': height,
                'ext': 'gif',
--- a/youtube_dlc/extractor/iqiyi.py
+++ b/youtube_dlc/extractor/iqiyi.py
@@ -373,7 +373,7 @@ class IqiyiIE(InfoExtractor):
                    'url': stream['m3utx'],
                    'format_id': vd,
                    'ext': 'mp4',
-                    'preference': self._FORMATS_MAP.get(vd, -1),
+                    'quality': self._FORMATS_MAP.get(vd, -1),
                    'protocol': 'm3u8_native',
                })

--- a/youtube_dlc/extractor/ivi.py
+++ b/youtube_dlc/extractor/ivi.py
@@ -163,7 +163,10 @@ class IviIE(InfoExtractor):
        for f in result.get('files', []):
            f_url = f.get('url')
            content_format = f.get('content_format')
-            if not f_url or '-MDRM-' in content_format or '-FPS-' in content_format:
+            if not f_url:
+                continue
+            if (not self._downloader.params.get('allow_unplayable_formats')
+                    and ('-MDRM-' in content_format or '-FPS-' in content_format)):
                continue
            formats.append({
                'url': f_url,
--- a/youtube_dlc/extractor/kaltura.py
+++ b/youtube_dlc/extractor/kaltura.py
@@ -309,7 +309,7 @@ class KalturaIE(InfoExtractor):
            if f.get('fileExt') == 'chun':
                continue
            # DRM-protected video, cannot be decrypted
-            if f.get('fileExt') == 'wvm':
+            if not self._downloader.params.get('allow_unplayable_formats') and f.get('fileExt') == 'wvm':
                continue
            if not f.get('fileExt'):
                # QT indicates QuickTime; some videos have broken fileExt
--- a/youtube_dlc/extractor/kuwo.py
+++ b/youtube_dlc/extractor/kuwo.py
@@ -49,7 +49,7 @@ class KuwoBaseIE(InfoExtractor):
                    'url': song_url,
                    'format_id': file_format['format'],
                    'format': file_format['format'],
-                    'preference': file_format['preference'],
+                    'quality': file_format['preference'],
                    'abr': file_format.get('abr'),
                })

--- a/youtube_dlc/extractor/leeco.py
+++ b/youtube_dlc/extractor/leeco.py
@@ -185,7 +185,7 @@ class LeIE(InfoExtractor):
                    f['height'] = int_or_none(format_id[:-1])

                formats.append(f)
-        self._sort_formats(formats, ('height', 'quality', 'format_id'))
+        self._sort_formats(formats, ('res', 'quality'))

        publish_time = parse_iso8601(self._html_search_regex(
            r'发布时间&nbsp;([^<>]+) ', page, 'publish time', default=None),
--- a/youtube_dlc/extractor/lifenews.py
+++ b/youtube_dlc/extractor/lifenews.py
@@ -201,7 +201,7 @@ class LifeEmbedIE(InfoExtractor):
            formats.append({
                'url': original_url,
                'format_id': determine_ext(original_url, None),
-                'preference': 1,
+                'quality': 1,
            })

        playlist = self._parse_json(
--- a/youtube_dlc/extractor/limelight.py
+++ b/youtube_dlc/extractor/limelight.py
@@ -96,7 +96,9 @@ class LimelightBaseIE(InfoExtractor):
        urls = []
        for stream in pc_item.get('streams', []):
            stream_url = stream.get('url')
-            if not stream_url or stream.get('drmProtected') or stream_url in urls:
+            if not stream_url or stream_url in urls:
+                continue
+            if not self._downloader.params.get('allow_unplayable_formats') and stream.get('drmProtected'):
                continue
            urls.append(stream_url)
            ext = determine_ext(stream_url)
@@ -173,7 +175,7 @@ class LimelightBaseIE(InfoExtractor):
                formats.append({
                    'url': media_url,
                    'format_id': format_id,
-                    'preference': -1,
+                    'quality': -10,
                    'ext': ext,
                })

--- a/youtube_dlc/extractor/linkedin.py
+++ b/youtube_dlc/extractor/linkedin.py
@@ -124,7 +124,10 @@ class LinkedInLearningIE(LinkedInLearningBaseIE):
                streaming_url, video_slug, 'mp4',
                'm3u8_native', m3u8_id='hls', fatal=False))

-        self._sort_formats(formats, ('width', 'height', 'source_preference', 'tbr', 'abr'))
+        # It seems like this would be correctly handled by default
+        # However, unless someone can confirm this, the old
+        # behaviour is being kept as-is
+        self._sort_formats(formats, ('res', 'source_preference'))

        return {
            'id': self._get_video_id(video_data, course_slug, video_slug),
--- a/youtube_dlc/extractor/liveleak.py
+++ b/youtube_dlc/extractor/liveleak.py
@@ -143,7 +143,7 @@ class LiveLeakIE(InfoExtractor):
                        formats.append({
                            'format_id': format_id,
                            'url': orig_url,
-                            'preference': 1,
+                            'quality': 1,
                        })
            self._sort_formats(formats)
            info_dict['formats'] = formats
--- a/youtube_dlc/extractor/livestream.py
+++ b/youtube_dlc/extractor/livestream.py
@@ -84,7 +84,7 @@ class LivestreamIE(InfoExtractor):
                'format_id': 'smil_%d' % tbr,
                'ext': 'flv',
                'tbr': tbr,
-                'preference': -1000,
+                'preference': -1000,  # Strictly inferior than all other formats?
            })
        return formats

--- a/youtube_dlc/extractor/massengeschmacktv.py
+++ b/youtube_dlc/extractor/massengeschmacktv.py
@@ -67,7 +67,7 @@ class MassengeschmackTVIE(InfoExtractor):
                'vcodec': 'none' if format_id.startswith('Audio') else None,
            })

-        self._sort_formats(formats, ('width', 'height', 'filesize', 'tbr'))
+        self._sort_formats(formats)

        return {
            'id': episode,
--- a/youtube_dlc/extractor/mdr.py
+++ b/youtube_dlc/extractor/mdr.py
@@ -137,11 +137,11 @@ class MDRIE(InfoExtractor):
                if ext == 'm3u8':
                    formats.extend(self._extract_m3u8_formats(
                        video_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                        preference=0, m3u8_id='HLS', fatal=False))
+                        quality=1, m3u8_id='HLS', fatal=False))
                elif ext == 'f4m':
                    formats.extend(self._extract_f4m_formats(
                        video_url + '?hdcore=3.7.0&plugin=aasp-3.7.0.39.44', video_id,
-                        preference=0, f4m_id='HDS', fatal=False))
+                        quality=1, f4m_id='HDS', fatal=False))
                else:
                    media_type = xpath_text(asset, './mediaType', 'media type', default='MP4')
                    vbr = int_or_none(xpath_text(asset, './bitrateVideo', 'vbr'), 1000)
--- a/youtube_dlc/extractor/mediasite.py
+++ b/youtube_dlc/extractor/mediasite.py
@@ -206,7 +206,7 @@ class MediasiteIE(InfoExtractor):
            # disprefer 'secondary' streams
            if stream_type != 0:
                for fmt in stream_formats:
-                    fmt['preference'] = -1
+                    fmt['quality'] = -10

            thumbnail_url = Stream.get('ThumbnailUrl')
            if thumbnail_url:
--- a/youtube_dlc/extractor/msn.py
+++ b/youtube_dlc/extractor/msn.py
@@ -132,7 +132,7 @@ class MSNIE(InfoExtractor):
                        'width': int_or_none(file_.get('width')),
                        'height': int_or_none(file_.get('height')),
                        'vbr': int_or_none(self._search_regex(r'_(\d+)\.mp4', format_url, 'vbr', default=None)),
-                        'preference': 1 if format_id == '1001' else None,
+                        'quality': 1 if format_id == '1001' else None,
                    })
            self._sort_formats(formats)

--- a/youtube_dlc/extractor/muenchentv.py
+++ b/youtube_dlc/extractor/muenchentv.py
@@ -61,7 +61,7 @@ class MuenchenTVIE(InfoExtractor):
                'tbr': int_or_none(s.get('label')),
                'ext': 'mp4',
                'format_id': format_id,
-                'preference': -100 if '.smil' in s['file'] else 0,
+                'preference': -100 if '.smil' in s['file'] else 0,  # Strictly inferior than all other formats?
            })
        self._sort_formats(formats)

--- a/youtube_dlc/extractor/nba.py
+++ b/youtube_dlc/extractor/nba.py
@@ -303,7 +303,7 @@ class NBABaseIE(NBACVPBaseIE):
                formats.append({
                    'format_id': 'source',
                    'url': source_url,
-                    'preference': 1,
+                    'quality': 1,
                })

            m3u8_url = video.get('m3u8')
--- a/youtube_dlc/extractor/niconico.py
+++ b/youtube_dlc/extractor/niconico.py
@@ -1,25 +1,25 @@
 # coding: utf-8
 from __future__ import unicode_literals

-import datetime
-import functools
+import re
 import json
-import math
+import datetime

 from .common import InfoExtractor
+from ..postprocessor.ffmpeg import FFmpegPostProcessor
 from ..compat import (
    compat_parse_qs,
    compat_urllib_parse_urlparse,
 )
 from ..utils import (
-    determine_ext,
    dict_get,
    ExtractorError,
-    float_or_none,
-    InAdvancePagedList,
    int_or_none,
+    float_or_none,
+    OnDemandPagedList,
    parse_duration,
    parse_iso8601,
+    PostProcessingError,
    remove_start,
    try_get,
    unified_timestamp,
@@ -191,37 +191,87 @@ class NiconicoIE(InfoExtractor):
            self._downloader.report_warning('unable to log in: bad username or password')
        return login_ok

-    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
-        def yesno(boolean):
-            return 'yes' if boolean else 'no'
+    def _get_heartbeat_info(self, info_dict):

-        session_api_data = api_data['video']['dmcInfo']['session_api']
-        session_api_endpoint = session_api_data['urls'][0]
+        video_id, video_src_id, audio_src_id = info_dict['url'].split(':')[1].split('/')

-        format_id = '-'.join(map(lambda s: remove_start(s['id'], 'archive_'), [video_quality, audio_quality]))
+        # Get video webpage for API data.
+        webpage, handle = self._download_webpage_handle(
+            'http://www.nicovideo.jp/watch/' + video_id, video_id)
+
+        api_data = self._parse_json(self._html_search_regex(
+            'data-api-data="([^"]+)"', webpage,
+            'API data', default='{}'), video_id)
+
+        session_api_data = try_get(api_data, lambda x: x['video']['dmcInfo']['session_api'])
+        session_api_endpoint = try_get(session_api_data, lambda x: x['urls'][0])
+
+        # ping
+        self._download_json(
+            'https://nvapi.nicovideo.jp/v1/2ab0cbaa/watch', video_id,
+            query={'t': try_get(api_data, lambda x: x['video']['dmcInfo']['tracking_id'])},
+            headers={
+                'Origin': 'https://www.nicovideo.jp',
+                'Referer': 'https://www.nicovideo.jp/watch/' + video_id,
+                'X-Frontend-Id': '6',
+                'X-Frontend-Version': '0'
+            })
+
+        yesno = lambda x: 'yes' if x else 'no'
+
+        # m3u8 (encryption)
+        if 'encryption' in try_get(api_data, lambda x: x['video']['dmcInfo']) or {}:
+            protocol = 'm3u8'
+            session_api_http_parameters = {
+                'parameters': {
+                    'hls_parameters': {
+                        'encryption': {
+                            'hls_encryption_v1': {
+                                'encrypted_key': try_get(api_data, lambda x: x['video']['dmcInfo']['encryption']['hls_encryption_v1']['encrypted_key']),
+                                'key_uri': try_get(api_data, lambda x: x['video']['dmcInfo']['encryption']['hls_encryption_v1']['key_uri'])
+                            }
+                        },
+                        'transfer_preset': '',
+                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
+                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
+                        'segment_duration': 6000
+                    }
+                }
+            }
+        # http
+        else:
+            protocol = 'http'
+            session_api_http_parameters = {
+                'parameters': {
+                    'http_output_download_parameters': {
+                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
+                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
+                    }
+                }
+            }

        session_response = self._download_json(
            session_api_endpoint['url'], video_id,
            query={'_format': 'json'},
            headers={'Content-Type': 'application/json'},
-            note='Downloading JSON metadata for %s' % format_id,
+            note='Downloading JSON metadata for %s' % info_dict['format_id'],
            data=json.dumps({
                'session': {
                    'client_info': {
-                        'player_id': session_api_data['player_id'],
+                        'player_id': session_api_data.get('player_id'),
                    },
                    'content_auth': {
-                        'auth_type': session_api_data['auth_types'][session_api_data['protocols'][0]],
-                        'content_key_timeout': session_api_data['content_key_timeout'],
+                        'auth_type': try_get(session_api_data, lambda x: x['auth_types'][session_api_data['protocols'][0]]),
+                        'content_key_timeout': session_api_data.get('content_key_timeout'),
                        'service_id': 'nicovideo',
-                        'service_user_id': session_api_data['service_user_id']
+                        'service_user_id': session_api_data.get('service_user_id')
                    },
-                    'content_id': session_api_data['content_id'],
+                    'content_id': session_api_data.get('content_id'),
                    'content_src_id_sets': [{
                        'content_src_ids': [{
                            'src_id_to_mux': {
-                                'audio_src_ids': [audio_quality['id']],
-                                'video_src_ids': [video_quality['id']],
+                                'audio_src_ids': [audio_src_id],
+                                'video_src_ids': [video_src_id],
                            }
                        }]
                    }],
@@ -229,52 +279,78 @@ class NiconicoIE(InfoExtractor):
                    'content_uri': '',
                    'keep_method': {
                        'heartbeat': {
-                            'lifetime': session_api_data['heartbeat_lifetime']
+                            'lifetime': session_api_data.get('heartbeat_lifetime')
                        }
                    },
-                    'priority': session_api_data['priority'],
+                    'priority': session_api_data.get('priority'),
                    'protocol': {
                        'name': 'http',
                        'parameters': {
-                            'http_parameters': {
-                                'parameters': {
-                                    'http_output_download_parameters': {
-                                        'use_ssl': yesno(session_api_endpoint['is_ssl']),
-                                        'use_well_known_port': yesno(session_api_endpoint['is_well_known_port']),
-                                    }
-                                }
-                            }
+                            'http_parameters': session_api_http_parameters
                        }
                    },
-                    'recipe_id': session_api_data['recipe_id'],
+                    'recipe_id': session_api_data.get('recipe_id'),
                    'session_operation_auth': {
                        'session_operation_auth_by_signature': {
-                            'signature': session_api_data['signature'],
-                            'token': session_api_data['token'],
+                            'signature': session_api_data.get('signature'),
+                            'token': session_api_data.get('token'),
                        }
                    },
                    'timing_constraint': 'unlimited'
                }
            }).encode())

-        resolution = video_quality.get('resolution', {})
+        info_dict['url'] = session_response['data']['session']['content_uri']
+        info_dict['protocol'] = protocol
+
+        # get heartbeat info
+        heartbeat_info_dict = {
+            'url': session_api_endpoint['url'] + '/' + session_response['data']['session']['id'] + '?_format=json&_method=PUT',
+            'data': json.dumps(session_response['data']),
+            # interval, convert milliseconds to seconds, then halve to make a buffer.
+            'interval': float_or_none(session_api_data.get('heartbeat_lifetime'), scale=2000),
+        }
+
+        return info_dict, heartbeat_info_dict
+
+    def _extract_format_for_quality(self, api_data, video_id, audio_quality, video_quality):
+        def parse_format_id(id_code):
+            mobj = re.match(r'''(?x)
+                    (?:archive_)?
+                    (?:(?P<codec>[^_]+)_)?
+                    (?:(?P<br>[\d]+)kbps_)?
+                    (?:(?P<res>[\d+]+)p_)?
+                ''', '%s_' % id_code)
+            return mobj.groupdict() if mobj else {}
+
+        protocol = 'niconico_dmc'
+        format_id = '-'.join(map(lambda s: remove_start(s['id'], 'archive_'), [video_quality, audio_quality]))
+        vdict = parse_format_id(video_quality['id'])
+        adict = parse_format_id(audio_quality['id'])
+        resolution = video_quality.get('resolution', {'height': vdict.get('res')})

        return {
-            'url': session_response['data']['session']['content_uri'],
+            'url': '%s:%s/%s/%s' % (protocol, video_id, video_quality['id'], audio_quality['id']),
            'format_id': format_id,
            'ext': 'mp4',  # Session API are used in HTML5, which always serves mp4
-            'abr': float_or_none(audio_quality.get('bitrate'), 1000),
-            'vbr': float_or_none(video_quality.get('bitrate'), 1000),
-            'height': resolution.get('height'),
-            'width': resolution.get('width'),
+            'vcodec': vdict.get('codec'),
+            'acodec': adict.get('codec'),
+            'vbr': float_or_none(video_quality.get('bitrate'), 1000) or float_or_none(vdict.get('br')),
+            'abr': float_or_none(audio_quality.get('bitrate'), 1000) or float_or_none(adict.get('br')),
+            'height': int_or_none(resolution.get('height', vdict.get('res'))),
+            'width': int_or_none(resolution.get('width')),
+            'quality': -2 if 'low' in format_id else -1,  # Default quality value is -1
+            'protocol': protocol,
+            'http_headers': {
+                'Origin': 'https://www.nicovideo.jp',
+                'Referer': 'https://www.nicovideo.jp/watch/' + video_id,
+            }
        }

    def _real_extract(self, url):
        video_id = self._match_id(url)

-        # Get video webpage. We are not actually interested in it for normal
-        # cases, but need the cookies in order to be able to download the
-        # info webpage
+        # Get video webpage for API data.
        webpage, handle = self._download_webpage_handle(
            'http://www.nicovideo.jp/watch/' + video_id, video_id)
        if video_id.startswith('so'):
@@ -284,80 +360,134 @@ class NiconicoIE(InfoExtractor):
            'data-api-data="([^"]+)"', webpage,
            'API data', default='{}'), video_id)

-        def _format_id_from_url(video_url):
-            return 'economy' if video_real_url.endswith('low') else 'normal'
+        def get_video_info_web(items):
+            return dict_get(api_data['video'], items)
+
+        # Get video info
+        video_info_xml = self._download_xml(
+            'http://ext.nicovideo.jp/api/getthumbinfo/' + video_id,
+            video_id, note='Downloading video info page')
+
+        def get_video_info_xml(items):
+            if not isinstance(items, list):
+                items = [items]
+            for item in items:
+                ret = xpath_text(video_info_xml, './/' + item)
+                if ret:
+                    return ret
+
+        if get_video_info_xml('error'):
+            error_code = get_video_info_xml('code')
+
+            if error_code == 'DELETED':
+                raise ExtractorError('The video has been deleted.',
+                                     expected=True)
+            elif error_code == 'NOT_FOUND':
+                raise ExtractorError('The video is not found.',
+                                     expected=True)
+            elif error_code == 'COMMUNITY':
+                self.to_screen('%s: The video is community members only.' % video_id)
+            else:
+                raise ExtractorError('%s reports error: %s' % (self.IE_NAME, error_code))
+
+        # Start extracting video formats
+        formats = []
+
+        # Get HTML5 videos info
+        try:
+            dmc_info = api_data['video']['dmcInfo']
+        except KeyError:
+            raise ExtractorError('The video can\'t downloaded.',
+                                 expected=True)
+
+        quality_info = dmc_info.get('quality')
+        for audio_quality in quality_info.get('audios') or {}:
+            for video_quality in quality_info.get('videos') or {}:
+                if not audio_quality.get('available') or not video_quality.get('available'):
+                    continue
+                formats.append(self._extract_format_for_quality(
+                    api_data, video_id, audio_quality, video_quality))
+
+        # Get flv/swf info
+        video_real_url = try_get(api_data, lambda x: x['video']['smileInfo']['url'])
+        is_economy = video_real_url.endswith('low')
+
+        if is_economy:
+            self.report_warning('Site is currently in economy mode! You will only have access to lower quality streams')
+
+        # Invoking ffprobe to determine resolution
+        pp = FFmpegPostProcessor(self._downloader)
+        cookies = self._get_cookies('https://nicovideo.jp').output(header='', sep='; path=/; domain=nicovideo.jp;\n')
+
+        self.to_screen('%s: %s' % (video_id, 'Checking smile format with ffprobe'))

        try:
-            video_real_url = api_data['video']['smileInfo']['url']
-        except KeyError:  # Flash videos
-            # Get flv info
-            flv_info_webpage = self._download_webpage(
-                'http://flapi.nicovideo.jp/api/getflv/' + video_id + '?as3=1',
-                video_id, 'Downloading flv info')
+            metadata = pp.get_metadata_object(video_real_url, ['-cookies', cookies])
+        except PostProcessingError as err:
+            raise ExtractorError(err.msg, expected=True)

-            flv_info = compat_parse_qs(flv_info_webpage)
-            if 'url' not in flv_info:
-                if 'deleted' in flv_info:
-                    raise ExtractorError('The video has been deleted.',
-                                         expected=True)
-                elif 'closed' in flv_info:
-                    raise ExtractorError('Niconico videos now require logging in',
-                                         expected=True)
-                elif 'error' in flv_info:
-                    raise ExtractorError('%s reports error: %s' % (
-                        self.IE_NAME, flv_info['error'][0]), expected=True)
-                else:
-                    raise ExtractorError('Unable to find video URL')
+        v_stream = a_stream = {}

-            video_info_xml = self._download_xml(
-                'http://ext.nicovideo.jp/api/getthumbinfo/' + video_id,
-                video_id, note='Downloading video info page')
+        # Some complex swf files doesn't have video stream (e.g. nm4809023)
+        for stream in metadata['streams']:
+            if stream['codec_type'] == 'video':
+                v_stream = stream
+            elif stream['codec_type'] == 'audio':
+                a_stream = stream

-            def get_video_info(items):
-                if not isinstance(items, list):
-                    items = [items]
-                for item in items:
-                    ret = xpath_text(video_info_xml, './/' + item)
-                    if ret:
-                        return ret
+        # Community restricted videos seem to have issues with the thumb API not returning anything at all
+        filesize = int(
+            (get_video_info_xml('size_high') if not is_economy else get_video_info_xml('size_low'))
+            or metadata['format']['size']
+        )
+        extension = (
+            get_video_info_xml('movie_type')
+            or 'mp4' if 'mp4' in metadata['format']['format_name'] else metadata['format']['format_name']
+        )

-            video_real_url = flv_info['url'][0]
+        # 'creation_time' tag on video stream of re-encoded SMILEVIDEO mp4 files are '1970-01-01T00:00:00.000000Z'.
+        timestamp = (
+            parse_iso8601(get_video_info_web('first_retrieve'))
+            or unified_timestamp(get_video_info_web('postedDateTime'))
+        )
+        metadata_timestamp = (
+            parse_iso8601(try_get(v_stream, lambda x: x['tags']['creation_time']))
+            or timestamp if extension != 'mp4' else 0
+        )

-            extension = get_video_info('movie_type')
-            if not extension:
-                extension = determine_ext(video_real_url)
+        # According to compconf, smile videos from pre-2017 are always better quality than their DMC counterparts
+        smile_threshold_timestamp = parse_iso8601('2016-12-08T00:00:00+09:00')

-            formats = [{
+        is_source = timestamp < smile_threshold_timestamp or metadata_timestamp > 0
+
+        # If movie file size is unstable, old server movie is not source movie.
+        if filesize > 1:
+            formats.append({
                'url': video_real_url,
+                'format_id': 'smile' if not is_economy else 'smile_low',
+                'format_note': 'SMILEVIDEO source' if not is_economy else 'SMILEVIDEO low quality',
                'ext': extension,
-                'format_id': _format_id_from_url(video_real_url),
-            }]
-        else:
-            formats = []
+                'container': extension,
+                'vcodec': v_stream.get('codec_name'),
+                'acodec': a_stream.get('codec_name'),
+                # Some complex swf files doesn't have total bit rate metadata (e.g. nm6049209)
+                'tbr': int_or_none(metadata['format'].get('bit_rate'), scale=1000),
+                'vbr': int_or_none(v_stream.get('bit_rate'), scale=1000),
+                'abr': int_or_none(a_stream.get('bit_rate'), scale=1000),
+                'height': int_or_none(v_stream.get('height')),
+                'width': int_or_none(v_stream.get('width')),
+                'source_preference': 5 if not is_economy else -2,
+                'quality': 5 if is_source and not is_economy else None,
+                'filesize': filesize
+            })

-            dmc_info = api_data['video'].get('dmcInfo')
-            if dmc_info:  # "New" HTML5 videos
-                quality_info = dmc_info['quality']
-                for audio_quality in quality_info['audios']:
-                    for video_quality in quality_info['videos']:
-                        if not audio_quality['available'] or not video_quality['available']:
-                            continue
-                        formats.append(self._extract_format_for_quality(
-                            api_data, video_id, audio_quality, video_quality))
+        if len(formats) == 0:
+            raise ExtractorError('Unable to find video info.')

-                self._sort_formats(formats)
-            else:  # "Old" HTML5 videos
-                formats = [{
-                    'url': video_real_url,
-                    'ext': 'mp4',
-                    'format_id': _format_id_from_url(video_real_url),
-                }]
-
-            def get_video_info(items):
-                return dict_get(api_data['video'], items)
+        self._sort_formats(formats)

        # Start extracting information
-        title = get_video_info('title')
+        title = get_video_info_web('originalTitle')
        if not title:
            title = self._og_search_title(webpage, default=None)
        if not title:
@@ -372,14 +502,13 @@ class NiconicoIE(InfoExtractor):
        video_detail = watch_api_data.get('videoDetail', {})

        thumbnail = (
-            get_video_info(['thumbnail_url', 'thumbnailURL'])
+            self._html_search_regex(r'<meta property="og:image" content="([^"]+)">', webpage, 'thumbnail data', default=None)
+            or get_video_info_web(['thumbnail_url', 'largeThumbnailURL', 'thumbnailURL'])
            or self._html_search_meta('image', webpage, 'thumbnail', default=None)
            or video_detail.get('thumbnail'))

-        description = get_video_info('description')
+        description = get_video_info_web('description')

-        timestamp = (parse_iso8601(get_video_info('first_retrieve'))
-                     or unified_timestamp(get_video_info('postedDateTime')))
        if not timestamp:
            match = self._html_search_meta('datePublished', webpage, 'date published', default=None)
            if match:
@@ -389,7 +518,7 @@ class NiconicoIE(InfoExtractor):
                video_detail['postedAt'].replace('/', '-'),
                delimiter=' ', timezone=datetime.timedelta(hours=9))

-        view_count = int_or_none(get_video_info(['view_counter', 'viewCount']))
+        view_count = int_or_none(get_video_info_web(['view_counter', 'viewCount']))
        if not view_count:
            match = self._html_search_regex(
                r'>Views: <strong[^>]*>([^<]+)</strong>',
@@ -398,7 +527,7 @@ class NiconicoIE(InfoExtractor):
                view_count = int_or_none(match.replace(',', ''))
        view_count = view_count or video_detail.get('viewCount')

-        comment_count = (int_or_none(get_video_info('comment_num'))
+        comment_count = (int_or_none(get_video_info_web('comment_num'))
                         or video_detail.get('commentCount')
                         or try_get(api_data, lambda x: x['thread']['commentCount']))
        if not comment_count:
@@ -409,19 +538,37 @@ class NiconicoIE(InfoExtractor):
                comment_count = int_or_none(match.replace(',', ''))

        duration = (parse_duration(
-            get_video_info('length')
+            get_video_info_web('length')
            or self._html_search_meta(
                'video:duration', webpage, 'video duration', default=None))
            or video_detail.get('length')
-            or get_video_info('duration'))
+            or get_video_info_web('duration'))

-        webpage_url = get_video_info('watch_url') or url
+        webpage_url = get_video_info_web('watch_url') or url
+
+        # for channel movie and community movie
+        channel_id = try_get(
+            api_data,
+            (lambda x: x['channel']['globalId'],
+             lambda x: x['community']['globalId']))
+        channel = try_get(
+            api_data,
+            (lambda x: x['channel']['name'],
+             lambda x: x['community']['name']))

        # Note: cannot use api_data.get('owner', {}) because owner may be set to "null"
        # in the JSON, which will cause None to be returned instead of {}.
        owner = try_get(api_data, lambda x: x.get('owner'), dict) or {}
-        uploader_id = get_video_info(['ch_id', 'user_id']) or owner.get('id')
-        uploader = get_video_info(['ch_name', 'user_nickname']) or owner.get('nickname')
+        uploader_id = (
+            get_video_info_web(['ch_id', 'user_id'])
+            or owner.get('id')
+            or channel_id
+        )
+        uploader = (
+            get_video_info_web(['ch_name', 'user_nickname'])
+            or owner.get('nickname')
+            or channel
+        )

        return {
            'id': video_id,
@@ -432,6 +579,8 @@ class NiconicoIE(InfoExtractor):
            'uploader': uploader,
            'timestamp': timestamp,
            'uploader_id': uploader_id,
+            'channel': channel,
+            'channel_id': channel_id,
            'view_count': view_count,
            'comment_count': comment_count,
            'duration': duration,
@@ -456,60 +605,45 @@ class NiconicoPlaylistIE(InfoExtractor):
        'url': 'https://www.nicovideo.jp/user/805442/mylist/27411728',
        'only_matching': True,
    }]
-    _PAGE_SIZE = 100
-
-    def _call_api(self, list_id, resource, query):
-        return self._download_json(
-            'https://nvapi.nicovideo.jp/v2/mylists/' + list_id, list_id,
-            'Downloading %s JSON metatdata' % resource, query=query,
-            headers={'X-Frontend-Id': 6})['data']['mylist']
-
-    def _parse_owner(self, item):
-        owner = item.get('owner') or {}
-        if owner:
-            return {
-                'uploader': owner.get('name'),
-                'uploader_id': owner.get('id'),
-            }
-        return {}
-
-    def _fetch_page(self, list_id, page):
-        page += 1
-        items = self._call_api(list_id, 'page %d' % page, {
-            'page': page,
-            'pageSize': self._PAGE_SIZE,
-        })['items']
-        for item in items:
-            video = item.get('video') or {}
-            video_id = video.get('id')
-            if not video_id:
-                continue
-            count = video.get('count') or {}
-            get_count = lambda x: int_or_none(count.get(x))
-            info = {
-                '_type': 'url',
-                'id': video_id,
-                'title': video.get('title'),
-                'url': 'https://www.nicovideo.jp/watch/' + video_id,
-                'description': video.get('shortDescription'),
-                'duration': int_or_none(video.get('duration')),
-                'view_count': get_count('view'),
-                'comment_count': get_count('comment'),
-                'ie_key': NiconicoIE.ie_key(),
-            }
-            info.update(self._parse_owner(video))
-            yield info

    def _real_extract(self, url):
        list_id = self._match_id(url)
-        mylist = self._call_api(list_id, 'list', {
-            'pageSize': 1,
-        })
-        entries = InAdvancePagedList(
-            functools.partial(self._fetch_page, list_id),
-            math.ceil(mylist['totalItemCount'] / self._PAGE_SIZE),
-            self._PAGE_SIZE)
-        result = self.playlist_result(
-            entries, list_id, mylist.get('name'), mylist.get('description'))
-        result.update(self._parse_owner(mylist))
-        return result
+        webpage = self._download_webpage(url, list_id)
+
+        header = self._parse_json(self._html_search_regex(
+            r'data-common-header="([^"]+)"', webpage,
+            'webpage header'), list_id)
+        frontendId = header.get('initConfig').get('frontendId')
+        frontendVersion = header.get('initConfig').get('frontendVersion')
+
+        def get_page_data(pagenum, pagesize):
+            return self._download_json(
+                'http://nvapi.nicovideo.jp/v2/mylists/' + list_id, list_id,
+                query={'page': 1 + pagenum, 'pageSize': pagesize},
+                headers={
+                    'X-Frontend-Id': frontendId,
+                    'X-Frontend-Version': frontendVersion,
+                }).get('data').get('mylist')
+
+        data = get_page_data(0, 1)
+        title = data.get('name')
+        description = data.get('description')
+        uploader = data.get('owner').get('name')
+        uploader_id = data.get('owner').get('id')
+
+        def pagefunc(pagenum):
+            data = get_page_data(pagenum, 25)
+            return ({
+                '_type': 'url',
+                'url': 'http://www.nicovideo.jp/watch/' + item.get('watchId'),
+            } for item in data.get('items'))
+
+        return {
+            '_type': 'playlist',
+            'id': list_id,
+            'title': title,
+            'description': description,
+            'uploader': uploader,
+            'uploader_id': uploader_id,
+            'entries': OnDemandPagedList(pagefunc, 25),
+        }
--- a/youtube_dlc/extractor/ninecninemedia.py
+++ b/youtube_dlc/extractor/ninecninemedia.py
@@ -36,7 +36,8 @@ class NineCNineMediaIE(InfoExtractor):
                '$include': '[HasClosedCaptions]',
            })

-        if try_get(content_package, lambda x: x['Constraints']['Security']['Type']):
+        if (not self._downloader.params.get('allow_unplayable_formats')
+                and try_get(content_package, lambda x: x['Constraints']['Security']['Type'])):
            raise ExtractorError('This video is DRM protected.', expected=True)

        manifest_base_url = content_package_url + 'manifest.'
--- a/youtube_dlc/extractor/ninegag.py
+++ b/youtube_dlc/extractor/ninegag.py
@@ -2,10 +2,11 @@ from __future__ import unicode_literals

 from .common import InfoExtractor
 from ..utils import (
-    determine_ext,
    ExtractorError,
+    determine_ext,
    int_or_none,
    try_get,
+    unescapeHTML,
    url_or_none,
 )

@@ -14,7 +15,7 @@ class NineGagIE(InfoExtractor):
    IE_NAME = '9gag'
    _VALID_URL = r'https?://(?:www\.)?9gag\.com/gag/(?P<id>[^/?&#]+)'

-    _TEST = {
+    _TESTS = [{
        'url': 'https://9gag.com/gag/ae5Ag7B',
        'info_dict': {
            'id': 'ae5Ag7B',
@@ -29,7 +30,11 @@ class NineGagIE(InfoExtractor):
            'dislike_count': int,
            'comment_count': int,
        }
-    }
+    }, {
+        # HTML escaped title
+        'url': 'https://9gag.com/gag/av5nvyb',
+        'only_matching': True,
+    }]

    def _real_extract(self, url):
        post_id = self._match_id(url)
@@ -43,7 +48,7 @@ class NineGagIE(InfoExtractor):
                'The given url does not contain a video',
                expected=True)

-        title = post['title']
+        title = unescapeHTML(post['title'])

        duration = None
        formats = []
--- a/youtube_dlc/extractor/ninenow.py
+++ b/youtube_dlc/extractor/ninenow.py
@@ -66,7 +66,7 @@ class NineNowIE(InfoExtractor):

        video_data = common_data['video']

-        if video_data.get('drm'):
+        if not self._downloader.params.get('allow_unplayable_formats') and video_data.get('drm'):
            raise ExtractorError('This video is DRM protected.', expected=True)

        brightcove_id = video_data.get('brightcoveId') or 'ref:' + video_data['referenceId']
--- a/youtube_dlc/extractor/noco.py
+++ b/youtube_dlc/extractor/noco.py
@@ -183,7 +183,7 @@ class NocoIE(InfoExtractor):
                        'filesize': int_or_none(fmt.get('filesize')),
                        'format_note': qualities[format_id].get('quality_name'),
                        'quality': qualities[format_id].get('priority'),
-                        'preference': preference,
+                        'language_preference': preference,
                    })

        self._sort_formats(formats)
--- a/youtube_dlc/extractor/npo.py
+++ b/youtube_dlc/extractor/npo.py
@@ -246,7 +246,7 @@ class NPOIE(NPOBaseIE):
                })

        if not formats:
-            if drm:
+            if not self._downloader.params.get('allow_unplayable_formats') and drm:
                raise ExtractorError('This video is DRM protected.', expected=True)
            return

@@ -425,7 +425,7 @@ class NPOIE(NPOBaseIE):
                        stream_url, video_id, fatal=False)
                    # f4m downloader downloads only piece of live stream
                    for f4m_format in f4m_formats:
-                        f4m_format['preference'] = -1
+                        f4m_format['preference'] = -5
                    formats.extend(f4m_formats)
                elif stream_type == 'hls':
                    formats.extend(self._extract_m3u8_formats(
--- a/youtube_dlc/extractor/ntvde.py
+++ b/youtube_dlc/extractor/ntvde.py
@@ -62,7 +62,7 @@ class NTVDeIE(InfoExtractor):
            m3u8_url = compat_urlparse.urljoin('http://video.n-tv.de', vdata['videoM3u8'])
            formats.extend(self._extract_m3u8_formats(
                m3u8_url, video_id, ext='mp4', entry_protocol='m3u8_native',
-                preference=0, m3u8_id='hls', fatal=False))
+                quality=1, m3u8_id='hls', fatal=False))
        self._sort_formats(formats)

        return {
--- a/youtube_dlc/extractor/nytimes.py
+++ b/youtube_dlc/extractor/nytimes.py
@@ -72,7 +72,7 @@ class NYTimesBaseIE(InfoExtractor):
                    'tbr': int_or_none(video.get('bitrate'), 1000) or None,
                    'ext': ext,
                })
-        self._sort_formats(formats, ('height', 'width', 'filesize', 'tbr', 'fps', 'format_id'))
+        self._sort_formats(formats)

        thumbnails = []
        for image in video_data.get('images', []):
--- a/youtube_dlc/extractor/ora.py
+++ b/youtube_dlc/extractor/ora.py
@@ -55,7 +55,7 @@ class OraTVIE(InfoExtractor):
                formats.append({
                    'url': http_template % q,
                    'format_id': q,
-                    'preference': preference(q),
+                    'quality': preference(q),
                })
            self._sort_formats(formats)
        else:
--- a/youtube_dlc/extractor/picarto.py
+++ b/youtube_dlc/extractor/picarto.py
@@ -78,7 +78,7 @@ class PicartoIE(InfoExtractor):
                        update_url_query(
                            'https://%s/hls/%s/index.m3u8'
                            % (edge_ep, channel_id), params),
-                        channel_id, 'mp4', preference=preference,
+                        channel_id, 'mp4', quality=preference,
                        m3u8_id='-'.join(format_id), fatal=False))
                    continue
                elif tech_type == 'video/mp4' or tech_label == 'MP4':
@@ -88,7 +88,7 @@ class PicartoIE(InfoExtractor):
                            'https://%s/mp4/%s.mp4' % (edge_ep, channel_id),
                            params),
                        'format_id': '-'.join(format_id),
-                        'preference': preference,
+                        'quality': preference,
                    })
                else:
                    # rtmp format does not seem to work
--- a/youtube_dlc/extractor/pinterest.py
+++ b/youtube_dlc/extractor/pinterest.py
@@ -54,8 +54,7 @@ class PinterestBaseIE(InfoExtractor):
                        'height': int_or_none(format_dict.get('height')),
                        'duration': duration,
                    })
-            self._sort_formats(
-                formats, field_preference=('height', 'width', 'tbr', 'format_id'))
+            self._sort_formats(formats)

        description = data.get('description') or data.get('description_html') or data.get('seo_description')
        timestamp = unified_timestamp(data.get('created_at'))
--- a/youtube_dlc/extractor/playtvak.py
+++ b/youtube_dlc/extractor/playtvak.py
@@ -150,7 +150,7 @@ class PlaytvakIE(InfoExtractor):
                ext = 'mp4'
                # Some streams have mp3 audio which does not play
                # well with ffmpeg filter aac_adtstoasc
-                preference = -1
+                preference = -10
            elif format_ == 'adobe':  # f4m manifest fails with 404 in 80% of requests
                continue
            else:  # Other formats not supported yet
--- a/youtube_dlc/extractor/prosiebensat1.py
+++ b/youtube_dlc/extractor/prosiebensat1.py
@@ -34,7 +34,7 @@ class ProSiebenSat1BaseIE(InfoExtractor):
                'ids': clip_id,
            })[0]

-        if video.get('is_protected') is True:
+        if not self._downloader.params.get('allow_unplayable_formats') and video.get('is_protected') is True:
            raise ExtractorError('This video is DRM protected.', expected=True)

        formats = []
--- a/youtube_dlc/extractor/qqmusic.py
+++ b/youtube_dlc/extractor/qqmusic.py
@@ -121,7 +121,7 @@ class QQMusicIE(InfoExtractor):
                       % (details['prefix'], mid, details['ext'], vkey, guid),
                'format': format_id,
                'format_id': format_id,
-                'preference': details['preference'],
+                'quality': details['preference'],
                'abr': details.get('abr'),
            })
        self._check_formats(formats, mid)
--- a/youtube_dlc/extractor/radiofrance.py
+++ b/youtube_dlc/extractor/radiofrance.py
@@ -43,7 +43,7 @@ class RadioFranceIE(InfoExtractor):
                'format_id': fm[0],
                'url': fm[1],
                'vcodec': 'none',
-                'preference': i,
+                'quality': i,
            }
            for i, fm in
            enumerate(re.findall(r"([a-z0-9]+)\s*:\s*'([^']+)'", formats_str))
--- a/youtube_dlc/extractor/reddit.py
+++ b/youtube_dlc/extractor/reddit.py
@@ -40,7 +40,7 @@ class RedditIE(InfoExtractor):
            'https://v.redd.it/%s/DASHPlaylist.mpd' % video_id, video_id,
            mpd_id='dash', fatal=False))

-        self._sort_formats(formats, ('height', 'width'))
+        self._sort_formats(formats)

        return {
            'id': video_id,
--- a/youtube_dlc/extractor/rtbf.py
+++ b/youtube_dlc/extractor/rtbf.py
@@ -125,7 +125,7 @@ class RTBFIE(InfoExtractor):
                })

        mpd_url = data.get('urlDash')
-        if not data.get('drm') and mpd_url:
+        if mpd_url and (self._downloader.params.get('allow_unplayable_formats') or not data.get('drm')):
            formats.extend(self._extract_mpd_formats(
                mpd_url, media_id, mpd_id='dash', fatal=False))

--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
pukkandan	4524baf056	Release 2021.02.19	2021-02-20 03:14:01 +05:30
pukkandan	bc2ca1bb75	Update to ytdl-commit-cf2dbec `cf2dbec630` Except: [kakao] improve info extraction and detect geo restriction `d8085580f6`	2021-02-20 02:32:22 +05:30
pukkandan	5e41dca334	[viki] Fix extractor (Closes #91 )	2021-02-19 18:21:29 +05:30
pukkandan	2a86f3da07	[build] Publish on PyPi only if token is set This allows forks to easily build releases :ci skip all	2021-02-19 17:04:25 +05:30
pukkandan	a40258a259	[documentation] Remove `--flat-videos` It does not work as documented It was an experimental option that I forgot to remove when making the fork public :ci skip all	2021-02-19 04:52:05 +05:30
pukkandan	ba7bf12d89	[youtube] Fix for empty comment text (Closes #97 )	2021-02-19 04:15:25 +05:30
pukkandan	f983b87567	[formatsort] Remove misuse of 'preference' 'preference' is to be used only when the format is better that ALL qualities of a lower preference irrespective of ANY sorting order the user requests. See deezer.py for correct use of this In the older sorting method, `preference`, `quality` and `language_preference` were functionally almost equivalent. So these disparities doesn't really matter there Also, despite what the documentation says, the default for `preference` was actually 0 and not -1. I have tried to correct this and also account for it when converting `preference` to `quality`	2021-02-19 03:33:45 +05:30
pukkandan	dca3ff4a5e	[formatsort] Remove forced priority of `quality` When making `FormatSort`, I misinterpreted the purpose `quality`	2021-02-19 00:12:21 +05:30
pukkandan	da9be05edf	[documentation] Better document `--prefer-free-formats` Also added `--no-prefer-free-formats`	2021-02-18 23:52:32 +05:30
pukkandan	155d2b48c5	[formatsort] Prefer vp9.2 over other vp9 codecs vp9.2 may contain HDR while vp9.0 doesn't	2021-02-18 23:52:31 +05:30
pukkandan	54f37eeabd	[formatsort] Remove unnecessary `field_preference` from extractors These were written with the old format sorting in mind and is no longer needed	2021-02-18 23:52:28 +05:30
pukkandan	9ba5705ac0	[youtube] Fix hashtag continuation Eg: https://www.youtube.com/hashtag/youtube :ci skip dl	2021-02-18 13:54:06 +05:30
pukkandan	c2934512c2	Option `--windows-filenames` to force use of windows compatible filenames * Also changed `--trim-file-name` to `--trim-filenames` to be similar to related options Related: https://web.archive.org/web/20210217190806/https://old.reddit.com/r/youtubedl/comments/llc4o5/do_you_guys_also_have_this_error :ci skip dl	2021-02-18 01:06:40 +05:30
shirt-dev	55e36f035c	#93 Build improvements * Lock all python package versions to the last officially supported releases for x86 * Bugfix for UNIX hash output * Use wheels to avoid compilation of python packages * Hash calculation on Windows now uses PowerShell rather than the legacy certutil Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com>	2021-02-17 11:40:39 +05:30
pukkandan	c86d5023d0	[youtube] Add more Invidious instances (Closes #92 ) :ci skip dl	2021-02-17 04:40:55 +05:30
pukkandan	42bb0c59f8	[MoveFiles] Fix when merger can't run :ci skip dl	2021-02-17 00:42:27 +05:30
pukkandan	c3e1f0c4f2	[contributors] update Forgot to do it when making release :ci skip dl	2021-02-16 17:09:54 +05:30
pukkandan	6b027907ce	Don't raise parser.error when exiting for update	2021-02-16 17:04:53 +05:30
pukkandan	f3b7c69377	[version] Set version number based on UTC time, not local time	2021-02-16 17:04:53 +05:30
Jody Bruchon	46261325be	#89 [pyinst.py] Exclude vcruntime140.dll from UPX (#89 ) Related: https://github.com/blackjack4494/yt-dlc/pull/182 (`7b400ac40b`) Authored by: jbruchon	2021-02-16 16:41:47 +05:30
kurumigi	78b9a616cc	#90 [niconico] Extract `channel` and `channel_id` (Closes #77 ) Authored by kurumigi	2021-02-16 16:19:37 +05:30
pukkandan	55b53b338b	[ExtractAudio] Bugfix for `1de75fa129` Fixes: #58 :ci skip dl	2021-02-16 15:00:54 +05:30
pukkandan	d16ab6ef1c	[version] update :ci skip dl	2021-02-16 04:17:55 +05:30
pukkandan	aa837ddf06	Release 2021.02.15	2021-02-16 04:04:27 +05:30
pukkandan	a718ef84c8	[youtube] Fix for new accounts Cookies for some new accounts doesn't work with age-gated videos without `has_verified=1`	2021-02-16 03:20:06 +05:30
shirt-dev	44f705d001	#88 Implement SHA256 checking for autoupdater * Also fix bugs from `e5813e53f0` Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> :ci skip dl	2021-02-16 02:36:42 +05:30
shirt-dev	47930b73a5	Fix build.yml hashing and crypto support (#87 ) Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com>	2021-02-16 00:46:23 +05:30
pukkandan	1de75fa129	[ExtractAudio] Don't re-encode when file is already in a common audio format (Closes #58 ) Fixes: https://github.com/blackjack4494/youtube-dlc/issues/214 Fixes: https://github.com/ytdl-org/youtube-dl/issues/28006	2021-02-15 23:22:11 +05:30
pukkandan	6285297795	[rumble] Add support for video page (Closes #80 )	2021-02-15 20:08:27 +05:30
pukkandan	e5813e53f0	Improve build/updater * Fix `get_executable_path` in UNIX * Update `x86.exe` correctly * Exit immediately in windows once the update process starts so that the file handle is released correctly * Show `exe`/`zip`/`source` and 32/64bit in verbose message * Look for both `yt-dlp` and `youtube-dlc` in releases. This ensures that the updater will keep working when the binary name is changed to yt-dlp * Disable pycryptodome in win_x86 since it causes `distutils.errors.DistutilsPlatformError: Microsoft Visual C++ 10.0 is required`	2021-02-15 15:41:40 +05:30
siikamiika	273762c8d0	#86 [youtube_live_chat] Use POST API (Closes #82 ) YouTube has removed support for the old GET based live chat API, and it's now returning 404 Authored by siikamiika	2021-02-15 15:27:21 +05:30
shirt-dev	7620cd46c3	#79 Fix HLS AES-128 with multiple keys in external downloaders Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com>	2021-02-13 21:45:41 +05:30
pukkandan	068693675e	Cleanup some code and fix typos :ci skip dl	2021-02-12 20:32:49 +05:30
pukkandan	1ea2412927	Minor bugfixes * `__real_download` should be false when ffmpeg unavailable and no download * Mistakes in #70 * `allow_playlist_files` was not correctly pass through	2021-02-12 20:29:29 +05:30
shirt-dev	63ad4d43eb	#70 Allow downloading of unplayable video formats Video postprocessors are also turned off when this option is used Co-authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> Co-authored-by: pukkandan <pukkandan@gmail.com>	2021-02-12 09:21:59 +05:30
pukkandan	584bab3766	[sponskrub] Print ffmpeg output and errors to terminal The ffmpeg run can be long when using `--sponskrub-cut`. So progress needs to be printed :ci skip dl	2021-02-12 01:40:08 +05:30
shirt-dev	fc2119f210	#76 Fix for empty HTTP head requests Related: https://github.com/ytdl-org/youtube-dl/issues/7181 Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> (shirt-dev)	2021-02-11 21:31:34 +05:30
shirt-dev	5d25607a3a	#75 Change optional dependency from `Crypto` to `pycryptodome` (Closes #74 ) Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> (shirt-dev) pycryptodome is an in-place replacement for Crypto and is more actively developed	2021-02-11 17:16:02 +05:30
pukkandan	a96c6d154a	[youtube] Fix search continuations	2021-02-11 17:10:38 +05:30
pukkandan	cc2db87805	Update to ytdl-2021.02.10 Except: [archiveorg] Fix and improve extraction (5fc53690cbe6abb11941a3f4846b566a7472753e)	2021-02-11 03:03:39 +05:30
shirt-dev	539d158c50	#72 Fix issue with unicode filenames in aria2c (Closes #71 ) Authored-by: shirtjs <2660574+shirtjs@users.noreply.github.com> (shirt-dev)	2021-02-11 02:27:18 +05:30
kurumigi	fb198a8a9c	#49 [niconico] Improved extraction and support encrypted/SMILE movies Co-authored-by: tsukumijima <tsukumijima@users.noreply.github.com> Co-authored-by: tsukumi <39271166+tsukumijima@users.noreply.github.com> Co-authored-by: Bepis <36346617+bbepis@users.noreply.github.com> Co-authored-by: pukkandan <pukkandan@gmail.com>	2021-02-10 12:15:20 +05:30
pukkandan	8d801631cf	[version] update :ci skip all	2021-02-10 02:06:49 +05:30