Release 2021.09.02

SponsorBlock options:
* The fetched sponsor sections are written to infojson
* `--sponsorblock-remove` removes specified chapters from file
* `--sponsorblock-mark` marks the specified sponsor sections as chapters
* `--sponsorblock-chapter-title` to specify sponsor chapter template
* `--sponsorblock-api` to use a different API

Related improvements:
* Split `--embed-chapters` from `--embed-metadata`
* Add `--remove-chapters` to remove arbitrary chapters
* Add `--force-keyframes-at-cuts` for more accurate cuts when removing and splitting chapters

Deprecates all `--sponskrub` options

Authored by: nihil-admirari, pukkandan

.github/FUNDING.yml

@@ -0,0 +1,13 @@
+# These are supported funding model platforms
+
+github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+
+custom: ['https://github.com/yt-dlp/yt-dlp/blob/master/Collaborators.md#collaborators']

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.08.10. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
+- [ ] I've verified that I'm running yt-dlp version **2021.08.10**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.07.21
+ [debug] yt-dlp version 2021.08.10
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.08.10. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,9 +29,10 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
+- [ ] I've verified that I'm running yt-dlp version **2021.08.10**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
+- [ ] The provided URLs do not contain any DRM to the best of my knowledge
 - [ ] I've searched the bugtracker for similar site support requests including closed ones
 
 

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.08.10. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
+- [ ] I've verified that I'm running yt-dlp version **2021.08.10**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.08.10. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,9 +29,10 @@ Carefully read and work through this check list in order to prevent the most com
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
-- [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
+- [ ] I'm reporting a bug unrelated to a specific site
+- [ ] I've verified that I'm running yt-dlp version **2021.08.10**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
+- [ ] The provided URLs do not contain any DRM to the best of my knowledge
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
 - [ ] I've read bugs section in FAQ
@@ -46,7 +47,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.07.21
+ [debug] yt-dlp version 2021.08.10
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.08.10. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
+- [ ] I've verified that I'm running yt-dlp version **2021.08.10**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/6_question.md

@@ -1,6 +1,6 @@
 ---
 name: Ask question
-about: Ask youtube-dl related question
+about: Ask yt-dlp related question
 title: "[Question]"
 labels: question
 assignees: ''

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md

@@ -32,6 +32,7 @@ Carefully read and work through this check list in order to prevent the most com
 - [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
+- [ ] The provided URLs do not contain any DRM to the best of my knowledge
 - [ ] I've searched the bugtracker for similar site support requests including closed ones
 
 

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md

@@ -29,9 +29,10 @@ Carefully read and work through this check list in order to prevent the most com
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
-- [ ] I'm reporting a broken site support issue
+- [ ] I'm reporting a bug unrelated to a specific site
 - [ ] I've verified that I'm running yt-dlp version **%(version)s**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
+- [ ] The provided URLs do not contain any DRM to the best of my knowledge
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
 - [ ] I've read bugs section in FAQ

.github/PULL_REQUEST_TEMPLATE.md

@@ -11,7 +11,7 @@
 - [ ] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
 - [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)
 
-### In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
+### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
 - [ ] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
 - [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)
 

.github/workflows/build.yml

@@ -103,7 +103,8 @@ jobs:
     - name: Upgrade pip and enable wheel support
       run: python -m pip install --upgrade pip setuptools wheel
     - name: Install Requirements
-      run: pip install pyinstaller mutagen pycryptodome websockets
+      # Custom pyinstaller built with https://github.com/yt-dlp/pyinstaller-builds
+      run: pip install "https://yt-dlp.github.io/pyinstaller-builds/x86_64/pyinstaller-4.5.1-py3-none-any.whl" mutagen pycryptodome websockets
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
@@ -147,7 +148,7 @@ jobs:
     - name: Upgrade pip and enable wheel support
       run: python -m pip install --upgrade pip setuptools wheel
     - name: Install Requirements
-      run: pip install pyinstaller mutagen pycryptodome websockets
+      run: pip install "https://yt-dlp.github.io/pyinstaller-builds/i686/pyinstaller-4.5.1-py3-none-any.whl" mutagen pycryptodome websockets
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py

.github/workflows/quick-test.yml

@@ -27,5 +27,7 @@ jobs:
         python-version: 3.9
     - name: Install flake8
       run: pip install flake8
+    - name: Make lazy extractors
+      run: python devscripts/make_lazy_extractors.py yt_dlp/extractor/lazy_extractors.py
     - name: Run flake8
       run: flake8 .

.gitignore

@@ -19,6 +19,8 @@ cookies.txt
 *.wav
 *.ape
 *.mkv
+*.flac
+*.avi
 *.swf
 *.part
 *.part-*

CONTRIBUTORS

@@ -1,6 +1,6 @@
 pukkandan (owner)
 shirt-dev (collaborator)
-colethedj (collaborator)
+coletdjnz/colethedj (collaborator)
 Ashish0804 (collaborator)
 h-h-h-h
 pauldubois98
@@ -22,7 +22,7 @@ Zocker1999NET
 nao20010128nao
 kurumigi
 bbepis
-animelover1984
+animelover1984/horahoradev
 Pccode66
 RobinD42
 hseg
@@ -63,3 +63,40 @@ TpmKranz
 mzbaulhaque
 zackmark29
 mbway
+zerodytrash
+wesnm
+pento
+rigstot
+dirkf
+funniray
+Jessecar96
+jhwgh1968
+kikuyan
+max-te
+nchilada
+pgaig
+PSlava
+stdedos
+u-spec-png
+Sipherdrakon
+kidonng
+smege1001
+tandy1000
+IONECarter
+capntrips
+mrfade
+ParadoxGBB
+wlritchi
+NeroBurner
+mahanstreamer
+alerikaisattera
+Derkades
+BunnyHelp
+i6t
+std-move
+Chocobozzz
+ouwou
+korli
+octotherp
+CeruleanSky
+zootedb0t

Changelog.md

@@ -19,11 +19,217 @@
 -->
 
 
+### 2021.09.02
+
+* **Native SponsorBlock** implementation by [nihil-admirari](https://github.com/nihil-admirari), [pukkandan](https://github.com/pukkandan)
+    * `--sponsorblock-remove CATS` removes specified chapters from file
+    * `--sponsorblock-mark CATS` marks the specified sponsor sections as chapters
+    * `--sponsorblock-chapter-title TMPL` to specify sponsor chapter template
+    * `--sponsorblock-api URL` to use a different API
+    * No re-encoding is done unless `--force-keyframes-at-cuts` is used
+    * The fetched sponsor sections are written to the infojson
+    * Deprecates: `--sponskrub`, `--no-sponskrub`, `--sponskrub-cut`, `--no-sponskrub-cut`, `--sponskrub-force`, `--no-sponskrub-force`, `--sponskrub-location`, `--sponskrub-args`
+* Split `--embed-chapters` from `--embed-metadata` (it still implies the former by default)
+* Add option `--remove-chapters` to remove arbitrary chapters by [nihil-admirari](https://github.com/nihil-admirari), pukkandan
+* Add option `--force-keyframes-at-cuts` for more accurate cuts when removing and splitting chapters by [nihil-admirari](https://github.com/nihil-admirari)
+* Let `--match-filter` reject entries early
+    * Makes redundant: `--match-title`, `--reject-title`, `--min-views`, `--max-views`
+* [lazy_extractor] Improvements (It now passes all tests)
+    * Bugfix for when plugin directory doesn't exist by [kidonng](https://github.com/kidonng)
+    * Create instance only after pre-checking archive
+    * Import actual class if an attribute is accessed
+    * Fix `suitable` and add flake8 test
+* [downloader/ffmpeg] Experimental support for DASH manifests (including live)
+    * Your ffmpeg must have [this patch](https://github.com/FFmpeg/FFmpeg/commit/3249c757aed678780e22e99a1a49f4672851bca9) applied for YouTube DASH to work
+* [downloader/ffmpeg] Allow passing custom arguments before `-i`
+
+* [BannedVideo] Add extractor by [smege1001](https://github.com/smege1001), [blackjack4494](https://github.com/blackjack4494), [pukkandan](https://github.com/pukkandan)
+* [bilibili] Add category extractor by [animelover1984](https://github.com/animelover1984)
+* [Epicon] Add extractors by [Ashish0804](https://github.com/Ashish0804)
+* [filmmodu] Add extractor by [mzbaulhaque](https://github.com/mzbaulhaque)
+* [GabTV] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Hungama] Fix `HungamaSongIE` and add `HungamaAlbumPlaylistIE` by [Ashish0804](https://github.com/Ashish0804)
+* [ManotoTV] Add new extractors by [tandy1000](https://github.com/tandy1000)
+* [Niconico] Add Search extractors by [animelover1984](https://github.com/animelover1984), [pukkandan](https://github.com/pukkandan)
+* [Patreon] Add `PatreonUserIE` by [zenerdi0de](https://github.com/zenerdi0de)
+* [peloton] Add extractor by [IONECarter](https://github.com/IONECarter), [capntrips](https://github.com/capntrips), [pukkandan](https://github.com/pukkandan)
+* [ProjectVeritas] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [radiko] Add extractors by [nao20010128nao](https://github.com/nao20010128nao)
+* [StarTV] Add extractor for `startv.com.tr` by [mrfade](https://github.com/mrfade), [coletdjnz](https://github.com/coletdjnz)
+* [tiktok] Add `TikTokUserIE` by [Ashish0804](https://github.com/Ashish0804), [pukkandan](https://github.com/pukkandan)
+* [Tokentube] Add extractor by [u-spec-png](https://github.com/u-spec-png)
+* [TV2Hu] Fix `TV2HuIE` and add `TV2HuSeriesIE` by [Ashish0804](https://github.com/Ashish0804)
+* [voicy] Add extractor by [nao20010128nao](https://github.com/nao20010128nao)
+
+* [adobepass] Fix Verizon SAML login by [nyuszika7h](https://github.com/nyuszika7h), [ParadoxGBB](https://github.com/ParadoxGBB)
+* [afreecatv] Fix adult VODs by [wlritchi](https://github.com/wlritchi)
+* [afreecatv] Tolerate failure to parse date string by [wlritchi](https://github.com/wlritchi)
+* [aljazeera] Fix extractor by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [ATV.at] Fix extractor for ATV.at by [NeroBurner](https://github.com/NeroBurner), [coletdjnz](https://github.com/coletdjnz)
+* [bitchute] Fix test by [mahanstreamer](https://github.com/mahanstreamer)
+* [camtube] Remove obsolete extractor by [alerikaisattera](https://github.com/alerikaisattera)
+* [CDA] Add more formats by [u-spec-png](https://github.com/u-spec-png)
+* [eroprofile] Fix page skipping in albums by [jhwgh1968](https://github.com/jhwgh1968)
+* [facebook] Fix format sorting
+* [facebook] Fix metadata extraction by [kikuyan](https://github.com/kikuyan)
+* [facebook] Update onion URL by [Derkades](https://github.com/Derkades)
+* [HearThisAtIE] Fix extractor by [Ashish0804](https://github.com/Ashish0804)
+* [instagram] Add referrer to prevent throttling by [u-spec-png](https://github.com/u-spec-png), [kikuyan](https://github.com/kikuyan)
+* [iwara.tv] Extract more metadata by [BunnyHelp](https://github.com/BunnyHelp)
+* [iwara] Add thumbnail by [i6t](https://github.com/i6t)
+* [kakao] Fix extractor
+* [mediaset] Fix extraction for some videos by [nyuszika7h](https://github.com/nyuszika7h)
+* [Motherless] Fix extractor by [coletdjnz](https://github.com/coletdjnz)
+* [Nova] fix extractor by [std-move](https://github.com/std-move)
+* [ParamountPlus] Fix geo verification by [shirt](https://github.com/shirt-dev)
+* [peertube] handle new video URL format by [Chocobozzz](https://github.com/Chocobozzz)
+* [pornhub] Separate and fix playlist extractor by [mzbaulhaque](https://github.com/mzbaulhaque)
+* [reddit] Fix for quarantined subreddits by [ouwou](https://github.com/ouwou)
+* [ShemarooMe] Fix extractor by [Ashish0804](https://github.com/Ashish0804)
+* [soundcloud] Refetch `client_id` on 403
+* [tiktok] Fix metadata extraction
+* [TV2] Fix extractor by [Ashish0804](https://github.com/Ashish0804)
+* [tv5mondeplus] Fix extractor by [korli](https://github.com/korli)
+* [VH1,TVLand] Fix extractors by [Sipherdrakon](https://github.com/Sipherdrakon)
+* [Viafree] Fix extractor and extract subtitles by [coletdjnz](https://github.com/coletdjnz)
+* [XHamster] Extract `uploader_id` by [octotherp](https://github.com/octotherp)
+* [youtube] Add `shorts` to `_VALID_URL`
+* [youtube] Add av01 itags to known formats list by [blackjack4494](https://github.com/blackjack4494)
+* [youtube] Extract error messages from HTTPError response by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Fix subtitle names
+* [youtube] Prefer audio stream that YouTube considers default
+* [youtube] Remove annotations and deprecate `--write-annotations` by [coletdjnz](https://github.com/coletdjnz)
+* [Zee5] Fix extractor and add subtitles by [Ashish0804](https://github.com/Ashish0804)
+
+* [aria2c] Obey `--rate-limit`
+* [EmbedSubtitle] Continue even if some files are missing
+* [extractor] Better error message for DRM
+* [extractor] Common function `_match_valid_url`
+* [extractor] Show video id in error messages if possible
+* [FormatSort] Remove priority of `lang`
+* [options] Add `_set_from_options_callback`
+* [SubtitleConvertor] Fix bug during subtitle conversion
+* [utils] Add `parse_qs`
+* [webvtt] Fix timestamp overflow adjustment by [fstirlitz](https://github.com/fstirlitz)
+* Bugfix for `--replace-in-metadata`
+* Don't try to merge with final extension
+* Fix `--force-overwrites` when using `-k`
+* Fix `--no-prefer-free-formats` by [CeruleanSky](https://github.com/CeruleanSky)
+* Fix `-F` for extractors that directly return url
+* Fix `-J` when there are failed videos
+* Fix `extra_info` being reused across runs
+* Fix `playlist_index` not obeying `playlist_start` and add tests
+* Fix resuming of single formats when using `--no-part`
+* Revert erroneous use of the `Content-Length` header by [fstirlitz](https://github.com/fstirlitz)
+* Use `os.replace` where applicable by; paulwrubel
+* [build] Add homebrew taps `yt-dlp/taps/yt-dlp` by [nao20010128nao](https://github.com/nao20010128nao)
+* [build] Fix bug in making `yt-dlp.tar.gz`
+* [docs] Fix some typos by [pukkandan](https://github.com/pukkandan), [zootedb0t](https://github.com/zootedb0t)
+* [cleanup] Replace improper use of tab in trovo by [glenn-slayden](https://github.com/glenn-slayden)
+
+
+### 2021.08.10
+
+* Add option `--replace-in-metadata`
+* Add option `--no-simulate` to not simulate even when `--print` or `--list...` are used - Deprecates `--print-json`
+* Allow entire infodict to be printed using `%()s` - makes `--dump-json` redundant
+* Allow multiple `--exec` and `--exec-before-download`
+* Add regex to `--match-filter`
+* Add all format filtering operators also to `--match-filter` by [max-te](https://github.com/max-te)
+* Add compat-option `no-keep-subs`
+* [adobepass] Add MSO Cablevision by [Jessecar96](https://github.com/Jessecar96)
+* [BandCamp] Add BandcampMusicIE by [Ashish0804](https://github.com/Ashish0804)
+* [blackboardcollaborate] Add new extractor by [mzbaulhaque](https://github.com/mzbaulhaque)
+* [eroprofile] Add album downloader by [jhwgh1968](https://github.com/jhwgh1968)
+* [mirrativ] Add extractors by [nao20010128nao](https://github.com/nao20010128nao)
+* [openrec] Add extractors by [nao20010128nao](https://github.com/nao20010128nao)
+* [nbcolympics:stream] Fix extractor by [nchilada](https://github.com/nchilada), [pukkandan](https://github.com/pukkandan)
+* [nbcolympics] Update extractor for 2020 olympics by [wesnm](https://github.com/wesnm)
+* [paramountplus] Separate extractor and fix some titles by [shirt](https://github.com/shirt-dev), [pukkandan](https://github.com/pukkandan)
+* [RCTIPlus] Support events and TV by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [Newgrounds] Improve extractor and fix playlist by [u-spec-png](https://github.com/u-spec-png)
+* [aenetworks] Update `_THEPLATFORM_KEY` and `_THEPLATFORM_SECRET` by [wesnm](https://github.com/wesnm)
+* [crunchyroll] Fix thumbnail by [funniray](https://github.com/funniray)
+* [HotStar] Use API for metadata and extract subtitles by [Ashish0804](https://github.com/Ashish0804)
+* [instagram] Fix comments extraction by [u-spec-png](https://github.com/u-spec-png)
+* [peertube] Fix videos without description by [u-spec-png](https://github.com/u-spec-png)
+* [twitch:clips] Extract `display_id` by [dirkf](https://github.com/dirkf)
+* [viki] Print error message from API request
+* [Vine] Remove invalid formats by [u-spec-png](https://github.com/u-spec-png)
+* [VrtNU] Fix XSRF token by [pgaig](https://github.com/pgaig)
+* [vrv] Fix thumbnail extraction by [funniray](https://github.com/funniray)
+* [youtube] Add extractor-arg `include-live-dash` to show live dash formats
+* [youtube] Improve signature function detection by [PSlava](https://github.com/PSlava)
+* [youtube] Raise appropriate error when API pages can't be downloaded
+* Ensure `_write_ytdl_file` closes file handle on error
+* Fix `--compat-options filename` by [stdedos](https://github.com/stdedos)
+* Fix issues with infodict sanitization
+* Fix resuming when using `--no-part`
+* Fix wrong extension for intermediate files
+* Handle `BrokenPipeError` by [kikuyan](https://github.com/kikuyan)
+* Show libraries present in verbose head
+* [extractor] Detect `sttp` as subtitles in MPD by [fstirlitz](https://github.com/fstirlitz)
+* [extractor] Reset non-repeating warnings per video
+* [ffmpeg] Fix streaming `mp4` to `stdout`
+* [ffpmeg] Allow `--ffmpeg-location` to be a file with different name
+* [utils] Fix `InAdvancePagedList.__getitem__`
+* [utils] Fix `traverse_obj` depth when `is_user_input`
+* [webvtt] Merge daisy-chained duplicate cues by [fstirlitz](https://github.com/fstirlitz)
+* [build] Use custom build of `pyinstaller` by [shirt](https://github.com/shirt-dev)
+* [tests:download] Add batch testing for extractors (`test_YourExtractor_all`)
+* [docs] Document which fields `--add-metadata` adds to the file
+* [docs] Fix some mistakes and improve doc
+* [cleanup] Misc code cleanup
+
+
+### 2021.08.02
+
+* Add logo, banner and donate links
+* Expand and escape environment variables correctly in output template
+* Add format types `j` (json), `l` (comma delimited list), `q` (quoted for terminal) in output template
+* [downloader] Allow streaming some unmerged formats to stdout using ffmpeg
+* [youtube] **Age-gate bypass**
+    * Add `agegate` clients by [pukkandan](https://github.com/pukkandan), [MinePlayersPE](https://github.com/MinePlayersPE)
+    * Add `thirdParty` to agegate clients to bypass more videos
+    * Simplify client definitions, expose `embedded` clients
+    * Improve age-gate detection by [coletdjnz](https://github.com/coletdjnz)
+    * Fix default global API key by [coletdjnz](https://github.com/coletdjnz)
+    * Add `creator` clients for age-gate bypass using unverified accounts by [zerodytrash](https://github.com/zerodytrash), [coletdjnz](https://github.com/coletdjnz), [pukkandan](https://github.com/pukkandan)
+* [adobepass] Add MSO Sling TV by [wesnm](https://github.com/wesnm)
+* [CBS] Add ParamountPlusSeriesIE by [Ashish0804](https://github.com/Ashish0804)
+* [dplay] Add `ScienceChannelIE` by [Sipherdrakon](https://github.com/Sipherdrakon)
+* [UtreonIE] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [youtube] Add `mweb` client by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Add `player_client=all`
+* [youtube] Force `hl=en` for comments by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Fix format sorting when using alternate clients
+* [youtube] Misc cleanup by [pukkandan](https://github.com/pukkandan), [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Extract SAPISID only once
+* [CBS] Add fallback by [llacb47](https://github.com/llacb47), [pukkandan](https://github.com/pukkandan)
+* [Hotstar] Support cookies by [Ashish0804](https://github.com/Ashish0804)
+* [HotStarSeriesIE] Fix regex by [Ashish0804](https://github.com/Ashish0804)
+* [bilibili] Improve `_VALID_URL`
+* [mediaset] Fix extraction by [nixxo](https://github.com/nixxo)
+* [Mxplayer] Add h265 formats by [Ashish0804](https://github.com/Ashish0804)
+* [RCTIPlus] Remove PhantomJS dependency by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [tenplay] Add MA15+ age limit by [pento](https://github.com/pento)
+* [vidio] Fix login error detection by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [vimeo] Better extraction of original file by [Ashish0804](https://github.com/Ashish0804)
+* [generic] Support KVS player (replaces ThisVidIE) by [rigstot](https://github.com/rigstot)
+* Add compat-option `no-clean-infojson`
+* Remove `asr` appearing twice in `-F`
+* Set `home:` as the default key for `-P`
+* [utils] Fix slicing of reversed `LazyList`
+* [FormatSort] Fix bug for audio with unknown codec
+* [test:download] Support testing with `ignore_no_formats_error`
+* [cleanup] Refactor some code
+
+
 ### 2021.07.24
 
 * [youtube:tab] Extract video duration early
 * [downloader] Pass `info_dict` to `progress_hook`s
-* [youtube] Fix age-gated videos for API clients when cookies are supplied by [colethedj](https://github.com/colethedj)
+* [youtube] Fix age-gated videos for API clients when cookies are supplied by [coletdjnz](https://github.com/coletdjnz)
 * [youtube] Disable `get_video_info` age-gate workaround - This endpoint seems to be completely dead
 * [youtube] Try all clients even if age-gated
 * [youtube] Fix subtitles only being extracted from the first client
@@ -43,7 +249,7 @@
 * [FFmpegMetadata] Add language of each stream and some refactoring
 * [douyin] Add extractor by [pukkandan](https://github.com/pukkandan), [pyx](https://github.com/pyx)
 * [pornflip] Add extractor by [mzbaulhaque](https://github.com/mzbaulhaque)
-* **[youtube] Extract data from multiple clients** by [pukkandan](https://github.com/pukkandan), [colethedj](https://github.com/colethedj)
+* **[youtube] Extract data from multiple clients** by [pukkandan](https://github.com/pukkandan), [coletdjnz](https://github.com/coletdjnz)
     * `player_client` now accepts multiple clients
     * Default `player_client` = `android,web`
         * This uses twice as many requests, but avoids throttling for most videos while also not losing any formats
@@ -56,19 +262,19 @@
 * [youtube] Misc fixes
     * Improve extraction of livestream metadata by [pukkandan](https://github.com/pukkandan), [krichbanana](https://github.com/krichbanana)
     * Hide live dash formats since they can't be downloaded anyway
-    * Fix authentication when using multiple accounts by [colethedj](https://github.com/colethedj)
-    * Fix controversial videos when requested via API by [colethedj](https://github.com/colethedj)
-    * Fix session index extraction and headers for non-web player clients by [colethedj](https://github.com/colethedj)
+    * Fix authentication when using multiple accounts by [coletdjnz](https://github.com/coletdjnz)
+    * Fix controversial videos when requested via API by [coletdjnz](https://github.com/coletdjnz)
+    * Fix session index extraction and headers for non-web player clients by [coletdjnz](https://github.com/coletdjnz)
     * Make `--extractor-retries` work for more errors
     * Fix sorting of 3gp format
     * Sanity check `chapters` (and refactor related code)
     * Make `parse_time_text` and `_extract_chapters` non-fatal
-    * Misc cleanup and bug fixes by [colethedj](https://github.com/colethedj)
+    * Misc cleanup and bug fixes by [coletdjnz](https://github.com/coletdjnz)
 * [youtube:tab] Fix channels tab
-* [youtube:tab] Extract playlist availability by [colethedj](https://github.com/colethedj)
-* **[youtube:comments] Move comment extraction to new API** by [colethedj](https://github.com/colethedj)
+* [youtube:tab] Extract playlist availability by [coletdjnz](https://github.com/coletdjnz)
+* **[youtube:comments] Move comment extraction to new API** by [coletdjnz](https://github.com/coletdjnz)
     * Adds extractor-args `comment_sort` (`top`/`new`), `max_comments`, `max_comment_depth`
-* [youtube:comments] Fix `is_favorited`, improve `like_count` parsing by [colethedj](https://github.com/colethedj)
+* [youtube:comments] Fix `is_favorited`, improve `like_count` parsing by [coletdjnz](https://github.com/coletdjnz)
 * [BravoTV] Improve metadata extraction by [kevinoconnor7](https://github.com/kevinoconnor7)
 * [crunchyroll:playlist] Force http
 * [yahoo:gyao:player] Relax `_VALID_URL` by [nao20010128nao](https://github.com/nao20010128nao)
@@ -94,7 +300,7 @@
 * [utils] Improve `js_to_json` comment regex by [fstirlitz](https://github.com/fstirlitz)
 * [webtt] Fix timestamps
 * [compat] Remove unnecessary code
-* [doc] fix default of multistreams
+* [docs] fix default of multistreams
 
 
 ### 2021.07.07
@@ -104,7 +310,7 @@
     * Add extractor option `skip` for `youtube`. Eg: `--extractor-args youtube:skip=hls,dash`
     * Deprecates `--youtube-skip-dash-manifest`, `--youtube-skip-hls-manifest`, `--youtube-include-dash-manifest`, `--youtube-include-hls-manifest`
 * Allow `--list...` options to work with `--print`, `--quiet` and other `--list...` options
-* [youtube] Use `player` API for additional video extraction requests by [colethedj](https://github.com/colethedj)
+* [youtube] Use `player` API for additional video extraction requests by [coletdjnz](https://github.com/coletdjnz)
     * **Fixes youtube premium music** (format 141) extraction
     * Adds extractor option `player_client` = `web`/`android`
         * **`--extractor-args youtube:player_client=android` works around the throttling** for the time-being
@@ -112,7 +318,7 @@
     * Adds age-gate fallback using embedded client
 * [youtube] Choose correct Live chat API for upcoming streams by [krichbanana](https://github.com/krichbanana)
 * [youtube] Fix subtitle names for age-gated videos
-* [youtube:comments] Fix error handling and add `itct` to params by [colethedj](https://github.com/colethedj)
+* [youtube:comments] Fix error handling and add `itct` to params by [coletdjnz](https://github.com/coletdjnz)
 * [youtube_live_chat] Fix download with cookies by [siikamiika](https://github.com/siikamiika)
 * [youtube_live_chat] use `clickTrackingParams` by [siikamiika](https://github.com/siikamiika)
 * [Funimation] Rewrite extractor
@@ -159,9 +365,9 @@
 * [downloader/mhtml] Add new downloader for slideshows/storyboards by [fstirlitz](https://github.com/fstirlitz)
 * [youtube] Temporary **fix for age-gate**
 * [youtube] Support ongoing live chat by [siikamiika](https://github.com/siikamiika)
-* [youtube] Improve SAPISID cookie handling by [colethedj](https://github.com/colethedj)
+* [youtube] Improve SAPISID cookie handling by [coletdjnz](https://github.com/coletdjnz)
 * [youtube] Login is not needed for `:ytrec`
-* [youtube] Non-fatal alert reporting for unavailable videos page by [colethedj](https://github.com/colethedj)
+* [youtube] Non-fatal alert reporting for unavailable videos page by [coletdjnz](https://github.com/coletdjnz)
 * [twitcasting] Websocket support by [nao20010128nao](https://github.com/nao20010128nao)
 * [mediasite] Extract slides by [fstirlitz](https://github.com/fstirlitz)
 * [funimation] Extract subtitles 
@@ -219,7 +425,7 @@
 * Merge youtube-dl: Upto [commit/d495292](https://github.com/ytdl-org/youtube-dl/commit/d495292852b6c2f1bd58bc2141ff2b0265c952cf)
 * Pre-check archive and filters during playlist extraction
 * Handle Basic Auth `user:pass` in URLs by [hhirtz](https://github.com/hhirtz) and [pukkandan](https://github.com/pukkandan)
-* [archiveorg] Add YoutubeWebArchiveIE by [colethedj](https://github.com/colethedj) and [alex-gedeon](https://github.com/alex-gedeon)
+* [archiveorg] Add YoutubeWebArchiveIE by [coletdjnz](https://github.com/coletdjnz) and [alex-gedeon](https://github.com/alex-gedeon)
 * [fancode] Add extractor by [rhsmachine](https://github.com/rhsmachine)
 * [patreon] Support vimeo embeds by [rhsmachine](https://github.com/rhsmachine)
 * [Saitosan] Add new extractor by [llacb47](https://github.com/llacb47)
@@ -262,7 +468,7 @@
 
 * **Youtube improvements**: 
     * Support youtube music `MP`, `VL` and `browse` pages
-    * Extract more formats for youtube music by [craftingmod](https://github.com/craftingmod), [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
+    * Extract more formats for youtube music by [craftingmod](https://github.com/craftingmod), [coletdjnz](https://github.com/coletdjnz) and [pukkandan](https://github.com/pukkandan)
     * Extract multiple subtitles in same language by [pukkandan](https://github.com/pukkandan) and [tpikonen](https://github.com/tpikonen)
     * Redirect channels that doesn't have a `videos` tab to their `UU` playlists
     * Support in-channel search
@@ -271,10 +477,10 @@
     * Extract audio language
     * Add subtitle language names by [nixxo](https://github.com/nixxo) and [tpikonen](https://github.com/tpikonen)
     * Show alerts only from the final webpage
-    * Add `html5=1` param to `get_video_info` page requests by [colethedj](https://github.com/colethedj)
+    * Add `html5=1` param to `get_video_info` page requests by [coletdjnz](https://github.com/coletdjnz)
     * Better message when login required
 * **Add option `--print`**: to print any field/template
-    * Deprecates: `--get-description`, `--get-duration`, `--get-filename`, `--get-format`, `--get-id`, `--get-thumbnail`, `--get-title`, `--get-url`
+    * Makes redundant: `--get-description`, `--get-duration`, `--get-filename`, `--get-format`, `--get-id`, `--get-thumbnail`, `--get-title`, `--get-url`
 * Field `additional_urls` to download additional videos from metadata using [`--parse-metadata`](https://github.com/yt-dlp/yt-dlp#modifying-metadata)
 * Merge youtube-dl: Upto [commit/dfbbe29](https://github.com/ytdl-org/youtube-dl/commit/dfbbe2902fc67f0f93ee47a8077c148055c67a9b)
 * Write thumbnail of playlist and add `pl_thumbnail` outtmpl key
@@ -368,11 +574,11 @@
 * [TubiTv] Add TubiTvShowIE by [Ashish0804](https://github.com/Ashish0804)
 * [twitcasting] Fix extractor
 * [viu:ott] Fix extractor and support series by [lkho](https://github.com/lkho) and [pukkandan](https://github.com/pukkandan)
-* [youtube:tab] Show unavailable videos in playlists by [colethedj](https://github.com/colethedj)
+* [youtube:tab] Show unavailable videos in playlists by [coletdjnz](https://github.com/coletdjnz)
 * [youtube:tab] Reload with unavailable videos for all playlists
 * [youtube] Ignore invalid stretch ratio
-* [youtube] Improve channel syncid extraction to support ytcfg by [colethedj](https://github.com/colethedj)
-* [youtube] Standardize API calls for tabs, mixes and search by [colethedj](https://github.com/colethedj)
+* [youtube] Improve channel syncid extraction to support ytcfg by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Standardize API calls for tabs, mixes and search by [coletdjnz](https://github.com/coletdjnz)
 * [youtube] Bugfix in `_extract_ytcfg`
 * [mildom:user:vod] Download only necessary amount of pages
 * [mildom] Remove proxy completely by [fstirlitz](https://github.com/fstirlitz)
@@ -384,8 +590,8 @@
 * Improve the yt-dlp.sh script by [fstirlitz](https://github.com/fstirlitz)
 * [lazy_extractor] Do not load plugins
 * [ci] Disable fail-fast
-* [documentation] Clarify which deprecated options still work
-* [documentation] Fix typos
+* [docs] Clarify which deprecated options still work
+* [docs] Fix typos
 
 
 ### 2021.04.11
@@ -402,17 +608,17 @@
 * [nitter] Fix extraction of reply tweets and update instance list by [B0pol](https://github.com/B0pol)
 * [nitter] Fix thumbnails by [B0pol](https://github.com/B0pol)
 * [youtube] Fix thumbnail URL
-* [youtube] Parse API parameters from initial webpage by [colethedj](https://github.com/colethedj)
-* [youtube] Extract comments' approximate timestamp by [colethedj](https://github.com/colethedj)
+* [youtube] Parse API parameters from initial webpage by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Extract comments' approximate timestamp by [coletdjnz](https://github.com/coletdjnz)
 * [youtube] Fix alert extraction
 * [bilibili] Fix uploader
-* [utils] Add `datetime_from_str` and `datetime_add_months` by [colethedj](https://github.com/colethedj)
+* [utils] Add `datetime_from_str` and `datetime_add_months` by [coletdjnz](https://github.com/coletdjnz)
 * Run some `postprocessors` before actual download
 * Improve argument parsing for `-P`, `-o`, `-S`
 * Fix some `m3u8` not obeying `--allow-unplayable-formats`
 * Fix default of `dynamic_mpd`
 * Deprecate `--all-formats`, `--include-ads`, `--hls-prefer-native`, `--hls-prefer-ffmpeg`
-* [documentation] Improvements
+* [docs] Improvements
 
 ### 2021.04.03
 * Merge youtube-dl: Upto [commit/654b4f4](https://github.com/ytdl-org/youtube-dl/commit/654b4f4ff2718f38b3182c1188c5d569c14cc70a)
@@ -423,10 +629,10 @@
 * [mildom] Update extractor with current proxy by [nao20010128nao](https://github.com/nao20010128nao)
 * [ard:mediathek] Fix video id extraction
 * [generic] Detect Invidious' link element
-* [youtube] Show premium state in `availability` by [colethedj](https://github.com/colethedj)
+* [youtube] Show premium state in `availability` by [coletdjnz](https://github.com/coletdjnz)
 * [viewsource] Add extractor to handle `view-source:`
 * [sponskrub] Run before embedding thumbnail
-* [documentation] Improve `--parse-metadata` documentation
+* [docs] Improve `--parse-metadata` documentation
 
 
 ### 2021.03.24.1
@@ -458,8 +664,8 @@
 * Use headers and cookies when downloading subtitles by [damianoamatruda](https://github.com/damianoamatruda)
 * Parse resolution in info dictionary by [damianoamatruda](https://github.com/damianoamatruda)
 * More consistent warning messages by [damianoamatruda](https://github.com/damianoamatruda) and [pukkandan](https://github.com/pukkandan)
-* [documentation] Add deprecated options and aliases in readme
-* [documentation] Fix some minor mistakes
+* [docs] Add deprecated options and aliases in readme
+* [docs] Fix some minor mistakes
 
 * [niconico] Partial fix adapted from [animelover1984/youtube-dl@b5eff52](https://github.com/animelover1984/youtube-dl/commit/b5eff52dd9ed5565672ea1694b38c9296db3fade) (login and smile formats still don't work)
 * [niconico] Add user extractor by [animelover1984](https://github.com/animelover1984)
@@ -468,7 +674,7 @@
 * [stitcher] Merge from youtube-dl by [nixxo](https://github.com/nixxo)
 * [rcs] Improved extraction by [nixxo](https://github.com/nixxo)
 * [linuxacadamy] Improve regex
-* [youtube] Show if video is `private`, `unlisted` etc in info (`availability`) by [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
+* [youtube] Show if video is `private`, `unlisted` etc in info (`availability`) by [coletdjnz](https://github.com/coletdjnz) and [pukkandan](https://github.com/pukkandan)
 * [youtube] bugfix for channel playlist extraction
 * [nbc] Improve metadata extraction by [2ShedsJackson](https://github.com/2ShedsJackson)
 
@@ -485,15 +691,15 @@
 * [wimtv] Add extractor by [nixxo](https://github.com/nixxo)
 * [mtv] Add mtv.it and extract series metadata by [nixxo](https://github.com/nixxo)
 * [pluto.tv] Add extractor by [kevinoconnor7](https://github.com/kevinoconnor7)
-* [youtube] Rewrite comment extraction by [colethedj](https://github.com/colethedj)
+* [youtube] Rewrite comment extraction by [coletdjnz](https://github.com/coletdjnz)
 * [embedthumbnail] Set mtime correctly
 * Refactor some postprocessor/downloader code by [pukkandan](https://github.com/pukkandan) and [shirt](https://github.com/shirt-dev)
 
 
 ### 2021.03.07
-* [youtube] Fix history, mixes, community pages and trending by [pukkandan](https://github.com/pukkandan) and [colethedj](https://github.com/colethedj)
-* [youtube] Fix private feeds/playlists on multi-channel accounts by [colethedj](https://github.com/colethedj)
-* [youtube] Extract alerts from continuation by [colethedj](https://github.com/colethedj)
+* [youtube] Fix history, mixes, community pages and trending by [pukkandan](https://github.com/pukkandan) and [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Fix private feeds/playlists on multi-channel accounts by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Extract alerts from continuation by [coletdjnz](https://github.com/coletdjnz)
 * [cbs] Add support for ParamountPlus by [shirt](https://github.com/shirt-dev)
 * [mxplayer] Rewrite extractor with show support by [pukkandan](https://github.com/pukkandan) and [Ashish0804](https://github.com/Ashish0804)
 * [gedi] Improvements from youtube-dl by [nixxo](https://github.com/nixxo)
@@ -505,7 +711,7 @@
 * [downloader] Fix bug for `ffmpeg`/`httpie`
 * [update] Fix updater removing the executable bit on some UNIX distros
 * [update] Fix current build hash for UNIX
-* [documentation] Include wget/curl/aria2c install instructions for Unix by [Ashish0804](https://github.com/Ashish0804)
+* [docs] Include wget/curl/aria2c install instructions for Unix by [Ashish0804](https://github.com/Ashish0804)
 * Fix some videos downloading with `m3u8` extension
 * Remove "fixup is ignored" warning when fixup wasn't passed by user
 
@@ -514,7 +720,7 @@
 * [build] Fix bug
 
 ### 2021.03.03
-* [youtube] Use new browse API for continuation page extraction by [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
+* [youtube] Use new browse API for continuation page extraction by [coletdjnz](https://github.com/coletdjnz) and [pukkandan](https://github.com/pukkandan)
 * Fix HLS playlist downloading by [shirt](https://github.com/shirt-dev)
 * Merge youtube-dl: Upto [2021.03.03](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.03.03)
 * [mtv] Fix extractor
@@ -562,7 +768,7 @@
 * [ffmpeg] Allow passing custom arguments before -i using `--ppa "ffmpeg_i1:ARGS"` syntax
 * Fix `--windows-filenames` removing `/` from UNIX paths
 * [hls] Show warning if pycryptodome is not found
-* [documentation] Improvements
+* [docs] Improvements
     * Fix documentation of `Extractor Options`
     * Document `all` in format selection
     * Document `playable_in_embed` in output templates
@@ -590,7 +796,7 @@
     * Exclude `vcruntime140.dll` from UPX by [jbruchon](https://github.com/jbruchon)
     * Set version number based on UTC time, not local time
     * Publish on PyPi only if token is set
-* [documentation] Better document `--prefer-free-formats` and add `--no-prefer-free-format`
+* [docs] Better document `--prefer-free-formats` and add `--no-prefer-free-format`
 
 
 ### 2021.02.15
@@ -633,7 +839,7 @@
 * [movefiles] Fix compatibility with python2
 * [remuxvideo] Fix validation of conditional remux
 * [sponskrub] Don't raise error when the video does not exist
-* [documentation] Crypto is an optional dependency
+* [docs] Crypto is an optional dependency
 
 
 ### 2021.02.04
@@ -694,10 +900,10 @@
 * Merge youtube-dl: Upto [2021.01.24](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
 * Plugin support ([documentation](https://github.com/yt-dlp/yt-dlp#plugins))
 * **Multiple paths**: New option `-P`/`--paths` to give different paths for different types of files
-    * The syntax is `-P "type:path" -P "type:path"` ([documentation](https://github.com/yt-dlp/yt-dlp#:~:text=-P,%20--paths%20TYPE:PATH))
+    * The syntax is `-P "type:path" -P "type:path"`
     * Valid types are: home, temp, description, annotation, subtitle, infojson, thumbnail
-    * Additionally, configuration file is taken from home directory or current directory ([documentation](https://github.com/yt-dlp/yt-dlp#:~:text=Home%20Configuration))
-* Allow passing different arguments to different external downloaders ([documentation](https://github.com/yt-dlp/yt-dlp#:~:text=--downloader-args%20NAME:ARGS))
+    * Additionally, configuration file is taken from home directory or current directory
+* Allow passing different arguments to different external downloaders
 * [mildom] Add extractor by [nao20010128nao](https://github.com/nao20010128nao)
 * Warn when using old style `--external-downloader-args` and `--post-processor-args`
 * Fix `--no-overwrite` when using `--write-link`
@@ -732,9 +938,9 @@
 * [roosterteeth.com] Fix for bonus episodes by [Zocker1999NET](https://github.com/Zocker1999NET)
 * [tiktok] Fix for when share_info is empty
 * [EmbedThumbnail] Fix bug due to incorrect function name
-* [documentation] Changed sponskrub links to point to [yt-dlp/SponSkrub](https://github.com/yt-dlp/SponSkrub) since I am now providing both linux and windows releases
-* [documentation] Change all links to correctly point to new fork URL
-* [documentation] Fixes typos
+* [docs] Changed sponskrub links to point to [yt-dlp/SponSkrub](https://github.com/yt-dlp/SponSkrub) since I am now providing both linux and windows releases
+* [docs] Change all links to correctly point to new fork URL
+* [docs] Fixes typos
 
 
 ### 2021.01.12
@@ -830,7 +1036,7 @@
     * Redirect channel home to /video
     * Print youtube's warning message
     * Handle Multiple pages for feeds better
-* [youtube] Fix ytsearch not returning results sometimes due to promoted content by [colethedj](https://github.com/colethedj)
+* [youtube] Fix ytsearch not returning results sometimes due to promoted content by [coletdjnz](https://github.com/coletdjnz)
 * [youtube] Temporary fix for automatic captions - disable json3 by [blackjack4494](https://github.com/blackjack4494)
 * Add --break-on-existing by [gergesh](https://github.com/gergesh)
 * Pre-check video IDs in the archive before downloading by [pukkandan](https://github.com/pukkandan)

Collaborators.md

@@ -0,0 +1,39 @@
+# Collaborators
+
+This is a list of the collaborators of the project and their major contributions. See the [Changelog](Changelog.md) for more details.
+
+You can also find lists of all [contributors of yt-dlp](CONTRIBUTORS) and [authors of youtube-dl](https://github.com/ytdl-org/youtube-dl/blob/master/AUTHORS)
+
+
+## [pukkandan](https://github.com/pukkandan)
+
+[![ko-fi](https://img.shields.io/badge/_-Ko--fi-red.svg?logo=kofi&labelColor=555555&style=for-the-badge)](https://ko-fi.com/pukkandan)
+
+* Owner of the fork
+
+
+
+## [shirt](https://github.com/shirt-dev)
+
+[![ko-fi](https://img.shields.io/badge/_-Ko--fi-red.svg?logo=kofi&labelColor=555555&style=for-the-badge)](https://ko-fi.com/shirt)
+
+* Multithreading (`-N`) and aria2c support for fragment downloads
+* Support for media initialization and discontinuity in HLS
+* The self-updater (`-U`)
+
+
+
+## [coletdjnz](https://github.com/coletdjnz)
+
+[![gh-sponsor](https://img.shields.io/badge/_-Sponsor-red.svg?logo=githubsponsors&labelColor=555555&style=for-the-badge)](https://github.com/sponsors/coletdjnz)
+
+* YouTube improvements including: age-gate bypass, private playlists, multiple-clients (to avoid throttling) and a lot of under-the-hood improvements
+
+
+
+## [Ashish0804](https://github.com/Ashish0804)
+
+[![ko-fi](https://img.shields.io/badge/_-Ko--fi-red.svg?logo=kofi&labelColor=555555&style=for-the-badge)](https://ko-fi.com/ashish0804)
+
+* Added support for new websites Zee5, MXPlayer, DiscoveryPlusIndia, ShemarooMe, Utreon etc
+* Added playlist/series downloads for TubiTv, SonyLIV, Voot, HotStar etc

Makefile

@@ -13,7 +13,7 @@ pypi-files: AUTHORS Changelog.md LICENSE README.md README.txt supportedsites com
 .PHONY: all clean install test tar pypi-files completions ot offlinetest codetest supportedsites
 
 clean-test:
-	rm -rf *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.frag *.frag.urls *.frag.aria2 test/testdata/player-*.js
+	rm -rf *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.frag *.frag.urls *.frag.aria2 test/testdata/player-*.js *.opus *.webp *.ttml *.vtt *.jpeg
 clean-dist:
 	rm -rf yt-dlp.1.temp.md yt-dlp.1 README.txt MANIFEST build/ dist/ .coverage cover/ yt-dlp.tar.gz completions/ yt_dlp/extractor/lazy_extractors.py *.spec CONTRIBUTING.md.tmp yt-dlp yt-dlp.exe yt_dlp.egg-info/ AUTHORS .mailmap
 clean-cache:
@@ -110,7 +110,7 @@ _EXTRACTOR_FILES = $(shell find yt_dlp/extractor -iname '*.py' -and -not -iname
 yt_dlp/extractor/lazy_extractors.py: devscripts/make_lazy_extractors.py devscripts/lazy_load_template.py $(_EXTRACTOR_FILES)
 	$(PYTHON) devscripts/make_lazy_extractors.py $@
 
-yt-dlp.tar.gz: README.md yt-dlp.1 completions Changelog.md AUTHORS
+yt-dlp.tar.gz: yt-dlp README.md supportedsites.md yt-dlp.1 completions Changelog.md AUTHORS
 	@tar -czf $(DESTDIR)/yt-dlp.tar.gz --transform "s|^|yt-dlp/|" --owner 0 --group 0 \
 		--exclude '*.DS_Store' \
 		--exclude '*.kate-swp' \
@@ -124,7 +124,7 @@ yt-dlp.tar.gz: README.md yt-dlp.1 completions Changelog.md AUTHORS
 		devscripts test \
 		Changelog.md AUTHORS LICENSE README.md supportedsites.md \
 		Makefile MANIFEST.in yt-dlp.1 completions \
-		setup.py setup.cfg yt-dlp
+		setup.py setup.cfg yt-dlp yt_dlp
 
 AUTHORS: .mailmap
 	git shortlog -s -n | cut -f2 | sort > AUTHORS

README.md

@@ -1,17 +1,16 @@
 <div align="center">
 
-# YT-DLP
-A command-line program to download videos from YouTube and many other [video platforms](supportedsites.md)
+[![YT-DLP](https://raw.githubusercontent.com/yt-dlp/yt-dlp/master/.github/banner.svg)](#readme)
 
-<!-- GHA doesn't have for-the-badge style
-[![CI Status](https://github.com/yt-dlp/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/yt-dlp/yt-dlp/actions)
--->
-[![Release version](https://img.shields.io/github/v/release/yt-dlp/yt-dlp?color=brightgreen&label=Release&style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/releases/latest)
-[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg?style=for-the-badge)](LICENSE)
+[![Release version](https://img.shields.io/github/v/release/yt-dlp/yt-dlp?color=blue&label=&style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/releases/latest)
+[![CI Status](https://img.shields.io/github/workflow/status/yt-dlp/yt-dlp/Core%20Tests/master?label=&style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/actions)
+[![License: Unlicense](https://img.shields.io/badge/-Unlicense-blue.svg?style=for-the-badge)](LICENSE)
+[![Donate](https://img.shields.io/badge/_-Donate-red.svg?logo=githubsponsors&labelColor=555555&style=for-the-badge)](Collaborators.md#collaborators)
+[![Supported Sites](https://img.shields.io/badge/-Supported_Sites-brightgreen.svg?style=for-the-badge)](supportedsites.md)
+[![Discord](https://img.shields.io/discord/807245652072857610?color=blue&label=​&logo=discord&style=for-the-badge)](https://discord.gg/H5MNcFW63r)
 [![Doc Status](https://readthedocs.org/projects/yt-dlp/badge/?version=latest&style=for-the-badge)](https://yt-dlp.readthedocs.io)
-[![Discord](https://img.shields.io/discord/807245652072857610?color=blue&label=discord&logo=discord&style=for-the-badge)](https://discord.gg/H5MNcFW63r)
 [![Commits](https://img.shields.io/github/commit-activity/m/yt-dlp/yt-dlp?label=commits&style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/commits)
-[![Last Commit](https://img.shields.io/github/last-commit/yt-dlp/yt-dlp/master?style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/commits)
+[![Last Commit](https://img.shields.io/github/last-commit/yt-dlp/yt-dlp/master?label=&style=for-the-badge)](https://github.com/yt-dlp/yt-dlp/commits)
 [![Downloads](https://img.shields.io/github/downloads/yt-dlp/yt-dlp/total?style=for-the-badge&color=blue)](https://github.com/yt-dlp/yt-dlp/releases/latest)
 [![PyPi Downloads](https://img.shields.io/pypi/dm/yt-dlp?label=PyPi&style=for-the-badge)](https://pypi.org/project/yt-dlp)
 
@@ -40,7 +39,7 @@ yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on t
     * [Subtitle Options](#subtitle-options)
     * [Authentication Options](#authentication-options)
     * [Post-processing Options](#post-processing-options)
-    * [SponSkrub (SponsorBlock) Options](#sponskrub-sponsorblock-options)
+    * [SponsorBlock Options](#sponsorblock-options)
     * [Extractor Options](#extractor-options)
 * [CONFIGURATION](#configuration)
     * [Authentication with .netrc file](#authentication-with-netrc-file)
@@ -63,22 +62,23 @@ yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on t
 # NEW FEATURES
 The major new features from the latest release of [blackjack4494/yt-dlc](https://github.com/blackjack4494/yt-dlc) are:
 
-* **[SponSkrub Integration](#sponskrub-sponsorblock-options)**: You can use [SponSkrub](https://github.com/yt-dlp/SponSkrub) to mark/remove sponsor sections in youtube videos by utilizing the [SponsorBlock](https://sponsor.ajay.app) API
+* **[SponsorBlock Integration](#sponsorblock-options)**: You can mark/remove sponsor sections in youtube videos by utilizing the [SponsorBlock](https://sponsor.ajay.app) API
 
-* **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
+* **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection than what is possible by simply using `--format` ([examples](#format-selection-examples))
 
 * **Merged with youtube-dl [commit/379f52a](https://github.com/ytdl-org/youtube-dl/commit/379f52a4954013767219d25099cce9e0f9401961)**: (v2021.06.06) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
 
 * **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--write-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.
 
 * **Youtube improvements**:
-    * All Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) supports downloading multiple pages of content
+    * All Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) and private playlists supports downloading multiple pages of content
     * Search (`ytsearch:`, `ytsearchdate:`), search URLs and in-channel search works
     * Mixes supports downloading multiple pages of content
+    * Most (but not all) age-gated content can be downloaded without cookies
     * Partial workaround for throttling issue
     * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
     * `255kbps` audio is extracted from youtube music if premium cookies are given
-    * Youtube music Albums, channels etc can be downloaded
+    * Youtube music Albums, channels etc can be downloaded ([except self-uploaded music](https://github.com/yt-dlp/yt-dlp/issues/723))
 
 * **Cookies from browser**: Cookies can be automatically extracted from all major web browsers using `--cookies-from-browser BROWSER[:PROFILE]`
 
@@ -88,9 +88,9 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **Aria2c with HLS/DASH**: You can use `aria2c` as the external downloader for DASH(mpd) and HLS(m3u8) formats
 
-* **New extractors**: AnimeLab, Philo MSO, Spectrum MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries, VidioPremier, VidioLive, RCTIPlus, TBS Live, douyin, pornflip
+* **New extractors**: AnimeLab, Philo MSO, Spectrum MSO, SlingTV MSO, Cablevision MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries, VidioPremier, VidioLive, RCTIPlus, TBS Live, douyin, pornflip, ParamountPlusSeries, ScienceChannel, Utreon, OpenRec, BandcampMusic, blackboardcollaborate, eroprofile albums, mirrativ, BannedVideo, bilibili categories, Epicon, filmmodu, GabTV, HungamaAlbum, ManotoTV, Niconico search, Patreon User, peloton, ProjectVeritas, radiko, StarTV, tiktok user, Tokentube, voicy, TV2HuSeries
 
-* **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay, patreon, videa, yahoo, BravoTV, crunchyroll playlist, RTP, viki
+* **Fixed/improved extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay, patreon, videa, yahoo, BravoTV, crunchyroll playlist, RTP, viki, Hotstar, vidio, vimeo, mediaset, Mxplayer, nbcolympics, ParamountPlus, Newgrounds, SAML Verizon login, Hungama, afreecatv, aljazeera, ATV, bitchute, camtube, CDA, eroprofile, facebook, HearThisAtIE, iwara, kakao, Motherless, Nova, peertube, pornhub, reddit, tiktok, TV2, TV2Hu, tv5mondeplus, VH1, Viafree, XHamster
 
 * **Subtitle extraction from manifests**: Subtitles can be extracted from streaming media manifests. See [commit/be6202f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
 
@@ -98,11 +98,11 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **Portable Configuration**: Configuration files are automatically loaded from the home and root directories. See [configuration](#configuration) for details
 
-* **Output template improvements**: Output templates can now have date-time formatting, numeric offsets, object traversal etc. See [output template](#output-template) for details. Even more advanced operations can also be done with the help of `--parse-metadata`
+* **Output template improvements**: Output templates can now have date-time formatting, numeric offsets, object traversal etc. See [output template](#output-template) for details. Even more advanced operations can also be done with the help of `--parse-metadata` and `--replace-in-metadata`
 
-* **Other new options**: `--sleep-requests`, `--convert-thumbnails`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc
+* **Other new options**: `--print`, `--sleep-requests`, `--convert-thumbnails`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc
 
-* **Improvements**: Multiple `--postprocessor-args` and `--downloader-args`, faster archive checking, more [format selection options](#format-selection) etc
+* **Improvements**: Regex and other operators in `--match-filter`, multiple `--postprocessor-args` and `--downloader-args`, faster archive checking, more [format selection options](#format-selection) etc
 
 * **Plugin extractors**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
 
@@ -123,7 +123,7 @@ Some of yt-dlp's default options are different from that of youtube-dl and youtu
 * The options `--id`, `--auto-number` (`-A`), `--title` (`-t`) and `--literal` (`-l`), no longer work. See [removed options](#Removed) for details
 * `avconv` is not supported as as an alternative to `ffmpeg`
 * The default [output template](#output-template) is `%(title)s [%(id)s].%(ext)s`. There is no real reason for this change. This was changed before yt-dlp was ever made public and now there are no plans to change it back to `%(title)s.%(id)s.%(ext)s`. Instead, you may use `--compat-options filename`
-* The default [format sorting](sorting-formats) is different from youtube-dl and prefers higher resolution and better codecs rather than higher bitrates. You can use the `--format-sort` option to change this to any order you prefer, or use `--compat-options format-sort` to use youtube-dl's sorting order
+* The default [format sorting](#sorting-formats) is different from youtube-dl and prefers higher resolution and better codecs rather than higher bitrates. You can use the `--format-sort` option to change this to any order you prefer, or use `--compat-options format-sort` to use youtube-dl's sorting order
 * The default format selector is `bv*+ba/b`. This means that if a combined video + audio format that is better than the best video-only format is found, the former will be prefered. Use `-f bv+ba/b` or `--compat-options format-spec` to revert this
 * Unlike youtube-dlc, yt-dlp does not allow merging multiple audio/video streams into one file by default (since this conflicts with the use of `-f bv*+ba`). If needed, this feature must be enabled using `--audio-multistreams` and `--video-multistreams`. You can also use `--compat-options multistreams` to enable both
 * `--ignore-errors` is enabled by default. Use `--abort-on-error` or `--compat-options abort-on-error` to abort on errors instead
@@ -137,6 +137,8 @@ Some of yt-dlp's default options are different from that of youtube-dl and youtu
 * Unavailable videos are also listed for youtube playlists. Use `--compat-options no-youtube-unavailable-videos` to remove this
 * If `ffmpeg` is used as the downloader, the downloading and merging of formats happen in a single step when possible. Use `--compat-options no-direct-merge` to revert this
 * Thumbnail embedding in `mp4` is done with mutagen if possible. Use `--compat-options embed-thumbnail-atomicparsley` to force the use of AtomicParsley instead
+* Some private fields such as filenames are removed by default from the infojson. Use `--no-clean-infojson` or `--compat-options no-clean-infojson` to revert this
+* When `--embed-subs` and `--write-subs` are used together, the subtitles are written to disk and also embedded in the media file. You can use just `--embed-subs` to embed the subs and automatically delete the seperate file. See [#630 (comment)](https://github.com/yt-dlp/yt-dlp/issues/630#issuecomment-893659460) for more info. `--compat-options no-keep-subs` can be used to revert this.
 
 For ease of use, a few more compat options are available:
 * `--compat-options all`: Use all compat options
@@ -149,6 +151,7 @@ yt-dlp is not platform specific. So it should work on your Unix box, on Windows
 
 You can install yt-dlp using one of the following methods:
 * Download the binary from the [latest release](https://github.com/yt-dlp/yt-dlp/releases/latest) (recommended method)
+* With Homebrew, `brew install yt-dlp/taps/yt-dlp`
 * Use [PyPI package](https://pypi.org/project/yt-dlp): `python3 -m pip install --upgrade yt-dlp`
 * Use pip+git: `python3 -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp.git@release`
 * Install master branch: `python3 -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp`
@@ -172,9 +175,16 @@ sudo aria2c https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o
 sudo chmod a+rx /usr/local/bin/yt-dlp
 ```
 
+macOS or Linux users that are using Homebrew (formerly known as Linuxbrew for Linux users) can also install it by:
+
+```
+brew install yt-dlp/taps/yt-dlp
+```
+
 ### UPDATE
 You can use `yt-dlp -U` to update if you are using the provided release.
 If you are using `pip`, simply re-run the same command that was used to install the program.
+If you have installed using Homebrew, run `brew upgrade yt-dlp/taps/yt-dlp`
 
 ### DEPENDENCIES
 Python versions 3.6+ (CPython and PyPy) are supported. Other versions and implementations may or may not work correctly.
@@ -184,7 +194,6 @@ On windows, [Microsoft Visual C++ 2010 SP1 Redistributable Package (x86)](https:
 
 While all the other dependancies are optional, `ffmpeg` and `ffprobe` are highly recommended
 * [**ffmpeg** and **ffprobe**](https://www.ffmpeg.org) - Required for [merging seperate video and audio files](#format-selection) as well as for various [post-processing](#post-processing-options) tasks. Licence [depends on the build](https://www.ffmpeg.org/legal.html)
-* [**sponskrub**](https://github.com/faissaloo/SponSkrub) - For using the [sponskrub options](#sponskrub-sponsorblock-options). Licenced under [GPLv3+](https://github.com/faissaloo/SponSkrub/blob/master/LICENCE.md)
 * [**mutagen**](https://github.com/quodlibet/mutagen) - For embedding thumbnail in certain formats. Licenced under [GPLv2+](https://github.com/quodlibet/mutagen/blob/master/COPYING)
 * [**pycryptodome**](https://github.com/Legrandin/pycryptodome) - For decrypting various data. Licenced under [BSD2](https://github.com/Legrandin/pycryptodome/blob/master/LICENSE.rst)
 * [**websockets**](https://github.com/aaugustin/websockets) - For downloading over websocket. Licenced under [BSD3](https://github.com/aaugustin/websockets/blob/main/LICENSE)
@@ -193,6 +202,7 @@ While all the other dependancies are optional, `ffmpeg` and `ffprobe` are highly
 * [**rtmpdump**](http://rtmpdump.mplayerhq.hu) - For downloading `rtmp` streams. ffmpeg will be used as a fallback. Licenced under [GPLv2+](http://rtmpdump.mplayerhq.hu)
 * [**mplayer**](http://mplayerhq.hu/design7/info.html) or [**mpv**](https://mpv.io) - For downloading `rstp` streams. ffmpeg will be used as a fallback. Licenced under [GPLv2+](https://github.com/mpv-player/mpv/blob/master/Copyright)
 * [**phantomjs**](https://github.com/ariya/phantomjs) - Used in extractors where javascript needs to be run. Licenced under [BSD3](https://github.com/ariya/phantomjs/blob/master/LICENSE.BSD)
+* [**sponskrub**](https://github.com/faissaloo/SponSkrub) - For using the now **deprecated** [sponskrub options](#sponskrub-options). Licenced under [GPLv3+](https://github.com/faissaloo/SponSkrub/blob/master/LICENCE.md)
 * Any external downloader that you want to use with `--downloader`
 
 To use or redistribute the dependencies, you must agree to their respective licensing terms.
@@ -211,7 +221,7 @@ Once you have all the necessary dependencies installed, just run `py pyinst.py`.
 You can also build the executable without any version info or metadata by using:
 
     pyinstaller.exe yt_dlp\__main__.py --onefile --name yt-dlp
-    
+
 Note that pyinstaller [does not support](https://github.com/pyinstaller/pyinstaller#requirements-and-tested-platforms) Python installed from the Windows store without using a virtual environment
 
 **For Unix**:
@@ -238,17 +248,17 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      (default) (Alias: --no-abort-on-error)
     --abort-on-error                 Abort downloading of further videos if an
                                      error occurs (Alias: --no-ignore-errors)
-    --dump-user-agent                Display the current browser identification
-    --list-extractors                List all supported extractors
+    --dump-user-agent                Display the current user-agent and exit
+    --list-extractors                List all supported extractors and exit
     --extractor-descriptions         Output descriptions of all supported
-                                     extractors
+                                     extractors and exit
     --force-generic-extractor        Force extraction to use the generic
                                      extractor
     --default-search PREFIX          Use this prefix for unqualified URLs. For
                                      example "gvsearch2:" downloads two videos
-                                     from google videos for youtube-dl "large
-                                     apple". Use the value "auto" to let
-                                     youtube-dl guess ("auto_warning" to emit a
+                                     from google videos for the search term
+                                     "large apple". Use the value "auto" to let
+                                     yt-dlp guess ("auto_warning" to emit a
                                      warning when guessing). "error" just throws
                                      an error. The default value "fixup_error"
                                      repairs broken URLs, but emits an error if
@@ -271,7 +281,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --no-mark-watched                Do not mark videos watched (default)
     --no-colors                      Do not emit color codes in output
     --compat-options OPTS            Options that can help keep compatibility
-                                     with youtube-dl and youtube-dlc
+                                     with youtube-dl or youtube-dlc
                                      configurations by reverting some of the
                                      changes made in yt-dlp. See "Differences in
                                      default behavior" for details
@@ -315,10 +325,6 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      specify range: "--playlist-items
                                      1-3,7,10-13", it will download the videos
                                      at index 1, 2, 3, 7, 10, 11, 12 and 13
-    --match-title REGEX              Download only matching titles (regex or
-                                     caseless sub-string)
-    --reject-title REGEX             Skip download for matching titles (regex or
-                                     caseless sub-string)
     --max-downloads NUMBER           Abort after downloading NUMBER files
     --min-filesize SIZE              Do not download any videos smaller than
                                      SIZE (e.g. 50k or 44.6m)
@@ -333,29 +339,24 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --dateafter DATE                 Download only videos uploaded on or after
                                      this date. The date formats accepted is the
                                      same as --date
-    --min-views COUNT                Do not download any videos with less than
-                                     COUNT views
-    --max-views COUNT                Do not download any videos with more than
-                                     COUNT views
-    --match-filter FILTER            Generic video filter. Specify any key (see
-                                     "OUTPUT TEMPLATE" for a list of available
-                                     keys) to match if the key is present, !key
-                                     to check if the key is not present,
-                                     key>NUMBER (like "view_count > 12", also
-                                     works with >=, <, <=, !=, =) to compare
-                                     against a number, key = 'LITERAL' (like
-                                     "uploader = 'Mike Smith'", also works with
-                                     !=) to match against a string literal and &
-                                     to require multiple matches. Values which
-                                     are not known are excluded unless you put a
-                                     question mark (?) after the operator. For
-                                     example, to only match videos that have
-                                     been liked more than 100 times and disliked
-                                     less than 50 times (or the dislike
-                                     functionality is not available at the given
-                                     service), but who also have a description,
-                                     use --match-filter "like_count > 100 &
-                                     dislike_count <? 50 & description"
+    --match-filter FILTER            Generic video filter. Any field (see
+                                     "OUTPUT TEMPLATE") can be compared with a
+                                     number or a string using the operators
+                                     defined in "Filtering formats". You can
+                                     also simply specify a field to match if the
+                                     field is present and "!field" to check if
+                                     the field is not present. In addition,
+                                     Python style regular expression matching
+                                     can be done using "~=", and multiple
+                                     filters can be checked with "&". Use a "\"
+                                     to escape "&" or quotes if needed. Eg:
+                                     --match-filter "!is_live & like_count>?100
+                                     & description~='(?i)\bcats \& dogs\b'"
+                                     matches only videos that are not live, has
+                                     a like count more than 100 (or the like
+                                     field is not available), and also has a
+                                     description that contains the phrase "cats
+                                     & dogs" (ignoring case)
     --no-match-filter                Do not use generic video filter (default)
     --no-playlist                    Download only the video, if the URL refers
                                      to a video and a playlist
@@ -438,9 +439,12 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      (Alias: --external-downloader)
     --downloader-args NAME:ARGS      Give these arguments to the external
                                      downloader. Specify the downloader name and
-                                     the arguments separated by a colon ":". You
-                                     can use this option multiple times to give
-                                     different arguments to different downloaders
+                                     the arguments separated by a colon ":". For
+                                     ffmpeg, arguments can be passed to
+                                     different positions using the same syntax
+                                     as --postprocessor-args. You can use this
+                                     option multiple times to give different
+                                     arguments to different downloaders
                                      (Alias: --external-downloader-args)
 
 ## Filesystem Options:
@@ -448,17 +452,17 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      stdin), one URL per line. Lines starting
                                      with '#', ';' or ']' are considered as
                                      comments and ignored
-    -P, --paths TYPES:PATH           The paths where the files should be
+    -P, --paths [TYPES:]PATH         The paths where the files should be
                                      downloaded. Specify the type of file and
                                      the path separated by a colon ":". All the
                                      same types as --output are supported.
                                      Additionally, you can also provide "home"
-                                     and "temp" paths. All intermediary files
-                                     are first downloaded to the temp path and
-                                     then the final files are moved over to the
-                                     home path after download is finished. This
-                                     option is ignored if --output is an
-                                     absolute path
+                                     (default) and "temp" paths. All
+                                     intermediary files are first downloaded to
+                                     the temp path and then the final files are
+                                     moved over to the home path after download
+                                     is finished. This option is ignored if
+                                     --output is an absolute path
     -o, --output [TYPES:]TEMPLATE    Output filename template; see "OUTPUT
                                      TEMPLATE" for details
     --output-na-placeholder TEXT     Placeholder value for unavailable meta
@@ -499,9 +503,6 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --write-info-json                Write video metadata to a .info.json file
                                      (this may contain personal information)
     --no-write-info-json             Do not write video metadata (default)
-    --write-annotations              Write video annotations to a
-                                     .annotations.xml file
-    --no-write-annotations           Do not write video annotations (default)
     --write-playlist-metafiles       Write playlist metadata in addition to the
                                      video metadata when using --write-info-json,
                                      --write-description etc. (default)
@@ -540,8 +541,8 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --cache-dir DIR                  Location in the filesystem where youtube-dl
                                      can store some downloaded information (such
                                      as client ids and signatures) permanently.
-                                     By default $XDG_CACHE_HOME/youtube-dl or
-                                     ~/.cache/youtube-dl
+                                     By default $XDG_CACHE_HOME/yt-dlp or
+                                     ~/.cache/yt-dlp
     --no-cache-dir                   Disable filesystem caching
     --rm-cache-dir                   Delete all filesystem cache files
 
@@ -550,8 +551,8 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --no-write-thumbnail             Do not write thumbnail image to disk
                                      (default)
     --write-all-thumbnails           Write all thumbnail image formats to disk
-    --list-thumbnails                Simulate and list all available thumbnail
-                                     formats
+    --list-thumbnails                List available thumbnails of each video.
+                                     Simulate unless --no-simulate is used
 
 ## Internet Shortcut Options:
     --write-link                     Write an internet shortcut file, depending
@@ -563,30 +564,34 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --write-desktop-link             Write a .desktop Linux internet shortcut
 
 ## Verbosity and Simulation Options:
-    -q, --quiet                      Activate quiet mode
+    -q, --quiet                      Activate quiet mode. If used with
+                                     --verbose, print the log to stderr
     --no-warnings                    Ignore warnings
     -s, --simulate                   Do not download the video and do not write
                                      anything to disk
+    --no-simulate                    Download the video even if printing/listing
+                                     options are used
     --ignore-no-formats-error        Ignore "No video formats" error. Usefull
-                                     for extracting metadata even if the video
-                                     is not actually available for download
+                                     for extracting metadata even if the videos
+                                     are not actually available for download
                                      (experimental)
     --no-ignore-no-formats-error     Throw error when no downloadable video
                                      formats are found (default)
     --skip-download                  Do not download the video but write all
                                      related files (Alias: --no-download)
-    -O, --print TEMPLATE             Simulate, quiet but print the given fields.
-                                     Either a field name or similar formatting
-                                     as the output template can be used
-    -j, --dump-json                  Simulate, quiet but print JSON information.
-                                     See "OUTPUT TEMPLATE" for a description of
-                                     available keys
-    -J, --dump-single-json           Simulate, quiet but print JSON information
-                                     for each command-line argument. If the URL
-                                     refers to a playlist, dump the whole
-                                     playlist information in a single line
-    --print-json                     Be quiet and print the video information as
-                                     JSON (video is still being downloaded)
+    -O, --print TEMPLATE             Quiet, but print the given fields for each
+                                     video. Simulate unless --no-simulate is
+                                     used. Either a field name or same syntax as
+                                     the output template can be used
+    -j, --dump-json                  Quiet, but print JSON information for each
+                                     video. Simulate unless --no-simulate is
+                                     used. See "OUTPUT TEMPLATE" for a
+                                     description of available keys
+    -J, --dump-single-json           Quiet, but print JSON information for each
+                                     url or infojson passed. Simulate unless
+                                     --no-simulate is used. If the URL refers to
+                                     a playlist, the whole playlist information
+                                     is dumped in a single line
     --force-write-archive            Force download archive entries to be
                                      written as far as no errors occur, even if
                                      -s or another simulation option is used
@@ -657,17 +662,12 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      actually downloadable
     --no-check-formats               Do not check that the formats selected are
                                      actually downloadable
-    -F, --list-formats               List all available formats of requested
-                                     videos
+    -F, --list-formats               List available formats of each video.
+                                     Simulate unless --no-simulate is used
     --merge-output-format FORMAT     If a merge is required (e.g.
                                      bestvideo+bestaudio), output to given
                                      container format. One of mkv, mp4, ogg,
                                      webm, flv. Ignored if no merge is required
-    --allow-unplayable-formats       Allow unplayable formats to be listed and
-                                     downloaded. All video post-processing will
-                                     also be turned off
-    --no-allow-unplayable-formats    Do not allow unplayable formats to be
-                                     listed or downloaded (default)
 
 ## Subtitle Options:
     --write-subs                     Write subtitle file
@@ -676,7 +676,8 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      (Alias: --write-automatic-subs)
     --no-write-auto-subs             Do not write auto-generated subtitles
                                      (default) (Alias: --no-write-automatic-subs)
-    --list-subs                      List all available subtitles for the video
+    --list-subs                      List available subtitles of each video.
+                                     Simulate unless --no-simulate is used
     --sub-format FORMAT              Subtitle format, accepts formats
                                      preference, for example: "srt" or
                                      "ass/srt/best"
@@ -711,7 +712,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --audio-format FORMAT            Specify audio format to convert the audio
                                      to when -x is used. Currently supported
                                      formats are: best (default) or one of
-                                     aac|flac|mp3|m4a|opus|vorbis|wav
+                                     best|aac|flac|mp3|m4a|opus|vorbis|wav
     --audio-quality QUALITY          Specify ffmpeg audio quality, insert a
                                      value between 0 (better) and 9 (worse) for
                                      VBR or a specific bitrate like 128K
@@ -732,24 +733,23 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      and the arguments separated by a colon ":"
                                      to give the argument to the specified
                                      postprocessor/executable. Supported PP are:
-                                     Merger, ExtractAudio, SplitChapters,
+                                     Merger, ModifyChapters, SplitChapters,
+                                     ExtractAudio, VideoRemuxer, VideoConvertor,
                                      Metadata, EmbedSubtitle, EmbedThumbnail,
                                      SubtitlesConvertor, ThumbnailsConvertor,
-                                     VideoRemuxer, VideoConvertor, SponSkrub,
                                      FixupStretched, FixupM4a, FixupM3u8,
                                      FixupTimestamp and FixupDuration. The
                                      supported executables are: AtomicParsley,
-                                     FFmpeg, FFprobe, and SponSkrub. You can
-                                     also specify "PP+EXE:ARGS" to give the
-                                     arguments to the specified executable only
-                                     when being used by the specified
-                                     postprocessor. Additionally, for
-                                     ffmpeg/ffprobe, "_i"/"_o" can be appended
-                                     to the prefix optionally followed by a
-                                     number to pass the argument before the
-                                     specified input/output file. Eg: --ppa
-                                     "Merger+ffmpeg_i1:-v quiet". You can use
-                                     this option multiple times to give
+                                     FFmpeg and FFprobe. You can also specify
+                                     "PP+EXE:ARGS" to give the arguments to the
+                                     specified executable only when being used
+                                     by the specified postprocessor.
+                                     Additionally, for ffmpeg/ffprobe, "_i"/"_o"
+                                     can be appended to the prefix optionally
+                                     followed by a number to pass the argument
+                                     before the specified input/output file. Eg:
+                                     --ppa "Merger+ffmpeg_i1:-v quiet". You can
+                                     use this option multiple times to give
                                      different arguments to different
                                      postprocessors. (Alias: --ppa)
     -k, --keep-video                 Keep the intermediate video file on disk
@@ -763,14 +763,22 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --no-embed-subs                  Do not embed subtitles (default)
     --embed-thumbnail                Embed thumbnail in the video as cover art
     --no-embed-thumbnail             Do not embed thumbnail (default)
-    --embed-metadata                 Embed metadata including chapter markers
-                                     (if supported by the format) to the video
-                                     file (Alias: --add-metadata)
-    --no-embed-metadata              Do not write metadata (default)
+    --embed-metadata                 Embed metadata to the video file. Also adds
+                                     chapters to file unless --no-add-chapters
+                                     is used (Alias: --add-metadata)
+    --no-embed-metadata              Do not add metadata to file (default)
                                      (Alias: --no-add-metadata)
+    --embed-chapters                 Add chapter markers to the video file
+                                     (Alias: --add-chapters)
+    --no-embed-chapters              Do not add chapter markers (default)
+                                     (Alias: --no-add-chapters)
     --parse-metadata FROM:TO         Parse additional metadata like title/artist
                                      from other fields; see "MODIFYING METADATA"
                                      for details
+    --replace-in-metadata FIELDS REGEX REPLACE
+                                     Replace text in a metadata field using the
+                                     given regex. This option can be used
+                                     multiple times
     --xattrs                         Write metadata to the video file's xattrs
                                      (using dublin core and xdg standards)
     --fixup POLICY                   Automatically correct known faults of the
@@ -783,16 +791,22 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      path to the binary or its containing
                                      directory
     --exec CMD                       Execute a command on the file after
-                                     downloading and post-processing. Similar
-                                     syntax to the output template can be used
+                                     downloading and post-processing. Same
+                                     syntax as the output template can be used
                                      to pass any field as arguments to the
                                      command. An additional field "filepath"
                                      that contains the final path of the
                                      downloaded file is also available. If no
-                                     fields are passed, "%(filepath)s" is
-                                     appended to the end of the command
+                                     fields are passed, %(filepath)q is appended
+                                     to the end of the command. This option can
+                                     be used multiple times
+    --no-exec                        Remove any previously defined --exec
     --exec-before-download CMD       Execute a command before the actual
                                      download. The syntax is the same as --exec
+                                     but "filepath" is not available. This
+                                     option can be used multiple times
+    --no-exec-before-download        Remove any previously defined
+                                     --exec-before-download
     --convert-subs FORMAT            Convert the subtitles to another format
                                      (currently supported: srt|vtt|ass|lrc)
                                      (Alias: --convert-subtitles)
@@ -805,27 +819,51 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      files. See "OUTPUT TEMPLATE" for details
     --no-split-chapters              Do not split video based on chapters
                                      (default)
+    --remove-chapters REGEX          Remove chapters whose title matches the
+                                     given regular expression. This option can
+                                     be used multiple times
+    --no-remove-chapters             Do not remove any chapters from the file
+                                     (default)
+    --force-keyframes-at-cuts        Force keyframes around the chapters before
+                                     removing/splitting them. Requires a
+                                     reencode and thus is very slow, but the
+                                     resulting video may have fewer artifacts
+                                     around the cuts
+    --no-force-keyframes-at-cuts     Do not force keyframes around the chapters
+                                     when cutting/splitting (default)
 
-## SponSkrub (SponsorBlock) Options:
-[SponSkrub](https://github.com/yt-dlp/SponSkrub) is a utility to
-    mark/remove sponsor segments from downloaded YouTube videos using
+## SponsorBlock Options:
+Make chapter entries for, or remove various segments (sponsor,
+    introductions, etc.) from downloaded YouTube videos using the
     [SponsorBlock API](https://sponsor.ajay.app)
 
-    --sponskrub                      Use sponskrub to mark sponsored sections.
-                                     This is enabled by default if the sponskrub
-                                     binary exists (Youtube only)
-    --no-sponskrub                   Do not use sponskrub
-    --sponskrub-cut                  Cut out the sponsor sections instead of
-                                     simply marking them
-    --no-sponskrub-cut               Simply mark the sponsor sections, not cut
-                                     them out (default)
-    --sponskrub-force                Run sponskrub even if the video was already
-                                     downloaded
-    --no-sponskrub-force             Do not cut out the sponsor sections if the
-                                     video was already downloaded (default)
-    --sponskrub-location PATH        Location of the sponskrub binary; either
-                                     the path to the binary or its containing
-                                     directory
+    --sponsorblock-mark CATS         SponsorBlock categories to create chapters
+                                     for, separated by commas. Available
+                                     categories are all, sponsor, intro, outro,
+                                     selfpromo, interaction, preview,
+                                     music_offtopic. You can prefix the category
+                                     with a "-" to exempt it. See 
+                                     https://wiki.sponsor.ajay.app/index.php/Segment_Categories
+                                     for description of the categories. Eg:
+                                     --sponsorblock-query all,-preview
+    --sponsorblock-remove CATS       SponsorBlock categories to be removed from
+                                     the video file, separated by commas. If a
+                                     category is present in both mark and
+                                     remove, remove takes precedence. The syntax
+                                     and available categories are the same as
+                                     for --sponsorblock-mark
+    --sponsorblock-chapter-title TEMPLATE
+                                     The title template for SponsorBlock
+                                     chapters created by --sponsorblock-mark.
+                                     The same syntax as the output template is
+                                     used, but the only available fields are
+                                     start_time, end_time, category, categories,
+                                     name, category_names. Defaults to
+                                     "[SponsorBlock]: %(category_names)l"
+    --no-sponsorblock                Disable both --sponsorblock-mark and
+                                     --sponsorblock-remove
+    --sponsorblock-api URL           SponsorBlock API location, defaults to
+                                     https://sponsor.ajay.app
 
 ## Extractor Options:
     --extractor-retries RETRIES      Number of retries for known extractor
@@ -917,10 +955,11 @@ The simplest usage of `-o` is not to set any template arguments when downloading
 It may however also contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by formatting operations.
 
 The field names themselves (the part inside the parenthesis) can also have some special formatting:
-1. **Object traversal**: The dictionaries and lists available in metadata can be traversed by using a `.` (dot) separator. You can also do python slicing using `:`. Eg: `%(tags.0)s`, `%(subtitles.en.-1.ext)`, `%(id.3:7:-1)s`. Note that the fields that become available using this method are not listed below. Use `-j` to see such fields
+1. **Object traversal**: The dictionaries and lists available in metadata can be traversed by using a `.` (dot) separator. You can also do python slicing using `:`. Eg: `%(tags.0)s`, `%(subtitles.en.-1.ext)s`, `%(id.3:7:-1)s`, `%(formats.:.format_id)s`. `%()s` refers to the entire infodict. Note that all the fields that become available using this method are not listed below. Use `-j` to see such fields
 1. **Addition**: Addition and subtraction of numeric fields can be done using `+` and `-` respectively. Eg: `%(playlist_index+10)03d`, `%(n_entries+1-playlist_index)d`
 1. **Date/time Formatting**: Date/time fields can be formatted according to [strftime formatting](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) by specifying it separated from the field name using a `>`. Eg: `%(duration>%H-%M-%S)s`, `%(upload_date>%Y-%m-%d)s`, `%(epoch-3600>%H-%M-%S)s`
 1. **Default**: A default value can be specified for when the field is empty using a `|` seperator. This overrides `--output-na-template`. Eg: `%(uploader|Unknown)s`
+1. **More Conversions**: In addition to the normal format types `diouxXeEfFgGcrs`, `j`, `l`, `q` can be used for converting to **j**son, a comma seperated **l**ist and a string **q**uoted for the terminal respectively
 
 To summarize, the general syntax for a field is:
 ```
@@ -957,7 +996,7 @@ The available fields are:
  - `average_rating` (numeric): Average rating give by users, the scale used depends on the webpage
  - `comment_count` (numeric): Number of comments on the video (For some extractors, comments are only downloaded at the end, and so this field cannot be used)
  - `age_limit` (numeric): Age restriction for the video (years)
- - `live_status` (string): One of 'is_live', 'was_live', 'upcoming', 'not_live'
+ - `live_status` (string): One of 'is_live', 'was_live', 'is_upcoming', 'not_live'
  - `is_live` (boolean): Whether this video is a live stream or a fixed-length video
  - `was_live` (boolean): Whether this video was originally a live stream
  - `playable_in_embed` (string): Whether this video is allowed to play in embedded players on other sites
@@ -1034,6 +1073,15 @@ Available only when used in `--print`:
 
  - `urls` (string): The URLs of all requested formats, one in each line
  - `filename` (string): Name of the video file. Note that the actual filename may be different due to post-processing. Use `--exec echo` to get the name after all postprocessing is complete
+ 
+Available only in `--sponsorblock-chapter-title`:
+
+ - `start_time` (numeric): Start time of the chapter in seconds
+ - `end_time` (numeric): End time of the chapter in seconds
+ - `categories` (list): The SponsorBlock categories the chapter belongs to
+ - `category` (string): The smallest SponsorBlock category the chapter belongs to
+ - `category_names` (list): Friendly names of the categories
+ - `name` (string): Friendly name of the smallest category
 
 Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default).
 
@@ -1158,7 +1206,9 @@ Format selectors can also be grouped using parentheses, for example if you want
 
 ## Sorting Formats
 
-You can change the criteria for being considered the `best` by using `-S` (`--format-sort`). The general format for this is `--format-sort field1,field2...`. The available fields are:
+You can change the criteria for being considered the `best` by using `-S` (`--format-sort`). The general format for this is `--format-sort field1,field2...`.
+
+The available fields are:
 
  - `hasvid`: Gives priority to formats that has a video stream
  - `hasaud`: Gives priority to formats that has a audio stream
@@ -1186,9 +1236,11 @@ You can change the criteria for being considered the `best` by using `-S` (`--fo
  - `br`: Equivalent to using `tbr,vbr,abr`
  - `asr`: Audio sample rate in Hz
 
-Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in descending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a preferred value for the fields, separated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two preferred values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
+All fields, unless specified otherwise, are sorted in descending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a preferred value for the fields, separated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two preferred values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
 
-The fields `hasvid`, `ie_pref`, `lang` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `quality,res,fps,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
+The fields `hasvid` and `ie_pref` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `lang,quality,res,fps,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. The extractors may override this default order, but they cannot override the user-provided order.
+
+Note that the default has `codec:vp9.2`; i.e. `av1` is not prefered
 
 If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all respects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
 
@@ -1320,13 +1372,39 @@ $ yt-dlp -S '+res:480,codec,br'
 
 # MODIFYING METADATA
 
-The metadata obtained the the extractors can be modified by using `--parse-metadata FROM:TO`. The general syntax is to give the name of a field or a template (with similar syntax to [output template](#output-template)) to extract data from, and the format to interpret it as, separated by a colon `:`. Either a [python regular expression](https://docs.python.org/3/library/re.html#regular-expression-syntax) with named capture groups or a similar syntax to the [output template](#output-template) (only `%(field)s` formatting is supported) can be used for `TO`. The option can be used multiple times to parse and modify various fields.
+The metadata obtained the the extractors can be modified by using `--parse-metadata` and `--replace-in-metadata`
+
+`--replace-in-metadata FIELDS REGEX REPLACE` is used to replace text in any metadata field using [python regular expression](https://docs.python.org/3/library/re.html#regular-expression-syntax). [Backreferences](https://docs.python.org/3/library/re.html?highlight=backreferences#re.sub) can be used in the replace string for advanced use.
+
+The general syntax of `--parse-metadata FROM:TO` is to give the name of a field or an [output template](#output-template) to extract data from, and the format to interpret it as, separated by a colon `:`. Either a [python regular expression](https://docs.python.org/3/library/re.html#regular-expression-syntax) with named capture groups or a similar syntax to the [output template](#output-template) (only `%(field)s` formatting is supported) can be used for `TO`. The option can be used multiple times to parse and modify various fields.
 
 Note that any field created by this can be used in the [output template](#output-template) and will also affect the media file's metadata added when using `--add-metadata`.
 
 This option also has a few special uses:
-* You can use this to change the metadata that is embedded in the media file. To do this, set the value of the corresponding field with a `meta_` prefix. For example, any value you set to `meta_description` field will be added to the `description` field in the file. You can use this to set a different "description" and "synopsis", for example
 * You can download an additional URL based on the metadata of the currently downloaded video. To do this, set the field `additional_urls` to the URL that you want to download. Eg: `--parse-metadata "description:(?P<additional_urls>https?://www\.vimeo\.com/\d+)` will download the first vimeo video found in the description
+* You can use this to change the metadata that is embedded in the media file. To do this, set the value of the corresponding field with a `meta_` prefix. For example, any value you set to `meta_description` field will be added to the `description` field in the file. For example, you can use this to set a different "description" and "synopsis"
+
+For reference, these are the fields yt-dlp adds by default to the file metadata:
+
+Metadata fields|From
+:---|:---
+`title`|`track` or `title`
+`date`|`upload_date`
+`description`,  `synopsis`|`description`
+`purl`, `comment`|`webpage_url`
+`track`|`track_number`
+`artist`|`artist`, `creator`, `uploader` or `uploader_id`
+`genre`|`genre`
+`album`|`album`
+`album_artist`|`album_artist`
+`disc`|`disc_number`
+`show`|`series`
+`season_number`|`season_number`
+`episode_id`|`episode` or `episode_id`
+`episode_sort`|`episode_number`
+`language` of each stream|From the format's `language`
+**Note**: The file format may not support some of these fields
+
 
 ## Modifying metadata examples
 
@@ -1345,20 +1423,24 @@ $ yt-dlp --parse-metadata '%(series)s S%(season_number)02dE%(episode_number)02d:
 # Set "comment" field in video metadata using description instead of webpage_url
 $ yt-dlp --parse-metadata 'description:(?s)(?P<meta_comment>.+)' --add-metadata
 
+# Replace all spaces and "_" in title and uploader with a `-`
+$ yt-dlp --replace-in-metadata 'title,uploader' '[ _]' '-'
+
 ```
 
 # EXTRACTOR ARGUMENTS
 
-Some extractors accept additional arguments which can be passed using `--extractor-args KEY:ARGS`. `ARGS` is a `;` (semicolon) seperated string of `ARG=VAL1,VAL2`. Eg: `--extractor-args "youtube:skip=dash,hls;player_client=android" --extractor-args "funimation:version=uncut"`
+Some extractors accept additional arguments which can be passed using `--extractor-args KEY:ARGS`. `ARGS` is a `;` (semicolon) seperated string of `ARG=VAL1,VAL2`. Eg: `--extractor-args "youtube:player_client=android_agegate,web;include_live_dash" --extractor-args "funimation:version=uncut"`
 
 The following extractors use this feature:
 * **youtube**
     * `skip`: `hls` or `dash` (or both) to skip download of the respective manifests
-    * `player_client`: Clients to extract video data from - one or more of `web`, `android`, `ios`, `web_music`, `android_music`, `ios_music`. By default, `android,web` is used. If the URL is from `music.youtube.com`, `android,web,android_music,web_music` is used
+    * `player_client`: Clients to extract video data from. The main clients are `web`, `android`, `ios`, `mweb`. These also have `_music`, `_embedded`, `_agegate`, and `_creator` variants (Eg: `web_embedded`) (`mweb` has only `_agegate`). By default, `android,web` is used, but the agegate and creator variants are added as required for age-gated videos. Similarly the music variants are added for `music.youtube.com` urls. You can also use `all` to use all the clients
     * `player_skip`: `configs` - skip any requests for client configs and use defaults
+    * `include_live_dash`: Include live dash formats (These formats don't download properly)
     * `comment_sort`: `top` or `new` (default) - choose comment sorting mode (on YouTube's side).
-    * `max_comments`: maximum amount of comments to download (default all).
-    * `max_comment_depth`: maximum depth for nested comments. YouTube supports depths 1 or 2 (default). 
+    * `max_comments`: Maximum amount of comments to download (default all).
+    * `max_comment_depth`: Maximum depth for nested comments. YouTube supports depths 1 or 2 (default).
 
 * **funimation**
     * `language`: Languages to extract. Eg: `funimation:language=english,japanese`
@@ -1380,8 +1462,8 @@ Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`. Currently
 
 These are all the deprecated options and the current alternative to achieve the same effect
 
-#### Not recommended
-While these options still work, their use is not recommended since there are other alternatives to achieve the same
+#### Redundant options
+While these options are redundant, they are still expected to be used due to their ease of use
 
     --get-description                --print description
     --get-duration                   --print duration_string
@@ -1391,8 +1473,19 @@ While these options still work, their use is not recommended since there are oth
     --get-thumbnail                  --print thumbnail
     -e, --get-title                  --print title
     -g, --get-url                    --print urls
+    -j, --dump-json                  --print "%()j"
+    --match-title REGEX              --match-filter "title ~= (?i)REGEX"
+    --reject-title REGEX             --match-filter "title !~= (?i)REGEX"
+    --min-views COUNT                --match-filter "view_count >=? COUNT"
+    --max-views COUNT                --match-filter "view_count <=? COUNT"
+
+
+#### Not recommended
+While these options still work, their use is not recommended since there are other alternatives to achieve the same
+
     --all-formats                    -f all
     --all-subs                       --sub-langs all --write-subs
+    --print-json                     -j --no-simulate
     --autonumber-size NUMBER         Use string formatting. Eg: %(autonumber)03d
     --autonumber-start NUMBER        Use internal field formatting like %(autonumber+NUMBER)s
     --metadata-from-title FORMAT     --parse-metadata "%(title)s:FORMAT"
@@ -1400,13 +1493,19 @@ While these options still work, their use is not recommended since there are oth
     --hls-prefer-ffmpeg              --downloader "m3u8:ffmpeg"
     --list-formats-old               --compat-options list-formats (Alias: --no-list-formats-as-table)
     --list-formats-as-table          --compat-options -list-formats [Default] (Alias: --no-list-formats-old)
-    --sponskrub-args ARGS            --ppa "sponskrub:ARGS"
     --youtube-skip-dash-manifest     --extractor-args "youtube:skip=dash" (Alias: --no-youtube-include-dash-manifest)
     --youtube-skip-hls-manifest      --extractor-args "youtube:skip=hls" (Alias: --no-youtube-include-hls-manifest)
     --youtube-include-dash-manifest  Default (Alias: --no-youtube-skip-dash-manifest)
     --youtube-include-hls-manifest   Default (Alias: --no-youtube-skip-hls-manifest)
-    --test                           Used by developers for testing extractors. Not intended for the end user
-    --youtube-print-sig-code         Used for testing youtube signatures
+
+
+#### Developer options
+These options are not intended to be used by the end-user
+
+    --test                           Download only part of video for testing extractors
+    --youtube-print-sig-code         For testing youtube signatures
+    --allow-unplayable-formats       List unplayable formats also
+    --no-allow-unplayable-formats    Default
 
 
 #### Old aliases
@@ -1428,6 +1527,18 @@ These are aliases that are no longer documented for various reasons
     --write-srt                      --write-subs
     --yes-overwrites                 --force-overwrites
 
+#### Sponskrub Options
+Support for [SponSkrub](https://github.com/faissaloo/SponSkrub) has been deprecated in favor of `--sponsorblock`
+
+    --sponskrub                      --sponsorblock-mark all
+    --no-sponskrub                   --no-sponsorblock
+    --sponskrub-cut                  --sponsorblock-remove all
+    --no-sponskrub-cut               --sponsorblock-remove -all
+    --sponskrub-force                Not applicable
+    --no-sponskrub-force             Not applicable
+    --sponskrub-location             Not applicable
+    --sponskrub-args                 Not applicable
+
 #### No longer supported
 These options may no longer work as intended
 
@@ -1437,6 +1548,8 @@ These options may no longer work as intended
     --no-call-home                   Default
     --include-ads                    No longer supported
     --no-include-ads                 Default
+    --write-annotations              No supported site has annotations now
+    --no-write-annotations           Default
 
 #### Removed
 These options were deprecated since 2014 and have now been entirely removed

devscripts/lazy_load_template.py

@@ -1,20 +1,25 @@
-#!/usr/bin/env python3
 # coding: utf-8
-from __future__ import unicode_literals
-
 import re
 
 
-class LazyLoadExtractor(object):
+class LazyLoadMetaClass(type):
+    def __getattr__(cls, name):
+        return getattr(cls._get_real_class(), name)
+
+
+class LazyLoadExtractor(metaclass=LazyLoadMetaClass):
     _module = None
+    _WORKING = True
 
     @classmethod
-    def ie_key(cls):
-        return cls.__name__[:-2]
+    def _get_real_class(cls):
+        if '__real_class' not in cls.__dict__:
+            mod = __import__(cls._module, fromlist=(cls.__name__,))
+            cls.__real_class = getattr(mod, cls.__name__)
+        return cls.__real_class
 
     def __new__(cls, *args, **kwargs):
-        mod = __import__(cls._module, fromlist=(cls.__name__,))
-        real_cls = getattr(mod, cls.__name__)
+        real_cls = cls._get_real_class()
         instance = real_cls.__new__(real_cls)
         instance.__init__(*args, **kwargs)
         return instance

devscripts/make_lazy_extractors.py

@@ -16,23 +16,28 @@ if os.path.exists(lazy_extractors_filename):
     os.remove(lazy_extractors_filename)
 
 # Block plugins from loading
-os.rename('ytdlp_plugins', 'ytdlp_plugins_blocked')
+plugins_dirname = 'ytdlp_plugins'
+plugins_blocked_dirname = 'ytdlp_plugins_blocked'
+if os.path.exists(plugins_dirname):
+    os.rename(plugins_dirname, plugins_blocked_dirname)
 
 from yt_dlp.extractor import _ALL_CLASSES
 from yt_dlp.extractor.common import InfoExtractor, SearchInfoExtractor
 
-os.rename('ytdlp_plugins_blocked', 'ytdlp_plugins')
+if os.path.exists(plugins_blocked_dirname):
+    os.rename(plugins_blocked_dirname, plugins_dirname)
 
 with open('devscripts/lazy_load_template.py', 'rt') as f:
     module_template = f.read()
 
+CLASS_PROPERTIES = ['ie_key', 'working', '_match_valid_url', 'suitable', '_match_id', 'get_temp_id']
 module_contents = [
-    module_template + '\n' + getsource(InfoExtractor.suitable) + '\n',
-    'class LazyLoadSearchExtractor(LazyLoadExtractor):\n    pass\n']
+    module_template,
+    *[getsource(getattr(InfoExtractor, k)) for k in CLASS_PROPERTIES],
+    '\nclass LazyLoadSearchExtractor(LazyLoadExtractor):\n    pass\n']
 
 ie_template = '''
 class {name}({bases}):
-    _VALID_URL = {valid_url!r}
     _module = '{module}'
 '''
 
@@ -53,14 +58,17 @@ def get_base_name(base):
 
 
 def build_lazy_ie(ie, name):
-    valid_url = getattr(ie, '_VALID_URL', None)
     s = ie_template.format(
         name=name,
         bases=', '.join(map(get_base_name, ie.__bases__)),
-        valid_url=valid_url,
         module=ie.__module__)
+    valid_url = getattr(ie, '_VALID_URL', None)
+    if valid_url:
+        s += f'    _VALID_URL = {valid_url!r}\n'
+    if not ie._WORKING:
+        s += '    _WORKING = False\n'
     if ie.suitable.__func__ is not InfoExtractor.suitable.__func__:
-        s += '\n' + getsource(ie.suitable)
+        s += f'\n{getsource(ie.suitable)}'
     if hasattr(ie, '_make_valid_url'):
         # search extractors
         s += make_valid_template.format(valid_url=ie._make_valid_url())
@@ -98,7 +106,7 @@ for ie in ordered_cls:
         names.append(name)
 
 module_contents.append(
-    '_ALL_CLASSES = [{0}]'.format(', '.join(names)))
+    '\n_ALL_CLASSES = [{0}]'.format(', '.join(names)))
 
 module_src = '\n'.join(module_contents) + '\n'
 

devscripts/run_tests.sh

@@ -11,5 +11,4 @@ else
     exit 1
 fi
 
-echo python3 -m pytest -k $test_set
 python3 -m pytest -k "$test_set"

devscripts/update-formulae.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+from __future__ import unicode_literals
+
+import json
+import os
+import re
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from yt_dlp.compat import compat_urllib_request
+
+
+# usage: python3 ./devscripts/update-formulae.py <path-to-formulae-rb> <version>
+# version can be either 0-aligned (yt-dlp version) or normalized (PyPl version)
+
+filename, version = sys.argv[1:]
+
+normalized_version = '.'.join(str(int(x)) for x in version.split('.'))
+
+pypi_release = json.loads(compat_urllib_request.urlopen(
+    'https://pypi.org/pypi/yt-dlp/%s/json' % normalized_version
+).read().decode('utf-8'))
+
+tarball_file = next(x for x in pypi_release['urls'] if x['filename'].endswith('.tar.gz'))
+
+sha256sum = tarball_file['digests']['sha256']
+url = tarball_file['url']
+
+with open(filename, 'r') as r:
+    formulae_text = r.read()
+
+formulae_text = re.sub(r'sha256 "[0-9a-f]*?"', 'sha256 "%s"' % sha256sum, formulae_text)
+formulae_text = re.sub(r'url "[^"]*?"', 'url "%s"' % url, formulae_text)
+
+with open(filename, 'w') as w:
+    w.write(formulae_text)

docs/Collaborators.md

@@ -0,0 +1,5 @@
+---
+orphan: true
+---
+```{include} ../Collaborators.md
+```

pyinst.py

@@ -73,7 +73,7 @@ excluded_modules = ['test', 'ytdlp_plugins', 'youtube-dl', 'youtube-dlc']
 PyInstaller.__main__.run([
     '--name=yt-dlp%s' % _x86,
     '--onefile',
-    '--icon=devscripts/cloud.ico',
+    '--icon=devscripts/logo.ico',
     *[f'--exclude-module={module}' for module in excluded_modules],
     *[f'--hidden-import={module}' for module in dependancies],
     '--upx-exclude=vcruntime140.dll',

supportedsites.md

@@ -95,7 +95,9 @@
  - **Bandcamp**
  - **Bandcamp:album**
  - **Bandcamp:weekly**
+ - **BandcampMusic**
  - **bangumi.bilibili.com**: BiliBili番剧
+ - **BannedVideo**
  - **bbc**: BBC
  - **bbc.co.uk**: BBC iPlayer
  - **bbc.co.uk:article**: BBC articles
@@ -117,6 +119,7 @@
  - **Bigflix**
  - **Bild**: Bild.de
  - **BiliBili**
+ - **Bilibili category extractor**
  - **BilibiliAudio**
  - **BilibiliAudioAlbum**
  - **BilibiliChannel**
@@ -129,6 +132,7 @@
  - **BitChuteChannel**
  - **bitwave:replay**
  - **bitwave:stream**
+ - **BlackboardCollaborate**
  - **BleacherReport**
  - **BleacherReportCMS**
  - **Bloomberg**
@@ -151,7 +155,6 @@
  - **Camdemy**
  - **CamdemyFolder**
  - **CamModels**
- - **CamTube**
  - **CamWithHer**
  - **canalc2.tv**
  - **Canalplus**: mycanal.fr and piwiplus.fr
@@ -293,8 +296,11 @@
  - **Embedly**
  - **EMPFlix**
  - **Engadget**
+ - **Epicon**
+ - **EpiconSeries**
  - **Eporner**
  - **EroProfile**
+ - **EroProfile:album**
  - **Escapist**
  - **ESPN**
  - **ESPNArticle**
@@ -313,6 +319,7 @@
  - **fc2**
  - **fc2:embed**
  - **Fczenit**
+ - **Filmmodu**
  - **filmon**
  - **filmon:channel**
  - **Filmweb**
@@ -350,6 +357,7 @@
  - **Funk**
  - **Fusion**
  - **Fux**
+ - **GabTV**
  - **Gaia**
  - **GameInformer**
  - **GameSpot**
@@ -405,6 +413,7 @@
  - **Huajiao**: 花椒直播
  - **HuffPost**: Huffington Post
  - **Hungama**
+ - **HungamaAlbumPlaylist**
  - **HungamaSong**
  - **Hypem**
  - **ign.com**
@@ -517,6 +526,9 @@
  - **MallTV**
  - **mangomolo:live**
  - **mangomolo:video**
+ - **ManotoTV**: Manoto TV (Episode)
+ - **ManotoTVLive**: Manoto TV (Live)
+ - **ManotoTVShow**: Manoto TV (Show)
  - **ManyVids**
  - **MaoriTV**
  - **Markiza**
@@ -552,6 +564,8 @@
  - **MinistryGrid**
  - **Minoto**
  - **miomio.tv**
+ - **mirrativ**
+ - **mirrativ:user**
  - **MiTele**: mitele.es
  - **mixcloud**
  - **mixcloud:playlist**
@@ -653,6 +667,9 @@
  - **niconico**: ニコニコ動画
  - **NiconicoPlaylist**
  - **NiconicoUser**
+ - **nicovideo:search**: Nico video searches
+ - **nicovideo:search:date**: Nico video searches, newest first
+ - **nicovideo:search_url**: Nico video search URLs
  - **Nintendo**
  - **Nitter**
  - **njoy**: N-JOY
@@ -703,6 +720,8 @@
  - **OnionStudios**
  - **Ooyala**
  - **OoyalaExternal**
+ - **openrec**
+ - **openrec:capture**
  - **OraTV**
  - **orf:burgenland**: Radio Burgenland
  - **orf:fm4**: radio FM4
@@ -728,12 +747,17 @@
  - **PalcoMP3:video**
  - **pandora.tv**: 판도라TV
  - **ParamountNetwork**
+ - **ParamountPlus**
+ - **ParamountPlusSeries**
  - **parliamentlive.tv**: UK parliament videos
  - **Parlview**
  - **Patreon**
+ - **PatreonUser**
  - **pbs**: Public Broadcasting Service (PBS) and member stations: PBS: Public Broadcasting Service, APT - Alabama Public Television (WBIQ), GPB/Georgia Public Broadcasting (WGTV), Mississippi Public Broadcasting (WMPN), Nashville Public Television (WNPT), WFSU-TV (WFSU), WSRE (WSRE), WTCI (WTCI), WPBA/Channel 30 (WPBA), Alaska Public Media (KAKM), Arizona PBS (KAET), KNME-TV/Channel 5 (KNME), Vegas PBS (KLVX), AETN/ARKANSAS ETV NETWORK (KETS), KET (WKLE), WKNO/Channel 10 (WKNO), LPB/LOUISIANA PUBLIC BROADCASTING (WLPB), OETA (KETA), Ozarks Public Television (KOZK), WSIU Public Broadcasting (WSIU), KEET TV (KEET), KIXE/Channel 9 (KIXE), KPBS San Diego (KPBS), KQED (KQED), KVIE Public Television (KVIE), PBS SoCal/KOCE (KOCE), ValleyPBS (KVPT), CONNECTICUT PUBLIC TELEVISION (WEDH), KNPB Channel 5 (KNPB), SOPTV (KSYS), Rocky Mountain PBS (KRMA), KENW-TV3 (KENW), KUED Channel 7 (KUED), Wyoming PBS (KCWC), Colorado Public Television / KBDI 12 (KBDI), KBYU-TV (KBYU), Thirteen/WNET New York (WNET), WGBH/Channel 2 (WGBH), WGBY (WGBY), NJTV Public Media NJ (WNJT), WLIW21 (WLIW), mpt/Maryland Public Television (WMPB), WETA Television and Radio (WETA), WHYY (WHYY), PBS 39 (WLVT), WVPT - Your Source for PBS and More! (WVPT), Howard University Television (WHUT), WEDU PBS (WEDU), WGCU Public Media (WGCU), WPBT2 (WPBT), WUCF TV (WUCF), WUFT/Channel 5 (WUFT), WXEL/Channel 42 (WXEL), WLRN/Channel 17 (WLRN), WUSF Public Broadcasting (WUSF), ETV (WRLK), UNC-TV (WUNC), PBS Hawaii - Oceanic Cable Channel 10 (KHET), Idaho Public Television (KAID), KSPS (KSPS), OPB (KOPB), KWSU/Channel 10 & KTNW/Channel 31 (KWSU), WILL-TV (WILL), Network Knowledge - WSEC/Springfield (WSEC), WTTW11 (WTTW), Iowa Public Television/IPTV (KDIN), Nine Network (KETC), PBS39 Fort Wayne (WFWA), WFYI Indianapolis (WFYI), Milwaukee Public Television (WMVS), WNIN (WNIN), WNIT Public Television (WNIT), WPT (WPNE), WVUT/Channel 22 (WVUT), WEIU/Channel 51 (WEIU), WQPT-TV (WQPT), WYCC PBS Chicago (WYCC), WIPB-TV (WIPB), WTIU (WTIU), CET  (WCET), ThinkTVNetwork (WPTD), WBGU-TV (WBGU), WGVU TV (WGVU), NET1 (KUON), Pioneer Public Television (KWCM), SDPB Television (KUSD), TPT (KTCA), KSMQ (KSMQ), KPTS/Channel 8 (KPTS), KTWU/Channel 11 (KTWU), East Tennessee PBS (WSJK), WCTE-TV (WCTE), WLJT, Channel 11 (WLJT), WOSU TV (WOSU), WOUB/WOUC (WOUB), WVPB (WVPB), WKYU-PBS (WKYU), KERA 13 (KERA), MPBN (WCBB), Mountain Lake PBS (WCFE), NHPTV (WENH), Vermont PBS (WETK), witf (WITF), WQED Multimedia (WQED), WMHT Educational Telecommunications (WMHT), Q-TV (WDCQ), WTVS Detroit Public TV (WTVS), CMU Public Television (WCMU), WKAR-TV (WKAR), WNMU-TV Public TV 13 (WNMU), WDSE - WRPT (WDSE), WGTE TV (WGTE), Lakeland Public Television (KAWE), KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS), MontanaPBS (KUSM), KRWG/Channel 22 (KRWG), KACV (KACV), KCOS/Channel 13 (KCOS), WCNY/Channel 24 (WCNY), WNED (WNED), WPBS (WPBS), WSKG Public TV (WSKG), WXXI (WXXI), WPSU (WPSU), WVIA Public Media Studios (WVIA), WTVI (WTVI), Western Reserve PBS (WNEO), WVIZ/PBS ideastream (WVIZ), KCTS 9 (KCTS), Basin PBS (KPBT), KUHT / Channel 8 (KUHT), KLRN (KLRN), KLRU (KLRU), WTJX Channel 12 (WTJX), WCVE PBS (WCVE), KBTC Public Television (KBTC)
  - **PearVideo**
  - **PeerTube**
+ - **peloton**
+ - **peloton:live**: Peloton Live
  - **People**
  - **PerformGroup**
  - **periscope**: Periscope
@@ -774,6 +798,7 @@
  - **PornHd**
  - **PornHub**: PornHub and Thumbzilla
  - **PornHubPagedVideoList**
+ - **PornHubPlaylist**
  - **PornHubUser**
  - **PornHubUserVideosUpload**
  - **Pornotube**
@@ -781,6 +806,7 @@
  - **PornoXO**
  - **PornTube**
  - **PressTV**
+ - **ProjectVeritas**
  - **prosiebensat1**: ProSiebenSat.1 Digital
  - **puhutv**
  - **puhutv:serie**
@@ -797,6 +823,8 @@
  - **QuicklineLive**
  - **R7**
  - **R7Article**
+ - **Radiko**
+ - **RadikoRadio**
  - **radio.de**
  - **radiobremen**
  - **radiocanada**
@@ -815,6 +843,7 @@
  - **RCSVarious**
  - **RCTIPlus**
  - **RCTIPlusSeries**
+ - **RCTIPlusTV**
  - **RDS**: RDS.ca
  - **RedBull**
  - **RedBullEmbed**
@@ -873,6 +902,7 @@
  - **savefrom.net**
  - **SBS**: sbs.com.au
  - **schooltv**
+ - **ScienceChannel**
  - **screen.yahoo:search**: Yahoo screen search
  - **Screencast**
  - **ScreencastOMatic**
@@ -945,6 +975,7 @@
  - **SRGSSR**
  - **SRGSSRPlay**: srf.ch, rts.ch, rsi.ch, rtr.ch and swissinfo.ch play sites
  - **stanfordoc**: Stanford Open ClassRoom
+ - **startv**
  - **Steam**
  - **Stitcher**
  - **StitcherShow**
@@ -1011,13 +1042,15 @@
  - **ThisAmericanLife**
  - **ThisAV**
  - **ThisOldHouse**
- - **ThisVid**
  - **TikTok**
+ - **tiktok:user**
  - **tinypic**: tinypic.com videos
  - **TMZ**
  - **TNAFlix**
  - **TNAFlixNetworkEmbed**
  - **toggle**
+ - **Tokentube**
+ - **Tokentube:channel**
  - **ToonGoggles**
  - **tou.tv**
  - **Toypics**: Toypics video
@@ -1040,10 +1073,11 @@
  - **Turbo**
  - **tv.dfb.de**
  - **TV2**
- - **tv2.hu**
  - **TV2Article**
  - **TV2DK**
  - **TV2DKBornholmPlay**
+ - **tv2play.hu**
+ - **tv2playseries.hu**
  - **TV4**: tv4.se and tv4play.se
  - **TV5MondePlus**: TV5MONDE+
  - **tv5unis**
@@ -1108,6 +1142,7 @@
  - **ustream:channel**
  - **ustudio**
  - **ustudio:embed**
+ - **Utreon**
  - **Varzesh3**
  - **Vbox7**
  - **VeeHD**
@@ -1176,6 +1211,8 @@
  - **VODPl**
  - **VODPlatform**
  - **VoiceRepublic**
+ - **voicy**
+ - **voicy:channel**
  - **Voot**
  - **VootSeries**
  - **VoxMedia**

test/helper.py

@@ -198,7 +198,10 @@ def expect_info_dict(self, got_dict, expected_dict):
     expect_dict(self, got_dict, expected_dict)
     # Check for the presence of mandatory fields
     if got_dict.get('_type') not in ('playlist', 'multi_video'):
-        for key in ('id', 'url', 'title', 'ext'):
+        mandatory_fields = ['id', 'title']
+        if expected_dict.get('ext'):
+            mandatory_fields.extend(('url', 'ext'))
+        for key in mandatory_fields:
             self.assertTrue(got_dict.get(key), 'Missing mandatory field %s' % key)
     # Check for mandatory fields that are automatically set by YoutubeDL
     for key in ['webpage_url', 'extractor', 'extractor_key']:

test/test_YoutubeDL.py

@@ -10,14 +10,15 @@ import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 import copy
+import json
 
 from test.helper import FakeYDL, assertRegexpMatches
 from yt_dlp import YoutubeDL
-from yt_dlp.compat import compat_str, compat_urllib_error
+from yt_dlp.compat import compat_os_name, compat_setenv, compat_str, compat_urllib_error
 from yt_dlp.extractor import YoutubeIE
 from yt_dlp.extractor.common import InfoExtractor
 from yt_dlp.postprocessor.common import PostProcessor
-from yt_dlp.utils import ExtractorError, int_or_none, match_filter_func
+from yt_dlp.utils import ExtractorError, int_or_none, match_filter_func, LazyList
 
 TEST_URL = 'http://localhost/sample.mp4'
 
@@ -647,6 +648,7 @@ class TestYoutubeDL(unittest.TestCase):
         'title1': '$PATH',
         'title2': '%PATH%',
         'title3': 'foo/bar\\test',
+        'title4': 'foo "bar" test',
         'timestamp': 1618488000,
         'duration': 100000,
         'playlist_index': 1,
@@ -663,21 +665,28 @@ class TestYoutubeDL(unittest.TestCase):
             self.assertEqual(ydl.validate_outtmpl(tmpl), None)
 
             outtmpl, tmpl_dict = ydl.prepare_outtmpl(tmpl, info or self.outtmpl_info)
-            out = outtmpl % tmpl_dict
+            out = ydl.escape_outtmpl(outtmpl) % tmpl_dict
             fname = ydl.prepare_filename(info or self.outtmpl_info)
 
-            if callable(expected):
-                self.assertTrue(expected(out))
-                self.assertTrue(expected(fname))
-            elif isinstance(expected, compat_str):
-                self.assertEqual((out, fname), (expected, expected))
-            else:
-                self.assertEqual((out, fname), expected)
+            if not isinstance(expected, (list, tuple)):
+                expected = (expected, expected)
+            for (name, got), expect in zip((('outtmpl', out), ('filename', fname)), expected):
+                if callable(expect):
+                    self.assertTrue(expect(got), f'Wrong {name} from {tmpl}')
+                else:
+                    self.assertEqual(got, expect, f'Wrong {name} from {tmpl}')
+
+        # Side-effects
+        original_infodict = dict(self.outtmpl_info)
+        test('foo.bar', 'foo.bar')
+        original_infodict['epoch'] = self.outtmpl_info.get('epoch')
+        self.assertTrue(isinstance(original_infodict['epoch'], int))
+        test('%(epoch)d', int_or_none)
+        self.assertEqual(original_infodict, self.outtmpl_info)
 
         # Auto-generated fields
         test('%(id)s.%(ext)s', '1234.mp4')
         test('%(duration_string)s', ('27:46:40', '27-46-40'))
-        test('%(epoch)d', int_or_none)
         test('%(resolution)s', '1080p')
         test('%(playlist_index)s', '001')
         test('%(autonumber)s', '00001')
@@ -685,9 +694,15 @@ class TestYoutubeDL(unittest.TestCase):
         test('%(autonumber)s', '001', autonumber_size=3)
 
         # Escaping %
+        test('%', '%')
         test('%%', '%')
         test('%%%%', '%%')
+        test('%s', '%s')
+        test('%%%s', '%%s')
+        test('%d', '%d')
+        test('%abc%', '%abc%')
         test('%%(width)06d.%(ext)s', '%(width)06d.mp4')
+        test('%%%(height)s', '%1080')
         test('%(width)06d.%(ext)s', 'NA.mp4')
         test('%(width)06d.%%(ext)s', 'NA.%(ext)s')
         test('%%(width)06d.%(ext)s', '%(width)06d.mp4')
@@ -702,12 +717,18 @@ class TestYoutubeDL(unittest.TestCase):
         test('%(id)s', ('ab:cd', 'ab -cd'), info={'id': 'ab:cd'})
 
         # Invalid templates
-        self.assertTrue(isinstance(YoutubeDL.validate_outtmpl('%'), ValueError))
         self.assertTrue(isinstance(YoutubeDL.validate_outtmpl('%(title)'), ValueError))
         test('%(invalid@tmpl|def)s', 'none', outtmpl_na_placeholder='none')
-        test('%()s', 'NA')
-        test('%s', '%s')
-        test('%d', '%d')
+        test('%(..)s', 'NA')
+
+        # Entire info_dict
+        def expect_same_infodict(out):
+            got_dict = json.loads(out)
+            for info_field, expected in self.outtmpl_info.items():
+                self.assertEqual(got_dict.get(info_field), expected, info_field)
+            return True
+
+        test('%()j', (expect_same_infodict, str))
 
         # NA placeholder
         NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(x|def)s-%(id)s.%(ext)s'
@@ -738,13 +759,26 @@ class TestYoutubeDL(unittest.TestCase):
         test('%(width|0)04d', '0000')
         test('a%(width|)d', 'a', outtmpl_na_placeholder='none')
 
-        # Internal formatting
         FORMATS = self.outtmpl_info['formats']
+        sanitize = lambda x: x.replace(':', ' -').replace('"', "'")
+
+        # Custom type casting
+        test('%(formats.:.id)l', 'id1, id2, id3')
+        test('%(ext)l', 'mp4')
+        test('%(formats.:.id) 15l', '  id1, id2, id3')
+        test('%(formats)j', (json.dumps(FORMATS), sanitize(json.dumps(FORMATS))))
+        if compat_os_name == 'nt':
+            test('%(title4)q', ('"foo \\"bar\\" test"', "'foo _'bar_' test'"))
+        else:
+            test('%(title4)q', ('\'foo "bar" test\'', "'foo 'bar' test'"))
+
+        # Internal formatting
         test('%(timestamp-1000>%H-%M-%S)s', '11-43-20')
+        test('%(title|%)s %(title|%%)s', '% %%')
         test('%(id+1-height+3)05d', '00158')
         test('%(width+100)05d', 'NA')
-        test('%(formats.0) 15s', ('% 15s' % FORMATS[0], '% 15s' % str(FORMATS[0]).replace(':', ' -')))
-        test('%(formats.0)r', (repr(FORMATS[0]), repr(FORMATS[0]).replace(':', ' -')))
+        test('%(formats.0) 15s', ('% 15s' % FORMATS[0], '% 15s' % sanitize(str(FORMATS[0]))))
+        test('%(formats.0)r', (repr(FORMATS[0]), sanitize(repr(FORMATS[0]))))
         test('%(height.0)03d', '001')
         test('%(-height.0)04d', '-001')
         test('%(formats.-1.id)s', FORMATS[-1]['id'])
@@ -754,11 +788,22 @@ class TestYoutubeDL(unittest.TestCase):
         test('%(formats.0.id.-1+id)f', '1235.000000')
         test('%(formats.0.id.-1+formats.1.id.-1)d', '3')
 
+        # Laziness
+        def gen():
+            yield from range(5)
+            raise self.assertTrue(False, 'LazyList should not be evaluated till here')
+        test('%(key.4)s', '4', info={'key': LazyList(gen())})
+
         # Empty filename
         test('%(foo|)s-%(bar|)s.%(ext)s', '-.mp4')
         # test('%(foo|)s.%(ext)s', ('.mp4', '_.mp4'))  # fixme
         # test('%(foo|)s', ('', '_'))  # fixme
 
+        # Environment variable expansion for prepare_filename
+        compat_setenv('__yt_dlp_var', 'expanded')
+        envvar = '%__yt_dlp_var%' if compat_os_name == 'nt' else '$__yt_dlp_var'
+        test(envvar, (envvar, 'expanded'))
+
         # Path expansion and escaping
         test('Hello %(title1)s', 'Hello $PATH')
         test('Hello %(title2)s', 'Hello %PATH%')
@@ -933,54 +978,31 @@ class TestYoutubeDL(unittest.TestCase):
             ydl.process_ie_result(copy.deepcopy(playlist))
             return ydl.downloaded_info_dicts
 
-        def get_ids(params):
-            return [int(v['id']) for v in get_downloaded_info_dicts(params)]
+        def test_selection(params, expected_ids):
+            results = [
+                (v['playlist_autonumber'] - 1, (int(v['id']), v['playlist_index']))
+                for v in get_downloaded_info_dicts(params)]
+            self.assertEqual(results, list(enumerate(zip(expected_ids, expected_ids))))
 
-        result = get_ids({})
-        self.assertEqual(result, [1, 2, 3, 4])
-
-        result = get_ids({'playlistend': 10})
-        self.assertEqual(result, [1, 2, 3, 4])
-
-        result = get_ids({'playlistend': 2})
-        self.assertEqual(result, [1, 2])
-
-        result = get_ids({'playliststart': 10})
-        self.assertEqual(result, [])
-
-        result = get_ids({'playliststart': 2})
-        self.assertEqual(result, [2, 3, 4])
-
-        result = get_ids({'playlist_items': '2-4'})
-        self.assertEqual(result, [2, 3, 4])
-
-        result = get_ids({'playlist_items': '2,4'})
-        self.assertEqual(result, [2, 4])
-
-        result = get_ids({'playlist_items': '10'})
-        self.assertEqual(result, [])
-
-        result = get_ids({'playlist_items': '3-10'})
-        self.assertEqual(result, [3, 4])
-
-        result = get_ids({'playlist_items': '2-4,3-4,3'})
-        self.assertEqual(result, [2, 3, 4])
+        test_selection({}, [1, 2, 3, 4])
+        test_selection({'playlistend': 10}, [1, 2, 3, 4])
+        test_selection({'playlistend': 2}, [1, 2])
+        test_selection({'playliststart': 10}, [])
+        test_selection({'playliststart': 2}, [2, 3, 4])
+        test_selection({'playlist_items': '2-4'}, [2, 3, 4])
+        test_selection({'playlist_items': '2,4'}, [2, 4])
+        test_selection({'playlist_items': '10'}, [])
 
         # Tests for https://github.com/ytdl-org/youtube-dl/issues/10591
-        # @{
-        result = get_downloaded_info_dicts({'playlist_items': '2-4,3-4,3'})
-        self.assertEqual(result[0]['playlist_index'], 2)
-        self.assertEqual(result[1]['playlist_index'], 3)
+        test_selection({'playlist_items': '2-4,3-4,3'}, [2, 3, 4])
+        test_selection({'playlist_items': '4,2'}, [4, 2])
 
-        result = get_downloaded_info_dicts({'playlist_items': '2-4,3-4,3'})
-        self.assertEqual(result[0]['playlist_index'], 2)
-        self.assertEqual(result[1]['playlist_index'], 3)
-        self.assertEqual(result[2]['playlist_index'], 4)
-
-        result = get_downloaded_info_dicts({'playlist_items': '4,2'})
-        self.assertEqual(result[0]['playlist_index'], 4)
-        self.assertEqual(result[1]['playlist_index'], 2)
-        # @}
+        # Tests for https://github.com/yt-dlp/yt-dlp/issues/720
+        # https://github.com/yt-dlp/yt-dlp/issues/302
+        test_selection({'playlistreverse': True}, [4, 3, 2, 1])
+        test_selection({'playliststart': 2, 'playlistreverse': True}, [4, 3, 2])
+        test_selection({'playlist_items': '2,4', 'playlistreverse': True}, [4, 2])
+        test_selection({'playlist_items': '4,2'}, [4, 2])
 
     def test_urlopen_no_file_protocol(self):
         # see https://github.com/ytdl-org/youtube-dl/issues/8227

test/test_download.py

@@ -73,6 +73,8 @@ class TestDownload(unittest.TestCase):
 
     maxDiff = None
 
+    COMPLETED_TESTS = {}
+
     def __str__(self):
         """Identify each test with the `add_ie` attribute, if available."""
 
@@ -94,6 +96,9 @@ class TestDownload(unittest.TestCase):
 def generator(test_case, tname):
 
     def test_template(self):
+        if self.COMPLETED_TESTS.get(tname):
+            return
+        self.COMPLETED_TESTS[tname] = True
         ie = yt_dlp.extractor.get_info_extractor(test_case['name'])()
         other_ies = [get_info_extractor(ie_key)() for ie_key in test_case.get('add_ie', [])]
         is_playlist = any(k.startswith('playlist') for k in test_case)
@@ -108,8 +113,13 @@ def generator(test_case, tname):
 
         for tc in test_cases:
             info_dict = tc.get('info_dict', {})
-            if not (info_dict.get('id') and info_dict.get('ext')):
-                raise Exception('Test definition incorrect. The output file cannot be known. Are both \'id\' and \'ext\' keys present?')
+            params = tc.get('params', {})
+            if not info_dict.get('id'):
+                raise Exception('Test definition incorrect. \'id\' key is not present')
+            elif not info_dict.get('ext'):
+                if params.get('skip_download') and params.get('ignore_no_formats_error'):
+                    continue
+                raise Exception('Test definition incorrect. The output file cannot be known. \'ext\' key is not present')
 
         if 'skip' in test_case:
             print_skipping(test_case['skip'])
@@ -137,7 +147,7 @@ def generator(test_case, tname):
         expect_warnings(ydl, test_case.get('expected_warnings', []))
 
         def get_tc_filename(tc):
-            return ydl.prepare_filename(tc.get('info_dict', {}))
+            return ydl.prepare_filename(dict(tc.get('info_dict', {})))
 
         res_dict = None
 
@@ -250,12 +260,12 @@ def generator(test_case, tname):
 
 
 # And add them to TestDownload
-for n, test_case in enumerate(defs):
-    tname = 'test_' + str(test_case['name'])
-    i = 1
-    while hasattr(TestDownload, tname):
-        tname = 'test_%s_%d' % (test_case['name'], i)
-        i += 1
+tests_counter = {}
+for test_case in defs:
+    name = test_case['name']
+    i = tests_counter.get(name, 0)
+    tests_counter[name] = i + 1
+    tname = f'test_{name}_{i}' if i else f'test_{name}'
     test_method = generator(test_case, tname)
     test_method.__name__ = str(tname)
     ie_list = test_case.get('add_ie')
@@ -264,5 +274,22 @@ for n, test_case in enumerate(defs):
     del test_method
 
 
+def batch_generator(name, num_tests):
+
+    def test_template(self):
+        for i in range(num_tests):
+            getattr(self, f'test_{name}_{i}' if i else f'test_{name}')()
+
+    return test_template
+
+
+for name, num_tests in tests_counter.items():
+    test_method = batch_generator(name, num_tests)
+    test_method.__name__ = f'test_{name}_all'
+    test_method.add_ie = ''
+    setattr(TestDownload, test_method.__name__, test_method)
+    del test_method
+
+
 if __name__ == '__main__':
     unittest.main()

test/test_overwrites.py

@@ -8,13 +8,14 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import try_rm
+from test.helper import is_download_test, try_rm
 
 
 root_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 download_file = join(root_dir, 'test.webm')
 
 
+@is_download_test
 class TestOverwrites(unittest.TestCase):
     def setUp(self):
         # create an empty file

test/test_postprocessors.py

@@ -6,37 +6,38 @@ from __future__ import unicode_literals
 import os
 import sys
 import unittest
+
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from yt_dlp import YoutubeDL
 from yt_dlp.compat import compat_shlex_quote
 from yt_dlp.postprocessor import (
-    ExecAfterDownloadPP,
+    ExecPP,
     FFmpegThumbnailsConvertorPP,
     MetadataFromFieldPP,
-    MetadataFromTitlePP,
+    MetadataParserPP,
+    ModifyChaptersPP
 )
 
 
 class TestMetadataFromField(unittest.TestCase):
+
     def test_format_to_regex(self):
-        pp = MetadataFromFieldPP(None, ['title:%(title)s - %(artist)s'])
-        self.assertEqual(pp._data[0]['regex'], r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+        self.assertEqual(
+            MetadataParserPP.format_to_regex('%(title)s - %(artist)s'),
+            r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+        self.assertEqual(MetadataParserPP.format_to_regex(r'(?P<x>.+)'), r'(?P<x>.+)')
 
-    def test_field_to_outtmpl(self):
-        pp = MetadataFromFieldPP(None, ['title:%(title)s : %(artist)s'])
-        self.assertEqual(pp._data[0]['tmpl'], '%(title)s')
+    def test_field_to_template(self):
+        self.assertEqual(MetadataParserPP.field_to_template('title'), '%(title)s')
+        self.assertEqual(MetadataParserPP.field_to_template('1'), '1')
+        self.assertEqual(MetadataParserPP.field_to_template('foo bar'), 'foo bar')
+        self.assertEqual(MetadataParserPP.field_to_template(' literal'), ' literal')
 
-    def test_in_out_seperation(self):
-        pp = MetadataFromFieldPP(None, ['%(title)s \\: %(artist)s:%(title)s : %(artist)s'])
-        self.assertEqual(pp._data[0]['in'], '%(title)s : %(artist)s')
-        self.assertEqual(pp._data[0]['out'], '%(title)s : %(artist)s')
-
-
-class TestMetadataFromTitle(unittest.TestCase):
-    def test_format_to_regex(self):
-        pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
-        self.assertEqual(pp._titleregex, r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+    def test_metadatafromfield(self):
+        self.assertEqual(
+            MetadataFromFieldPP.to_action('%(title)s \\: %(artist)s:%(title)s : %(artist)s'),
+            (MetadataParserPP.Actions.INTERPRET, '%(title)s : %(artist)s', '%(title)s : %(artist)s'))
 
 
 class TestConvertThumbnail(unittest.TestCase):
@@ -60,12 +61,470 @@ class TestConvertThumbnail(unittest.TestCase):
             os.remove(file.format(out))
 
 
-class TestExecAfterDownload(unittest.TestCase):
+class TestExec(unittest.TestCase):
     def test_parse_cmd(self):
-        pp = ExecAfterDownloadPP(YoutubeDL(), '')
+        pp = ExecPP(YoutubeDL(), '')
         info = {'filepath': 'file name'}
-        quoted_filepath = compat_shlex_quote(info['filepath'])
+        cmd = 'echo %s' % compat_shlex_quote(info['filepath'])
 
-        self.assertEqual(pp.parse_cmd('echo', info), 'echo %s' % quoted_filepath)
-        self.assertEqual(pp.parse_cmd('echo.{}', info), 'echo.%s' % quoted_filepath)
-        self.assertEqual(pp.parse_cmd('echo "%(filepath)s"', info), 'echo "%s"' % info['filepath'])
+        self.assertEqual(pp.parse_cmd('echo', info), cmd)
+        self.assertEqual(pp.parse_cmd('echo {}', info), cmd)
+        self.assertEqual(pp.parse_cmd('echo %(filepath)q', info), cmd)
+
+
+class TestModifyChaptersPP(unittest.TestCase):
+    def setUp(self):
+        self._pp = ModifyChaptersPP(YoutubeDL())
+
+    @staticmethod
+    def _sponsor_chapter(start, end, cat, remove=False):
+        c = {'start_time': start, 'end_time': end, '_categories': [(cat, start, end)]}
+        if remove:
+            c['remove'] = True
+        return c
+
+    @staticmethod
+    def _chapter(start, end, title=None, remove=False):
+        c = {'start_time': start, 'end_time': end}
+        if title is not None:
+            c['title'] = title
+        if remove:
+            c['remove'] = True
+        return c
+
+    def _chapters(self, ends, titles):
+        self.assertEqual(len(ends), len(titles))
+        start = 0
+        chapters = []
+        for e, t in zip(ends, titles):
+            chapters.append(self._chapter(start, e, t))
+            start = e
+        return chapters
+
+    def _remove_marked_arrange_sponsors_test_impl(
+            self, chapters, expected_chapters, expected_removed):
+        actual_chapters, actual_removed = (
+            self._pp._remove_marked_arrange_sponsors(chapters))
+        for c in actual_removed:
+            c.pop('title', None)
+            c.pop('_categories', None)
+        actual_chapters = [{
+            'start_time': c['start_time'],
+            'end_time': c['end_time'],
+            'title': c['title'],
+        } for c in actual_chapters]
+        self.assertSequenceEqual(expected_chapters, actual_chapters)
+        self.assertSequenceEqual(expected_removed, actual_removed)
+
+    def test_remove_marked_arrange_sponsors_CanGetThroughUnaltered(self):
+        chapters = self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, chapters, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithSponsors(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 20, 'sponsor'),
+            self._sponsor_chapter(30, 40, 'preview'),
+            self._sponsor_chapter(50, 60, 'sponsor')]
+        expected = self._chapters(
+            [10, 20, 30, 40, 50, 60, 70],
+            ['c', '[SponsorBlock]: Sponsor', 'c', '[SponsorBlock]: Preview/Recap',
+             'c', '[SponsorBlock]: Sponsor', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_UniqueNamesForOverlappingSponsors(self):
+        chapters = self._chapters([120], ['c']) + [
+            self._sponsor_chapter(10, 45, 'sponsor'), self._sponsor_chapter(20, 40, 'selfpromo'),
+            self._sponsor_chapter(50, 70, 'sponsor'), self._sponsor_chapter(60, 85, 'selfpromo'),
+            self._sponsor_chapter(90, 120, 'selfpromo'), self._sponsor_chapter(100, 110, 'sponsor')]
+        expected = self._chapters(
+            [10, 20, 40, 45, 50, 60, 70, 85, 90, 100, 110, 120],
+            ['c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Sponsor, Unpaid/Self Promotion',
+             '[SponsorBlock]: Sponsor',
+             'c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Sponsor, Unpaid/Self Promotion',
+             '[SponsorBlock]: Unpaid/Self Promotion',
+             'c', '[SponsorBlock]: Unpaid/Self Promotion', '[SponsorBlock]: Unpaid/Self Promotion, Sponsor',
+             '[SponsorBlock]: Unpaid/Self Promotion'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithCuts(self):
+        cuts = [self._chapter(10, 20, remove=True),
+                self._sponsor_chapter(30, 40, 'sponsor', remove=True),
+                self._chapter(50, 60, remove=True)]
+        chapters = self._chapters([70], ['c']) + cuts
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([40], ['c']), cuts)
+
+    def test_remove_marked_arrange_sponsors_ChapterWithSponsorsAndCuts(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 20, 'sponsor'),
+            self._sponsor_chapter(30, 40, 'selfpromo', remove=True),
+            self._sponsor_chapter(50, 60, 'interaction')]
+        expected = self._chapters([10, 20, 40, 50, 60],
+                                  ['c', '[SponsorBlock]: Sponsor', 'c',
+                                   '[SponsorBlock]: Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(30, 40, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithSponsorCutInTheMiddle(self):
+        cuts = [self._sponsor_chapter(20, 30, 'selfpromo', remove=True),
+                self._chapter(40, 50, remove=True)]
+        chapters = self._chapters([70], ['c']) + [self._sponsor_chapter(10, 60, 'sponsor')] + cuts
+        expected = self._chapters(
+            [10, 40, 50], ['c', '[SponsorBlock]: Sponsor', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_ChapterWithCutHidingSponsor(self):
+        cuts = [self._sponsor_chapter(20, 50, 'selpromo', remove=True)]
+        chapters = self._chapters([60], ['c']) + [
+            self._sponsor_chapter(10, 20, 'intro'),
+            self._sponsor_chapter(30, 40, 'sponsor'),
+            self._sponsor_chapter(50, 60, 'outro'),
+        ] + cuts
+        expected = self._chapters(
+            [10, 20, 30], ['c', '[SponsorBlock]: Intermission/Intro Animation', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_ChapterWithAdjacentSponsors(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 20, 'sponsor'),
+            self._sponsor_chapter(20, 30, 'selfpromo'),
+            self._sponsor_chapter(30, 40, 'interaction')]
+        expected = self._chapters(
+            [10, 20, 30, 40, 70],
+            ['c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Unpaid/Self Promotion',
+             '[SponsorBlock]: Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithAdjacentCuts(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 20, 'sponsor'),
+            self._sponsor_chapter(20, 30, 'interaction', remove=True),
+            self._chapter(30, 40, remove=True),
+            self._sponsor_chapter(40, 50, 'selpromo', remove=True),
+            self._sponsor_chapter(50, 60, 'interaction')]
+        expected = self._chapters([10, 20, 30, 40],
+                                  ['c', '[SponsorBlock]: Sponsor',
+                                   '[SponsorBlock]: Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(20, 50, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithOverlappingSponsors(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 30, 'sponsor'),
+            self._sponsor_chapter(20, 50, 'selfpromo'),
+            self._sponsor_chapter(40, 60, 'interaction')]
+        expected = self._chapters(
+            [10, 20, 30, 40, 50, 60, 70],
+            ['c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Sponsor, Unpaid/Self Promotion',
+             '[SponsorBlock]: Unpaid/Self Promotion', '[SponsorBlock]: Unpaid/Self Promotion, Interaction Reminder',
+             '[SponsorBlock]: Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithOverlappingCuts(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 30, 'sponsor', remove=True),
+            self._sponsor_chapter(20, 50, 'selfpromo', remove=True),
+            self._sponsor_chapter(40, 60, 'interaction', remove=True)]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([20], ['c']), [self._chapter(10, 60, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithRunsOfOverlappingSponsors(self):
+        chapters = self._chapters([170], ['c']) + [
+            self._sponsor_chapter(0, 30, 'intro'),
+            self._sponsor_chapter(20, 50, 'sponsor'),
+            self._sponsor_chapter(40, 60, 'selfpromo'),
+            self._sponsor_chapter(70, 90, 'sponsor'),
+            self._sponsor_chapter(80, 100, 'sponsor'),
+            self._sponsor_chapter(90, 110, 'sponsor'),
+            self._sponsor_chapter(120, 140, 'selfpromo'),
+            self._sponsor_chapter(130, 160, 'interaction'),
+            self._sponsor_chapter(150, 170, 'outro')]
+        expected = self._chapters(
+            [20, 30, 40, 50, 60, 70, 110, 120, 130, 140, 150, 160, 170],
+            ['[SponsorBlock]: Intermission/Intro Animation', '[SponsorBlock]: Intermission/Intro Animation, Sponsor', '[SponsorBlock]: Sponsor',
+             '[SponsorBlock]: Sponsor, Unpaid/Self Promotion', '[SponsorBlock]: Unpaid/Self Promotion', 'c',
+             '[SponsorBlock]: Sponsor', 'c', '[SponsorBlock]: Unpaid/Self Promotion',
+             '[SponsorBlock]: Unpaid/Self Promotion, Interaction Reminder',
+             '[SponsorBlock]: Interaction Reminder',
+             '[SponsorBlock]: Interaction Reminder, Endcards/Credits', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithRunsOfOverlappingCuts(self):
+        chapters = self._chapters([170], ['c']) + [
+            self._chapter(0, 30, remove=True),
+            self._sponsor_chapter(20, 50, 'sponsor', remove=True),
+            self._chapter(40, 60, remove=True),
+            self._sponsor_chapter(70, 90, 'sponsor', remove=True),
+            self._chapter(80, 100, remove=True),
+            self._chapter(90, 110, remove=True),
+            self._sponsor_chapter(120, 140, 'sponsor', remove=True),
+            self._sponsor_chapter(130, 160, 'selfpromo', remove=True),
+            self._chapter(150, 170, remove=True)]
+        expected_cuts = [self._chapter(0, 60, remove=True),
+                         self._chapter(70, 110, remove=True),
+                         self._chapter(120, 170, remove=True)]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([20], ['c']), expected_cuts)
+
+    def test_remove_marked_arrange_sponsors_OverlappingSponsorsDifferentTitlesAfterCut(self):
+        chapters = self._chapters([60], ['c']) + [
+            self._sponsor_chapter(10, 60, 'sponsor'),
+            self._sponsor_chapter(10, 40, 'intro'),
+            self._sponsor_chapter(30, 50, 'interaction'),
+            self._sponsor_chapter(30, 50, 'selfpromo', remove=True),
+            self._sponsor_chapter(40, 50, 'interaction'),
+            self._sponsor_chapter(50, 60, 'outro')]
+        expected = self._chapters(
+            [10, 30, 40], ['c', '[SponsorBlock]: Sponsor, Intermission/Intro Animation', '[SponsorBlock]: Sponsor, Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(30, 50, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_SponsorsNoLongerOverlapAfterCut(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 30, 'sponsor'),
+            self._sponsor_chapter(20, 50, 'interaction'),
+            self._sponsor_chapter(30, 50, 'selpromo', remove=True),
+            self._sponsor_chapter(40, 60, 'sponsor'),
+            self._sponsor_chapter(50, 60, 'interaction')]
+        expected = self._chapters(
+            [10, 20, 40, 50], ['c', '[SponsorBlock]: Sponsor',
+                               '[SponsorBlock]: Sponsor, Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(30, 50, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_SponsorsStillOverlapAfterCut(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 60, 'sponsor'),
+            self._sponsor_chapter(20, 60, 'interaction'),
+            self._sponsor_chapter(30, 50, 'selfpromo', remove=True)]
+        expected = self._chapters(
+            [10, 20, 40, 50], ['c', '[SponsorBlock]: Sponsor',
+                               '[SponsorBlock]: Sponsor, Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(30, 50, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithRunsOfOverlappingSponsorsAndCuts(self):
+        chapters = self._chapters([200], ['c']) + [
+            self._sponsor_chapter(10, 40, 'sponsor'),
+            self._sponsor_chapter(10, 30, 'intro'),
+            self._chapter(20, 30, remove=True),
+            self._sponsor_chapter(30, 40, 'selfpromo'),
+            self._sponsor_chapter(50, 70, 'sponsor'),
+            self._sponsor_chapter(60, 80, 'interaction'),
+            self._chapter(70, 80, remove=True),
+            self._sponsor_chapter(70, 90, 'sponsor'),
+            self._sponsor_chapter(80, 100, 'interaction'),
+            self._sponsor_chapter(120, 170, 'selfpromo'),
+            self._sponsor_chapter(130, 180, 'outro'),
+            self._chapter(140, 150, remove=True),
+            self._chapter(150, 160, remove=True)]
+        expected = self._chapters(
+            [10, 20, 30, 40, 50, 70, 80, 100, 110, 130, 140, 160],
+            ['c', '[SponsorBlock]: Sponsor, Intermission/Intro Animation', '[SponsorBlock]: Sponsor, Unpaid/Self Promotion',
+             'c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Sponsor, Interaction Reminder',
+             '[SponsorBlock]: Interaction Reminder', 'c', '[SponsorBlock]: Unpaid/Self Promotion',
+             '[SponsorBlock]: Unpaid/Self Promotion, Endcards/Credits', '[SponsorBlock]: Endcards/Credits', 'c'])
+        expected_cuts = [self._chapter(20, 30, remove=True),
+                         self._chapter(70, 80, remove=True),
+                         self._chapter(140, 160, remove=True)]
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, expected_cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorOverlapsMultipleChapters(self):
+        chapters = (self._chapters([20, 40, 60, 80, 100], ['c1', 'c2', 'c3', 'c4', 'c5'])
+                    + [self._sponsor_chapter(10, 90, 'sponsor')])
+        expected = self._chapters([10, 90, 100], ['c1', '[SponsorBlock]: Sponsor', 'c5'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutOverlapsMultipleChapters(self):
+        cuts = [self._chapter(10, 90, remove=True)]
+        chapters = self._chapters([20, 40, 60, 80, 100], ['c1', 'c2', 'c3', 'c4', 'c5']) + cuts
+        expected = self._chapters([10, 20], ['c1', 'c5'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorsWithinSomeChaptersAndOverlappingOthers(self):
+        chapters = (self._chapters([10, 40, 60, 80], ['c1', 'c2', 'c3', 'c4'])
+                    + [self._sponsor_chapter(20, 30, 'sponsor'),
+                       self._sponsor_chapter(50, 70, 'selfpromo')])
+        expected = self._chapters([10, 20, 30, 40, 50, 70, 80],
+                                  ['c1', 'c2', '[SponsorBlock]: Sponsor', 'c2', 'c3',
+                                   '[SponsorBlock]: Unpaid/Self Promotion', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutsWithinSomeChaptersAndOverlappingOthers(self):
+        cuts = [self._chapter(20, 30, remove=True), self._chapter(50, 70, remove=True)]
+        chapters = self._chapters([10, 40, 60, 80], ['c1', 'c2', 'c3', 'c4']) + cuts
+        expected = self._chapters([10, 30, 40, 50], ['c1', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_ChaptersAfterLastSponsor(self):
+        chapters = (self._chapters([20, 40, 50, 60], ['c1', 'c2', 'c3', 'c4'])
+                    + [self._sponsor_chapter(10, 30, 'music_offtopic')])
+        expected = self._chapters(
+            [10, 30, 40, 50, 60],
+            ['c1', '[SponsorBlock]: Non-Music Section', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChaptersAfterLastCut(self):
+        cuts = [self._chapter(10, 30, remove=True)]
+        chapters = self._chapters([20, 40, 50, 60], ['c1', 'c2', 'c3', 'c4']) + cuts
+        expected = self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorStartsAtChapterStart(self):
+        chapters = (self._chapters([10, 20, 40], ['c1', 'c2', 'c3'])
+                    + [self._sponsor_chapter(20, 30, 'sponsor')])
+        expected = self._chapters([10, 20, 30, 40], ['c1', 'c2', '[SponsorBlock]: Sponsor', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutStartsAtChapterStart(self):
+        cuts = [self._chapter(20, 30, remove=True)]
+        chapters = self._chapters([10, 20, 40], ['c1', 'c2', 'c3']) + cuts
+        expected = self._chapters([10, 20, 30], ['c1', 'c2', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorEndsAtChapterEnd(self):
+        chapters = (self._chapters([10, 30, 40], ['c1', 'c2', 'c3'])
+                    + [self._sponsor_chapter(20, 30, 'sponsor')])
+        expected = self._chapters([10, 20, 30, 40], ['c1', 'c2', '[SponsorBlock]: Sponsor', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutEndsAtChapterEnd(self):
+        cuts = [self._chapter(20, 30, remove=True)]
+        chapters = self._chapters([10, 30, 40], ['c1', 'c2', 'c3']) + cuts
+        expected = self._chapters([10, 20, 30], ['c1', 'c2', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorCoincidesWithChapters(self):
+        chapters = (self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4'])
+                    + [self._sponsor_chapter(10, 30, 'sponsor')])
+        expected = self._chapters([10, 30, 40], ['c1', '[SponsorBlock]: Sponsor', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutCoincidesWithChapters(self):
+        cuts = [self._chapter(10, 30, remove=True)]
+        chapters = self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4']) + cuts
+        expected = self._chapters([10, 20], ['c1', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorsAtVideoBoundaries(self):
+        chapters = (self._chapters([20, 40, 60], ['c1', 'c2', 'c3'])
+                    + [self._sponsor_chapter(0, 10, 'intro'), self._sponsor_chapter(50, 60, 'outro')])
+        expected = self._chapters(
+            [10, 20, 40, 50, 60], ['[SponsorBlock]: Intermission/Intro Animation', 'c1', 'c2', 'c3', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutsAtVideoBoundaries(self):
+        cuts = [self._chapter(0, 10, remove=True), self._chapter(50, 60, remove=True)]
+        chapters = self._chapters([20, 40, 60], ['c1', 'c2', 'c3']) + cuts
+        expected = self._chapters([10, 30, 40], ['c1', 'c2', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorsOverlapChaptersAtVideoBoundaries(self):
+        chapters = (self._chapters([10, 40, 50], ['c1', 'c2', 'c3'])
+                    + [self._sponsor_chapter(0, 20, 'intro'), self._sponsor_chapter(30, 50, 'outro')])
+        expected = self._chapters(
+            [20, 30, 50], ['[SponsorBlock]: Intermission/Intro Animation', 'c2', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutsOverlapChaptersAtVideoBoundaries(self):
+        cuts = [self._chapter(0, 20, remove=True), self._chapter(30, 50, remove=True)]
+        chapters = self._chapters([10, 40, 50], ['c1', 'c2', 'c3']) + cuts
+        expected = self._chapters([10], ['c2'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_EverythingSponsored(self):
+        chapters = (self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4'])
+                    + [self._sponsor_chapter(0, 20, 'intro'), self._sponsor_chapter(20, 40, 'outro')])
+        expected = self._chapters([20, 40], ['[SponsorBlock]: Intermission/Intro Animation', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_EverythingCut(self):
+        cuts = [self._chapter(0, 20, remove=True), self._chapter(20, 40, remove=True)]
+        chapters = self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4']) + cuts
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, [], [self._chapter(0, 40, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_TinyChaptersInTheOriginalArePreserved(self):
+        chapters = self._chapters([0.1, 0.2, 0.3, 0.4], ['c1', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, chapters, [])
+
+    def test_remove_marked_arrange_sponsors_TinySponsorsAreIgnored(self):
+        chapters = [self._sponsor_chapter(0, 0.1, 'intro'), self._chapter(0.1, 0.2, 'c1'),
+                    self._sponsor_chapter(0.2, 0.3, 'sponsor'), self._chapter(0.3, 0.4, 'c2'),
+                    self._sponsor_chapter(0.4, 0.5, 'outro')]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([0.3, 0.5], ['c1', 'c2']), [])
+
+    def test_remove_marked_arrange_sponsors_TinyChaptersResultingFromCutsAreIgnored(self):
+        cuts = [self._chapter(1.5, 2.5, remove=True)]
+        chapters = self._chapters([2, 3, 3.5], ['c1', 'c2', 'c3']) + cuts
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([2, 2.5], ['c1', 'c3']), cuts)
+
+    def test_remove_marked_arrange_sponsors_TinyChaptersResultingFromSponsorOverlapAreIgnored(self):
+        chapters = self._chapters([1, 3, 4], ['c1', 'c2', 'c3']) + [
+            self._sponsor_chapter(1.5, 2.5, 'sponsor')]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([1.5, 3, 4], ['c1', '[SponsorBlock]: Sponsor', 'c3']), [])
+
+    def test_remove_marked_arrange_sponsors_TinySponsorsOverlapsAreIgnored(self):
+        chapters = self._chapters([2, 3, 5], ['c1', 'c2', 'c3']) + [
+            self._sponsor_chapter(1, 3, 'sponsor'),
+            self._sponsor_chapter(2.5, 4, 'selfpromo')
+        ]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([1, 3, 4, 5], [
+                'c1', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Unpaid/Self Promotion', 'c3']), [])
+
+    def test_make_concat_opts_CommonCase(self):
+        sponsor_chapters = [self._chapter(1, 2, 's1'), self._chapter(10, 20, 's2')]
+        expected = '''ffconcat version 1.0
+file 'file:test'
+outpoint 1.000000
+file 'file:test'
+inpoint 2.000000
+outpoint 10.000000
+file 'file:test'
+inpoint 20.000000
+'''
+        opts = self._pp._make_concat_opts(sponsor_chapters, 30)
+        self.assertEqual(expected, ''.join(self._pp._concat_spec(['test'] * len(opts), opts)))
+
+    def test_make_concat_opts_NoZeroDurationChunkAtVideoStart(self):
+        sponsor_chapters = [self._chapter(0, 1, 's1'), self._chapter(10, 20, 's2')]
+        expected = '''ffconcat version 1.0
+file 'file:test'
+inpoint 1.000000
+outpoint 10.000000
+file 'file:test'
+inpoint 20.000000
+'''
+        opts = self._pp._make_concat_opts(sponsor_chapters, 30)
+        self.assertEqual(expected, ''.join(self._pp._concat_spec(['test'] * len(opts), opts)))
+
+    def test_make_concat_opts_NoZeroDurationChunkAtVideoEnd(self):
+        sponsor_chapters = [self._chapter(1, 2, 's1'), self._chapter(10, 20, 's2')]
+        expected = '''ffconcat version 1.0
+file 'file:test'
+outpoint 1.000000
+file 'file:test'
+inpoint 2.000000
+outpoint 10.000000
+'''
+        opts = self._pp._make_concat_opts(sponsor_chapters, 20)
+        self.assertEqual(expected, ''.join(self._pp._concat_spec(['test'] * len(opts), opts)))
+
+    def test_quote_for_concat_RunsOfQuotes(self):
+        self.assertEqual(
+            r"'special '\'' '\'\''characters'\'\'\''galore'",
+            self._pp._quote_for_ffmpeg("special ' ''characters'''galore"))
+
+    def test_quote_for_concat_QuotesAtStart(self):
+        self.assertEqual(
+            r"\'\'\''special '\'' characters '\'' galore'",
+            self._pp._quote_for_ffmpeg("'''special ' characters ' galore"))
+
+    def test_quote_for_concat_QuotesAtEnd(self):
+        self.assertEqual(
+            r"'special '\'' characters '\'' galore'\'\'\'",
+            self._pp._quote_for_ffmpeg("special ' characters ' galore'''"))

test/test_utils.py

@@ -62,6 +62,7 @@ from yt_dlp.utils import (
     parse_iso8601,
     parse_resolution,
     parse_bitrate,
+    parse_qs,
     pkcs1pad,
     read_batch_urls,
     sanitize_filename,
@@ -117,8 +118,6 @@ from yt_dlp.compat import (
     compat_getenv,
     compat_os_name,
     compat_setenv,
-    compat_urlparse,
-    compat_parse_qs,
 )
 
 
@@ -688,38 +687,36 @@ class TestUtil(unittest.TestCase):
         self.assertTrue(isinstance(data, bytes))
 
     def test_update_url_query(self):
-        def query_dict(url):
-            return compat_parse_qs(compat_urlparse.urlparse(url).query)
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'quality': ['HD'], 'format': ['mp4']})),
-            query_dict('http://example.com/path?quality=HD&format=mp4'))
-        self.assertEqual(query_dict(update_url_query(
+            parse_qs('http://example.com/path?quality=HD&format=mp4'))
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'system': ['LINUX', 'WINDOWS']})),
-            query_dict('http://example.com/path?system=LINUX&system=WINDOWS'))
-        self.assertEqual(query_dict(update_url_query(
+            parse_qs('http://example.com/path?system=LINUX&system=WINDOWS'))
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'fields': 'id,formats,subtitles'})),
-            query_dict('http://example.com/path?fields=id,formats,subtitles'))
-        self.assertEqual(query_dict(update_url_query(
+            parse_qs('http://example.com/path?fields=id,formats,subtitles'))
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'fields': ('id,formats,subtitles', 'thumbnails')})),
-            query_dict('http://example.com/path?fields=id,formats,subtitles&fields=thumbnails'))
-        self.assertEqual(query_dict(update_url_query(
+            parse_qs('http://example.com/path?fields=id,formats,subtitles&fields=thumbnails'))
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path?manifest=f4m', {'manifest': []})),
-            query_dict('http://example.com/path'))
-        self.assertEqual(query_dict(update_url_query(
+            parse_qs('http://example.com/path'))
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path?system=LINUX&system=WINDOWS', {'system': 'LINUX'})),
-            query_dict('http://example.com/path?system=LINUX'))
-        self.assertEqual(query_dict(update_url_query(
+            parse_qs('http://example.com/path?system=LINUX'))
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'fields': b'id,formats,subtitles'})),
-            query_dict('http://example.com/path?fields=id,formats,subtitles'))
-        self.assertEqual(query_dict(update_url_query(
+            parse_qs('http://example.com/path?fields=id,formats,subtitles'))
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'width': 1080, 'height': 720})),
-            query_dict('http://example.com/path?width=1080&height=720'))
-        self.assertEqual(query_dict(update_url_query(
+            parse_qs('http://example.com/path?width=1080&height=720'))
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'bitrate': 5020.43})),
-            query_dict('http://example.com/path?bitrate=5020.43'))
-        self.assertEqual(query_dict(update_url_query(
+            parse_qs('http://example.com/path?bitrate=5020.43'))
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'test': '第二行тест'})),
-            query_dict('http://example.com/path?test=%E7%AC%AC%E4%BA%8C%E8%A1%8C%D1%82%D0%B5%D1%81%D1%82'))
+            parse_qs('http://example.com/path?test=%E7%AC%AC%E4%BA%8C%E8%A1%8C%D1%82%D0%B5%D1%81%D1%82'))
 
     def test_multipart_encode(self):
         self.assertEqual(
@@ -1207,35 +1204,12 @@ ffmpeg version 2.4.4 Copyright (c) 2000-2014 the FFmpeg ...'''), '2.4.4')
             '9999 51')
 
     def test_match_str(self):
-        self.assertRaises(ValueError, match_str, 'xy>foobar', {})
+        # Unary
         self.assertFalse(match_str('xy', {'x': 1200}))
         self.assertTrue(match_str('!xy', {'x': 1200}))
         self.assertTrue(match_str('x', {'x': 1200}))
         self.assertFalse(match_str('!x', {'x': 1200}))
         self.assertTrue(match_str('x', {'x': 0}))
-        self.assertFalse(match_str('x>0', {'x': 0}))
-        self.assertFalse(match_str('x>0', {}))
-        self.assertTrue(match_str('x>?0', {}))
-        self.assertTrue(match_str('x>1K', {'x': 1200}))
-        self.assertFalse(match_str('x>2K', {'x': 1200}))
-        self.assertTrue(match_str('x>=1200 & x < 1300', {'x': 1200}))
-        self.assertFalse(match_str('x>=1100 & x < 1200', {'x': 1200}))
-        self.assertFalse(match_str('y=a212', {'y': 'foobar42'}))
-        self.assertTrue(match_str('y=foobar42', {'y': 'foobar42'}))
-        self.assertFalse(match_str('y!=foobar42', {'y': 'foobar42'}))
-        self.assertTrue(match_str('y!=foobar2', {'y': 'foobar42'}))
-        self.assertFalse(match_str(
-            'like_count > 100 & dislike_count <? 50 & description',
-            {'like_count': 90, 'description': 'foo'}))
-        self.assertTrue(match_str(
-            'like_count > 100 & dislike_count <? 50 & description',
-            {'like_count': 190, 'description': 'foo'}))
-        self.assertFalse(match_str(
-            'like_count > 100 & dislike_count <? 50 & description',
-            {'like_count': 190, 'dislike_count': 60, 'description': 'foo'}))
-        self.assertFalse(match_str(
-            'like_count > 100 & dislike_count <? 50 & description',
-            {'like_count': 190, 'dislike_count': 10}))
         self.assertTrue(match_str('is_live', {'is_live': True}))
         self.assertFalse(match_str('is_live', {'is_live': False}))
         self.assertFalse(match_str('is_live', {'is_live': None}))
@@ -1249,6 +1223,75 @@ ffmpeg version 2.4.4 Copyright (c) 2000-2014 the FFmpeg ...'''), '2.4.4')
         self.assertFalse(match_str('!title', {'title': 'abc'}))
         self.assertFalse(match_str('!title', {'title': ''}))
 
+        # Numeric
+        self.assertFalse(match_str('x>0', {'x': 0}))
+        self.assertFalse(match_str('x>0', {}))
+        self.assertTrue(match_str('x>?0', {}))
+        self.assertTrue(match_str('x>1K', {'x': 1200}))
+        self.assertFalse(match_str('x>2K', {'x': 1200}))
+        self.assertTrue(match_str('x>=1200 & x < 1300', {'x': 1200}))
+        self.assertFalse(match_str('x>=1100 & x < 1200', {'x': 1200}))
+
+        # String
+        self.assertFalse(match_str('y=a212', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y=foobar42', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y!=foobar42', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y!=foobar2', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y^=foo', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y!^=foo', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y^=bar', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y!^=bar', {'y': 'foobar42'}))
+        self.assertRaises(ValueError, match_str, 'x^=42', {'x': 42})
+        self.assertTrue(match_str('y*=bar', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y!*=bar', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y*=baz', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y!*=baz', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y$=42', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y$=43', {'y': 'foobar42'}))
+
+        # And
+        self.assertFalse(match_str(
+            'like_count > 100 & dislike_count <? 50 & description',
+            {'like_count': 90, 'description': 'foo'}))
+        self.assertTrue(match_str(
+            'like_count > 100 & dislike_count <? 50 & description',
+            {'like_count': 190, 'description': 'foo'}))
+        self.assertFalse(match_str(
+            'like_count > 100 & dislike_count <? 50 & description',
+            {'like_count': 190, 'dislike_count': 60, 'description': 'foo'}))
+        self.assertFalse(match_str(
+            'like_count > 100 & dislike_count <? 50 & description',
+            {'like_count': 190, 'dislike_count': 10}))
+
+        # Regex
+        self.assertTrue(match_str(r'x~=\bbar', {'x': 'foo bar'}))
+        self.assertFalse(match_str(r'x~=\bbar.+', {'x': 'foo bar'}))
+        self.assertFalse(match_str(r'x~=^FOO', {'x': 'foo bar'}))
+        self.assertTrue(match_str(r'x~=(?i)^FOO', {'x': 'foo bar'}))
+
+        # Quotes
+        self.assertTrue(match_str(r'x^="foo"', {'x': 'foo "bar"'}))
+        self.assertFalse(match_str(r'x^="foo  "', {'x': 'foo "bar"'}))
+        self.assertFalse(match_str(r'x$="bar"', {'x': 'foo "bar"'}))
+        self.assertTrue(match_str(r'x$=" \"bar\""', {'x': 'foo "bar"'}))
+
+        # Escaping &
+        self.assertFalse(match_str(r'x=foo & bar', {'x': 'foo & bar'}))
+        self.assertTrue(match_str(r'x=foo \& bar', {'x': 'foo & bar'}))
+        self.assertTrue(match_str(r'x=foo \& bar & x^=foo', {'x': 'foo & bar'}))
+        self.assertTrue(match_str(r'x="foo \& bar" & x^=foo', {'x': 'foo & bar'}))
+
+        # Example from docs
+        self.assertTrue(match_str(
+            r"!is_live & like_count>?100 & description~='(?i)\bcats \& dogs\b'",
+            {'description': 'Raining Cats & Dogs'}))
+
+        # Incomplete
+        self.assertFalse(match_str('id!=foo', {'id': 'foo'}, True))
+        self.assertTrue(match_str('x', {'id': 'foo'}, True))
+        self.assertTrue(match_str('!x', {'id': 'foo'}, True))
+        self.assertFalse(match_str('x', {'id': 'foo'}, False))
+
     def test_parse_dfxp_time_expr(self):
         self.assertEqual(parse_dfxp_time_expr(None), None)
         self.assertEqual(parse_dfxp_time_expr(''), None)
@@ -1537,8 +1580,11 @@ Line 1
         self.assertEqual(LazyList(it).exhaust(), it)
         self.assertEqual(LazyList(it)[5], it[5])
 
+        self.assertEqual(LazyList(it)[5:], it[5:])
+        self.assertEqual(LazyList(it)[:5], it[:5])
         self.assertEqual(LazyList(it)[::2], it[::2])
         self.assertEqual(LazyList(it)[1::2], it[1::2])
+        self.assertEqual(LazyList(it)[5::-1], it[5::-1])
         self.assertEqual(LazyList(it)[6:2:-2], it[6:2:-2])
         self.assertEqual(LazyList(it)[::-1], it[::-1])
 
@@ -1550,6 +1596,7 @@ Line 1
 
         self.assertEqual(list(LazyList(it).reverse()), it[::-1])
         self.assertEqual(list(LazyList(it).reverse()[1:3:7]), it[::-1][1:3:7])
+        self.assertEqual(list(LazyList(it).reverse()[::-1]), it)
 
     def test_LazyList_laziness(self):
 

yt_dlp/YoutubeDL.py

@@ -35,6 +35,7 @@ from .compat import (
     compat_kwargs,
     compat_numeric_types,
     compat_os_name,
+    compat_shlex_quote,
     compat_str,
     compat_tokenize_tokenize,
     compat_urllib_error,
@@ -65,7 +66,8 @@ from .utils import (
     float_or_none,
     format_bytes,
     format_field,
-    STR_FORMAT_RE,
+    STR_FORMAT_RE_TMPL,
+    STR_FORMAT_TYPES,
     formatSeconds,
     GeoRestrictedError,
     HEADRequest,
@@ -107,6 +109,7 @@ from .utils import (
     try_get,
     UnavailableVideoError,
     url_basename,
+    variadic,
     version_tuple,
     write_json_file,
     write_string,
@@ -123,6 +126,7 @@ from .extractor import (
 )
 from .extractor.openload import PhantomJSwrapper
 from .downloader import (
+    FFmpegFD,
     get_suitable_downloader,
     shorten_protocol_name
 )
@@ -194,7 +198,8 @@ class YoutubeDL(object):
                        (or video) as a single JSON line.
     force_write_download_archive: Force writing download archive regardless
                        of 'skip_download' or 'simulate'.
-    simulate:          Do not download the video files.
+    simulate:          Do not download the video files. If unset (or None),
+                       simulate only if listsubtitles, listformats or list_thumbnails is used
     format:            Video format code. see "FORMAT SELECTION" for more details.
     allow_unplayable_formats:   Allow unplayable formats to be extracted and downloaded.
     ignore_no_formats_error: Ignore "No video formats" error. Usefull for
@@ -215,7 +220,7 @@ class YoutubeDL(object):
                        'temp' and the keys of OUTTMPL_TYPES (in utils.py)
     outtmpl:           Dictionary of templates for output names. Allowed keys
                        are 'default' and the keys of OUTTMPL_TYPES (in utils.py).
-                       A string a also accepted for backward compatibility
+                       For compatibility with youtube-dl, a single string can also be used
     outtmpl_na_placeholder: Placeholder for unavailable meta fields.
     restrictfilenames: Do not allow "&" and spaces in file names
     trim_file_name:    Limit length of filename (extension excluded)
@@ -229,6 +234,8 @@ class YoutubeDL(object):
     overwrites:        Overwrite all video and metadata files if True,
                        overwrite only non-video files if None
                        and don't overwrite any file if False
+                       For compatibility with youtube-dl,
+                       "nooverwrites" may also be used instead
     playliststart:     Playlist item to start at.
     playlistend:       Playlist item to end at.
     playlist_items:    Specific indices of playlist to download.
@@ -241,7 +248,7 @@ class YoutubeDL(object):
     writedescription:  Write the video description to a .description file
     writeinfojson:     Write the video description to a .info.json file
     clean_infojson:    Remove private fields from the infojson
-    writecomments:     Extract video comments. This will not be written to disk
+    getcomments:       Extract video comments. This will not be written to disk
                        unless writeinfojson is also given
     writeannotations:  Write the video annotations to a .annotations.xml file
     writethumbnail:    Write the thumbnail image to a file
@@ -400,7 +407,8 @@ class YoutubeDL(object):
     compat_opts:       Compatibility options. See "Differences in default behavior".
                        The following options do not work when used through the API:
                        filename, abort-on-error, multistreams, no-live-chat,
-                       no-playlist-metafiles. Refer __init__.py for their implementation
+                       no-clean-infojson, no-playlist-metafiles, no-keep-subs.
+                       Refer __init__.py for their implementation
 
     The following parameters are not used by YoutubeDL itself, they are used by
     the downloader (see yt_dlp/downloader/common.py):
@@ -414,10 +422,12 @@ class YoutubeDL(object):
     ffmpeg_location:   Location of the ffmpeg/avconv binary; either the path
                        to the binary or its containing directory.
     postprocessor_args: A dictionary of postprocessor/executable keys (in lower case)
-                        and a list of additional command-line arguments for the
-                        postprocessor/executable. The dict can also have "PP+EXE" keys
-                        which are used when the given exe is used by the given PP.
-                        Use 'default' as the name for arguments to passed to all PP
+                       and a list of additional command-line arguments for the
+                       postprocessor/executable. The dict can also have "PP+EXE" keys
+                       which are used when the given exe is used by the given PP.
+                       Use 'default' as the name for arguments to passed to all PP
+                       For compatibility with youtube-dl, a single list of args
+                       can also be used
 
     The following options are used by the extractors:
     extractor_retries: Number of times to retry for known errors
@@ -451,7 +461,7 @@ class YoutubeDL(object):
     ))
 
     params = None
-    _ies = []
+    _ies = {}
     _pps = {'pre_process': [], 'before_dl': [], 'after_move': [], 'post_process': []}
     _printed_messages = set()
     _first_webpage_request = True
@@ -465,7 +475,7 @@ class YoutubeDL(object):
         """Create a FileDownloader object with the given options."""
         if params is None:
             params = {}
-        self._ies = []
+        self._ies = {}
         self._ies_instances = {}
         self._pps = {'pre_process': [], 'before_dl': [], 'after_move': [], 'post_process': []}
         self._printed_messages = set()
@@ -487,6 +497,12 @@ class YoutubeDL(object):
             self.report_warning(
                 'Python version %d.%d is not supported! Please update to Python 3.6 or above' % sys.version_info[:2])
 
+        if self.params.get('allow_unplayable_formats'):
+            self.report_warning(
+                'You have asked for unplayable formats to be listed/downloaded. '
+                'This is a developer option intended for debugging. '
+                'If you experience any issues while using this option, DO NOT open a bug report')
+
         def check_deprecated(param, option, suggestion):
             if self.params.get(param) is not None:
                 self.report_warning('%s is deprecated. Use %s instead' % (option, suggestion))
@@ -504,13 +520,15 @@ class YoutubeDL(object):
         for msg in self.params.get('warnings', []):
             self.report_warning(msg)
 
-        if self.params.get('final_ext'):
-            if self.params.get('merge_output_format'):
-                self.report_warning('--merge-output-format will be ignored since --remux-video or --recode-video is given')
-            self.params['merge_output_format'] = self.params['final_ext']
-
-        if 'overwrites' in self.params and self.params['overwrites'] is None:
-            del self.params['overwrites']
+        if self.params.get('overwrites') is None:
+            self.params.pop('overwrites', None)
+        elif self.params.get('nooverwrites') is not None:
+            # nooverwrites was unnecessarily changed to overwrites
+            # in 0c3d0f51778b153f65c21906031c2e091fcfb641
+            # This ensures compatibility with both keys
+            self.params['overwrites'] = not self.params['nooverwrites']
+        else:
+            self.params['nooverwrites'] = not self.params['overwrites']
 
         if params.get('bidi_workaround', False):
             try:
@@ -613,11 +631,19 @@ class YoutubeDL(object):
 
     def add_info_extractor(self, ie):
         """Add an InfoExtractor object to the end of the list."""
-        self._ies.append(ie)
+        ie_key = ie.ie_key()
+        self._ies[ie_key] = ie
         if not isinstance(ie, type):
-            self._ies_instances[ie.ie_key()] = ie
+            self._ies_instances[ie_key] = ie
             ie.set_downloader(self)
 
+    def _get_info_extractor_class(self, ie_key):
+        ie = self._ies.get(ie_key)
+        if ie is None:
+            ie = get_info_extractor(ie_key)
+            self.add_info_extractor(ie)
+        return ie
+
     def get_info_extractor(self, ie_key):
         """
         Get an instance of an IE with name ie_key, it will try to get one from
@@ -701,7 +727,7 @@ class YoutubeDL(object):
     def save_console_title(self):
         if not self.params.get('consoletitle', False):
             return
-        if self.params.get('simulate', False):
+        if self.params.get('simulate'):
             return
         if compat_os_name != 'nt' and 'TERM' in os.environ:
             # Save the title on stack
@@ -710,7 +736,7 @@ class YoutubeDL(object):
     def restore_console_title(self):
         if not self.params.get('consoletitle', False):
             return
-        if self.params.get('simulate', False):
+        if self.params.get('simulate'):
             return
         if compat_os_name != 'nt' and 'TERM' in os.environ:
             # Restore the title from stack
@@ -815,6 +841,16 @@ class YoutubeDL(object):
         except UnicodeEncodeError:
             self.to_screen('Deleting existing file')
 
+    def raise_no_formats(self, info, forced=False):
+        has_drm = info.get('__has_drm')
+        msg = 'This video is DRM protected' if has_drm else 'No video formats found!'
+        expected = self.params.get('ignore_no_formats_error')
+        if forced or not expected:
+            raise ExtractorError(msg, video_id=info['id'], ie=info['extractor'],
+                                 expected=has_drm or expected)
+        else:
+            self.report_warning(msg)
+
     def parse_outtmpl(self):
         outtmpl_dict = self.params.get('outtmpl', {})
         if not isinstance(outtmpl_dict, dict):
@@ -845,28 +881,52 @@ class YoutubeDL(object):
         return sanitize_path(path, force=self.params.get('windowsfilenames'))
 
     @staticmethod
-    def validate_outtmpl(tmpl):
+    def _outtmpl_expandpath(outtmpl):
+        # expand_path translates '%%' into '%' and '$$' into '$'
+        # correspondingly that is not what we want since we need to keep
+        # '%%' intact for template dict substitution step. Working around
+        # with boundary-alike separator hack.
+        sep = ''.join([random.choice(ascii_letters) for _ in range(32)])
+        outtmpl = outtmpl.replace('%%', '%{0}%'.format(sep)).replace('$$', '${0}$'.format(sep))
+
+        # outtmpl should be expand_path'ed before template dict substitution
+        # because meta fields may contain env variables we don't want to
+        # be expanded. For example, for outtmpl "%(title)s.%(ext)s" and
+        # title "Hello $PATH", we don't want `$PATH` to be expanded.
+        return expand_path(outtmpl).replace(sep, '')
+
+    @staticmethod
+    def escape_outtmpl(outtmpl):
+        ''' Escape any remaining strings like %s, %abc% etc. '''
+        return re.sub(
+            STR_FORMAT_RE_TMPL.format('', '(?![%(\0])'),
+            lambda mobj: ('' if mobj.group('has_key') else '%') + mobj.group(0),
+            outtmpl)
+
+    @classmethod
+    def validate_outtmpl(cls, outtmpl):
         ''' @return None or Exception object '''
+        outtmpl = re.sub(
+            STR_FORMAT_RE_TMPL.format('[^)]*', '[ljq]'),
+            lambda mobj: f'{mobj.group(0)[:-1]}s',
+            cls._outtmpl_expandpath(outtmpl))
         try:
-            re.sub(
-                STR_FORMAT_RE.format(''),
-                lambda mobj: ('%' if not mobj.group('has_key') else '') + mobj.group(0),
-                tmpl
-            ) % collections.defaultdict(int)
+            cls.escape_outtmpl(outtmpl) % collections.defaultdict(int)
             return None
         except ValueError as err:
             return err
 
     def prepare_outtmpl(self, outtmpl, info_dict, sanitize=None):
-        """ Make the template and info_dict suitable for substitution (outtmpl % info_dict)"""
-        info_dict = dict(info_dict)
-        na = self.params.get('outtmpl_na_placeholder', 'NA')
+        """ Make the template and info_dict suitable for substitution : ydl.outtmpl_escape(outtmpl) % info_dict """
+        info_dict.setdefault('epoch', int(time.time()))  # keep epoch consistent once set
 
+        info_dict = dict(info_dict)  # Do not sanitize so as not to consume LazyList
+        for key in ('__original_infodict', '__postprocessors'):
+            info_dict.pop(key, None)
         info_dict['duration_string'] = (  # %(duration>%H-%M-%S)s is wrong if duration > 24hrs
             formatSeconds(info_dict['duration'], '-' if sanitize else ':')
             if info_dict.get('duration', None) is not None
             else None)
-        info_dict['epoch'] = int(time.time())
         info_dict['autonumber'] = self.params.get('autonumber_start', 1) - 1 + self._num_downloads
         if info_dict.get('resolution') is None:
             info_dict['resolution'] = self.format_resolution(info_dict, default=None)
@@ -879,14 +939,14 @@ class YoutubeDL(object):
         }
 
         TMPL_DICT = {}
-        EXTERNAL_FORMAT_RE = re.compile(STR_FORMAT_RE.format('[^)]*'))
+        EXTERNAL_FORMAT_RE = re.compile(STR_FORMAT_RE_TMPL.format('[^)]*', f'[{STR_FORMAT_TYPES}ljq]'))
         MATH_FUNCTIONS = {
             '+': float.__add__,
             '-': float.__sub__,
         }
         # Field is of the form key1.key2...
         # where keys (except first) can be string, int or slice
-        FIELD_RE = r'\w+(?:\.(?:\w+|{num}|{num}?(?::{num}?){{1,2}}))*'.format(num=r'(?:-?\d+)')
+        FIELD_RE = r'\w*(?:\.(?:\w+|{num}|{num}?(?::{num}?){{1,2}}))*'.format(num=r'(?:-?\d+)')
         MATH_FIELD_RE = r'''{field}|{num}'''.format(field=FIELD_RE, num=r'-?\d+(?:.\d+)?')
         MATH_OPERATORS_RE = r'(?:%s)' % '|'.join(map(re.escape, MATH_FUNCTIONS.keys()))
         INTERNAL_FORMAT_RE = re.compile(r'''(?x)
@@ -897,12 +957,15 @@ class YoutubeDL(object):
             (?:\|(?P<default>.*?))?
             $'''.format(field=FIELD_RE, math_op=MATH_OPERATORS_RE, math_field=MATH_FIELD_RE))
 
-        get_key = lambda k: traverse_obj(
-            info_dict, k.split('.'), is_user_input=True, traverse_string=True)
+        def _traverse_infodict(k):
+            k = k.split('.')
+            if k[0] == '':
+                k.pop(0)
+            return traverse_obj(info_dict, k, is_user_input=True, traverse_string=True)
 
         def get_value(mdict):
             # Object traversal
-            value = get_key(mdict['fields'])
+            value = _traverse_infodict(mdict['fields'])
             # Negative
             if mdict['negate']:
                 value = float_or_none(value)
@@ -924,7 +987,7 @@ class YoutubeDL(object):
                     item, multiplier = (item[1:], -1) if item[0] == '-' else (item, 1)
                     offset = float_or_none(item)
                     if offset is None:
-                        offset = float_or_none(get_key(item))
+                        offset = float_or_none(_traverse_infodict(item))
                     try:
                         value = operator(value, multiplier * offset)
                     except (TypeError, ZeroDivisionError):
@@ -936,12 +999,17 @@ class YoutubeDL(object):
 
             return value
 
+        na = self.params.get('outtmpl_na_placeholder', 'NA')
+
+        def _dumpjson_default(obj):
+            if isinstance(obj, (set, LazyList)):
+                return list(obj)
+            raise TypeError(f'Object of type {type(obj).__name__} is not JSON serializable')
+
         def create_key(outer_mobj):
             if not outer_mobj.group('has_key'):
-                return '%{}'.format(outer_mobj.group(0))
-
+                return f'%{outer_mobj.group(0)}'
             key = outer_mobj.group('key')
-            fmt = outer_mobj.group('format')
             mobj = re.match(INTERNAL_FORMAT_RE, key)
             if mobj is None:
                 value, default, mobj = None, na, {'fields': ''}
@@ -950,13 +1018,21 @@ class YoutubeDL(object):
                 default = mobj['default'] if mobj['default'] is not None else na
                 value = get_value(mobj)
 
+            fmt = outer_mobj.group('format')
             if fmt == 's' and value is not None and key in field_size_compat_map.keys():
                 fmt = '0{:d}d'.format(field_size_compat_map[key])
 
             value = default if value is None else value
 
-            if fmt == 'c':
-                value = compat_str(value)
+            str_fmt = f'{fmt[:-1]}s'
+            if fmt[-1] == 'l':
+                value, fmt = ', '.join(variadic(value)), str_fmt
+            elif fmt[-1] == 'j':
+                value, fmt = json.dumps(value, default=_dumpjson_default), str_fmt
+            elif fmt[-1] == 'q':
+                value, fmt = compat_shlex_quote(str(value)), str_fmt
+            elif fmt[-1] == 'c':
+                value = str(value)
                 if value is None:
                     value, fmt = default, 's'
                 else:
@@ -965,16 +1041,18 @@ class YoutubeDL(object):
                 value = float_or_none(value)
                 if value is None:
                     value, fmt = default, 's'
+
             if sanitize:
                 if fmt[-1] == 'r':
                     # If value is an object, sanitize might convert it to a string
                     # So we convert it to repr first
-                    value, fmt = repr(value), '%ss' % fmt[:-1]
+                    value, fmt = repr(value), str_fmt
                 if fmt[-1] in 'csr':
                     value = sanitize(mobj['fields'].split('.')[-1], value)
-            key += '\0%s' % fmt
+
+            key = '%s\0%s' % (key.replace('%', '%\0'), outer_mobj.group('format'))
             TMPL_DICT[key] = value
-            return '%({key}){fmt}'.format(key=key, fmt=fmt)
+            return '{prefix}%({key}){fmt}'.format(key=key, fmt=fmt, prefix=outer_mobj.group('prefix'))
 
         return EXTERNAL_FORMAT_RE.sub(create_key, outtmpl), TMPL_DICT
 
@@ -986,19 +1064,8 @@ class YoutubeDL(object):
                 is_id=(k == 'id' or k.endswith('_id')))
             outtmpl = self.outtmpl_dict.get(tmpl_type, self.outtmpl_dict['default'])
             outtmpl, template_dict = self.prepare_outtmpl(outtmpl, info_dict, sanitize)
-
-            # expand_path translates '%%' into '%' and '$$' into '$'
-            # correspondingly that is not what we want since we need to keep
-            # '%%' intact for template dict substitution step. Working around
-            # with boundary-alike separator hack.
-            sep = ''.join([random.choice(ascii_letters) for _ in range(32)])
-            outtmpl = outtmpl.replace('%%', '%{0}%'.format(sep)).replace('$$', '${0}$'.format(sep))
-
-            # outtmpl should be expand_path'ed before template dict substitution
-            # because meta fields may contain env variables we don't want to
-            # be expanded. For example, for outtmpl "%(title)s.%(ext)s" and
-            # title "Hello $PATH", we don't want `$PATH` to be expanded.
-            filename = expand_path(outtmpl).replace(sep, '') % template_dict
+            outtmpl = self.escape_outtmpl(self._outtmpl_expandpath(outtmpl))
+            filename = outtmpl % template_dict
 
             force_ext = OUTTMPL_TYPES.get(tmpl_type)
             if force_ext is not None:
@@ -1031,7 +1098,6 @@ class YoutubeDL(object):
                 self.report_warning('--paths is ignored when an outputting to stdout', only_once=True)
             elif os.path.isabs(filename):
                 self.report_warning('--paths is ignored since an absolute path is given in output template', only_once=True)
-            self.__prepare_filename_warned = True
         if filename == '-' or not filename:
             return filename
 
@@ -1070,12 +1136,15 @@ class YoutubeDL(object):
             if age_restricted(info_dict.get('age_limit'), self.params.get('age_limit')):
                 return 'Skipping "%s" because it is age restricted' % video_title
 
-            if not incomplete:
-                match_filter = self.params.get('match_filter')
-                if match_filter is not None:
-                    ret = match_filter(info_dict)
-                    if ret is not None:
-                        return ret
+            match_filter = self.params.get('match_filter')
+            if match_filter is not None:
+                try:
+                    ret = match_filter(info_dict, incomplete=incomplete)
+                except TypeError:
+                    # For backward compatibility
+                    ret = None if incomplete else match_filter(info_dict)
+                if ret is not None:
+                    return ret
             return None
 
         if self.in_download_archive(info_dict):
@@ -1118,31 +1187,24 @@ class YoutubeDL(object):
             ie_key = 'Generic'
 
         if ie_key:
-            ies = [self.get_info_extractor(ie_key)]
+            ies = {ie_key: self._get_info_extractor_class(ie_key)}
         else:
             ies = self._ies
 
-        for ie in ies:
+        for ie_key, ie in ies.items():
             if not ie.suitable(url):
                 continue
 
-            ie_key = ie.ie_key()
-            ie = self.get_info_extractor(ie_key)
             if not ie.working():
                 self.report_warning('The program functionality for this site has been marked as broken, '
                                     'and will probably not work.')
 
-            try:
-                temp_id = str_or_none(
-                    ie.extract_id(url) if callable(getattr(ie, 'extract_id', None))
-                    else ie._match_id(url))
-            except (AssertionError, IndexError, AttributeError):
-                temp_id = None
+            temp_id = ie.get_temp_id(url)
             if temp_id is not None and self.in_download_archive({'id': temp_id, 'ie_key': ie_key}):
                 self.to_screen("[%s] %s: has already been recorded in archive" % (
                                ie_key, temp_id))
                 break
-            return self.__extract_info(url, ie, download, extra_info, process)
+            return self.__extract_info(url, self.get_info_extractor(ie_key), download, extra_info, process)
         else:
             self.report_error('no suitable InfoExtractor for URL %s' % url)
 
@@ -1204,7 +1266,7 @@ class YoutubeDL(object):
                 'extractor_key': ie.ie_key(),
             })
 
-    def process_ie_result(self, ie_result, download=True, extra_info={}):
+    def process_ie_result(self, ie_result, download=True, extra_info=None):
         """
         Take the result of the ie(may be modified) and resolve all unresolved
         references (URLs, playlist items).
@@ -1212,6 +1274,8 @@ class YoutubeDL(object):
         It will also download the videos if 'download'.
         Returns the resolved ie_result.
         """
+        if extra_info is None:
+            extra_info = {}
         result_type = ie_result.get('_type', 'video')
 
         if result_type in ('url', 'url_transparent'):
@@ -1234,7 +1298,7 @@ class YoutubeDL(object):
             ie_result = self.process_video_result(ie_result, download=download)
             additional_urls = (ie_result or {}).get('additional_urls')
             if additional_urls:
-                # TODO: Improve MetadataFromFieldPP to allow setting a list
+                # TODO: Improve MetadataParserPP to allow setting a list
                 if isinstance(additional_urls, compat_str):
                     additional_urls = [additional_urls]
                 self.to_screen(
@@ -1310,15 +1374,12 @@ class YoutubeDL(object):
                 'It needs to be updated.' % ie_result.get('extractor'))
 
             def _fixup(r):
-                self.add_extra_info(
-                    r,
-                    {
-                        'extractor': ie_result['extractor'],
-                        'webpage_url': ie_result['webpage_url'],
-                        'webpage_url_basename': url_basename(ie_result['webpage_url']),
-                        'extractor_key': ie_result['extractor_key'],
-                    }
-                )
+                self.add_extra_info(r, {
+                    'extractor': ie_result['extractor'],
+                    'webpage_url': ie_result['webpage_url'],
+                    'webpage_url_basename': url_basename(ie_result['webpage_url']),
+                    'extractor_key': ie_result['extractor_key'],
+                })
                 return r
             ie_result['entries'] = [
                 self.process_ie_result(_fixup(r), download, extra_info)
@@ -1405,7 +1466,7 @@ class YoutubeDL(object):
 
         # Save playlist_index before re-ordering
         entries = [
-            ((playlistitems[i - 1] if playlistitems else i), entry)
+            ((playlistitems[i - 1] if playlistitems else i + playliststart - 1), entry)
             for i, entry in enumerate(entries, 1)
             if entry is not None]
         n_entries = len(entries)
@@ -1434,7 +1495,7 @@ class YoutubeDL(object):
                 else:
                     self.to_screen('[info] Writing playlist metadata as JSON to: ' + infofn)
                     try:
-                        write_json_file(self.filter_requested_info(ie_result, self.params.get('clean_infojson', True)), infofn)
+                        write_json_file(self.sanitize_info(ie_result, self.params.get('clean_infojson', True)), infofn)
                     except (OSError, IOError):
                         self.report_error('Cannot write playlist metadata to JSON file ' + infofn)
 
@@ -1470,7 +1531,7 @@ class YoutubeDL(object):
         max_failures = self.params.get('skip_playlist_after_errors') or float('inf')
         for i, entry_tuple in enumerate(entries, 1):
             playlist_index, entry = entry_tuple
-            if 'playlist_index' in self.params.get('compat_options', []):
+            if 'playlist-index' in self.params.get('compat_options', []):
                 playlist_index = playlistitems[i - 1] if playlistitems else i
             self.to_screen('[download] Downloading video %s of %s' % (i, n_entries))
             # This __x_forwarded_for_ip thing is a bit ugly but requires
@@ -1582,7 +1643,7 @@ class YoutubeDL(object):
             return merger.available and merger.can_merge()
 
         prefer_best = (
-            not self.params.get('simulate', False)
+            not self.params.get('simulate')
             and download
             and (
                 not can_merge()
@@ -1730,6 +1791,7 @@ class YoutubeDL(object):
                         if not allow_multiple_streams[aud_vid] and fmt_info.get(aud_vid[0] + 'codec') != 'none':
                             if get_no_more[aud_vid]:
                                 formats_info.pop(i)
+                                break
                             get_no_more[aud_vid] = True
 
             if len(formats_info) == 1:
@@ -2005,7 +2067,8 @@ class YoutubeDL(object):
         if 'id' not in info_dict:
             raise ExtractorError('Missing "id" field in extractor result')
         if 'title' not in info_dict:
-            raise ExtractorError('Missing "title" field in extractor result')
+            raise ExtractorError('Missing "title" field in extractor result',
+                                 video_id=info_dict['id'], ie=info_dict['extractor'])
 
         def report_force_conversion(field, field_not, conversion):
             self.report_warning(
@@ -2106,11 +2169,12 @@ class YoutubeDL(object):
         else:
             formats = info_dict['formats']
 
+        info_dict['__has_drm'] = any(f.get('has_drm') for f in formats)
+        if not self.params.get('allow_unplayable_formats'):
+            formats = [f for f in formats if not f.get('has_drm')]
+
         if not formats:
-            if not self.params.get('ignore_no_formats_error'):
-                raise ExtractorError('No video formats found!')
-            else:
-                self.report_warning('No video formats found!')
+            self.raise_no_formats(info_dict)
 
         def is_wellformed(f):
             url = f.get('url')
@@ -2154,7 +2218,7 @@ class YoutubeDL(object):
                 format['format'] = '{id} - {res}{note}'.format(
                     id=format['format_id'],
                     res=self.format_resolution(format),
-                    note=' ({0})'.format(format['format_note']) if format.get('format_note') is not None else '',
+                    note=format_field(format, 'format_note', ' (%s)'),
                 )
             # Automatically determine file extension if missing
             if format.get('ext') is None:
@@ -2174,7 +2238,7 @@ class YoutubeDL(object):
 
         # TODO Central sorting goes here
 
-        if formats and formats[0] is not info_dict:
+        if not formats or formats[0] is not info_dict:
             # only set the 'formats' fields if the original info_dict list them
             # otherwise we end up with a circular reference, the first (and unique)
             # element in the 'formats' field in info_dict is info_dict itself,
@@ -2183,20 +2247,23 @@ class YoutubeDL(object):
 
         info_dict, _ = self.pre_process(info_dict)
 
-        list_only = self.params.get('list_thumbnails') or self.params.get('listformats') or self.params.get('listsubtitles')
-        if list_only:
-            self.__forced_printings(info_dict, self.prepare_filename(info_dict), incomplete=True)
-            if self.params.get('list_thumbnails'):
-                self.list_thumbnails(info_dict)
-            if self.params.get('listformats'):
-                if not info_dict.get('formats'):
-                    raise ExtractorError('No video formats found', expected=True)
+        if self.params.get('list_thumbnails'):
+            self.list_thumbnails(info_dict)
+        if self.params.get('listformats'):
+            if not info_dict.get('formats') and not info_dict.get('url'):
+                self.to_screen('%s has no formats' % info_dict['id'])
+            else:
                 self.list_formats(info_dict)
-            if self.params.get('listsubtitles'):
-                if 'automatic_captions' in info_dict:
-                    self.list_subtitles(
-                        info_dict['id'], automatic_captions, 'automatic captions')
-                self.list_subtitles(info_dict['id'], subtitles, 'subtitles')
+        if self.params.get('listsubtitles'):
+            if 'automatic_captions' in info_dict:
+                self.list_subtitles(
+                    info_dict['id'], automatic_captions, 'automatic captions')
+            self.list_subtitles(info_dict['id'], subtitles, 'subtitles')
+        list_only = self.params.get('simulate') is None and (
+            self.params.get('list_thumbnails') or self.params.get('listformats') or self.params.get('listsubtitles'))
+        if list_only:
+            # Without this printing, -F --print-json will not work
+            self.__forced_printings(info_dict, self.prepare_filename(info_dict), incomplete=True)
             return
 
         format_selector = self.format_selector
@@ -2234,7 +2301,8 @@ class YoutubeDL(object):
         formats_to_download = list(format_selector(ctx))
         if not formats_to_download:
             if not self.params.get('ignore_no_formats_error'):
-                raise ExtractorError('Requested format is not available', expected=True)
+                raise ExtractorError('Requested format is not available', expected=True,
+                                     video_id=info_dict['id'], ie=info_dict['extractor'])
             else:
                 self.report_warning('Requested format is not available')
                 # Process what we can, even without any available formats.
@@ -2292,7 +2360,8 @@ class YoutubeDL(object):
             requested_langs = ['en']
         else:
             requested_langs = [list(all_sub_langs)[0]]
-        self.write_debug('Downloading subtitles: %s' % ', '.join(requested_langs))
+        if requested_langs:
+            self.write_debug('Downloading subtitles: %s' % ', '.join(requested_langs))
 
         formats_query = self.params.get('subtitlesformat', 'best')
         formats_preference = formats_query.split('/') if formats_query else []
@@ -2340,11 +2409,13 @@ class YoutubeDL(object):
         elif 'url' in info_dict:
             info_dict['urls'] = info_dict['url'] + info_dict.get('play_path', '')
 
+        if self.params.get('forceprint') or self.params.get('forcejson'):
+            self.post_extract(info_dict)
         for tmpl in self.params.get('forceprint', []):
             if re.match(r'\w+$', tmpl):
                 tmpl = '%({})s'.format(tmpl)
             tmpl, info_copy = self.prepare_outtmpl(tmpl, info_dict)
-            self.to_stdout(tmpl % info_copy)
+            self.to_stdout(self.escape_outtmpl(tmpl) % info_copy)
 
         print_mandatory('title')
         print_mandatory('id')
@@ -2352,15 +2423,16 @@ class YoutubeDL(object):
         print_optional('thumbnail')
         print_optional('description')
         print_optional('filename')
-        if self.params.get('forceduration', False) and info_dict.get('duration') is not None:
+        if self.params.get('forceduration') and info_dict.get('duration') is not None:
             self.to_stdout(formatSeconds(info_dict['duration']))
         print_mandatory('format')
 
-        if self.params.get('forcejson', False):
-            self.post_extract(info_dict)
-            self.to_stdout(json.dumps(info_dict, default=repr))
+        if self.params.get('forcejson'):
+            self.to_stdout(json.dumps(self.sanitize_info(info_dict)))
 
     def dl(self, name, info, subtitle=False, test=False):
+        if not info.get('url'):
+            self.raise_no_formats(info, True)
 
         if test:
             verbose = self.params.get('verbose')
@@ -2377,7 +2449,7 @@ class YoutubeDL(object):
             }
         else:
             params = self.params
-        fd = get_suitable_downloader(info, params)(self, params)
+        fd = get_suitable_downloader(info, params, to_stdout=(name == '-'))(self, params)
         if not test:
             for ph in self._progress_hooks:
                 fd.add_progress_hook(ph)
@@ -2393,8 +2465,6 @@ class YoutubeDL(object):
 
         assert info_dict.get('_type', 'video') == 'video'
 
-        info_dict.setdefault('__postprocessors', [])
-
         max_downloads = self.params.get('max_downloads')
         if max_downloads is not None:
             if self._num_downloads >= int(max_downloads):
@@ -2420,7 +2490,7 @@ class YoutubeDL(object):
         # Forced printings
         self.__forced_printings(info_dict, full_filename, incomplete=('format' not in info_dict))
 
-        if self.params.get('simulate', False):
+        if self.params.get('simulate'):
             if self.params.get('force_write_download_archive', False):
                 self.record_download_archive(info_dict)
 
@@ -2520,7 +2590,7 @@ class YoutubeDL(object):
             else:
                 self.to_screen('[info] Writing video metadata as JSON to: ' + infofn)
                 try:
-                    write_json_file(self.filter_requested_info(info_dict, self.params.get('clean_infojson', True)), infofn)
+                    write_json_file(self.sanitize_info(info_dict, self.params.get('clean_infojson', True)), infofn)
                 except (OSError, IOError):
                     self.report_error('Cannot write video metadata to JSON file ' + infofn)
                     return
@@ -2595,6 +2665,7 @@ class YoutubeDL(object):
             info_dict = self.run_pp(MoveFilesAfterDownloadPP(self, False), info_dict)
         else:
             # Download
+            info_dict.setdefault('__postprocessors', [])
             try:
 
                 def existing_file(*filepaths):
@@ -2615,7 +2686,6 @@ class YoutubeDL(object):
                             os.remove(encodeFilename(file))
                         return None
 
-                    self.report_file_already_downloaded(existing_files[0])
                     info_dict['ext'] = os.path.splitext(existing_files[0])[1][1:]
                     return existing_files[0]
 
@@ -2647,14 +2717,17 @@ class YoutubeDL(object):
                         info_dict['ext'] = 'mkv'
                         self.report_warning(
                             'Requested formats are incompatible for merge and will be merged into mkv.')
+                    new_ext = info_dict['ext']
 
-                    def correct_ext(filename):
+                    def correct_ext(filename, ext=new_ext):
+                        if filename == '-':
+                            return filename
                         filename_real_ext = os.path.splitext(filename)[1][1:]
                         filename_wo_ext = (
                             os.path.splitext(filename)[0]
-                            if filename_real_ext == old_ext
+                            if filename_real_ext in (old_ext, new_ext)
                             else filename)
-                        return '%s.%s' % (filename_wo_ext, info_dict['ext'])
+                        return '%s.%s' % (filename_wo_ext, ext)
 
                     # Ensure filename always has a correct extension for successful merge
                     full_filename = correct_ext(full_filename)
@@ -2663,20 +2736,16 @@ class YoutubeDL(object):
                     info_dict['__real_download'] = False
 
                     _protocols = set(determine_protocol(f) for f in requested_formats)
-                    if len(_protocols) == 1:
+                    if len(_protocols) == 1:  # All requested formats have same protocol
                         info_dict['protocol'] = _protocols.pop()
-                    directly_mergable = (
-                        'no-direct-merge' not in self.params.get('compat_opts', [])
-                        and info_dict.get('protocol') is not None  # All requested formats have same protocol
-                        and not self.params.get('allow_unplayable_formats')
-                        and get_suitable_downloader(info_dict, self.params).__name__ == 'FFmpegFD')
-                    if directly_mergable:
-                        info_dict['url'] = requested_formats[0]['url']
-                        # Treat it as a single download
-                        dl_filename = existing_file(full_filename, temp_filename)
-                        if dl_filename is None:
-                            success, real_download = self.dl(temp_filename, info_dict)
-                            info_dict['__real_download'] = real_download
+                    directly_mergable = FFmpegFD.can_merge_formats(info_dict)
+                    if dl_filename is not None:
+                        self.report_file_already_downloaded(dl_filename)
+                    elif (directly_mergable and get_suitable_downloader(
+                            info_dict, self.params, to_stdout=(temp_filename == '-')) == FFmpegFD):
+                        info_dict['url'] = '\n'.join(f['url'] for f in requested_formats)
+                        success, real_download = self.dl(temp_filename, info_dict)
+                        info_dict['__real_download'] = real_download
                     else:
                         downloaded = []
                         merger = FFmpegMergerPP(self)
@@ -2690,34 +2759,46 @@ class YoutubeDL(object):
                                 'You have requested merging of multiple formats but ffmpeg is not installed. '
                                 'The formats won\'t be merged.')
 
-                        if dl_filename is None:
-                            for f in requested_formats:
-                                new_info = dict(info_dict)
-                                del new_info['requested_formats']
-                                new_info.update(f)
+                        if temp_filename == '-':
+                            reason = ('using a downloader other than ffmpeg' if directly_mergable
+                                      else 'but the formats are incompatible for simultaneous download' if merger.available
+                                      else 'but ffmpeg is not installed')
+                            self.report_warning(
+                                f'You have requested downloading multiple formats to stdout {reason}. '
+                                'The formats will be streamed one after the other')
+                            fname = temp_filename
+                        for f in requested_formats:
+                            new_info = dict(info_dict)
+                            del new_info['requested_formats']
+                            new_info.update(f)
+                            if temp_filename != '-':
                                 fname = prepend_extension(
-                                    self.prepare_filename(new_info, 'temp'),
+                                    correct_ext(temp_filename, new_info['ext']),
                                     'f%s' % f['format_id'], new_info['ext'])
                                 if not self._ensure_dir_exists(fname):
                                     return
                                 downloaded.append(fname)
-                                partial_success, real_download = self.dl(fname, new_info)
-                                info_dict['__real_download'] = info_dict['__real_download'] or real_download
-                                success = success and partial_success
-                            if merger.available and not self.params.get('allow_unplayable_formats'):
-                                info_dict['__postprocessors'].append(merger)
-                                info_dict['__files_to_merge'] = downloaded
-                                # Even if there were no downloads, it is being merged only now
-                                info_dict['__real_download'] = True
-                            else:
-                                for file in downloaded:
-                                    files_to_move[file] = None
+                            partial_success, real_download = self.dl(fname, new_info)
+                            info_dict['__real_download'] = info_dict['__real_download'] or real_download
+                            success = success and partial_success
+                        if merger.available and not self.params.get('allow_unplayable_formats'):
+                            info_dict['__postprocessors'].append(merger)
+                            info_dict['__files_to_merge'] = downloaded
+                            # Even if there were no downloads, it is being merged only now
+                            info_dict['__real_download'] = True
+                        else:
+                            for file in downloaded:
+                                files_to_move[file] = None
                 else:
                     # Just a single file
                     dl_filename = existing_file(full_filename, temp_filename)
-                    if dl_filename is None:
+                    if dl_filename is None or dl_filename == temp_filename:
+                        # dl_filename == temp_filename could mean that the file was partially downloaded with --no-part.
+                        # So we should try to resume the download
                         success, real_download = self.dl(temp_filename, info_dict)
                         info_dict['__real_download'] = real_download
+                    else:
+                        self.report_file_already_downloaded(dl_filename)
 
                 dl_filename = dl_filename or temp_filename
                 info_dict['__finaldir'] = os.path.dirname(os.path.abspath(encodeFilename(full_filename)))
@@ -2815,18 +2896,18 @@ class YoutubeDL(object):
             except UnavailableVideoError:
                 self.report_error('unable to download video')
             except MaxDownloadsReached:
-                self.to_screen('[info] Maximum number of downloaded files reached')
+                self.to_screen('[info] Maximum number of downloads reached')
                 raise
             except ExistingVideoReached:
-                self.to_screen('[info] Encountered a file that is already in the archive, stopping due to --break-on-existing')
+                self.to_screen('[info] Encountered a video that is already in the archive, stopping due to --break-on-existing')
                 raise
             except RejectedVideoReached:
-                self.to_screen('[info] Encountered a file that did not match filter, stopping due to --break-on-reject')
+                self.to_screen('[info] Encountered a video that did not match filter, stopping due to --break-on-reject')
                 raise
             else:
                 if self.params.get('dump_single_json', False):
                     self.post_extract(res)
-                    self.to_stdout(json.dumps(res, default=repr))
+                    self.to_stdout(json.dumps(self.sanitize_info(res)))
 
         return self._download_retcode
 
@@ -2835,7 +2916,7 @@ class YoutubeDL(object):
                 [info_filename], mode='r',
                 openhook=fileinput.hook_encoded('utf-8'))) as f:
             # FileInput doesn't have a read method, we can't call json.load
-            info = self.filter_requested_info(json.loads('\n'.join(f)), self.params.get('clean_infojson', True))
+            info = self.sanitize_info(json.loads('\n'.join(f)), self.params.get('clean_infojson', True))
         try:
             self.process_ie_result(info, download=True)
         except (DownloadError, EntryNotInPlaylist, ThrottledDownload):
@@ -2848,16 +2929,22 @@ class YoutubeDL(object):
         return self._download_retcode
 
     @staticmethod
-    def filter_requested_info(info_dict, actually_filter=True):
-        remove_keys = ['__original_infodict']  # Always remove this since this may contain a copy of the entire dict
+    def sanitize_info(info_dict, remove_private_keys=False):
+        ''' Sanitize the infodict for converting to json '''
+        if info_dict is None:
+            return info_dict
+        info_dict.setdefault('epoch', int(time.time()))
+        remove_keys = {'__original_infodict'}  # Always remove this since this may contain a copy of the entire dict
         keep_keys = ['_type'],  # Always keep this to facilitate load-info-json
-        if actually_filter:
-            remove_keys += ('requested_formats', 'requested_subtitles', 'requested_entries', 'filepath', 'entries', 'original_url')
+        if remove_private_keys:
+            remove_keys |= {
+                'requested_formats', 'requested_subtitles', 'requested_entries',
+                'filepath', 'entries', 'original_url', 'playlist_autonumber',
+            }
             empty_values = (None, {}, [], set(), tuple())
             reject = lambda k, v: k not in keep_keys and (
                 k.startswith('_') or k in remove_keys or v in empty_values)
         else:
-            info_dict['epoch'] = int(time.time())
             reject = lambda k, v: k in remove_keys
         filter_fn = lambda obj: (
             list(map(filter_fn, obj)) if isinstance(obj, (LazyList, list, tuple, set))
@@ -2865,6 +2952,11 @@ class YoutubeDL(object):
             else dict((k, filter_fn(v)) for k, v in obj.items() if not reject(k, v)))
         return filter_fn(info_dict)
 
+    @staticmethod
+    def filter_requested_info(info_dict, actually_filter=True):
+        ''' Alias of sanitize_info for backward compatibility '''
+        return YoutubeDL.sanitize_info(info_dict, actually_filter)
+
     def run_pp(self, pp, infodict):
         files_to_delete = []
         if '__files_to_move' not in infodict:
@@ -2939,9 +3031,9 @@ class YoutubeDL(object):
             if not url:
                 return
             # Try to find matching extractor for the URL and take its ie_key
-            for ie in self._ies:
+            for ie_key, ie in self._ies.items():
                 if ie.suitable(url):
-                    extractor = ie.ie_key()
+                    extractor = ie_key
                     break
             else:
                 return
@@ -3069,7 +3161,7 @@ class YoutubeDL(object):
                         format_field(f, 'language', '[%s]'),
                         format_field(f, 'format_note'),
                         format_field(f, 'container', ignore=(None, f.get('ext'))),
-                        format_field(f, 'asr', '%5dHz')))),
+                    ))),
                 ] for f in formats if f.get('preference') is None or f['preference'] >= -1000]
             header_line = ['ID', 'EXT', 'RESOLUTION', 'FPS', '|', ' FILESIZE', '  TBR', 'PROTO',
                            '|', 'VCODEC', '  VBR', 'ACODEC', ' ABR', ' ASR', 'MORE INFO']
@@ -3129,11 +3221,6 @@ class YoutubeDL(object):
         if not self.params.get('verbose'):
             return
 
-        if type('') is not compat_str:
-            # Python 2.6 on SLES11 SP1 (https://github.com/ytdl-org/youtube-dl/issues/3326)
-            self.report_warning(
-                'Your Python is broken! Update to a newer and supported version')
-
         stdout_encoding = getattr(
             sys.stdout, 'encoding', 'missing (%s)' % type(sys.stdout).__name__)
         encoding_str = (
@@ -3189,14 +3276,24 @@ class YoutubeDL(object):
         exe_versions['rtmpdump'] = rtmpdump_version()
         exe_versions['phantomjs'] = PhantomJSwrapper._version()
         exe_str = ', '.join(
-            '%s %s' % (exe, v)
-            for exe, v in sorted(exe_versions.items())
-            if v
-        )
-        if not exe_str:
-            exe_str = 'none'
+            f'{exe} {v}' for exe, v in sorted(exe_versions.items()) if v
+        ) or 'none'
         self._write_string('[debug] exe versions: %s\n' % exe_str)
 
+        from .downloader.fragment import can_decrypt_frag
+        from .downloader.websocket import has_websockets
+        from .postprocessor.embedthumbnail import has_mutagen
+        from .cookies import SQLITE_AVAILABLE, KEYRING_AVAILABLE
+
+        lib_str = ', '.join(sorted(filter(None, (
+            can_decrypt_frag and 'pycryptodome',
+            has_websockets and 'websockets',
+            has_mutagen and 'mutagen',
+            SQLITE_AVAILABLE and 'sqlite',
+            KEYRING_AVAILABLE and 'keyring',
+        )))) or 'none'
+        self._write_string('[debug] Optional libraries: %s\n' % lib_str)
+
         proxy_map = {}
         for handler in self._opener.handlers:
             if hasattr(handler, 'proxies'):

yt_dlp/__init__.py

@@ -1,23 +1,24 @@
 #!/usr/bin/env python3
 # coding: utf-8
 
-from __future__ import unicode_literals
+f'You are using an unsupported version of Python. Only Python versions 3.6 and above are supported by yt-dlp'  # noqa: F541
 
 __license__ = 'Public Domain'
 
 import codecs
 import io
+import itertools
 import os
 import random
 import re
 import sys
 
-
 from .options import (
     parseOpts,
 )
 from .compat import (
     compat_getpass,
+    compat_shlex_quote,
     workaround_optparse_bug9161,
 )
 from .cookies import SUPPORTED_BROWSERS
@@ -46,14 +47,15 @@ from .downloader import (
 from .extractor import gen_extractors, list_extractors
 from .extractor.common import InfoExtractor
 from .extractor.adobepass import MSO_INFO
-from .postprocessor.ffmpeg import (
+from .postprocessor import (
     FFmpegExtractAudioPP,
     FFmpegSubtitlesConvertorPP,
     FFmpegThumbnailsConvertorPP,
     FFmpegVideoConvertorPP,
     FFmpegVideoRemuxerPP,
+    MetadataFromFieldPP,
+    MetadataParserPP,
 )
-from .postprocessor.metadatafromfield import MetadataFromFieldPP
 from .YoutubeDL import YoutubeDL
 
 
@@ -107,14 +109,14 @@ def _real_main(argv=None):
 
     if opts.list_extractors:
         for ie in list_extractors(opts.age_limit):
-            write_string(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie._WORKING else '') + '\n', out=sys.stdout)
+            write_string(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie.working() else '') + '\n', out=sys.stdout)
             matchedUrls = [url for url in all_urls if ie.suitable(url)]
             for mu in matchedUrls:
                 write_string('  ' + mu + '\n', out=sys.stdout)
         sys.exit(0)
     if opts.list_extractor_descriptions:
         for ie in list_extractors(opts.age_limit):
-            if not ie._WORKING:
+            if not ie.working():
                 continue
             desc = getattr(ie, 'IE_DESC', ie.IE_NAME)
             if desc is False:
@@ -254,35 +256,7 @@ def _real_main(argv=None):
     else:
         date = DateRange(opts.dateafter, opts.datebefore)
 
-    def parse_compat_opts():
-        parsed_compat_opts, compat_opts = set(), opts.compat_opts[::-1]
-        while compat_opts:
-            actual_opt = opt = compat_opts.pop().lower()
-            if opt == 'youtube-dl':
-                compat_opts.extend(['-multistreams', 'all'])
-            elif opt == 'youtube-dlc':
-                compat_opts.extend(['-no-youtube-channel-redirect', '-no-live-chat', 'all'])
-            elif opt == 'all':
-                parsed_compat_opts.update(all_compat_opts)
-            elif opt == '-all':
-                parsed_compat_opts = set()
-            else:
-                if opt[0] == '-':
-                    opt = opt[1:]
-                    parsed_compat_opts.discard(opt)
-                else:
-                    parsed_compat_opts.update([opt])
-                if opt not in all_compat_opts:
-                    parser.error('Invalid compatibility option %s' % actual_opt)
-        return parsed_compat_opts
-
-    all_compat_opts = [
-        'filename', 'format-sort', 'abort-on-error', 'format-spec', 'no-playlist-metafiles',
-        'multistreams', 'no-live-chat', 'playlist-index', 'list-formats', 'no-direct-merge',
-        'no-youtube-channel-redirect', 'no-youtube-unavailable-videos', 'no-attach-info-json',
-        'embed-thumbnail-atomicparsley', 'seperate-video-versions',
-    ]
-    compat_opts = parse_compat_opts()
+    compat_opts = opts.compat_opts
 
     def _unused_compat_opt(name):
         if name not in compat_opts:
@@ -291,7 +265,7 @@ def _real_main(argv=None):
         compat_opts.update(['*%s' % name])
         return True
 
-    def set_default_compat(compat_name, opt_name, default=True, remove_compat=False):
+    def set_default_compat(compat_name, opt_name, default=True, remove_compat=True):
         attr = getattr(opts, opt_name)
         if compat_name in compat_opts:
             if attr is None:
@@ -307,6 +281,7 @@ def _real_main(argv=None):
 
     set_default_compat('abort-on-error', 'ignoreerrors')
     set_default_compat('no-playlist-metafiles', 'allow_playlist_files')
+    set_default_compat('no-clean-infojson', 'clean_infojson')
     if 'format-sort' in compat_opts:
         opts.format_sort.extend(InfoExtractor.FormatSort.ytdl_default)
     _video_multistreams_set = set_default_compat('multistreams', 'allow_multiple_video_streams', False, remove_compat=False)
@@ -316,7 +291,7 @@ def _real_main(argv=None):
     outtmpl_default = opts.outtmpl.get('default')
     if 'filename' in compat_opts:
         if outtmpl_default is None:
-            outtmpl_default = '%(title)s.%(id)s.%(ext)s'
+            outtmpl_default = '%(title)s-%(id)s.%(ext)s'
             opts.outtmpl.update({'default': outtmpl_default})
         else:
             _unused_compat_opt('filename')
@@ -328,8 +303,10 @@ def _real_main(argv=None):
 
     for k, tmpl in opts.outtmpl.items():
         validate_outtmpl(tmpl, '%s output template' % k)
-    for tmpl in opts.forceprint:
+    opts.forceprint = opts.forceprint or []
+    for tmpl in opts.forceprint or []:
         validate_outtmpl(tmpl, 'print template')
+    validate_outtmpl(opts.sponsorblock_chapter_title, 'SponsorBlock chapter title')
 
     if opts.extractaudio and not opts.keepvideo and opts.format is None:
         opts.format = 'bestaudio/best'
@@ -343,13 +320,29 @@ def _real_main(argv=None):
         if re.match(InfoExtractor.FormatSort.regex, f) is None:
             parser.error('invalid format sort string "%s" specified' % f)
 
-    if opts.metafromfield is None:
-        opts.metafromfield = []
+    def metadataparser_actions(f):
+        if isinstance(f, str):
+            cmd = '--parse-metadata %s' % compat_shlex_quote(f)
+            try:
+                actions = [MetadataFromFieldPP.to_action(f)]
+            except Exception as err:
+                parser.error(f'{cmd} is invalid; {err}')
+        else:
+            cmd = '--replace-in-metadata %s' % ' '.join(map(compat_shlex_quote, f))
+            actions = ((MetadataParserPP.Actions.REPLACE, x, *f[1:]) for x in f[0].split(','))
+
+        for action in actions:
+            try:
+                MetadataParserPP.validate_action(*action)
+            except Exception as err:
+                parser.error(f'{cmd} is invalid; {err}')
+            yield action
+
+    if opts.parse_metadata is None:
+        opts.parse_metadata = []
     if opts.metafromtitle is not None:
-        opts.metafromfield.append('title:%s' % opts.metafromtitle)
-    for f in opts.metafromfield:
-        if re.match(MetadataFromFieldPP.regex, f) is None:
-            parser.error('invalid format string "%s" specified for --parse-metadata' % f)
+        opts.parse_metadata.append('title:%s' % opts.metafromtitle)
+    opts.parse_metadata = list(itertools.chain(*map(metadataparser_actions, opts.parse_metadata)))
 
     any_getting = opts.forceprint or opts.geturl or opts.gettitle or opts.getid or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat or opts.getduration or opts.dumpjson or opts.dump_single_json
     any_printing = opts.print_json
@@ -360,15 +353,34 @@ def _real_main(argv=None):
     if opts.getcomments and not printing_json:
         opts.writeinfojson = True
 
+    if opts.no_sponsorblock:
+        opts.sponsorblock_mark = set()
+        opts.sponsorblock_remove = set()
+    sponsorblock_query = opts.sponsorblock_mark | opts.sponsorblock_remove
+
+    if (opts.addmetadata or opts.sponsorblock_mark) and opts.addchapters is None:
+        opts.addchapters = True
+    opts.remove_chapters = opts.remove_chapters or []
+
     def report_conflict(arg1, arg2):
         warnings.append('%s is ignored since %s was given' % (arg2, arg1))
 
+    if (opts.remove_chapters or sponsorblock_query) and opts.sponskrub is not False:
+        if opts.sponskrub:
+            if opts.remove_chapters:
+                report_conflict('--remove-chapters', '--sponskrub')
+            if opts.sponsorblock_mark:
+                report_conflict('--sponsorblock-mark', '--sponskrub')
+            if opts.sponsorblock_remove:
+                report_conflict('--sponsorblock-remove', '--sponskrub')
+        opts.sponskrub = False
+    if opts.sponskrub_cut and opts.split_chapters and opts.sponskrub is not False:
+        report_conflict('--split-chapter', '--sponskrub-cut')
+        opts.sponskrub_cut = False
+
     if opts.remuxvideo and opts.recodevideo:
         report_conflict('--recode-video', '--remux-video')
         opts.remuxvideo = False
-    if opts.sponskrub_cut and opts.split_chapters and opts.sponskrub is not False:
-        report_conflict('--split-chapter', '--sponskrub-cut')
-        opts.sponskrub_cut = False
 
     if opts.allow_unplayable_formats:
         if opts.extractaudio:
@@ -395,16 +407,30 @@ def _real_main(argv=None):
         if opts.fixup and opts.fixup.lower() not in ('never', 'ignore'):
             report_conflict('--allow-unplayable-formats', '--fixup')
         opts.fixup = 'never'
+        if opts.remove_chapters:
+            report_conflict('--allow-unplayable-formats', '--remove-chapters')
+            opts.remove_chapters = []
+        if opts.sponsorblock_remove:
+            report_conflict('--allow-unplayable-formats', '--sponsorblock-remove')
+            opts.sponsorblock_remove = set()
         if opts.sponskrub:
             report_conflict('--allow-unplayable-formats', '--sponskrub')
         opts.sponskrub = False
 
     # PostProcessors
     postprocessors = []
-    if opts.metafromfield:
+    if sponsorblock_query:
         postprocessors.append({
-            'key': 'MetadataFromField',
-            'formats': opts.metafromfield,
+            'key': 'SponsorBlock',
+            'categories': sponsorblock_query,
+            'api': opts.sponsorblock_api,
+            # Run this immediately after extraction is complete
+            'when': 'pre_process'
+        })
+    if opts.parse_metadata:
+        postprocessors.append({
+            'key': 'MetadataParser',
+            'actions': opts.parse_metadata,
             # Run this immediately after extraction is complete
             'when': 'pre_process'
         })
@@ -425,7 +451,7 @@ def _real_main(argv=None):
     # Must be after all other before_dl
     if opts.exec_before_dl_cmd:
         postprocessors.append({
-            'key': 'ExecAfterDownload',
+            'key': 'Exec',
             'exec_cmd': opts.exec_before_dl_cmd,
             'when': 'before_dl'
         })
@@ -446,29 +472,47 @@ def _real_main(argv=None):
             'key': 'FFmpegVideoConvertor',
             'preferedformat': opts.recodevideo,
         })
-    # FFmpegMetadataPP should be run after FFmpegVideoConvertorPP and
-    # FFmpegExtractAudioPP as containers before conversion may not support
-    # metadata (3gp, webm, etc.)
-    # And this post-processor should be placed before other metadata
-    # manipulating post-processors (FFmpegEmbedSubtitle) to prevent loss of
-    # extra metadata. By default ffmpeg preserves metadata applicable for both
-    # source and target containers. From this point the container won't change,
-    # so metadata can be added here.
-    if opts.addmetadata:
-        postprocessors.append({'key': 'FFmpegMetadata'})
+    # If ModifyChapters is going to remove chapters, subtitles must already be in the container.
     if opts.embedsubtitles:
-        already_have_subtitle = opts.writesubtitles
+        already_have_subtitle = opts.writesubtitles and 'no-keep-subs' not in compat_opts
         postprocessors.append({
             'key': 'FFmpegEmbedSubtitle',
             # already_have_subtitle = True prevents the file from being deleted after embedding
             'already_have_subtitle': already_have_subtitle
         })
-        if not already_have_subtitle:
+        if not opts.writeautomaticsub and 'no-keep-subs' not in compat_opts:
             opts.writesubtitles = True
     # --all-sub automatically sets --write-sub if --write-auto-sub is not given
     # this was the old behaviour if only --all-sub was given.
     if opts.allsubtitles and not opts.writeautomaticsub:
         opts.writesubtitles = True
+    # ModifyChapters must run before FFmpegMetadataPP
+    remove_chapters_patterns = []
+    for regex in opts.remove_chapters:
+        try:
+            remove_chapters_patterns.append(re.compile(regex))
+        except re.error as err:
+            parser.error(f'invalid --remove-chapters regex {regex!r} - {err}')
+    if opts.remove_chapters or sponsorblock_query:
+        postprocessors.append({
+            'key': 'ModifyChapters',
+            'remove_chapters_patterns': remove_chapters_patterns,
+            'remove_sponsor_segments': opts.sponsorblock_remove,
+            'sponsorblock_chapter_title': opts.sponsorblock_chapter_title,
+            'force_keyframes': opts.force_keyframes_at_cuts
+        })
+    # FFmpegMetadataPP should be run after FFmpegVideoConvertorPP and
+    # FFmpegExtractAudioPP as containers before conversion may not support
+    # metadata (3gp, webm, etc.)
+    # By default ffmpeg preserves metadata applicable for both
+    # source and target containers. From this point the container won't change,
+    # so metadata can be added here.
+    if opts.addmetadata or opts.addchapters:
+        postprocessors.append({
+            'key': 'FFmpegMetadata',
+            'add_chapters': opts.addchapters,
+            'add_metadata': opts.addmetadata,
+        })
     # This should be above EmbedThumbnail since sponskrub removes the thumbnail attachment
     # but must be below EmbedSubtitle and FFmpegMetadata
     # See https://github.com/yt-dlp/yt-dlp/issues/204 , https://github.com/faissaloo/SponSkrub/issues/29
@@ -492,14 +536,17 @@ def _real_main(argv=None):
         if not already_have_thumbnail:
             opts.writethumbnail = True
     if opts.split_chapters:
-        postprocessors.append({'key': 'FFmpegSplitChapters'})
+        postprocessors.append({
+            'key': 'FFmpegSplitChapters',
+            'force_keyframes': opts.force_keyframes_at_cuts,
+        })
     # XAttrMetadataPP should be run after post-processors that may change file contents
     if opts.xattrs:
         postprocessors.append({'key': 'XAttrMetadata'})
-    # ExecAfterDownload must be the last PP
+    # Exec must be the last PP
     if opts.exec_cmd:
         postprocessors.append({
-            'key': 'ExecAfterDownload',
+            'key': 'Exec',
             'exec_cmd': opts.exec_cmd,
             # Run this only after the files have been moved to their final locations
             'when': 'after_move'
@@ -549,7 +596,7 @@ def _real_main(argv=None):
         'forcejson': opts.dumpjson or opts.print_json,
         'dump_single_json': opts.dump_single_json,
         'force_write_download_archive': opts.force_write_download_archive,
-        'simulate': opts.simulate or any_getting,
+        'simulate': (any_getting or None) if opts.simulate is None else opts.simulate,
         'skip_download': opts.skip_download,
         'format': opts.format,
         'allow_unplayable_formats': opts.allow_unplayable_formats,
@@ -733,6 +780,11 @@ def main(argv=None):
         sys.exit('ERROR: fixed output name but more than one file to download')
     except KeyboardInterrupt:
         sys.exit('\nERROR: Interrupted by user')
+    except BrokenPipeError:
+        # https://docs.python.org/3/library/signal.html#note-on-sigpipe
+        devnull = os.open(os.devnull, os.O_WRONLY)
+        os.dup2(devnull, sys.stdout.fileno())
+        sys.exit(r'\nERROR: {err}')
 
 
 __all__ = ['main', 'YoutubeDL', 'gen_extractors', 'list_extractors']

yt_dlp/downloader/__init__.py

@@ -3,17 +3,20 @@ from __future__ import unicode_literals
 from ..compat import compat_str
 from ..utils import (
     determine_protocol,
+    NO_DEFAULT
 )
 
 
-def _get_real_downloader(info_dict, protocol=None, *args, **kwargs):
+def get_suitable_downloader(info_dict, params={}, default=NO_DEFAULT, protocol=None, to_stdout=False):
+    info_dict['protocol'] = determine_protocol(info_dict)
     info_copy = info_dict.copy()
     if protocol:
         info_copy['protocol'] = protocol
-    return get_suitable_downloader(info_copy, *args, **kwargs)
+    info_copy['to_stdout'] = to_stdout
+    return _get_suitable_downloader(info_copy, params, default)
 
 
-# Some of these require _get_real_downloader
+# Some of these require get_suitable_downloader
 from .common import FileDownloader
 from .dash import DashSegmentsFD
 from .f4m import F4mFD
@@ -69,32 +72,39 @@ def shorten_protocol_name(proto, simplify=False):
     return short_protocol_names.get(proto, proto)
 
 
-def get_suitable_downloader(info_dict, params={}, default=HttpFD):
+def _get_suitable_downloader(info_dict, params, default):
     """Get the downloader class that can handle the info dict."""
-    protocol = determine_protocol(info_dict)
-    info_dict['protocol'] = protocol
+    if default is NO_DEFAULT:
+        default = HttpFD
 
     # if (info_dict.get('start_time') or info_dict.get('end_time')) and not info_dict.get('requested_formats') and FFmpegFD.can_download(info_dict):
     #     return FFmpegFD
 
+    protocol = info_dict['protocol']
     downloaders = params.get('external_downloader')
     external_downloader = (
         downloaders if isinstance(downloaders, compat_str) or downloaders is None
         else downloaders.get(shorten_protocol_name(protocol, True), downloaders.get('default')))
-    if external_downloader and external_downloader.lower() == 'native':
-        external_downloader = 'native'
 
-    if external_downloader not in (None, 'native'):
+    if external_downloader is None:
+        if info_dict['to_stdout'] and FFmpegFD.can_merge_formats(info_dict, params):
+            return FFmpegFD
+    elif external_downloader.lower() != 'native':
         ed = get_external_downloader(external_downloader)
         if ed.can_download(info_dict, external_downloader):
             return ed
 
+    if protocol == 'http_dash_segments':
+        if info_dict.get('is_live') and (external_downloader or '').lower() != 'native':
+            return FFmpegFD
+
     if protocol in ('m3u8', 'm3u8_native'):
         if info_dict.get('is_live'):
             return FFmpegFD
-        elif external_downloader == 'native':
+        elif (external_downloader or '').lower() == 'native':
             return HlsFD
-        elif _get_real_downloader(info_dict, 'm3u8_frag_urls', params, None):
+        elif get_suitable_downloader(
+                info_dict, params, None, protocol='m3u8_frag_urls', to_stdout=info_dict['to_stdout']):
             return HlsFD
         elif params.get('hls_prefer_native') is True:
             return HlsFD

yt_dlp/downloader/common.py

@@ -47,8 +47,11 @@ class FileDownloader(object):
     min_filesize:       Skip files smaller than this size
     max_filesize:       Skip files larger than this size
     xattr_set_filesize: Set ytdl.filesize user xattribute with expected size.
-    external_downloader_args:  A list of additional command-line arguments for the
-                        external downloader.
+    external_downloader_args:  A dictionary of downloader keys (in lower case)
+                        and a list of additional command-line arguments for the
+                        executable. Use 'default' as the name for arguments to be
+                        passed to all downloaders. For compatibility with youtube-dl,
+                        a single list of args can also be used
     hls_use_mpegts:     Use the mpegts container for HLS videos.
     http_chunk_size:    Size of a chunk for chunk-based HTTP downloading. May be
                         useful for bypassing bandwidth throttling imposed by
@@ -201,12 +204,12 @@ class FileDownloader(object):
         return filename + '.ytdl'
 
     def try_rename(self, old_filename, new_filename):
+        if old_filename == new_filename:
+            return
         try:
-            if old_filename == new_filename:
-                return
-            os.rename(encodeFilename(old_filename), encodeFilename(new_filename))
+            os.replace(old_filename, new_filename)
         except (IOError, OSError) as err:
-            self.report_error('unable to rename file: %s' % error_to_compat_str(err))
+            self.report_error(f'unable to rename file: {err}')
 
     def try_utime(self, filename, last_modified_hdr):
         """Try to set the last-modified time of the given file."""
@@ -320,12 +323,9 @@ class FileDownloader(object):
             '[download] Got server HTTP error: %s. Retrying (attempt %d of %s) ...'
             % (error_to_compat_str(err), count, self.format_retries(retries)))
 
-    def report_file_already_downloaded(self, file_name):
+    def report_file_already_downloaded(self, *args, **kwargs):
         """Report file has already been fully downloaded."""
-        try:
-            self.to_screen('[download] %s has already been downloaded' % file_name)
-        except UnicodeEncodeError:
-            self.to_screen('[download] The file has already been downloaded')
+        return self.ydl.report_file_already_downloaded(*args, **kwargs)
 
     def report_unable_to_resume(self):
         """Report it was impossible to resume download."""
@@ -343,7 +343,7 @@ class FileDownloader(object):
         """
 
         nooverwrites_and_exists = (
-            not self.params.get('overwrites', subtitle)
+            not self.params.get('overwrites', True)
             and os.path.exists(encodeFilename(filename))
         )
 

yt_dlp/downloader/dash.py

@@ -1,6 +1,6 @@
 from __future__ import unicode_literals
 
-from ..downloader import _get_real_downloader
+from ..downloader import get_suitable_downloader
 from .fragment import FragmentFD
 
 from ..utils import urljoin
@@ -15,11 +15,15 @@ class DashSegmentsFD(FragmentFD):
     FD_NAME = 'dashsegments'
 
     def real_download(self, filename, info_dict):
+        if info_dict.get('is_live'):
+            self.report_error('Live DASH videos are not supported')
+
         fragment_base_url = info_dict.get('fragment_base_url')
         fragments = info_dict['fragments'][:1] if self.params.get(
             'test', False) else info_dict['fragments']
 
-        real_downloader = _get_real_downloader(info_dict, 'dash_frag_urls', self.params, None)
+        real_downloader = get_suitable_downloader(
+            info_dict, self.params, None, protocol='dash_frag_urls', to_stdout=(filename == '-'))
 
         ctx = {
             'filename': filename,
@@ -54,9 +58,6 @@ class DashSegmentsFD(FragmentFD):
             info_copy = info_dict.copy()
             info_copy['fragments'] = fragments_to_download
             fd = real_downloader(self.ydl, self.params)
-            # TODO: Make progress updates work without hooking twice
-            # for ph in self._progress_hooks:
-            #     fd.add_progress_hook(ph)
             return fd.real_download(filename, info_copy)
 
         return self.download_and_append_fragments(ctx, fragments_to_download, info_dict)

yt_dlp/downloader/external.py

@@ -22,7 +22,7 @@ from ..utils import (
     cli_option,
     cli_valueless_option,
     cli_bool_option,
-    cli_configuration_args,
+    _configuration_args,
     encodeFilename,
     encodeArgument,
     handle_youtubedl_headers,
@@ -36,6 +36,7 @@ from ..utils import (
 
 class ExternalFD(FileDownloader):
     SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps')
+    can_download_to_stdout = False
 
     def real_download(self, filename, info_dict):
         self.report_destination(filename)
@@ -93,7 +94,9 @@ class ExternalFD(FileDownloader):
 
     @classmethod
     def supports(cls, info_dict):
-        return info_dict['protocol'] in cls.SUPPORTED_PROTOCOLS
+        return (
+            (cls.can_download_to_stdout or not info_dict.get('to_stdout'))
+            and info_dict['protocol'] in cls.SUPPORTED_PROTOCOLS)
 
     @classmethod
     def can_download(cls, info_dict, path=None):
@@ -108,11 +111,10 @@ class ExternalFD(FileDownloader):
     def _valueless_option(self, command_option, param, expected_value=True):
         return cli_valueless_option(self.params, command_option, param, expected_value)
 
-    def _configuration_args(self, *args, **kwargs):
-        return cli_configuration_args(
-            self.params.get('external_downloader_args'),
-            [self.get_basename(), 'default'],
-            *args, **kwargs)
+    def _configuration_args(self, keys=None, *args, **kwargs):
+        return _configuration_args(
+            self.get_basename(), self.params.get('external_downloader_args'), self.get_basename(),
+            keys, *args, **kwargs)
 
     def _call_downloader(self, tmpfilename, info_dict):
         """ Either overwrite this or implement _make_cmd """
@@ -286,6 +288,7 @@ class Aria2cFD(ExternalFD):
         if info_dict.get('http_headers') is not None:
             for key, val in info_dict['http_headers'].items():
                 cmd += ['--header', '%s: %s' % (key, val)]
+        cmd += self._option('--max-overall-download-limit', 'ratelimit')
         cmd += self._option('--interface', 'source_address')
         cmd += self._option('--all-proxy', 'proxy')
         cmd += self._bool_option('--check-certificate', 'nocheckcertificate', 'false', 'true', '=')
@@ -340,17 +343,28 @@ class HttpieFD(ExternalFD):
 
 
 class FFmpegFD(ExternalFD):
-    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'm3u8_native', 'rtsp', 'rtmp', 'rtmp_ffmpeg', 'mms')
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'm3u8_native', 'rtsp', 'rtmp', 'rtmp_ffmpeg', 'mms', 'http_dash_segments')
+    can_download_to_stdout = True
 
     @classmethod
     def available(cls, path=None):
         # TODO: Fix path for ffmpeg
+        # Fixme: This may be wrong when --ffmpeg-location is used
         return FFmpegPostProcessor().available
 
     def on_process_started(self, proc, stdin):
         """ Override this in subclasses  """
         pass
 
+    @classmethod
+    def can_merge_formats(cls, info_dict, params={}):
+        return (
+            info_dict.get('requested_formats')
+            and info_dict.get('protocol')
+            and not params.get('allow_unplayable_formats')
+            and 'no-direct-merge' not in params.get('compat_opts', [])
+            and cls.can_download(info_dict))
+
     def _call_downloader(self, tmpfilename, info_dict):
         urls = [f['url'] for f in info_dict.get('requested_formats', [])] or [info_dict['url']]
         ffpp = FFmpegPostProcessor(downloader=self)
@@ -368,6 +382,9 @@ class FFmpegFD(ExternalFD):
         if not self.params.get('verbose'):
             args += ['-hide_banner']
 
+        args += info_dict.get('_ffmpeg_args', [])
+
+        # This option exists only for compatibility. Extractors should use `_ffmpeg_args` instead
         seekable = info_dict.get('_seekable')
         if seekable is not None:
             # setting -seekable prevents ffmpeg from guessing if the server
@@ -442,20 +459,20 @@ class FFmpegFD(ExternalFD):
             elif isinstance(conn, compat_str):
                 args += ['-rtmp_conn', conn]
 
-        for url in urls:
-            args += ['-i', url]
+        for i, url in enumerate(urls):
+            args += self._configuration_args((f'_i{i + 1}', '_i')) + ['-i', url]
 
-        args += self._configuration_args() + ['-c', 'copy']
-        if info_dict.get('requested_formats'):
-            for (i, fmt) in enumerate(info_dict['requested_formats']):
-                if fmt.get('acodec') != 'none':
-                    args.extend(['-map', '%d:a:0' % i])
-                if fmt.get('vcodec') != 'none':
-                    args.extend(['-map', '%d:v:0' % i])
+        args += ['-c', 'copy']
+        if info_dict.get('requested_formats') or protocol == 'http_dash_segments':
+            for (i, fmt) in enumerate(info_dict.get('requested_formats') or [info_dict]):
+                stream_number = fmt.get('manifest_stream_number', 0)
+                a_or_v = 'a' if fmt.get('acodec') != 'none' else 'v'
+                args.extend(['-map', f'{i}:{a_or_v}:{stream_number}'])
 
         if self.params.get('test', False):
             args += ['-fs', compat_str(self._TEST_FILE_SIZE)]
 
+        ext = info_dict['ext']
         if protocol in ('m3u8', 'm3u8_native'):
             use_mpegts = (tmpfilename == '-') or self.params.get('hls_use_mpegts')
             if use_mpegts is None:
@@ -468,12 +485,15 @@ class FFmpegFD(ExternalFD):
                     args += ['-bsf:a', 'aac_adtstoasc']
         elif protocol == 'rtmp':
             args += ['-f', 'flv']
+        elif ext == 'mp4' and tmpfilename == '-':
+            args += ['-f', 'mpegts']
         else:
-            args += ['-f', EXT_TO_OUT_FORMATS.get(info_dict['ext'], info_dict['ext'])]
+            args += ['-f', EXT_TO_OUT_FORMATS.get(ext, ext)]
+
+        args += self._configuration_args(('_o1', '_o', ''))
 
         args = [encodeArgument(opt) for opt in args]
         args.append(encodeFilename(ffpp._ffmpeg_filename_argument(tmpfilename), True))
-
         self._debug_cmd(args)
 
         proc = subprocess.Popen(args, stdin=subprocess.PIPE, env=env)

yt_dlp/downloader/fragment.py

@@ -105,17 +105,19 @@ class FragmentFD(FileDownloader):
 
     def _write_ytdl_file(self, ctx):
         frag_index_stream, _ = sanitize_open(self.ytdl_filename(ctx['filename']), 'w')
-        downloader = {
-            'current_fragment': {
-                'index': ctx['fragment_index'],
-            },
-        }
-        if 'extra_state' in ctx:
-            downloader['extra_state'] = ctx['extra_state']
-        if ctx.get('fragment_count') is not None:
-            downloader['fragment_count'] = ctx['fragment_count']
-        frag_index_stream.write(json.dumps({'downloader': downloader}))
-        frag_index_stream.close()
+        try:
+            downloader = {
+                'current_fragment': {
+                    'index': ctx['fragment_index'],
+                },
+            }
+            if 'extra_state' in ctx:
+                downloader['extra_state'] = ctx['extra_state']
+            if ctx.get('fragment_count') is not None:
+                downloader['fragment_count'] = ctx['fragment_count']
+            frag_index_stream.write(json.dumps({'downloader': downloader}))
+        finally:
+            frag_index_stream.close()
 
     def _download_fragment(self, ctx, frag_url, info_dict, headers=None, request_data=None):
         fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], ctx['fragment_index'])
@@ -327,7 +329,7 @@ class FragmentFD(FileDownloader):
             'fragment_index': 0,
         })
 
-    def download_and_append_fragments(self, ctx, fragments, info_dict, pack_func=None):
+    def download_and_append_fragments(self, ctx, fragments, info_dict, *, pack_func=None, finish_func=None):
         fragment_retries = self.params.get('fragment_retries', 0)
         is_fatal = (lambda idx: idx == 0) if self.params.get('skip_unavailable_fragments', True) else (lambda _: True)
         if not pack_func:
@@ -422,5 +424,8 @@ class FragmentFD(FileDownloader):
                 if not result:
                     return False
 
+        if finish_func is not None:
+            ctx['dest_stream'].write(finish_func())
+            ctx['dest_stream'].flush()
         self._finish_frag_download(ctx, info_dict)
         return True

yt_dlp/downloader/hls.py

@@ -4,7 +4,7 @@ import re
 import io
 import binascii
 
-from ..downloader import _get_real_downloader
+from ..downloader import get_suitable_downloader
 from .fragment import FragmentFD, can_decrypt_frag
 from .external import FFmpegFD
 
@@ -80,16 +80,14 @@ class HlsFD(FragmentFD):
             fd = FFmpegFD(self.ydl, self.params)
             self.report_warning(
                 '%s detected unsupported features; extraction will be delegated to %s' % (self.FD_NAME, fd.get_basename()))
-            # TODO: Make progress updates work without hooking twice
-            # for ph in self._progress_hooks:
-            #     fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
 
         is_webvtt = info_dict['ext'] == 'vtt'
         if is_webvtt:
             real_downloader = None  # Packing the fragments is not currently supported for external downloader
         else:
-            real_downloader = _get_real_downloader(info_dict, 'm3u8_frag_urls', self.params, None)
+            real_downloader = get_suitable_downloader(
+                info_dict, self.params, None, protocol='m3u8_frag_urls', to_stdout=(filename == '-'))
         if real_downloader and not real_downloader.supports_manifest(s):
             real_downloader = None
         if real_downloader:
@@ -256,35 +254,47 @@ class HlsFD(FragmentFD):
             def pack_fragment(frag_content, frag_index):
                 output = io.StringIO()
                 adjust = 0
+                overflow = False
+                mpegts_last = None
                 for block in webvtt.parse_fragment(frag_content):
                     if isinstance(block, webvtt.CueBlock):
+                        extra_state['webvtt_mpegts_last'] = mpegts_last
+                        if overflow:
+                            extra_state['webvtt_mpegts_adjust'] += 1
+                            overflow = False
                         block.start += adjust
                         block.end += adjust
 
                         dedup_window = extra_state.setdefault('webvtt_dedup_window', [])
-                        cue = block.as_json
 
-                        # skip the cue if an identical one appears
-                        # in the window of potential duplicates
-                        # and prune the window of unviable candidates
+                        ready = []
+
                         i = 0
-                        skip = True
+                        is_new = True
                         while i < len(dedup_window):
-                            window_cue = dedup_window[i]
-                            if window_cue == cue:
-                                break
-                            if window_cue['end'] >= cue['start']:
-                                i += 1
+                            wcue = dedup_window[i]
+                            wblock = webvtt.CueBlock.from_json(wcue)
+                            i += 1
+                            if wblock.hinges(block):
+                                wcue['end'] = block.end
+                                is_new = False
                                 continue
+                            if wblock == block:
+                                is_new = False
+                                continue
+                            if wblock.end > block.start:
+                                continue
+                            ready.append(wblock)
+                            i -= 1
                             del dedup_window[i]
-                        else:
-                            skip = False
 
-                        if skip:
-                            continue
+                        if is_new:
+                            dedup_window.append(block.as_json)
+                        for block in ready:
+                            block.write_into(output)
 
-                        # add the cue to the window
-                        dedup_window.append(cue)
+                        # we only emit cues once they fall out of the duplicate window
+                        continue
                     elif isinstance(block, webvtt.Magic):
                         # take care of MPEG PES timestamp overflow
                         if block.mpegts is None:
@@ -292,9 +302,9 @@ class HlsFD(FragmentFD):
                         extra_state.setdefault('webvtt_mpegts_adjust', 0)
                         block.mpegts += extra_state['webvtt_mpegts_adjust'] << 33
                         if block.mpegts < extra_state.get('webvtt_mpegts_last', 0):
-                            extra_state['webvtt_mpegts_adjust'] += 1
+                            overflow = True
                             block.mpegts += 1 << 33
-                        extra_state['webvtt_mpegts_last'] = block.mpegts
+                        mpegts_last = block.mpegts
 
                         if frag_index == 1:
                             extra_state['webvtt_mpegts'] = block.mpegts or 0
@@ -319,6 +329,19 @@ class HlsFD(FragmentFD):
                     block.write_into(output)
 
                 return output.getvalue().encode('utf-8')
+
+            def fin_fragments():
+                dedup_window = extra_state.get('webvtt_dedup_window')
+                if not dedup_window:
+                    return b''
+
+                output = io.StringIO()
+                for cue in dedup_window:
+                    webvtt.CueBlock.from_json(cue).write_into(output)
+
+                return output.getvalue().encode('utf-8')
+
+            self.download_and_append_fragments(
+                ctx, fragments, info_dict, pack_func=pack_fragment, finish_func=fin_fragments)
         else:
-            pack_fragment = None
-        return self.download_and_append_fragments(ctx, fragments, info_dict, pack_fragment)
+            return self.download_and_append_fragments(ctx, fragments, info_dict)

yt_dlp/downloader/http.py

@@ -238,7 +238,7 @@ class HttpFD(FileDownloader):
             while True:
                 try:
                     # Download and write
-                    data_block = ctx.data.read(block_size if data_len is None else min(block_size, data_len - byte_counter))
+                    data_block = ctx.data.read(block_size if not is_test else min(block_size, data_len - byte_counter))
                 # socket.timeout is a subclass of socket.error but may not have
                 # errno set
                 except socket.timeout as e:

yt_dlp/downloader/niconico.py

@@ -4,7 +4,7 @@ from __future__ import unicode_literals
 import threading
 
 from .common import FileDownloader
-from ..downloader import _get_real_downloader
+from ..downloader import get_suitable_downloader
 from ..extractor.niconico import NiconicoIE
 from ..compat import compat_urllib_request
 
@@ -20,7 +20,7 @@ class NiconicoDmcFD(FileDownloader):
         ie = NiconicoIE(self.ydl)
         info_dict, heartbeat_info_dict = ie._get_heartbeat_info(info_dict)
 
-        fd = _get_real_downloader(info_dict, params=self.params)(self.ydl, self.params)
+        fd = get_suitable_downloader(info_dict, params=self.params)(self.ydl, self.params)
 
         success = download_complete = False
         timer = [None]

yt_dlp/extractor/abcnews.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .amp import AMPIE
 from .common import InfoExtractor
@@ -59,7 +58,7 @@ class AbcNewsVideoIE(AMPIE):
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         display_id = mobj.group('display_id')
         video_id = mobj.group('id')
         info_dict = self._extract_feed_info(

yt_dlp/extractor/abcotvs.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..compat import compat_str
@@ -55,7 +54,7 @@ class ABCOTVSIE(InfoExtractor):
     }
 
     def _real_extract(self, url):
-        site, display_id, video_id = re.match(self._VALID_URL, url).groups()
+        site, display_id, video_id = self._match_valid_url(url).groups()
         display_id = display_id or video_id
         station = self._SITE_MAP[site]
 

yt_dlp/extractor/acast.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..utils import (
@@ -80,7 +79,7 @@ class ACastIE(ACastBaseIE):
     }]
 
     def _real_extract(self, url):
-        channel, display_id = re.match(self._VALID_URL, url).groups()
+        channel, display_id = self._match_valid_url(url).groups()
         episode = self._call_api(
             '%s/episodes/%s' % (channel, display_id),
             display_id, {'showInfo': 'true'})

yt_dlp/extractor/adobepass.py

@@ -76,6 +76,11 @@ MSO_INFO = {
         'username_field': 'IDToken1',
         'password_field': 'IDToken2',
     },
+    'Cablevision': {
+        'name': 'Optimum/Cablevision',
+        'username_field': 'j_username',
+        'password_field': 'j_password',
+    },
     'thr030': {
         'name': '3 Rivers Communications'
     },
@@ -1330,6 +1335,11 @@ MSO_INFO = {
     'cou060': {
         'name': 'Zito Media'
     },
+    'slingtv': {
+        'name': 'Sling TV',
+        'username_field': 'username',
+        'password_field': 'password',
+    },
 }
 
 
@@ -1498,7 +1508,8 @@ class AdobePassIE(InfoExtractor):
                     # In general, if you're connecting from a Verizon-assigned IP,
                     # you will not actually pass your credentials.
                     provider_redirect_page, urlh = provider_redirect_page_res
-                    if 'Please wait ...' in provider_redirect_page:
+                    # From non-Verizon IP, still gave 'Please wait', but noticed N==Y; will need to try on Verizon IP
+                    if 'Please wait ...' in provider_redirect_page and '\'N\'== "Y"' not in provider_redirect_page:
                         saml_redirect_url = self._html_search_regex(
                             r'self\.parent\.location=(["\'])(?P<url>.+?)\1',
                             provider_redirect_page,
@@ -1506,7 +1517,8 @@ class AdobePassIE(InfoExtractor):
                         saml_login_page = self._download_webpage(
                             saml_redirect_url, video_id,
                             'Downloading SAML Login Page')
-                    else:
+                    elif 'Verizon FiOS - sign in' in provider_redirect_page:
+                        # FXNetworks from non-Verizon IP
                         saml_login_page_res = post_form(
                             provider_redirect_page_res, 'Logging in', {
                                 mso_info['username_field']: username,
@@ -1516,6 +1528,26 @@ class AdobePassIE(InfoExtractor):
                         if 'Please try again.' in saml_login_page:
                             raise ExtractorError(
                                 'We\'re sorry, but either the User ID or Password entered is not correct.')
+                    else:
+                        # ABC from non-Verizon IP
+                        saml_redirect_url = self._html_search_regex(
+                            r'var\surl\s*=\s*(["\'])(?P<url>.+?)\1',
+                            provider_redirect_page,
+                            'SAML Redirect URL', group='url')
+                        saml_redirect_url = saml_redirect_url.replace(r'\/', '/')
+                        saml_redirect_url = saml_redirect_url.replace(r'\-', '-')
+                        saml_redirect_url = saml_redirect_url.replace(r'\x26', '&')
+                        saml_login_page = self._download_webpage(
+                            saml_redirect_url, video_id,
+                            'Downloading SAML Login Page')
+                        saml_login_page, urlh = post_form(
+                            [saml_login_page, saml_redirect_url], 'Logging in', {
+                                mso_info['username_field']: username,
+                                mso_info['password_field']: password,
+                            })
+                        if 'Please try again.' in saml_login_page:
+                            raise ExtractorError(
+                                'Failed to login, incorrect User ID or Password.')
                     saml_login_url = self._search_regex(
                         r'xmlHttp\.open\("POST"\s*,\s*(["\'])(?P<url>.+?)\1',
                         saml_login_page, 'SAML Login URL', group='url')
@@ -1565,6 +1597,40 @@ class AdobePassIE(InfoExtractor):
                         }), headers={
                             'Content-Type': 'application/x-www-form-urlencoded'
                         })
+                elif mso_id == 'slingtv':
+                    # SlingTV has a meta-refresh based authentication, but also
+                    # looks at the tab history to count the number of times the
+                    # browser has been on a page
+
+                    first_bookend_page, urlh = provider_redirect_page_res
+
+                    hidden_data = self._hidden_inputs(first_bookend_page)
+                    hidden_data['history'] = 1
+
+                    provider_login_page_res = self._download_webpage_handle(
+                        urlh.geturl(), video_id, 'Sending first bookend',
+                        query=hidden_data)
+
+                    provider_association_redirect, urlh = post_form(
+                        provider_login_page_res, 'Logging in', {
+                            mso_info['username_field']: username,
+                            mso_info['password_field']: password
+                        })
+
+                    provider_refresh_redirect_url = extract_redirect_url(
+                        provider_association_redirect, url=urlh.geturl())
+
+                    last_bookend_page, urlh = self._download_webpage_handle(
+                        provider_refresh_redirect_url, video_id,
+                        'Downloading Auth Association Redirect Page')
+                    hidden_data = self._hidden_inputs(last_bookend_page)
+                    hidden_data['history'] = 3
+
+                    mvpd_confirm_page_res = self._download_webpage_handle(
+                        urlh.geturl(), video_id, 'Sending final bookend',
+                        query=hidden_data)
+
+                    post_form(mvpd_confirm_page_res, 'Confirming Login')
                 else:
                     # Some providers (e.g. DIRECTV NOW) have another meta refresh
                     # based redirect that should be followed.
@@ -1577,10 +1643,13 @@ class AdobePassIE(InfoExtractor):
                             'Downloading Provider Redirect Page (meta refresh)')
                     provider_login_page_res = post_form(
                         provider_redirect_page_res, self._DOWNLOADING_LOGIN_PAGE)
-                    mvpd_confirm_page_res = post_form(provider_login_page_res, 'Logging in', {
+                    form_data = {
                         mso_info.get('username_field', 'username'): username,
-                        mso_info.get('password_field', 'password'): password,
-                    })
+                        mso_info.get('password_field', 'password'): password
+                    }
+                    if mso_id == 'Cablevision':
+                        form_data['_eventId_proceed'] = ''
+                    mvpd_confirm_page_res = post_form(provider_login_page_res, 'Logging in', form_data)
                     if mso_id != 'Rogers':
                         post_form(mvpd_confirm_page_res, 'Confirming Login')
 

yt_dlp/extractor/adobetv.py

@@ -132,7 +132,7 @@ class AdobeTVIE(AdobeTVBaseIE):
     }
 
     def _real_extract(self, url):
-        language, show_urlname, urlname = re.match(self._VALID_URL, url).groups()
+        language, show_urlname, urlname = self._match_valid_url(url).groups()
         if not language:
             language = 'en'
 
@@ -178,7 +178,7 @@ class AdobeTVShowIE(AdobeTVPlaylistBaseIE):
     _process_data = AdobeTVBaseIE._parse_video_data
 
     def _real_extract(self, url):
-        language, show_urlname = re.match(self._VALID_URL, url).groups()
+        language, show_urlname = self._match_valid_url(url).groups()
         if not language:
             language = 'en'
         query = {
@@ -215,7 +215,7 @@ class AdobeTVChannelIE(AdobeTVPlaylistBaseIE):
             show_data['url'], 'AdobeTVShow', str_or_none(show_data.get('id')))
 
     def _real_extract(self, url):
-        language, channel_urlname, category_urlname = re.match(self._VALID_URL, url).groups()
+        language, channel_urlname, category_urlname = self._match_valid_url(url).groups()
         if not language:
             language = 'en'
         query = {

yt_dlp/extractor/adultswim.py

@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 
 import json
-import re
 
 from .turner import TurnerBaseIE
 from ..utils import (
@@ -89,7 +88,7 @@ class AdultSwimIE(TurnerBaseIE):
     }]
 
     def _real_extract(self, url):
-        show_path, episode_path = re.match(self._VALID_URL, url).groups()
+        show_path, episode_path = self._match_valid_url(url).groups()
         display_id = episode_path or show_path
         query = '''query {
   getShowBySlug(slug:"%s") {

yt_dlp/extractor/aenetworks.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .theplatform import ThePlatformIE
 from ..utils import (
@@ -20,8 +19,8 @@ class AENetworksBaseIE(ThePlatformIE):
             (?:history(?:vault)?|aetv|mylifetime|lifetimemovieclub)\.com|
             fyi\.tv
         )/'''
-    _THEPLATFORM_KEY = 'crazyjava'
-    _THEPLATFORM_SECRET = 's3cr3t'
+    _THEPLATFORM_KEY = '43jXaGRQud'
+    _THEPLATFORM_SECRET = 'S10BPXHMlb'
     _DOMAIN_MAP = {
         'history.com': ('HISTORY', 'history'),
         'aetv.com': ('AETV', 'aetv'),
@@ -170,7 +169,7 @@ class AENetworksIE(AENetworksBaseIE):
     }]
 
     def _real_extract(self, url):
-        domain, canonical = re.match(self._VALID_URL, url).groups()
+        domain, canonical = self._match_valid_url(url).groups()
         return self._extract_aetn_info(domain, 'canonical', '/' + canonical, url)
 
 
@@ -187,7 +186,7 @@ class AENetworksListBaseIE(AENetworksBaseIE):
             }))['data'][resource]
 
     def _real_extract(self, url):
-        domain, slug = re.match(self._VALID_URL, url).groups()
+        domain, slug = self._match_valid_url(url).groups()
         _, brand = self._DOMAIN_MAP[domain]
         playlist = self._call_api(self._RESOURCE, slug, brand, self._FIELDS)
         base_url = 'http://watch.%s' % domain
@@ -309,7 +308,7 @@ class HistoryPlayerIE(AENetworksBaseIE):
     _TESTS = []
 
     def _real_extract(self, url):
-        domain, video_id = re.match(self._VALID_URL, url).groups()
+        domain, video_id = self._match_valid_url(url).groups()
         return self._extract_aetn_info(domain, 'id', video_id, url)
 
 

yt_dlp/extractor/afreecatv.py

@@ -6,9 +6,11 @@ import re
 from .common import InfoExtractor
 from ..compat import compat_xpath
 from ..utils import (
+    date_from_str,
     determine_ext,
     ExtractorError,
     int_or_none,
+    unified_strdate,
     url_or_none,
     urlencode_postdata,
     xpath_text,
@@ -237,6 +239,7 @@ class AfreecaTVIE(InfoExtractor):
             r'nTitleNo\s*=\s*(\d+)', webpage, 'title', default=video_id)
 
         partial_view = False
+        adult_view = False
         for _ in range(2):
             query = {
                 'nTitleNo': video_id,
@@ -245,6 +248,8 @@ class AfreecaTVIE(InfoExtractor):
             }
             if partial_view:
                 query['partialView'] = 'SKIP_ADULT'
+            if adult_view:
+                query['adultView'] = 'ADULT_VIEW'
             video_xml = self._download_xml(
                 'http://afbbs.afreecatv.com:8080/api/video/get_video_info.php',
                 video_id, 'Downloading video info XML%s'
@@ -264,6 +269,9 @@ class AfreecaTVIE(InfoExtractor):
                 partial_view = True
                 continue
             elif flag == 'ADULT':
+                if not adult_view:
+                    adult_view = True
+                    continue
                 error = 'Only users older than 19 are able to watch this video. Provide account credentials to download this content.'
             else:
                 error = flag
@@ -309,8 +317,15 @@ class AfreecaTVIE(InfoExtractor):
                 if not file_url:
                     continue
                 key = file_element.get('key', '')
-                upload_date = self._search_regex(
-                    r'^(\d{8})_', key, 'upload date', default=None)
+                upload_date = unified_strdate(self._search_regex(
+                    r'^(\d{8})_', key, 'upload date', default=None))
+                if upload_date is not None:
+                    # sometimes the upload date isn't included in the file name
+                    # instead, another random ID is, which may parse as a valid
+                    # date but be wildly out of a reasonable range
+                    parsed_date = date_from_str(upload_date)
+                    if parsed_date.year < 2000 or parsed_date.year >= 2100:
+                        upload_date = None
                 file_duration = int_or_none(file_element.get('duration'))
                 format_id = key if key else '%s_%s' % (video_id, file_num)
                 if determine_ext(file_url) == 'm3u8':

yt_dlp/extractor/aljazeera.py

@@ -1,7 +1,6 @@
 from __future__ import unicode_literals
 
 import json
-import re
 
 from .common import InfoExtractor
 
@@ -32,7 +31,7 @@ class AlJazeeraIE(InfoExtractor):
     BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_default/index.html?videoId=%s'
 
     def _real_extract(self, url):
-        post_type, name = re.match(self._VALID_URL, url).groups()
+        post_type, name = self._match_valid_url(url).groups()
         post_type = {
             'features': 'post',
             'program': 'episode',
@@ -40,7 +39,7 @@ class AlJazeeraIE(InfoExtractor):
         }[post_type.split('/')[0]]
         video = self._download_json(
             'https://www.aljazeera.com/graphql', name, query={
-                'operationName': 'SingleArticleQuery',
+                'operationName': 'ArchipelagoSingleArticleQuery',
                 'variables': json.dumps({
                     'name': name,
                     'postType': post_type,

yt_dlp/extractor/alura.py

@@ -42,8 +42,7 @@ class AluraIE(InfoExtractor):
 
     def _real_extract(self, url):
 
-        video_id = self._match_id(url)
-        course = self._search_regex(self._VALID_URL, url, 'post url', group='course_name')
+        course, video_id = self._match_valid_url(url)
         video_url = self._VIDEO_URL % (course, video_id)
 
         video_dict = self._download_json(video_url, video_id, 'Searching for videos')

yt_dlp/extractor/amcnetworks.py

@@ -63,7 +63,7 @@ class AMCNetworksIE(ThePlatformIE):
     }
 
     def _real_extract(self, url):
-        site, display_id = re.match(self._VALID_URL, url).groups()
+        site, display_id = self._match_valid_url(url).groups()
         requestor_id = self._REQUESTOR_ID_MAP[site]
         page_data = self._download_json(
             'https://content-delivery-gw.svc.ds.amcn.com/api/v2/content/amcn/%s/url/%s'

yt_dlp/extractor/americastestkitchen.py

@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 
 import json
-import re
 
 from .common import InfoExtractor
 from ..utils import (
@@ -69,7 +68,7 @@ class AmericasTestKitchenIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        resource_type, video_id = re.match(self._VALID_URL, url).groups()
+        resource_type, video_id = self._match_valid_url(url).groups()
         is_episode = resource_type == 'episode'
         if is_episode:
             resource_type = 'episodes'
@@ -114,7 +113,7 @@ class AmericasTestKitchenSeasonIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        show_name, season_number = re.match(self._VALID_URL, url).groups()
+        show_name, season_number = self._match_valid_url(url).groups()
         season_number = int(season_number)
 
         slug = 'atk' if show_name == 'americastestkitchen' else 'cco'

yt_dlp/extractor/anvato.py

@@ -390,7 +390,7 @@ class AnvatoIE(InfoExtractor):
             'countries': smuggled_data.get('geo_countries'),
         })
 
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         access_key, video_id = mobj.group('access_key_or_mcp', 'id')
         if access_key not in self._ANVACK_TABLE:
             access_key = self._MCP_TO_ACCESS_KEY_TABLE.get(

yt_dlp/extractor/aol.py

@@ -4,13 +4,10 @@ from __future__ import unicode_literals
 import re
 
 from .yahoo import YahooIE
-from ..compat import (
-    compat_parse_qs,
-    compat_urllib_parse_urlparse,
-)
 from ..utils import (
     ExtractorError,
     int_or_none,
+    parse_qs,
     url_or_none,
 )
 
@@ -119,7 +116,7 @@ class AolIE(YahooIE):
                         'height': int(mobj.group(2)),
                     })
                 else:
-                    qs = compat_parse_qs(compat_urllib_parse_urlparse(video_url).query)
+                    qs = parse_qs(video_url)
                     f.update({
                         'width': int_or_none(qs.get('w', [None])[0]),
                         'height': int_or_none(qs.get('h', [None])[0]),

yt_dlp/extractor/apa.py

@@ -42,7 +42,7 @@ class APAIE(InfoExtractor):
                 webpage)]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         video_id, base_url = mobj.group('id', 'base_url')
 
         webpage = self._download_webpage(

yt_dlp/extractor/appletrailers.py

@@ -94,7 +94,7 @@ class AppleTrailersIE(InfoExtractor):
     _JSON_RE = r'iTunes.playURL\((.*?)\);'
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         movie = mobj.group('movie')
         uploader_id = mobj.group('company')
 

yt_dlp/extractor/archiveorg.py

@@ -9,8 +9,6 @@ from .youtube import YoutubeIE
 from ..compat import (
     compat_urllib_parse_unquote,
     compat_urllib_parse_unquote_plus,
-    compat_urlparse,
-    compat_parse_qs,
     compat_HTTPError
 )
 from ..utils import (
@@ -25,6 +23,7 @@ from ..utils import (
     merge_dicts,
     mimetype2ext,
     parse_duration,
+    parse_qs,
     RegexNotFoundError,
     str_to_int,
     str_or_none,
@@ -399,7 +398,7 @@ class YoutubeWebArchiveIE(InfoExtractor):
                     expected=True)
             raise
         video_file_url = compat_urllib_parse_unquote(video_file_webpage.url)
-        video_file_url_qs = compat_parse_qs(compat_urlparse.urlparse(video_file_url).query)
+        video_file_url_qs = parse_qs(video_file_url)
 
         # Attempt to recover any ext & format info from playback url
         format = {'url': video_file_url}

yt_dlp/extractor/arcpublishing.py

@@ -86,7 +86,7 @@ class ArcPublishingIE(InfoExtractor):
         return entries
 
     def _real_extract(self, url):
-        org, uuid = re.match(self._VALID_URL, url).groups()
+        org, uuid = self._match_valid_url(url).groups()
         for orgs, tmpl in self._POWA_DEFAULTS:
             if org in orgs:
                 base_api_tmpl = tmpl

yt_dlp/extractor/ard.py

@@ -199,7 +199,7 @@ class ARDMediathekIE(ARDMediathekBaseIE):
 
     def _real_extract(self, url):
         # determine video id from url
-        m = re.match(self._VALID_URL, url)
+        m = self._match_valid_url(url)
 
         document_id = None
 
@@ -325,7 +325,7 @@ class ARDIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         display_id = mobj.group('id')
 
         player_url = mobj.group('mainurl') + '~playerXml.xml'
@@ -525,7 +525,7 @@ class ARDBetaMediathekIE(ARDMediathekBaseIE):
         return self.playlist_result(entries, playlist_title=display_id)
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         video_id = mobj.group('video_id')
         display_id = mobj.group('display_id')
         if display_id:

yt_dlp/extractor/arkena.py

@@ -4,12 +4,12 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..compat import compat_urlparse
 from ..utils import (
     ExtractorError,
     float_or_none,
     int_or_none,
     parse_iso8601,
+    parse_qs,
     try_get,
 )
 
@@ -63,13 +63,13 @@ class ArkenaIE(InfoExtractor):
             return mobj.group('url')
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         video_id = mobj.group('id')
         account_id = mobj.group('account_id')
 
         # Handle http://video.arkena.com/play2/embed/player URL
         if not video_id:
-            qs = compat_urlparse.parse_qs(compat_urlparse.urlparse(url).query)
+            qs = parse_qs(url)
             video_id = qs.get('mediaId', [None])[0]
             account_id = qs.get('accountId', [None])[0]
             if not video_id or not account_id:

yt_dlp/extractor/arte.py

@@ -6,11 +6,11 @@ import re
 from .common import InfoExtractor
 from ..compat import (
     compat_str,
-    compat_urlparse,
 )
 from ..utils import (
     ExtractorError,
     int_or_none,
+    parse_qs,
     qualities,
     try_get,
     unified_strdate,
@@ -49,7 +49,7 @@ class ArteTVIE(ArteTVBaseIE):
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         video_id = mobj.group('id')
         lang = mobj.group('lang') or mobj.group('lang_2')
 
@@ -204,7 +204,7 @@ class ArteTVEmbedIE(InfoExtractor):
             webpage)]
 
     def _real_extract(self, url):
-        qs = compat_urlparse.parse_qs(compat_urlparse.urlparse(url).query)
+        qs = parse_qs(url)
         json_url = qs['json_url'][0]
         video_id = ArteTVIE._match_id(json_url)
         return self.url_result(
@@ -227,7 +227,7 @@ class ArteTVPlaylistIE(ArteTVBaseIE):
     }]
 
     def _real_extract(self, url):
-        lang, playlist_id = re.match(self._VALID_URL, url).groups()
+        lang, playlist_id = self._match_valid_url(url).groups()
         collection = self._download_json(
             '%s/collectionData/%s/%s?source=videos'
             % (self._API_BASE, lang, playlist_id), playlist_id)

yt_dlp/extractor/asiancrush.py

@@ -111,7 +111,7 @@ class AsianCrushIE(AsianCrushBaseIE):
     }]
 
     def _real_extract(self, url):
-        host, video_id = re.match(self._VALID_URL, url).groups()
+        host, video_id = self._match_valid_url(url).groups()
 
         if host == 'cocoro.tv':
             webpage = self._download_webpage(url, video_id)
@@ -161,7 +161,7 @@ class AsianCrushPlaylistIE(AsianCrushBaseIE):
             yield self._parse_video_data(video)
 
     def _real_extract(self, url):
-        host, playlist_id = re.match(self._VALID_URL, url).groups()
+        host, playlist_id = self._match_valid_url(url).groups()
 
         if host == 'cocoro.tv':
             webpage = self._download_webpage(url, playlist_id)

yt_dlp/extractor/atresplayer.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..compat import compat_HTTPError
@@ -75,7 +74,7 @@ class AtresPlayerIE(InfoExtractor):
         self._request_webpage(target_url, None, 'Following Target URL')
 
     def _real_extract(self, url):
-        display_id, video_id = re.match(self._VALID_URL, url).groups()
+        display_id, video_id = self._match_valid_url(url).groups()
 
         try:
             episode = self._download_json(

yt_dlp/extractor/atvat.py

@@ -4,6 +4,7 @@ from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
     determine_ext,
+    dict_get,
     int_or_none,
     unescapeHTML,
 )
@@ -12,64 +13,62 @@ from ..utils import (
 class ATVAtIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?atv\.at/(?:[^/]+/){2}(?P<id>[dv]\d+)'
     _TESTS = [{
-        'url': 'http://atv.at/aktuell/di-210317-2005-uhr/v1698449/',
-        'md5': 'c3b6b975fb3150fc628572939df205f2',
+        'url': 'https://www.atv.at/bauer-sucht-frau-die-zweite-chance/folge-1/d3390693/',
+        'md5': 'c471605591009dfb6e6c54f7e62e2807',
         'info_dict': {
-            'id': '1698447',
+            'id': '3390684',
             'ext': 'mp4',
-            'title': 'DI, 21.03.17 | 20:05 Uhr 1/1',
+            'title': 'Bauer sucht Frau - Die zweite Chance Folge 1',
         }
     }, {
-        'url': 'http://atv.at/aktuell/meinrad-knapp/d8416/',
+        'url': 'https://www.atv.at/bauer-sucht-frau-staffel-17/fuenfte-eventfolge/d3339537/',
         'only_matching': True,
     }]
 
+    def _process_source_entry(self, source, part_id):
+        source_url = source.get('url')
+        if not source_url:
+            return
+        if determine_ext(source_url) == 'm3u8':
+            return self._extract_m3u8_formats(
+                source_url, part_id, 'mp4', 'm3u8_native',
+                m3u8_id='hls', fatal=False)
+        else:
+            return [{
+                'url': source_url,
+            }]
+
+    def _process_entry(self, entry):
+        part_id = entry.get('id')
+        if not part_id:
+            return
+        formats = []
+        for source in entry.get('sources', []):
+            formats.extend(self._process_source_entry(source, part_id) or [])
+
+        self._sort_formats(formats)
+        return {
+            'id': part_id,
+            'title': entry.get('title'),
+            'duration': int_or_none(entry.get('duration')),
+            'formats': formats
+        }
+
     def _real_extract(self, url):
         display_id = self._match_id(url)
         webpage = self._download_webpage(url, display_id)
         video_data = self._parse_json(unescapeHTML(self._search_regex(
-            [r'flashPlayerOptions\s*=\s*(["\'])(?P<json>(?:(?!\1).)+)\1',
-             r'class="[^"]*jsb_video/FlashPlayer[^"]*"[^>]+data-jsb="(?P<json>[^"]+)"'],
+            r'var\splaylist\s*=\s*(?P<json>\[.*\]);',
             webpage, 'player data', group='json')),
-            display_id)['config']['initial_video']
+            display_id)
 
-        video_id = video_data['id']
-        video_title = video_data['title']
-
-        parts = []
-        for part in video_data.get('parts', []):
-            part_id = part['id']
-            part_title = part['title']
-
-            formats = []
-            for source in part.get('sources', []):
-                source_url = source.get('src')
-                if not source_url:
-                    continue
-                ext = determine_ext(source_url)
-                if ext == 'm3u8':
-                    formats.extend(self._extract_m3u8_formats(
-                        source_url, part_id, 'mp4', 'm3u8_native',
-                        m3u8_id='hls', fatal=False))
-                else:
-                    formats.append({
-                        'format_id': source.get('delivery'),
-                        'url': source_url,
-                    })
-            self._sort_formats(formats)
-
-            parts.append({
-                'id': part_id,
-                'title': part_title,
-                'thumbnail': part.get('preview_image_url'),
-                'duration': int_or_none(part.get('duration')),
-                'is_live': part.get('is_livestream'),
-                'formats': formats,
-            })
+        first_video = video_data[0]
+        video_id = first_video['id']
+        video_title = dict_get(first_video, ('tvShowTitle', 'title'))
 
         return {
             '_type': 'multi_video',
             'id': video_id,
             'title': video_title,
-            'entries': parts,
+            'entries': (self._process_entry(entry) for entry in video_data),
         }

yt_dlp/extractor/audius.py

@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 
 import random
-import re
 
 from .common import InfoExtractor
 from ..utils import ExtractorError, try_get, compat_str, str_or_none
@@ -124,7 +123,7 @@ class AudiusIE(AudiusBaseIE):
     }
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         track_id = try_get(mobj, lambda x: x.group('track_id'))
         if track_id is None:
             title = mobj.group('title')
@@ -217,7 +216,7 @@ class AudiusPlaylistIE(AudiusBaseIE):
 
     def _real_extract(self, url):
         self._select_api_base()
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         title = mobj.group('title')
         # uploader = mobj.group('uploader')
         url = self._prepare_url(url, title)

yt_dlp/extractor/awaan.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 import base64
 
 from .common import InfoExtractor
@@ -22,7 +21,7 @@ class AWAANIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?(?:awaan|dcndigital)\.ae/(?:#/)?show/(?P<show_id>\d+)/[^/]+(?:/(?P<id>\d+)/(?P<season_id>\d+))?'
 
     def _real_extract(self, url):
-        show_id, video_id, season_id = re.match(self._VALID_URL, url).groups()
+        show_id, video_id, season_id = self._match_valid_url(url).groups()
         if video_id and int(video_id) > 0:
             return self.url_result(
                 'http://awaan.ae/media/%s' % video_id, 'AWAANVideo')
@@ -154,7 +153,7 @@ class AWAANSeasonIE(InfoExtractor):
 
     def _real_extract(self, url):
         url, smuggled_data = unsmuggle_url(url, {})
-        show_id, season_id = re.match(self._VALID_URL, url).groups()
+        show_id, season_id = self._match_valid_url(url).groups()
 
         data = {}
         if season_id:

yt_dlp/extractor/azmedien.py

@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 
 import json
-import re
 
 from .common import InfoExtractor
 from .kaltura import KalturaIE
@@ -51,7 +50,7 @@ class AZMedienIE(InfoExtractor):
     _PARTNER_ID = '1719221'
 
     def _real_extract(self, url):
-        host, display_id, article_id, entry_id = re.match(self._VALID_URL, url).groups()
+        host, display_id, article_id, entry_id = self._match_valid_url(url).groups()
 
         if not entry_id:
             entry_id = self._download_json(

yt_dlp/extractor/baidu.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..utils import unescapeHTML
@@ -33,7 +32,7 @@ class BaiduVideoIE(InfoExtractor):
             path, category, playlist_id), playlist_id, note)
 
     def _real_extract(self, url):
-        category, playlist_id = re.match(self._VALID_URL, url).groups()
+        category, playlist_id = self._match_valid_url(url).groups()
         if category == 'show':
             category = 'tvshow'
         if category == 'tv':

yt_dlp/extractor/bandcamp.py

@@ -212,7 +212,7 @@ class BandcampIE(InfoExtractor):
 
 class BandcampAlbumIE(BandcampIE):
     IE_NAME = 'Bandcamp:album'
-    _VALID_URL = r'https?://(?:(?P<subdomain>[^.]+)\.)?bandcamp\.com(?:/album/(?P<id>[^/?#&]+))?'
+    _VALID_URL = r'https?://(?:(?P<subdomain>[^.]+)\.)?bandcamp\.com(?!/music)(?:/album/(?P<id>[^/?#&]+))?'
 
     _TESTS = [{
         'url': 'http://blazo.bandcamp.com/album/jazz-format-mixtape-vol-1',
@@ -294,7 +294,7 @@ class BandcampAlbumIE(BandcampIE):
                 else super(BandcampAlbumIE, cls).suitable(url))
 
     def _real_extract(self, url):
-        uploader_id, album_id = re.match(self._VALID_URL, url).groups()
+        uploader_id, album_id = self._match_valid_url(url).groups()
         playlist_id = album_id or uploader_id
         webpage = self._download_webpage(url, playlist_id)
         tralbum = self._extract_data_attr(webpage, playlist_id)
@@ -389,3 +389,43 @@ class BandcampWeeklyIE(BandcampIE):
             'episode_id': show_id,
             'formats': formats
         }
+
+
+class BandcampMusicIE(InfoExtractor):
+    _VALID_URL = r'https?://(?P<id>[^/]+)\.bandcamp\.com/music'
+    _TESTS = [{
+        'url': 'https://steviasphere.bandcamp.com/music',
+        'playlist_mincount': 47,
+        'info_dict': {
+            'id': 'steviasphere',
+        },
+    }, {
+        'url': 'https://coldworldofficial.bandcamp.com/music',
+        'playlist_mincount': 10,
+        'info_dict': {
+            'id': 'coldworldofficial',
+        },
+    }, {
+        'url': 'https://nuclearwarnowproductions.bandcamp.com/music',
+        'playlist_mincount': 399,
+        'info_dict': {
+            'id': 'nuclearwarnowproductions',
+        },
+    }
+    ]
+
+    _TYPE_IE_DICT = {
+        'album': BandcampAlbumIE.ie_key(),
+        'track': BandcampIE.ie_key()
+    }
+
+    def _real_extract(self, url):
+        id = self._match_id(url)
+        webpage = self._download_webpage(url, id)
+        items = re.findall(r'href\=\"\/(?P<path>(?P<type>album|track)+/[^\"]+)', webpage)
+        entries = [
+            self.url_result(
+                f'https://{id}.bandcamp.com/{item[0]}',
+                ie=self._TYPE_IE_DICT[item[1]])
+            for item in items]
+        return self.playlist_result(entries, id)

yt_dlp/extractor/bannedvideo.py

@@ -0,0 +1,165 @@
+from __future__ import unicode_literals
+
+import json
+
+from .common import InfoExtractor
+from ..utils import (
+    try_get,
+    int_or_none,
+    url_or_none,
+    float_or_none,
+    unified_timestamp,
+)
+
+
+class BannedVideoIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?banned\.video/watch\?id=(?P<id>[0-f]{24})'
+    _TESTS = [{
+        'url': 'https://banned.video/watch?id=5e7a859644e02200c6ef5f11',
+        'md5': '14b6e81d41beaaee2215cd75c6ed56e4',
+        'info_dict': {
+            'id': '5e7a859644e02200c6ef5f11',
+            'ext': 'mp4',
+            'title': 'China Discovers Origin of Corona Virus: Issues Emergency Statement',
+            'thumbnail': r're:^https?://(?:www\.)?assets\.infowarsmedia.com/images/',
+            'description': 'md5:560d96f02abbebe6c6b78b47465f6b28',
+            'upload_date': '20200324',
+            'timestamp': 1585087895,
+        }
+    }]
+
+    _GRAPHQL_GETMETADATA_QUERY = '''
+query GetVideoAndComments($id: String!) {
+    getVideo(id: $id) {
+        streamUrl
+        directUrl
+        unlisted
+        live
+        tags {
+            name
+        }
+        title
+        summary
+        playCount
+        largeImage
+        videoDuration
+        channel {
+            _id
+            title
+        }
+        createdAt
+    }
+    getVideoComments(id: $id, limit: 999999, offset: 0) {
+        _id
+        content
+        user {
+            _id
+            username
+        }
+        voteCount {
+            positive
+        }
+        createdAt
+        replyCount
+    }
+}'''
+
+    _GRAPHQL_GETCOMMENTSREPLIES_QUERY = '''
+query GetCommentReplies($id: String!) {
+    getCommentReplies(id: $id, limit: 999999, offset: 0) {
+        _id
+        content
+        user {
+            _id
+            username
+        }
+        voteCount {
+            positive
+        }
+        createdAt
+        replyCount
+    }
+}'''
+
+    _GRAPHQL_QUERIES = {
+        'GetVideoAndComments': _GRAPHQL_GETMETADATA_QUERY,
+        'GetCommentReplies': _GRAPHQL_GETCOMMENTSREPLIES_QUERY,
+    }
+
+    def _call_api(self, video_id, id, operation, note):
+        return self._download_json(
+            'https://api.infowarsmedia.com/graphql', video_id, note=note,
+            headers={
+                'Content-Type': 'application/json; charset=utf-8'
+            }, data=json.dumps({
+                'variables': {'id': id},
+                'operationName': operation,
+                'query': self._GRAPHQL_QUERIES[operation]
+            }).encode('utf8')).get('data')
+
+    def _extract_comments(self, video_id, comments, comment_data):
+        for comment in comment_data.copy():
+            comment_id = comment.get('_id')
+            if comment.get('replyCount') > 0:
+                reply_json = self._call_api(
+                    video_id, comment_id, 'GetCommentReplies',
+                    f'Downloading replies for comment {comment_id}')
+                comments.extend(
+                    self._parse_comment(reply, comment_id)
+                    for reply in reply_json.get('getCommentReplies'))
+
+        return {
+            'comments': comments,
+            'comment_count': len(comments),
+        }
+
+    @staticmethod
+    def _parse_comment(comment_data, parent):
+        return {
+            'id': comment_data.get('_id'),
+            'text': comment_data.get('content'),
+            'author': try_get(comment_data, lambda x: x['user']['username']),
+            'author_id': try_get(comment_data, lambda x: x['user']['_id']),
+            'timestamp': unified_timestamp(comment_data.get('createdAt')),
+            'parent': parent,
+            'like_count': try_get(comment_data, lambda x: x['voteCount']['positive']),
+        }
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        video_json = self._call_api(video_id, video_id, 'GetVideoAndComments', 'Downloading video metadata')
+        video_info = video_json['getVideo']
+        is_live = video_info.get('live')
+        comments = [self._parse_comment(comment, 'root') for comment in video_json.get('getVideoComments')]
+
+        formats = [{
+            'format_id': 'direct',
+            'quality': 1,
+            'url': video_info.get('directUrl'),
+            'ext': 'mp4',
+        }] if url_or_none(video_info.get('directUrl')) else []
+        if video_info.get('streamUrl'):
+            formats.extend(self._extract_m3u8_formats(
+                video_info.get('streamUrl'), video_id, 'mp4',
+                entry_protocol='m3u8_native', m3u8_id='hls', live=True))
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': video_info.get('title')[:-1],
+            'formats': formats,
+            'is_live': is_live,
+            'description': video_info.get('summary'),
+            'channel': try_get(video_info, lambda x: x['channel']['title']),
+            'channel_id': try_get(video_info, lambda x: x['channel']['_id']),
+            'view_count': int_or_none(video_info.get('playCount')),
+            'thumbnail': url_or_none(video_info.get('largeImage')),
+            'duration': float_or_none(video_info.get('videoDuration')),
+            'timestamp': unified_timestamp(video_info.get('createdAt')),
+            'tags': [tag.get('name') for tag in video_info.get('tags')],
+            'availability': self._availability(is_unlisted=video_info.get('unlisted')),
+            'comments': comments,
+            '__post_extractor': (
+                (lambda: self._extract_comments(video_id, comments, video_json.get('getVideoComments')))
+                if self.get_param('getcomments') else None)
+        }

yt_dlp/extractor/bbc.py

@@ -10,9 +10,7 @@ from .common import InfoExtractor
 from ..compat import (
     compat_etree_Element,
     compat_HTTPError,
-    compat_parse_qs,
     compat_str,
-    compat_urllib_parse_urlparse,
     compat_urlparse,
 )
 from ..utils import (
@@ -26,6 +24,7 @@ from ..utils import (
     js_to_json,
     parse_duration,
     parse_iso8601,
+    parse_qs,
     strip_or_none,
     try_get,
     unescapeHTML,
@@ -1410,7 +1409,7 @@ class BBCCoUkIPlayerPlaylistBaseIE(InfoExtractor):
 
     def _real_extract(self, url):
         pid = self._match_id(url)
-        qs = compat_parse_qs(compat_urllib_parse_urlparse(url).query)
+        qs = parse_qs(url)
         series_id = qs.get('seriesId', [None])[0]
         page = qs.get('page', [None])[0]
         per_page = 36 if page else self._PAGE_SIZE

yt_dlp/extractor/beatport.py

@@ -40,7 +40,7 @@ class BeatportIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         track_id = mobj.group('id')
         display_id = mobj.group('display_id')
 

yt_dlp/extractor/beeg.py

@@ -3,10 +3,10 @@ from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..compat import (
     compat_str,
-    compat_urlparse,
 )
 from ..utils import (
     int_or_none,
+    parse_qs,
     unified_timestamp,
 )
 
@@ -57,7 +57,7 @@ class BeegIE(InfoExtractor):
             query = {
                 'v': 2,
             }
-            qs = compat_urlparse.parse_qs(compat_urlparse.urlparse(url).query)
+            qs = parse_qs(url)
             t = qs.get('t', [''])[0].split('-')
             if len(t) > 1:
                 query.update({

yt_dlp/extractor/behindkink.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..utils import url_basename
@@ -24,7 +23,7 @@ class BehindKinkIE(InfoExtractor):
     }
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         display_id = mobj.group('id')
 
         webpage = self._download_webpage(url, display_id)

yt_dlp/extractor/bellmedia.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 
@@ -78,7 +77,7 @@ class BellMediaIE(InfoExtractor):
     }
 
     def _real_extract(self, url):
-        domain, video_id = re.match(self._VALID_URL, url).groups()
+        domain, video_id = self._match_valid_url(url).groups()
         domain = domain.split('.')[0]
         return {
             '_type': 'url_transparent',

yt_dlp/extractor/bilibili.py

@@ -4,13 +4,16 @@ from __future__ import unicode_literals
 import hashlib
 import itertools
 import json
+import functools
 import re
+import math
 
 from .common import InfoExtractor, SearchInfoExtractor
 from ..compat import (
     compat_str,
     compat_parse_qs,
     compat_urlparse,
+    compat_urllib_parse_urlparse
 )
 from ..utils import (
     ExtractorError,
@@ -24,6 +27,7 @@ from ..utils import (
     unified_timestamp,
     unsmuggle_url,
     urlencode_postdata,
+    OnDemandPagedList
 )
 
 
@@ -37,7 +41,7 @@ class BiliBiliIE(InfoExtractor):
                                 video/[aA][vV]|
                                 anime/(?P<anime_id>\d+)/play\#
                             )(?P<id>\d+)|
-                            video/[bB][vV](?P<id_bv>[^/?#&]+)
+                            (s/)?video/[bB][vV](?P<id_bv>[^/?#&]+)
                         )
                         (?:/?\?p=(?P<page>\d+))?
                     '''
@@ -140,7 +144,7 @@ class BiliBiliIE(InfoExtractor):
     def _real_extract(self, url):
         url, smuggled_data = unsmuggle_url(url, {})
 
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         video_id = mobj.group('id_bv') or mobj.group('id')
 
         av_id, bv_id = self._get_video_id_set(video_id, mobj.group('id_bv') is not None)
@@ -535,6 +539,75 @@ class BilibiliChannelIE(InfoExtractor):
         return self.playlist_result(self._entries(list_id), list_id)
 
 
+class BilibiliCategoryIE(InfoExtractor):
+    IE_NAME = 'Bilibili category extractor'
+    _MAX_RESULTS = 1000000
+    _VALID_URL = r'https?://www\.bilibili\.com/v/[a-zA-Z]+\/[a-zA-Z]+'
+    _TESTS = [{
+        'url': 'https://www.bilibili.com/v/kichiku/mad',
+        'info_dict': {
+            'id': 'kichiku: mad',
+            'title': 'kichiku: mad'
+        },
+        'playlist_mincount': 45,
+        'params': {
+            'playlistend': 45
+        }
+    }]
+
+    def _fetch_page(self, api_url, num_pages, query, page_num):
+        parsed_json = self._download_json(
+            api_url, query, query={'Search_key': query, 'pn': page_num},
+            note='Extracting results from page %s of %s' % (page_num, num_pages))
+
+        video_list = try_get(parsed_json, lambda x: x['data']['archives'], list)
+        if not video_list:
+            raise ExtractorError('Failed to retrieve video list for page %d' % page_num)
+
+        for video in video_list:
+            yield self.url_result(
+                'https://www.bilibili.com/video/%s' % video['bvid'], 'BiliBili', video['bvid'])
+
+    def _entries(self, category, subcategory, query):
+        # map of categories : subcategories : RIDs
+        rid_map = {
+            'kichiku': {
+                'mad': 26,
+                'manual_vocaloid': 126,
+                'guide': 22,
+                'theatre': 216,
+                'course': 127
+            },
+        }
+
+        if category not in rid_map:
+            raise ExtractorError('The supplied category, %s, is not supported. List of supported categories: %s' % (category, list(rid_map.keys())))
+
+        if subcategory not in rid_map[category]:
+            raise ExtractorError('The subcategory, %s, isn\'t supported for this category. Supported subcategories: %s' % (subcategory, list(rid_map[category].keys())))
+
+        rid_value = rid_map[category][subcategory]
+
+        api_url = 'https://api.bilibili.com/x/web-interface/newlist?rid=%d&type=1&ps=20&jsonp=jsonp' % rid_value
+        page_json = self._download_json(api_url, query, query={'Search_key': query, 'pn': '1'})
+        page_data = try_get(page_json, lambda x: x['data']['page'], dict)
+        count, size = int_or_none(page_data.get('count')), int_or_none(page_data.get('size'))
+        if count is None or not size:
+            raise ExtractorError('Failed to calculate either page count or size')
+
+        num_pages = math.ceil(count / size)
+
+        return OnDemandPagedList(functools.partial(
+            self._fetch_page, api_url, num_pages, query), size)
+
+    def _real_extract(self, url):
+        u = compat_urllib_parse_urlparse(url)
+        category, subcategory = u.path.split('/')[2:4]
+        query = '%s: %s' % (category, subcategory)
+
+        return self.playlist_result(self._entries(category, subcategory, query), query, query)
+
+
 class BiliBiliSearchIE(SearchInfoExtractor):
     IE_DESC = 'Bilibili video search, "bilisearch" keyword'
     _MAX_RESULTS = 100000

yt_dlp/extractor/bitchute.py

@@ -17,16 +17,16 @@ from ..utils import (
 class BitChuteIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?bitchute\.com/(?:video|embed|torrent/[^/]+)/(?P<id>[^/?#&]+)'
     _TESTS = [{
-        'url': 'https://www.bitchute.com/video/szoMrox2JEI/',
-        'md5': '66c4a70e6bfc40dcb6be3eb1d74939eb',
+        'url': 'https://www.bitchute.com/video/UGlrF9o9b-Q/',
+        'md5': '7e427d7ed7af5a75b5855705ec750e2b',
         'info_dict': {
             'id': 'szoMrox2JEI',
             'ext': 'mp4',
-            'title': 'Fuck bitches get money',
-            'description': 'md5:3f21f6fb5b1d17c3dee9cf6b5fe60b3a',
+            'title': 'This is the first video on #BitChute !',
+            'description': 'md5:a0337e7b1fe39e32336974af8173a034',
             'thumbnail': r're:^https?://.*\.jpg$',
-            'uploader': 'Victoria X Rave',
-            'upload_date': '20170813',
+            'uploader': 'BitChute',
+            'upload_date': '20170103',
         },
     }, {
         'url': 'https://www.bitchute.com/embed/lbb5G1hjPhw/',

yt_dlp/extractor/blackboardcollaborate.py

@@ -0,0 +1,67 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+
+from .common import InfoExtractor
+from ..utils import parse_iso8601
+
+
+class BlackboardCollaborateIE(InfoExtractor):
+    _VALID_URL = r'''(?x)
+                        https?://
+                        (?P<region>[a-z-]+)\.bbcollab\.com/
+                        (?:
+                            collab/ui/session/playback/load|
+                            recording
+                        )/
+                        (?P<id>[^/]+)'''
+    _TESTS = [
+        {
+            'url': 'https://us-lti.bbcollab.com/collab/ui/session/playback/load/0a633b6a88824deb8c918f470b22b256',
+            'md5': 'bb7a055682ee4f25fdb5838cdf014541',
+            'info_dict': {
+                'id': '0a633b6a88824deb8c918f470b22b256',
+                'title': 'HESI A2 Information Session - Thursday, May 6, 2021 - recording_1',
+                'ext': 'mp4',
+                'duration': 1896000,
+                'timestamp': 1620331399,
+                'upload_date': '20210506',
+            },
+        },
+        {
+            'url': 'https://us.bbcollab.com/collab/ui/session/playback/load/76761522adfe4345a0dee6794bbcabda',
+            'only_matching': True,
+        },
+        {
+            'url': 'https://ca.bbcollab.com/collab/ui/session/playback/load/b6399dcb44df4f21b29ebe581e22479d',
+            'only_matching': True,
+        },
+        {
+            'url': 'https://eu.bbcollab.com/recording/51ed7b50810c4444a106e48cefb3e6b5',
+            'only_matching': True,
+        },
+        {
+            'url': 'https://au.bbcollab.com/collab/ui/session/playback/load/2bccf7165d7c419ab87afc1ec3f3bb15',
+            'only_matching': True,
+        },
+    ]
+
+    def _real_extract(self, url):
+        mobj = self._match_valid_url(url)
+        region = mobj.group('region')
+        video_id = mobj.group('id')
+        info = self._download_json(
+            'https://{}.bbcollab.com/collab/api/csa/recordings/{}/data'.format(region, video_id), video_id)
+        duration = info.get('duration')
+        title = info['name']
+        upload_date = info.get('created')
+        streams = info['streams']
+        formats = [{'format_id': k, 'url': url} for k, url in streams.items()]
+
+        return {
+            'duration': duration,
+            'formats': formats,
+            'id': video_id,
+            'timestamp': parse_iso8601(upload_date),
+            'title': title,
+        }

yt_dlp/extractor/bokecc.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..compat import compat_parse_qs
@@ -45,7 +44,7 @@ class BokeCCIE(BokeCCBaseIE):
     }]
 
     def _real_extract(self, url):
-        qs = compat_parse_qs(re.match(self._VALID_URL, url).group('query'))
+        qs = compat_parse_qs(self._match_valid_url(url).group('query'))
         if not qs.get('vid') or not qs.get('uid'):
             raise ExtractorError('Invalid URL', expected=True)
 

yt_dlp/extractor/bongacams.py

@@ -1,6 +1,5 @@
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..compat import compat_str
@@ -22,7 +21,7 @@ class BongaCamsIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         host = mobj.group('host')
         channel_id = mobj.group('id')
 

yt_dlp/extractor/box.py

@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 
 import json
-import re
 
 from .common import InfoExtractor
 from ..utils import (
@@ -30,7 +29,7 @@ class BoxIE(InfoExtractor):
     }
 
     def _real_extract(self, url):
-        shared_name, file_id = re.match(self._VALID_URL, url).groups()
+        shared_name, file_id = self._match_valid_url(url).groups()
         webpage = self._download_webpage(url, file_id)
         request_token = self._parse_json(self._search_regex(
             r'Box\.config\s*=\s*({.+?});', webpage,

yt_dlp/extractor/br.py

@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 
 import json
-import re
 
 from .common import InfoExtractor
 from ..utils import (
@@ -86,7 +85,7 @@ class BRIE(InfoExtractor):
     ]
 
     def _real_extract(self, url):
-        base_url, display_id = re.search(self._VALID_URL, url).groups()
+        base_url, display_id = self._match_valid_url(url).groups()
         page = self._download_webpage(url, display_id)
         xml_url = self._search_regex(
             r"return BRavFramework\.register\(BRavFramework\('avPlayer_(?:[a-f0-9-]{36})'\)\.setup\({dataURL:'(/(?:[a-z0-9\-]+/)+[a-z0-9/~_.-]+)'}\)\);", page, 'XMLURL')

yt_dlp/extractor/bravotv.py

@@ -42,7 +42,7 @@ class BravoTVIE(AdobePassIE):
     }]
 
     def _real_extract(self, url):
-        site, display_id = re.match(self._VALID_URL, url).groups()
+        site, display_id = self._match_valid_url(url).groups()
         webpage = self._download_webpage(url, display_id)
         settings = self._parse_json(self._search_regex(
             r'<script[^>]+data-drupal-selector="drupal-settings-json"[^>]*>({.+?})</script>', webpage, 'drupal settings'),

yt_dlp/extractor/breakcom.py

@@ -1,6 +1,5 @@
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from .youtube import YoutubeIE
@@ -41,7 +40,7 @@ class BreakIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        display_id, video_id = re.match(self._VALID_URL, url).groups()
+        display_id, video_id = self._match_valid_url(url).groups()
 
         webpage = self._download_webpage(url, display_id)
 

yt_dlp/extractor/brightcove.py

@@ -11,7 +11,6 @@ from ..compat import (
     compat_etree_fromstring,
     compat_HTTPError,
     compat_parse_qs,
-    compat_urllib_parse_urlparse,
     compat_urlparse,
     compat_xml_parse_error,
 )
@@ -26,6 +25,7 @@ from ..utils import (
     js_to_json,
     mimetype2ext,
     parse_iso8601,
+    parse_qs,
     smuggle_url,
     str_or_none,
     try_get,
@@ -177,7 +177,7 @@ class BrightcoveLegacyIE(InfoExtractor):
             flashvars = {}
 
         data_url = object_doc.attrib.get('data', '')
-        data_url_params = compat_parse_qs(compat_urllib_parse_urlparse(data_url).query)
+        data_url_params = parse_qs(data_url)
 
         def find_param(name):
             if name in flashvars:
@@ -290,7 +290,7 @@ class BrightcoveLegacyIE(InfoExtractor):
         url = re.sub(r'(?<=[?&])(videoI(d|D)|idVideo|bctid)', '%40videoPlayer', url)
         # Change bckey (used by bcove.me urls) to playerKey
         url = re.sub(r'(?<=[?&])bckey', 'playerKey', url)
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         query_str = mobj.group('query')
         query = compat_urlparse.parse_qs(query_str)
 
@@ -549,7 +549,7 @@ class BrightcoveNewIE(AdobePassIE):
                     error.get('message') or error.get('error_subcode') or error['error_code'], expected=True)
             elif (not self.get_param('allow_unplayable_formats')
                     and sources and num_drm_sources == len(sources)):
-                raise ExtractorError('This video is DRM protected.', expected=True)
+                self.report_drm(video_id)
 
         self._sort_formats(formats)
 
@@ -595,7 +595,7 @@ class BrightcoveNewIE(AdobePassIE):
             'ip_blocks': smuggled_data.get('geo_ip_blocks'),
         })
 
-        account_id, player_id, embed, content_type, video_id = re.match(self._VALID_URL, url).groups()
+        account_id, player_id, embed, content_type, video_id = self._match_valid_url(url).groups()
 
         policy_key_id = '%s_%s' % (account_id, player_id)
         policy_key = self._downloader.cache.load('brightcove', policy_key_id)

yt_dlp/extractor/byutv.py

@@ -1,6 +1,5 @@
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..utils import (
@@ -52,7 +51,7 @@ class BYUtvIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         video_id = mobj.group('id')
         display_id = mobj.group('display_id') or video_id
 

yt_dlp/extractor/c56.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..utils import js_to_json
@@ -31,7 +30,7 @@ class C56IE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url, flags=re.VERBOSE)
+        mobj = self._match_valid_url(url)
         text_id = mobj.group('textid')
 
         webpage = self._download_webpage(url, text_id)

yt_dlp/extractor/camtube.py

@@ -1,71 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..utils import (
-    int_or_none,
-    unified_timestamp,
-)
-
-
-class CamTubeIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:(?:www|api)\.)?camtube\.co/recordings?/(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'https://camtube.co/recording/minafay-030618-1136-chaturbate-female',
-        'info_dict': {
-            'id': '42ad3956-dd5b-445a-8313-803ea6079fac',
-            'display_id': 'minafay-030618-1136-chaturbate-female',
-            'ext': 'mp4',
-            'title': 'minafay-030618-1136-chaturbate-female',
-            'duration': 1274,
-            'timestamp': 1528018608,
-            'upload_date': '20180603',
-            'age_limit': 18
-        },
-        'params': {
-            'skip_download': True,
-        },
-    }]
-
-    _API_BASE = 'https://api.camtube.co'
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-
-        token = self._download_json(
-            '%s/rpc/session/new' % self._API_BASE, display_id,
-            'Downloading session token')['token']
-
-        self._set_cookie('api.camtube.co', 'session', token)
-
-        video = self._download_json(
-            '%s/recordings/%s' % (self._API_BASE, display_id), display_id,
-            headers={'Referer': url})
-
-        video_id = video['uuid']
-        timestamp = unified_timestamp(video.get('createdAt'))
-        duration = int_or_none(video.get('duration'))
-        view_count = int_or_none(video.get('viewCount'))
-        like_count = int_or_none(video.get('likeCount'))
-        creator = video.get('stageName')
-
-        formats = [{
-            'url': '%s/recordings/%s/manifest.m3u8'
-                   % (self._API_BASE, video_id),
-            'format_id': 'hls',
-            'ext': 'mp4',
-            'protocol': 'm3u8_native',
-        }]
-
-        return {
-            'id': video_id,
-            'display_id': display_id,
-            'title': display_id,
-            'timestamp': timestamp,
-            'duration': duration,
-            'view_count': view_count,
-            'like_count': like_count,
-            'creator': creator,
-            'formats': formats,
-            'age_limit': 18
-        }

yt_dlp/extractor/canalplus.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..utils import (
@@ -50,7 +49,7 @@ class CanalplusIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        site, display_id, video_id = re.match(self._VALID_URL, url).groups()
+        site, display_id, video_id = self._match_valid_url(url).groups()
 
         site_id = self._SITE_ID_MAP[site]
 

yt_dlp/extractor/canvas.py

@@ -1,7 +1,5 @@
 from __future__ import unicode_literals
 
-import re
-import json
 
 from .common import InfoExtractor
 from .gigya import GigyaBaseIE
@@ -17,6 +15,7 @@ from ..utils import (
     str_or_none,
     strip_or_none,
     url_or_none,
+    urlencode_postdata
 )
 
 
@@ -47,7 +46,7 @@ class CanvasIE(InfoExtractor):
     _REST_API_BASE = 'https://media-services-public.vrt.be/vualto-video-aggregator-web/rest/external/v1'
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         site_id, video_id = mobj.group('site_id'), mobj.group('id')
 
         data = None
@@ -192,7 +191,7 @@ class CanvasEenIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         site_id, display_id = mobj.group('site_id'), mobj.group('id')
 
         webpage = self._download_webpage(url, display_id)
@@ -265,7 +264,7 @@ class VrtNUIE(GigyaBaseIE):
         'expected_warnings': ['Unable to download asset JSON', 'is not a supported codec', 'Unknown MIME type'],
     }]
     _NETRC_MACHINE = 'vrtnu'
-    _APIKEY = '3_0Z2HujMtiWq_pkAjgnS2Md2E11a1AwZjYiBETtwNE-EoEHDINgtnvcAOpNgmrVGy'
+    _APIKEY = '3_qhEcPa5JGFROVwu5SWKqJ4mVOIkwlFNMSKwzPDAh8QZOtHqu6L4nD5Q7lk0eXOOG'
     _CONTEXT_ID = 'R3595707040'
 
     def _real_initialize(self):
@@ -276,35 +275,38 @@ class VrtNUIE(GigyaBaseIE):
         if username is None:
             return
 
-        auth_data = {
-            'APIKey': self._APIKEY,
-            'targetEnv': 'jssdk',
-            'loginID': username,
-            'password': password,
-            'authMode': 'cookie',
-        }
-
-        auth_info = self._gigya_login(auth_data)
+        auth_info = self._download_json(
+            'https://accounts.vrt.be/accounts.login', None,
+            note='Login data', errnote='Could not get Login data',
+            headers={}, data=urlencode_postdata({
+                'loginID': username,
+                'password': password,
+                'sessionExpiration': '-2',
+                'APIKey': self._APIKEY,
+                'targetEnv': 'jssdk',
+            }))
 
         # Sometimes authentication fails for no good reason, retry
         login_attempt = 1
         while login_attempt <= 3:
             try:
-                # When requesting a token, no actual token is returned, but the
-                # necessary cookies are set.
+                self._request_webpage('https://token.vrt.be/vrtnuinitlogin',
+                                      None, note='Requesting XSRF Token', errnote='Could not get XSRF Token',
+                                      query={'provider': 'site', 'destination': 'https://www.vrt.be/vrtnu/'})
+
+                post_data = {
+                    'UID': auth_info['UID'],
+                    'UIDSignature': auth_info['UIDSignature'],
+                    'signatureTimestamp': auth_info['signatureTimestamp'],
+                    'client_id': 'vrtnu-site',
+                    '_csrf': self._get_cookies('https://login.vrt.be').get('OIDCXSRF').value,
+                }
+
                 self._request_webpage(
-                    'https://token.vrt.be',
+                    'https://login.vrt.be/perform_login',
                     None, note='Requesting a token', errnote='Could not get a token',
-                    headers={
-                        'Content-Type': 'application/json',
-                        'Referer': 'https://www.vrt.be/vrtnu/',
-                    },
-                    data=json.dumps({
-                        'uid': auth_info['UID'],
-                        'uidsig': auth_info['UIDSignature'],
-                        'ts': auth_info['signatureTimestamp'],
-                        'email': auth_info['profile']['email'],
-                    }).encode('utf-8'))
+                    headers={}, data=urlencode_postdata(post_data))
+
             except ExtractorError as e:
                 if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
                     login_attempt += 1

yt_dlp/extractor/cbs.py

@@ -8,6 +8,7 @@ from ..utils import (
     xpath_element,
     xpath_text,
     update_url_query,
+    url_or_none,
 )
 
 
@@ -25,16 +26,62 @@ class CBSBaseIE(ThePlatformFeedIE):
                     })
         return subtitles
 
+    def _extract_common_video_info(self, content_id, asset_types, mpx_acc, extra_info):
+        tp_path = 'dJ5BDC/media/guid/%d/%s' % (mpx_acc, content_id)
+        tp_release_url = f'https://link.theplatform.com/s/{tp_path}'
+        info = self._extract_theplatform_metadata(tp_path, content_id)
+
+        formats, subtitles = [], {}
+        last_e = None
+        for asset_type, query in asset_types.items():
+            try:
+                tp_formats, tp_subtitles = self._extract_theplatform_smil(
+                    update_url_query(tp_release_url, query), content_id,
+                    'Downloading %s SMIL data' % asset_type)
+            except ExtractorError as e:
+                last_e = e
+                if asset_type != 'fallback':
+                    continue
+                query['formats'] = ''  # blank query to check if expired
+                try:
+                    tp_formats, tp_subtitles = self._extract_theplatform_smil(
+                        update_url_query(tp_release_url, query), content_id,
+                        'Downloading %s SMIL data, trying again with another format' % asset_type)
+                except ExtractorError as e:
+                    last_e = e
+                    continue
+            formats.extend(tp_formats)
+            subtitles = self._merge_subtitles(subtitles, tp_subtitles)
+        if last_e and not formats:
+            self.raise_no_formats(last_e, True, content_id)
+        self._sort_formats(formats)
+
+        extra_info.update({
+            'id': content_id,
+            'formats': formats,
+            'subtitles': subtitles,
+        })
+        info.update({k: v for k, v in extra_info.items() if v is not None})
+        return info
+
+    def _extract_video_info(self, *args, **kwargs):
+        # Extract assets + metadata and call _extract_common_video_info
+        raise NotImplementedError('This method must be implemented by subclasses')
+
+    def _real_extract(self, url):
+        return self._extract_video_info(self._match_id(url))
+
 
 class CBSIE(CBSBaseIE):
     _VALID_URL = r'''(?x)
         (?:
             cbs:|
             https?://(?:www\.)?(?:
-                (?:cbs|paramountplus)\.com/(?:shows/[^/]+/video|movies/[^/]+)/|
+                cbs\.com/(?:shows/[^/]+/video|movies/[^/]+)/|
                 colbertlateshow\.com/(?:video|podcasts)/)
         )(?P<id>[\w-]+)'''
 
+    # All tests are blocked outside US
     _TESTS = [{
         'url': 'https://www.cbs.com/shows/garth-brooks/video/_u7W953k6la293J7EPTd9oHkSPs6Xn6_/connect-chat-feat-garth-brooks/',
         'info_dict': {
@@ -51,19 +98,28 @@ class CBSIE(CBSBaseIE):
             # m3u8 download
             'skip_download': True,
         },
-        '_skip': 'Blocked outside the US',
+    }, {
+        'url': 'https://www.cbs.com/shows/the-late-show-with-stephen-colbert/video/60icOhMb9NcjbcWnF_gub9XXHdeBcNk2/the-late-show-6-23-21-christine-baranski-joy-oladokun-',
+        'info_dict': {
+            'id': '60icOhMb9NcjbcWnF_gub9XXHdeBcNk2',
+            'title': 'The Late Show - 6/23/21 (Christine Baranski, Joy Oladokun)',
+            'timestamp': 1624507140,
+            'description': 'md5:e01af24e95c74d55e8775aef86117b95',
+            'uploader': 'CBSI-NEW',
+            'upload_date': '20210624',
+        },
+        'params': {
+            'ignore_no_formats_error': True,
+            'skip_download': True,
+        },
+        'expected_warnings': [
+            'This content expired on', 'No video formats found', 'Requested format is not available'],
     }, {
         'url': 'http://colbertlateshow.com/video/8GmB0oY0McANFvp2aEffk9jZZZ2YyXxy/the-colbeard/',
         'only_matching': True,
     }, {
         'url': 'http://www.colbertlateshow.com/podcasts/dYSwjqPs_X1tvbV_P2FcPWRa_qT6akTC/in-the-bad-room-with-stephen/',
         'only_matching': True,
-    }, {
-        'url': 'https://www.paramountplus.com/shows/all-rise/video/QmR1WhNkh1a_IrdHZrbcRklm176X_rVc/all-rise-space/',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.paramountplus.com/movies/million-dollar-american-princesses-meghan-and-harry/C0LpgNwXYeB8txxycdWdR9TjxpJOsdCq',
-        'only_matching': True,
     }]
 
     def _extract_video_info(self, content_id, site='cbs', mpx_acc=2198311517):
@@ -72,53 +128,34 @@ class CBSIE(CBSBaseIE):
             content_id, query={'partner': site, 'contentId': content_id})
         video_data = xpath_element(items_data, './/item')
         title = xpath_text(video_data, 'videoTitle', 'title') or xpath_text(video_data, 'videotitle', 'title')
-        tp_path = 'dJ5BDC/media/guid/%d/%s' % (mpx_acc, content_id)
-        tp_release_url = 'https://link.theplatform.com/s/' + tp_path
 
-        asset_types = []
-        subtitles = {}
-        formats = []
-        last_e = None
+        asset_types = {}
         for item in items_data.findall('.//item'):
             asset_type = xpath_text(item, 'assetType')
-            if not asset_type or asset_type in asset_types or 'HLS_FPS' in asset_type or 'DASH_CENC' in asset_type:
-                continue
-            asset_types.append(asset_type)
             query = {
                 'mbr': 'true',
                 'assetTypes': asset_type,
             }
-            if asset_type.startswith('HLS') or asset_type in ('OnceURL', 'StreamPack'):
+            if not asset_type:
+                # fallback for content_ids that videoPlayerService doesn't return anything for
+                asset_type = 'fallback'
+                query['formats'] = 'M3U+none,MPEG4,M3U+appleHlsEncryption,MP3'
+                del query['assetTypes']
+            if asset_type in asset_types:
+                continue
+            elif any(excluded in asset_type for excluded in ('HLS_FPS', 'DASH_CENC', 'OnceURL')):
+                continue
+            if asset_type.startswith('HLS') or 'StreamPack' in asset_type:
                 query['formats'] = 'MPEG4,M3U'
             elif asset_type in ('RTMP', 'WIFI', '3G'):
                 query['formats'] = 'MPEG4,FLV'
-            try:
-                tp_formats, tp_subtitles = self._extract_theplatform_smil(
-                    update_url_query(tp_release_url, query), content_id,
-                    'Downloading %s SMIL data' % asset_type)
-            except ExtractorError as e:
-                last_e = e
-                continue
-            formats.extend(tp_formats)
-            subtitles = self._merge_subtitles(subtitles, tp_subtitles)
-        if last_e and not formats:
-            raise last_e
-        self._sort_formats(formats)
+            asset_types[asset_type] = query
 
-        info = self._extract_theplatform_metadata(tp_path, content_id)
-        info.update({
-            'id': content_id,
+        return self._extract_common_video_info(content_id, asset_types, mpx_acc, extra_info={
             'title': title,
             'series': xpath_text(video_data, 'seriesTitle'),
             'season_number': int_or_none(xpath_text(video_data, 'seasonNumber')),
             'episode_number': int_or_none(xpath_text(video_data, 'episodeNumber')),
             'duration': int_or_none(xpath_text(video_data, 'videoLength'), 1000),
-            'thumbnail': xpath_text(video_data, 'previewImageURL'),
-            'formats': formats,
-            'subtitles': subtitles,
+            'thumbnail': url_or_none(xpath_text(video_data, 'previewImageURL')),
         })
-        return info
-
-    def _real_extract(self, url):
-        content_id = self._match_id(url)
-        return self._extract_video_info(content_id)

yt_dlp/extractor/cbsinteractive.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .cbs import CBSIE
 from ..utils import int_or_none
@@ -71,7 +70,7 @@ class CBSInteractiveIE(CBSIE):
     }
 
     def _real_extract(self, url):
-        site, display_id = re.match(self._VALID_URL, url).groups()
+        site, display_id = self._match_valid_url(url).groups()
         webpage = self._download_webpage(url, display_id)
 
         data_json = self._html_search_regex(

yt_dlp/extractor/cbssports.py

@@ -1,6 +1,5 @@
 from __future__ import unicode_literals
 
-import re
 
 # from .cbs import CBSBaseIE
 from .common import InfoExtractor
@@ -30,7 +29,7 @@ class CBSSportsEmbedIE(InfoExtractor):
     #     return self._extract_feed_info('dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id)
 
     def _real_extract(self, url):
-        uuid, pcid = re.match(self._VALID_URL, url).groups()
+        uuid, pcid = self._match_valid_url(url).groups()
         query = {'id': uuid} if uuid else {'pcid': pcid}
         video = self._download_json(
             'https://www.cbssports.com/api/content/video/',

yt_dlp/extractor/ccma.py

@@ -3,7 +3,6 @@ from __future__ import unicode_literals
 
 import calendar
 import datetime
-import re
 
 from .common import InfoExtractor
 from ..utils import (
@@ -61,7 +60,7 @@ class CCMAIE(InfoExtractor):
     }]
 
     def _real_extract(self, url):
-        media_type, media_id = re.match(self._VALID_URL, url).groups()
+        media_type, media_id = self._match_valid_url(url).groups()
 
         media = self._download_json(
             'http://dinamics.ccma.cat/pvideo/media.jsp', media_id, query={

yt_dlp/extractor/cda.py

@@ -3,6 +3,7 @@ from __future__ import unicode_literals
 
 import codecs
 import re
+import json
 
 from .common import InfoExtractor
 from ..compat import (
@@ -19,6 +20,7 @@ from ..utils import (
     parse_duration,
     random_birthday,
     urljoin,
+    try_get,
 )
 
 
@@ -38,6 +40,8 @@ class CDAIE(InfoExtractor):
             'average_rating': float,
             'duration': 39,
             'age_limit': 0,
+            'upload_date': '20160221',
+            'timestamp': 1456078244,
         }
     }, {
         'url': 'http://www.cda.pl/video/57413289',
@@ -143,7 +147,7 @@ class CDAIE(InfoExtractor):
             b = []
             for c in a:
                 f = compat_ord(c)
-                b.append(compat_chr(33 + (f + 14) % 94) if 33 <= f and 126 >= f else compat_chr(f))
+                b.append(compat_chr(33 + (f + 14) % 94) if 33 <= f <= 126 else compat_chr(f))
             a = ''.join(b)
             a = a.replace('.cda.mp4', '')
             for p in ('.2cda.pl', '.3cda.pl'):
@@ -173,18 +177,34 @@ class CDAIE(InfoExtractor):
                     video['file'] = video['file'].replace('adc.mp4', '.mp4')
             elif not video['file'].startswith('http'):
                 video['file'] = decrypt_file(video['file'])
-            f = {
+            video_quality = video.get('quality')
+            qualities = video.get('qualities', {})
+            video_quality = next((k for k, v in qualities.items() if v == video_quality), video_quality)
+            info_dict['formats'].append({
                 'url': video['file'],
-            }
-            m = re.search(
-                r'<a[^>]+data-quality="(?P<format_id>[^"]+)"[^>]+href="[^"]+"[^>]+class="[^"]*quality-btn-active[^"]*">(?P<height>[0-9]+)p',
-                page)
-            if m:
-                f.update({
-                    'format_id': m.group('format_id'),
-                    'height': int(m.group('height')),
-                })
-            info_dict['formats'].append(f)
+                'format_id': video_quality,
+                'height': int_or_none(video_quality[:-1]),
+            })
+            for quality, cda_quality in qualities.items():
+                if quality == video_quality:
+                    continue
+                data = {'jsonrpc': '2.0', 'method': 'videoGetLink', 'id': 2,
+                        'params': [video_id, cda_quality, video.get('ts'), video.get('hash2'), {}]}
+                data = json.dumps(data).encode('utf-8')
+                video_url = self._download_json(
+                    f'https://www.cda.pl/video/{video_id}', video_id, headers={
+                        'Content-Type': 'application/json',
+                        'X-Requested-With': 'XMLHttpRequest'
+                    }, data=data, note=f'Fetching {quality} url',
+                    errnote=f'Failed to fetch {quality} url', fatal=False)
+                if try_get(video_url, lambda x: x['result']['status']) == 'ok':
+                    video_url = try_get(video_url, lambda x: x['result']['resp'])
+                    info_dict['formats'].append({
+                        'url': video_url,
+                        'format_id': quality,
+                        'height': int_or_none(quality[:-1])
+                    })
+
             if not info_dict['duration']:
                 info_dict['duration'] = parse_duration(video.get('duration'))