[version] update

:ci skip all

.github/FUNDING.yml

@@ -0,0 +1,13 @@
+# These are supported funding model platforms
+
+github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
+patreon: # Replace with a single Patreon username
+open_collective: # Replace with a single Open Collective username
+ko_fi: # Replace with a single Ko-fi username
+tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
+community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
+liberapay: # Replace with a single Liberapay username
+issuehunt: # Replace with a single IssueHunt username
+otechie: # Replace with a single Otechie username
+
+custom: ['https://github.com/yt-dlp/yt-dlp/blob/master/Collaborators.md#collaborators']

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -1,70 +0,0 @@
----
-name: Broken site support
-about: Report broken or misfunctioning site
-title: "[Broken]"
-labels: Broken
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
-- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar issues including closed ones
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.06.09
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-
-```
-<!--
-Do not remove the above ```
--->
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/1_broken_site.yml

@@ -0,0 +1,63 @@
+name: Broken site support
+description: Report broken or misfunctioning site
+labels: [triage, extractor-bug]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a broken site
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.11.10**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your issue in an arbitrary form.
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version 2021.11.10 (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (2021.11.10)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -1,56 +0,0 @@
----
-name: Site support request
-about: Request support for a new site
-title: "[Site Request]"
-labels: Request
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that none of provided URLs violate any copyrights
-- [ ] I've searched the bugtracker for similar site support requests including closed ones
-
-
-## Example URLs
-
-<!--
-Provide all kinds of example URLs support for which should be included. Replace following example URLs by yours.
--->
-
-- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
-- Single video: https://youtu.be/BaW_jenozKc
-- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
-
-
-## Description
-
-<!--
-Provide any additional information.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/2_site_support_request.yml

@@ -0,0 +1,74 @@
+name: Site support request
+description: Request support for a new site
+labels: [triage, site-request]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a new site support request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.11.10**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and am willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: example-urls
+    attributes:
+      label: Example URLs
+      description: |
+        Provide all kinds of example URLs for which support should be added
+      value: |
+        - Single video: https://www.youtube.com/watch?v=BaW_jenozKc
+        - Single video: https://youtu.be/BaW_jenozKc
+        - Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
+    validations:
+      required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide any additional information
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output **using one of the example URLs provided above**.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version 2021.11.10 (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (2021.11.10)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -1,40 +0,0 @@
----
-name: Site feature request
-about: Request a new functionality for a site
-title: "[Site Request]"
-labels: Request
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
-- [ ] I've searched the bugtracker for similar site feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your site feature request in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/3_site_feature_request.yml

@@ -0,0 +1,49 @@
+name: Site feature request
+description: Request a new functionality for a site
+labels: [triage, site-enhancement]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a site feature request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.11.10**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: example-urls
+    attributes:
+      label: Example URLs
+      description: |
+        Example URLs that can be used to demonstrate the requested feature
+      value: |
+        https://www.youtube.com/watch?v=BaW_jenozKc
+    validations:
+      required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your site feature request in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -1,72 +0,0 @@
----
-name: Bug report
-about: Report a bug unrelated to any particular site or extractor
-title: ''
-labels: ''
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
-- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Read bugs section in FAQ: https://github.com/yt-dlp/yt-dlp
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar bug reports including closed ones
-- [ ] I've read bugs section in FAQ
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.06.09
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-
-```
-<!--
-Do not remove the above ```
--->
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/4_bug_report.yml

@@ -0,0 +1,57 @@
+name: Bug report
+description: Report a bug unrelated to any particular site or extractor
+labels: [triage,bug]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a bug unrelated to a specific site
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.11.10**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your issue in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**.
+        Add the `-Uv` flag to **your** command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version 2021.11.10 (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (2021.11.10)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -1,40 +0,0 @@
----
-name: Feature request
-about: Request a new functionality unrelated to any particular site or extractor
-title: "[Feature Request]"
-labels: Request
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.09. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.06.09**
-- [ ] I've searched the bugtracker for similar feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/5_feature_request.yml

@@ -0,0 +1,30 @@
+name: Feature request request
+description: Request a new functionality unrelated to any particular site or extractor
+labels: [triage, enhancement]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a feature request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.11.10**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your site feature request in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/6_question.md

@@ -1,40 +0,0 @@
----
-name: Ask question
-about: Ask youtube-dl related question
-title: "[Question]"
-labels: question
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- Look through the README (https://github.com/yt-dlp/yt-dlp) and FAQ (https://github.com/yt-dlp/yt-dlp) for similar questions
-- Search the bugtracker for similar questions: https://github.com/yt-dlp/yt-dlp
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm asking a question
-- [ ] I've looked through the README and FAQ for similar questions
-- [ ] I've searched the bugtracker for similar questions including closed ones
-
-
-## Question
-
-<!--
-Ask your question in an arbitrary form. Please make sure it's worded well enough to be understood, see https://github.com/yt-dlp/yt-dlp.
--->
-
-WRITE QUESTION HERE

.github/ISSUE_TEMPLATE/6_question.yml

@@ -0,0 +1,30 @@
+name: Ask question
+description: Ask yt-dlp related question
+labels: [question]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm asking a question and not reporting a bug/feature request
+          required: true
+        - label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions including closed ones
+          required: true
+  - type: textarea
+    id: question
+    attributes:
+      label: Question
+      description: |
+        Ask your question in an arbitrary form.
+        Please make sure it's worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information and as much context and examples as possible
+      placeholder: WRITE QUESTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,5 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Get help from the community on Discord
+    url: https://discord.gg/H5MNcFW63r
+    about: Join the yt-dlp Discord for community-powered support!

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.md

@@ -1,70 +0,0 @@
----
-name: Broken site support
-about: Report broken or misfunctioning site
-title: "[Broken]"
-labels: Broken
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
-- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar issues including closed ones
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version %(version)s
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-
-```
-<!--
-Do not remove the above ```
--->
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml

@@ -0,0 +1,63 @@
+name: Broken site support
+description: Report broken or misfunctioning site
+labels: [triage, extractor-bug]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a broken site
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your issue in an arbitrary form.
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version %(version)s (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (%(version)s)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md

@@ -1,56 +0,0 @@
----
-name: Site support request
-about: Request support for a new site
-title: "[Site Request]"
-labels: Request
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that none of provided URLs violate any copyrights
-- [ ] I've searched the bugtracker for similar site support requests including closed ones
-
-
-## Example URLs
-
-<!--
-Provide all kinds of example URLs support for which should be included. Replace following example URLs by yours.
--->
-
-- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
-- Single video: https://youtu.be/BaW_jenozKc
-- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
-
-
-## Description
-
-<!--
-Provide any additional information.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml

@@ -0,0 +1,74 @@
+name: Site support request
+description: Request support for a new site
+labels: [triage, site-request]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a new site support request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and am willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: example-urls
+    attributes:
+      label: Example URLs
+      description: |
+        Provide all kinds of example URLs for which support should be added
+      value: |
+        - Single video: https://www.youtube.com/watch?v=BaW_jenozKc
+        - Single video: https://youtu.be/BaW_jenozKc
+        - Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
+    validations:
+      required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide any additional information
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output **using one of the example URLs provided above**.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version %(version)s (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (%(version)s)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md

@@ -1,40 +0,0 @@
----
-name: Site feature request
-about: Request a new functionality for a site
-title: "[Site Request]"
-labels: Request
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've searched the bugtracker for similar site feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your site feature request in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml

@@ -0,0 +1,49 @@
+name: Site feature request
+description: Request a new functionality for a site
+labels: [triage, site-enhancement]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a site feature request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: example-urls
+    attributes:
+      label: Example URLs
+      description: |
+        Example URLs that can be used to demonstrate the requested feature
+      value: |
+        https://www.youtube.com/watch?v=BaW_jenozKc
+    validations:
+      required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your site feature request in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md

@@ -1,72 +0,0 @@
----
-name: Bug report
-about: Report a bug unrelated to any particular site or extractor
-title: ''
-labels: ''
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
-- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Read bugs section in FAQ: https://github.com/yt-dlp/yt-dlp
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar bug reports including closed ones
-- [ ] I've read bugs section in FAQ
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version %(version)s
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-
-```
-<!--
-Do not remove the above ```
--->
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml

@@ -0,0 +1,57 @@
+name: Bug report
+description: Report a bug unrelated to any particular site or extractor
+labels: [triage,bug]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a bug unrelated to a specific site
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your issue in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output of yt-dlp **that clearly demonstrates the problem**.
+        Add the `-Uv` flag to **your** command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version %(version)s (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (%(version)s)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/5_feature_request.md

@@ -1,40 +0,0 @@
----
-name: Feature request
-about: Request a new functionality unrelated to any particular site or extractor
-title: "[Feature Request]"
-labels: Request
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've searched the bugtracker for similar feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml

@@ -0,0 +1,30 @@
+name: Feature request request
+description: Request a new functionality unrelated to any particular site or extractor
+labels: [triage, enhancement]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a feature request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your site feature request in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/6_question.yml

@@ -0,0 +1,30 @@
+name: Ask question
+description: Ask yt-dlp related question
+labels: [question]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm asking a question and not reporting a bug/feature request
+          required: true
+        - label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions including closed ones
+          required: true
+  - type: textarea
+    id: question
+    attributes:
+      label: Question
+      description: |
+        Ask your question in an arbitrary form.
+        Please make sure it's worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information and as much context and examples as possible
+      placeholder: WRITE QUESTION HERE
+    validations:
+      required: true

.github/PULL_REQUEST_TEMPLATE.md

@@ -7,11 +7,11 @@
 ---
 
 ### Before submitting a *pull request* make sure you have:
-- [ ] At least skimmed through [adding new extractor tutorial](https://github.com/ytdl-org/youtube-dl#adding-support-for-a-new-site) and [youtube-dl coding conventions](https://github.com/ytdl-org/youtube-dl#youtube-dl-coding-conventions) sections
+- [ ] At least skimmed through [contributing guidelines](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) including [yt-dlp coding conventions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#yt-dlp-coding-conventions)
 - [ ] [Searched](https://github.com/yt-dlp/yt-dlp/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
 - [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)
 
-### In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
+### In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:
 - [ ] I am the original author of this code and I am willing to release it under [Unlicense](http://unlicense.org/)
 - [ ] I am not the original author of this code but it is in public domain or released under [Unlicense](http://unlicense.org/) (provide reliable evidence)
 

.github/workflows/build.yml

@@ -8,15 +8,18 @@ on:
 jobs:
   build_unix:
     runs-on: ubuntu-latest
-
     outputs:
       ytdlp_version: ${{ steps.bump_version.outputs.ytdlp_version }}
       upload_url: ${{ steps.create_release.outputs.upload_url }}
-      sha256_unix: ${{ steps.sha256_file.outputs.sha256_unix }}
+      sha256_bin: ${{ steps.sha256_bin.outputs.sha256_bin }}
-      sha512_unix: ${{ steps.sha512_file.outputs.sha512_unix }}
+      sha512_bin: ${{ steps.sha512_bin.outputs.sha512_bin }}
+      sha256_tar: ${{ steps.sha256_tar.outputs.sha256_tar }}
+      sha512_tar: ${{ steps.sha512_tar.outputs.sha512_tar }}
 
     steps:
     - uses: actions/checkout@v2
+      with:
+        fetch-depth: 0
     - name: Set up Python
       uses: actions/setup-python@v2
       with:
@@ -25,11 +28,83 @@ jobs:
       run: sudo apt-get -y install zip pandoc man
     - name: Bump version
       id: bump_version
-      run: python devscripts/update-version.py
+      run: |
+        python devscripts/update-version.py
+        make issuetemplates
     - name: Print version
       run: echo "${{ steps.bump_version.outputs.ytdlp_version }}"
+    - name: Update master
+      id: push_update
+      run: |
+        git config --global user.email "${{ github.event.pusher.email }}"
+        git config --global user.name "${{ github.event.pusher.name }}"
+        git add -u
+        git commit -m "[version] update" -m ":ci skip all"
+        git pull --rebase origin ${{ github.event.repository.master_branch }}
+        git push origin ${{ github.event.ref }}:${{ github.event.repository.master_branch }}
+        echo ::set-output name=head_sha::$(git rev-parse HEAD)
+    - name: Get Changelog
+      id: get_changelog
+      run: |
+        changelog=$(cat Changelog.md | grep -oPz '(?s)(?<=### ${{ steps.bump_version.outputs.ytdlp_version }}\n{2}).+?(?=\n{2,3}###)') || true
+        echo "changelog<<EOF" >> $GITHUB_ENV
+        echo "$changelog" >> $GITHUB_ENV
+        echo "EOF" >> $GITHUB_ENV
+
+    - name: Build lazy extractors
+      id: lazy_extractors
+      run: python devscripts/make_lazy_extractors.py
     - name: Run Make
       run: make all tar
+    - name: Get SHA2-256SUMS for yt-dlp
+      id: sha256_bin
+      run: echo "::set-output name=sha256_bin::$(sha256sum yt-dlp | awk '{print $1}')"
+    - name: Get SHA2-256SUMS for yt-dlp.tar.gz
+      id: sha256_tar
+      run: echo "::set-output name=sha256_tar::$(sha256sum yt-dlp.tar.gz | awk '{print $1}')"
+    - name: Get SHA2-512SUMS for yt-dlp
+      id: sha512_bin
+      run: echo "::set-output name=sha512_bin::$(sha512sum yt-dlp | awk '{print $1}')"
+    - name: Get SHA2-512SUMS for yt-dlp.tar.gz
+      id: sha512_tar
+      run: echo "::set-output name=sha512_tar::$(sha512sum yt-dlp.tar.gz | awk '{print $1}')"
+
+    - name: Install dependencies for pypi
+      env:
+        PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
+      if: "env.PYPI_TOKEN != ''"
+      run: |
+        python -m pip install --upgrade pip
+        pip install setuptools wheel twine
+    - name: Build and publish on pypi
+      env:
+        TWINE_USERNAME: __token__
+        TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
+      if: "env.TWINE_PASSWORD != ''"
+      run: |
+        rm -rf dist/*
+        python setup.py sdist bdist_wheel
+        twine upload dist/*
+
+    - name: Install SSH private key
+      env:
+        BREW_TOKEN: ${{ secrets.BREW_TOKEN }}
+      if: "env.BREW_TOKEN != ''"
+      uses: webfactory/ssh-agent@v0.5.3
+      with:
+          ssh-private-key: ${{ env.BREW_TOKEN }}
+    - name: Update Homebrew Formulae
+      env:
+        BREW_TOKEN: ${{ secrets.BREW_TOKEN }}
+      if: "env.BREW_TOKEN != ''"
+      run: |
+        git clone git@github.com:yt-dlp/homebrew-taps taps/
+        python3 devscripts/update-formulae.py taps/Formula/yt-dlp.rb "${{ steps.bump_version.outputs.ytdlp_version }}"
+        git -C taps/ config user.name github-actions
+        git -C taps/ config user.email github-actions@example.com
+        git -C taps/ commit -am 'yt-dlp: ${{ steps.bump_version.outputs.ytdlp_version }}'
+        git -C taps/ push
+
     - name: Create Release
       id: create_release
       uses: actions/create-release@v1
@@ -38,9 +113,14 @@ jobs:
       with:
         tag_name: ${{ steps.bump_version.outputs.ytdlp_version }}
         release_name: yt-dlp ${{ steps.bump_version.outputs.ytdlp_version }}
+        commitish: ${{ steps.push_update.outputs.head_sha }}
         body: |
-          Changelog:
+          #### [A description of the various files]((https://github.com/yt-dlp/yt-dlp#release-files)) are in the README
-          PLACEHOLDER
+
+          ---
+
+          ### Changelog:
+          ${{ env.changelog }}
         draft: false
         prerelease: false
     - name: Upload yt-dlp Unix binary
@@ -62,36 +142,81 @@ jobs:
         asset_path: ./yt-dlp.tar.gz
         asset_name: yt-dlp.tar.gz
         asset_content_type: application/gzip
-    - name: Get SHA2-256SUMS for yt-dlp
+
-      id: sha256_file
+  build_macos:
-      run: echo "::set-output name=sha256_unix::$(sha256sum yt-dlp | awk '{print $1}')"
+    runs-on: macos-11
-    - name: Get SHA2-512SUMS for yt-dlp
+    needs: build_unix
-      id: sha512_file
+    outputs:
-      run: echo "::set-output name=sha512_unix::$(sha512sum yt-dlp | awk '{print $1}')"
+      sha256_macos: ${{ steps.sha256_macos.outputs.sha256_macos }}
-    - name: Install dependencies for pypi
+      sha512_macos: ${{ steps.sha512_macos.outputs.sha512_macos }}
-      env:
+      sha256_macos_zip: ${{ steps.sha256_macos_zip.outputs.sha256_macos_zip }}
-        PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
+      sha512_macos_zip: ${{ steps.sha512_macos_zip.outputs.sha512_macos_zip }}
-      if: "env.PYPI_TOKEN != ''"
+
+    steps:
+    - uses: actions/checkout@v2
+    # In order to create a universal2 application, the version of python3 in /usr/bin has to be used
+    - name: Install Requirements
       run: |
-        python -m pip install --upgrade pip
+          brew install coreutils
-        pip install setuptools wheel twine
+          /usr/bin/python3 -m pip install -U --user pip Pyinstaller mutagen pycryptodomex websockets
-    - name: Build and publish on pypi
+    - name: Bump version
+      id: bump_version
+      run: /usr/bin/python3 devscripts/update-version.py
+    - name: Build lazy extractors
+      id: lazy_extractors
+      run: /usr/bin/python3 devscripts/make_lazy_extractors.py
+    - name: Run PyInstaller Script
+      run: /usr/bin/python3 pyinst.py --target-architecture universal2 --onefile
+    - name: Upload yt-dlp MacOS binary
+      id: upload-release-macos
+      uses: actions/upload-release-asset@v1
       env:
-        TWINE_USERNAME: __token__
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}
+      with:
-      if: "env.TWINE_PASSWORD != ''"
+        upload_url: ${{ needs.build_unix.outputs.upload_url }}
-      run: |
+        asset_path: ./dist/yt-dlp_macos
-        rm -rf dist/*
+        asset_name: yt-dlp_macos
-        python setup.py sdist bdist_wheel
+        asset_content_type: application/octet-stream
-        twine upload dist/*
+    - name: Get SHA2-256SUMS for yt-dlp_macos
+      id: sha256_macos
+      run: echo "::set-output name=sha256_macos::$(sha256sum dist/yt-dlp_macos | awk '{print $1}')"
+    - name: Get SHA2-512SUMS for yt-dlp_macos
+      id: sha512_macos
+      run: echo "::set-output name=sha512_macos::$(sha512sum dist/yt-dlp_macos | awk '{print $1}')"
+
+    - name: Run PyInstaller Script with --onedir
+      run: /usr/bin/python3 pyinst.py --target-architecture universal2 --onedir
+    - uses: papeloto/action-zip@v1
+      with:
+        files: ./dist/yt-dlp_macos
+        dest: ./dist/yt-dlp_macos.zip
+    - name: Upload yt-dlp MacOS onedir
+      id: upload-release-macos-zip
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ needs.build_unix.outputs.upload_url }}
+        asset_path: ./dist/yt-dlp_macos.zip
+        asset_name: yt-dlp_macos.zip
+        asset_content_type: application/zip
+    - name: Get SHA2-256SUMS for yt-dlp_macos.zip
+      id: sha256_macos_zip
+      run: echo "::set-output name=sha256_macos_zip::$(sha256sum dist/yt-dlp_macos.zip | awk '{print $1}')"
+    - name: Get SHA2-512SUMS for yt-dlp_macos
+      id: sha512_macos_zip
+      run: echo "::set-output name=sha512_macos_zip::$(sha512sum dist/yt-dlp_macos.zip | awk '{print $1}')"
 
   build_windows:
     runs-on: windows-latest
     needs: build_unix
-
     outputs:
-      sha256_windows: ${{ steps.sha256_file_win.outputs.sha256_windows }}
+      sha256_win: ${{ steps.sha256_win.outputs.sha256_win }}
-      sha512_windows: ${{ steps.sha512_file_win.outputs.sha512_windows }}
+      sha512_win: ${{ steps.sha512_win.outputs.sha512_win }}
+      sha256_py2exe: ${{ steps.sha256_py2exe.outputs.sha256_py2exe }}
+      sha512_py2exe: ${{ steps.sha512_py2exe.outputs.sha512_py2exe }}
+      sha256_win_zip: ${{ steps.sha256_win_zip.outputs.sha256_win_zip }}
+      sha512_win_zip: ${{ steps.sha512_win_zip.outputs.sha512_win_zip }}
 
     steps:
     - uses: actions/checkout@v2
@@ -100,17 +225,19 @@ jobs:
       uses: actions/setup-python@v2
       with:
           python-version: '3.8'
-    - name: Upgrade pip and enable wheel support
-      run: python -m pip install --upgrade pip setuptools wheel
     - name: Install Requirements
-      run: pip install pyinstaller mutagen pycryptodome websockets
+      # Custom pyinstaller built with https://github.com/yt-dlp/pyinstaller-builds
+      run: |
+          python -m pip install --upgrade pip setuptools wheel py2exe
+          pip install "https://yt-dlp.github.io/Pyinstaller-Builds/x86_64/pyinstaller-4.5.1-py3-none-any.whl" mutagen pycryptodomex websockets
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
-    - name: Print version
+    - name: Build lazy extractors
-      run: echo "${{ steps.bump_version.outputs.ytdlp_version }}"
+      id: lazy_extractors
+      run: python devscripts/make_lazy_extractors.py
     - name: Run PyInstaller Script
-      run: python pyinst.py 64
+      run: python pyinst.py
     - name: Upload yt-dlp.exe Windows binary
       id: upload-release-windows
       uses: actions/upload-release-asset@v1
@@ -122,19 +249,61 @@ jobs:
         asset_name: yt-dlp.exe
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for yt-dlp.exe
-      id: sha256_file_win
+      id: sha256_win
-      run: echo "::set-output name=sha256_windows::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())"
+      run: echo "::set-output name=sha256_win::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())"
     - name: Get SHA2-512SUMS for yt-dlp.exe
-      id: sha512_file_win
+      id: sha512_win
-      run: echo "::set-output name=sha512_windows::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA512).Hash.ToLower())"
+      run: echo "::set-output name=sha512_win::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA512).Hash.ToLower())"
+
+    - name: Run PyInstaller Script with --onedir
+      run: python pyinst.py --onedir
+    - uses: papeloto/action-zip@v1
+      with:
+        files: ./dist/yt-dlp
+        dest: ./dist/yt-dlp_win.zip
+    - name: Upload yt-dlp Windows onedir
+      id: upload-release-windows-zip
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ needs.build_unix.outputs.upload_url }}
+        asset_path: ./dist/yt-dlp_win.zip
+        asset_name: yt-dlp_win.zip
+        asset_content_type: application/zip
+    - name: Get SHA2-256SUMS for yt-dlp_win.zip
+      id: sha256_win_zip
+      run: echo "::set-output name=sha256_win_zip::$((Get-FileHash dist\yt-dlp_win.zip -Algorithm SHA256).Hash.ToLower())"
+    - name: Get SHA2-512SUMS for yt-dlp_win.zip
+      id: sha512_win_zip
+      run: echo "::set-output name=sha512_win_zip::$((Get-FileHash dist\yt-dlp_win.zip -Algorithm SHA512).Hash.ToLower())"
+
+    - name: Run py2exe Script
+      run: python setup.py py2exe
+    - name: Upload yt-dlp_min.exe Windows binary
+      id: upload-release-windows-py2exe
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ needs.build_unix.outputs.upload_url }}
+        asset_path: ./dist/yt-dlp.exe
+        asset_name: yt-dlp_min.exe
+        asset_content_type: application/vnd.microsoft.portable-executable
+    - name: Get SHA2-256SUMS for yt-dlp_min.exe
+      id: sha256_py2exe
+      run: echo "::set-output name=sha256_py2exe::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())"
+    - name: Get SHA2-512SUMS for yt-dlp_min.exe
+      id: sha512_py2exe
+      run: echo "::set-output name=sha512_py2exe::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA512).Hash.ToLower())"
 
   build_windows32:
     runs-on: windows-latest
-    needs: [build_unix, build_windows]
+    needs: build_unix
 
     outputs:
-      sha256_windows32: ${{ steps.sha256_file_win32.outputs.sha256_windows32 }}
+      sha256_win32: ${{ steps.sha256_win32.outputs.sha256_win32 }}
-      sha512_windows32: ${{ steps.sha512_file_win32.outputs.sha512_windows32 }}
+      sha512_win32: ${{ steps.sha512_win32.outputs.sha512_win32 }}
 
     steps:
     - uses: actions/checkout@v2
@@ -144,17 +313,18 @@ jobs:
       with:
           python-version: '3.7'
           architecture: 'x86'
-    - name: Upgrade pip and enable wheel support
-      run: python -m pip install --upgrade pip setuptools wheel
     - name: Install Requirements
-      run: pip install pyinstaller mutagen pycryptodome websockets
+      run: |
+          python -m pip install --upgrade pip setuptools wheel
+          pip install "https://yt-dlp.github.io/Pyinstaller-Builds/i686/pyinstaller-4.5.1-py3-none-any.whl" mutagen pycryptodomex websockets
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
-    - name: Print version
+    - name: Build lazy extractors
-      run: echo "${{ steps.bump_version.outputs.ytdlp_version }}"
+      id: lazy_extractors
+      run: python devscripts/make_lazy_extractors.py
     - name: Run PyInstaller Script for 32 Bit
-      run: python pyinst.py 32
+      run: python pyinst.py
     - name: Upload Executable yt-dlp_x86.exe
       id: upload-release-windows32
       uses: actions/upload-release-asset@v1
@@ -166,28 +336,36 @@ jobs:
         asset_name: yt-dlp_x86.exe
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for yt-dlp_x86.exe
-      id: sha256_file_win32
+      id: sha256_win32
-      run: echo "::set-output name=sha256_windows32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA256).Hash.ToLower())"
+      run: echo "::set-output name=sha256_win32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA256).Hash.ToLower())"
     - name: Get SHA2-512SUMS for yt-dlp_x86.exe
-      id: sha512_file_win32
+      id: sha512_win32
-      run: echo "::set-output name=sha512_windows32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA512).Hash.ToLower())"
+      run: echo "::set-output name=sha512_win32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA512).Hash.ToLower())"
 
   finish:
     runs-on: ubuntu-latest
-    needs: [build_unix, build_windows, build_windows32]
+    needs: [build_unix, build_windows, build_windows32, build_macos]
 
     steps:
     - name: Make SHA2-256SUMS file
       env:
-        SHA256_WINDOWS: ${{ needs.build_windows.outputs.sha256_windows }}
+        SHA256_BIN: ${{ needs.build_unix.outputs.sha256_bin }}
-        SHA256_WINDOWS32: ${{ needs.build_windows32.outputs.sha256_windows32 }}
+        SHA256_TAR: ${{ needs.build_unix.outputs.sha256_tar }}
-        SHA256_UNIX: ${{ needs.build_unix.outputs.sha256_unix }}
+        SHA256_WIN: ${{ needs.build_windows.outputs.sha256_win }}
-        YTDLP_VERSION: ${{ needs.build_unix.outputs.ytdlp_version }}
+        SHA256_PY2EXE: ${{ needs.build_windows.outputs.sha256_py2exe }}
+        SHA256_WIN_ZIP: ${{ needs.build_windows.outputs.sha256_win_zip }}
+        SHA256_WIN32: ${{ needs.build_windows32.outputs.sha256_win32 }}
+        SHA256_MACOS: ${{ needs.build_macos.outputs.sha256_macos }}
+        SHA256_MACOS_ZIP: ${{ needs.build_macos.outputs.sha256_macos_zip }}
       run: |
-        echo "version:${{ env.YTDLP_VERSION }}" >> SHA2-256SUMS
+        echo "${{ env.SHA256_BIN }}  yt-dlp" >> SHA2-256SUMS
-        echo "yt-dlp.exe:${{ env.SHA256_WINDOWS }}" >> SHA2-256SUMS
+        echo "${{ env.SHA256_TAR }}  yt-dlp.tar.gz" >> SHA2-256SUMS
-        echo "yt-dlp_x86.exe:${{ env.SHA256_WINDOWS32 }}" >> SHA2-256SUMS
+        echo "${{ env.SHA256_WIN }}  yt-dlp.exe" >> SHA2-256SUMS
-        echo "yt-dlp:${{ env.SHA256_UNIX }}" >> SHA2-256SUMS
+        echo "${{ env.SHA256_PY2EXE }}  yt-dlp_min.exe" >> SHA2-256SUMS
+        echo "${{ env.SHA256_WIN32 }}  yt-dlp_x86.exe" >> SHA2-256SUMS
+        echo "${{ env.SHA256_WIN_ZIP }}  yt-dlp_win.zip" >> SHA2-256SUMS
+        echo "${{ env.SHA256_MACOS }}  yt-dlp_macos" >> SHA2-256SUMS
+        echo "${{ env.SHA256_MACOS_ZIP }}  yt-dlp_macos.zip" >> SHA2-256SUMS
     - name: Upload 256SUMS file
       id: upload-sums
       uses: actions/upload-release-asset@v1
@@ -200,13 +378,23 @@ jobs:
         asset_content_type: text/plain
     - name: Make SHA2-512SUMS file
       env:
-        SHA512_WINDOWS: ${{ needs.build_windows.outputs.sha512_windows }}
+        SHA512_BIN: ${{ needs.build_unix.outputs.sha512_bin }}
-        SHA512_WINDOWS32: ${{ needs.build_windows32.outputs.sha512_windows32 }}
+        SHA512_TAR: ${{ needs.build_unix.outputs.sha512_tar }}
-        SHA512_UNIX: ${{ needs.build_unix.outputs.sha512_unix }}
+        SHA512_WIN: ${{ needs.build_windows.outputs.sha512_win }}
+        SHA512_PY2EXE: ${{ needs.build_windows.outputs.sha512_py2exe }}
+        SHA512_WIN_ZIP: ${{ needs.build_windows.outputs.sha512_win_zip }}
+        SHA512_WIN32: ${{ needs.build_windows32.outputs.sha512_win32 }}
+        SHA512_MACOS: ${{ needs.build_macos.outputs.sha512_macos }}
+        SHA512_MACOS_ZIP: ${{ needs.build_macos.outputs.sha512_macos_zip }}
       run: |
-        echo "${{ env.SHA512_WINDOWS }}  yt-dlp.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_BIN }}  yt-dlp" >> SHA2-512SUMS
-        echo "${{ env.SHA512_WINDOWS32 }}  yt-dlp_x86.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_TAR }}  yt-dlp.tar.gz" >> SHA2-512SUMS
-        echo "${{ env.SHA512_UNIX }}  yt-dlp" >> SHA2-512SUMS
+        echo "${{ env.SHA512_WIN }}  yt-dlp.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_WIN_ZIP }}  yt-dlp_win.zip" >> SHA2-512SUMS
+        echo "${{ env.SHA512_PY2EXE }}  yt-dlp_min.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_WIN32 }}  yt-dlp_x86.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_MACOS }}  yt-dlp_macos" >> SHA2-512SUMS
+        echo "${{ env.SHA512_MACOS_ZIP }}  yt-dlp_macos.zip" >> SHA2-512SUMS
     - name: Upload 512SUMS file
       id: upload-512sums
       uses: actions/upload-release-asset@v1

.github/workflows/core.yml

@@ -10,7 +10,7 @@ jobs:
       matrix:
         os: [ubuntu-18.04]
         # py3.9 is in quick-test
-        python-version: [3.7, 3.8, pypy-3.6, pypy-3.7]
+        python-version: [3.7, 3.8, 3.10-dev, pypy-3.6, pypy-3.7]
         run-tests-ext: [sh]
         include:
         # atleast one of the tests must be in windows
@@ -23,11 +23,9 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install nose
+    - name: Install pytest
-      run: pip install nose
+      run: pip install pytest
     - name: Run tests
       continue-on-error: False
-      env:
+      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }} core
-        YTDL_TEST_SET: core
-      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}
   # Linter is in quick-test

.github/workflows/download.yml

@@ -9,7 +9,7 @@ jobs:
       fail-fast: true
       matrix:
         os: [ubuntu-18.04]
-        python-version: [3.7, 3.8, 3.9, pypy-3.6, pypy-3.7]
+        python-version: [3.7, 3.8, 3.9, 3.10-dev, pypy-3.6, pypy-3.7]
         run-tests-ext: [sh]
         include:
         - os: windows-latest
@@ -21,10 +21,8 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install nose
+    - name: Install pytest
-      run: pip install nose
+      run: pip install pytest
     - name: Run tests
       continue-on-error: true
-      env:
+      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }} download
-        YTDL_TEST_SET: download
-      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}

.github/workflows/quick-test.yml

@@ -11,12 +11,10 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: 3.9
-    - name: Install nose
+    - name: Install test requirements
-      run: pip install nose
+      run: pip install pytest pycryptodomex
     - name: Run tests
-      env:
+      run: ./devscripts/run_tests.sh core
-        YTDL_TEST_SET: core
-      run: ./devscripts/run_tests.sh
   flake8:
     name: Linter
     if: "!contains(github.event.head_commit.message, 'ci skip all')"
@@ -29,5 +27,7 @@ jobs:
         python-version: 3.9
     - name: Install flake8
       run: pip install flake8
+    - name: Make lazy extractors
+      run: python devscripts/make_lazy_extractors.py
     - name: Run flake8
-      run: flake8 .
+      run: flake8 .

.gitignore

@@ -2,7 +2,8 @@
 *.conf
 *.spec
 cookies
-cookies.txt
+*cookies.txt
+.netrc
 
 # Downloaded
 *.srt
@@ -19,6 +20,8 @@ cookies.txt
 *.wav
 *.ape
 *.mkv
+*.flac
+*.avi
 *.swf
 *.part
 *.part-*
@@ -33,17 +36,20 @@ cookies.txt
 *.info.json
 *.live_chat.json
 *.jpg
+*.jpeg
 *.png
 *.webp
 *.annotations.xml
 *.description
+.cache/
 
 # Allow config/media files in testdata
-!test/testdata/**
+!test/**
 
 # Python
 *.pyc
 *.pyo
+.pytest_cache
 wine-py2exe/
 py2exe.log
 build/
@@ -78,6 +84,7 @@ README.txt
 *.tar.gz
 *.zsh
 *.spec
+test/testdata/player-*.js
 
 # Binary
 /youtube-dl

CONTRIBUTING.md

@@ -1,26 +1,59 @@
-**Please include the full output of youtube-dl when run with `-v`**, i.e. **add** `-v` flag to **your command line**, copy the **whole** output and post it in the issue body wrapped in \`\`\` for better formatting. It should look similar to this:
+# CONTRIBUTING TO YT-DLP
+
+- [OPENING AN ISSUE](#opening-an-issue)
+    - [Is the description of the issue itself sufficient?](#is-the-description-of-the-issue-itself-sufficient)
+    - [Are you using the latest version?](#are-you-using-the-latest-version)
+    - [Is the issue already documented?](#is-the-issue-already-documented)
+    - [Why are existing options not enough?](#why-are-existing-options-not-enough)
+    - [Have you read and understood the changes, between youtube-dl and yt-dlp](#have-you-read-and-understood-the-changes-between-youtube-dl-and-yt-dlp)
+    - [Is there enough context in your bug report?](#is-there-enough-context-in-your-bug-report)
+    - [Does the issue involve one problem, and one problem only?](#does-the-issue-involve-one-problem-and-one-problem-only)
+    - [Is anyone going to need the feature?](#is-anyone-going-to-need-the-feature)
+    - [Is your question about yt-dlp?](#is-your-question-about-yt-dlp)
+- [DEVELOPER INSTRUCTIONS](#developer-instructions)
+    - [Adding new feature or making overarching changes](#adding-new-feature-or-making-overarching-changes)
+    - [Adding support for a new site](#adding-support-for-a-new-site)
+    - [yt-dlp coding conventions](#yt-dlp-coding-conventions)
+        - [Mandatory and optional metafields](#mandatory-and-optional-metafields)
+        - [Provide fallbacks](#provide-fallbacks)
+        - [Regular expressions](#regular-expressions)
+        - [Long lines policy](#long-lines-policy)
+        - [Inline values](#inline-values)
+        - [Collapse fallbacks](#collapse-fallbacks)
+        - [Trailing parentheses](#trailing-parentheses)
+        - [Use convenience conversion and parsing functions](#use-convenience-conversion-and-parsing-functions)
+- [EMBEDDING YT-DLP](README.md#embedding-yt-dlp)
+
+
+
+# OPENING AN ISSUE
+
+Bugs and suggestions should be reported at: [yt-dlp/yt-dlp/issues](https://github.com/yt-dlp/yt-dlp/issues). Unless you were prompted to or there is another pertinent reason (e.g. GitHub fails to accept the bug report), please do not send bug reports via personal email. For discussions, join us in our [discord server](https://discord.gg/H5MNcFW63r).
+
+**Please include the full output of yt-dlp when run with `-Uv`**, i.e. **add** `-Uv` flag to **your command line**, copy the **whole** output and post it in the issue body wrapped in \`\`\` for better formatting. It should look similar to this:
 ```
-$ youtube-dl -v <your command line>
+$ yt-dlp -Uv <your command line>
-[debug] System config: []
+[debug] Command-line config: ['-v', 'demo.com']
-[debug] User config: []
+[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
-[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=BaW_jenozKcj']
+[debug] yt-dlp version 2021.09.25 (zip)
-[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
+[debug] Python version 3.8.10 (CPython 64bit) - Linux-5.4.0-74-generic-x86_64-with-glibc2.29
-[debug] youtube-dl version 2015.12.06
+[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4
-[debug] Git HEAD: 135392e
-[debug] Python version 2.6.6 - Windows-2003Server-5.2.3790-SP2
-[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
+Current Build Hash 25cc412d1d3c0725a1f2f5b7e4682f6fb40e6d15f7024e96f7afd572e9919535
+yt-dlp is up to date (2021.09.25)
 ...
 ```
 **Do not post screenshots of verbose logs; only plain text is acceptable.**
 
-The output (including the first lines) contains important debugging information. Issues without the full output are often not reproducible and therefore do not get solved in short order, if ever.
+The output (including the first lines) contains important debugging information. Issues without the full output are often not reproducible and therefore will be closed as `incomplete`.
+
+The templates provided for the Issues, should be completed and **not removed**, this helps aide the resolution of the issue.
 
 Please re-read your issue once again to avoid a couple of common mistakes (you can and should use this as a checklist):
 
 ### Is the description of the issue itself sufficient?
 
-We often get issue reports that we cannot really decipher. While in most cases we eventually get the required information after asking back multiple times, this poses an unnecessary drain on our resources. Many contributors, including myself, are also not native speakers, so we may misread some parts.
+We often get issue reports that we cannot really decipher. While in most cases we eventually get the required information after asking back multiple times, this poses an unnecessary drain on our resources.
 
 So please elaborate on what feature you are requesting, or what bug you want to be fixed. Make sure that it's obvious
 
@@ -28,25 +61,31 @@ So please elaborate on what feature you are requesting, or what bug you want to
 - How it could be fixed
 - How your proposed solution would look like
 
-If your report is shorter than two lines, it is almost certainly missing some of these, which makes it hard for us to respond to it. We're often too polite to close the issue outright, but the missing info makes misinterpretation likely. As a committer myself, I often get frustrated by these issues, since the only possible way for me to move forward on them is to ask for clarification over and over.
+If your report is shorter than two lines, it is almost certainly missing some of these, which makes it hard for us to respond to it. We're often too polite to close the issue outright, but the missing info makes misinterpretation likely. We often get frustrated by these issues, since the only possible way for us to move forward on them is to ask for clarification over and over.
 
-For bug reports, this means that your report should contain the *complete* output of youtube-dl when called with the `-v` flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information.
+For bug reports, this means that your report should contain the **complete** output of yt-dlp when called with the `-Uv` flag. The error message you get for (most) bugs even says so, but you would not believe how many of our bug reports do not contain this information.
 
-If your server has multiple IPs or you suspect censorship, adding `--call-home` may be a good idea to get more diagnostics. If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--dump-pages` (warning: this will yield a rather large output, redirect it to the file `log.txt` by adding `>log.txt 2>&1` to your command-line) or upload the `.dump` files you get when you add `--write-pages` [somewhere](https://gist.github.com/).
+If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--write-pages` and upload the `.dump` files you get [somewhere](https://gist.github.com).
 
 **Site support requests must contain an example URL**. An example URL is a URL you might want to download, like `https://www.youtube.com/watch?v=BaW_jenozKc`. There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. `https://www.youtube.com/`) is *not* an example URL.
 
 ###  Are you using the latest version?
 
-Before reporting any issue, type `youtube-dl -U`. This should report that you're up-to-date. About 20% of the reports we receive are already fixed, but people are using outdated versions. This goes for feature requests as well.
+Before reporting any issue, type `yt-dlp -U`. This should report that you're up-to-date. This goes for feature requests as well.
 
 ###  Is the issue already documented?
 
-Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/ytdl-org/youtube-dl/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
+Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/yt-dlp/yt-dlp/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2021.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
+
+Additionally, it is also helpful to see if the issue has already been documented in the [youtube-dl issue tracker](https://github.com/ytdl-org/youtube-dl/issues). If similar issues have already been reported in youtube-dl (but not in our issue tracker), links to them can be included in your issue report here.
 
 ###  Why are existing options not enough?
 
-Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
+Before requesting a new feature, please have a quick peek at [the list of supported options](README.md#usage-and-options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
+
+###  Have you read and understood the changes, between youtube-dl and yt-dlp
+
+There are many changes between youtube-dl and yt-dlp [(changes to default behavior)](README.md#differences-in-default-behavior), and some of the options available have a different behaviour in yt-dlp, or have been removed all together [(list of changes to options)](README.md#deprecated-options). Make sure you have read and understand the differences in the options and how this may impact your downloads before opening an issue.
 
 ###  Is there enough context in your bug report?
 
@@ -58,68 +97,86 @@ We are then presented with a very complicated request when the original problem
 
 Some of our users seem to think there is a limit of issues they can or should open. There is no limit of issues they can or should open. While it may seem appealing to be able to dump all your issues into one ticket, that means that someone who solves one of your issues cannot mark the issue as closed. Typically, reporting a bunch of issues leads to the ticket lingering since nobody wants to attack that behemoth, until someone mercifully splits the issue into multiple ones.
 
-In particular, every site support request issue should only pertain to services at one site (generally under a common domain, but always using the same backend technology). Do not request support for vimeo user videos, White house podcasts, and Google Plus pages in the same issue. Also, make sure that you don't post bug reports alongside feature requests. As a rule of thumb, a feature request does not include outputs of youtube-dl that are not immediately related to the feature at hand. Do not post reports of a network error alongside the request for a new video service.
+In particular, every site support request issue should only pertain to services at one site (generally under a common domain, but always using the same backend technology). Do not request support for vimeo user videos, White house podcasts, and Google Plus pages in the same issue. Also, make sure that you don't post bug reports alongside feature requests. As a rule of thumb, a feature request does not include outputs of yt-dlp that are not immediately related to the feature at hand. Do not post reports of a network error alongside the request for a new video service.
 
 ###  Is anyone going to need the feature?
 
 Only post features that you (or an incapacitated friend you can personally talk to) require. Do not post features because they seem like a good idea. If they are really useful, they will be requested by someone who requires them.
 
-###  Is your question about youtube-dl?
+###  Is your question about yt-dlp?
+
+Some bug reports are completely unrelated to yt-dlp and relate to a different, or even the reporter's own, application. Please make sure that you are actually using yt-dlp. If you are using a UI for yt-dlp, report the bug to the maintainer of the actual application providing the UI. In general, if you are unable to provide the verbose log, you should not be opening the issue here.
+
+If the issue is with `youtube-dl` (the upstream fork of yt-dlp) and not with yt-dlp, the issue should be raised in the youtube-dl project.
+
+### Are you willing to share account details if needed?
+
+The maintainers and potential contributors of the project often do not have an account for the website you are asking support for. So any developer interested in solving your issue may ask you for account details. It is your personal discression whether you are willing to share the account in order for the developer to try and solve your issue. However, if you are unwilling or unable to provide details, they obviously cannot work on the issue and it cannot be solved unless some developer who both has an account and is willing/able to contribute decides to solve it.
+
+By sharing an account with anyone, you agree to bear all risks associated with it. The maintainers and yt-dlp can't be held responsible for any misuse of the credentials.
+
+While these steps won't necessarily ensure that no misuse of the account takes place, these are still some good practices to follow.
+
+- Look for people with `Member` (maintainers of the project) or `Contributor` (people who have previously contributed code) tag on their messages.
+- Change the password before sharing the account to something random (use [this](https://passwordsgenerator.net/) if you don't have a random password generator).
+- Change the password after receiving the account back.
+
+
 
-It may sound strange, but some bug reports we receive are completely unrelated to youtube-dl and relate to a different, or even the reporter's own, application. Please make sure that you are actually using youtube-dl. If you are using a UI for youtube-dl, report the bug to the maintainer of the actual application providing the UI. On the other hand, if your UI for youtube-dl fails in some way you believe is related to youtube-dl, by all means, go ahead and report the bug.
 
 # DEVELOPER INSTRUCTIONS
 
-Most users do not need to build youtube-dl and can [download the builds](https://ytdl-org.github.io/youtube-dl/download.html) or get them from their distribution.
+Most users do not need to build yt-dlp and can [download the builds](https://github.com/yt-dlp/yt-dlp/releases) or get them via [the other installation methods](README.md#installation).
 
-To run youtube-dl as a developer, you don't need to build anything either. Simply execute
+To run yt-dlp as a developer, you don't need to build anything either. Simply execute
 
-    python -m youtube_dl
+    python -m yt_dlp
 
 To run the test, simply invoke your favorite test runner, or execute a test file directly; any of the following work:
 
     python -m unittest discover
     python test/test_download.py
     nosetests
+    pytest
 
 See item 6 of [new extractor tutorial](#adding-support-for-a-new-site) for how to run extractor specific test cases.
 
-If you want to create a build of youtube-dl yourself, you'll need
+If you want to create a build of yt-dlp yourself, you can follow the instructions [here](README.md#compile).
 
-* python
-* make (only GNU make is supported)
-* pandoc
-* zip
-* nosetests
 
-### Adding support for a new site
+## Adding new feature or making overarching changes
 
-If you want to add support for a new site, first of all **make sure** this site is **not dedicated to [copyright infringement](README.md#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. youtube-dl does **not support** such sites thus pull requests adding support for them **will be rejected**.
+Before you start writing code for implementing a new feature, open an issue explaining your feature request and atleast one use case. This allows the maintainers to decide whether such a feature is desired for the project in the first place, and will provide an avenue to discuss some implementation details. If you open a pull request for a new feature without discussing with us first, do not be surprised when we ask for large changes to the code, or even reject it outright.
+
+The same applies for changes to the documentation, code style, or overarching changes to the architecture
+
+
+## Adding support for a new site
+
+If you want to add support for a new site, first of all **make sure** this site is **not dedicated to [copyright infringement](https://www.github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. yt-dlp does **not support** such sites thus pull requests adding support for them **will be rejected**.
 
 After you have ensured this site is distributing its content legally, you can follow this quick list (assuming your service is called `yourextractor`):
 
-1. [Fork this repository](https://github.com/ytdl-org/youtube-dl/fork)
+1. [Fork this repository](https://github.com/yt-dlp/yt-dlp/fork)
-2. Check out the source code with:
+1. Check out the source code with:
 
-        git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git
+        git clone git@github.com:YOUR_GITHUB_USERNAME/yt-dlp.git
 
-3. Start a new git branch with
+1. Start a new git branch with
 
-        cd youtube-dl
+        cd yt-dlp
         git checkout -b yourextractor
 
-4. Start with this simple template and save it to `youtube_dl/extractor/yourextractor.py`:
+1. Start with this simple template and save it to `yt_dlp/extractor/yourextractor.py`:
 
     ```python
     # coding: utf-8
-    from __future__ import unicode_literals
-
     from .common import InfoExtractor
-
+    
-
+    
     class YourExtractorIE(InfoExtractor):
         _VALID_URL = r'https?://(?:www\.)?yourextractor\.com/watch/(?P<id>[0-9]+)'
-        _TEST = {
+        _TESTS = [{
             'url': 'https://yourextractor.com/watch/42',
             'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)',
             'info_dict': {
@@ -133,12 +190,12 @@ After you have ensured this site is distributing its content legally, you can fo
                 # * A regular expression; start the string with re:
                 # * Any Python type (for example int or float)
             }
-        }
+        }]
 
         def _real_extract(self, url):
             video_id = self._match_id(url)
             webpage = self._download_webpage(url, video_id)
-
+    
             # TODO more code goes here, for example ...
             title = self._html_search_regex(r'<h1>(.+?)</h1>', webpage, 'title')
 
@@ -147,45 +204,48 @@ After you have ensured this site is distributing its content legally, you can fo
                 'title': title,
                 'description': self._og_search_description(webpage),
                 'uploader': self._search_regex(r'<div[^>]+id="uploader"[^>]*>([^<]+)<', webpage, 'uploader', fatal=False),
-                # TODO more properties (see youtube_dl/extractor/common.py)
+                # TODO more properties (see yt_dlp/extractor/common.py)
             }
     ```
-5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
+1. Add an import in [`yt_dlp/extractor/extractors.py`](yt_dlp/extractor/extractors.py).
-6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in.
+1. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, the tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in. You can also run all the tests in one go with `TestDownload.test_YourExtractor_all`
-7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303). Add tests and code for as many as you want.
+1. Make sure you have atleast one test for your extractor. Even if all videos covered by the extractor are expected to be inaccessible for automated testing, tests should still be added with a `skip` parameter indicating why the purticular test is disabled from running.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart):
+1. Have a look at [`yt_dlp/extractor/common.py`](yt_dlp/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](yt_dlp/extractor/common.py#L91-L426). Add tests and code for as many as you want.
+1. Make sure your code follows [yt-dlp coding conventions](#yt-dlp-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart):
 
-        $ flake8 youtube_dl/extractor/yourextractor.py
+        $ flake8 yt_dlp/extractor/yourextractor.py
 
-9. Make sure your code works under all [Python](https://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
+1. Make sure your code works under all [Python](https://www.python.org/) versions supported by yt-dlp, namely CPython and PyPy for Python 3.6 and above. Backward compatability is not required for even older versions of Python.
-10. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
+1. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files, [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
 
-        $ git add youtube_dl/extractor/extractors.py
+        $ git add yt_dlp/extractor/extractors.py
-        $ git add youtube_dl/extractor/yourextractor.py
+        $ git add yt_dlp/extractor/yourextractor.py
-        $ git commit -m '[yourextractor] Add new extractor'
+        $ git commit -m '[yourextractor] Add extractor'
         $ git push origin yourextractor
 
-11. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
+1. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
 
 In any case, thank you very much for your contributions!
 
-## youtube-dl coding conventions
+
+## yt-dlp coding conventions
 
 This section introduces a guide lines for writing idiomatic, robust and future-proof extractor code.
 
-Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hosters out of your control and this layout tends to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize dependency on the source's layout and even to make the code foresee potential future changes and be ready for that. This is important because it will allow the extractor not to break on minor layout changes thus keeping old youtube-dl versions working. Even though this breakage issue is easily fixed by emitting a new version of youtube-dl with a fix incorporated, all the previous versions become broken in all repositories and distros' packages that may not be so prompt in fetching the update from us. Needless to say, some non rolling release distros may never receive an update at all.
+Extractors are very fragile by nature since they depend on the layout of the source data provided by 3rd party media hosters out of your control and this layout tends to change. As an extractor implementer your task is not only to write code that will extract media links and metadata correctly but also to minimize dependency on the source's layout and even to make the code foresee potential future changes and be ready for that. This is important because it will allow the extractor not to break on minor layout changes thus keeping old yt-dlp versions working. Even though this breakage issue may be easily fixed by a new version of yt-dlp, this could take some time, during which the the extractor will remain broken.
+
 
 ### Mandatory and optional metafields
 
-For extraction to work youtube-dl relies on metadata your extractor extracts and provides to youtube-dl expressed by an [information dictionary](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by youtube-dl:
+For extraction to work yt-dlp relies on metadata your extractor extracts and provides to yt-dlp expressed by an [information dictionary](yt_dlp/extractor/common.py#L91-L426) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by yt-dlp:
 
  - `id` (media identifier)
  - `title` (media title)
  - `url` (media download URL) or `formats`
 
-In fact only the last option is technically mandatory (i.e. if you can't figure out the download location of the media the extraction does not make any sense). But by convention youtube-dl also treats `id` and `title` as mandatory. Thus the aforementioned metafields are the critical data that the extraction does not make any sense without and if any of them fail to be extracted then the extractor is considered completely broken.
+The aforementioned metafields are the critical data that the extraction does not make any sense without and if any of them fail to be extracted then the extractor is considered completely broken. While, in fact, only `id` is technically mandatory, due to compatability reasons, yt-dlp also treats `title` as mandatory. The extractor is allowed to return the info dict without url or formats in some special cases if it allows the user to extract usefull information with `--ignore-no-formats-error` - Eg: when the video is a live stream that has not started yet.
 
-[Any field](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L188-L303) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
+[Any field](yt_dlp/extractor/common.py#219-L426) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
 
 #### Example
 
@@ -199,8 +259,10 @@ Assume at this point `meta`'s layout is:
 
 ```python
 {
-    ...
     "summary": "some fancy summary text",
+    "user": {
+        "name": "uploader name"
+    },
     ...
 }
 ```
@@ -219,6 +281,30 @@ description = meta['summary']  # incorrect
 
 The latter will break extraction process with `KeyError` if `summary` disappears from `meta` at some later time but with the former approach extraction will just go ahead with `description` set to `None` which is perfectly fine (remember `None` is equivalent to the absence of data).
 
+
+If the data is nested, do not use `.get` chains, but instead make use of the utility functions `try_get` or `traverse_obj`
+
+Considering the above `meta` again, assume you want to extract `["user"]["name"]` and put it in the resulting info dict as `uploader`
+
+```python
+uploader = try_get(meta, lambda x: x['user']['name'])  # correct
+```
+or
+```python
+uploader = traverse_obj(meta, ('user', 'name'))  # correct
+```
+
+and not like:
+
+```python
+uploader = meta['user']['name']  # incorrect
+```
+or
+```python
+uploader = meta.get('user', {}).get('name')  # incorrect
+```
+
+
 Similarly, you should pass `fatal=False` when extracting optional data from a webpage with `_search_regex`, `_html_search_regex` or similar methods, for instance:
 
 ```python
@@ -238,11 +324,36 @@ description = self._search_regex(
 ```
 
 On failure this code will silently continue the extraction with `description` set to `None`. That is useful for metafields that may or may not be present.
- 
+
+
+Another thing to remember is not to try to iterate over `None`
+
+Say you extracted a list of thumbnails into `thumbnail_data` using `try_get` and now want to iterate over them
+
+```python
+thumbnail_data = try_get(...)
+thumbnails = [{
+    'url': item['url']
+} for item in thumbnail_data or []]  # correct
+```
+
+and not like:
+
+```python
+thumbnail_data = try_get(...)
+thumbnails = [{
+    'url': item['url']
+} for item in thumbnail_data]  # incorrect
+```
+
+In the later case, `thumbnail_data` will be `None` if the field was not found and this will cause the loop `for item in thumbnail_data` to raise a fatal error. Using `for item in thumbnail_data or []` avoids this error and results in setting an empty list in `thumbnails` instead.
+
+
 ### Provide fallbacks
 
 When extracting metadata try to do so from multiple sources. For example if `title` is present in several places, try extracting from at least some of them. This makes it more future-proof in case some of the sources become unavailable.
 
+
 #### Example
 
 Say `meta` from the previous example has a `title` and you are about to extract it. Since `title` is a mandatory meta field you should end up with something like:
@@ -261,6 +372,7 @@ title = meta.get('title') or self._og_search_title(webpage)
 
 This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`.
 
+
 ### Regular expressions
 
 #### Don't capture groups you don't use
@@ -282,11 +394,10 @@ Incorrect:
 r'(id|ID)=(?P<id>\d+)'
 ```
 
-
 #### Make regular expressions relaxed and flexible
 
 When using regular expressions try to write them fuzzy, relaxed and flexible, skipping insignificant parts that are more likely to change, allowing both single and double quotes for quoted values and so on.
- 
+
 ##### Example
 
 Say you need to extract `title` from the following HTML code:
@@ -298,14 +409,14 @@ Say you need to extract `title` from the following HTML code:
 The code for that task should look similar to:
 
 ```python
-title = self._search_regex(
+title = self._search_regex(  # correct
     r'<span[^>]+class="title"[^>]*>([^<]+)', webpage, 'title')
 ```
 
 Or even better:
 
 ```python
-title = self._search_regex(
+title = self._search_regex(  # correct
     r'<span[^>]+class=(["\'])title\1[^>]*>(?P<title>[^<]+)',
     webpage, 'title', group='title')
 ```
@@ -315,14 +426,25 @@ Note how you tolerate potential changes in the `style` attribute's value or swit
 The code definitely should not look like:
 
 ```python
-title = self._search_regex(
+title = self._search_regex(  # incorrect
     r'<span style="position: absolute; left: 910px; width: 90px; float: right; z-index: 9999;" class="title">(.*?)</span>',
     webpage, 'title', group='title')
 ```
 
+or even
+
+```python
+title = self._search_regex(  # incorrect
+    r'<span style=".*?" class="title">(.*?)</span>',
+    webpage, 'title', group='title')
+```
+
+Here the presence or absence of other attributes including `style` is irrelevent for the data we need, and so the regex must not depend on it
+
+
 ### Long lines policy
 
-There is a soft limit to keep lines of code under 80 characters long. This means it should be respected if possible and if it does not make readability and code maintenance worse.
+There is a soft limit to keep lines of code under 100 characters long. This means it should be respected if possible and if it does not make readability and code maintenance worse. Sometimes, it may be reasonable to go upto 120 characters and sometimes even 80 can be unreadable. Keep in mind that this is not a hard limit and is just one of many tools to make the code more readable
 
 For example, you should **never** split long string literals like URLs or some other often copied entities over multiple lines to fit this limit:
 
@@ -359,6 +481,7 @@ TITLE_RE = r'<title>([^<]+)</title>'
 title = self._html_search_regex(TITLE_RE, webpage, 'title')
 ```
 
+
 ### Collapse fallbacks
 
 Multiple fallback values can quickly become unwieldy. Collapse multiple fallback values into a single expression via a list of patterns.
@@ -384,10 +507,13 @@ description = (
 
 Methods supporting list of patterns are: `_search_regex`, `_html_search_regex`, `_og_search_property`, `_html_search_meta`.
 
+
 ### Trailing parentheses
 
 Always move trailing parentheses after the last argument.
 
+Note that this *does not* apply to braces `}` or square brackets `]` both of which should closed be in a new line
+
 #### Example
 
 Correct:
@@ -405,30 +531,36 @@ Incorrect:
 )
 ```
 
+
 ### Use convenience conversion and parsing functions
 
-Wrap all extracted numeric data into safe functions from [`youtube_dl/utils.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
+Wrap all extracted numeric data into safe functions from [`yt_dlp/utils.py`](yt_dlp/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
 
 Use `url_or_none` for safe URL processing.
 
-Use `try_get` for safe metadata extraction from parsed JSON.
+Use `try_get`, `dict_get` and `traverse_obj` for safe metadata extraction from parsed JSON.
 
 Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction. 
 
-Explore [`youtube_dl/utils.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/utils.py) for more useful convenience functions.
+Explore [`yt_dlp/utils.py`](yt_dlp/utils.py) for more useful convenience functions.
 
 #### More examples
 
 ##### Safely extract optional description from parsed JSON
 ```python
-description = try_get(response, lambda x: x['result']['video'][0]['summary'], compat_str)
+description = traverse_obj(response, ('result', 'video', 'summary'), expected_type=str)
 ```
 
 ##### Safely extract more optional metadata
 ```python
-video = try_get(response, lambda x: x['result']['video'][0], dict) or {}
+video = traverse_obj(response, ('result', 'video', 0), default={}, expected_type=dict)
 description = video.get('summary')
 duration = float_or_none(video.get('durationMs'), scale=1000)
 view_count = int_or_none(video.get('views'))
 ```
 
+
+
+
+# EMBEDDING YT-DLP
+See [README.md#embedding-yt-dlp](README.md#embedding-yt-dlp) for instructions on how to embed yt-dlp in another Python program

CONTRIBUTORS

@@ -1,6 +1,6 @@
 pukkandan (owner)
 shirt-dev (collaborator)
-colethedj (collaborator)
+coletdjnz/colethedj (collaborator)
 Ashish0804 (collaborator)
 h-h-h-h
 pauldubois98
@@ -22,7 +22,7 @@ Zocker1999NET
 nao20010128nao
 kurumigi
 bbepis
-animelover1984
+animelover1984/horahoradev
 Pccode66
 RobinD42
 hseg
@@ -52,5 +52,90 @@ hhirtz
 louie-github
 MinePlayersPE
 olifre
-rhsmachine
+rhsmachine/zenerdi0de
 nihil-admirari
+krichbanana
+ohmybahgosh
+nyuszika7h
+blackjack4494
+pyx
+TpmKranz
+mzbaulhaque
+zackmark29
+mbway
+zerodytrash
+wesnm
+pento
+rigstot
+dirkf
+funniray
+Jessecar96
+jhwgh1968
+kikuyan
+max-te
+nchilada
+pgaig
+PSlava
+stdedos
+u-spec-png
+Sipherdrakon
+kidonng
+smege1001
+tandy1000
+IONECarter
+capntrips
+mrfade
+ParadoxGBB
+wlritchi
+NeroBurner
+mahanstreamer
+alerikaisattera
+Derkades
+BunnyHelp
+i6t
+std-move
+Chocobozzz
+ouwou
+korli
+octotherp
+CeruleanSky
+zootedb0t
+chao813
+ChillingPepper
+ConquerorDopy
+dalanmiller
+DigitalDJ
+f4pp3rk1ng
+gesa
+Jules-A
+makeworld-the-better-one
+MKSherbini
+mrx23dot
+poschi3
+raphaeldore
+renalid
+sleaux-meaux
+sulyi
+tmarki
+Vangelis66
+AjaxGb
+ajj8
+jakubadamw
+jfogelman
+timethrow
+sarnoud
+Bojidarist
+18928172992817182/gustaf
+nixklai
+smplayer-dev
+Zirro
+CrypticSignal
+flashdagger
+fractalf
+frafra
+kaz-us
+ozburo
+rhendric
+sdomi
+selfisekai
+stanoarn

Changelog.md

@@ -7,31 +7,709 @@
 * Update Changelog.md and CONTRIBUTORS
 * Change "Merged with ytdl" version in Readme.md if needed
 * Add new/fixed extractors in "new features" section of Readme.md
-* Commit to master as `Release <version>`
+* Commit as `Release <version>`
 * Push to origin/release using `git push origin master:release`
     build task will now run
-* Update version.py using `devscripts\update-version.py`
-* Run `make issuetemplates`
-* Commit to master as `[version] update :ci skip all`
-* Push to origin/master
-* Update changelog in /releases
 
 -->
 
 
+### 2021.11.10
+
+* [youtube] **Fix throttling by decrypting n-sig**
+* Merging extractors from [haruhi-dl](https://git.sakamoto.pl/laudom/haruhi-dl) by [selfisekai](https://github.com/selfisekai)
+    * [extractor] Add `_search_nextjs_data`
+    * [tvp] Fix extractors
+    * [tvp] Add TVPStreamIE
+    * [wppilot] Add extractors
+    * [polskieradio] Add extractors
+    * [radiokapital] Add extractors
+    * [polsatgo] Add extractor by [selfisekai](https://github.com/selfisekai), [sdomi](https://github.com/sdomi)
+* Separate `--check-all-formats` from `--check-formats`
+* Approximate filesize from bitrate
+* Don't create console in `windows_enable_vt_mode`
+* Fix bug in `--load-infojson` of playlists
+* [minicurses] Add colors to `-F` and standardize color-printing code
+* [outtmpl] Add type `link` for internet shortcut files
+* [outtmpl] Add alternate forms for `q` and `j`
+* [outtmpl] Do not traverse `None`
+* [fragment] Fix progress display in fragmented downloads
+* [downloader/ffmpeg] Fix vtt download with ffmpeg
+* [ffmpeg] Detect presence of setts and libavformat version
+* [ExtractAudio] Rescale --audio-quality correctly by [CrypticSignal](https://github.com/CrypticSignal), [pukkandan](https://github.com/pukkandan)
+* [ExtractAudio] Use `libfdk_aac` if available by [CrypticSignal](https://github.com/CrypticSignal)
+* [FormatSort] `eac3` is better than `ac3`
+* [FormatSort] Fix some fields' defaults
+* [generic] Detect more json_ld
+* [generic] parse jwplayer with only the json URL
+* [extractor] Add keyword automatically to SearchIE descriptions
+* [extractor] Fix some errors being converted to `ExtractorError`
+* [utils] Add `join_nonempty`
+* [utils] Add `jwt_decode_hs256` by [Ashish0804](https://github.com/Ashish0804)
+* [utils] Create `DownloadCancelled` exception
+* [utils] Parse `vp09` as vp9
+* [utils] Sanitize URL when determining protocol
+* [test/download] Fallback test to `bv`
+* [docs] Minor documentation improvements
+* [cleanup] Improvements to error and debug messages
+* [cleanup] Minor fixes and cleanup
+* [3speak] Add extractors by [Ashish0804](https://github.com/Ashish0804)
+* [AmazonStore] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Gab] Add extractor by [u-spec-png](https://github.com/u-spec-png)
+* [mediaset] Add playlist support by [nixxo](https://github.com/nixxo)
+* [MLSScoccer] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [N1] Add support for nova.rs by [u-spec-png](https://github.com/u-spec-png)
+* [PlanetMarathi] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [RaiplayRadio] Add extractors by [frafra](https://github.com/frafra)
+* [roosterteeth] Add series extractor
+* [sky] Add `SkyNewsStoryIE` by [ajj8](https://github.com/ajj8)
+* [youtube] Fix sorting for some videos
+* [youtube] Populate `thumbnail` with the best "known" thumbnail
+* [youtube] Refactor itag processing
+* [youtube] Remove unnecessary no-playlist warning
+* [youtube:tab] Add Invidious list for playlists/channels by [rhendric](https://github.com/rhendric)
+* [Bilibili:comments] Fix infinite loop by [u-spec-png](https://github.com/u-spec-png)
+* [ceskatelevize] Fix extractor by [flashdagger](https://github.com/flashdagger)
+* [Coub] Fix media format identification by [wlritchi](https://github.com/wlritchi)
+* [crunchyroll] Add extractor-args `language` and `hardsub`
+* [DiscoveryPlus] Allow language codes in URL
+* [imdb] Fix thumbnail by [ozburo](https://github.com/ozburo)
+* [instagram] Add IOS URL support by [u-spec-png](https://github.com/u-spec-png)
+* [instagram] Improve login code by [u-spec-png](https://github.com/u-spec-png)
+* [Instagram] Improve metadata extraction by [u-spec-png](https://github.com/u-spec-png)
+* [iPrima] Fix extractor by [stanoarn](https://github.com/stanoarn)
+* [itv] Add support for ITV News by [ajj8](https://github.com/ajj8)
+* [la7] Fix extractor by [nixxo](https://github.com/nixxo)
+* [linkedin] Don't login multiple times
+* [mtv] Fix some videos by [Sipherdrakon](https://github.com/Sipherdrakon)
+* [Newgrounds] Fix description by [u-spec-png](https://github.com/u-spec-png)
+* [Nrk] Minor fixes by [fractalf](https://github.com/fractalf)
+* [Olympics] Fix extractor by [u-spec-png](https://github.com/u-spec-png)
+* [piksel] Fix sorting
+* [twitter] Do not sort by codec
+* [viewlift] Add cookie-based login and series support by [Ashish0804](https://github.com/Ashish0804), [pukkandan](https://github.com/pukkandan)
+* [vimeo] Detect source extension and misc cleanup by [flashdagger](https://github.com/flashdagger)
+* [vimeo] Fix ondemand videos and direct URLs with hash
+* [vk] Fix login and add subtitles by [kaz-us](https://github.com/kaz-us)
+* [VLive] Add upload_date and thumbnail by [Ashish0804](https://github.com/Ashish0804)
+* [VRT] Fix login by [pgaig](https://github.com/pgaig)
+* [Vupload] Fix extractor by [u-spec-png](https://github.com/u-spec-png)
+* [wakanim] Add support for MPD manifests by [nyuszika7h](https://github.com/nyuszika7h)
+* [wakanim] Detect geo-restriction by [nyuszika7h](https://github.com/nyuszika7h)
+* [ZenYandex] Fix extractor by [u-spec-png](https://github.com/u-spec-png)
+
+
+### 2021.10.22
+
+* [build] Improvements
+    * Build standalone MacOS packages by [smplayer-dev](https://github.com/smplayer-dev)
+    * Release windows exe built with `py2exe`
+    * Enable lazy-extractors in releases. 
+        * Set env var `YTDLP_NO_LAZY_EXTRACTORS` to forcefully disable this (experimental)
+    * Clean up error reporting in update
+    * Refactor `pyinst.py`, misc cleanup and improve docs
+* [docs] Migrate issues to use forms by [Ashish0804](https://github.com/Ashish0804)
+* [downloader] **Fix slow progress hooks**
+    * This was causing HLS/DASH downloads to be extremely slow in some situations
+* [downloader/ffmpeg] Improve simultaneous download and merge
+* [EmbedMetadata] Allow overwriting all default metadata with `meta_default` key
+* [ModifyChapters] Add ability for `--remove-chapters` to remove sections by timestamp
+* [utils] Allow duration strings in `--match-filter`
+* Add HDR information to formats
+* Add negative option `--no-batch-file` by [Zirro](https://github.com/Zirro)
+* Calculate more fields for merged formats
+* Do not verify thumbnail URLs unless `--check-formats` is specified
+* Don't create console for subprocesses on Windows
+* Fix `--restrict-filename` when used with default template
+* Fix `check_formats` output being written to stdout when `-qv`
+* Fix bug in storyboards
+* Fix conflict b/w id and ext in format selection
+* Fix verbose head not showing custom configs
+* Load archive only after printing verbose head
+* Make `duration_string` and `resolution` available in --match-filter
+* Re-implement deprecated option `--id`
+* Reduce default `--socket-timeout`
+* Write verbose header to logger
+* [outtmpl] Fix bug in expanding environment variables
+* [cookies] Local State should be opened as utf-8
+* [extractor,utils] Detect more codecs/mimetypes
+* [extractor] Detect `EXT-X-KEY` Apple FairPlay
+* [utils] Use `importlib` to load plugins by [sulyi](https://github.com/sulyi)
+* [http] Retry on socket timeout and show the last encountered error
+* [fragment] Print error message when skipping fragment
+* [aria2c] Fix `--skip-unavailable-fragment`
+* [SponsorBlock] Obey `extractor-retries` and `sleep-requests`
+* [Merger] Do not add `aac_adtstoasc` to non-hls audio
+* [ModifyChapters] Do not mutate original chapters by [nihil-admirari](https://github.com/nihil-admirari)
+* [devscripts/run_tests] Use markers to filter tests by [sulyi](https://github.com/sulyi)
+* [7plus] Add cookie based authentication by [nyuszika7h](https://github.com/nyuszika7h)
+* [AdobePass] Fix RCN MSO by [jfogelman](https://github.com/jfogelman)
+* [CBC] Fix Gem livestream by [makeworld-the-better-one](https://github.com/makeworld-the-better-one)
+* [CBC] Support CBC Gem member content by [makeworld-the-better-one](https://github.com/makeworld-the-better-one)
+* [crunchyroll] Add season to flat-playlist
+* [crunchyroll] Add support for `beta.crunchyroll` URLs and fix series URLs with language code
+* [EUScreen] Add Extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Gronkh] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [hidive] Fix typo
+* [Hotstar] Mention Dynamic Range in `format_id` by [Ashish0804](https://github.com/Ashish0804)
+* [Hotstar] Raise appropriate error for DRM
+* [instagram] Add login by [u-spec-png](https://github.com/u-spec-png)
+* [instagram] Show appropriate error when login is needed
+* [microsoftstream] Add extractor by [damianoamatruda](https://github.com/damianoamatruda), [nixklai](https://github.com/nixklai)
+* [on24] Add extractor by [damianoamatruda](https://github.com/damianoamatruda)
+* [patreon] Fix vimeo player regex by [zenerdi0de](https://github.com/zenerdi0de)
+* [SkyNewsAU] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [tagesschau] Fix extractor by [u-spec-png](https://github.com/u-spec-png)
+* [tbs] Add tbs live streams by [llacb47](https://github.com/llacb47)
+* [tiktok] Fix typo and update tests
+* [trovo] Support channel clips and VODs by [Ashish0804](https://github.com/Ashish0804)
+* [Viafree] Add support for Finland by [18928172992817182](https://github.com/18928172992817182)
+* [vimeo] Fix embedded `player.vimeo`
+* [vlive:channel] Fix extraction by [kikuyan](https://github.com/kikuyan), [pukkandan](https://github.com/pukkandan)
+* [youtube] Add auto-translated subtitles
+* [youtube] Expose different formats with same itag
+* [youtube:comments] Fix for new layout by [coletdjnz](https://github.com/coletdjnz)
+* [cleanup] Cleanup bilibili code by [pukkandan](https://github.com/pukkandan), [u-spec-png](https://github.com/u-spec-png)
+* [cleanup] Remove broken youtube login code
+* [cleanup] Standardize timestamp formatting code
+* [cleanup] Generalize `getcomments` implementation for extractors
+* [cleanup] Simplify search extractors code
+* [cleanup] misc
+
+
+### 2021.10.10
+
+* [downloader/ffmpeg] Fix bug in initializing `FFmpegPostProcessor`
+* [minicurses] Fix when printing to file
+* [downloader] Fix throttledratelimit
+* [francetv] Fix extractor by [fstirlitz](https://github.com/fstirlitz), [sarnoud](https://github.com/sarnoud)
+* [NovaPlay] Add extractor by [Bojidarist](https://github.com/Bojidarist)
+* [ffmpeg] Revert "Set max probesize" - No longer needed
+* [docs] Remove incorrect dependency on VC++10
+* [build] Allow to release without changelog
+
+### 2021.10.09
+
+* Improved progress reporting
+    * Separate `--console-title` and `--no-progress`
+    * Add option `--progress` to show progress-bar even in quiet mode
+    * Fix and refactor `minicurses` and use it for all progress reporting
+    * Standardize use of terminal sequences and enable color support for windows 10
+    * Add option `--progress-template` to customize progress-bar and console-title
+    * Add postprocessor hooks and progress reporting
+* [postprocessor] Add plugin support with option `--use-postprocessor`
+* [extractor] Extract storyboards from SMIL manifests by [fstirlitz](https://github.com/fstirlitz)
+* [outtmpl] Alternate form of format type `l` for `\n` delimited list
+* [outtmpl] Format type `U` for unicode normalization
+* [outtmpl] Allow empty output template to skip a type of file
+* Merge webm formats into mkv if thumbnails are to be embedded
+* [adobepass] Add RCN as MSO by [jfogelman](https://github.com/jfogelman)
+* [ciscowebex] Add extractor by [damianoamatruda](https://github.com/damianoamatruda)
+* [Gettr] Add extractor by [i6t](https://github.com/i6t)
+* [GoPro] Add extractor by [i6t](https://github.com/i6t)
+* [N1] Add extractor by [u-spec-png](https://github.com/u-spec-png)
+* [Theta] Add video extractor by [alerikaisattera](https://github.com/alerikaisattera)
+* [Veo] Add extractor by [i6t](https://github.com/i6t)
+* [Vupload] Add extractor by [u-spec-png](https://github.com/u-spec-png)
+* [bbc] Extract better quality videos by [ajj8](https://github.com/ajj8)
+* [Bilibili] Add subtitle converter by [u-spec-png](https://github.com/u-spec-png)
+* [CBC] Cleanup tests by [makeworld-the-better-one](https://github.com/makeworld-the-better-one)
+* [Douyin] Rewrite extractor by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [Funimation] Fix for /v/ urls by [pukkandan](https://github.com/pukkandan), [Jules-A](https://github.com/Jules-A)
+* [Funimation] Sort formats according to the relevant extractor-args
+* [Hidive] Fix duplicate and incorrect formats
+* [HotStarSeries] Fix cookies by [Ashish0804](https://github.com/Ashish0804)
+* [LinkedInLearning] Add subtitles by [Ashish0804](https://github.com/Ashish0804)
+* [Mediaite] Relax valid url by [coletdjnz](https://github.com/coletdjnz)
+* [Newgrounds] Add age_limit and fix duration by [u-spec-png](https://github.com/u-spec-png)
+* [Newgrounds] Fix view count on songs by [u-spec-png](https://github.com/u-spec-png)
+* [parliamentlive.tv] Fix extractor by [u-spec-png](https://github.com/u-spec-png)
+* [PolskieRadio] Fix extractors by [jakubadamw](https://github.com/jakubadamw), [u-spec-png](https://github.com/u-spec-png)
+* [reddit] Add embedded url by [u-spec-png](https://github.com/u-spec-png)
+* [reddit] Fix 429 by generating a random `reddit_session` by [AjaxGb](https://github.com/AjaxGb)
+* [Rumble] Add RumbleChannelIE by [Ashish0804](https://github.com/Ashish0804)
+* [soundcloud:playlist] Detect last page correctly
+* [SovietsCloset] Add duration from m3u8 by [ChillingPepper](https://github.com/ChillingPepper)
+* [Streamable] Add codecs by [u-spec-png](https://github.com/u-spec-png)
+* [vidme] Remove extractor by [alerikaisattera](https://github.com/alerikaisattera)
+* [youtube:tab] Fallback to API when webpage fails to download by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Fix non-fatal errors in fetching player
+* Fix `--flat-playlist` when neither IE nor id is known
+* Fix `-f mp4` behaving differently from youtube-dl
+* Workaround for bug in `ssl.SSLContext.load_default_certs`
+* [aes] Improve performance slightly by [sulyi](https://github.com/sulyi)
+* [cookies] Fix keyring fallback by [mbway](https://github.com/mbway)
+* [embedsubtitle] Fix error when duration is unknown
+* [ffmpeg] Fix error when subtitle file is missing
+* [ffmpeg] Set max probesize to workaround AAC HLS stream issues by [shirt](https://github.com/shirt-dev)
+* [FixupM3u8] Remove redundant run if merged is needed
+* [hls] Fix decryption issues by [shirt](https://github.com/shirt-dev), [pukkandan](https://github.com/pukkandan)
+* [http] Respect user-provided chunk size over extractor's
+* [utils] Let traverse_obj accept functions as keys
+* [docs] Add note about our custom ffmpeg builds
+* [docs] Write embedding and contributing documentation by [pukkandan](https://github.com/pukkandan), [timethrow](https://github.com/timethrow)
+* [update] Check for new version even if not updateable
+* [build] Add more files to the tarball
+* [build] Allow building with py2exe (and misc fixes)
+* [build] Use pycryptodomex by [shirt](https://github.com/shirt-dev), [pukkandan](https://github.com/pukkandan)
+* [cleanup] Some minor refactoring, improve docs and misc cleanup
+
+
+### 2021.09.25
+
+* Add new option `--netrc-location`
+* [outtmpl] Allow alternate fields using `,`
+* [outtmpl] Add format type `B` to treat the value as bytes (eg: to limit the filename to a certain number of bytes)
+* Separate the options `--ignore-errors` and `--no-abort-on-error`
+* Basic framework for simultaneous download of multiple formats by [nao20010128nao](https://github.com/nao20010128nao)
+* [17live] Add 17.live extractor by [nao20010128nao](https://github.com/nao20010128nao)
+* [bilibili] Add BiliIntlIE and BiliIntlSeriesIE by [Ashish0804](https://github.com/Ashish0804)
+* [CAM4] Add extractor by [alerikaisattera](https://github.com/alerikaisattera)
+* [Chingari] Add extractors by [Ashish0804](https://github.com/Ashish0804)
+* [CGTN] Add extractor by [chao813](https://github.com/chao813)
+* [damtomo] Add extractor by [nao20010128nao](https://github.com/nao20010128nao)
+* [gotostage] Add extractor by [poschi3](https://github.com/poschi3)
+* [Koo] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Mediaite] Add Extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Mediaklikk] Add Extractor by [tmarki](https://github.com/tmarki), [mrx23dot](https://github.com/mrx23dot), [coletdjnz](https://github.com/coletdjnz)
+* [MuseScore] Add Extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Newgrounds] Add NewgroundsUserIE and improve extractor by [u-spec-png](https://github.com/u-spec-png)
+* [nzherald] Add NZHeraldIE by [coletdjnz](https://github.com/coletdjnz)
+* [Olympics] Add replay extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Peertube] Add channel and playlist extractors by [u-spec-png](https://github.com/u-spec-png)
+* [radlive] Add extractor by [nyuszika7h](https://github.com/nyuszika7h)
+* [SovietsCloset] Add extractor by [ChillingPepper](https://github.com/ChillingPepper)
+* [Streamanity] Add Extractor by [alerikaisattera](https://github.com/alerikaisattera)
+* [Theta] Add extractor by [alerikaisattera](https://github.com/alerikaisattera)
+* [Yandex] Add ZenYandexIE and ZenYandexChannelIE by [Ashish0804](https://github.com/Ashish0804)
+* [9Now] handle episodes of series by [dalanmiller](https://github.com/dalanmiller)
+* [AnimalPlanet] Fix extractor by [Sipherdrakon](https://github.com/Sipherdrakon)
+* [Arte] Improve description extraction by [renalid](https://github.com/renalid)
+* [atv.at] Use jwt for API by [NeroBurner](https://github.com/NeroBurner)
+* [brightcove] Extract subtitles from manifests
+* [CBC] Fix CBC Gem extractors by [makeworld-the-better-one](https://github.com/makeworld-the-better-one)
+* [cbs] Report appropriate error for DRM
+* [comedycentral] Support `collection-playlist` by [nixxo](https://github.com/nixxo)
+* [DIYNetwork] Support new format by [Sipherdrakon](https://github.com/Sipherdrakon)
+* [downloader/niconico] Pass custom headers by [nao20010128nao](https://github.com/nao20010128nao)
+* [dw] Fix extractor
+* [Fancode] Fix live streams by [zenerdi0de](https://github.com/zenerdi0de)
+* [funimation] Fix for locations outside US by [Jules-A](https://github.com/Jules-A), [pukkandan](https://github.com/pukkandan)
+* [globo] Fix GloboIE by [Ashish0804](https://github.com/Ashish0804)
+* [HiDive] Fix extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Hotstar] Add referer for subs by [Ashish0804](https://github.com/Ashish0804)
+* [itv] Fix extractor, add subtitles and thumbnails by [coletdjnz](https://github.com/coletdjnz), [sleaux-meaux](https://github.com/sleaux-meaux), [Vangelis66](https://github.com/Vangelis66)
+* [lbry] Show error message from API response
+* [Mxplayer] Use mobile API by [Ashish0804](https://github.com/Ashish0804)
+* [NDR] Rewrite NDRIE by [Ashish0804](https://github.com/Ashish0804)
+* [Nuvid] Fix extractor by [u-spec-png](https://github.com/u-spec-png)
+* [Oreilly] Handle new web url by [MKSherbini](https://github.com/MKSherbini)
+* [pbs] Fix subtitle extraction by [coletdjnz](https://github.com/coletdjnz), [gesa](https://github.com/gesa), [raphaeldore](https://github.com/raphaeldore)
+* [peertube] Update instances by [u-spec-png](https://github.com/u-spec-png)
+* [plutotv] Fix extractor for URLs with `/en`
+* [reddit] Workaround for 429 by redirecting to old.reddit.com
+* [redtube] Fix exts
+* [soundcloud] Make playlist extraction lazy
+* [soundcloud] Retry playlist pages on `502` error and update `_CLIENT_ID`
+* [southpark] Fix SouthParkDE by [coletdjnz](https://github.com/coletdjnz)
+* [SovietsCloset] Fix playlists for games with only named categories by [ConquerorDopy](https://github.com/ConquerorDopy)
+* [SpankBang] Fix uploader by [f4pp3rk1ng](https://github.com/f4pp3rk1ng), [coletdjnz](https://github.com/coletdjnz)
+* [tiktok] Use API to fetch higher quality video by [MinePlayersPE](https://github.com/MinePlayersPE), [llacb47](https://github.com/llacb47)
+* [TikTokUser] Fix extractor using mobile API by [MinePlayersPE](https://github.com/MinePlayersPE), [llacb47](https://github.com/llacb47)
+* [videa] Fix some extraction errors by [nyuszika7h](https://github.com/nyuszika7h)
+* [VrtNU] Handle login errors by [llacb47](https://github.com/llacb47)
+* [vrv] Don't raise error when thumbnails are missing
+* [youtube] Cleanup authentication code by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Fix `--mark-watched` with `--cookies-from-browser`
+* [youtube] Improvements to JS player extraction and add extractor-args to skip it by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Retry on 'Unknown Error' by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Return full URL instead of just ID
+* [youtube] Warn when trying to download clips
+* [zdf] Improve format sorting
+* [zype] Extract subtitles from the m3u8 manifest by [fstirlitz](https://github.com/fstirlitz)
+* Allow `--force-write-archive` to work with `--flat-playlist`
+* Download subtitles in order of `--sub-langs`
+* Allow `0` in `--playlist-items`
+* Handle more playlist errors with `-i`
+* Fix `--no-get-comments`
+* Fix `extra_info` being reused across runs
+* Fix compat options `no-direct-merge` and `playlist-index`
+* Dump files should obey `--trim-filename` by [sulyi](https://github.com/sulyi)
+* [aes] Add `aes_gcm_decrypt_and_verify` by [sulyi](https://github.com/sulyi), [pukkandan](https://github.com/pukkandan)
+* [aria2c] Fix IV for some AES-128 streams by [shirt](https://github.com/shirt-dev)
+* [compat] Don't ignore `HOME` (if set) on windows
+* [cookies] Make browser names case insensitive
+* [cookies] Print warning for cookie decoding error only once
+* [extractor] Fix root-relative URLs in MPD by [DigitalDJ](https://github.com/DigitalDJ)
+* [ffmpeg] Add `aac_adtstoasc` when merging if needed
+* [fragment,aria2c] Generalize and refactor some code
+* [fragment] Avoid repeated request for AES key
+* [fragment] Fix range header when using `-N` and media sequence by [shirt](https://github.com/shirt-dev)
+* [hls,aes] Fallback to native implementation for AES-CBC and detect `Cryptodome` in addition to `Crypto`
+* [hls] Byterange + AES128 is supported by native downloader
+* [ModifyChapters] Improve sponsor chapter merge algorithm by [nihil-admirari](https://github.com/nihil-admirari)
+* [ModifyChapters] Minor fixes
+* [WebVTT] Adjust parser to accommodate PBS subtitles
+* [utils] Improve `extract_timezone` by [dirkf](https://github.com/dirkf)
+* [options] Fix `--no-config` and refactor reading of config files
+* [options] Strip spaces and ignore empty entries in list-like switches
+* [test/cookies] Improve logging
+* [build] Automate more of the release process by [animelover1984](https://github.com/animelover1984), [pukkandan](https://github.com/pukkandan)
+* [build] Fix sha256 by [nihil-admirari](https://github.com/nihil-admirari)
+* [build] Bring back brew taps by [nao20010128nao](https://github.com/nao20010128nao)
+* [build] Provide `--onedir` zip for windows by [pukkandan](https://github.com/pukkandan)
+* [cleanup,docs] Add deprecation warning in docs for some counter intuitive behaviour
+* [cleanup] Fix line endings for `nebula.py` by [glenn-slayden](https://github.com/glenn-slayden)
+* [cleanup] Improve `make clean-test` by [sulyi](https://github.com/sulyi)
+* [cleanup] Misc
+
+
+### 2021.09.02
+
+* **Native SponsorBlock** implementation by [nihil-admirari](https://github.com/nihil-admirari), [pukkandan](https://github.com/pukkandan)
+    * `--sponsorblock-remove CATS` removes specified chapters from file
+    * `--sponsorblock-mark CATS` marks the specified sponsor sections as chapters
+    * `--sponsorblock-chapter-title TMPL` to specify sponsor chapter template
+    * `--sponsorblock-api URL` to use a different API
+    * No re-encoding is done unless `--force-keyframes-at-cuts` is used
+    * The fetched sponsor sections are written to the infojson
+    * Deprecates: `--sponskrub`, `--no-sponskrub`, `--sponskrub-cut`, `--no-sponskrub-cut`, `--sponskrub-force`, `--no-sponskrub-force`, `--sponskrub-location`, `--sponskrub-args`
+* Split `--embed-chapters` from `--embed-metadata` (it still implies the former by default)
+* Add option `--remove-chapters` to remove arbitrary chapters by [nihil-admirari](https://github.com/nihil-admirari), [pukkandan](https://github.com/pukkandan)
+* Add option `--force-keyframes-at-cuts` for more accurate cuts when removing and splitting chapters by [nihil-admirari](https://github.com/nihil-admirari)
+* Let `--match-filter` reject entries early
+    * Makes redundant: `--match-title`, `--reject-title`, `--min-views`, `--max-views`
+* [lazy_extractor] Improvements (It now passes all tests)
+    * Bugfix for when plugin directory doesn't exist by [kidonng](https://github.com/kidonng)
+    * Create instance only after pre-checking archive
+    * Import actual class if an attribute is accessed
+    * Fix `suitable` and add flake8 test
+* [downloader/ffmpeg] Experimental support for DASH manifests (including live)
+    * Your ffmpeg must have [this patch](https://github.com/FFmpeg/FFmpeg/commit/3249c757aed678780e22e99a1a49f4672851bca9) applied for YouTube DASH to work
+* [downloader/ffmpeg] Allow passing custom arguments before `-i`
+* [BannedVideo] Add extractor by [smege1001](https://github.com/smege1001), [blackjack4494](https://github.com/blackjack4494), [pukkandan](https://github.com/pukkandan)
+* [bilibili] Add category extractor by [animelover1984](https://github.com/animelover1984)
+* [Epicon] Add extractors by [Ashish0804](https://github.com/Ashish0804)
+* [filmmodu] Add extractor by [mzbaulhaque](https://github.com/mzbaulhaque)
+* [GabTV] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Hungama] Fix `HungamaSongIE` and add `HungamaAlbumPlaylistIE` by [Ashish0804](https://github.com/Ashish0804)
+* [ManotoTV] Add new extractors by [tandy1000](https://github.com/tandy1000)
+* [Niconico] Add Search extractors by [animelover1984](https://github.com/animelover1984), [pukkandan](https://github.com/pukkandan)
+* [Patreon] Add `PatreonUserIE` by [zenerdi0de](https://github.com/zenerdi0de)
+* [peloton] Add extractor by [IONECarter](https://github.com/IONECarter), [capntrips](https://github.com/capntrips), [pukkandan](https://github.com/pukkandan)
+* [ProjectVeritas] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [radiko] Add extractors by [nao20010128nao](https://github.com/nao20010128nao)
+* [StarTV] Add extractor for `startv.com.tr` by [mrfade](https://github.com/mrfade), [coletdjnz](https://github.com/coletdjnz)
+* [tiktok] Add `TikTokUserIE` by [Ashish0804](https://github.com/Ashish0804), [pukkandan](https://github.com/pukkandan)
+* [Tokentube] Add extractor by [u-spec-png](https://github.com/u-spec-png)
+* [TV2Hu] Fix `TV2HuIE` and add `TV2HuSeriesIE` by [Ashish0804](https://github.com/Ashish0804)
+* [voicy] Add extractor by [nao20010128nao](https://github.com/nao20010128nao)
+* [adobepass] Fix Verizon SAML login by [nyuszika7h](https://github.com/nyuszika7h), [ParadoxGBB](https://github.com/ParadoxGBB)
+* [afreecatv] Fix adult VODs by [wlritchi](https://github.com/wlritchi)
+* [afreecatv] Tolerate failure to parse date string by [wlritchi](https://github.com/wlritchi)
+* [aljazeera] Fix extractor by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [ATV.at] Fix extractor for ATV.at by [NeroBurner](https://github.com/NeroBurner), [coletdjnz](https://github.com/coletdjnz)
+* [bitchute] Fix test by [mahanstreamer](https://github.com/mahanstreamer)
+* [camtube] Remove obsolete extractor by [alerikaisattera](https://github.com/alerikaisattera)
+* [CDA] Add more formats by [u-spec-png](https://github.com/u-spec-png)
+* [eroprofile] Fix page skipping in albums by [jhwgh1968](https://github.com/jhwgh1968)
+* [facebook] Fix format sorting
+* [facebook] Fix metadata extraction by [kikuyan](https://github.com/kikuyan)
+* [facebook] Update onion URL by [Derkades](https://github.com/Derkades)
+* [HearThisAtIE] Fix extractor by [Ashish0804](https://github.com/Ashish0804)
+* [instagram] Add referrer to prevent throttling by [u-spec-png](https://github.com/u-spec-png), [kikuyan](https://github.com/kikuyan)
+* [iwara.tv] Extract more metadata by [BunnyHelp](https://github.com/BunnyHelp)
+* [iwara] Add thumbnail by [i6t](https://github.com/i6t)
+* [kakao] Fix extractor
+* [mediaset] Fix extraction for some videos by [nyuszika7h](https://github.com/nyuszika7h)
+* [Motherless] Fix extractor by [coletdjnz](https://github.com/coletdjnz)
+* [Nova] fix extractor by [std-move](https://github.com/std-move)
+* [ParamountPlus] Fix geo verification by [shirt](https://github.com/shirt-dev)
+* [peertube] handle new video URL format by [Chocobozzz](https://github.com/Chocobozzz)
+* [pornhub] Separate and fix playlist extractor by [mzbaulhaque](https://github.com/mzbaulhaque)
+* [reddit] Fix for quarantined subreddits by [ouwou](https://github.com/ouwou)
+* [ShemarooMe] Fix extractor by [Ashish0804](https://github.com/Ashish0804)
+* [soundcloud] Refetch `client_id` on 403
+* [tiktok] Fix metadata extraction
+* [TV2] Fix extractor by [Ashish0804](https://github.com/Ashish0804)
+* [tv5mondeplus] Fix extractor by [korli](https://github.com/korli)
+* [VH1,TVLand] Fix extractors by [Sipherdrakon](https://github.com/Sipherdrakon)
+* [Viafree] Fix extractor and extract subtitles by [coletdjnz](https://github.com/coletdjnz)
+* [XHamster] Extract `uploader_id` by [octotherp](https://github.com/octotherp)
+* [youtube] Add `shorts` to `_VALID_URL`
+* [youtube] Add av01 itags to known formats list by [blackjack4494](https://github.com/blackjack4494)
+* [youtube] Extract error messages from HTTPError response by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Fix subtitle names
+* [youtube] Prefer audio stream that YouTube considers default
+* [youtube] Remove annotations and deprecate `--write-annotations` by [coletdjnz](https://github.com/coletdjnz)
+* [Zee5] Fix extractor and add subtitles by [Ashish0804](https://github.com/Ashish0804)
+* [aria2c] Obey `--rate-limit`
+* [EmbedSubtitle] Continue even if some files are missing
+* [extractor] Better error message for DRM
+* [extractor] Common function `_match_valid_url`
+* [extractor] Show video id in error messages if possible
+* [FormatSort] Remove priority of `lang`
+* [options] Add `_set_from_options_callback`
+* [SubtitleConvertor] Fix bug during subtitle conversion
+* [utils] Add `parse_qs`
+* [webvtt] Fix timestamp overflow adjustment by [fstirlitz](https://github.com/fstirlitz)
+* Bugfix for `--replace-in-metadata`
+* Don't try to merge with final extension
+* Fix `--force-overwrites` when using `-k`
+* Fix `--no-prefer-free-formats` by [CeruleanSky](https://github.com/CeruleanSky)
+* Fix `-F` for extractors that directly return url
+* Fix `-J` when there are failed videos
+* Fix `extra_info` being reused across runs
+* Fix `playlist_index` not obeying `playlist_start` and add tests
+* Fix resuming of single formats when using `--no-part`
+* Revert erroneous use of the `Content-Length` header by [fstirlitz](https://github.com/fstirlitz)
+* Use `os.replace` where applicable by; paulwrubel
+* [build] Add homebrew taps `yt-dlp/taps/yt-dlp` by [nao20010128nao](https://github.com/nao20010128nao)
+* [build] Fix bug in making `yt-dlp.tar.gz`
+* [docs] Fix some typos by [pukkandan](https://github.com/pukkandan), [zootedb0t](https://github.com/zootedb0t)
+* [cleanup] Replace improper use of tab in trovo by [glenn-slayden](https://github.com/glenn-slayden)
+
+
+### 2021.08.10
+
+* Add option `--replace-in-metadata`
+* Add option `--no-simulate` to not simulate even when `--print` or `--list...` are used - Deprecates `--print-json`
+* Allow entire infodict to be printed using `%()s` - makes `--dump-json` redundant
+* Allow multiple `--exec` and `--exec-before-download`
+* Add regex to `--match-filter`
+* Add all format filtering operators also to `--match-filter` by [max-te](https://github.com/max-te)
+* Add compat-option `no-keep-subs`
+* [adobepass] Add MSO Cablevision by [Jessecar96](https://github.com/Jessecar96)
+* [BandCamp] Add BandcampMusicIE by [Ashish0804](https://github.com/Ashish0804)
+* [blackboardcollaborate] Add new extractor by [mzbaulhaque](https://github.com/mzbaulhaque)
+* [eroprofile] Add album downloader by [jhwgh1968](https://github.com/jhwgh1968)
+* [mirrativ] Add extractors by [nao20010128nao](https://github.com/nao20010128nao)
+* [openrec] Add extractors by [nao20010128nao](https://github.com/nao20010128nao)
+* [nbcolympics:stream] Fix extractor by [nchilada](https://github.com/nchilada), [pukkandan](https://github.com/pukkandan)
+* [nbcolympics] Update extractor for 2020 olympics by [wesnm](https://github.com/wesnm)
+* [paramountplus] Separate extractor and fix some titles by [shirt](https://github.com/shirt-dev), [pukkandan](https://github.com/pukkandan)
+* [RCTIPlus] Support events and TV by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [Newgrounds] Improve extractor and fix playlist by [u-spec-png](https://github.com/u-spec-png)
+* [aenetworks] Update `_THEPLATFORM_KEY` and `_THEPLATFORM_SECRET` by [wesnm](https://github.com/wesnm)
+* [crunchyroll] Fix thumbnail by [funniray](https://github.com/funniray)
+* [HotStar] Use API for metadata and extract subtitles by [Ashish0804](https://github.com/Ashish0804)
+* [instagram] Fix comments extraction by [u-spec-png](https://github.com/u-spec-png)
+* [peertube] Fix videos without description by [u-spec-png](https://github.com/u-spec-png)
+* [twitch:clips] Extract `display_id` by [dirkf](https://github.com/dirkf)
+* [viki] Print error message from API request
+* [Vine] Remove invalid formats by [u-spec-png](https://github.com/u-spec-png)
+* [VrtNU] Fix XSRF token by [pgaig](https://github.com/pgaig)
+* [vrv] Fix thumbnail extraction by [funniray](https://github.com/funniray)
+* [youtube] Add extractor-arg `include-live-dash` to show live dash formats
+* [youtube] Improve signature function detection by [PSlava](https://github.com/PSlava)
+* [youtube] Raise appropriate error when API pages can't be downloaded
+* Ensure `_write_ytdl_file` closes file handle on error
+* Fix `--compat-options filename` by [stdedos](https://github.com/stdedos)
+* Fix issues with infodict sanitization
+* Fix resuming when using `--no-part`
+* Fix wrong extension for intermediate files
+* Handle `BrokenPipeError` by [kikuyan](https://github.com/kikuyan)
+* Show libraries present in verbose head
+* [extractor] Detect `sttp` as subtitles in MPD by [fstirlitz](https://github.com/fstirlitz)
+* [extractor] Reset non-repeating warnings per video
+* [ffmpeg] Fix streaming `mp4` to `stdout`
+* [ffpmeg] Allow `--ffmpeg-location` to be a file with different name
+* [utils] Fix `InAdvancePagedList.__getitem__`
+* [utils] Fix `traverse_obj` depth when `is_user_input`
+* [webvtt] Merge daisy-chained duplicate cues by [fstirlitz](https://github.com/fstirlitz)
+* [build] Use custom build of `pyinstaller` by [shirt](https://github.com/shirt-dev)
+* [tests:download] Add batch testing for extractors (`test_YourExtractor_all`)
+* [docs] Document which fields `--add-metadata` adds to the file
+* [docs] Fix some mistakes and improve doc
+* [cleanup] Misc code cleanup
+
+
+### 2021.08.02
+
+* Add logo, banner and donate links
+* [outtmpl] Expand and escape environment variables
+* [outtmpl] Add format types `j` (json), `l` (comma delimited list), `q` (quoted for terminal)
+* [downloader] Allow streaming some unmerged formats to stdout using ffmpeg
+* [youtube] **Age-gate bypass**
+    * Add `agegate` clients by [pukkandan](https://github.com/pukkandan), [MinePlayersPE](https://github.com/MinePlayersPE)
+    * Add `thirdParty` to agegate clients to bypass more videos
+    * Simplify client definitions, expose `embedded` clients
+    * Improve age-gate detection by [coletdjnz](https://github.com/coletdjnz)
+    * Fix default global API key by [coletdjnz](https://github.com/coletdjnz)
+    * Add `creator` clients for age-gate bypass using unverified accounts by [zerodytrash](https://github.com/zerodytrash), [coletdjnz](https://github.com/coletdjnz), [pukkandan](https://github.com/pukkandan)
+* [adobepass] Add MSO Sling TV by [wesnm](https://github.com/wesnm)
+* [CBS] Add ParamountPlusSeriesIE by [Ashish0804](https://github.com/Ashish0804)
+* [dplay] Add `ScienceChannelIE` by [Sipherdrakon](https://github.com/Sipherdrakon)
+* [UtreonIE] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [youtube] Add `mweb` client by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Add `player_client=all`
+* [youtube] Force `hl=en` for comments by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Fix format sorting when using alternate clients
+* [youtube] Misc cleanup by [pukkandan](https://github.com/pukkandan), [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Extract SAPISID only once
+* [CBS] Add fallback by [llacb47](https://github.com/llacb47), [pukkandan](https://github.com/pukkandan)
+* [Hotstar] Support cookies by [Ashish0804](https://github.com/Ashish0804)
+* [HotStarSeriesIE] Fix regex by [Ashish0804](https://github.com/Ashish0804)
+* [bilibili] Improve `_VALID_URL`
+* [mediaset] Fix extraction by [nixxo](https://github.com/nixxo)
+* [Mxplayer] Add h265 formats by [Ashish0804](https://github.com/Ashish0804)
+* [RCTIPlus] Remove PhantomJS dependency by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [tenplay] Add MA15+ age limit by [pento](https://github.com/pento)
+* [vidio] Fix login error detection by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [vimeo] Better extraction of original file by [Ashish0804](https://github.com/Ashish0804)
+* [generic] Support KVS player (replaces ThisVidIE) by [rigstot](https://github.com/rigstot)
+* Add compat-option `no-clean-infojson`
+* Remove `asr` appearing twice in `-F`
+* Set `home:` as the default key for `-P`
+* [utils] Fix slicing of reversed `LazyList`
+* [FormatSort] Fix bug for audio with unknown codec
+* [test:download] Support testing with `ignore_no_formats_error`
+* [cleanup] Refactor some code
+
+
+### 2021.07.24
+
+* [youtube:tab] Extract video duration early
+* [downloader] Pass `info_dict` to `progress_hook`s
+* [youtube] Fix age-gated videos for API clients when cookies are supplied by [coletdjnz](https://github.com/coletdjnz)
+* [youtube] Disable `get_video_info` age-gate workaround - This endpoint seems to be completely dead
+* [youtube] Try all clients even if age-gated
+* [youtube] Fix subtitles only being extracted from the first client
+* [youtube] Simplify `_get_text`
+* [cookies] bugfix for microsoft edge on macOS
+* [cookies] Handle `sqlite` `ImportError` gracefully by [mbway](https://github.com/mbway)
+* [cookies] Handle errors when importing `keyring`
+
+### 2021.07.21
+
+* **Add option `--cookies-from-browser`** to load cookies from a browser by [mbway](https://github.com/mbway)
+    * Usage: `--cookies-from-browser BROWSER[:PROFILE_NAME_OR_PATH]`
+    * Also added `--no-cookies-from-browser`
+    * To decrypt chromium cookies, `keyring` is needed for UNIX and `pycryptodome` for Windows
+* Add option `--exec-before-download`
+* Add field `live_status`
+* [FFmpegMetadata] Add language of each stream and some refactoring
+* [douyin] Add extractor by [pukkandan](https://github.com/pukkandan), [pyx](https://github.com/pyx)
+* [pornflip] Add extractor by [mzbaulhaque](https://github.com/mzbaulhaque)
+* **[youtube] Extract data from multiple clients** by [pukkandan](https://github.com/pukkandan), [coletdjnz](https://github.com/coletdjnz)
+    * `player_client` now accepts multiple clients
+    * Default `player_client` = `android,web`
+        * This uses twice as many requests, but avoids throttling for most videos while also not losing any formats
+    * Music clients can be specifically requested and is enabled by default if `music.youtube.com`
+    * Added `player_client=ios` (Known issue: formats from ios are not sorted correctly)
+    * Add age-gate bypass for android and ios clients
+* [youtube] Extract more thumbnails
+    * The thumbnail URLs are hard-coded and their actual existence is tested lazily
+    * Added option `--no-check-formats` to not test them
+* [youtube] Misc fixes
+    * Improve extraction of livestream metadata by [pukkandan](https://github.com/pukkandan), [krichbanana](https://github.com/krichbanana)
+    * Hide live dash formats since they can't be downloaded anyway
+    * Fix authentication when using multiple accounts by [coletdjnz](https://github.com/coletdjnz)
+    * Fix controversial videos when requested via API by [coletdjnz](https://github.com/coletdjnz)
+    * Fix session index extraction and headers for non-web player clients by [coletdjnz](https://github.com/coletdjnz)
+    * Make `--extractor-retries` work for more errors
+    * Fix sorting of 3gp format
+    * Sanity check `chapters` (and refactor related code)
+    * Make `parse_time_text` and `_extract_chapters` non-fatal
+    * Misc cleanup and bug fixes by [coletdjnz](https://github.com/coletdjnz)
+* [youtube:tab] Fix channels tab
+* [youtube:tab] Extract playlist availability by [coletdjnz](https://github.com/coletdjnz)
+* **[youtube:comments] Move comment extraction to new API** by [coletdjnz](https://github.com/coletdjnz)
+    * Adds extractor-args `comment_sort` (`top`/`new`), `max_comments`, `max_comment_depth`
+* [youtube:comments] Fix `is_favorited`, improve `like_count` parsing by [coletdjnz](https://github.com/coletdjnz)
+* [BravoTV] Improve metadata extraction by [kevinoconnor7](https://github.com/kevinoconnor7)
+* [crunchyroll:playlist] Force http
+* [yahoo:gyao:player] Relax `_VALID_URL` by [nao20010128nao](https://github.com/nao20010128nao)
+* [nebula] Authentication via tokens from cookie jar by [hheimbuerger](https://github.com/hheimbuerger), [TpmKranz](https://github.com/TpmKranz)
+* [RTP] Fix extraction and add subtitles by [fstirlitz](https://github.com/fstirlitz)
+* [viki] Rewrite extractors and add extractor-arg `video_types` to `vikichannel` by [zackmark29](https://github.com/zackmark29), [pukkandan](https://github.com/pukkandan)
+* [vlive] Extract thumbnail directly in addition to the one from Naver
+* [generic] Extract previously missed subtitles by [fstirlitz](https://github.com/fstirlitz)
+* [generic] Extract everything in the SMIL manifest and detect discarded subtitles by [fstirlitz](https://github.com/fstirlitz)
+* [embedthumbnail] Fix `_get_thumbnail_resolution`
+* [metadatafromfield] Do not detect numbers as field names
+* Fix selectors `all`, `mergeall` and add tests
+* Errors in playlist extraction should obey `--ignore-errors`
+* Fix bug where `original_url` was not propagated when `_type`=`url`
+* Revert "Merge webm formats into mkv if thumbnails are to be embedded (#173)"
+    * This was wrongly checking for `write_thumbnail`
+* Improve `extractor_args` parsing
+* Rename `NOTE` in `-F` to `MORE INFO` since it's often confused to be the same as `format_note`
+* Add `only_once` param for `write_debug` and `report_warning`
+* [extractor] Allow extracting multiple groups in `_search_regex` by [fstirlitz](https://github.com/fstirlitz)
+* [utils] Improve `traverse_obj`
+* [utils] Add `variadic`
+* [utils] Improve `js_to_json` comment regex by [fstirlitz](https://github.com/fstirlitz)
+* [webtt] Fix timestamps
+* [compat] Remove unnecessary code
+* [docs] fix default of multistreams
+
+
+### 2021.07.07
+
+* Merge youtube-dl: Upto [commit/a803582](https://github.com/ytdl-org/youtube-dl/commit/a8035827177d6b59aca03bd717acb6a9bdd75ada)
+* Add `--extractor-args` to pass some extractor-specific arguments. See [readme](https://github.com/yt-dlp/yt-dlp#extractor-arguments)
+    * Add extractor option `skip` for `youtube`. Eg: `--extractor-args youtube:skip=hls,dash`
+    * Deprecates `--youtube-skip-dash-manifest`, `--youtube-skip-hls-manifest`, `--youtube-include-dash-manifest`, `--youtube-include-hls-manifest`
+* Allow `--list...` options to work with `--print`, `--quiet` and other `--list...` options
+* [youtube] Use `player` API for additional video extraction requests by [coletdjnz](https://github.com/coletdjnz)
+    * **Fixes youtube premium music** (format 141) extraction
+    * Adds extractor option `player_client` = `web`/`android`
+        * **`--extractor-args youtube:player_client=android` works around the throttling** for the time-being
+    * Adds extractor option `player_skip=config`
+    * Adds age-gate fallback using embedded client
+* [youtube] Choose correct Live chat API for upcoming streams by [krichbanana](https://github.com/krichbanana)
+* [youtube] Fix subtitle names for age-gated videos
+* [youtube:comments] Fix error handling and add `itct` to params by [coletdjnz](https://github.com/coletdjnz)
+* [youtube_live_chat] Fix download with cookies by [siikamiika](https://github.com/siikamiika)
+* [youtube_live_chat] use `clickTrackingParams` by [siikamiika](https://github.com/siikamiika)
+* [Funimation] Rewrite extractor
+    * Add `FunimationShowIE` by [Mevious](https://github.com/Mevious)
+    * **Treat the different versions of an episode as different formats of a single video**
+        * This changes the video `id` and will break break existing archives
+        * Compat option `seperate-video-versions` to fall back to old behavior including using the old video ids
+    * Support direct `/player/` URL
+    * Extractor options `language` and `version` to pre-select them during extraction
+        * These options may be removed in the future if we can extract all formats without additional network requests
+        * Do not rely on these for format selection and use `-f` filters instead
+* [AdobePass] Add Spectrum MSO by [kevinoconnor7](https://github.com/kevinoconnor7), [ohmybahgosh](https://github.com/ohmybahgosh)
+* [facebook] Extract description and fix title
+* [fancode] Fix extraction, support live and allow login with refresh token by [zenerdi0de](https://github.com/zenerdi0de)
+* [plutotv] Improve `_VALID_URL`
+* [RCTIPlus] Add extractor by [MinePlayersPE](https://github.com/MinePlayersPE)
+* [Soundcloud] Allow login using oauth token by [blackjack4494](https://github.com/blackjack4494)
+* [TBS] Support livestreams by [llacb47](https://github.com/llacb47)
+* [videa] Fix extraction by [nyuszika7h](https://github.com/nyuszika7h)
+* [yahoo] Fix extraction by [llacb47](https://github.com/llacb47), [pukkandan](https://github.com/pukkandan)
+* Process videos when using `--ignore-no-formats-error` by [krichbanana](https://github.com/krichbanana)
+* Fix `--throttled-rate` when using `--load-info-json`
+* Fix `--flat-playlist` when entry has no `ie_key`
+* Fix `check_formats` catching `ExtractorError` instead of `DownloadError`
+* Fix deprecated option `--list-formats-old`
+* [downloader/ffmpeg] Fix `--ppa` when using simultaneous download
+* [extractor] Prevent unnecessary download of hls manifests and refactor `hls_split_discontinuity`
+* [fragment] Handle status of download and errors in threads correctly; and minor refactoring
+* [thumbnailsconvertor] Treat `jpeg` as `jpg`
+* [utils] Fix issues with `LazyList` reversal
+* [extractor] Allow extractors to set their own login hint
+* [cleanup] Simplify format selector code with `LazyList` and `yield from`
+* [cleanup] Clean `extractor.common._merge_subtitles` signature
+* [cleanup] Fix some typos
+
+
 ### 2021.06.23
 
 * Merge youtube-dl: Upto [commit/379f52a](https://github.com/ytdl-org/youtube-dl/commit/379f52a4954013767219d25099cce9e0f9401961)
 * **Add option `--throttled-rate`** below which video data is re-extracted
 * [fragment] **Merge during download for `-N`**, and refactor `hls`/`dash`
-* [websockets] Add `WebSocketFragmentFD`by [nao20010128nao](https://github.com/nao20010128nao), [pukkandan](https://github.com/pukkandan)
+* [websockets] Add `WebSocketFragmentFD` by [nao20010128nao](https://github.com/nao20010128nao), [pukkandan](https://github.com/pukkandan)
 * Allow `images` formats in addition to video/audio
 * [downloader/mhtml] Add new downloader for slideshows/storyboards by [fstirlitz](https://github.com/fstirlitz)
 * [youtube] Temporary **fix for age-gate**
 * [youtube] Support ongoing live chat by [siikamiika](https://github.com/siikamiika)
-* [youtube] Improve SAPISID cookie handling by [colethedj](https://github.com/colethedj)
+* [youtube] Improve SAPISID cookie handling by [coletdjnz](https://github.com/coletdjnz)
 * [youtube] Login is not needed for `:ytrec`
-* [youtube] Non-fatal alert reporting for unavailable videos page by [colethedj](https://github.com/colethedj)
+* [youtube] Non-fatal alert reporting for unavailable videos page by [coletdjnz](https://github.com/coletdjnz)
 * [twitcasting] Websocket support by [nao20010128nao](https://github.com/nao20010128nao)
 * [mediasite] Extract slides by [fstirlitz](https://github.com/fstirlitz)
 * [funimation] Extract subtitles 
@@ -55,7 +733,7 @@
 ### 2021.06.09
 
 * Fix bug where `%(field)d` in filename template throws error
-* Improve offset parsing in outtmpl
+* [outtmpl] Improve offset parsing
 * [test] More rigorous tests for `prepare_filename`
 
 ### 2021.06.08
@@ -89,7 +767,7 @@
 * Merge youtube-dl: Upto [commit/d495292](https://github.com/ytdl-org/youtube-dl/commit/d495292852b6c2f1bd58bc2141ff2b0265c952cf)
 * Pre-check archive and filters during playlist extraction
 * Handle Basic Auth `user:pass` in URLs by [hhirtz](https://github.com/hhirtz) and [pukkandan](https://github.com/pukkandan)
-* [archiveorg] Add YoutubeWebArchiveIE by [colethedj](https://github.com/colethedj) and [alex-gedeon](https://github.com/alex-gedeon)
+* [archiveorg] Add YoutubeWebArchiveIE by [coletdjnz](https://github.com/coletdjnz) and [alex-gedeon](https://github.com/alex-gedeon)
 * [fancode] Add extractor by [rhsmachine](https://github.com/rhsmachine)
 * [patreon] Support vimeo embeds by [rhsmachine](https://github.com/rhsmachine)
 * [Saitosan] Add new extractor by [llacb47](https://github.com/llacb47)
@@ -132,7 +810,7 @@
 
 * **Youtube improvements**: 
     * Support youtube music `MP`, `VL` and `browse` pages
-    * Extract more formats for youtube music by [craftingmod](https://github.com/craftingmod), [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
+    * Extract more formats for youtube music by [craftingmod](https://github.com/craftingmod), [coletdjnz](https://github.com/coletdjnz) and [pukkandan](https://github.com/pukkandan)
     * Extract multiple subtitles in same language by [pukkandan](https://github.com/pukkandan) and [tpikonen](https://github.com/tpikonen)
     * Redirect channels that doesn't have a `videos` tab to their `UU` playlists
     * Support in-channel search
@@ -141,10 +819,10 @@
     * Extract audio language
     * Add subtitle language names by [nixxo](https://github.com/nixxo) and [tpikonen](https://github.com/tpikonen)
     * Show alerts only from the final webpage
-    * Add `html5=1` param to `get_video_info` page requests by [colethedj](https://github.com/colethedj)
+    * Add `html5=1` param to `get_video_info` page requests by [coletdjnz](https://github.com/coletdjnz)
     * Better message when login required
 * **Add option `--print`**: to print any field/template
-    * Deprecates: `--get-description`, `--get-duration`, `--get-filename`, `--get-format`, `--get-id`, `--get-thumbnail`, `--get-title`, `--get-url`
+    * Makes redundant: `--get-description`, `--get-duration`, `--get-filename`, `--get-format`, `--get-id`, `--get-thumbnail`, `--get-title`, `--get-url`
 * Field `additional_urls` to download additional videos from metadata using [`--parse-metadata`](https://github.com/yt-dlp/yt-dlp#modifying-metadata)
 * Merge youtube-dl: Upto [commit/dfbbe29](https://github.com/ytdl-org/youtube-dl/commit/dfbbe2902fc67f0f93ee47a8077c148055c67a9b)
 * Write thumbnail of playlist and add `pl_thumbnail` outtmpl key
@@ -238,11 +916,11 @@
 * [TubiTv] Add TubiTvShowIE by [Ashish0804](https://github.com/Ashish0804)
 * [twitcasting] Fix extractor
 * [viu:ott] Fix extractor and support series by [lkho](https://github.com/lkho) and [pukkandan](https://github.com/pukkandan)
-* [youtube:tab] Show unavailable videos in playlists by [colethedj](https://github.com/colethedj)
+* [youtube:tab] Show unavailable videos in playlists by [coletdjnz](https://github.com/coletdjnz)
 * [youtube:tab] Reload with unavailable videos for all playlists
 * [youtube] Ignore invalid stretch ratio
-* [youtube] Improve channel syncid extraction to support ytcfg by [colethedj](https://github.com/colethedj)
+* [youtube] Improve channel syncid extraction to support ytcfg by [coletdjnz](https://github.com/coletdjnz)
-* [youtube] Standardize API calls for tabs, mixes and search by [colethedj](https://github.com/colethedj)
+* [youtube] Standardize API calls for tabs, mixes and search by [coletdjnz](https://github.com/coletdjnz)
 * [youtube] Bugfix in `_extract_ytcfg`
 * [mildom:user:vod] Download only necessary amount of pages
 * [mildom] Remove proxy completely by [fstirlitz](https://github.com/fstirlitz)
@@ -254,8 +932,8 @@
 * Improve the yt-dlp.sh script by [fstirlitz](https://github.com/fstirlitz)
 * [lazy_extractor] Do not load plugins
 * [ci] Disable fail-fast
-* [documentation] Clarify which deprecated options still work
+* [docs] Clarify which deprecated options still work
-* [documentation] Fix typos
+* [docs] Fix typos
 
 
 ### 2021.04.11
@@ -272,17 +950,17 @@
 * [nitter] Fix extraction of reply tweets and update instance list by [B0pol](https://github.com/B0pol)
 * [nitter] Fix thumbnails by [B0pol](https://github.com/B0pol)
 * [youtube] Fix thumbnail URL
-* [youtube] Parse API parameters from initial webpage by [colethedj](https://github.com/colethedj)
+* [youtube] Parse API parameters from initial webpage by [coletdjnz](https://github.com/coletdjnz)
-* [youtube] Extract comments' approximate timestamp by [colethedj](https://github.com/colethedj)
+* [youtube] Extract comments' approximate timestamp by [coletdjnz](https://github.com/coletdjnz)
 * [youtube] Fix alert extraction
 * [bilibili] Fix uploader
-* [utils] Add `datetime_from_str` and `datetime_add_months` by [colethedj](https://github.com/colethedj)
+* [utils] Add `datetime_from_str` and `datetime_add_months` by [coletdjnz](https://github.com/coletdjnz)
 * Run some `postprocessors` before actual download
 * Improve argument parsing for `-P`, `-o`, `-S`
 * Fix some `m3u8` not obeying `--allow-unplayable-formats`
 * Fix default of `dynamic_mpd`
 * Deprecate `--all-formats`, `--include-ads`, `--hls-prefer-native`, `--hls-prefer-ffmpeg`
-* [documentation] Improvements
+* [docs] Improvements
 
 ### 2021.04.03
 * Merge youtube-dl: Upto [commit/654b4f4](https://github.com/ytdl-org/youtube-dl/commit/654b4f4ff2718f38b3182c1188c5d569c14cc70a)
@@ -293,10 +971,10 @@
 * [mildom] Update extractor with current proxy by [nao20010128nao](https://github.com/nao20010128nao)
 * [ard:mediathek] Fix video id extraction
 * [generic] Detect Invidious' link element
-* [youtube] Show premium state in `availability` by [colethedj](https://github.com/colethedj)
+* [youtube] Show premium state in `availability` by [coletdjnz](https://github.com/coletdjnz)
 * [viewsource] Add extractor to handle `view-source:`
 * [sponskrub] Run before embedding thumbnail
-* [documentation] Improve `--parse-metadata` documentation
+* [docs] Improve `--parse-metadata` documentation
 
 
 ### 2021.03.24.1
@@ -328,8 +1006,8 @@
 * Use headers and cookies when downloading subtitles by [damianoamatruda](https://github.com/damianoamatruda)
 * Parse resolution in info dictionary by [damianoamatruda](https://github.com/damianoamatruda)
 * More consistent warning messages by [damianoamatruda](https://github.com/damianoamatruda) and [pukkandan](https://github.com/pukkandan)
-* [documentation] Add deprecated options and aliases in readme
+* [docs] Add deprecated options and aliases in readme
-* [documentation] Fix some minor mistakes
+* [docs] Fix some minor mistakes
 
 * [niconico] Partial fix adapted from [animelover1984/youtube-dl@b5eff52](https://github.com/animelover1984/youtube-dl/commit/b5eff52dd9ed5565672ea1694b38c9296db3fade) (login and smile formats still don't work)
 * [niconico] Add user extractor by [animelover1984](https://github.com/animelover1984)
@@ -338,7 +1016,7 @@
 * [stitcher] Merge from youtube-dl by [nixxo](https://github.com/nixxo)
 * [rcs] Improved extraction by [nixxo](https://github.com/nixxo)
 * [linuxacadamy] Improve regex
-* [youtube] Show if video is `private`, `unlisted` etc in info (`availability`) by [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
+* [youtube] Show if video is `private`, `unlisted` etc in info (`availability`) by [coletdjnz](https://github.com/coletdjnz) and [pukkandan](https://github.com/pukkandan)
 * [youtube] bugfix for channel playlist extraction
 * [nbc] Improve metadata extraction by [2ShedsJackson](https://github.com/2ShedsJackson)
 
@@ -355,15 +1033,15 @@
 * [wimtv] Add extractor by [nixxo](https://github.com/nixxo)
 * [mtv] Add mtv.it and extract series metadata by [nixxo](https://github.com/nixxo)
 * [pluto.tv] Add extractor by [kevinoconnor7](https://github.com/kevinoconnor7)
-* [youtube] Rewrite comment extraction by [colethedj](https://github.com/colethedj)
+* [youtube] Rewrite comment extraction by [coletdjnz](https://github.com/coletdjnz)
 * [embedthumbnail] Set mtime correctly
 * Refactor some postprocessor/downloader code by [pukkandan](https://github.com/pukkandan) and [shirt](https://github.com/shirt-dev)
 
 
 ### 2021.03.07
-* [youtube] Fix history, mixes, community pages and trending by [pukkandan](https://github.com/pukkandan) and [colethedj](https://github.com/colethedj)
+* [youtube] Fix history, mixes, community pages and trending by [pukkandan](https://github.com/pukkandan) and [coletdjnz](https://github.com/coletdjnz)
-* [youtube] Fix private feeds/playlists on multi-channel accounts by [colethedj](https://github.com/colethedj)
+* [youtube] Fix private feeds/playlists on multi-channel accounts by [coletdjnz](https://github.com/coletdjnz)
-* [youtube] Extract alerts from continuation by [colethedj](https://github.com/colethedj)
+* [youtube] Extract alerts from continuation by [coletdjnz](https://github.com/coletdjnz)
 * [cbs] Add support for ParamountPlus by [shirt](https://github.com/shirt-dev)
 * [mxplayer] Rewrite extractor with show support by [pukkandan](https://github.com/pukkandan) and [Ashish0804](https://github.com/Ashish0804)
 * [gedi] Improvements from youtube-dl by [nixxo](https://github.com/nixxo)
@@ -375,7 +1053,7 @@
 * [downloader] Fix bug for `ffmpeg`/`httpie`
 * [update] Fix updater removing the executable bit on some UNIX distros
 * [update] Fix current build hash for UNIX
-* [documentation] Include wget/curl/aria2c install instructions for Unix by [Ashish0804](https://github.com/Ashish0804)
+* [docs] Include wget/curl/aria2c install instructions for Unix by [Ashish0804](https://github.com/Ashish0804)
 * Fix some videos downloading with `m3u8` extension
 * Remove "fixup is ignored" warning when fixup wasn't passed by user
 
@@ -384,7 +1062,7 @@
 * [build] Fix bug
 
 ### 2021.03.03
-* [youtube] Use new browse API for continuation page extraction by [colethedj](https://github.com/colethedj) and [pukkandan](https://github.com/pukkandan)
+* [youtube] Use new browse API for continuation page extraction by [coletdjnz](https://github.com/coletdjnz) and [pukkandan](https://github.com/pukkandan)
 * Fix HLS playlist downloading by [shirt](https://github.com/shirt-dev)
 * Merge youtube-dl: Upto [2021.03.03](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.03.03)
 * [mtv] Fix extractor
@@ -432,7 +1110,7 @@
 * [ffmpeg] Allow passing custom arguments before -i using `--ppa "ffmpeg_i1:ARGS"` syntax
 * Fix `--windows-filenames` removing `/` from UNIX paths
 * [hls] Show warning if pycryptodome is not found
-* [documentation] Improvements
+* [docs] Improvements
     * Fix documentation of `Extractor Options`
     * Document `all` in format selection
     * Document `playable_in_embed` in output templates
@@ -460,7 +1138,7 @@
     * Exclude `vcruntime140.dll` from UPX by [jbruchon](https://github.com/jbruchon)
     * Set version number based on UTC time, not local time
     * Publish on PyPi only if token is set
-* [documentation] Better document `--prefer-free-formats` and add `--no-prefer-free-format`
+* [docs] Better document `--prefer-free-formats` and add `--no-prefer-free-format`
 
 
 ### 2021.02.15
@@ -503,7 +1181,7 @@
 * [movefiles] Fix compatibility with python2
 * [remuxvideo] Fix validation of conditional remux
 * [sponskrub] Don't raise error when the video does not exist
-* [documentation] Crypto is an optional dependency
+* [docs] Crypto is an optional dependency
 
 
 ### 2021.02.04
@@ -564,10 +1242,10 @@
 * Merge youtube-dl: Upto [2021.01.24](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
 * Plugin support ([documentation](https://github.com/yt-dlp/yt-dlp#plugins))
 * **Multiple paths**: New option `-P`/`--paths` to give different paths for different types of files
-    * The syntax is `-P "type:path" -P "type:path"` ([documentation](https://github.com/yt-dlp/yt-dlp#:~:text=-P,%20--paths%20TYPE:PATH))
+    * The syntax is `-P "type:path" -P "type:path"`
     * Valid types are: home, temp, description, annotation, subtitle, infojson, thumbnail
-    * Additionally, configuration file is taken from home directory or current directory ([documentation](https://github.com/yt-dlp/yt-dlp#:~:text=Home%20Configuration))
+    * Additionally, configuration file is taken from home directory or current directory
-* Allow passing different arguments to different external downloaders ([documentation](https://github.com/yt-dlp/yt-dlp#:~:text=--downloader-args%20NAME:ARGS))
+* Allow passing different arguments to different external downloaders
 * [mildom] Add extractor by [nao20010128nao](https://github.com/nao20010128nao)
 * Warn when using old style `--external-downloader-args` and `--post-processor-args`
 * Fix `--no-overwrite` when using `--write-link`
@@ -602,9 +1280,9 @@
 * [roosterteeth.com] Fix for bonus episodes by [Zocker1999NET](https://github.com/Zocker1999NET)
 * [tiktok] Fix for when share_info is empty
 * [EmbedThumbnail] Fix bug due to incorrect function name
-* [documentation] Changed sponskrub links to point to [yt-dlp/SponSkrub](https://github.com/yt-dlp/SponSkrub) since I am now providing both linux and windows releases
+* [docs] Changed sponskrub links to point to [yt-dlp/SponSkrub](https://github.com/yt-dlp/SponSkrub) since I am now providing both linux and windows releases
-* [documentation] Change all links to correctly point to new fork URL
+* [docs] Change all links to correctly point to new fork URL
-* [documentation] Fixes typos
+* [docs] Fixes typos
 
 
 ### 2021.01.12
@@ -700,7 +1378,7 @@
     * Redirect channel home to /video
     * Print youtube's warning message
     * Handle Multiple pages for feeds better
-* [youtube] Fix ytsearch not returning results sometimes due to promoted content by [colethedj](https://github.com/colethedj)
+* [youtube] Fix ytsearch not returning results sometimes due to promoted content by [coletdjnz](https://github.com/coletdjnz)
 * [youtube] Temporary fix for automatic captions - disable json3 by [blackjack4494](https://github.com/blackjack4494)
 * Add --break-on-existing by [gergesh](https://github.com/gergesh)
 * Pre-check video IDs in the archive before downloading by [pukkandan](https://github.com/pukkandan)

Collaborators.md

@@ -0,0 +1,39 @@
+# Collaborators
+
+This is a list of the collaborators of the project and their major contributions. See the [Changelog](Changelog.md) for more details.
+
+You can also find lists of all [contributors of yt-dlp](CONTRIBUTORS) and [authors of youtube-dl](https://github.com/ytdl-org/youtube-dl/blob/master/AUTHORS)
+
+
+## [pukkandan](https://github.com/pukkandan)
+
+[![ko-fi](https://img.shields.io/badge/_-Ko--fi-red.svg?logo=kofi&labelColor=555555&style=for-the-badge)](https://ko-fi.com/pukkandan)
+
+* Owner of the fork
+
+
+
+## [shirt](https://github.com/shirt-dev)
+
+[![ko-fi](https://img.shields.io/badge/_-Ko--fi-red.svg?logo=kofi&labelColor=555555&style=for-the-badge)](https://ko-fi.com/shirt)
+
+* Multithreading (`-N`) and aria2c support for fragment downloads
+* Support for media initialization and discontinuity in HLS
+* The self-updater (`-U`)
+
+
+
+## [coletdjnz](https://github.com/coletdjnz)
+
+[![gh-sponsor](https://img.shields.io/badge/_-Sponsor-red.svg?logo=githubsponsors&labelColor=555555&style=for-the-badge)](https://github.com/sponsors/coletdjnz)
+
+* YouTube improvements including: age-gate bypass, private playlists, multiple-clients (to avoid throttling) and a lot of under-the-hood improvements
+
+
+
+## [Ashish0804](https://github.com/Ashish0804)
+
+[![ko-fi](https://img.shields.io/badge/_-Ko--fi-red.svg?logo=kofi&labelColor=555555&style=for-the-badge)](https://ko-fi.com/ashish0804)
+
+* Added support for new websites Zee5, MXPlayer, DiscoveryPlusIndia, ShemarooMe, Utreon etc
+* Added playlist/series downloads for TubiTv, SonyLIV, Voot, HotStar etc

Makefile

@@ -1,4 +1,4 @@
-all: yt-dlp doc pypi-files
+all: lazy-extractors yt-dlp doc pypi-files
 clean: clean-test clean-dist clean-cache
 completions: completion-bash completion-fish completion-zsh
 doc: README.md CONTRIBUTING.md issuetemplates supportedsites
@@ -13,7 +13,9 @@ pypi-files: AUTHORS Changelog.md LICENSE README.md README.txt supportedsites com
 .PHONY: all clean install test tar pypi-files completions ot offlinetest codetest supportedsites
 
 clean-test:
-	rm -rf *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.frag *.frag.urls *.frag.aria2
+	rm -rf *.3gp *.annotations.xml *.ape *.avi *.description *.dump *.flac *.flv *.frag *.frag.aria2 *.frag.urls \
+	*.info.json *.jpeg *.jpg *.live_chat.json *.m4a *.m4v *.mkv *.mp3 *.mp4 *.ogg *.opus *.part* *.png *.sbv *.srt \
+	*.swf *.swp *.ttml *.vtt *.wav *.webm *.webp *.ytdl test/testdata/player-*.js
 clean-dist:
 	rm -rf yt-dlp.1.temp.md yt-dlp.1 README.txt MANIFEST build/ dist/ .coverage cover/ yt-dlp.tar.gz completions/ yt_dlp/extractor/lazy_extractors.py *.spec CONTRIBUTING.md.tmp yt-dlp yt-dlp.exe yt_dlp.egg-info/ AUTHORS .mailmap
 clean-cache:
@@ -38,9 +40,9 @@ SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then ech
 # set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2
 MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi)
 
-install: yt-dlp yt-dlp.1 completions
+install: lazy-extractors yt-dlp yt-dlp.1 completions
-	install -Dm755 yt-dlp $(DESTDIR)$(BINDIR)
+	install -Dm755 yt-dlp $(DESTDIR)$(BINDIR)/yt-dlp
-	install -Dm644 yt-dlp.1 $(DESTDIR)$(MANDIR)/man1
+	install -Dm644 yt-dlp.1 $(DESTDIR)$(MANDIR)/man1/yt-dlp.1
 	install -Dm644 completions/bash/yt-dlp $(DESTDIR)$(SHAREDIR)/bash-completion/completions/yt-dlp
 	install -Dm644 completions/zsh/_yt-dlp $(DESTDIR)$(SHAREDIR)/zsh/site-functions/_yt-dlp
 	install -Dm644 completions/fish/yt-dlp.fish $(DESTDIR)$(SHAREDIR)/fish/vendor_completions.d/yt-dlp.fish
@@ -49,23 +51,11 @@ codetest:
 	flake8 .
 
 test:
-	#nosetests --with-coverage --cover-package=yt_dlp --cover-html --verbose --processes 4 test
+	$(PYTHON) -m pytest
-	nosetests --verbose test
 	$(MAKE) codetest
 
-# Keep this list in sync with devscripts/run_tests.sh
 offlinetest: codetest
-	$(PYTHON) -m nose --verbose test \
+	$(PYTHON) -m pytest -k "not download"
-		--exclude test_age_restriction.py \
-		--exclude test_download.py \
-		--exclude test_iqiyi_sdk_interpreter.py \
-		--exclude test_overwrites.py \
-		--exclude test_socks.py \
-		--exclude test_subtitles.py \
-		--exclude test_write_annotations.py \
-		--exclude test_youtube_lists.py \
-		--exclude test_youtube_signature.py \
-		--exclude test_post_hooks.py
 
 yt-dlp: yt_dlp/*.py yt_dlp/*/*.py
 	mkdir -p zip
@@ -88,12 +78,13 @@ README.md: yt_dlp/*.py yt_dlp/*/*.py
 CONTRIBUTING.md: README.md
 	$(PYTHON) devscripts/make_contributing.py README.md CONTRIBUTING.md
 
-issuetemplates: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.md .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md .github/ISSUE_TEMPLATE_tmpl/4_bug_report.md .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md yt_dlp/version.py
+issuetemplates: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml .github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml .github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml yt_dlp/version.py
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.md .github/ISSUE_TEMPLATE/1_broken_site.md
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml .github/ISSUE_TEMPLATE/1_broken_site.yml
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md .github/ISSUE_TEMPLATE/2_site_support_request.md
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml .github/ISSUE_TEMPLATE/2_site_support_request.yml
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md .github/ISSUE_TEMPLATE/3_site_feature_request.md
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml .github/ISSUE_TEMPLATE/3_site_feature_request.yml
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/4_bug_report.md .github/ISSUE_TEMPLATE/4_bug_report.md
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml .github/ISSUE_TEMPLATE/4_bug_report.yml
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md .github/ISSUE_TEMPLATE/5_feature_request.md
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml .github/ISSUE_TEMPLATE/5_feature_request.yml
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/6_question.yml .github/ISSUE_TEMPLATE/6_question.yml
 
 supportedsites:
 	$(PYTHON) devscripts/make_supportedsites.py supportedsites.md
@@ -122,7 +113,7 @@ _EXTRACTOR_FILES = $(shell find yt_dlp/extractor -iname '*.py' -and -not -iname
 yt_dlp/extractor/lazy_extractors.py: devscripts/make_lazy_extractors.py devscripts/lazy_load_template.py $(_EXTRACTOR_FILES)
 	$(PYTHON) devscripts/make_lazy_extractors.py $@
 
-yt-dlp.tar.gz: README.md yt-dlp.1 completions Changelog.md AUTHORS
+yt-dlp.tar.gz: all
 	@tar -czf $(DESTDIR)/yt-dlp.tar.gz --transform "s|^|yt-dlp/|" --owner 0 --group 0 \
 		--exclude '*.DS_Store' \
 		--exclude '*.kate-swp' \
@@ -131,12 +122,12 @@ yt-dlp.tar.gz: README.md yt-dlp.1 completions Changelog.md AUTHORS
 		--exclude '*~' \
 		--exclude '__pycache__' \
 		--exclude '.git' \
-		--exclude 'docs/_build' \
 		-- \
-		devscripts test \
+		README.md supportedsites.md Changelog.md LICENSE \
-		Changelog.md AUTHORS LICENSE README.md supportedsites.md \
+		CONTRIBUTING.md Collaborators.md CONTRIBUTORS AUTHORS \
-		Makefile MANIFEST.in yt-dlp.1 completions \
+		Makefile MANIFEST.in yt-dlp.1 README.txt completions \
-		setup.py setup.cfg yt-dlp
+		setup.py setup.cfg yt-dlp yt_dlp requirements.txt \
+		devscripts test tox.ini pytest.ini
 
 AUTHORS: .mailmap
 	git shortlog -s -n | cut -f2 | sort > AUTHORS

devscripts/lazy_load_template.py

@@ -1,20 +1,31 @@
-#!/usr/bin/env python3
 # coding: utf-8
-from __future__ import unicode_literals
-
 import re
 
+from ..utils import bug_reports_message, write_string
 
-class LazyLoadExtractor(object):
+
+class LazyLoadMetaClass(type):
+    def __getattr__(cls, name):
+        if '_real_class' not in cls.__dict__:
+            write_string(
+                f'WARNING: Falling back to normal extractor since lazy extractor '
+                f'{cls.__name__} does not have attribute {name}{bug_reports_message()}')
+        return getattr(cls._get_real_class(), name)
+
+
+class LazyLoadExtractor(metaclass=LazyLoadMetaClass):
     _module = None
+    _WORKING = True
 
     @classmethod
-    def ie_key(cls):
+    def _get_real_class(cls):
-        return cls.__name__[:-2]
+        if '_real_class' not in cls.__dict__:
+            mod = __import__(cls._module, fromlist=(cls.__name__,))
+            cls._real_class = getattr(mod, cls.__name__)
+        return cls._real_class
 
     def __new__(cls, *args, **kwargs):
-        mod = __import__(cls._module, fromlist=(cls.__name__,))
+        real_cls = cls._get_real_class()
-        real_cls = getattr(mod, cls.__name__)
         instance = real_cls.__new__(real_cls)
         instance.__init__(*args, **kwargs)
         return instance

devscripts/make_contributing.py

@@ -1,33 +1,34 @@
 #!/usr/bin/env python3
 from __future__ import unicode_literals
 
-# import io
+import io
 import optparse
-# import re
+import re
 
 
 def main():
+    return  # This is unused in yt-dlp
+
     parser = optparse.OptionParser(usage='%prog INFILE OUTFILE')
     options, args = parser.parse_args()
     if len(args) != 2:
         parser.error('Expected an input and an output filename')
 
-
+    infile, outfile = args
-"""     infile, outfile = args
 
     with io.open(infile, encoding='utf-8') as inf:
         readme = inf.read()
 
-    bug_text = re.search( """
+    bug_text = re.search(
-# r'(?s)#\s*BUGS\s*[^\n]*\s*(.*?)#\s*COPYRIGHT', readme).group(1)
+        r'(?s)#\s*BUGS\s*[^\n]*\s*(.*?)#\s*COPYRIGHT', readme).group(1)
-# dev_text = re.search(
+    dev_text = re.search(
-# r'(?s)(#\s*DEVELOPER INSTRUCTIONS.*?)#\s*EMBEDDING yt-dlp',
+        r'(?s)(#\s*DEVELOPER INSTRUCTIONS.*?)#\s*EMBEDDING yt-dlp', readme).group(1)
-"""         readme).group(1)
 
     out = bug_text + dev_text
 
     with io.open(outfile, 'w', encoding='utf-8') as outf:
-        outf.write(out) """
+        outf.write(out)
+
 
 if __name__ == '__main__':
     main()

devscripts/make_lazy_extractors.py

@@ -7,32 +7,35 @@ import os
 from os.path import dirname as dirn
 import sys
 
-print('WARNING: Lazy loading extractors is an experimental feature that may not always work', file=sys.stderr)
-
 sys.path.insert(0, dirn(dirn((os.path.abspath(__file__)))))
 
-lazy_extractors_filename = sys.argv[1]
+lazy_extractors_filename = sys.argv[1] if len(sys.argv) > 1 else 'yt_dlp/extractor/lazy_extractors.py'
 if os.path.exists(lazy_extractors_filename):
     os.remove(lazy_extractors_filename)
 
 # Block plugins from loading
-os.rename('ytdlp_plugins', 'ytdlp_plugins_blocked')
+plugins_dirname = 'ytdlp_plugins'
+plugins_blocked_dirname = 'ytdlp_plugins_blocked'
+if os.path.exists(plugins_dirname):
+    os.rename(plugins_dirname, plugins_blocked_dirname)
 
 from yt_dlp.extractor import _ALL_CLASSES
 from yt_dlp.extractor.common import InfoExtractor, SearchInfoExtractor
 
-os.rename('ytdlp_plugins_blocked', 'ytdlp_plugins')
+if os.path.exists(plugins_blocked_dirname):
+    os.rename(plugins_blocked_dirname, plugins_dirname)
 
 with open('devscripts/lazy_load_template.py', 'rt') as f:
     module_template = f.read()
 
+CLASS_PROPERTIES = ['ie_key', 'working', '_match_valid_url', 'suitable', '_match_id', 'get_temp_id']
 module_contents = [
-    module_template + '\n' + getsource(InfoExtractor.suitable) + '\n',
+    module_template,
-    'class LazyLoadSearchExtractor(LazyLoadExtractor):\n    pass\n']
+    *[getsource(getattr(InfoExtractor, k)) for k in CLASS_PROPERTIES],
+    '\nclass LazyLoadSearchExtractor(LazyLoadExtractor):\n    pass\n']
 
 ie_template = '''
 class {name}({bases}):
-    _VALID_URL = {valid_url!r}
     _module = '{module}'
 '''
 
@@ -53,14 +56,17 @@ def get_base_name(base):
 
 
 def build_lazy_ie(ie, name):
-    valid_url = getattr(ie, '_VALID_URL', None)
     s = ie_template.format(
         name=name,
         bases=', '.join(map(get_base_name, ie.__bases__)),
-        valid_url=valid_url,
         module=ie.__module__)
+    valid_url = getattr(ie, '_VALID_URL', None)
+    if valid_url:
+        s += f'    _VALID_URL = {valid_url!r}\n'
+    if not ie._WORKING:
+        s += '    _WORKING = False\n'
     if ie.suitable.__func__ is not InfoExtractor.suitable.__func__:
-        s += '\n' + getsource(ie.suitable)
+        s += f'\n{getsource(ie.suitable)}'
     if hasattr(ie, '_make_valid_url'):
         # search extractors
         s += make_valid_template.format(valid_url=ie._make_valid_url())
@@ -98,7 +104,7 @@ for ie in ordered_cls:
         names.append(name)
 
 module_contents.append(
-    '_ALL_CLASSES = [{0}]'.format(', '.join(names)))
+    '\n_ALL_CLASSES = [{0}]'.format(', '.join(names)))
 
 module_src = '\n'.join(module_contents) + '\n'
 

devscripts/make_supportedsites.py

@@ -29,6 +29,9 @@ def main():
                 continue
             if ie_desc is not None:
                 ie_md += ': {0}'.format(ie.IE_DESC)
+            search_key = getattr(ie, 'SEARCH_KEY', None)
+            if search_key is not None:
+                ie_md += f'; "{ie.SEARCH_KEY}:" prefix'
             if not ie.working():
                 ie_md += ' (Currently broken)'
             yield ie_md

devscripts/run_tests.bat

@@ -1,17 +1,16 @@
+@setlocal
 @echo off
+cd /d %~dp0..
 
-rem Keep this list in sync with the `offlinetest` target in Makefile
+if ["%~1"]==[""] (
-set DOWNLOAD_TESTS="age_restriction^|download^|iqiyi_sdk_interpreter^|socks^|subtitles^|write_annotations^|youtube_lists^|youtube_signature^|post_hooks"
+    set "test_set="test""
-
+) else if ["%~1"]==["core"] (
-if "%YTDL_TEST_SET%" == "core" (
+    set "test_set="-m not download""
-    set test_set="-I test_("%DOWNLOAD_TESTS%")\.py"
+) else if ["%~1"]==["download"] (
-    set multiprocess_args=""
+    set "test_set="-m "download""
-) else if "%YTDL_TEST_SET%" == "download" (
-    set test_set="-I test_(?!"%DOWNLOAD_TESTS%").+\.py"
-    set multiprocess_args="--processes=4 --process-timeout=540"
 ) else (
-    echo YTDL_TEST_SET is not set or invalid
+    echo.Invalid test type "%~1". Use "core" ^| "download"
     exit /b 1
 )
 
-nosetests test --verbose %test_set:"=% %multiprocess_args:"=%
+pytest %test_set%

devscripts/run_tests.sh

@@ -1,22 +1,14 @@
-#!/bin/bash
+#!/bin/sh
 
-# Keep this list in sync with the `offlinetest` target in Makefile
+if [ -z $1 ]; then
-DOWNLOAD_TESTS="age_restriction|download|iqiyi_sdk_interpreter|overwrites|socks|subtitles|write_annotations|youtube_lists|youtube_signature|post_hooks"
+    test_set='test'
+elif [ $1 = 'core' ]; then
+    test_set="-m not download"
+elif [ $1 = 'download' ]; then
+    test_set="-m download"
+else
+    echo 'Invalid test type "'$1'". Use "core" | "download"'
+    exit 1
+fi
 
-test_set=""
+python3 -m pytest "$test_set"
-multiprocess_args=""
-
-case "$YTDL_TEST_SET" in
-    core)
-        test_set="-I test_($DOWNLOAD_TESTS)\.py"
-    ;;
-    download)
-        test_set="-I test_(?!$DOWNLOAD_TESTS).+\.py"
-        multiprocess_args="--processes=4 --process-timeout=540"
-    ;;
-    *)
-        break
-    ;;
-esac
-
-nosetests test --verbose $test_set $multiprocess_args

devscripts/update-formulae.py

@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+from __future__ import unicode_literals
+
+import json
+import os
+import re
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from yt_dlp.compat import compat_urllib_request
+
+
+# usage: python3 ./devscripts/update-formulae.py <path-to-formulae-rb> <version>
+# version can be either 0-aligned (yt-dlp version) or normalized (PyPl version)
+
+filename, version = sys.argv[1:]
+
+normalized_version = '.'.join(str(int(x)) for x in version.split('.'))
+
+pypi_release = json.loads(compat_urllib_request.urlopen(
+    'https://pypi.org/pypi/yt-dlp/%s/json' % normalized_version
+).read().decode('utf-8'))
+
+tarball_file = next(x for x in pypi_release['urls'] if x['filename'].endswith('.tar.gz'))
+
+sha256sum = tarball_file['digests']['sha256']
+url = tarball_file['url']
+
+with open(filename, 'r') as r:
+    formulae_text = r.read()
+
+formulae_text = re.sub(r'sha256 "[0-9a-f]*?"', 'sha256 "%s"' % sha256sum, formulae_text)
+formulae_text = re.sub(r'url "[^"]*?"', 'url "%s"' % url, formulae_text)
+
+with open(filename, 'w') as w:
+    w.write(formulae_text)

docs/Collaborators.md

@@ -0,0 +1,5 @@
+---
+orphan: true
+---
+```{include} ../Collaborators.md
+```

pyinst.py

@@ -1,82 +1,135 @@
 #!/usr/bin/env python3
 # coding: utf-8
-
+import os
-from __future__ import unicode_literals
-import sys
-# import os
 import platform
-
+import sys
 from PyInstaller.utils.hooks import collect_submodules
-from PyInstaller.utils.win32.versioninfo import (
-    VarStruct, VarFileInfo, StringStruct, StringTable,
-    StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
-)
-import PyInstaller.__main__
 
-arch = sys.argv[1] if len(sys.argv) > 1 else platform.architecture()[0][:2]
-assert arch in ('32', '64')
-print('Building %sbit version' % arch)
-_x86 = '_x86' if arch == '32' else ''
 
-FILE_DESCRIPTION = 'yt-dlp%s' % (' (32 Bit)' if _x86 else '')
+OS_NAME = platform.system()
+if OS_NAME == 'Windows':
+    from PyInstaller.utils.win32.versioninfo import (
+        VarStruct, VarFileInfo, StringStruct, StringTable,
+        StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
+    )
+elif OS_NAME == 'Darwin':
+    pass
+else:
+    raise Exception('{OS_NAME} is not supported')
 
-# root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
+ARCH = platform.architecture()[0][:2]
-# print('Changing working directory to %s' % root_dir)
-# os.chdir(root_dir)
 
-exec(compile(open('yt_dlp/version.py').read(), 'yt_dlp/version.py', 'exec'))
-VERSION = locals()['__version__']
 
-VERSION_LIST = VERSION.split('.')
+def main():
-VERSION_LIST = list(map(int, VERSION_LIST)) + [0] * (4 - len(VERSION_LIST))
+    opts = parse_options()
+    version = read_version()
 
-print('Version: %s%s' % (VERSION, _x86))
+    suffix = '_macos' if OS_NAME == 'Darwin' else '_x86' if ARCH == '32' else ''
-print('Remember to update the version using devscipts\\update-version.py')
+    final_file = 'dist/%syt-dlp%s%s' % (
+        'yt-dlp/' if '--onedir' in opts else '', suffix, '.exe' if OS_NAME == 'Windows' else '')
 
-VERSION_FILE = VSVersionInfo(
+    print(f'Building yt-dlp v{version} {ARCH}bit for {OS_NAME} with options {opts}')
-    ffi=FixedFileInfo(
+    print('Remember to update the version using  "devscripts/update-version.py"')
-        filevers=VERSION_LIST,
+    if not os.path.isfile('yt_dlp/extractor/lazy_extractors.py'):
-        prodvers=VERSION_LIST,
+        print('WARNING: Building without lazy_extractors. Run  '
-        mask=0x3F,
+              '"devscripts/make_lazy_extractors.py"  to build lazy extractors', file=sys.stderr)
-        flags=0x0,
+    print(f'Destination: {final_file}\n')
-        OS=0x4,
+
-        fileType=0x1,
+    opts = [
-        subtype=0x0,
+        f'--name=yt-dlp{suffix}',
-        date=(0, 0),
+        '--icon=devscripts/logo.ico',
-    ),
+        '--upx-exclude=vcruntime140.dll',
-    kids=[
+        '--noconfirm',
-        StringFileInfo([
+        *dependancy_options(),
-            StringTable(
+        *opts,
-                '040904B0', [
+        'yt_dlp/__main__.py',
-                    StringStruct('Comments', 'yt-dlp%s Command Line Interface.' % _x86),
-                    StringStruct('CompanyName', 'https://github.com/yt-dlp'),
-                    StringStruct('FileDescription', FILE_DESCRIPTION),
-                    StringStruct('FileVersion', VERSION),
-                    StringStruct('InternalName', 'yt-dlp%s' % _x86),
-                    StringStruct(
-                        'LegalCopyright',
-                        'pukkandan.ytdlp@gmail.com | UNLICENSE',
-                    ),
-                    StringStruct('OriginalFilename', 'yt-dlp%s.exe' % _x86),
-                    StringStruct('ProductName', 'yt-dlp%s' % _x86),
-                    StringStruct(
-                        'ProductVersion',
-                        '%s%s on Python %s' % (VERSION, _x86, platform.python_version())),
-                ])]),
-        VarFileInfo([VarStruct('Translation', [0, 1200])])
     ]
-)
+    print(f'Running PyInstaller with {opts}')
 
-dependancies = ['Crypto', 'mutagen'] + collect_submodules('websockets')
+    import PyInstaller.__main__
-excluded_modules = ['test', 'ytdlp_plugins', 'youtube-dl', 'youtube-dlc']
 
-PyInstaller.__main__.run([
+    PyInstaller.__main__.run(opts)
-    '--name=yt-dlp%s' % _x86,
+
-    '--onefile',
+    set_version_info(final_file, version)
-    '--icon=devscripts/cloud.ico',
+
-    *[f'--exclude-module={module}' for module in excluded_modules],
+
-    *[f'--hidden-import={module}' for module in dependancies],
+def parse_options():
-    '--upx-exclude=vcruntime140.dll',
+    # Compatability with older arguments
-    'yt_dlp/__main__.py',
+    opts = sys.argv[1:]
-])
+    if opts[0:1] in (['32'], ['64']):
-SetVersion('dist/yt-dlp%s.exe' % _x86, VERSION_FILE)
+        if ARCH != opts[0]:
+            raise Exception(f'{opts[0]}bit executable cannot be built on a {ARCH}bit system')
+        opts = opts[1:]
+    return opts or ['--onefile']
+
+
+def read_version():
+    exec(compile(open('yt_dlp/version.py').read(), 'yt_dlp/version.py', 'exec'))
+    return locals()['__version__']
+
+
+def version_to_list(version):
+    version_list = version.split('.')
+    return list(map(int, version_list)) + [0] * (4 - len(version_list))
+
+
+def dependancy_options():
+    dependancies = [pycryptodome_module(), 'mutagen'] + collect_submodules('websockets')
+    excluded_modules = ['test', 'ytdlp_plugins', 'youtube-dl', 'youtube-dlc']
+
+    yield from (f'--hidden-import={module}' for module in dependancies)
+    yield from (f'--exclude-module={module}' for module in excluded_modules)
+
+
+def pycryptodome_module():
+    try:
+        import Cryptodome  # noqa: F401
+    except ImportError:
+        try:
+            import Crypto  # noqa: F401
+            print('WARNING: Using Crypto since Cryptodome is not available. '
+                  'Install with: pip install pycryptodomex', file=sys.stderr)
+            return 'Crypto'
+        except ImportError:
+            pass
+    return 'Cryptodome'
+
+
+def set_version_info(exe, version):
+    if OS_NAME == 'Windows':
+        windows_set_version(exe, version)
+
+
+def windows_set_version(exe, version):
+    version_list = version_to_list(version)
+    suffix = '_x86' if ARCH == '32' else ''
+    SetVersion(exe, VSVersionInfo(
+        ffi=FixedFileInfo(
+            filevers=version_list,
+            prodvers=version_list,
+            mask=0x3F,
+            flags=0x0,
+            OS=0x4,
+            fileType=0x1,
+            subtype=0x0,
+            date=(0, 0),
+        ),
+        kids=[
+            StringFileInfo([StringTable('040904B0', [
+                StringStruct('Comments', 'yt-dlp%s Command Line Interface.' % suffix),
+                StringStruct('CompanyName', 'https://github.com/yt-dlp'),
+                StringStruct('FileDescription', 'yt-dlp%s' % (' (32 Bit)' if ARCH == '32' else '')),
+                StringStruct('FileVersion', version),
+                StringStruct('InternalName', f'yt-dlp{suffix}'),
+                StringStruct('LegalCopyright', 'pukkandan.ytdlp@gmail.com | UNLICENSE'),
+                StringStruct('OriginalFilename', f'yt-dlp{suffix}.exe'),
+                StringStruct('ProductName', f'yt-dlp{suffix}'),
+                StringStruct(
+                    'ProductVersion', f'{version}{suffix} on Python {platform.python_version()}'),
+            ])]), VarFileInfo([VarStruct('Translation', [0, 1200])])
+        ]
+    ))
+
+
+if __name__ == '__main__':
+    main()

pytest.ini

@@ -0,0 +1,4 @@
+[pytest]
+addopts = -ra -v --strict-markers
+markers =
+    download

requirements.txt

@@ -1,3 +1,3 @@
 mutagen
-pycryptodome
+pycryptodomex
 websockets

setup.py

@@ -1,52 +1,86 @@
 #!/usr/bin/env python3
 # coding: utf-8
-
-from setuptools import setup, Command, find_packages
 import os.path
 import warnings
 import sys
-from distutils.spawn import spawn
 
+try:
+    from setuptools import setup, Command, find_packages
+    setuptools_available = True
+except ImportError:
+    from distutils.core import setup, Command
+    setuptools_available = False
+from distutils.spawn import spawn
 
 # Get the version from yt_dlp/version.py without importing the package
 exec(compile(open('yt_dlp/version.py').read(), 'yt_dlp/version.py', 'exec'))
 
 
-DESCRIPTION = 'Command-line program to download videos from YouTube.com and many other other video platforms.'
+DESCRIPTION = 'A youtube-dl fork with additional features and patches'
 
 LONG_DESCRIPTION = '\n\n'.join((
     'Official repository: <https://github.com/yt-dlp/yt-dlp>',
     '**PS**: Some links in this document will not work since this is a copy of the README.md from Github',
     open('README.md', 'r', encoding='utf-8').read()))
 
-REQUIREMENTS = ['mutagen', 'pycryptodome', 'websockets']
+REQUIREMENTS = ['mutagen', 'pycryptodomex', 'websockets']
+
 
 if sys.argv[1:2] == ['py2exe']:
-    raise NotImplementedError('py2exe is not currently supported; instead, use "pyinst.py" to build with pyinstaller')
+    import py2exe
+    warnings.warn(
+        'py2exe builds do not support pycryptodomex and needs VC++14 to run. '
+        'The recommended way is to use "pyinst.py" to build using pyinstaller')
+    params = {
+        'console': [{
+            'script': './yt_dlp/__main__.py',
+            'dest_base': 'yt-dlp',
+            'version': __version__,
+            'description': DESCRIPTION,
+            'comments': LONG_DESCRIPTION.split('\n')[0],
+            'product_name': 'yt-dlp',
+            'product_version': __version__,
+        }],
+        'options': {
+            'py2exe': {
+                'bundle_files': 0,
+                'compressed': 1,
+                'optimize': 2,
+                'dist_dir': './dist',
+                'excludes': ['Crypto', 'Cryptodome'],  # py2exe cannot import Crypto
+                'dll_excludes': ['w9xpopen.exe', 'crypt32.dll'],
+            }
+        },
+        'zipfile': None
+    }
 
+else:
+    files_spec = [
+        ('share/bash-completion/completions', ['completions/bash/yt-dlp']),
+        ('share/zsh/site-functions', ['completions/zsh/_yt-dlp']),
+        ('share/fish/vendor_completions.d', ['completions/fish/yt-dlp.fish']),
+        ('share/doc/yt_dlp', ['README.txt']),
+        ('share/man/man1', ['yt-dlp.1'])
+    ]
+    root = os.path.dirname(os.path.abspath(__file__))
+    data_files = []
+    for dirname, files in files_spec:
+        resfiles = []
+        for fn in files:
+            if not os.path.exists(fn):
+                warnings.warn('Skipping file %s since it is not present. Try running `make pypi-files` first' % fn)
+            else:
+                resfiles.append(fn)
+        data_files.append((dirname, resfiles))
 
-files_spec = [
+    params = {
-    ('share/bash-completion/completions', ['completions/bash/yt-dlp']),
+        'data_files': data_files,
-    ('share/zsh/site-functions', ['completions/zsh/_yt-dlp']),
+    }
-    ('share/fish/vendor_completions.d', ['completions/fish/yt-dlp.fish']),
-    ('share/doc/yt_dlp', ['README.txt']),
-    ('share/man/man1', ['yt-dlp.1'])
-]
-root = os.path.dirname(os.path.abspath(__file__))
-data_files = []
-for dirname, files in files_spec:
-    resfiles = []
-    for fn in files:
-        if not os.path.exists(fn):
-            warnings.warn('Skipping file %s since it is not present. Try running `make pypi-files` first' % fn)
-        else:
-            resfiles.append(fn)
-    data_files.append((dirname, resfiles))
 
-params = {
+    if setuptools_available:
-    'data_files': data_files,
+        params['entry_points'] = {'console_scripts': ['yt-dlp = yt_dlp:main']}
-}
+    else:
-params['entry_points'] = {'console_scripts': ['yt-dlp = yt_dlp:main']}
+        params['scripts'] = ['yt-dlp']
 
 
 class build_lazy_extractors(Command):
@@ -64,7 +98,11 @@ class build_lazy_extractors(Command):
               dry_run=self.dry_run)
 
 
-packages = find_packages(exclude=('youtube_dl', 'test', 'ytdlp_plugins'))
+if setuptools_available:
+    packages = find_packages(exclude=('youtube_dl', 'youtube_dlc', 'test', 'ytdlp_plugins'))
+else:
+    packages = ['yt_dlp', 'yt_dlp.downloader', 'yt_dlp.extractor', 'yt_dlp.postprocessor']
+
 
 setup(
     name='yt-dlp',
@@ -81,7 +119,7 @@ setup(
         'Documentation': 'https://yt-dlp.readthedocs.io',
         'Source': 'https://github.com/yt-dlp/yt-dlp',
         'Tracker': 'https://github.com/yt-dlp/yt-dlp/issues',
-        #'Funding': 'https://donate.pypi.org',
+        'Funding': 'https://github.com/yt-dlp/yt-dlp/blob/master/Collaborators.md#collaborators',
     },
     classifiers=[
         'Topic :: Multimedia :: Video',

supportedsites.md

@@ -1,4 +1,6 @@
 # Supported sites
+ - **17live**
+ - **17live:clip**
  - **1tv**: Первый канал
  - **20min**
  - **220.ro**
@@ -46,10 +48,12 @@
  - **Alura**
  - **AluraCourse**
  - **Amara**
+ - **AmazonStore**
  - **AMCNetworks**
  - **AmericasTestKitchen**
  - **AmericasTestKitchenSeason**
  - **anderetijden**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
+ - **AnimalPlanet**
  - **AnimeLab**
  - **AnimeLabShows**
  - **AnimeOnDemand**
@@ -95,7 +99,9 @@
  - **Bandcamp**
  - **Bandcamp:album**
  - **Bandcamp:weekly**
+ - **BandcampMusic**
  - **bangumi.bilibili.com**: BiliBili番剧
+ - **BannedVideo**
  - **bbc**: BBC
  - **bbc.co.uk**: BBC iPlayer
  - **bbc.co.uk:article**: BBC articles
@@ -117,11 +123,14 @@
  - **Bigflix**
  - **Bild**: Bild.de
  - **BiliBili**
+ - **Bilibili category extractor**
  - **BilibiliAudio**
  - **BilibiliAudioAlbum**
  - **BilibiliChannel**
  - **BiliBiliPlayer**
- - **BiliBiliSearch**: Bilibili video search, "bilisearch" keyword
+ - **BiliBiliSearch**: Bilibili video search; "bilisearch:" prefix
+ - **BiliIntl**
+ - **BiliIntlSeries**
  - **BioBioChileTV**
  - **Biography**
  - **BIQLE**
@@ -129,6 +138,7 @@
  - **BitChuteChannel**
  - **bitwave:replay**
  - **bitwave:stream**
+ - **BlackboardCollaborate**
  - **BleacherReport**
  - **BleacherReportCMS**
  - **Bloomberg**
@@ -148,10 +158,10 @@
  - **BusinessInsider**
  - **BuzzFeed**
  - **BYUtv**
+ - **CAM4**
  - **Camdemy**
  - **CamdemyFolder**
  - **CamModels**
- - **CamTube**
  - **CamWithHer**
  - **canalc2.tv**
  - **Canalplus**: mycanal.fr and piwiplus.fr
@@ -161,10 +171,7 @@
  - **CarambaTVPage**
  - **CartoonNetwork**
  - **cbc.ca**
- - **cbc.ca:olympics**
  - **cbc.ca:player**
- - **cbc.ca:watch**
- - **cbc.ca:watch:video**
  - **CBS**
  - **CBSInteractive**
  - **CBSLocal**
@@ -178,11 +185,13 @@
  - **CCTV**: 央视网
  - **CDA**
  - **CeskaTelevize**
- - **CeskaTelevizePorady**
+ - **CGTN**
  - **channel9**: Channel 9
  - **CharlieRose**
  - **Chaturbate**
  - **Chilloutzone**
+ - **Chingari**
+ - **ChingariUser**
  - **chirbit**
  - **chirbit:profile**
  - **cielotv.it**
@@ -190,6 +199,7 @@
  - **Cinemax**
  - **CiscoLiveSearch**
  - **CiscoLiveSession**
+ - **ciscowebex**: Cisco Webex
  - **CJSW**
  - **cliphunter**
  - **Clippit**
@@ -216,13 +226,14 @@
  - **Crackle**
  - **CrooksAndLiars**
  - **crunchyroll**
+ - **crunchyroll:beta**
  - **crunchyroll:playlist**
+ - **crunchyroll:playlist:beta**
  - **CSpan**: C-SPAN
  - **CtsNews**: 華視新聞
  - **CTV**
  - **CTVNews**
  - **cu.ntv.co.jp**: Nippon Television Network
- - **Culturebox**
  - **CultureUnplugged**
  - **curiositystream**
  - **curiositystream:collection**
@@ -232,6 +243,8 @@
  - **dailymotion**
  - **dailymotion:playlist**
  - **dailymotion:user**
+ - **damtomo:record**
+ - **damtomo:video**
  - **daum.net**
  - **daum.net:clip**
  - **daum.net:playlist**
@@ -255,10 +268,12 @@
  - **DiscoveryPlusIndiaShow**
  - **DiscoveryVR**
  - **Disney**
+ - **DIYNetwork**
  - **dlive:stream**
  - **dlive:vod**
  - **DoodStream**
  - **Dotsub**
+ - **Douyin**
  - **DouyuShow**
  - **DouyuTV**: 斗鱼
  - **DPlay**
@@ -292,13 +307,17 @@
  - **Embedly**
  - **EMPFlix**
  - **Engadget**
+ - **Epicon**
+ - **EpiconSeries**
  - **Eporner**
  - **EroProfile**
+ - **EroProfile:album**
  - **Escapist**
  - **ESPN**
  - **ESPNArticle**
  - **EsriVideo**
  - **Europa**
+ - **EUScreen**
  - **EWETV**
  - **ExpoTV**
  - **Expressen**
@@ -306,11 +325,13 @@
  - **EyedoTV**
  - **facebook**
  - **FacebookPluginsVideo**
+ - **fancode:live**
  - **fancode:vod**
  - **faz.net**
  - **fc2**
  - **fc2:embed**
  - **Fczenit**
+ - **Filmmodu**
  - **filmon**
  - **filmon:channel**
  - **Filmweb**
@@ -327,13 +348,10 @@
  - **foxnews**: Fox News and Fox Business Video
  - **foxnews:article**
  - **FoxSports**
- - **france2.fr:generation-what**
  - **FranceCulture**
  - **FranceInter**
  - **FranceTV**
- - **FranceTVEmbed**
  - **francetvinfo.fr**
- - **FranceTVJeunesse**
  - **FranceTVSite**
  - **Freesound**
  - **freespeech.org**
@@ -343,9 +361,13 @@
  - **FrontendMastersLesson**
  - **FujiTVFODPlus7**
  - **Funimation**
+ - **funimation:page**
+ - **funimation:show**
  - **Funk**
  - **Fusion**
  - **Fux**
+ - **Gab**
+ - **GabTV**
  - **Gaia**
  - **GameInformer**
  - **GameSpot**
@@ -354,7 +376,11 @@
  - **Gazeta**
  - **GDCVault**
  - **GediDigital**
+ - **gem.cbc.ca**
+ - **gem.cbc.ca:live**
+ - **gem.cbc.ca:playlist**
  - **generic**: Generic downloader that works on some sites
+ - **Gettr**
  - **Gfycat**
  - **GiantBomb**
  - **Giga**
@@ -368,8 +394,11 @@
  - **google:podcasts**
  - **google:podcasts:feed**
  - **GoogleDrive**
+ - **GoPro**
  - **Goshgay**
+ - **GoToStage**
  - **GPUTechConf**
+ - **Gronkh**
  - **Groupon**
  - **hbo**
  - **HearThisAt**
@@ -401,6 +430,7 @@
  - **Huajiao**: 花椒直播
  - **HuffPost**: Huffington Post
  - **Hungama**
+ - **HungamaAlbumPlaylist**
  - **HungamaSong**
  - **Hypem**
  - **ign.com**
@@ -420,9 +450,11 @@
  - **Instagram**
  - **instagram:tag**: Instagram hashtag search
  - **instagram:user**: Instagram user profile
+ - **InstagramIOS**: IOS instagram:// URL
  - **Internazionale**
  - **InternetVideoArchive**
  - **IPrima**
+ - **IPrimaCNN**
  - **iqiyi**: 爱奇艺
  - **Ir90Tv**
  - **ITTF**
@@ -453,6 +485,7 @@
  - **KinjaEmbed**
  - **KinoPoisk**
  - **KonserthusetPlay**
+ - **Koo**
  - **KrasView**: Красвью
  - **Ku6**
  - **KUSI**
@@ -513,6 +546,9 @@
  - **MallTV**
  - **mangomolo:live**
  - **mangomolo:video**
+ - **ManotoTV**: Manoto TV (Episode)
+ - **ManotoTVLive**: Manoto TV (Live)
+ - **ManotoTVShow**: Manoto TV (Show)
  - **ManyVids**
  - **MaoriTV**
  - **Markiza**
@@ -523,8 +559,11 @@
  - **MedalTV**
  - **media.ccc.de**
  - **media.ccc.de:lists**
+ - **Mediaite**
+ - **MediaKlikk**
  - **Medialaan**
  - **Mediaset**
+ - **MediasetShow**
  - **Mediasite**
  - **MediasiteCatalog**
  - **MediasiteNamedCatalog**
@@ -539,6 +578,7 @@
  - **Mgoon**
  - **MGTV**: 芒果TV
  - **MiaoPai**
+ - **microsoftstream**: Microsoft Stream
  - **mildom**: Record ongoing live by specific user in Mildom
  - **mildom:user:vod**: Download all VODs from specific user in Mildom
  - **mildom:vod**: Download a VOD in Mildom
@@ -548,12 +588,15 @@
  - **MinistryGrid**
  - **Minoto**
  - **miomio.tv**
+ - **mirrativ**
+ - **mirrativ:user**
  - **MiTele**: mitele.es
  - **mixcloud**
  - **mixcloud:playlist**
  - **mixcloud:user**
  - **MLB**
  - **MLBVideo**
+ - **MLSSoccer**
  - **Mnet**
  - **MNetTV**
  - **MoeVideo**: LetitBit video services: moevideo.net, playreplay.net and videochart.net
@@ -579,6 +622,7 @@
  - **mtvservices:embedded**
  - **MTVUutisetArticle**
  - **MuenchenTV**: münchen.tv
+ - **MuseScore**
  - **mva**: Microsoft Virtual Academy videos
  - **mva:course**: Microsoft Virtual Academy courses
  - **Mwave**
@@ -595,6 +639,8 @@
  - **MyviEmbed**
  - **MyVisionTV**
  - **n-tv.de**
+ - **N1Info:article**
+ - **N1InfoAsset**
  - **natgeo:video**
  - **NationalGeographicTV**
  - **Naver**
@@ -628,7 +674,8 @@
  - **NetPlus**
  - **Netzkino**
  - **Newgrounds**
- - **NewgroundsPlaylist**
+ - **Newgrounds:playlist**
+ - **Newgrounds:user**
  - **Newstube**
  - **NextMedia**: 蘋果日報
  - **NextMediaActionNews**: 蘋果日報 - 動新聞
@@ -649,6 +696,9 @@
  - **niconico**: ニコニコ動画
  - **NiconicoPlaylist**
  - **NiconicoUser**
+ - **nicovideo:search**: Nico video searches; "nicosearch:" prefix
+ - **nicovideo:search:date**: Nico video searches, newest first; "nicosearchdate:" prefix
+ - **nicovideo:search_url**: Nico video search URLs
  - **Nintendo**
  - **Nitter**
  - **njoy**: N-JOY
@@ -661,6 +711,7 @@
  - **NosVideo**
  - **Nova**: TN.cz, Prásk.tv, Nova.cz, Novaplus.cz, FANDA.tv, Krásná.cz and Doma.cz
  - **NovaEmbed**
+ - **NovaPlay**
  - **nowness**
  - **nowness:playlist**
  - **nowness:series**
@@ -686,11 +737,14 @@
  - **NYTimes**
  - **NYTimesArticle**
  - **NYTimesCooking**
+ - **nzherald**
  - **NZZ**
  - **ocw.mit.edu**
  - **OdaTV**
  - **Odnoklassniki**
  - **OktoberfestTV**
+ - **OlympicsReplay**
+ - **on24**: ON24
  - **OnDemandKorea**
  - **onet.pl**
  - **onet.tv**
@@ -699,6 +753,8 @@
  - **OnionStudios**
  - **Ooyala**
  - **OoyalaExternal**
+ - **openrec**
+ - **openrec:capture**
  - **OraTV**
  - **orf:burgenland**: Radio Burgenland
  - **orf:fm4**: radio FM4
@@ -724,12 +780,18 @@
  - **PalcoMP3:video**
  - **pandora.tv**: 판도라TV
  - **ParamountNetwork**
+ - **ParamountPlus**
+ - **ParamountPlusSeries**
  - **parliamentlive.tv**: UK parliament videos
  - **Parlview**
  - **Patreon**
+ - **PatreonUser**
  - **pbs**: Public Broadcasting Service (PBS) and member stations: PBS: Public Broadcasting Service, APT - Alabama Public Television (WBIQ), GPB/Georgia Public Broadcasting (WGTV), Mississippi Public Broadcasting (WMPN), Nashville Public Television (WNPT), WFSU-TV (WFSU), WSRE (WSRE), WTCI (WTCI), WPBA/Channel 30 (WPBA), Alaska Public Media (KAKM), Arizona PBS (KAET), KNME-TV/Channel 5 (KNME), Vegas PBS (KLVX), AETN/ARKANSAS ETV NETWORK (KETS), KET (WKLE), WKNO/Channel 10 (WKNO), LPB/LOUISIANA PUBLIC BROADCASTING (WLPB), OETA (KETA), Ozarks Public Television (KOZK), WSIU Public Broadcasting (WSIU), KEET TV (KEET), KIXE/Channel 9 (KIXE), KPBS San Diego (KPBS), KQED (KQED), KVIE Public Television (KVIE), PBS SoCal/KOCE (KOCE), ValleyPBS (KVPT), CONNECTICUT PUBLIC TELEVISION (WEDH), KNPB Channel 5 (KNPB), SOPTV (KSYS), Rocky Mountain PBS (KRMA), KENW-TV3 (KENW), KUED Channel 7 (KUED), Wyoming PBS (KCWC), Colorado Public Television / KBDI 12 (KBDI), KBYU-TV (KBYU), Thirteen/WNET New York (WNET), WGBH/Channel 2 (WGBH), WGBY (WGBY), NJTV Public Media NJ (WNJT), WLIW21 (WLIW), mpt/Maryland Public Television (WMPB), WETA Television and Radio (WETA), WHYY (WHYY), PBS 39 (WLVT), WVPT - Your Source for PBS and More! (WVPT), Howard University Television (WHUT), WEDU PBS (WEDU), WGCU Public Media (WGCU), WPBT2 (WPBT), WUCF TV (WUCF), WUFT/Channel 5 (WUFT), WXEL/Channel 42 (WXEL), WLRN/Channel 17 (WLRN), WUSF Public Broadcasting (WUSF), ETV (WRLK), UNC-TV (WUNC), PBS Hawaii - Oceanic Cable Channel 10 (KHET), Idaho Public Television (KAID), KSPS (KSPS), OPB (KOPB), KWSU/Channel 10 & KTNW/Channel 31 (KWSU), WILL-TV (WILL), Network Knowledge - WSEC/Springfield (WSEC), WTTW11 (WTTW), Iowa Public Television/IPTV (KDIN), Nine Network (KETC), PBS39 Fort Wayne (WFWA), WFYI Indianapolis (WFYI), Milwaukee Public Television (WMVS), WNIN (WNIN), WNIT Public Television (WNIT), WPT (WPNE), WVUT/Channel 22 (WVUT), WEIU/Channel 51 (WEIU), WQPT-TV (WQPT), WYCC PBS Chicago (WYCC), WIPB-TV (WIPB), WTIU (WTIU), CET  (WCET), ThinkTVNetwork (WPTD), WBGU-TV (WBGU), WGVU TV (WGVU), NET1 (KUON), Pioneer Public Television (KWCM), SDPB Television (KUSD), TPT (KTCA), KSMQ (KSMQ), KPTS/Channel 8 (KPTS), KTWU/Channel 11 (KTWU), East Tennessee PBS (WSJK), WCTE-TV (WCTE), WLJT, Channel 11 (WLJT), WOSU TV (WOSU), WOUB/WOUC (WOUB), WVPB (WVPB), WKYU-PBS (WKYU), KERA 13 (KERA), MPBN (WCBB), Mountain Lake PBS (WCFE), NHPTV (WENH), Vermont PBS (WETK), witf (WITF), WQED Multimedia (WQED), WMHT Educational Telecommunications (WMHT), Q-TV (WDCQ), WTVS Detroit Public TV (WTVS), CMU Public Television (WCMU), WKAR-TV (WKAR), WNMU-TV Public TV 13 (WNMU), WDSE - WRPT (WDSE), WGTE TV (WGTE), Lakeland Public Television (KAWE), KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS), MontanaPBS (KUSM), KRWG/Channel 22 (KRWG), KACV (KACV), KCOS/Channel 13 (KCOS), WCNY/Channel 24 (WCNY), WNED (WNED), WPBS (WPBS), WSKG Public TV (WSKG), WXXI (WXXI), WPSU (WPSU), WVIA Public Media Studios (WVIA), WTVI (WTVI), Western Reserve PBS (WNEO), WVIZ/PBS ideastream (WVIZ), KCTS 9 (KCTS), Basin PBS (KPBT), KUHT / Channel 8 (KUHT), KLRN (KLRN), KLRU (KLRU), WTJX Channel 12 (WTJX), WCVE PBS (WCVE), KBTC Public Television (KBTC)
  - **PearVideo**
  - **PeerTube**
+ - **PeerTube:Playlist**
+ - **peloton**
+ - **peloton:live**: Peloton Live
  - **People**
  - **PerformGroup**
  - **periscope**: Periscope
@@ -744,6 +806,7 @@
  - **Pinterest**
  - **PinterestCollection**
  - **Pladform**
+ - **PlanetMarathi**
  - **Platzi**
  - **PlatziCourse**
  - **play.fm**
@@ -760,15 +823,22 @@
  - **podomatic**
  - **Pokemon**
  - **PokemonWatch**
+ - **PolsatGo**
  - **PolskieRadio**
+ - **polskieradio:kierowcow**
+ - **polskieradio:player**
+ - **polskieradio:podcast**
+ - **polskieradio:podcast:list**
  - **PolskieRadioCategory**
  - **Popcorntimes**
  - **PopcornTV**
  - **PornCom**
  - **PornerBros**
+ - **PornFlip**
  - **PornHd**
  - **PornHub**: PornHub and Thumbzilla
  - **PornHubPagedVideoList**
+ - **PornHubPlaylist**
  - **PornHubUser**
  - **PornHubUserVideosUpload**
  - **Pornotube**
@@ -776,6 +846,7 @@
  - **PornoXO**
  - **PornTube**
  - **PressTV**
+ - **ProjectVeritas**
  - **prosiebensat1**: ProSiebenSat.1 Digital
  - **puhutv**
  - **puhutv:serie**
@@ -792,22 +863,34 @@
  - **QuicklineLive**
  - **R7**
  - **R7Article**
+ - **Radiko**
+ - **RadikoRadio**
  - **radio.de**
  - **radiobremen**
  - **radiocanada**
  - **radiocanada:audiovideo**
  - **radiofrance**
  - **RadioJavan**
+ - **radiokapital**
+ - **radiokapital:show**
+ - **radlive**
+ - **radlive:channel**
+ - **radlive:season**
  - **Rai**
  - **RaiPlay**
  - **RaiPlayLive**
  - **RaiPlayPlaylist**
+ - **RaiPlayRadio**
+ - **RaiPlayRadioPlaylist**
  - **RayWenderlich**
  - **RayWenderlichCourse**
  - **RBMARadio**
  - **RCS**
  - **RCSEmbeds**
  - **RCSVarious**
+ - **RCTIPlus**
+ - **RCTIPlusSeries**
+ - **RCTIPlusTV**
  - **RDS**: RDS.ca
  - **RedBull**
  - **RedBullEmbed**
@@ -826,6 +909,7 @@
  - **RMCDecouverte**
  - **RockstarGames**
  - **RoosterTeeth**
+ - **RoosterTeethSeries**
  - **RottenTomatoes**
  - **Roxwel**
  - **Rozhlas**
@@ -845,6 +929,7 @@
  - **RTVNH**
  - **RTVS**
  - **RUHD**
+ - **RumbleChannel**
  - **RumbleEmbed**
  - **rutube**: Rutube videos
  - **rutube:channel**: Rutube channels
@@ -866,7 +951,8 @@
  - **savefrom.net**
  - **SBS**: sbs.com.au
  - **schooltv**
- - **screen.yahoo:search**: Yahoo screen search
+ - **ScienceChannel**
+ - **screen.yahoo:search**: Yahoo screen search; "yvsearch:" prefix
  - **Screencast**
  - **ScreencastOMatic**
  - **ScrippsNetworks**
@@ -891,12 +977,14 @@
  - **Sina**
  - **sky.it**
  - **sky:news**
+ - **sky:news:story**
  - **sky:sports**
  - **sky:sports:news**
  - **skyacademy.it**
  - **SkylineWebcams**
  - **skynewsarabia:article**
  - **skynewsarabia:video**
+ - **SkyNewsAU**
  - **Slideshare**
  - **SlidesLive**
  - **Slutload**
@@ -906,7 +994,7 @@
  - **SonyLIVSeries**
  - **soundcloud**
  - **soundcloud:playlist**
- - **soundcloud:search**: Soundcloud search
+ - **soundcloud:search**: Soundcloud search; "scsearch:" prefix
  - **soundcloud:set**
  - **soundcloud:trackstation**
  - **soundcloud:user**
@@ -918,11 +1006,12 @@
  - **southpark.de**
  - **southpark.nl**
  - **southparkstudios.dk**
+ - **SovietsCloset**
+ - **SovietsClosetPlaylist**
  - **SpankBang**
  - **SpankBangPlaylist**
  - **Spankwire**
  - **Spiegel**
- - **sport.francetvinfo.fr**
  - **Sport5**
  - **SportBox**
  - **SportDeutschland**
@@ -938,6 +1027,7 @@
  - **SRGSSR**
  - **SRGSSRPlay**: srf.ch, rts.ch, rsi.ch, rtr.ch and swissinfo.ch play sites
  - **stanfordoc**: Stanford Open ClassRoom
+ - **startv**
  - **Steam**
  - **Stitcher**
  - **StitcherShow**
@@ -945,6 +1035,7 @@
  - **StoryFireSeries**
  - **StoryFireUser**
  - **Streamable**
+ - **Streamanity**
  - **streamcloud.eu**
  - **StreamCZ**
  - **StreetVoice**
@@ -962,7 +1053,6 @@
  - **SztvHu**
  - **t-online.de**
  - **Tagesschau**
- - **tagesschau:player**
  - **Tass**
  - **TBS**
  - **TDSLifeway**
@@ -1000,17 +1090,23 @@
  - **TheScene**
  - **TheStar**
  - **TheSun**
+ - **ThetaStream**
+ - **ThetaVideo**
  - **TheWeatherChannel**
  - **ThisAmericanLife**
  - **ThisAV**
  - **ThisOldHouse**
- - **ThisVid**
+ - **ThreeSpeak**
+ - **ThreeSpeakUser**
  - **TikTok**
+ - **tiktok:user**
  - **tinypic**: tinypic.com videos
  - **TMZ**
  - **TNAFlix**
  - **TNAFlixNetworkEmbed**
  - **toggle**
+ - **Tokentube**
+ - **Tokentube:channel**
  - **ToonGoggles**
  - **tou.tv**
  - **Toypics**: Toypics video
@@ -1018,6 +1114,8 @@
  - **TrailerAddict** (Currently broken)
  - **Trilulilu**
  - **Trovo**
+ - **TrovoChannelClip**: All Clips of a trovo.live channel; "trovoclip:" prefix
+ - **TrovoChannelVod**: All VODs of a trovo.live channel; "trovovod:" prefix
  - **TrovoVod**
  - **TruNews**
  - **TruTV**
@@ -1033,10 +1131,11 @@
  - **Turbo**
  - **tv.dfb.de**
  - **TV2**
- - **tv2.hu**
  - **TV2Article**
  - **TV2DK**
  - **TV2DKBornholmPlay**
+ - **tv2play.hu**
+ - **tv2playseries.hu**
  - **TV4**: tv4.se and tv4play.se
  - **TV5MondePlus**: TV5MONDE+
  - **tv5unis**
@@ -1062,6 +1161,7 @@
  - **tvp**: Telewizja Polska
  - **tvp:embed**: Telewizja Polska
  - **tvp:series**
+ - **tvp:stream**
  - **TVPlayer**
  - **TVPlayHome**
  - **Tweakers**
@@ -1101,9 +1201,11 @@
  - **ustream:channel**
  - **ustudio**
  - **ustudio:embed**
+ - **Utreon**
  - **Varzesh3**
  - **Vbox7**
  - **VeeHD**
+ - **Veo**
  - **Veoh**
  - **Vesti**: Вести.Ru
  - **Vevo**
@@ -1119,7 +1221,7 @@
  - **Viddler**
  - **Videa**
  - **video.arnes.si**: Arnes Video
- - **video.google:search**: Google Video search
+ - **video.google:search**: Google Video search; "gvsearch:" prefix (Currently broken)
  - **video.sky.it**
  - **video.sky.it:live**
  - **VideoDetective**
@@ -1132,9 +1234,6 @@
  - **VidioLive**
  - **VidioPremier**
  - **VidLii**
- - **vidme**
- - **vidme:user**
- - **vidme:user:likes**
  - **vier**: vier.be and vijf.be
  - **vier:videos**
  - **viewlift**
@@ -1169,6 +1268,8 @@
  - **VODPl**
  - **VODPlatform**
  - **VoiceRepublic**
+ - **voicy**
+ - **voicy:channel**
  - **Voot**
  - **VootSeries**
  - **VoxMedia**
@@ -1184,6 +1285,7 @@
  - **VTXTV**
  - **vube**: Vube.com
  - **VuClip**
+ - **Vupload**
  - **VVVVID**
  - **VVVVIDShow**
  - **VyboryMos**
@@ -1214,6 +1316,8 @@
  - **WistiaPlaylist**
  - **wnl**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
  - **WorldStarHipHop**
+ - **wppilot**
+ - **wppilot:channels**
  - **WSJ**: Wall Street Journal
  - **WSJArticle**
  - **WWE**
@@ -1261,19 +1365,19 @@
  - **YouPorn**
  - **YourPorn**
  - **YourUpload**
- - **youtube**: YouTube.com
+ - **youtube**: YouTube
- - **youtube:favorites**: YouTube.com liked videos, ":ytfav" for short (requires authentication)
+ - **youtube:favorites**: YouTube liked videos; ":ytfav" keyword (requires cookies)
- - **youtube:history**: Youtube watch history, ":ythis" for short (requires authentication)
+ - **youtube:history**: Youtube watch history; ":ythis" keyword (requires cookies)
- - **youtube:playlist**: YouTube.com playlists
+ - **youtube:playlist**: YouTube playlists
- - **youtube:recommended**: YouTube.com recommended videos, ":ytrec" for short (requires authentication)
+ - **youtube:recommended**: YouTube recommended videos; ":ytrec" keyword
- - **youtube:search**: YouTube.com searches, "ytsearch" keyword
+ - **youtube:search**: YouTube searches; "ytsearch:" prefix
- - **youtube:search:date**: YouTube.com searches, newest videos first, "ytsearchdate" keyword
+ - **youtube:search:date**: YouTube searches, newest videos first; "ytsearchdate:" prefix
- - **youtube:search_url**: YouTube.com search URLs
+ - **youtube:search_url**: YouTube search URLs with sorting and filter support
- - **youtube:subscriptions**: YouTube.com subscriptions feed, ":ytsubs" for short (requires authentication)
+ - **youtube:subscriptions**: YouTube subscriptions feed; ":ytsubs" keyword (requires cookies)
- - **youtube:tab**: YouTube.com tab
+ - **youtube:tab**: YouTube Tabs
- - **youtube:watchlater**: Youtube watch later list, ":ytwatchlater" for short (requires authentication)
+ - **youtube:watchlater**: Youtube watch later list; ":ytwatchlater" keyword (requires cookies)
  - **YoutubeYtBe**: youtu.be
- - **YoutubeYtUser**: YouTube.com user videos, URL or "ytuser" keyword
+ - **YoutubeYtUser**: YouTube user videos; "ytuser:" prefix
  - **Zapiks**
  - **Zattoo**
  - **ZattooLive**
@@ -1281,6 +1385,8 @@
  - **ZDFChannel**
  - **Zee5**
  - **zee5:series**
+ - **ZenYandex**
+ - **ZenYandexChannel**
  - **Zhihu**
  - **zingmp3**: mp3.zing.vn
  - **zingmp3:album**

test/helper.py

@@ -22,11 +22,19 @@ from yt_dlp.utils import (
 )
 
 
+if 'pytest' in sys.modules:
+    import pytest
+    is_download_test = pytest.mark.download
+else:
+    def is_download_test(testClass):
+        return testClass
+
+
 def get_params(override=None):
     PARAMETERS_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
-                                   "parameters.json")
+                                   'parameters.json')
     LOCAL_PARAMETERS_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
-                                         "local_parameters.json")
+                                         'local_parameters.json')
     with io.open(PARAMETERS_FILE, encoding='utf-8') as pf:
         parameters = json.load(pf)
     if os.path.exists(LOCAL_PARAMETERS_FILE):
@@ -190,7 +198,10 @@ def expect_info_dict(self, got_dict, expected_dict):
     expect_dict(self, got_dict, expected_dict)
     # Check for the presence of mandatory fields
     if got_dict.get('_type') not in ('playlist', 'multi_video'):
-        for key in ('id', 'url', 'title', 'ext'):
+        mandatory_fields = ['id', 'title']
+        if expected_dict.get('ext'):
+            mandatory_fields.extend(('url', 'ext'))
+        for key in mandatory_fields:
             self.assertTrue(got_dict.get(key), 'Missing mandatory field %s' % key)
     # Check for mandatory fields that are automatically set by YoutubeDL
     for key in ['webpage_url', 'extractor', 'extractor_key']:

test/parameters.json

@@ -1,4 +1,5 @@
 {
+    "check_formats": false,
     "consoletitle": false,
     "continuedl": true,
     "forcedescription": false,
@@ -8,7 +9,7 @@
     "forcetitle": false,
     "forceurl": false,
     "force_write_download_archive": false,
-    "format": "best",
+    "format": "b/bv",
     "ignoreerrors": false,
     "listformats": null,
     "logtostderr": false,
@@ -43,6 +44,5 @@
     "writesubtitles": false,
     "allsubtitles": false,
     "listsubtitles": false,
-    "socket_timeout": 20,
     "fixup": "never"
 }

test/test_InfoExtractor.py

@@ -35,13 +35,13 @@ class InfoExtractorTestRequestHandler(compat_http_server.BaseHTTPRequestHandler)
             assert False
 
 
-class TestIE(InfoExtractor):
+class DummyIE(InfoExtractor):
     pass
 
 
 class TestInfoExtractor(unittest.TestCase):
     def setUp(self):
-        self.ie = TestIE(FakeYDL())
+        self.ie = DummyIE(FakeYDL())
 
     def test_ie_key(self):
         self.assertEqual(get_info_extractor(YoutubeIE.ie_key()), YoutubeIE)

test/test_YoutubeDL.py

@@ -10,14 +10,15 @@ import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 import copy
+import json
 
 from test.helper import FakeYDL, assertRegexpMatches
 from yt_dlp import YoutubeDL
-from yt_dlp.compat import compat_str, compat_urllib_error
+from yt_dlp.compat import compat_os_name, compat_setenv, compat_str, compat_urllib_error
 from yt_dlp.extractor import YoutubeIE
 from yt_dlp.extractor.common import InfoExtractor
 from yt_dlp.postprocessor.common import PostProcessor
-from yt_dlp.utils import ExtractorError, int_or_none, match_filter_func
+from yt_dlp.utils import ExtractorError, int_or_none, match_filter_func, LazyList
 
 TEST_URL = 'http://localhost/sample.mp4'
 
@@ -35,6 +36,9 @@ class YDL(FakeYDL):
     def to_screen(self, msg):
         self.msgs.append(msg)
 
+    def dl(self, *args, **kwargs):
+        assert False, 'Downloader must not be invoked for test_YoutubeDL'
+
 
 def _make_result(formats, **kwargs):
     res = {
@@ -117,35 +121,24 @@ class TestFormatSelection(unittest.TestCase):
         ]
         info_dict = _make_result(formats)
 
-        ydl = YDL({'format': '20/47'})
+        def test(inp, *expected, multi=False):
-        ydl.process_ie_result(info_dict.copy())
+            ydl = YDL({
-        downloaded = ydl.downloaded_info_dicts[0]
+                'format': inp,
-        self.assertEqual(downloaded['format_id'], '47')
+                'allow_multiple_video_streams': multi,
+                'allow_multiple_audio_streams': multi,
+            })
+            ydl.process_ie_result(info_dict.copy())
+            downloaded = map(lambda x: x['format_id'], ydl.downloaded_info_dicts)
+            self.assertEqual(list(downloaded), list(expected))
 
-        ydl = YDL({'format': '20/71/worst'})
+        test('20/47', '47')
-        ydl.process_ie_result(info_dict.copy())
+        test('20/71/worst', '35')
-        downloaded = ydl.downloaded_info_dicts[0]
+        test(None, '2')
-        self.assertEqual(downloaded['format_id'], '35')
+        test('webm/mp4', '47')
-
+        test('3gp/40/mp4', '35')
-        ydl = YDL()
+        test('example-with-dashes', 'example-with-dashes')
-        ydl.process_ie_result(info_dict.copy())
+        test('all', '35', 'example-with-dashes', '45', '47', '2')  # Order doesn't actually matter for this
-        downloaded = ydl.downloaded_info_dicts[0]
+        test('mergeall', '2+47+45+example-with-dashes+35', multi=True)
-        self.assertEqual(downloaded['format_id'], '2')
-
-        ydl = YDL({'format': 'webm/mp4'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], '47')
-
-        ydl = YDL({'format': '3gp/40/mp4'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], '35')
-
-        ydl = YDL({'format': 'example-with-dashes'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'example-with-dashes')
 
     def test_format_selection_audio(self):
         formats = [
@@ -655,12 +648,15 @@ class TestYoutubeDL(unittest.TestCase):
         'title1': '$PATH',
         'title2': '%PATH%',
         'title3': 'foo/bar\\test',
+        'title4': 'foo "bar" test',
+        'title5': 'áéí 𝐀',
         'timestamp': 1618488000,
         'duration': 100000,
         'playlist_index': 1,
+        'playlist_autonumber': 2,
         '_last_playlist_index': 100,
         'n_entries': 10,
-        'formats': [{'id': 'id1'}, {'id': 'id2'}, {'id': 'id3'}]
+        'formats': [{'id': 'id 1'}, {'id': 'id 2'}, {'id': 'id 3'}]
     }
 
     def test_prepare_outtmpl_and_filename(self):
@@ -670,32 +666,45 @@ class TestYoutubeDL(unittest.TestCase):
             ydl._num_downloads = 1
             self.assertEqual(ydl.validate_outtmpl(tmpl), None)
 
-            outtmpl, tmpl_dict = ydl.prepare_outtmpl(tmpl, info or self.outtmpl_info)
+            out = ydl.evaluate_outtmpl(tmpl, info or self.outtmpl_info)
-            out = outtmpl % tmpl_dict
             fname = ydl.prepare_filename(info or self.outtmpl_info)
 
-            if callable(expected):
+            if not isinstance(expected, (list, tuple)):
-                self.assertTrue(expected(out))
+                expected = (expected, expected)
-                self.assertTrue(expected(fname))
+            for (name, got), expect in zip((('outtmpl', out), ('filename', fname)), expected):
-            elif isinstance(expected, compat_str):
+                if callable(expect):
-                self.assertEqual((out, fname), (expected, expected))
+                    self.assertTrue(expect(got), f'Wrong {name} from {tmpl}')
-            else:
+                else:
-                self.assertEqual((out, fname), expected)
+                    self.assertEqual(got, expect, f'Wrong {name} from {tmpl}')
+
+        # Side-effects
+        original_infodict = dict(self.outtmpl_info)
+        test('foo.bar', 'foo.bar')
+        original_infodict['epoch'] = self.outtmpl_info.get('epoch')
+        self.assertTrue(isinstance(original_infodict['epoch'], int))
+        test('%(epoch)d', int_or_none)
+        self.assertEqual(original_infodict, self.outtmpl_info)
 
         # Auto-generated fields
         test('%(id)s.%(ext)s', '1234.mp4')
         test('%(duration_string)s', ('27:46:40', '27-46-40'))
-        test('%(epoch)d', int_or_none)
         test('%(resolution)s', '1080p')
         test('%(playlist_index)s', '001')
+        test('%(playlist_autonumber)s', '02')
         test('%(autonumber)s', '00001')
         test('%(autonumber+2)03d', '005', autonumber_start=3)
         test('%(autonumber)s', '001', autonumber_size=3)
 
         # Escaping %
+        test('%', '%')
         test('%%', '%')
         test('%%%%', '%%')
+        test('%s', '%s')
+        test('%%%s', '%%s')
+        test('%d', '%d')
+        test('%abc%', '%abc%')
         test('%%(width)06d.%(ext)s', '%(width)06d.mp4')
+        test('%%%(height)s', '%1080')
         test('%(width)06d.%(ext)s', 'NA.mp4')
         test('%(width)06d.%%(ext)s', 'NA.%(ext)s')
         test('%%(width)06d.%(ext)s', '%(width)06d.mp4')
@@ -710,18 +719,25 @@ class TestYoutubeDL(unittest.TestCase):
         test('%(id)s', ('ab:cd', 'ab -cd'), info={'id': 'ab:cd'})
 
         # Invalid templates
-        self.assertTrue(isinstance(YoutubeDL.validate_outtmpl('%'), ValueError))
         self.assertTrue(isinstance(YoutubeDL.validate_outtmpl('%(title)'), ValueError))
         test('%(invalid@tmpl|def)s', 'none', outtmpl_na_placeholder='none')
-        test('%()s', 'NA')
+        test('%(..)s', 'NA')
-        test('%s', '%s')
+
-        test('%d', '%d')
+        # Entire info_dict
+        def expect_same_infodict(out):
+            got_dict = json.loads(out)
+            for info_field, expected in self.outtmpl_info.items():
+                self.assertEqual(got_dict.get(info_field), expected, info_field)
+            return True
+
+        test('%()j', (expect_same_infodict, str))
 
         # NA placeholder
         NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(x|def)s-%(id)s.%(ext)s'
         test(NA_TEST_OUTTMPL, 'NA-NA-def-1234.mp4')
         test(NA_TEST_OUTTMPL, 'none-none-def-1234.mp4', outtmpl_na_placeholder='none')
         test(NA_TEST_OUTTMPL, '--def-1234.mp4', outtmpl_na_placeholder='')
+        test('%(non_existent.0)s', 'NA')
 
         # String formatting
         FMT_TEST_OUTTMPL = '%%(height)%s.%%(ext)s'
@@ -746,13 +762,37 @@ class TestYoutubeDL(unittest.TestCase):
         test('%(width|0)04d', '0000')
         test('a%(width|)d', 'a', outtmpl_na_placeholder='none')
 
-        # Internal formatting
         FORMATS = self.outtmpl_info['formats']
+        sanitize = lambda x: x.replace(':', ' -').replace('"', "'").replace('\n', ' ')
+
+        # Custom type casting
+        test('%(formats.:.id)l', 'id 1, id 2, id 3')
+        test('%(formats.:.id)#l', ('id 1\nid 2\nid 3', 'id 1 id 2 id 3'))
+        test('%(ext)l', 'mp4')
+        test('%(formats.:.id) 18l', '  id 1, id 2, id 3')
+        test('%(formats)j', (json.dumps(FORMATS), sanitize(json.dumps(FORMATS))))
+        test('%(formats)#j', (json.dumps(FORMATS, indent=4), sanitize(json.dumps(FORMATS, indent=4))))
+        test('%(title5).3B', 'á')
+        test('%(title5)U', 'áéí 𝐀')
+        test('%(title5)#U', 'a\u0301e\u0301i\u0301 𝐀')
+        test('%(title5)+U', 'áéí A')
+        test('%(title5)+#U', 'a\u0301e\u0301i\u0301 A')
+        if compat_os_name == 'nt':
+            test('%(title4)q', ('"foo \\"bar\\" test"', "'foo _'bar_' test'"))
+            test('%(formats.:.id)#q', ('"id 1" "id 2" "id 3"', "'id 1' 'id 2' 'id 3'"))
+            test('%(formats.0.id)#q', ('"id 1"', "'id 1'"))
+        else:
+            test('%(title4)q', ('\'foo "bar" test\'', "'foo 'bar' test'"))
+            test('%(formats.:.id)#q', "'id 1' 'id 2' 'id 3'")
+            test('%(formats.0.id)#q', "'id 1'")
+
+        # Internal formatting
         test('%(timestamp-1000>%H-%M-%S)s', '11-43-20')
+        test('%(title|%)s %(title|%%)s', '% %%')
         test('%(id+1-height+3)05d', '00158')
         test('%(width+100)05d', 'NA')
-        test('%(formats.0) 15s', ('% 15s' % FORMATS[0], '% 15s' % str(FORMATS[0]).replace(':', ' -')))
+        test('%(formats.0) 15s', ('% 15s' % FORMATS[0], '% 15s' % sanitize(str(FORMATS[0]))))
-        test('%(formats.0)r', (repr(FORMATS[0]), repr(FORMATS[0]).replace(':', ' -')))
+        test('%(formats.0)r', (repr(FORMATS[0]), sanitize(repr(FORMATS[0]))))
         test('%(height.0)03d', '001')
         test('%(-height.0)04d', '-001')
         test('%(formats.-1.id)s', FORMATS[-1]['id'])
@@ -762,11 +802,34 @@ class TestYoutubeDL(unittest.TestCase):
         test('%(formats.0.id.-1+id)f', '1235.000000')
         test('%(formats.0.id.-1+formats.1.id.-1)d', '3')
 
+        # Alternates
+        test('%(title,id)s', '1234')
+        test('%(width-100,height+20|def)d', '1100')
+        test('%(width-100,height+width|def)s', 'def')
+        test('%(timestamp-x>%H\\,%M\\,%S,timestamp>%H\\,%M\\,%S)s', '12,00,00')
+
+        # Laziness
+        def gen():
+            yield from range(5)
+            raise self.assertTrue(False, 'LazyList should not be evaluated till here')
+        test('%(key.4)s', '4', info={'key': LazyList(gen())})
+
         # Empty filename
         test('%(foo|)s-%(bar|)s.%(ext)s', '-.mp4')
         # test('%(foo|)s.%(ext)s', ('.mp4', '_.mp4'))  # fixme
         # test('%(foo|)s', ('', '_'))  # fixme
 
+        # Environment variable expansion for prepare_filename
+        compat_setenv('__yt_dlp_var', 'expanded')
+        envvar = '%__yt_dlp_var%' if compat_os_name == 'nt' else '$__yt_dlp_var'
+        test(envvar, (envvar, 'expanded'))
+        if compat_os_name == 'nt':
+            test('%s%', ('%s%', '%s%'))
+            compat_setenv('s', 'expanded')
+            test('%s%', ('%s%', 'expanded'))  # %s% should be expanded before escaping %s
+            compat_setenv('(test)s', 'expanded')
+            test('%(test)s%', ('NA%', 'expanded'))  # Environment should take priority over template
+
         # Path expansion and escaping
         test('Hello %(title1)s', 'Hello $PATH')
         test('Hello %(title2)s', 'Hello %PATH%')
@@ -941,54 +1004,32 @@ class TestYoutubeDL(unittest.TestCase):
             ydl.process_ie_result(copy.deepcopy(playlist))
             return ydl.downloaded_info_dicts
 
-        def get_ids(params):
+        def test_selection(params, expected_ids):
-            return [int(v['id']) for v in get_downloaded_info_dicts(params)]
+            results = [
+                (v['playlist_autonumber'] - 1, (int(v['id']), v['playlist_index']))
+                for v in get_downloaded_info_dicts(params)]
+            self.assertEqual(results, list(enumerate(zip(expected_ids, expected_ids))))
 
-        result = get_ids({})
+        test_selection({}, [1, 2, 3, 4])
-        self.assertEqual(result, [1, 2, 3, 4])
+        test_selection({'playlistend': 10}, [1, 2, 3, 4])
-
+        test_selection({'playlistend': 2}, [1, 2])
-        result = get_ids({'playlistend': 10})
+        test_selection({'playliststart': 10}, [])
-        self.assertEqual(result, [1, 2, 3, 4])
+        test_selection({'playliststart': 2}, [2, 3, 4])
-
+        test_selection({'playlist_items': '2-4'}, [2, 3, 4])
-        result = get_ids({'playlistend': 2})
+        test_selection({'playlist_items': '2,4'}, [2, 4])
-        self.assertEqual(result, [1, 2])
+        test_selection({'playlist_items': '10'}, [])
-
+        test_selection({'playlist_items': '0'}, [])
-        result = get_ids({'playliststart': 10})
-        self.assertEqual(result, [])
-
-        result = get_ids({'playliststart': 2})
-        self.assertEqual(result, [2, 3, 4])
-
-        result = get_ids({'playlist_items': '2-4'})
-        self.assertEqual(result, [2, 3, 4])
-
-        result = get_ids({'playlist_items': '2,4'})
-        self.assertEqual(result, [2, 4])
-
-        result = get_ids({'playlist_items': '10'})
-        self.assertEqual(result, [])
-
-        result = get_ids({'playlist_items': '3-10'})
-        self.assertEqual(result, [3, 4])
-
-        result = get_ids({'playlist_items': '2-4,3-4,3'})
-        self.assertEqual(result, [2, 3, 4])
 
         # Tests for https://github.com/ytdl-org/youtube-dl/issues/10591
-        # @{
+        test_selection({'playlist_items': '2-4,3-4,3'}, [2, 3, 4])
-        result = get_downloaded_info_dicts({'playlist_items': '2-4,3-4,3'})
+        test_selection({'playlist_items': '4,2'}, [4, 2])
-        self.assertEqual(result[0]['playlist_index'], 2)
-        self.assertEqual(result[1]['playlist_index'], 3)
 
-        result = get_downloaded_info_dicts({'playlist_items': '2-4,3-4,3'})
+        # Tests for https://github.com/yt-dlp/yt-dlp/issues/720
-        self.assertEqual(result[0]['playlist_index'], 2)
+        # https://github.com/yt-dlp/yt-dlp/issues/302
-        self.assertEqual(result[1]['playlist_index'], 3)
+        test_selection({'playlistreverse': True}, [4, 3, 2, 1])
-        self.assertEqual(result[2]['playlist_index'], 4)
+        test_selection({'playliststart': 2, 'playlistreverse': True}, [4, 3, 2])
-
+        test_selection({'playlist_items': '2,4', 'playlistreverse': True}, [4, 2])
-        result = get_downloaded_info_dicts({'playlist_items': '4,2'})
+        test_selection({'playlist_items': '4,2'}, [4, 2])
-        self.assertEqual(result[0]['playlist_index'], 4)
-        self.assertEqual(result[1]['playlist_index'], 2)
-        # @}
 
     def test_urlopen_no_file_protocol(self):
         # see https://github.com/ytdl-org/youtube-dl/issues/8227

test/test_aes.py

@@ -7,7 +7,19 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from yt_dlp.aes import aes_decrypt, aes_encrypt, aes_cbc_decrypt, aes_cbc_encrypt, aes_decrypt_text
+from yt_dlp.aes import (
+    aes_decrypt,
+    aes_encrypt,
+    aes_cbc_decrypt,
+    aes_cbc_decrypt_bytes,
+    aes_cbc_encrypt,
+    aes_ctr_decrypt,
+    aes_ctr_encrypt,
+    aes_gcm_decrypt_and_verify,
+    aes_gcm_decrypt_and_verify_bytes,
+    aes_decrypt_text
+)
+from yt_dlp.compat import compat_pycrypto_AES
 from yt_dlp.utils import bytes_to_intlist, intlist_to_bytes
 import base64
 
@@ -27,18 +39,43 @@ class TestAES(unittest.TestCase):
         self.assertEqual(decrypted, msg)
 
     def test_cbc_decrypt(self):
-        data = bytes_to_intlist(
+        data = b'\x97\x92+\xe5\x0b\xc3\x18\x91ky9m&\xb3\xb5@\xe6\x27\xc2\x96.\xc8u\x88\xab9-[\x9e|\xf1\xcd'
-            b"\x97\x92+\xe5\x0b\xc3\x18\x91ky9m&\xb3\xb5@\xe6'\xc2\x96.\xc8u\x88\xab9-[\x9e|\xf1\xcd"
+        decrypted = intlist_to_bytes(aes_cbc_decrypt(bytes_to_intlist(data), self.key, self.iv))
-        )
-        decrypted = intlist_to_bytes(aes_cbc_decrypt(data, self.key, self.iv))
         self.assertEqual(decrypted.rstrip(b'\x08'), self.secret_msg)
+        if compat_pycrypto_AES:
+            decrypted = aes_cbc_decrypt_bytes(data, intlist_to_bytes(self.key), intlist_to_bytes(self.iv))
+            self.assertEqual(decrypted.rstrip(b'\x08'), self.secret_msg)
 
     def test_cbc_encrypt(self):
         data = bytes_to_intlist(self.secret_msg)
         encrypted = intlist_to_bytes(aes_cbc_encrypt(data, self.key, self.iv))
         self.assertEqual(
             encrypted,
-            b"\x97\x92+\xe5\x0b\xc3\x18\x91ky9m&\xb3\xb5@\xe6'\xc2\x96.\xc8u\x88\xab9-[\x9e|\xf1\xcd")
+            b'\x97\x92+\xe5\x0b\xc3\x18\x91ky9m&\xb3\xb5@\xe6\'\xc2\x96.\xc8u\x88\xab9-[\x9e|\xf1\xcd')
+
+    def test_ctr_decrypt(self):
+        data = bytes_to_intlist(b'\x03\xc7\xdd\xd4\x8e\xb3\xbc\x1a*O\xdc1\x12+8Aio\xd1z\xb5#\xaf\x08')
+        decrypted = intlist_to_bytes(aes_ctr_decrypt(data, self.key, self.iv))
+        self.assertEqual(decrypted.rstrip(b'\x08'), self.secret_msg)
+
+    def test_ctr_encrypt(self):
+        data = bytes_to_intlist(self.secret_msg)
+        encrypted = intlist_to_bytes(aes_ctr_encrypt(data, self.key, self.iv))
+        self.assertEqual(
+            encrypted,
+            b'\x03\xc7\xdd\xd4\x8e\xb3\xbc\x1a*O\xdc1\x12+8Aio\xd1z\xb5#\xaf\x08')
+
+    def test_gcm_decrypt(self):
+        data = b'\x159Y\xcf5eud\x90\x9c\x85&]\x14\x1d\x0f.\x08\xb4T\xe4/\x17\xbd'
+        authentication_tag = b'\xe8&I\x80rI\x07\x9d}YWuU@:e'
+
+        decrypted = intlist_to_bytes(aes_gcm_decrypt_and_verify(
+            bytes_to_intlist(data), self.key, bytes_to_intlist(authentication_tag), self.iv[:12]))
+        self.assertEqual(decrypted.rstrip(b'\x08'), self.secret_msg)
+        if compat_pycrypto_AES:
+            decrypted = aes_gcm_decrypt_and_verify_bytes(
+                data, intlist_to_bytes(self.key), authentication_tag, intlist_to_bytes(self.iv[:12]))
+            self.assertEqual(decrypted.rstrip(b'\x08'), self.secret_msg)
 
     def test_decrypt_text(self):
         password = intlist_to_bytes(self.key).decode('utf-8')

test/test_age_restriction.py

@@ -7,8 +7,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import try_rm
+from test.helper import try_rm, is_download_test
-
 
 from yt_dlp import YoutubeDL
 
@@ -32,6 +31,7 @@ def _download_restricted(url, filename, age):
     return res
 
 
+@is_download_test
 class TestAgeRestriction(unittest.TestCase):
     def _assert_restricted(self, url, filename, age, old_age=None):
         self.assertTrue(_download_restricted(url, filename, old_age))

test/test_cookies.py

@@ -0,0 +1,107 @@
+import unittest
+from datetime import datetime, timezone
+
+from yt_dlp import cookies
+from yt_dlp.cookies import (
+    LinuxChromeCookieDecryptor,
+    MacChromeCookieDecryptor,
+    WindowsChromeCookieDecryptor,
+    parse_safari_cookies,
+    pbkdf2_sha1,
+)
+
+
+class Logger:
+    def debug(self, message):
+        print(f'[verbose] {message}')
+
+    def info(self, message):
+        print(message)
+
+    def warning(self, message, only_once=False):
+        self.error(message)
+
+    def error(self, message):
+        raise Exception(message)
+
+
+class MonkeyPatch:
+    def __init__(self, module, temporary_values):
+        self._module = module
+        self._temporary_values = temporary_values
+        self._backup_values = {}
+
+    def __enter__(self):
+        for name, temp_value in self._temporary_values.items():
+            self._backup_values[name] = getattr(self._module, name)
+            setattr(self._module, name, temp_value)
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        for name, backup_value in self._backup_values.items():
+            setattr(self._module, name, backup_value)
+
+
+class TestCookies(unittest.TestCase):
+    def test_chrome_cookie_decryptor_linux_derive_key(self):
+        key = LinuxChromeCookieDecryptor.derive_key(b'abc')
+        self.assertEqual(key, b'7\xa1\xec\xd4m\xfcA\xc7\xb19Z\xd0\x19\xdcM\x17')
+
+    def test_chrome_cookie_decryptor_mac_derive_key(self):
+        key = MacChromeCookieDecryptor.derive_key(b'abc')
+        self.assertEqual(key, b'Y\xe2\xc0\xd0P\xf6\xf4\xe1l\xc1\x8cQ\xcb|\xcdY')
+
+    def test_chrome_cookie_decryptor_linux_v10(self):
+        with MonkeyPatch(cookies, {'_get_linux_keyring_password': lambda *args, **kwargs: b''}):
+            encrypted_value = b'v10\xccW%\xcd\xe6\xe6\x9fM" \xa7\xb0\xca\xe4\x07\xd6'
+            value = 'USD'
+            decryptor = LinuxChromeCookieDecryptor('Chrome', Logger())
+            self.assertEqual(decryptor.decrypt(encrypted_value), value)
+
+    def test_chrome_cookie_decryptor_linux_v11(self):
+        with MonkeyPatch(cookies, {'_get_linux_keyring_password': lambda *args, **kwargs: b'',
+                                   'KEYRING_AVAILABLE': True}):
+            encrypted_value = b'v11#\x81\x10>`w\x8f)\xc0\xb2\xc1\r\xf4\x1al\xdd\x93\xfd\xf8\xf8N\xf2\xa9\x83\xf1\xe9o\x0elVQd'
+            value = 'tz=Europe.London'
+            decryptor = LinuxChromeCookieDecryptor('Chrome', Logger())
+            self.assertEqual(decryptor.decrypt(encrypted_value), value)
+
+    def test_chrome_cookie_decryptor_windows_v10(self):
+        with MonkeyPatch(cookies, {
+            '_get_windows_v10_key': lambda *args, **kwargs: b'Y\xef\xad\xad\xeerp\xf0Y\xe6\x9b\x12\xc2<z\x16]\n\xbb\xb8\xcb\xd7\x9bA\xc3\x14e\x99{\xd6\xf4&'
+        }):
+            encrypted_value = b'v10T\xb8\xf3\xb8\x01\xa7TtcV\xfc\x88\xb8\xb8\xef\x05\xb5\xfd\x18\xc90\x009\xab\xb1\x893\x85)\x87\xe1\xa9-\xa3\xad='
+            value = '32101439'
+            decryptor = WindowsChromeCookieDecryptor('', Logger())
+            self.assertEqual(decryptor.decrypt(encrypted_value), value)
+
+    def test_chrome_cookie_decryptor_mac_v10(self):
+        with MonkeyPatch(cookies, {'_get_mac_keyring_password': lambda *args, **kwargs: b'6eIDUdtKAacvlHwBVwvg/Q=='}):
+            encrypted_value = b'v10\xb3\xbe\xad\xa1[\x9fC\xa1\x98\xe0\x9a\x01\xd9\xcf\xbfc'
+            value = '2021-06-01-22'
+            decryptor = MacChromeCookieDecryptor('', Logger())
+            self.assertEqual(decryptor.decrypt(encrypted_value), value)
+
+    def test_safari_cookie_parsing(self):
+        cookies = \
+            b'cook\x00\x00\x00\x01\x00\x00\x00i\x00\x00\x01\x00\x01\x00\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00Y' \
+            b'\x00\x00\x00\x00\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x008\x00\x00\x00B\x00\x00\x00F\x00\x00\x00H' \
+            b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x80\x03\xa5>\xc3A\x00\x00\x80\xc3\x07:\xc3A' \
+            b'localhost\x00foo\x00/\x00test%20%3Bcookie\x00\x00\x00\x054\x07\x17 \x05\x00\x00\x00Kbplist00\xd1\x01' \
+            b'\x02_\x10\x18NSHTTPCookieAcceptPolicy\x10\x02\x08\x0b&\x00\x00\x00\x00\x00\x00\x01\x01\x00\x00\x00' \
+            b'\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00('
+
+        jar = parse_safari_cookies(cookies)
+        self.assertEqual(len(jar), 1)
+        cookie = list(jar)[0]
+        self.assertEqual(cookie.domain, 'localhost')
+        self.assertEqual(cookie.port, None)
+        self.assertEqual(cookie.path, '/')
+        self.assertEqual(cookie.name, 'foo')
+        self.assertEqual(cookie.value, 'test%20%3Bcookie')
+        self.assertFalse(cookie.secure)
+        expected_expiration = datetime(2021, 6, 18, 21, 39, 19, tzinfo=timezone.utc)
+        self.assertEqual(cookie.expires, int(expected_expiration.timestamp()))
+
+    def test_pbkdf2_sha1(self):
+        key = pbkdf2_sha1(b'peanuts', b' ' * 16, 1, 16)
+        self.assertEqual(key, b'g\xe1\x8e\x0fQ\x1c\x9b\xf3\xc9`!\xaa\x90\xd9\xd34')

test/test_download.py

@@ -10,12 +10,13 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from test.helper import (
     assertGreaterEqual,
+    expect_info_dict,
     expect_warnings,
     get_params,
     gettestcases,
-    expect_info_dict,
+    is_download_test,
-    try_rm,
     report_warning,
+    try_rm,
 )
 
 
@@ -64,6 +65,7 @@ def _file_md5(fn):
 defs = gettestcases()
 
 
+@is_download_test
 class TestDownload(unittest.TestCase):
     # Parallel testing in nosetests. See
     # http://nose.readthedocs.org/en/latest/doc_tests/test_multiprocess/multiprocess.html
@@ -71,6 +73,8 @@ class TestDownload(unittest.TestCase):
 
     maxDiff = None
 
+    COMPLETED_TESTS = {}
+
     def __str__(self):
         """Identify each test with the `add_ie` attribute, if available."""
 
@@ -92,6 +96,9 @@ class TestDownload(unittest.TestCase):
 def generator(test_case, tname):
 
     def test_template(self):
+        if self.COMPLETED_TESTS.get(tname):
+            return
+        self.COMPLETED_TESTS[tname] = True
         ie = yt_dlp.extractor.get_info_extractor(test_case['name'])()
         other_ies = [get_info_extractor(ie_key)() for ie_key in test_case.get('add_ie', [])]
         is_playlist = any(k.startswith('playlist') for k in test_case)
@@ -106,8 +113,13 @@ def generator(test_case, tname):
 
         for tc in test_cases:
             info_dict = tc.get('info_dict', {})
-            if not (info_dict.get('id') and info_dict.get('ext')):
+            params = tc.get('params', {})
-                raise Exception('Test definition incorrect. The output file cannot be known. Are both \'id\' and \'ext\' keys present?')
+            if not info_dict.get('id'):
+                raise Exception('Test definition incorrect. \'id\' key is not present')
+            elif not info_dict.get('ext'):
+                if params.get('skip_download') and params.get('ignore_no_formats_error'):
+                    continue
+                raise Exception('Test definition incorrect. The output file cannot be known. \'ext\' key is not present')
 
         if 'skip' in test_case:
             print_skipping(test_case['skip'])
@@ -135,7 +147,7 @@ def generator(test_case, tname):
         expect_warnings(ydl, test_case.get('expected_warnings', []))
 
         def get_tc_filename(tc):
-            return ydl.prepare_filename(tc.get('info_dict', {}))
+            return ydl.prepare_filename(dict(tc.get('info_dict', {})))
 
         res_dict = None
 
@@ -248,12 +260,12 @@ def generator(test_case, tname):
 
 
 # And add them to TestDownload
-for n, test_case in enumerate(defs):
+tests_counter = {}
-    tname = 'test_' + str(test_case['name'])
+for test_case in defs:
-    i = 1
+    name = test_case['name']
-    while hasattr(TestDownload, tname):
+    i = tests_counter.get(name, 0)
-        tname = 'test_%s_%d' % (test_case['name'], i)
+    tests_counter[name] = i + 1
-        i += 1
+    tname = f'test_{name}_{i}' if i else f'test_{name}'
     test_method = generator(test_case, tname)
     test_method.__name__ = str(tname)
     ie_list = test_case.get('add_ie')
@@ -262,5 +274,22 @@ for n, test_case in enumerate(defs):
     del test_method
 
 
+def batch_generator(name, num_tests):
+
+    def test_template(self):
+        for i in range(num_tests):
+            getattr(self, f'test_{name}_{i}' if i else f'test_{name}')()
+
+    return test_template
+
+
+for name, num_tests in tests_counter.items():
+    test_method = batch_generator(name, num_tests)
+    test_method.__name__ = f'test_{name}_all'
+    test_method.add_ie = ''
+    setattr(TestDownload, test_method.__name__, test_method)
+    del test_method
+
+
 if __name__ == '__main__':
     unittest.main()

test/test_iqiyi_sdk_interpreter.py

@@ -8,7 +8,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import FakeYDL
+from test.helper import FakeYDL, is_download_test
 from yt_dlp.extractor import IqiyiIE
 
 
@@ -31,6 +31,7 @@ class WarningLogger(object):
         pass
 
 
+@is_download_test
 class TestIqiyiSDKInterpreter(unittest.TestCase):
     def test_iqiyi_sdk_interpreter(self):
         '''

test/test_jsinterp.py

@@ -112,6 +112,71 @@ class TestJSInterpreter(unittest.TestCase):
         ''')
         self.assertEqual(jsi.call_function('z'), 5)
 
+    def test_for_loop(self):
+        jsi = JSInterpreter('''
+        function x() { a=0; for (i=0; i-10; i++) {a++} a }
+        ''')
+        self.assertEqual(jsi.call_function('x'), 10)
+
+    def test_switch(self):
+        jsi = JSInterpreter('''
+        function x(f) { switch(f){
+            case 1:f+=1;
+            case 2:f+=2;
+            case 3:f+=3;break;
+            case 4:f+=4;
+            default:f=0;
+        } return f }
+        ''')
+        self.assertEqual(jsi.call_function('x', 1), 7)
+        self.assertEqual(jsi.call_function('x', 3), 6)
+        self.assertEqual(jsi.call_function('x', 5), 0)
+
+    def test_switch_default(self):
+        jsi = JSInterpreter('''
+        function x(f) { switch(f){
+            case 2: f+=2;
+            default: f-=1;
+            case 5:
+            case 6: f+=6;
+            case 0: break;
+            case 1: f+=1;
+        } return f }
+        ''')
+        self.assertEqual(jsi.call_function('x', 1), 2)
+        self.assertEqual(jsi.call_function('x', 5), 11)
+        self.assertEqual(jsi.call_function('x', 9), 14)
+
+    def test_try(self):
+        jsi = JSInterpreter('''
+        function x() { try{return 10} catch(e){return 5} }
+        ''')
+        self.assertEqual(jsi.call_function('x'), 10)
+
+    def test_for_loop_continue(self):
+        jsi = JSInterpreter('''
+        function x() { a=0; for (i=0; i-10; i++) { continue; a++ } a }
+        ''')
+        self.assertEqual(jsi.call_function('x'), 0)
+
+    def test_for_loop_break(self):
+        jsi = JSInterpreter('''
+        function x() { a=0; for (i=0; i-10; i++) { break; a++ } a }
+        ''')
+        self.assertEqual(jsi.call_function('x'), 0)
+
+    def test_literal_list(self):
+        jsi = JSInterpreter('''
+        function x() { [1, 2, "asdf", [5, 6, 7]][3] }
+        ''')
+        self.assertEqual(jsi.call_function('x'), [5, 6, 7])
+
+    def test_comma(self):
+        jsi = JSInterpreter('''
+        function x() { a=5; a -= 1, a+=3; return a }
+        ''')
+        self.assertEqual(jsi.call_function('x'), 7)
+
 
 if __name__ == '__main__':
     unittest.main()

test/test_overwrites.py

@@ -8,13 +8,14 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import try_rm
+from test.helper import is_download_test, try_rm
 
 
 root_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 download_file = join(root_dir, 'test.webm')
 
 
+@is_download_test
 class TestOverwrites(unittest.TestCase):
     def setUp(self):
         # create an empty file

test/test_post_hooks.py

@@ -7,7 +7,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import get_params, try_rm
+from test.helper import get_params, try_rm, is_download_test
 import yt_dlp.YoutubeDL
 from yt_dlp.utils import DownloadError
 
@@ -22,6 +22,7 @@ TEST_ID = 'gr51aVj-mLg'
 EXPECTED_NAME = 'gr51aVj-mLg'
 
 
+@is_download_test
 class TestPostHooks(unittest.TestCase):
     def setUp(self):
         self.stored_name_1 = None

test/test_postprocessors.py

@@ -6,37 +6,38 @@ from __future__ import unicode_literals
 import os
 import sys
 import unittest
+
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from yt_dlp import YoutubeDL
 from yt_dlp.compat import compat_shlex_quote
 from yt_dlp.postprocessor import (
-    ExecAfterDownloadPP,
+    ExecPP,
     FFmpegThumbnailsConvertorPP,
     MetadataFromFieldPP,
-    MetadataFromTitlePP,
+    MetadataParserPP,
+    ModifyChaptersPP
 )
 
 
 class TestMetadataFromField(unittest.TestCase):
+
     def test_format_to_regex(self):
-        pp = MetadataFromFieldPP(None, ['title:%(title)s - %(artist)s'])
+        self.assertEqual(
-        self.assertEqual(pp._data[0]['regex'], r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+            MetadataParserPP.format_to_regex('%(title)s - %(artist)s'),
+            r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+        self.assertEqual(MetadataParserPP.format_to_regex(r'(?P<x>.+)'), r'(?P<x>.+)')
 
-    def test_field_to_outtmpl(self):
+    def test_field_to_template(self):
-        pp = MetadataFromFieldPP(None, ['title:%(title)s : %(artist)s'])
+        self.assertEqual(MetadataParserPP.field_to_template('title'), '%(title)s')
-        self.assertEqual(pp._data[0]['tmpl'], '%(title)s')
+        self.assertEqual(MetadataParserPP.field_to_template('1'), '1')
+        self.assertEqual(MetadataParserPP.field_to_template('foo bar'), 'foo bar')
+        self.assertEqual(MetadataParserPP.field_to_template(' literal'), ' literal')
 
-    def test_in_out_seperation(self):
+    def test_metadatafromfield(self):
-        pp = MetadataFromFieldPP(None, ['%(title)s \\: %(artist)s:%(title)s : %(artist)s'])
+        self.assertEqual(
-        self.assertEqual(pp._data[0]['in'], '%(title)s : %(artist)s')
+            MetadataFromFieldPP.to_action('%(title)s \\: %(artist)s:%(title)s : %(artist)s'),
-        self.assertEqual(pp._data[0]['out'], '%(title)s : %(artist)s')
+            (MetadataParserPP.Actions.INTERPRET, '%(title)s : %(artist)s', '%(title)s : %(artist)s'))
-
-
-class TestMetadataFromTitle(unittest.TestCase):
-    def test_format_to_regex(self):
-        pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
-        self.assertEqual(pp._titleregex, r'(?P<title>.+)\ \-\ (?P<artist>.+)')
 
 
 class TestConvertThumbnail(unittest.TestCase):
@@ -60,12 +61,502 @@ class TestConvertThumbnail(unittest.TestCase):
             os.remove(file.format(out))
 
 
-class TestExecAfterDownload(unittest.TestCase):
+class TestExec(unittest.TestCase):
     def test_parse_cmd(self):
-        pp = ExecAfterDownloadPP(YoutubeDL(), '')
+        pp = ExecPP(YoutubeDL(), '')
         info = {'filepath': 'file name'}
-        quoted_filepath = compat_shlex_quote(info['filepath'])
+        cmd = 'echo %s' % compat_shlex_quote(info['filepath'])
 
-        self.assertEqual(pp.parse_cmd('echo', info), 'echo %s' % quoted_filepath)
+        self.assertEqual(pp.parse_cmd('echo', info), cmd)
-        self.assertEqual(pp.parse_cmd('echo.{}', info), 'echo.%s' % quoted_filepath)
+        self.assertEqual(pp.parse_cmd('echo {}', info), cmd)
-        self.assertEqual(pp.parse_cmd('echo "%(filepath)s"', info), 'echo "%s"' % info['filepath'])
+        self.assertEqual(pp.parse_cmd('echo %(filepath)q', info), cmd)
+
+
+class TestModifyChaptersPP(unittest.TestCase):
+    def setUp(self):
+        self._pp = ModifyChaptersPP(YoutubeDL())
+
+    @staticmethod
+    def _sponsor_chapter(start, end, cat, remove=False):
+        c = {'start_time': start, 'end_time': end, '_categories': [(cat, start, end)]}
+        if remove:
+            c['remove'] = True
+        return c
+
+    @staticmethod
+    def _chapter(start, end, title=None, remove=False):
+        c = {'start_time': start, 'end_time': end}
+        if title is not None:
+            c['title'] = title
+        if remove:
+            c['remove'] = True
+        return c
+
+    def _chapters(self, ends, titles):
+        self.assertEqual(len(ends), len(titles))
+        start = 0
+        chapters = []
+        for e, t in zip(ends, titles):
+            chapters.append(self._chapter(start, e, t))
+            start = e
+        return chapters
+
+    def _remove_marked_arrange_sponsors_test_impl(
+            self, chapters, expected_chapters, expected_removed):
+        actual_chapters, actual_removed = (
+            self._pp._remove_marked_arrange_sponsors(chapters))
+        for c in actual_removed:
+            c.pop('title', None)
+            c.pop('_categories', None)
+        actual_chapters = [{
+            'start_time': c['start_time'],
+            'end_time': c['end_time'],
+            'title': c['title'],
+        } for c in actual_chapters]
+        self.assertSequenceEqual(expected_chapters, actual_chapters)
+        self.assertSequenceEqual(expected_removed, actual_removed)
+
+    def test_remove_marked_arrange_sponsors_CanGetThroughUnaltered(self):
+        chapters = self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, chapters, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithSponsors(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 20, 'sponsor'),
+            self._sponsor_chapter(30, 40, 'preview'),
+            self._sponsor_chapter(50, 60, 'sponsor')]
+        expected = self._chapters(
+            [10, 20, 30, 40, 50, 60, 70],
+            ['c', '[SponsorBlock]: Sponsor', 'c', '[SponsorBlock]: Preview/Recap',
+             'c', '[SponsorBlock]: Sponsor', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_UniqueNamesForOverlappingSponsors(self):
+        chapters = self._chapters([120], ['c']) + [
+            self._sponsor_chapter(10, 45, 'sponsor'), self._sponsor_chapter(20, 40, 'selfpromo'),
+            self._sponsor_chapter(50, 70, 'sponsor'), self._sponsor_chapter(60, 85, 'selfpromo'),
+            self._sponsor_chapter(90, 120, 'selfpromo'), self._sponsor_chapter(100, 110, 'sponsor')]
+        expected = self._chapters(
+            [10, 20, 40, 45, 50, 60, 70, 85, 90, 100, 110, 120],
+            ['c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Sponsor, Unpaid/Self Promotion',
+             '[SponsorBlock]: Sponsor',
+             'c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Sponsor, Unpaid/Self Promotion',
+             '[SponsorBlock]: Unpaid/Self Promotion',
+             'c', '[SponsorBlock]: Unpaid/Self Promotion', '[SponsorBlock]: Unpaid/Self Promotion, Sponsor',
+             '[SponsorBlock]: Unpaid/Self Promotion'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithCuts(self):
+        cuts = [self._chapter(10, 20, remove=True),
+                self._sponsor_chapter(30, 40, 'sponsor', remove=True),
+                self._chapter(50, 60, remove=True)]
+        chapters = self._chapters([70], ['c']) + cuts
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([40], ['c']), cuts)
+
+    def test_remove_marked_arrange_sponsors_ChapterWithSponsorsAndCuts(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 20, 'sponsor'),
+            self._sponsor_chapter(30, 40, 'selfpromo', remove=True),
+            self._sponsor_chapter(50, 60, 'interaction')]
+        expected = self._chapters([10, 20, 40, 50, 60],
+                                  ['c', '[SponsorBlock]: Sponsor', 'c',
+                                   '[SponsorBlock]: Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(30, 40, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithSponsorCutInTheMiddle(self):
+        cuts = [self._sponsor_chapter(20, 30, 'selfpromo', remove=True),
+                self._chapter(40, 50, remove=True)]
+        chapters = self._chapters([70], ['c']) + [self._sponsor_chapter(10, 60, 'sponsor')] + cuts
+        expected = self._chapters(
+            [10, 40, 50], ['c', '[SponsorBlock]: Sponsor', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_ChapterWithCutHidingSponsor(self):
+        cuts = [self._sponsor_chapter(20, 50, 'selpromo', remove=True)]
+        chapters = self._chapters([60], ['c']) + [
+            self._sponsor_chapter(10, 20, 'intro'),
+            self._sponsor_chapter(30, 40, 'sponsor'),
+            self._sponsor_chapter(50, 60, 'outro'),
+        ] + cuts
+        expected = self._chapters(
+            [10, 20, 30], ['c', '[SponsorBlock]: Intermission/Intro Animation', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_ChapterWithAdjacentSponsors(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 20, 'sponsor'),
+            self._sponsor_chapter(20, 30, 'selfpromo'),
+            self._sponsor_chapter(30, 40, 'interaction')]
+        expected = self._chapters(
+            [10, 20, 30, 40, 70],
+            ['c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Unpaid/Self Promotion',
+             '[SponsorBlock]: Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithAdjacentCuts(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 20, 'sponsor'),
+            self._sponsor_chapter(20, 30, 'interaction', remove=True),
+            self._chapter(30, 40, remove=True),
+            self._sponsor_chapter(40, 50, 'selpromo', remove=True),
+            self._sponsor_chapter(50, 60, 'interaction')]
+        expected = self._chapters([10, 20, 30, 40],
+                                  ['c', '[SponsorBlock]: Sponsor',
+                                   '[SponsorBlock]: Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(20, 50, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithOverlappingSponsors(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 30, 'sponsor'),
+            self._sponsor_chapter(20, 50, 'selfpromo'),
+            self._sponsor_chapter(40, 60, 'interaction')]
+        expected = self._chapters(
+            [10, 20, 30, 40, 50, 60, 70],
+            ['c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Sponsor, Unpaid/Self Promotion',
+             '[SponsorBlock]: Unpaid/Self Promotion', '[SponsorBlock]: Unpaid/Self Promotion, Interaction Reminder',
+             '[SponsorBlock]: Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithOverlappingCuts(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 30, 'sponsor', remove=True),
+            self._sponsor_chapter(20, 50, 'selfpromo', remove=True),
+            self._sponsor_chapter(40, 60, 'interaction', remove=True)]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([20], ['c']), [self._chapter(10, 60, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithRunsOfOverlappingSponsors(self):
+        chapters = self._chapters([170], ['c']) + [
+            self._sponsor_chapter(0, 30, 'intro'),
+            self._sponsor_chapter(20, 50, 'sponsor'),
+            self._sponsor_chapter(40, 60, 'selfpromo'),
+            self._sponsor_chapter(70, 90, 'sponsor'),
+            self._sponsor_chapter(80, 100, 'sponsor'),
+            self._sponsor_chapter(90, 110, 'sponsor'),
+            self._sponsor_chapter(120, 140, 'selfpromo'),
+            self._sponsor_chapter(130, 160, 'interaction'),
+            self._sponsor_chapter(150, 170, 'outro')]
+        expected = self._chapters(
+            [20, 30, 40, 50, 60, 70, 110, 120, 130, 140, 150, 160, 170],
+            ['[SponsorBlock]: Intermission/Intro Animation', '[SponsorBlock]: Intermission/Intro Animation, Sponsor', '[SponsorBlock]: Sponsor',
+             '[SponsorBlock]: Sponsor, Unpaid/Self Promotion', '[SponsorBlock]: Unpaid/Self Promotion', 'c',
+             '[SponsorBlock]: Sponsor', 'c', '[SponsorBlock]: Unpaid/Self Promotion',
+             '[SponsorBlock]: Unpaid/Self Promotion, Interaction Reminder',
+             '[SponsorBlock]: Interaction Reminder',
+             '[SponsorBlock]: Interaction Reminder, Endcards/Credits', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithRunsOfOverlappingCuts(self):
+        chapters = self._chapters([170], ['c']) + [
+            self._chapter(0, 30, remove=True),
+            self._sponsor_chapter(20, 50, 'sponsor', remove=True),
+            self._chapter(40, 60, remove=True),
+            self._sponsor_chapter(70, 90, 'sponsor', remove=True),
+            self._chapter(80, 100, remove=True),
+            self._chapter(90, 110, remove=True),
+            self._sponsor_chapter(120, 140, 'sponsor', remove=True),
+            self._sponsor_chapter(130, 160, 'selfpromo', remove=True),
+            self._chapter(150, 170, remove=True)]
+        expected_cuts = [self._chapter(0, 60, remove=True),
+                         self._chapter(70, 110, remove=True),
+                         self._chapter(120, 170, remove=True)]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([20], ['c']), expected_cuts)
+
+    def test_remove_marked_arrange_sponsors_OverlappingSponsorsDifferentTitlesAfterCut(self):
+        chapters = self._chapters([60], ['c']) + [
+            self._sponsor_chapter(10, 60, 'sponsor'),
+            self._sponsor_chapter(10, 40, 'intro'),
+            self._sponsor_chapter(30, 50, 'interaction'),
+            self._sponsor_chapter(30, 50, 'selfpromo', remove=True),
+            self._sponsor_chapter(40, 50, 'interaction'),
+            self._sponsor_chapter(50, 60, 'outro')]
+        expected = self._chapters(
+            [10, 30, 40], ['c', '[SponsorBlock]: Sponsor, Intermission/Intro Animation', '[SponsorBlock]: Sponsor, Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(30, 50, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_SponsorsNoLongerOverlapAfterCut(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 30, 'sponsor'),
+            self._sponsor_chapter(20, 50, 'interaction'),
+            self._sponsor_chapter(30, 50, 'selpromo', remove=True),
+            self._sponsor_chapter(40, 60, 'sponsor'),
+            self._sponsor_chapter(50, 60, 'interaction')]
+        expected = self._chapters(
+            [10, 20, 40, 50], ['c', '[SponsorBlock]: Sponsor',
+                               '[SponsorBlock]: Sponsor, Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(30, 50, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_SponsorsStillOverlapAfterCut(self):
+        chapters = self._chapters([70], ['c']) + [
+            self._sponsor_chapter(10, 60, 'sponsor'),
+            self._sponsor_chapter(20, 60, 'interaction'),
+            self._sponsor_chapter(30, 50, 'selfpromo', remove=True)]
+        expected = self._chapters(
+            [10, 20, 40, 50], ['c', '[SponsorBlock]: Sponsor',
+                               '[SponsorBlock]: Sponsor, Interaction Reminder', 'c'])
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, expected, [self._chapter(30, 50, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_ChapterWithRunsOfOverlappingSponsorsAndCuts(self):
+        chapters = self._chapters([200], ['c']) + [
+            self._sponsor_chapter(10, 40, 'sponsor'),
+            self._sponsor_chapter(10, 30, 'intro'),
+            self._chapter(20, 30, remove=True),
+            self._sponsor_chapter(30, 40, 'selfpromo'),
+            self._sponsor_chapter(50, 70, 'sponsor'),
+            self._sponsor_chapter(60, 80, 'interaction'),
+            self._chapter(70, 80, remove=True),
+            self._sponsor_chapter(70, 90, 'sponsor'),
+            self._sponsor_chapter(80, 100, 'interaction'),
+            self._sponsor_chapter(120, 170, 'selfpromo'),
+            self._sponsor_chapter(130, 180, 'outro'),
+            self._chapter(140, 150, remove=True),
+            self._chapter(150, 160, remove=True)]
+        expected = self._chapters(
+            [10, 20, 30, 40, 50, 70, 80, 100, 110, 130, 140, 160],
+            ['c', '[SponsorBlock]: Sponsor, Intermission/Intro Animation', '[SponsorBlock]: Sponsor, Unpaid/Self Promotion',
+             'c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Sponsor, Interaction Reminder',
+             '[SponsorBlock]: Interaction Reminder', 'c', '[SponsorBlock]: Unpaid/Self Promotion',
+             '[SponsorBlock]: Unpaid/Self Promotion, Endcards/Credits', '[SponsorBlock]: Endcards/Credits', 'c'])
+        expected_cuts = [self._chapter(20, 30, remove=True),
+                         self._chapter(70, 80, remove=True),
+                         self._chapter(140, 160, remove=True)]
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, expected_cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorOverlapsMultipleChapters(self):
+        chapters = (self._chapters([20, 40, 60, 80, 100], ['c1', 'c2', 'c3', 'c4', 'c5'])
+                    + [self._sponsor_chapter(10, 90, 'sponsor')])
+        expected = self._chapters([10, 90, 100], ['c1', '[SponsorBlock]: Sponsor', 'c5'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutOverlapsMultipleChapters(self):
+        cuts = [self._chapter(10, 90, remove=True)]
+        chapters = self._chapters([20, 40, 60, 80, 100], ['c1', 'c2', 'c3', 'c4', 'c5']) + cuts
+        expected = self._chapters([10, 20], ['c1', 'c5'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorsWithinSomeChaptersAndOverlappingOthers(self):
+        chapters = (self._chapters([10, 40, 60, 80], ['c1', 'c2', 'c3', 'c4'])
+                    + [self._sponsor_chapter(20, 30, 'sponsor'),
+                       self._sponsor_chapter(50, 70, 'selfpromo')])
+        expected = self._chapters([10, 20, 30, 40, 50, 70, 80],
+                                  ['c1', 'c2', '[SponsorBlock]: Sponsor', 'c2', 'c3',
+                                   '[SponsorBlock]: Unpaid/Self Promotion', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutsWithinSomeChaptersAndOverlappingOthers(self):
+        cuts = [self._chapter(20, 30, remove=True), self._chapter(50, 70, remove=True)]
+        chapters = self._chapters([10, 40, 60, 80], ['c1', 'c2', 'c3', 'c4']) + cuts
+        expected = self._chapters([10, 30, 40, 50], ['c1', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_ChaptersAfterLastSponsor(self):
+        chapters = (self._chapters([20, 40, 50, 60], ['c1', 'c2', 'c3', 'c4'])
+                    + [self._sponsor_chapter(10, 30, 'music_offtopic')])
+        expected = self._chapters(
+            [10, 30, 40, 50, 60],
+            ['c1', '[SponsorBlock]: Non-Music Section', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_ChaptersAfterLastCut(self):
+        cuts = [self._chapter(10, 30, remove=True)]
+        chapters = self._chapters([20, 40, 50, 60], ['c1', 'c2', 'c3', 'c4']) + cuts
+        expected = self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorStartsAtChapterStart(self):
+        chapters = (self._chapters([10, 20, 40], ['c1', 'c2', 'c3'])
+                    + [self._sponsor_chapter(20, 30, 'sponsor')])
+        expected = self._chapters([10, 20, 30, 40], ['c1', 'c2', '[SponsorBlock]: Sponsor', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutStartsAtChapterStart(self):
+        cuts = [self._chapter(20, 30, remove=True)]
+        chapters = self._chapters([10, 20, 40], ['c1', 'c2', 'c3']) + cuts
+        expected = self._chapters([10, 20, 30], ['c1', 'c2', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorEndsAtChapterEnd(self):
+        chapters = (self._chapters([10, 30, 40], ['c1', 'c2', 'c3'])
+                    + [self._sponsor_chapter(20, 30, 'sponsor')])
+        expected = self._chapters([10, 20, 30, 40], ['c1', 'c2', '[SponsorBlock]: Sponsor', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutEndsAtChapterEnd(self):
+        cuts = [self._chapter(20, 30, remove=True)]
+        chapters = self._chapters([10, 30, 40], ['c1', 'c2', 'c3']) + cuts
+        expected = self._chapters([10, 20, 30], ['c1', 'c2', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorCoincidesWithChapters(self):
+        chapters = (self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4'])
+                    + [self._sponsor_chapter(10, 30, 'sponsor')])
+        expected = self._chapters([10, 30, 40], ['c1', '[SponsorBlock]: Sponsor', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutCoincidesWithChapters(self):
+        cuts = [self._chapter(10, 30, remove=True)]
+        chapters = self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4']) + cuts
+        expected = self._chapters([10, 20], ['c1', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorsAtVideoBoundaries(self):
+        chapters = (self._chapters([20, 40, 60], ['c1', 'c2', 'c3'])
+                    + [self._sponsor_chapter(0, 10, 'intro'), self._sponsor_chapter(50, 60, 'outro')])
+        expected = self._chapters(
+            [10, 20, 40, 50, 60], ['[SponsorBlock]: Intermission/Intro Animation', 'c1', 'c2', 'c3', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutsAtVideoBoundaries(self):
+        cuts = [self._chapter(0, 10, remove=True), self._chapter(50, 60, remove=True)]
+        chapters = self._chapters([20, 40, 60], ['c1', 'c2', 'c3']) + cuts
+        expected = self._chapters([10, 30, 40], ['c1', 'c2', 'c3'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_SponsorsOverlapChaptersAtVideoBoundaries(self):
+        chapters = (self._chapters([10, 40, 50], ['c1', 'c2', 'c3'])
+                    + [self._sponsor_chapter(0, 20, 'intro'), self._sponsor_chapter(30, 50, 'outro')])
+        expected = self._chapters(
+            [20, 30, 50], ['[SponsorBlock]: Intermission/Intro Animation', 'c2', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_CutsOverlapChaptersAtVideoBoundaries(self):
+        cuts = [self._chapter(0, 20, remove=True), self._chapter(30, 50, remove=True)]
+        chapters = self._chapters([10, 40, 50], ['c1', 'c2', 'c3']) + cuts
+        expected = self._chapters([10], ['c2'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, cuts)
+
+    def test_remove_marked_arrange_sponsors_EverythingSponsored(self):
+        chapters = (self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4'])
+                    + [self._sponsor_chapter(0, 20, 'intro'), self._sponsor_chapter(20, 40, 'outro')])
+        expected = self._chapters([20, 40], ['[SponsorBlock]: Intermission/Intro Animation', '[SponsorBlock]: Endcards/Credits'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, expected, [])
+
+    def test_remove_marked_arrange_sponsors_EverythingCut(self):
+        cuts = [self._chapter(0, 20, remove=True), self._chapter(20, 40, remove=True)]
+        chapters = self._chapters([10, 20, 30, 40], ['c1', 'c2', 'c3', 'c4']) + cuts
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, [], [self._chapter(0, 40, remove=True)])
+
+    def test_remove_marked_arrange_sponsors_TinyChaptersInTheOriginalArePreserved(self):
+        chapters = self._chapters([0.1, 0.2, 0.3, 0.4], ['c1', 'c2', 'c3', 'c4'])
+        self._remove_marked_arrange_sponsors_test_impl(chapters, chapters, [])
+
+    def test_remove_marked_arrange_sponsors_TinySponsorsAreIgnored(self):
+        chapters = [self._sponsor_chapter(0, 0.1, 'intro'), self._chapter(0.1, 0.2, 'c1'),
+                    self._sponsor_chapter(0.2, 0.3, 'sponsor'), self._chapter(0.3, 0.4, 'c2'),
+                    self._sponsor_chapter(0.4, 0.5, 'outro')]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([0.3, 0.5], ['c1', 'c2']), [])
+
+    def test_remove_marked_arrange_sponsors_TinyChaptersResultingFromCutsAreIgnored(self):
+        cuts = [self._chapter(1.5, 2.5, remove=True)]
+        chapters = self._chapters([2, 3, 3.5], ['c1', 'c2', 'c3']) + cuts
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([2, 2.5], ['c1', 'c3']), cuts)
+
+    def test_remove_marked_arrange_sponsors_SingleTinyChapterIsPreserved(self):
+        cuts = [self._chapter(0.5, 2, remove=True)]
+        chapters = self._chapters([2], ['c']) + cuts
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([0.5], ['c']), cuts)
+
+    def test_remove_marked_arrange_sponsors_TinyChapterAtTheStartPrependedToTheNext(self):
+        cuts = [self._chapter(0.5, 2, remove=True)]
+        chapters = self._chapters([2, 4], ['c1', 'c2']) + cuts
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([2.5], ['c2']), cuts)
+
+    def test_remove_marked_arrange_sponsors_TinyChaptersResultingFromSponsorOverlapAreIgnored(self):
+        chapters = self._chapters([1, 3, 4], ['c1', 'c2', 'c3']) + [
+            self._sponsor_chapter(1.5, 2.5, 'sponsor')]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([1.5, 2.5, 4], ['c1', '[SponsorBlock]: Sponsor', 'c3']), [])
+
+    def test_remove_marked_arrange_sponsors_TinySponsorsOverlapsAreIgnored(self):
+        chapters = self._chapters([2, 3, 5], ['c1', 'c2', 'c3']) + [
+            self._sponsor_chapter(1, 3, 'sponsor'),
+            self._sponsor_chapter(2.5, 4, 'selfpromo')
+        ]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([1, 3, 4, 5], [
+                'c1', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Unpaid/Self Promotion', 'c3']), [])
+
+    def test_remove_marked_arrange_sponsors_TinySponsorsPrependedToTheNextSponsor(self):
+        chapters = self._chapters([4], ['c']) + [
+            self._sponsor_chapter(1.5, 2, 'sponsor'),
+            self._sponsor_chapter(2, 4, 'selfpromo')
+        ]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([1.5, 4], ['c', '[SponsorBlock]: Unpaid/Self Promotion']), [])
+
+    def test_remove_marked_arrange_sponsors_SmallestSponsorInTheOverlapGetsNamed(self):
+        self._pp._sponsorblock_chapter_title = '[SponsorBlock]: %(name)s'
+        chapters = self._chapters([10], ['c']) + [
+            self._sponsor_chapter(2, 8, 'sponsor'),
+            self._sponsor_chapter(4, 6, 'selfpromo')
+        ]
+        self._remove_marked_arrange_sponsors_test_impl(
+            chapters, self._chapters([2, 4, 6, 8, 10], [
+                'c', '[SponsorBlock]: Sponsor', '[SponsorBlock]: Unpaid/Self Promotion',
+                '[SponsorBlock]: Sponsor', 'c'
+            ]), [])
+
+    def test_make_concat_opts_CommonCase(self):
+        sponsor_chapters = [self._chapter(1, 2, 's1'), self._chapter(10, 20, 's2')]
+        expected = '''ffconcat version 1.0
+file 'file:test'
+outpoint 1.000000
+file 'file:test'
+inpoint 2.000000
+outpoint 10.000000
+file 'file:test'
+inpoint 20.000000
+'''
+        opts = self._pp._make_concat_opts(sponsor_chapters, 30)
+        self.assertEqual(expected, ''.join(self._pp._concat_spec(['test'] * len(opts), opts)))
+
+    def test_make_concat_opts_NoZeroDurationChunkAtVideoStart(self):
+        sponsor_chapters = [self._chapter(0, 1, 's1'), self._chapter(10, 20, 's2')]
+        expected = '''ffconcat version 1.0
+file 'file:test'
+inpoint 1.000000
+outpoint 10.000000
+file 'file:test'
+inpoint 20.000000
+'''
+        opts = self._pp._make_concat_opts(sponsor_chapters, 30)
+        self.assertEqual(expected, ''.join(self._pp._concat_spec(['test'] * len(opts), opts)))
+
+    def test_make_concat_opts_NoZeroDurationChunkAtVideoEnd(self):
+        sponsor_chapters = [self._chapter(1, 2, 's1'), self._chapter(10, 20, 's2')]
+        expected = '''ffconcat version 1.0
+file 'file:test'
+outpoint 1.000000
+file 'file:test'
+inpoint 2.000000
+outpoint 10.000000
+'''
+        opts = self._pp._make_concat_opts(sponsor_chapters, 20)
+        self.assertEqual(expected, ''.join(self._pp._concat_spec(['test'] * len(opts), opts)))
+
+    def test_quote_for_concat_RunsOfQuotes(self):
+        self.assertEqual(
+            r"'special '\'' '\'\''characters'\'\'\''galore'",
+            self._pp._quote_for_ffmpeg("special ' ''characters'''galore"))
+
+    def test_quote_for_concat_QuotesAtStart(self):
+        self.assertEqual(
+            r"\'\'\''special '\'' characters '\'' galore'",
+            self._pp._quote_for_ffmpeg("'''special ' characters ' galore"))
+
+    def test_quote_for_concat_QuotesAtEnd(self):
+        self.assertEqual(
+            r"'special '\'' characters '\'' galore'\'\'\'",
+            self._pp._quote_for_ffmpeg("special ' characters ' galore'''"))

test/test_socks.py

@@ -14,6 +14,7 @@ import subprocess
 from test.helper import (
     FakeYDL,
     get_params,
+    is_download_test,
 )
 from yt_dlp.compat import (
     compat_str,
@@ -21,6 +22,7 @@ from yt_dlp.compat import (
 )
 
 
+@is_download_test
 class TestMultipleSocks(unittest.TestCase):
     @staticmethod
     def _check_params(attrs):
@@ -76,6 +78,7 @@ class TestMultipleSocks(unittest.TestCase):
             params['secondary_server_ip'])
 
 
+@is_download_test
 class TestSocks(unittest.TestCase):
     _SKIP_SOCKS_TEST = True
 

test/test_subtitles.py

@@ -7,7 +7,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import FakeYDL, md5
+from test.helper import FakeYDL, md5, is_download_test
 
 
 from yt_dlp.extractor import (
@@ -19,6 +19,7 @@ from yt_dlp.extractor import (
     CeskaTelevizeIE,
     LyndaIE,
     NPOIE,
+    PBSIE,
     ComedyCentralIE,
     NRKTVIE,
     RaiPlayIE,
@@ -30,6 +31,7 @@ from yt_dlp.extractor import (
 )
 
 
+@is_download_test
 class BaseTestSubtitles(unittest.TestCase):
     url = None
     IE = None
@@ -55,6 +57,7 @@ class BaseTestSubtitles(unittest.TestCase):
         return dict((l, sub_info['data']) for l, sub_info in subtitles.items())
 
 
+@is_download_test
 class TestYoutubeSubtitles(BaseTestSubtitles):
     url = 'QRS8MkLhQmM'
     IE = YoutubeIE
@@ -111,6 +114,7 @@ class TestYoutubeSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestDailymotionSubtitles(BaseTestSubtitles):
     url = 'http://www.dailymotion.com/video/xczg00'
     IE = DailymotionIE
@@ -134,6 +138,7 @@ class TestDailymotionSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestTedSubtitles(BaseTestSubtitles):
     url = 'http://www.ted.com/talks/dan_dennett_on_our_consciousness.html'
     IE = TEDIE
@@ -149,6 +154,7 @@ class TestTedSubtitles(BaseTestSubtitles):
             self.assertTrue(subtitles.get(lang) is not None, 'Subtitles for \'%s\' not extracted' % lang)
 
 
+@is_download_test
 class TestVimeoSubtitles(BaseTestSubtitles):
     url = 'http://vimeo.com/76979871'
     IE = VimeoIE
@@ -170,6 +176,7 @@ class TestVimeoSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestWallaSubtitles(BaseTestSubtitles):
     url = 'http://vod.walla.co.il/movie/2705958/the-yes-men'
     IE = WallaIE
@@ -191,6 +198,7 @@ class TestWallaSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestCeskaTelevizeSubtitles(BaseTestSubtitles):
     url = 'http://www.ceskatelevize.cz/ivysilani/10600540290-u6-uzasny-svet-techniky'
     IE = CeskaTelevizeIE
@@ -212,6 +220,7 @@ class TestCeskaTelevizeSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestLyndaSubtitles(BaseTestSubtitles):
     url = 'http://www.lynda.com/Bootstrap-tutorials/Using-exercise-files/110885/114408-4.html'
     IE = LyndaIE
@@ -224,6 +233,7 @@ class TestLyndaSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '09bbe67222259bed60deaa26997d73a7')
 
 
+@is_download_test
 class TestNPOSubtitles(BaseTestSubtitles):
     url = 'http://www.npo.nl/nos-journaal/28-08-2014/POW_00722860'
     IE = NPOIE
@@ -236,6 +246,7 @@ class TestNPOSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['nl']), 'fc6435027572b63fb4ab143abd5ad3f4')
 
 
+@is_download_test
 class TestMTVSubtitles(BaseTestSubtitles):
     url = 'http://www.cc.com/video-clips/p63lk0/adam-devine-s-house-party-chasing-white-swans'
     IE = ComedyCentralIE
@@ -251,6 +262,7 @@ class TestMTVSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '78206b8d8a0cfa9da64dc026eea48961')
 
 
+@is_download_test
 class TestNRKSubtitles(BaseTestSubtitles):
     url = 'http://tv.nrk.no/serie/ikke-gjoer-dette-hjemme/DMPV73000411/sesong-2/episode-1'
     IE = NRKTVIE
@@ -263,6 +275,7 @@ class TestNRKSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['no']), '544fa917d3197fcbee64634559221cc2')
 
 
+@is_download_test
 class TestRaiPlaySubtitles(BaseTestSubtitles):
     IE = RaiPlayIE
 
@@ -283,6 +296,7 @@ class TestRaiPlaySubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['it']), '4b3264186fbb103508abe5311cfcb9cd')
 
 
+@is_download_test
 class TestVikiSubtitles(BaseTestSubtitles):
     url = 'http://www.viki.com/videos/1060846v-punch-episode-18'
     IE = VikiIE
@@ -295,6 +309,7 @@ class TestVikiSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '53cb083a5914b2d84ef1ab67b880d18a')
 
 
+@is_download_test
 class TestThePlatformSubtitles(BaseTestSubtitles):
     # from http://www.3playmedia.com/services-features/tools/integrations/theplatform/
     # (see http://theplatform.com/about/partners/type/subtitles-closed-captioning/)
@@ -309,6 +324,7 @@ class TestThePlatformSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '97e7670cbae3c4d26ae8bcc7fdd78d4b')
 
 
+@is_download_test
 class TestThePlatformFeedSubtitles(BaseTestSubtitles):
     url = 'http://feed.theplatform.com/f/7wvmTC/msnbc_video-p-test?form=json&pretty=true&range=-40&byGuid=n_hardball_5biden_140207'
     IE = ThePlatformFeedIE
@@ -321,6 +337,7 @@ class TestThePlatformFeedSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '48649a22e82b2da21c9a67a395eedade')
 
 
+@is_download_test
 class TestRtveSubtitles(BaseTestSubtitles):
     url = 'http://www.rtve.es/alacarta/videos/los-misterios-de-laura/misterios-laura-capitulo-32-misterio-del-numero-17-2-parte/2428621/'
     IE = RTVEALaCartaIE
@@ -335,6 +352,7 @@ class TestRtveSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['es']), '69e70cae2d40574fb7316f31d6eb7fca')
 
 
+@is_download_test
 class TestDemocracynowSubtitles(BaseTestSubtitles):
     url = 'http://www.democracynow.org/shows/2015/7/3'
     IE = DemocracynowIE
@@ -355,5 +373,42 @@ class TestDemocracynowSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), 'acaca989e24a9e45a6719c9b3d60815c')
 
 
+@is_download_test
+class TestPBSSubtitles(BaseTestSubtitles):
+    url = 'https://www.pbs.org/video/how-fantasy-reflects-our-world-picecq/'
+    IE = PBSIE
+
+    def test_allsubtitles(self):
+        self.DL.params['writesubtitles'] = True
+        self.DL.params['allsubtitles'] = True
+        subtitles = self.getSubtitles()
+        self.assertEqual(set(subtitles.keys()), set(['en']))
+
+    def test_subtitles_dfxp_format(self):
+        self.DL.params['writesubtitles'] = True
+        self.DL.params['subtitlesformat'] = 'dfxp'
+        subtitles = self.getSubtitles()
+        self.assertIn(md5(subtitles['en']), ['643b034254cdc3768ff1e750b6b5873b'])
+
+    def test_subtitles_vtt_format(self):
+        self.DL.params['writesubtitles'] = True
+        self.DL.params['subtitlesformat'] = 'vtt'
+        subtitles = self.getSubtitles()
+        self.assertIn(
+            md5(subtitles['en']), ['937a05711555b165d4c55a9667017045', 'f49ea998d6824d94959c8152a368ff73'])
+
+    def test_subtitles_srt_format(self):
+        self.DL.params['writesubtitles'] = True
+        self.DL.params['subtitlesformat'] = 'srt'
+        subtitles = self.getSubtitles()
+        self.assertIn(md5(subtitles['en']), ['2082c21b43759d9bf172931b2f2ca371'])
+
+    def test_subtitles_sami_format(self):
+        self.DL.params['writesubtitles'] = True
+        self.DL.params['subtitlesformat'] = 'sami'
+        subtitles = self.getSubtitles()
+        self.assertIn(md5(subtitles['en']), ['4256b16ac7da6a6780fafd04294e85cd'])
+
+
 if __name__ == '__main__':
     unittest.main()

test/test_utils.py

@@ -62,6 +62,7 @@ from yt_dlp.utils import (
     parse_iso8601,
     parse_resolution,
     parse_bitrate,
+    parse_qs,
     pkcs1pad,
     read_batch_urls,
     sanitize_filename,
@@ -117,8 +118,6 @@ from yt_dlp.compat import (
     compat_getenv,
     compat_os_name,
     compat_setenv,
-    compat_urlparse,
-    compat_parse_qs,
 )
 
 
@@ -688,38 +687,36 @@ class TestUtil(unittest.TestCase):
         self.assertTrue(isinstance(data, bytes))
 
     def test_update_url_query(self):
-        def query_dict(url):
+        self.assertEqual(parse_qs(update_url_query(
-            return compat_parse_qs(compat_urlparse.urlparse(url).query)
-        self.assertEqual(query_dict(update_url_query(
             'http://example.com/path', {'quality': ['HD'], 'format': ['mp4']})),
-            query_dict('http://example.com/path?quality=HD&format=mp4'))
+            parse_qs('http://example.com/path?quality=HD&format=mp4'))
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'system': ['LINUX', 'WINDOWS']})),
-            query_dict('http://example.com/path?system=LINUX&system=WINDOWS'))
+            parse_qs('http://example.com/path?system=LINUX&system=WINDOWS'))
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'fields': 'id,formats,subtitles'})),
-            query_dict('http://example.com/path?fields=id,formats,subtitles'))
+            parse_qs('http://example.com/path?fields=id,formats,subtitles'))
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'fields': ('id,formats,subtitles', 'thumbnails')})),
-            query_dict('http://example.com/path?fields=id,formats,subtitles&fields=thumbnails'))
+            parse_qs('http://example.com/path?fields=id,formats,subtitles&fields=thumbnails'))
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path?manifest=f4m', {'manifest': []})),
-            query_dict('http://example.com/path'))
+            parse_qs('http://example.com/path'))
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path?system=LINUX&system=WINDOWS', {'system': 'LINUX'})),
-            query_dict('http://example.com/path?system=LINUX'))
+            parse_qs('http://example.com/path?system=LINUX'))
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'fields': b'id,formats,subtitles'})),
-            query_dict('http://example.com/path?fields=id,formats,subtitles'))
+            parse_qs('http://example.com/path?fields=id,formats,subtitles'))
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'width': 1080, 'height': 720})),
-            query_dict('http://example.com/path?width=1080&height=720'))
+            parse_qs('http://example.com/path?width=1080&height=720'))
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'bitrate': 5020.43})),
-            query_dict('http://example.com/path?bitrate=5020.43'))
+            parse_qs('http://example.com/path?bitrate=5020.43'))
-        self.assertEqual(query_dict(update_url_query(
+        self.assertEqual(parse_qs(update_url_query(
             'http://example.com/path', {'test': '第二行тест'})),
-            query_dict('http://example.com/path?test=%E7%AC%AC%E4%BA%8C%E8%A1%8C%D1%82%D0%B5%D1%81%D1%82'))
+            parse_qs('http://example.com/path?test=%E7%AC%AC%E4%BA%8C%E8%A1%8C%D1%82%D0%B5%D1%81%D1%82'))
 
     def test_multipart_encode(self):
         self.assertEqual(
@@ -851,30 +848,52 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(parse_codecs('avc1.77.30, mp4a.40.2'), {
             'vcodec': 'avc1.77.30',
             'acodec': 'mp4a.40.2',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('mp4a.40.2'), {
             'vcodec': 'none',
             'acodec': 'mp4a.40.2',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('mp4a.40.5,avc1.42001e'), {
             'vcodec': 'avc1.42001e',
             'acodec': 'mp4a.40.5',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('avc3.640028'), {
             'vcodec': 'avc3.640028',
             'acodec': 'none',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs(', h264,,newcodec,aac'), {
             'vcodec': 'h264',
             'acodec': 'aac',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('av01.0.05M.08'), {
             'vcodec': 'av01.0.05M.08',
             'acodec': 'none',
+            'dynamic_range': None,
+        })
+        self.assertEqual(parse_codecs('vp9.2'), {
+            'vcodec': 'vp9.2',
+            'acodec': 'none',
+            'dynamic_range': 'HDR10',
+        })
+        self.assertEqual(parse_codecs('av01.0.12M.10.0.110.09.16.09.0'), {
+            'vcodec': 'av01.0.12M.10',
+            'acodec': 'none',
+            'dynamic_range': 'HDR10',
+        })
+        self.assertEqual(parse_codecs('dvhe'), {
+            'vcodec': 'dvhe',
+            'acodec': 'none',
+            'dynamic_range': 'DV',
         })
         self.assertEqual(parse_codecs('theora, vorbis'), {
             'vcodec': 'theora',
             'acodec': 'vorbis',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('unknownvcodec, unknownacodec'), {
             'vcodec': 'unknownvcodec',
@@ -1054,6 +1073,9 @@ class TestUtil(unittest.TestCase):
         on = js_to_json('{ "040": "040" }')
         self.assertEqual(json.loads(on), {'040': '040'})
 
+        on = js_to_json('[1,//{},\n2]')
+        self.assertEqual(json.loads(on), [1, 2])
+
     def test_js_to_json_malformed(self):
         self.assertEqual(js_to_json('42a1'), '42"a1"')
         self.assertEqual(js_to_json('42a-1'), '42"a"-1')
@@ -1141,12 +1163,15 @@ class TestUtil(unittest.TestCase):
     def test_parse_resolution(self):
         self.assertEqual(parse_resolution(None), {})
         self.assertEqual(parse_resolution(''), {})
-        self.assertEqual(parse_resolution('1920x1080'), {'width': 1920, 'height': 1080})
+        self.assertEqual(parse_resolution(' 1920x1080'), {'width': 1920, 'height': 1080})
-        self.assertEqual(parse_resolution('1920×1080'), {'width': 1920, 'height': 1080})
+        self.assertEqual(parse_resolution('1920×1080 '), {'width': 1920, 'height': 1080})
         self.assertEqual(parse_resolution('1920 x 1080'), {'width': 1920, 'height': 1080})
         self.assertEqual(parse_resolution('720p'), {'height': 720})
         self.assertEqual(parse_resolution('4k'), {'height': 2160})
         self.assertEqual(parse_resolution('8K'), {'height': 4320})
+        self.assertEqual(parse_resolution('pre_1920x1080_post'), {'width': 1920, 'height': 1080})
+        self.assertEqual(parse_resolution('ep1x2'), {})
+        self.assertEqual(parse_resolution('1920, 1080'), {'width': 1920, 'height': 1080})
 
     def test_parse_bitrate(self):
         self.assertEqual(parse_bitrate(None), None)
@@ -1204,35 +1229,12 @@ ffmpeg version 2.4.4 Copyright (c) 2000-2014 the FFmpeg ...'''), '2.4.4')
             '9999 51')
 
     def test_match_str(self):
-        self.assertRaises(ValueError, match_str, 'xy>foobar', {})
+        # Unary
         self.assertFalse(match_str('xy', {'x': 1200}))
         self.assertTrue(match_str('!xy', {'x': 1200}))
         self.assertTrue(match_str('x', {'x': 1200}))
         self.assertFalse(match_str('!x', {'x': 1200}))
         self.assertTrue(match_str('x', {'x': 0}))
-        self.assertFalse(match_str('x>0', {'x': 0}))
-        self.assertFalse(match_str('x>0', {}))
-        self.assertTrue(match_str('x>?0', {}))
-        self.assertTrue(match_str('x>1K', {'x': 1200}))
-        self.assertFalse(match_str('x>2K', {'x': 1200}))
-        self.assertTrue(match_str('x>=1200 & x < 1300', {'x': 1200}))
-        self.assertFalse(match_str('x>=1100 & x < 1200', {'x': 1200}))
-        self.assertFalse(match_str('y=a212', {'y': 'foobar42'}))
-        self.assertTrue(match_str('y=foobar42', {'y': 'foobar42'}))
-        self.assertFalse(match_str('y!=foobar42', {'y': 'foobar42'}))
-        self.assertTrue(match_str('y!=foobar2', {'y': 'foobar42'}))
-        self.assertFalse(match_str(
-            'like_count > 100 & dislike_count <? 50 & description',
-            {'like_count': 90, 'description': 'foo'}))
-        self.assertTrue(match_str(
-            'like_count > 100 & dislike_count <? 50 & description',
-            {'like_count': 190, 'description': 'foo'}))
-        self.assertFalse(match_str(
-            'like_count > 100 & dislike_count <? 50 & description',
-            {'like_count': 190, 'dislike_count': 60, 'description': 'foo'}))
-        self.assertFalse(match_str(
-            'like_count > 100 & dislike_count <? 50 & description',
-            {'like_count': 190, 'dislike_count': 10}))
         self.assertTrue(match_str('is_live', {'is_live': True}))
         self.assertFalse(match_str('is_live', {'is_live': False}))
         self.assertFalse(match_str('is_live', {'is_live': None}))
@@ -1246,6 +1248,76 @@ ffmpeg version 2.4.4 Copyright (c) 2000-2014 the FFmpeg ...'''), '2.4.4')
         self.assertFalse(match_str('!title', {'title': 'abc'}))
         self.assertFalse(match_str('!title', {'title': ''}))
 
+        # Numeric
+        self.assertFalse(match_str('x>0', {'x': 0}))
+        self.assertFalse(match_str('x>0', {}))
+        self.assertTrue(match_str('x>?0', {}))
+        self.assertTrue(match_str('x>1K', {'x': 1200}))
+        self.assertFalse(match_str('x>2K', {'x': 1200}))
+        self.assertTrue(match_str('x>=1200 & x < 1300', {'x': 1200}))
+        self.assertFalse(match_str('x>=1100 & x < 1200', {'x': 1200}))
+        self.assertTrue(match_str('x > 1:0:0', {'x': 3700}))
+
+        # String
+        self.assertFalse(match_str('y=a212', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y=foobar42', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y!=foobar42', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y!=foobar2', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y^=foo', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y!^=foo', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y^=bar', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y!^=bar', {'y': 'foobar42'}))
+        self.assertRaises(ValueError, match_str, 'x^=42', {'x': 42})
+        self.assertTrue(match_str('y*=bar', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y!*=bar', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y*=baz', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y!*=baz', {'y': 'foobar42'}))
+        self.assertTrue(match_str('y$=42', {'y': 'foobar42'}))
+        self.assertFalse(match_str('y$=43', {'y': 'foobar42'}))
+
+        # And
+        self.assertFalse(match_str(
+            'like_count > 100 & dislike_count <? 50 & description',
+            {'like_count': 90, 'description': 'foo'}))
+        self.assertTrue(match_str(
+            'like_count > 100 & dislike_count <? 50 & description',
+            {'like_count': 190, 'description': 'foo'}))
+        self.assertFalse(match_str(
+            'like_count > 100 & dislike_count <? 50 & description',
+            {'like_count': 190, 'dislike_count': 60, 'description': 'foo'}))
+        self.assertFalse(match_str(
+            'like_count > 100 & dislike_count <? 50 & description',
+            {'like_count': 190, 'dislike_count': 10}))
+
+        # Regex
+        self.assertTrue(match_str(r'x~=\bbar', {'x': 'foo bar'}))
+        self.assertFalse(match_str(r'x~=\bbar.+', {'x': 'foo bar'}))
+        self.assertFalse(match_str(r'x~=^FOO', {'x': 'foo bar'}))
+        self.assertTrue(match_str(r'x~=(?i)^FOO', {'x': 'foo bar'}))
+
+        # Quotes
+        self.assertTrue(match_str(r'x^="foo"', {'x': 'foo "bar"'}))
+        self.assertFalse(match_str(r'x^="foo  "', {'x': 'foo "bar"'}))
+        self.assertFalse(match_str(r'x$="bar"', {'x': 'foo "bar"'}))
+        self.assertTrue(match_str(r'x$=" \"bar\""', {'x': 'foo "bar"'}))
+
+        # Escaping &
+        self.assertFalse(match_str(r'x=foo & bar', {'x': 'foo & bar'}))
+        self.assertTrue(match_str(r'x=foo \& bar', {'x': 'foo & bar'}))
+        self.assertTrue(match_str(r'x=foo \& bar & x^=foo', {'x': 'foo & bar'}))
+        self.assertTrue(match_str(r'x="foo \& bar" & x^=foo', {'x': 'foo & bar'}))
+
+        # Example from docs
+        self.assertTrue(match_str(
+            r"!is_live & like_count>?100 & description~='(?i)\bcats \& dogs\b'",
+            {'description': 'Raining Cats & Dogs'}))
+
+        # Incomplete
+        self.assertFalse(match_str('id!=foo', {'id': 'foo'}, True))
+        self.assertTrue(match_str('x', {'id': 'foo'}, True))
+        self.assertTrue(match_str('!x', {'id': 'foo'}, True))
+        self.assertFalse(match_str('x', {'id': 'foo'}, False))
+
     def test_parse_dfxp_time_expr(self):
         self.assertEqual(parse_dfxp_time_expr(None), None)
         self.assertEqual(parse_dfxp_time_expr(''), None)
@@ -1321,21 +1393,21 @@ The first line
   </body>
 </tt>'''.encode('utf-8')
         srt_data = '''1
-00:00:02,080 --> 00:00:05,839
+00:00:02,080 --> 00:00:05,840
 <font color="white" face="sansSerif" size="16">default style<font color="red">custom style</font></font>
 
 2
-00:00:02,080 --> 00:00:05,839
+00:00:02,080 --> 00:00:05,840
 <b><font color="cyan" face="sansSerif" size="16"><font color="lime">part 1
 </font>part 2</font></b>
 
 3
-00:00:05,839 --> 00:00:09,560
+00:00:05,840 --> 00:00:09,560
 <u><font color="lime">line 3
 part 3</font></u>
 
 4
-00:00:09,560 --> 00:00:12,359
+00:00:09,560 --> 00:00:12,360
 <i><u><font color="yellow"><font color="lime">inner
  </font>style</font></u></i>
 
@@ -1534,8 +1606,11 @@ Line 1
         self.assertEqual(LazyList(it).exhaust(), it)
         self.assertEqual(LazyList(it)[5], it[5])
 
+        self.assertEqual(LazyList(it)[5:], it[5:])
+        self.assertEqual(LazyList(it)[:5], it[:5])
         self.assertEqual(LazyList(it)[::2], it[::2])
         self.assertEqual(LazyList(it)[1::2], it[1::2])
+        self.assertEqual(LazyList(it)[5::-1], it[5::-1])
         self.assertEqual(LazyList(it)[6:2:-2], it[6:2:-2])
         self.assertEqual(LazyList(it)[::-1], it[::-1])
 
@@ -1545,8 +1620,9 @@ Line 1
         self.assertEqual(repr(LazyList(it)), repr(it))
         self.assertEqual(str(LazyList(it)), str(it))
 
-        self.assertEqual(list(reversed(LazyList(it))), it[::-1])
+        self.assertEqual(list(LazyList(it).reverse()), it[::-1])
-        self.assertEqual(list(reversed(LazyList(it))[1:3:7]), it[::-1][1:3:7])
+        self.assertEqual(list(LazyList(it).reverse()[1:3:7]), it[::-1][1:3:7])
+        self.assertEqual(list(LazyList(it).reverse()[::-1]), it)
 
     def test_LazyList_laziness(self):
 
@@ -1559,13 +1635,13 @@ Line 1
         test(ll, 5, 5, range(6))
         test(ll, -3, 7, range(10))
 
-        ll = reversed(LazyList(range(10)))
+        ll = LazyList(range(10)).reverse()
         test(ll, -1, 0, range(1))
         test(ll, 3, 6, range(10))
 
         ll = LazyList(itertools.count())
         test(ll, 10, 10, range(11))
-        reversed(ll)
+        ll.reverse()
         test(ll, -15, 14, range(15))
 
 

test/test_write_annotations.py.disabled

@@ -8,7 +8,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import get_params, try_rm
+from test.helper import get_params, try_rm, is_download_test
 
 
 import io
@@ -38,6 +38,7 @@ ANNOTATIONS_FILE = TEST_ID + '.annotations.xml'
 EXPECTED_ANNOTATIONS = ['Speech bubble', 'Note', 'Title', 'Spotlight', 'Label']
 
 
+@is_download_test
 class TestAnnotations(unittest.TestCase):
     def setUp(self):
         # Clear old files

test/test_youtube_lists.py

@@ -7,7 +7,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import FakeYDL
+from test.helper import FakeYDL, is_download_test
 
 
 from yt_dlp.extractor import (
@@ -17,6 +17,7 @@ from yt_dlp.extractor import (
 )
 
 
+@is_download_test
 class TestYoutubeLists(unittest.TestCase):
     def assertIsPlaylist(self, info):
         """Make sure the info has '_type' set to 'playlist'"""

test/test_youtube_signature.py

@@ -12,11 +12,12 @@ import io
 import re
 import string
 
-from test.helper import FakeYDL
+from test.helper import FakeYDL, is_download_test
 from yt_dlp.extractor import YoutubeIE
+from yt_dlp.jsinterp import JSInterpreter
 from yt_dlp.compat import compat_str, compat_urlretrieve
 
-_TESTS = [
+_SIG_TESTS = [
     (
         'https://s.ytimg.com/yts/jsbin/html5player-vflHOr_nV.js',
         86,
@@ -64,7 +65,19 @@ _TESTS = [
     )
 ]
 
+_NSIG_TESTS = [
+    (
+        'https://www.youtube.com/s/player/9216d1f7/player_ias.vflset/en_US/base.js',
+        'SLp9F5bwjAdhE9F-', 'gWnb9IK2DJ8Q1w',
+    ),
+    (
+        'https://www.youtube.com/s/player/f8cb7a3b/player_ias.vflset/en_US/base.js',
+        'oBo2h5euWy6osrUt', 'ivXHpm7qJjJN',
+    ),
+]
 
+
+@is_download_test
 class TestPlayerInfo(unittest.TestCase):
     def test_youtube_extract_player_info(self):
         PLAYER_URLS = (
@@ -87,6 +100,7 @@ class TestPlayerInfo(unittest.TestCase):
             self.assertEqual(player_id, expected_player_id)
 
 
+@is_download_test
 class TestSignature(unittest.TestCase):
     def setUp(self):
         TEST_DIR = os.path.dirname(os.path.abspath(__file__))
@@ -95,35 +109,49 @@ class TestSignature(unittest.TestCase):
             os.mkdir(self.TESTDATA_DIR)
 
 
-def make_tfunc(url, sig_input, expected_sig):
+def t_factory(name, sig_func, url_pattern):
-    m = re.match(r'.*-([a-zA-Z0-9_-]+)(?:/watch_as3|/html5player)?\.[a-z]+$', url)
+    def make_tfunc(url, sig_input, expected_sig):
-    assert m, '%r should follow URL format' % url
+        m = url_pattern.match(url)
-    test_id = m.group(1)
+        assert m, '%r should follow URL format' % url
+        test_id = m.group('id')
 
-    def test_func(self):
+        def test_func(self):
-        basename = 'player-%s.js' % test_id
+            basename = f'player-{name}-{test_id}.js'
-        fn = os.path.join(self.TESTDATA_DIR, basename)
+            fn = os.path.join(self.TESTDATA_DIR, basename)
 
-        if not os.path.exists(fn):
+            if not os.path.exists(fn):
-            compat_urlretrieve(url, fn)
+                compat_urlretrieve(url, fn)
+            with io.open(fn, encoding='utf-8') as testf:
+                jscode = testf.read()
+            self.assertEqual(sig_func(jscode, sig_input), expected_sig)
 
-        ydl = FakeYDL()
+        test_func.__name__ = f'test_{name}_js_{test_id}'
-        ie = YoutubeIE(ydl)
+        setattr(TestSignature, test_func.__name__, test_func)
-        with io.open(fn, encoding='utf-8') as testf:
+    return make_tfunc
-            jscode = testf.read()
-        func = ie._parse_sig_js(jscode)
-        src_sig = (
-            compat_str(string.printable[:sig_input])
-            if isinstance(sig_input, int) else sig_input)
-        got_sig = func(src_sig)
-        self.assertEqual(got_sig, expected_sig)
-
-    test_func.__name__ = str('test_signature_js_' + test_id)
-    setattr(TestSignature, test_func.__name__, test_func)
 
 
-for test_spec in _TESTS:
+def signature(jscode, sig_input):
-    make_tfunc(*test_spec)
+    func = YoutubeIE(FakeYDL())._parse_sig_js(jscode)
+    src_sig = (
+        compat_str(string.printable[:sig_input])
+        if isinstance(sig_input, int) else sig_input)
+    return func(src_sig)
+
+
+def n_sig(jscode, sig_input):
+    funcname = YoutubeIE(FakeYDL())._extract_n_function_name(jscode)
+    return JSInterpreter(jscode).call_function(funcname, sig_input)
+
+
+make_sig_test = t_factory(
+    'signature', signature, re.compile(r'.*-(?P<id>[a-zA-Z0-9_-]+)(?:/watch_as3|/html5player)?\.[a-z]+$'))
+for test_spec in _SIG_TESTS:
+    make_sig_test(*test_spec)
+
+make_nsig_test = t_factory(
+    'nsig', n_sig, re.compile(r'.+/player/(?P<id>[a-zA-Z0-9_-]+)/.+.js$'))
+for test_spec in _NSIG_TESTS:
+    make_nsig_test(*test_spec)
 
 
 if __name__ == '__main__':

tox.ini

@@ -1,5 +1,7 @@
 [tox]
 envlist = py26,py27,py33,py34,py35
+
+# Needed?
 [testenv]
 deps =
    nose

yt_dlp/__init__.py

@@ -1,25 +1,27 @@
 #!/usr/bin/env python3
 # coding: utf-8
 
-from __future__ import unicode_literals
+f'You are using an unsupported version of Python. Only Python versions 3.6 and above are supported by yt-dlp'  # noqa: F541
 
 __license__ = 'Public Domain'
 
 import codecs
 import io
+import itertools
 import os
 import random
 import re
 import sys
 
-
 from .options import (
     parseOpts,
 )
 from .compat import (
     compat_getpass,
+    compat_shlex_quote,
     workaround_optparse_bug9161,
 )
+from .cookies import SUPPORTED_BROWSERS
 from .utils import (
     DateRange,
     decodeOption,
@@ -27,8 +29,11 @@ from .utils import (
     error_to_compat_str,
     ExistingVideoReached,
     expand_path,
+    float_or_none,
+    int_or_none,
     match_filter_func,
     MaxDownloadsReached,
+    parse_duration,
     preferredencoding,
     read_batch_urls,
     RejectedVideoReached,
@@ -45,14 +50,15 @@ from .downloader import (
 from .extractor import gen_extractors, list_extractors
 from .extractor.common import InfoExtractor
 from .extractor.adobepass import MSO_INFO
-from .postprocessor.ffmpeg import (
+from .postprocessor import (
     FFmpegExtractAudioPP,
     FFmpegSubtitlesConvertorPP,
     FFmpegThumbnailsConvertorPP,
     FFmpegVideoConvertorPP,
     FFmpegVideoRemuxerPP,
+    MetadataFromFieldPP,
+    MetadataParserPP,
 )
-from .postprocessor.metadatafromfield import MetadataFromFieldPP
 from .YoutubeDL import YoutubeDL
 
 
@@ -106,22 +112,22 @@ def _real_main(argv=None):
 
     if opts.list_extractors:
         for ie in list_extractors(opts.age_limit):
-            write_string(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie._WORKING else '') + '\n', out=sys.stdout)
+            write_string(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie.working() else '') + '\n', out=sys.stdout)
             matchedUrls = [url for url in all_urls if ie.suitable(url)]
             for mu in matchedUrls:
                 write_string('  ' + mu + '\n', out=sys.stdout)
         sys.exit(0)
     if opts.list_extractor_descriptions:
         for ie in list_extractors(opts.age_limit):
-            if not ie._WORKING:
+            if not ie.working():
                 continue
             desc = getattr(ie, 'IE_DESC', ie.IE_NAME)
             if desc is False:
                 continue
-            if hasattr(ie, 'SEARCH_KEY'):
+            if getattr(ie, 'SEARCH_KEY', None) is not None:
                 _SEARCHES = ('cute kittens', 'slithering pythons', 'falling cat', 'angry poodle', 'purple fish', 'running tortoise', 'sleeping bunny', 'burping cow')
                 _COUNTS = ('', '5', '10', 'all')
-                desc += ' (Example: "%s%s:%s" )' % (ie.SEARCH_KEY, random.choice(_COUNTS), random.choice(_SEARCHES))
+                desc += f'; "{ie.SEARCH_KEY}:" prefix (Example: "{ie.SEARCH_KEY}{random.choice(_COUNTS)}:{random.choice(_SEARCHES)}")'
             write_string(desc + '\n', out=sys.stdout)
         sys.exit(0)
     if opts.ap_list_mso:
@@ -221,11 +227,13 @@ def _real_main(argv=None):
     if opts.playlistend not in (-1, None) and opts.playlistend < opts.playliststart:
         raise ValueError('Playlist end must be greater than playlist start')
     if opts.extractaudio:
+        opts.audioformat = opts.audioformat.lower()
         if opts.audioformat not in ['best'] + list(FFmpegExtractAudioPP.SUPPORTED_EXTS):
             parser.error('invalid audio format specified')
     if opts.audioquality:
         opts.audioquality = opts.audioquality.strip('k').strip('K')
-        if not opts.audioquality.isdigit():
+        audioquality = int_or_none(float_or_none(opts.audioquality))  # int_or_none prevents inf, nan
+        if audioquality is None or audioquality < 0:
             parser.error('invalid audio quality specified')
     if opts.recodevideo is not None:
         opts.recodevideo = opts.recodevideo.replace(' ', '')
@@ -242,40 +250,21 @@ def _real_main(argv=None):
         if opts.convertthumbnails not in FFmpegThumbnailsConvertorPP.SUPPORTED_EXTS:
             parser.error('invalid thumbnail format specified')
 
+    if opts.cookiesfrombrowser is not None:
+        opts.cookiesfrombrowser = [
+            part.strip() or None for part in opts.cookiesfrombrowser.split(':', 1)]
+        if opts.cookiesfrombrowser[0].lower() not in SUPPORTED_BROWSERS:
+            parser.error('unsupported browser specified for cookies')
+
     if opts.date is not None:
         date = DateRange.day(opts.date)
     else:
         date = DateRange(opts.dateafter, opts.datebefore)
 
-    def parse_compat_opts():
+    compat_opts = opts.compat_opts
-        parsed_compat_opts, compat_opts = set(), opts.compat_opts[::-1]
-        while compat_opts:
-            actual_opt = opt = compat_opts.pop().lower()
-            if opt == 'youtube-dl':
-                compat_opts.extend(['-multistreams', 'all'])
-            elif opt == 'youtube-dlc':
-                compat_opts.extend(['-no-youtube-channel-redirect', '-no-live-chat', 'all'])
-            elif opt == 'all':
-                parsed_compat_opts.update(all_compat_opts)
-            elif opt == '-all':
-                parsed_compat_opts = set()
-            else:
-                if opt[0] == '-':
-                    opt = opt[1:]
-                    parsed_compat_opts.discard(opt)
-                else:
-                    parsed_compat_opts.update([opt])
-                if opt not in all_compat_opts:
-                    parser.error('Invalid compatibility option %s' % actual_opt)
-        return parsed_compat_opts
 
-    all_compat_opts = [
+    def report_conflict(arg1, arg2):
-        'filename', 'format-sort', 'abort-on-error', 'format-spec', 'no-playlist-metafiles',
+        warnings.append(f'{arg2} is ignored since {arg1} was given')
-        'multistreams', 'no-live-chat', 'playlist-index', 'list-formats', 'no-direct-merge',
-        'no-youtube-channel-redirect', 'no-youtube-unavailable-videos', 'no-attach-info-json',
-        'embed-thumbnail-atomicparsley',
-    ]
-    compat_opts = parse_compat_opts()
 
     def _unused_compat_opt(name):
         if name not in compat_opts:
@@ -284,7 +273,7 @@ def _real_main(argv=None):
         compat_opts.update(['*%s' % name])
         return True
 
-    def set_default_compat(compat_name, opt_name, default=True, remove_compat=False):
+    def set_default_compat(compat_name, opt_name, default=True, remove_compat=True):
         attr = getattr(opts, opt_name)
         if compat_name in compat_opts:
             if attr is None:
@@ -298,8 +287,9 @@ def _real_main(argv=None):
             setattr(opts, opt_name, default)
         return None
 
-    set_default_compat('abort-on-error', 'ignoreerrors')
+    set_default_compat('abort-on-error', 'ignoreerrors', 'only_download')
     set_default_compat('no-playlist-metafiles', 'allow_playlist_files')
+    set_default_compat('no-clean-infojson', 'clean_infojson')
     if 'format-sort' in compat_opts:
         opts.format_sort.extend(InfoExtractor.FormatSort.ytdl_default)
     _video_multistreams_set = set_default_compat('multistreams', 'allow_multiple_video_streams', False, remove_compat=False)
@@ -307,10 +297,14 @@ def _real_main(argv=None):
     if _video_multistreams_set is False and _audio_multistreams_set is False:
         _unused_compat_opt('multistreams')
     outtmpl_default = opts.outtmpl.get('default')
+    if opts.useid:
+        if outtmpl_default is None:
+            outtmpl_default = opts.outtmpl['default'] = '%(id)s.%(ext)s'
+        else:
+            report_conflict('--output', '--id')
     if 'filename' in compat_opts:
         if outtmpl_default is None:
-            outtmpl_default = '%(title)s.%(id)s.%(ext)s'
+            outtmpl_default = opts.outtmpl['default'] = '%(title)s-%(id)s.%(ext)s'
-            opts.outtmpl.update({'default': outtmpl_default})
         else:
             _unused_compat_opt('filename')
 
@@ -320,9 +314,14 @@ def _real_main(argv=None):
             parser.error('invalid %s %r: %s' % (msg, tmpl, error_to_compat_str(err)))
 
     for k, tmpl in opts.outtmpl.items():
-        validate_outtmpl(tmpl, '%s output template' % k)
+        validate_outtmpl(tmpl, f'{k} output template')
-    for tmpl in opts.forceprint:
+    opts.forceprint = opts.forceprint or []
+    for tmpl in opts.forceprint or []:
         validate_outtmpl(tmpl, 'print template')
+    validate_outtmpl(opts.sponsorblock_chapter_title, 'SponsorBlock chapter title')
+    for k, tmpl in opts.progress_template.items():
+        k = f'{k[:-6]} console title' if '-title' in k else f'{k} progress'
+        validate_outtmpl(tmpl, f'{k} template')
 
     if opts.extractaudio and not opts.keepvideo and opts.format is None:
         opts.format = 'bestaudio/best'
@@ -336,13 +335,29 @@ def _real_main(argv=None):
         if re.match(InfoExtractor.FormatSort.regex, f) is None:
             parser.error('invalid format sort string "%s" specified' % f)
 
-    if opts.metafromfield is None:
+    def metadataparser_actions(f):
-        opts.metafromfield = []
+        if isinstance(f, str):
+            cmd = '--parse-metadata %s' % compat_shlex_quote(f)
+            try:
+                actions = [MetadataFromFieldPP.to_action(f)]
+            except Exception as err:
+                parser.error(f'{cmd} is invalid; {err}')
+        else:
+            cmd = '--replace-in-metadata %s' % ' '.join(map(compat_shlex_quote, f))
+            actions = ((MetadataParserPP.Actions.REPLACE, x, *f[1:]) for x in f[0].split(','))
+
+        for action in actions:
+            try:
+                MetadataParserPP.validate_action(*action)
+            except Exception as err:
+                parser.error(f'{cmd} is invalid; {err}')
+            yield action
+
+    if opts.parse_metadata is None:
+        opts.parse_metadata = []
     if opts.metafromtitle is not None:
-        opts.metafromfield.append('title:%s' % opts.metafromtitle)
+        opts.parse_metadata.append('title:%s' % opts.metafromtitle)
-    for f in opts.metafromfield:
+    opts.parse_metadata = list(itertools.chain(*map(metadataparser_actions, opts.parse_metadata)))
-        if re.match(MetadataFromFieldPP.regex, f) is None:
-            parser.error('invalid format string "%s" specified for --parse-metadata' % f)
 
     any_getting = opts.forceprint or opts.geturl or opts.gettitle or opts.getid or opts.getthumbnail or opts.getdescription or opts.getfilename or opts.getformat or opts.getduration or opts.dumpjson or opts.dump_single_json
     any_printing = opts.print_json
@@ -353,15 +368,31 @@ def _real_main(argv=None):
     if opts.getcomments and not printing_json:
         opts.writeinfojson = True
 
-    def report_conflict(arg1, arg2):
+    if opts.no_sponsorblock:
-        warnings.append('%s is ignored since %s was given' % (arg2, arg1))
+        opts.sponsorblock_mark = set()
+        opts.sponsorblock_remove = set()
+    sponsorblock_query = opts.sponsorblock_mark | opts.sponsorblock_remove
+
+    if (opts.addmetadata or opts.sponsorblock_mark) and opts.addchapters is None:
+        opts.addchapters = True
+    opts.remove_chapters = opts.remove_chapters or []
+
+    if (opts.remove_chapters or sponsorblock_query) and opts.sponskrub is not False:
+        if opts.sponskrub:
+            if opts.remove_chapters:
+                report_conflict('--remove-chapters', '--sponskrub')
+            if opts.sponsorblock_mark:
+                report_conflict('--sponsorblock-mark', '--sponskrub')
+            if opts.sponsorblock_remove:
+                report_conflict('--sponsorblock-remove', '--sponskrub')
+        opts.sponskrub = False
+    if opts.sponskrub_cut and opts.split_chapters and opts.sponskrub is not False:
+        report_conflict('--split-chapter', '--sponskrub-cut')
+        opts.sponskrub_cut = False
 
     if opts.remuxvideo and opts.recodevideo:
         report_conflict('--recode-video', '--remux-video')
         opts.remuxvideo = False
-    if opts.sponskrub_cut and opts.split_chapters and opts.sponskrub is not False:
-        report_conflict('--split-chapter', '--sponskrub-cut')
-        opts.sponskrub_cut = False
 
     if opts.allow_unplayable_formats:
         if opts.extractaudio:
@@ -388,16 +419,30 @@ def _real_main(argv=None):
         if opts.fixup and opts.fixup.lower() not in ('never', 'ignore'):
             report_conflict('--allow-unplayable-formats', '--fixup')
         opts.fixup = 'never'
+        if opts.remove_chapters:
+            report_conflict('--allow-unplayable-formats', '--remove-chapters')
+            opts.remove_chapters = []
+        if opts.sponsorblock_remove:
+            report_conflict('--allow-unplayable-formats', '--sponsorblock-remove')
+            opts.sponsorblock_remove = set()
         if opts.sponskrub:
             report_conflict('--allow-unplayable-formats', '--sponskrub')
         opts.sponskrub = False
 
     # PostProcessors
-    postprocessors = []
+    postprocessors = list(opts.add_postprocessors)
-    if opts.metafromfield:
+    if sponsorblock_query:
         postprocessors.append({
-            'key': 'MetadataFromField',
+            'key': 'SponsorBlock',
-            'formats': opts.metafromfield,
+            'categories': sponsorblock_query,
+            'api': opts.sponsorblock_api,
+            # Run this immediately after extraction is complete
+            'when': 'pre_process'
+        })
+    if opts.parse_metadata:
+        postprocessors.append({
+            'key': 'MetadataParser',
+            'actions': opts.parse_metadata,
             # Run this immediately after extraction is complete
             'when': 'pre_process'
         })
@@ -415,6 +460,13 @@ def _real_main(argv=None):
             # Run this before the actual video download
             'when': 'before_dl'
         })
+    # Must be after all other before_dl
+    if opts.exec_before_dl_cmd:
+        postprocessors.append({
+            'key': 'Exec',
+            'exec_cmd': opts.exec_before_dl_cmd,
+            'when': 'before_dl'
+        })
     if opts.extractaudio:
         postprocessors.append({
             'key': 'FFmpegExtractAudio',
@@ -432,29 +484,55 @@ def _real_main(argv=None):
             'key': 'FFmpegVideoConvertor',
             'preferedformat': opts.recodevideo,
         })
-    # FFmpegMetadataPP should be run after FFmpegVideoConvertorPP and
+    # If ModifyChapters is going to remove chapters, subtitles must already be in the container.
-    # FFmpegExtractAudioPP as containers before conversion may not support
-    # metadata (3gp, webm, etc.)
-    # And this post-processor should be placed before other metadata
-    # manipulating post-processors (FFmpegEmbedSubtitle) to prevent loss of
-    # extra metadata. By default ffmpeg preserves metadata applicable for both
-    # source and target containers. From this point the container won't change,
-    # so metadata can be added here.
-    if opts.addmetadata:
-        postprocessors.append({'key': 'FFmpegMetadata'})
     if opts.embedsubtitles:
-        already_have_subtitle = opts.writesubtitles
+        already_have_subtitle = opts.writesubtitles and 'no-keep-subs' not in compat_opts
         postprocessors.append({
             'key': 'FFmpegEmbedSubtitle',
             # already_have_subtitle = True prevents the file from being deleted after embedding
             'already_have_subtitle': already_have_subtitle
         })
-        if not already_have_subtitle:
+        if not opts.writeautomaticsub and 'no-keep-subs' not in compat_opts:
             opts.writesubtitles = True
     # --all-sub automatically sets --write-sub if --write-auto-sub is not given
     # this was the old behaviour if only --all-sub was given.
     if opts.allsubtitles and not opts.writeautomaticsub:
         opts.writesubtitles = True
+    # ModifyChapters must run before FFmpegMetadataPP
+    remove_chapters_patterns, remove_ranges = [], []
+    for regex in opts.remove_chapters:
+        if regex.startswith('*'):
+            dur = list(map(parse_duration, regex[1:].split('-')))
+            if len(dur) == 2 and all(t is not None for t in dur):
+                remove_ranges.append(tuple(dur))
+                continue
+            parser.error(f'invalid --remove-chapters time range {regex!r}. Must be of the form ?start-end')
+        try:
+            remove_chapters_patterns.append(re.compile(regex))
+        except re.error as err:
+            parser.error(f'invalid --remove-chapters regex {regex!r} - {err}')
+    if opts.remove_chapters or sponsorblock_query:
+        postprocessors.append({
+            'key': 'ModifyChapters',
+            'remove_chapters_patterns': remove_chapters_patterns,
+            'remove_sponsor_segments': opts.sponsorblock_remove,
+            'remove_ranges': remove_ranges,
+            'sponsorblock_chapter_title': opts.sponsorblock_chapter_title,
+            'force_keyframes': opts.force_keyframes_at_cuts
+        })
+    # FFmpegMetadataPP should be run after FFmpegVideoConvertorPP and
+    # FFmpegExtractAudioPP as containers before conversion may not support
+    # metadata (3gp, webm, etc.)
+    # By default ffmpeg preserves metadata applicable for both
+    # source and target containers. From this point the container won't change,
+    # so metadata can be added here.
+    if opts.addmetadata or opts.addchapters:
+        postprocessors.append({
+            'key': 'FFmpegMetadata',
+            'add_chapters': opts.addchapters,
+            'add_metadata': opts.addmetadata,
+        })
+    # Note: Deprecated
     # This should be above EmbedThumbnail since sponskrub removes the thumbnail attachment
     # but must be below EmbedSubtitle and FFmpegMetadata
     # See https://github.com/yt-dlp/yt-dlp/issues/204 , https://github.com/faissaloo/SponSkrub/issues/29
@@ -477,15 +555,19 @@ def _real_main(argv=None):
         })
         if not already_have_thumbnail:
             opts.writethumbnail = True
+            opts.outtmpl['pl_thumbnail'] = ''
     if opts.split_chapters:
-        postprocessors.append({'key': 'FFmpegSplitChapters'})
+        postprocessors.append({
+            'key': 'FFmpegSplitChapters',
+            'force_keyframes': opts.force_keyframes_at_cuts,
+        })
     # XAttrMetadataPP should be run after post-processors that may change file contents
     if opts.xattrs:
         postprocessors.append({'key': 'XAttrMetadata'})
-    # ExecAfterDownload must be the last PP
+    # Exec must be the last PP
     if opts.exec_cmd:
         postprocessors.append({
-            'key': 'ExecAfterDownload',
+            'key': 'Exec',
             'exec_cmd': opts.exec_cmd,
             # Run this only after the files have been moved to their final locations
             'when': 'after_move'
@@ -514,6 +596,7 @@ def _real_main(argv=None):
 
     ydl_opts = {
         'usenetrc': opts.usenetrc,
+        'netrc_location': opts.netrc_location,
         'username': opts.username,
         'password': opts.password,
         'twofactor': opts.twofactor,
@@ -535,7 +618,7 @@ def _real_main(argv=None):
         'forcejson': opts.dumpjson or opts.print_json,
         'dump_single_json': opts.dump_single_json,
         'force_write_download_archive': opts.force_write_download_archive,
-        'simulate': opts.simulate or any_getting,
+        'simulate': (any_getting or None) if opts.simulate is None else opts.simulate,
         'skip_download': opts.skip_download,
         'format': opts.format,
         'allow_unplayable_formats': opts.allow_unplayable_formats,
@@ -569,8 +652,9 @@ def _real_main(argv=None):
         'noresizebuffer': opts.noresizebuffer,
         'http_chunk_size': opts.http_chunk_size,
         'continuedl': opts.continue_dl,
-        'noprogress': opts.noprogress,
+        'noprogress': opts.quiet if opts.noprogress is None else opts.noprogress,
         'progress_with_newline': opts.progress_with_newline,
+        'progress_template': opts.progress_template,
         'playliststart': opts.playliststart,
         'playlistend': opts.playlistend,
         'playlistreverse': opts.playlist_reverse,
@@ -621,6 +705,7 @@ def _real_main(argv=None):
         'break_on_reject': opts.break_on_reject,
         'skip_playlist_after_errors': opts.skip_playlist_after_errors,
         'cookiefile': opts.cookiefile,
+        'cookiesfrombrowser': opts.cookiesfrombrowser,
         'nocheckcertificate': opts.no_check_certificate,
         'prefer_insecure': opts.prefer_insecure,
         'proxy': opts.proxy,
@@ -631,6 +716,7 @@ def _real_main(argv=None):
         'include_ads': opts.include_ads,
         'default_search': opts.default_search,
         'dynamic_mpd': opts.dynamic_mpd,
+        'extractor_args': opts.extractor_args,
         'youtube_include_dash_manifest': opts.youtube_include_dash_manifest,
         'youtube_include_hls_manifest': opts.youtube_include_hls_manifest,
         'encoding': opts.encoding,
@@ -663,12 +749,8 @@ def _real_main(argv=None):
         'geo_bypass': opts.geo_bypass,
         'geo_bypass_country': opts.geo_bypass_country,
         'geo_bypass_ip_block': opts.geo_bypass_ip_block,
-        'warnings': warnings,
+        '_warnings': warnings,
         'compat_opts': compat_opts,
-        # just for deprecation check
-        'autonumber': opts.autonumber or None,
-        'usetitle': opts.usetitle or None,
-        'useid': opts.useid or None,
     }
 
     with YoutubeDL(ydl_opts) as ydl:
@@ -713,10 +795,15 @@ def main(argv=None):
         _real_main(argv)
     except DownloadError:
         sys.exit(1)
-    except SameFileError:
+    except SameFileError as e:
-        sys.exit('ERROR: fixed output name but more than one file to download')
+        sys.exit(f'ERROR: {e}')
     except KeyboardInterrupt:
         sys.exit('\nERROR: Interrupted by user')
+    except BrokenPipeError as e:
+        # https://docs.python.org/3/library/signal.html#note-on-sigpipe
+        devnull = os.open(os.devnull, os.O_WRONLY)
+        os.dup2(devnull, sys.stdout.fileno())
+        sys.exit(f'\nERROR: {e}')
 
 
 __all__ = ['main', 'YoutubeDL', 'gen_extractors', 'list_extractors']

yt_dlp/aes.py

@@ -2,36 +2,68 @@ from __future__ import unicode_literals
 
 from math import ceil
 
-from .compat import compat_b64decode
+from .compat import compat_b64decode, compat_pycrypto_AES
 from .utils import bytes_to_intlist, intlist_to_bytes
 
+
+if compat_pycrypto_AES:
+    def aes_cbc_decrypt_bytes(data, key, iv):
+        """ Decrypt bytes with AES-CBC using pycryptodome """
+        return compat_pycrypto_AES.new(key, compat_pycrypto_AES.MODE_CBC, iv).decrypt(data)
+
+    def aes_gcm_decrypt_and_verify_bytes(data, key, tag, nonce):
+        """ Decrypt bytes with AES-GCM using pycryptodome """
+        return compat_pycrypto_AES.new(key, compat_pycrypto_AES.MODE_GCM, nonce).decrypt_and_verify(data, tag)
+
+else:
+    def aes_cbc_decrypt_bytes(data, key, iv):
+        """ Decrypt bytes with AES-CBC using native implementation since pycryptodome is unavailable """
+        return intlist_to_bytes(aes_cbc_decrypt(*map(bytes_to_intlist, (data, key, iv))))
+
+    def aes_gcm_decrypt_and_verify_bytes(data, key, tag, nonce):
+        """ Decrypt bytes with AES-GCM using native implementation since pycryptodome is unavailable """
+        return intlist_to_bytes(aes_gcm_decrypt_and_verify(*map(bytes_to_intlist, (data, key, tag, nonce))))
+
+
 BLOCK_SIZE_BYTES = 16
 
 
-def aes_ctr_decrypt(data, key, counter):
+def aes_ctr_decrypt(data, key, iv):
     """
     Decrypt with aes in counter mode
 
     @param {int[]} data        cipher
     @param {int[]} key         16/24/32-Byte cipher key
-    @param {instance} counter  Instance whose next_value function (@returns {int[]}  16-Byte block)
+    @param {int[]} iv          16-Byte initialization vector
-                               returns the next counter block
     @returns {int[]}           decrypted data
     """
+    return aes_ctr_encrypt(data, key, iv)
+
+
+def aes_ctr_encrypt(data, key, iv):
+    """
+    Encrypt with aes in counter mode
+
+    @param {int[]} data        cleartext
+    @param {int[]} key         16/24/32-Byte cipher key
+    @param {int[]} iv          16-Byte initialization vector
+    @returns {int[]}           encrypted data
+    """
     expanded_key = key_expansion(key)
     block_count = int(ceil(float(len(data)) / BLOCK_SIZE_BYTES))
+    counter = iter_vector(iv)
 
-    decrypted_data = []
+    encrypted_data = []
     for i in range(block_count):
-        counter_block = counter.next_value()
+        counter_block = next(counter)
         block = data[i * BLOCK_SIZE_BYTES: (i + 1) * BLOCK_SIZE_BYTES]
         block += [0] * (BLOCK_SIZE_BYTES - len(block))
 
         cipher_counter_block = aes_encrypt(counter_block, expanded_key)
-        decrypted_data += xor(block, cipher_counter_block)
+        encrypted_data += xor(block, cipher_counter_block)
-    decrypted_data = decrypted_data[:len(data)]
+    encrypted_data = encrypted_data[:len(data)]
 
-    return decrypted_data
+    return encrypted_data
 
 
 def aes_cbc_decrypt(data, key, iv):
@@ -88,39 +120,47 @@ def aes_cbc_encrypt(data, key, iv):
     return encrypted_data
 
 
-def key_expansion(data):
+def aes_gcm_decrypt_and_verify(data, key, tag, nonce):
     """
-    Generate key schedule
+    Decrypt with aes in GBM mode and checks authenticity using tag
 
-    @param {int[]} data  16/24/32-Byte cipher key
+    @param {int[]} data        cipher
-    @returns {int[]}     176/208/240-Byte expanded key
+    @param {int[]} key         16-Byte cipher key
+    @param {int[]} tag         authentication tag
+    @param {int[]} nonce       IV (recommended 12-Byte)
+    @returns {int[]}           decrypted data
     """
-    data = data[:]  # copy
-    rcon_iteration = 1
-    key_size_bytes = len(data)
-    expanded_key_size_bytes = (key_size_bytes // 4 + 7) * BLOCK_SIZE_BYTES
 
-    while len(data) < expanded_key_size_bytes:
+    # XXX: check aes, gcm param
-        temp = data[-4:]
-        temp = key_schedule_core(temp, rcon_iteration)
-        rcon_iteration += 1
-        data += xor(temp, data[-key_size_bytes: 4 - key_size_bytes])
 
-        for _ in range(3):
+    hash_subkey = aes_encrypt([0] * BLOCK_SIZE_BYTES, key_expansion(key))
-            temp = data[-4:]
-            data += xor(temp, data[-key_size_bytes: 4 - key_size_bytes])
 
-        if key_size_bytes == 32:
+    if len(nonce) == 12:
-            temp = data[-4:]
+        j0 = nonce + [0, 0, 0, 1]
-            temp = sub_bytes(temp)
+    else:
-            data += xor(temp, data[-key_size_bytes: 4 - key_size_bytes])
+        fill = (BLOCK_SIZE_BYTES - (len(nonce) % BLOCK_SIZE_BYTES)) % BLOCK_SIZE_BYTES + 8
+        ghash_in = nonce + [0] * fill + bytes_to_intlist((8 * len(nonce)).to_bytes(8, 'big'))
+        j0 = ghash(hash_subkey, ghash_in)
 
-        for _ in range(3 if key_size_bytes == 32 else 2 if key_size_bytes == 24 else 0):
+    # TODO: add nonce support to aes_ctr_decrypt
-            temp = data[-4:]
-            data += xor(temp, data[-key_size_bytes: 4 - key_size_bytes])
-    data = data[:expanded_key_size_bytes]
 
-    return data
+    # nonce_ctr = j0[:12]
+    iv_ctr = inc(j0)
+
+    decrypted_data = aes_ctr_decrypt(data, key, iv_ctr + [0] * (BLOCK_SIZE_BYTES - len(iv_ctr)))
+    pad_len = len(data) // 16 * 16
+    s_tag = ghash(
+        hash_subkey,
+        data
+        + [0] * (BLOCK_SIZE_BYTES - len(data) + pad_len)        # pad
+        + bytes_to_intlist((0 * 8).to_bytes(8, 'big')           # length of associated data
+                           + ((len(data) * 8).to_bytes(8, 'big')))  # length of data
+    )
+
+    if tag != aes_ctr_encrypt(s_tag, key, j0):
+        raise ValueError("Mismatching authentication tag")
+
+    return decrypted_data
 
 
 def aes_encrypt(data, expanded_key):
@@ -138,7 +178,7 @@ def aes_encrypt(data, expanded_key):
         data = sub_bytes(data)
         data = shift_rows(data)
         if i != rounds:
-            data = mix_columns(data)
+            data = list(iter_mix_columns(data, MIX_COLUMN_MATRIX))
         data = xor(data, expanded_key[i * BLOCK_SIZE_BYTES: (i + 1) * BLOCK_SIZE_BYTES])
 
     return data
@@ -157,7 +197,7 @@ def aes_decrypt(data, expanded_key):
     for i in range(rounds, 0, -1):
         data = xor(data, expanded_key[i * BLOCK_SIZE_BYTES: (i + 1) * BLOCK_SIZE_BYTES])
         if i != rounds:
-            data = mix_columns_inv(data)
+            data = list(iter_mix_columns(data, MIX_COLUMN_MATRIX_INV))
         data = shift_rows_inv(data)
         data = sub_bytes_inv(data)
     data = xor(data, expanded_key[:BLOCK_SIZE_BYTES])
@@ -189,15 +229,7 @@ def aes_decrypt_text(data, password, key_size_bytes):
     nonce = data[:NONCE_LENGTH_BYTES]
     cipher = data[NONCE_LENGTH_BYTES:]
 
-    class Counter(object):
+    decrypted_data = aes_ctr_decrypt(cipher, key, nonce + [0] * (BLOCK_SIZE_BYTES - NONCE_LENGTH_BYTES))
-        __value = nonce + [0] * (BLOCK_SIZE_BYTES - NONCE_LENGTH_BYTES)
-
-        def next_value(self):
-            temp = self.__value
-            self.__value = inc(self.__value)
-            return temp
-
-    decrypted_data = aes_ctr_decrypt(cipher, key, Counter())
     plaintext = intlist_to_bytes(decrypted_data)
 
     return plaintext
@@ -278,6 +310,47 @@ RIJNDAEL_LOG_TABLE = (0x00, 0x00, 0x19, 0x01, 0x32, 0x02, 0x1a, 0xc6, 0x4b, 0xc7
                       0x67, 0x4a, 0xed, 0xde, 0xc5, 0x31, 0xfe, 0x18, 0x0d, 0x63, 0x8c, 0x80, 0xc0, 0xf7, 0x70, 0x07)
 
 
+def key_expansion(data):
+    """
+    Generate key schedule
+
+    @param {int[]} data  16/24/32-Byte cipher key
+    @returns {int[]}     176/208/240-Byte expanded key
+    """
+    data = data[:]  # copy
+    rcon_iteration = 1
+    key_size_bytes = len(data)
+    expanded_key_size_bytes = (key_size_bytes // 4 + 7) * BLOCK_SIZE_BYTES
+
+    while len(data) < expanded_key_size_bytes:
+        temp = data[-4:]
+        temp = key_schedule_core(temp, rcon_iteration)
+        rcon_iteration += 1
+        data += xor(temp, data[-key_size_bytes: 4 - key_size_bytes])
+
+        for _ in range(3):
+            temp = data[-4:]
+            data += xor(temp, data[-key_size_bytes: 4 - key_size_bytes])
+
+        if key_size_bytes == 32:
+            temp = data[-4:]
+            temp = sub_bytes(temp)
+            data += xor(temp, data[-key_size_bytes: 4 - key_size_bytes])
+
+        for _ in range(3 if key_size_bytes == 32 else 2 if key_size_bytes == 24 else 0):
+            temp = data[-4:]
+            data += xor(temp, data[-key_size_bytes: 4 - key_size_bytes])
+    data = data[:expanded_key_size_bytes]
+
+    return data
+
+
+def iter_vector(iv):
+    while True:
+        yield iv
+        iv = inc(iv)
+
+
 def sub_bytes(data):
     return [SBOX[x] for x in data]
 
@@ -302,48 +375,36 @@ def xor(data1, data2):
     return [x ^ y for x, y in zip(data1, data2)]
 
 
-def rijndael_mul(a, b):
+def iter_mix_columns(data, matrix):
-    if(a == 0 or b == 0):
+    for i in (0, 4, 8, 12):
-        return 0
+        for row in matrix:
-    return RIJNDAEL_EXP_TABLE[(RIJNDAEL_LOG_TABLE[a] + RIJNDAEL_LOG_TABLE[b]) % 0xFF]
+            mixed = 0
-
+            for j in range(4):
-
+                # xor is (+) and (-)
-def mix_column(data, matrix):
+                mixed ^= (0 if data[i:i + 4][j] == 0 or row[j] == 0 else
-    data_mixed = []
+                          RIJNDAEL_EXP_TABLE[(RIJNDAEL_LOG_TABLE[data[i + j]] + RIJNDAEL_LOG_TABLE[row[j]]) % 0xFF])
-    for row in range(4):
+            yield mixed
-        mixed = 0
-        for column in range(4):
-            # xor is (+) and (-)
-            mixed ^= rijndael_mul(data[column], matrix[row][column])
-        data_mixed.append(mixed)
-    return data_mixed
-
-
-def mix_columns(data, matrix=MIX_COLUMN_MATRIX):
-    data_mixed = []
-    for i in range(4):
-        column = data[i * 4: (i + 1) * 4]
-        data_mixed += mix_column(column, matrix)
-    return data_mixed
-
-
-def mix_columns_inv(data):
-    return mix_columns(data, MIX_COLUMN_MATRIX_INV)
 
 
 def shift_rows(data):
-    data_shifted = []
+    return [data[((column + row) & 0b11) * 4 + row] for column in range(4) for row in range(4)]
-    for column in range(4):
-        for row in range(4):
-            data_shifted.append(data[((column + row) & 0b11) * 4 + row])
-    return data_shifted
 
 
 def shift_rows_inv(data):
+    return [data[((column - row) & 0b11) * 4 + row] for column in range(4) for row in range(4)]
+
+
+def shift_block(data):
     data_shifted = []
-    for column in range(4):
+
-        for row in range(4):
+    bit = 0
-            data_shifted.append(data[((column - row) & 0b11) * 4 + row])
+    for n in data:
+        if bit:
+            n |= 0x100
+        bit = n & 1
+        n >>= 1
+        data_shifted.append(n)
+
     return data_shifted
 
 
@@ -358,4 +419,50 @@ def inc(data):
     return data
 
 
-__all__ = ['aes_encrypt', 'key_expansion', 'aes_ctr_decrypt', 'aes_cbc_decrypt', 'aes_decrypt_text']
+def block_product(block_x, block_y):
+    # NIST SP 800-38D, Algorithm 1
+
+    if len(block_x) != BLOCK_SIZE_BYTES or len(block_y) != BLOCK_SIZE_BYTES:
+        raise ValueError("Length of blocks need to be %d bytes" % BLOCK_SIZE_BYTES)
+
+    block_r = [0xE1] + [0] * (BLOCK_SIZE_BYTES - 1)
+    block_v = block_y[:]
+    block_z = [0] * BLOCK_SIZE_BYTES
+
+    for i in block_x:
+        for bit in range(7, -1, -1):
+            if i & (1 << bit):
+                block_z = xor(block_z, block_v)
+
+            do_xor = block_v[-1] & 1
+            block_v = shift_block(block_v)
+            if do_xor:
+                block_v = xor(block_v, block_r)
+
+    return block_z
+
+
+def ghash(subkey, data):
+    # NIST SP 800-38D, Algorithm 2
+
+    if len(data) % BLOCK_SIZE_BYTES:
+        raise ValueError("Length of data should be %d bytes" % BLOCK_SIZE_BYTES)
+
+    last_y = [0] * BLOCK_SIZE_BYTES
+    for i in range(0, len(data), BLOCK_SIZE_BYTES):
+        block = data[i : i + BLOCK_SIZE_BYTES]  # noqa: E203
+        last_y = block_product(xor(last_y, block), subkey)
+
+    return last_y
+
+
+__all__ = [
+    'aes_ctr_decrypt',
+    'aes_cbc_decrypt',
+    'aes_cbc_decrypt_bytes',
+    'aes_decrypt_text',
+    'aes_encrypt',
+    'aes_gcm_decrypt_and_verify',
+    'aes_gcm_decrypt_and_verify_bytes',
+    'key_expansion'
+]

yt_dlp/cache.py

@@ -50,6 +50,7 @@ class Cache(object):
             except OSError as ose:
                 if ose.errno != errno.EEXIST:
                     raise
+            self._ydl.write_debug(f'Saving {section}.{key} to cache')
             write_json_file(data, fn)
         except Exception:
             tb = traceback.format_exc()
@@ -66,6 +67,7 @@ class Cache(object):
         try:
             try:
                 with io.open(cache_fn, 'r', encoding='utf-8') as cachef:
+                    self._ydl.write_debug(f'Loading {section}.{key} from cache')
                     return json.load(cachef)
             except ValueError:
                 try:

yt_dlp/cookies.py

@@ -0,0 +1,745 @@
+import ctypes
+import json
+import os
+import shutil
+import struct
+import subprocess
+import sys
+import tempfile
+from datetime import datetime, timedelta, timezone
+from hashlib import pbkdf2_hmac
+
+from .aes import aes_cbc_decrypt_bytes, aes_gcm_decrypt_and_verify_bytes
+from .compat import (
+    compat_b64decode,
+    compat_cookiejar_Cookie,
+)
+from .utils import (
+    bug_reports_message,
+    expand_path,
+    Popen,
+    YoutubeDLCookieJar,
+)
+
+try:
+    import sqlite3
+    SQLITE_AVAILABLE = True
+except ImportError:
+    # although sqlite3 is part of the standard library, it is possible to compile python without
+    # sqlite support. See: https://github.com/yt-dlp/yt-dlp/issues/544
+    SQLITE_AVAILABLE = False
+
+
+try:
+    import keyring
+    KEYRING_AVAILABLE = True
+    KEYRING_UNAVAILABLE_REASON = f'due to unknown reasons{bug_reports_message()}'
+except ImportError:
+    KEYRING_AVAILABLE = False
+    KEYRING_UNAVAILABLE_REASON = (
+        'as the `keyring` module is not installed. '
+        'Please install by running `python3 -m pip install keyring`. '
+        'Depending on your platform, additional packages may be required '
+        'to access the keyring; see  https://pypi.org/project/keyring')
+except Exception as _err:
+    KEYRING_AVAILABLE = False
+    KEYRING_UNAVAILABLE_REASON = 'as the `keyring` module could not be initialized: %s' % _err
+
+
+CHROMIUM_BASED_BROWSERS = {'brave', 'chrome', 'chromium', 'edge', 'opera', 'vivaldi'}
+SUPPORTED_BROWSERS = CHROMIUM_BASED_BROWSERS | {'firefox', 'safari'}
+
+
+class YDLLogger:
+    def __init__(self, ydl=None):
+        self._ydl = ydl
+
+    def debug(self, message):
+        if self._ydl:
+            self._ydl.write_debug(message)
+
+    def info(self, message):
+        if self._ydl:
+            self._ydl.to_screen(f'[Cookies] {message}')
+
+    def warning(self, message, only_once=False):
+        if self._ydl:
+            self._ydl.report_warning(message, only_once)
+
+    def error(self, message):
+        if self._ydl:
+            self._ydl.report_error(message)
+
+
+def load_cookies(cookie_file, browser_specification, ydl):
+    cookie_jars = []
+    if browser_specification is not None:
+        browser_name, profile = _parse_browser_specification(*browser_specification)
+        cookie_jars.append(extract_cookies_from_browser(browser_name, profile, YDLLogger(ydl)))
+
+    if cookie_file is not None:
+        cookie_file = expand_path(cookie_file)
+        jar = YoutubeDLCookieJar(cookie_file)
+        if os.access(cookie_file, os.R_OK):
+            jar.load(ignore_discard=True, ignore_expires=True)
+        cookie_jars.append(jar)
+
+    return _merge_cookie_jars(cookie_jars)
+
+
+def extract_cookies_from_browser(browser_name, profile=None, logger=YDLLogger()):
+    if browser_name == 'firefox':
+        return _extract_firefox_cookies(profile, logger)
+    elif browser_name == 'safari':
+        return _extract_safari_cookies(profile, logger)
+    elif browser_name in CHROMIUM_BASED_BROWSERS:
+        return _extract_chrome_cookies(browser_name, profile, logger)
+    else:
+        raise ValueError('unknown browser: {}'.format(browser_name))
+
+
+def _extract_firefox_cookies(profile, logger):
+    logger.info('Extracting cookies from firefox')
+    if not SQLITE_AVAILABLE:
+        logger.warning('Cannot extract cookies from firefox without sqlite3 support. '
+                       'Please use a python interpreter compiled with sqlite3 support')
+        return YoutubeDLCookieJar()
+
+    if profile is None:
+        search_root = _firefox_browser_dir()
+    elif _is_path(profile):
+        search_root = profile
+    else:
+        search_root = os.path.join(_firefox_browser_dir(), profile)
+
+    cookie_database_path = _find_most_recently_used_file(search_root, 'cookies.sqlite')
+    if cookie_database_path is None:
+        raise FileNotFoundError('could not find firefox cookies database in {}'.format(search_root))
+    logger.debug('Extracting cookies from: "{}"'.format(cookie_database_path))
+
+    with tempfile.TemporaryDirectory(prefix='yt_dlp') as tmpdir:
+        cursor = None
+        try:
+            cursor = _open_database_copy(cookie_database_path, tmpdir)
+            cursor.execute('SELECT host, name, value, path, expiry, isSecure FROM moz_cookies')
+            jar = YoutubeDLCookieJar()
+            for host, name, value, path, expiry, is_secure in cursor.fetchall():
+                cookie = compat_cookiejar_Cookie(
+                    version=0, name=name, value=value, port=None, port_specified=False,
+                    domain=host, domain_specified=bool(host), domain_initial_dot=host.startswith('.'),
+                    path=path, path_specified=bool(path), secure=is_secure, expires=expiry, discard=False,
+                    comment=None, comment_url=None, rest={})
+                jar.set_cookie(cookie)
+            logger.info('Extracted {} cookies from firefox'.format(len(jar)))
+            return jar
+        finally:
+            if cursor is not None:
+                cursor.connection.close()
+
+
+def _firefox_browser_dir():
+    if sys.platform in ('linux', 'linux2'):
+        return os.path.expanduser('~/.mozilla/firefox')
+    elif sys.platform == 'win32':
+        return os.path.expandvars(r'%APPDATA%\Mozilla\Firefox\Profiles')
+    elif sys.platform == 'darwin':
+        return os.path.expanduser('~/Library/Application Support/Firefox')
+    else:
+        raise ValueError('unsupported platform: {}'.format(sys.platform))
+
+
+def _get_chromium_based_browser_settings(browser_name):
+    # https://chromium.googlesource.com/chromium/src/+/HEAD/docs/user_data_dir.md
+    if sys.platform in ('linux', 'linux2'):
+        config = _config_home()
+        browser_dir = {
+            'brave': os.path.join(config, 'BraveSoftware/Brave-Browser'),
+            'chrome': os.path.join(config, 'google-chrome'),
+            'chromium': os.path.join(config, 'chromium'),
+            'edge': os.path.join(config, 'microsoft-edge'),
+            'opera': os.path.join(config, 'opera'),
+            'vivaldi': os.path.join(config, 'vivaldi'),
+        }[browser_name]
+
+    elif sys.platform == 'win32':
+        appdata_local = os.path.expandvars('%LOCALAPPDATA%')
+        appdata_roaming = os.path.expandvars('%APPDATA%')
+        browser_dir = {
+            'brave': os.path.join(appdata_local, r'BraveSoftware\Brave-Browser\User Data'),
+            'chrome': os.path.join(appdata_local, r'Google\Chrome\User Data'),
+            'chromium': os.path.join(appdata_local, r'Chromium\User Data'),
+            'edge': os.path.join(appdata_local, r'Microsoft\Edge\User Data'),
+            'opera': os.path.join(appdata_roaming, r'Opera Software\Opera Stable'),
+            'vivaldi': os.path.join(appdata_local, r'Vivaldi\User Data'),
+        }[browser_name]
+
+    elif sys.platform == 'darwin':
+        appdata = os.path.expanduser('~/Library/Application Support')
+        browser_dir = {
+            'brave': os.path.join(appdata, 'BraveSoftware/Brave-Browser'),
+            'chrome': os.path.join(appdata, 'Google/Chrome'),
+            'chromium': os.path.join(appdata, 'Chromium'),
+            'edge': os.path.join(appdata, 'Microsoft Edge'),
+            'opera': os.path.join(appdata, 'com.operasoftware.Opera'),
+            'vivaldi': os.path.join(appdata, 'Vivaldi'),
+        }[browser_name]
+
+    else:
+        raise ValueError('unsupported platform: {}'.format(sys.platform))
+
+    # Linux keyring names can be determined by snooping on dbus while opening the browser in KDE:
+    # dbus-monitor "interface='org.kde.KWallet'" "type=method_return"
+    keyring_name = {
+        'brave': 'Brave',
+        'chrome': 'Chrome',
+        'chromium': 'Chromium',
+        'edge': 'Microsoft Edge' if sys.platform == 'darwin' else 'Chromium',
+        'opera': 'Opera' if sys.platform == 'darwin' else 'Chromium',
+        'vivaldi': 'Vivaldi' if sys.platform == 'darwin' else 'Chrome',
+    }[browser_name]
+
+    browsers_without_profiles = {'opera'}
+
+    return {
+        'browser_dir': browser_dir,
+        'keyring_name': keyring_name,
+        'supports_profiles': browser_name not in browsers_without_profiles
+    }
+
+
+def _extract_chrome_cookies(browser_name, profile, logger):
+    logger.info('Extracting cookies from {}'.format(browser_name))
+
+    if not SQLITE_AVAILABLE:
+        logger.warning(('Cannot extract cookies from {} without sqlite3 support. '
+                        'Please use a python interpreter compiled with sqlite3 support').format(browser_name))
+        return YoutubeDLCookieJar()
+
+    config = _get_chromium_based_browser_settings(browser_name)
+
+    if profile is None:
+        search_root = config['browser_dir']
+    elif _is_path(profile):
+        search_root = profile
+        config['browser_dir'] = os.path.dirname(profile) if config['supports_profiles'] else profile
+    else:
+        if config['supports_profiles']:
+            search_root = os.path.join(config['browser_dir'], profile)
+        else:
+            logger.error('{} does not support profiles'.format(browser_name))
+            search_root = config['browser_dir']
+
+    cookie_database_path = _find_most_recently_used_file(search_root, 'Cookies')
+    if cookie_database_path is None:
+        raise FileNotFoundError('could not find {} cookies database in "{}"'.format(browser_name, search_root))
+    logger.debug('Extracting cookies from: "{}"'.format(cookie_database_path))
+
+    decryptor = get_cookie_decryptor(config['browser_dir'], config['keyring_name'], logger)
+
+    with tempfile.TemporaryDirectory(prefix='yt_dlp') as tmpdir:
+        cursor = None
+        try:
+            cursor = _open_database_copy(cookie_database_path, tmpdir)
+            cursor.connection.text_factory = bytes
+            column_names = _get_column_names(cursor, 'cookies')
+            secure_column = 'is_secure' if 'is_secure' in column_names else 'secure'
+            cursor.execute('SELECT host_key, name, value, encrypted_value, path, '
+                           'expires_utc, {} FROM cookies'.format(secure_column))
+            jar = YoutubeDLCookieJar()
+            failed_cookies = 0
+            for host_key, name, value, encrypted_value, path, expires_utc, is_secure in cursor.fetchall():
+                host_key = host_key.decode('utf-8')
+                name = name.decode('utf-8')
+                value = value.decode('utf-8')
+                path = path.decode('utf-8')
+
+                if not value and encrypted_value:
+                    value = decryptor.decrypt(encrypted_value)
+                    if value is None:
+                        failed_cookies += 1
+                        continue
+
+                cookie = compat_cookiejar_Cookie(
+                    version=0, name=name, value=value, port=None, port_specified=False,
+                    domain=host_key, domain_specified=bool(host_key), domain_initial_dot=host_key.startswith('.'),
+                    path=path, path_specified=bool(path), secure=is_secure, expires=expires_utc, discard=False,
+                    comment=None, comment_url=None, rest={})
+                jar.set_cookie(cookie)
+            if failed_cookies > 0:
+                failed_message = ' ({} could not be decrypted)'.format(failed_cookies)
+            else:
+                failed_message = ''
+            logger.info('Extracted {} cookies from {}{}'.format(len(jar), browser_name, failed_message))
+            return jar
+        finally:
+            if cursor is not None:
+                cursor.connection.close()
+
+
+class ChromeCookieDecryptor:
+    """
+    Overview:
+
+        Linux:
+        - cookies are either v10 or v11
+            - v10: AES-CBC encrypted with a fixed key
+            - v11: AES-CBC encrypted with an OS protected key (keyring)
+            - v11 keys can be stored in various places depending on the activate desktop environment [2]
+
+        Mac:
+        - cookies are either v10 or not v10
+            - v10: AES-CBC encrypted with an OS protected key (keyring) and more key derivation iterations than linux
+            - not v10: 'old data' stored as plaintext
+
+        Windows:
+        - cookies are either v10 or not v10
+            - v10: AES-GCM encrypted with a key which is encrypted with DPAPI
+            - not v10: encrypted with DPAPI
+
+    Sources:
+    - [1] https://chromium.googlesource.com/chromium/src/+/refs/heads/main/components/os_crypt/
+    - [2] https://chromium.googlesource.com/chromium/src/+/refs/heads/main/components/os_crypt/key_storage_linux.cc
+        - KeyStorageLinux::CreateService
+    """
+
+    def decrypt(self, encrypted_value):
+        raise NotImplementedError
+
+
+def get_cookie_decryptor(browser_root, browser_keyring_name, logger):
+    if sys.platform in ('linux', 'linux2'):
+        return LinuxChromeCookieDecryptor(browser_keyring_name, logger)
+    elif sys.platform == 'darwin':
+        return MacChromeCookieDecryptor(browser_keyring_name, logger)
+    elif sys.platform == 'win32':
+        return WindowsChromeCookieDecryptor(browser_root, logger)
+    else:
+        raise NotImplementedError('Chrome cookie decryption is not supported '
+                                  'on this platform: {}'.format(sys.platform))
+
+
+class LinuxChromeCookieDecryptor(ChromeCookieDecryptor):
+    def __init__(self, browser_keyring_name, logger):
+        self._logger = logger
+        self._v10_key = self.derive_key(b'peanuts')
+        if KEYRING_AVAILABLE:
+            self._v11_key = self.derive_key(_get_linux_keyring_password(browser_keyring_name))
+        else:
+            self._v11_key = None
+
+    @staticmethod
+    def derive_key(password):
+        # values from
+        # https://chromium.googlesource.com/chromium/src/+/refs/heads/main/components/os_crypt/os_crypt_linux.cc
+        return pbkdf2_sha1(password, salt=b'saltysalt', iterations=1, key_length=16)
+
+    def decrypt(self, encrypted_value):
+        version = encrypted_value[:3]
+        ciphertext = encrypted_value[3:]
+
+        if version == b'v10':
+            return _decrypt_aes_cbc(ciphertext, self._v10_key, self._logger)
+
+        elif version == b'v11':
+            if self._v11_key is None:
+                self._logger.warning(f'cannot decrypt cookie {KEYRING_UNAVAILABLE_REASON}', only_once=True)
+                return None
+            return _decrypt_aes_cbc(ciphertext, self._v11_key, self._logger)
+
+        else:
+            return None
+
+
+class MacChromeCookieDecryptor(ChromeCookieDecryptor):
+    def __init__(self, browser_keyring_name, logger):
+        self._logger = logger
+        password = _get_mac_keyring_password(browser_keyring_name, logger)
+        self._v10_key = None if password is None else self.derive_key(password)
+
+    @staticmethod
+    def derive_key(password):
+        # values from
+        # https://chromium.googlesource.com/chromium/src/+/refs/heads/main/components/os_crypt/os_crypt_mac.mm
+        return pbkdf2_sha1(password, salt=b'saltysalt', iterations=1003, key_length=16)
+
+    def decrypt(self, encrypted_value):
+        version = encrypted_value[:3]
+        ciphertext = encrypted_value[3:]
+
+        if version == b'v10':
+            if self._v10_key is None:
+                self._logger.warning('cannot decrypt v10 cookies: no key found', only_once=True)
+                return None
+
+            return _decrypt_aes_cbc(ciphertext, self._v10_key, self._logger)
+
+        else:
+            # other prefixes are considered 'old data' which were stored as plaintext
+            # https://chromium.googlesource.com/chromium/src/+/refs/heads/main/components/os_crypt/os_crypt_mac.mm
+            return encrypted_value
+
+
+class WindowsChromeCookieDecryptor(ChromeCookieDecryptor):
+    def __init__(self, browser_root, logger):
+        self._logger = logger
+        self._v10_key = _get_windows_v10_key(browser_root, logger)
+
+    def decrypt(self, encrypted_value):
+        version = encrypted_value[:3]
+        ciphertext = encrypted_value[3:]
+
+        if version == b'v10':
+            if self._v10_key is None:
+                self._logger.warning('cannot decrypt v10 cookies: no key found', only_once=True)
+                return None
+
+            # https://chromium.googlesource.com/chromium/src/+/refs/heads/main/components/os_crypt/os_crypt_win.cc
+            #   kNonceLength
+            nonce_length = 96 // 8
+            # boringssl
+            #   EVP_AEAD_AES_GCM_TAG_LEN
+            authentication_tag_length = 16
+
+            raw_ciphertext = ciphertext
+            nonce = raw_ciphertext[:nonce_length]
+            ciphertext = raw_ciphertext[nonce_length:-authentication_tag_length]
+            authentication_tag = raw_ciphertext[-authentication_tag_length:]
+
+            return _decrypt_aes_gcm(ciphertext, self._v10_key, nonce, authentication_tag, self._logger)
+
+        else:
+            # any other prefix means the data is DPAPI encrypted
+            # https://chromium.googlesource.com/chromium/src/+/refs/heads/main/components/os_crypt/os_crypt_win.cc
+            return _decrypt_windows_dpapi(encrypted_value, self._logger).decode('utf-8')
+
+
+def _extract_safari_cookies(profile, logger):
+    if profile is not None:
+        logger.error('safari does not support profiles')
+    if sys.platform != 'darwin':
+        raise ValueError('unsupported platform: {}'.format(sys.platform))
+
+    cookies_path = os.path.expanduser('~/Library/Cookies/Cookies.binarycookies')
+
+    if not os.path.isfile(cookies_path):
+        raise FileNotFoundError('could not find safari cookies database')
+
+    with open(cookies_path, 'rb') as f:
+        cookies_data = f.read()
+
+    jar = parse_safari_cookies(cookies_data, logger=logger)
+    logger.info('Extracted {} cookies from safari'.format(len(jar)))
+    return jar
+
+
+class ParserError(Exception):
+    pass
+
+
+class DataParser:
+    def __init__(self, data, logger):
+        self._data = data
+        self.cursor = 0
+        self._logger = logger
+
+    def read_bytes(self, num_bytes):
+        if num_bytes < 0:
+            raise ParserError('invalid read of {} bytes'.format(num_bytes))
+        end = self.cursor + num_bytes
+        if end > len(self._data):
+            raise ParserError('reached end of input')
+        data = self._data[self.cursor:end]
+        self.cursor = end
+        return data
+
+    def expect_bytes(self, expected_value, message):
+        value = self.read_bytes(len(expected_value))
+        if value != expected_value:
+            raise ParserError('unexpected value: {} != {} ({})'.format(value, expected_value, message))
+
+    def read_uint(self, big_endian=False):
+        data_format = '>I' if big_endian else '<I'
+        return struct.unpack(data_format, self.read_bytes(4))[0]
+
+    def read_double(self, big_endian=False):
+        data_format = '>d' if big_endian else '<d'
+        return struct.unpack(data_format, self.read_bytes(8))[0]
+
+    def read_cstring(self):
+        buffer = []
+        while True:
+            c = self.read_bytes(1)
+            if c == b'\x00':
+                return b''.join(buffer).decode('utf-8')
+            else:
+                buffer.append(c)
+
+    def skip(self, num_bytes, description='unknown'):
+        if num_bytes > 0:
+            self._logger.debug('skipping {} bytes ({}): {}'.format(
+                num_bytes, description, self.read_bytes(num_bytes)))
+        elif num_bytes < 0:
+            raise ParserError('invalid skip of {} bytes'.format(num_bytes))
+
+    def skip_to(self, offset, description='unknown'):
+        self.skip(offset - self.cursor, description)
+
+    def skip_to_end(self, description='unknown'):
+        self.skip_to(len(self._data), description)
+
+
+def _mac_absolute_time_to_posix(timestamp):
+    return int((datetime(2001, 1, 1, 0, 0, tzinfo=timezone.utc) + timedelta(seconds=timestamp)).timestamp())
+
+
+def _parse_safari_cookies_header(data, logger):
+    p = DataParser(data, logger)
+    p.expect_bytes(b'cook', 'database signature')
+    number_of_pages = p.read_uint(big_endian=True)
+    page_sizes = [p.read_uint(big_endian=True) for _ in range(number_of_pages)]
+    return page_sizes, p.cursor
+
+
+def _parse_safari_cookies_page(data, jar, logger):
+    p = DataParser(data, logger)
+    p.expect_bytes(b'\x00\x00\x01\x00', 'page signature')
+    number_of_cookies = p.read_uint()
+    record_offsets = [p.read_uint() for _ in range(number_of_cookies)]
+    if number_of_cookies == 0:
+        logger.debug('a cookies page of size {} has no cookies'.format(len(data)))
+        return
+
+    p.skip_to(record_offsets[0], 'unknown page header field')
+
+    for record_offset in record_offsets:
+        p.skip_to(record_offset, 'space between records')
+        record_length = _parse_safari_cookies_record(data[record_offset:], jar, logger)
+        p.read_bytes(record_length)
+    p.skip_to_end('space in between pages')
+
+
+def _parse_safari_cookies_record(data, jar, logger):
+    p = DataParser(data, logger)
+    record_size = p.read_uint()
+    p.skip(4, 'unknown record field 1')
+    flags = p.read_uint()
+    is_secure = bool(flags & 0x0001)
+    p.skip(4, 'unknown record field 2')
+    domain_offset = p.read_uint()
+    name_offset = p.read_uint()
+    path_offset = p.read_uint()
+    value_offset = p.read_uint()
+    p.skip(8, 'unknown record field 3')
+    expiration_date = _mac_absolute_time_to_posix(p.read_double())
+    _creation_date = _mac_absolute_time_to_posix(p.read_double())  # noqa: F841
+
+    try:
+        p.skip_to(domain_offset)
+        domain = p.read_cstring()
+
+        p.skip_to(name_offset)
+        name = p.read_cstring()
+
+        p.skip_to(path_offset)
+        path = p.read_cstring()
+
+        p.skip_to(value_offset)
+        value = p.read_cstring()
+    except UnicodeDecodeError:
+        logger.warning('failed to parse Safari cookie because UTF-8 decoding failed', only_once=True)
+        return record_size
+
+    p.skip_to(record_size, 'space at the end of the record')
+
+    cookie = compat_cookiejar_Cookie(
+        version=0, name=name, value=value, port=None, port_specified=False,
+        domain=domain, domain_specified=bool(domain), domain_initial_dot=domain.startswith('.'),
+        path=path, path_specified=bool(path), secure=is_secure, expires=expiration_date, discard=False,
+        comment=None, comment_url=None, rest={})
+    jar.set_cookie(cookie)
+    return record_size
+
+
+def parse_safari_cookies(data, jar=None, logger=YDLLogger()):
+    """
+    References:
+        - https://github.com/libyal/dtformats/blob/main/documentation/Safari%20Cookies.asciidoc
+            - this data appears to be out of date but the important parts of the database structure is the same
+            - there are a few bytes here and there which are skipped during parsing
+    """
+    if jar is None:
+        jar = YoutubeDLCookieJar()
+    page_sizes, body_start = _parse_safari_cookies_header(data, logger)
+    p = DataParser(data[body_start:], logger)
+    for page_size in page_sizes:
+        _parse_safari_cookies_page(p.read_bytes(page_size), jar, logger)
+    p.skip_to_end('footer')
+    return jar
+
+
+def _get_linux_keyring_password(browser_keyring_name):
+    password = keyring.get_password('{} Keys'.format(browser_keyring_name),
+                                    '{} Safe Storage'.format(browser_keyring_name))
+    if password is None:
+        # this sometimes occurs in KDE because chrome does not check hasEntry and instead
+        # just tries to read the value (which kwallet returns "") whereas keyring checks hasEntry
+        # to verify this:
+        # dbus-monitor "interface='org.kde.KWallet'" "type=method_return"
+        # while starting chrome.
+        # this may be a bug as the intended behaviour is to generate a random password and store
+        # it, but that doesn't matter here.
+        password = ''
+    return password.encode('utf-8')
+
+
+def _get_mac_keyring_password(browser_keyring_name, logger):
+    if KEYRING_AVAILABLE:
+        logger.debug('using keyring to obtain password')
+        password = keyring.get_password('{} Safe Storage'.format(browser_keyring_name), browser_keyring_name)
+        return password.encode('utf-8')
+    else:
+        logger.debug('using find-generic-password to obtain password')
+        proc = Popen(
+            ['security', 'find-generic-password',
+             '-w',  # write password to stdout
+             '-a', browser_keyring_name,  # match 'account'
+             '-s', '{} Safe Storage'.format(browser_keyring_name)],  # match 'service'
+            stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+        try:
+            stdout, stderr = proc.communicate_or_kill()
+            if stdout[-1:] == b'\n':
+                stdout = stdout[:-1]
+            return stdout
+        except BaseException as e:
+            logger.warning(f'exception running find-generic-password: {type(e).__name__}({e})')
+            return None
+
+
+def _get_windows_v10_key(browser_root, logger):
+    path = _find_most_recently_used_file(browser_root, 'Local State')
+    if path is None:
+        logger.error('could not find local state file')
+        return None
+    with open(path, 'r', encoding='utf8') as f:
+        data = json.load(f)
+    try:
+        base64_key = data['os_crypt']['encrypted_key']
+    except KeyError:
+        logger.error('no encrypted key in Local State')
+        return None
+    encrypted_key = compat_b64decode(base64_key)
+    prefix = b'DPAPI'
+    if not encrypted_key.startswith(prefix):
+        logger.error('invalid key')
+        return None
+    return _decrypt_windows_dpapi(encrypted_key[len(prefix):], logger)
+
+
+def pbkdf2_sha1(password, salt, iterations, key_length):
+    return pbkdf2_hmac('sha1', password, salt, iterations, key_length)
+
+
+def _decrypt_aes_cbc(ciphertext, key, logger, initialization_vector=b' ' * 16):
+    plaintext = aes_cbc_decrypt_bytes(ciphertext, key, initialization_vector)
+    padding_length = plaintext[-1]
+    try:
+        return plaintext[:-padding_length].decode('utf-8')
+    except UnicodeDecodeError:
+        logger.warning('failed to decrypt cookie (AES-CBC) because UTF-8 decoding failed. Possibly the key is wrong?', only_once=True)
+        return None
+
+
+def _decrypt_aes_gcm(ciphertext, key, nonce, authentication_tag, logger):
+    try:
+        plaintext = aes_gcm_decrypt_and_verify_bytes(ciphertext, key, authentication_tag, nonce)
+    except ValueError:
+        logger.warning('failed to decrypt cookie (AES-GCM) because the MAC check failed. Possibly the key is wrong?', only_once=True)
+        return None
+
+    try:
+        return plaintext.decode('utf-8')
+    except UnicodeDecodeError:
+        logger.warning('failed to decrypt cookie (AES-GCM) because UTF-8 decoding failed. Possibly the key is wrong?', only_once=True)
+        return None
+
+
+def _decrypt_windows_dpapi(ciphertext, logger):
+    """
+    References:
+        - https://docs.microsoft.com/en-us/windows/win32/api/dpapi/nf-dpapi-cryptunprotectdata
+    """
+    from ctypes.wintypes import DWORD
+
+    class DATA_BLOB(ctypes.Structure):
+        _fields_ = [('cbData', DWORD),
+                    ('pbData', ctypes.POINTER(ctypes.c_char))]
+
+    buffer = ctypes.create_string_buffer(ciphertext)
+    blob_in = DATA_BLOB(ctypes.sizeof(buffer), buffer)
+    blob_out = DATA_BLOB()
+    ret = ctypes.windll.crypt32.CryptUnprotectData(
+        ctypes.byref(blob_in),  # pDataIn
+        None,  # ppszDataDescr: human readable description of pDataIn
+        None,  # pOptionalEntropy: salt?
+        None,  # pvReserved: must be NULL
+        None,  # pPromptStruct: information about prompts to display
+        0,  # dwFlags
+        ctypes.byref(blob_out)  # pDataOut
+    )
+    if not ret:
+        logger.warning('failed to decrypt with DPAPI', only_once=True)
+        return None
+
+    result = ctypes.string_at(blob_out.pbData, blob_out.cbData)
+    ctypes.windll.kernel32.LocalFree(blob_out.pbData)
+    return result
+
+
+def _config_home():
+    return os.environ.get('XDG_CONFIG_HOME', os.path.expanduser('~/.config'))
+
+
+def _open_database_copy(database_path, tmpdir):
+    # cannot open sqlite databases if they are already in use (e.g. by the browser)
+    database_copy_path = os.path.join(tmpdir, 'temporary.sqlite')
+    shutil.copy(database_path, database_copy_path)
+    conn = sqlite3.connect(database_copy_path)
+    return conn.cursor()
+
+
+def _get_column_names(cursor, table_name):
+    table_info = cursor.execute('PRAGMA table_info({})'.format(table_name)).fetchall()
+    return [row[1].decode('utf-8') for row in table_info]
+
+
+def _find_most_recently_used_file(root, filename):
+    # if there are multiple browser profiles, take the most recently used one
+    paths = []
+    for root, dirs, files in os.walk(root):
+        for file in files:
+            if file == filename:
+                paths.append(os.path.join(root, file))
+    return None if not paths else max(paths, key=lambda path: os.lstat(path).st_mtime)
+
+
+def _merge_cookie_jars(jars):
+    output_jar = YoutubeDLCookieJar()
+    for jar in jars:
+        for cookie in jar:
+            output_jar.set_cookie(cookie)
+        if jar.filename is not None:
+            output_jar.filename = jar.filename
+    return output_jar
+
+
+def _is_path(value):
+    return os.path.sep in value
+
+
+def _parse_browser_specification(browser_name, profile=None):
+    browser_name = browser_name.lower()
+    if browser_name not in SUPPORTED_BROWSERS:
+        raise ValueError(f'unsupported browser: "{browser_name}"')
+    if profile is not None and _is_path(profile):
+        profile = os.path.expanduser(profile)
+    return browser_name, profile

yt_dlp/downloader/__init__.py

@@ -3,17 +3,25 @@ from __future__ import unicode_literals
 from ..compat import compat_str
 from ..utils import (
     determine_protocol,
+    NO_DEFAULT
 )
 
 
-def _get_real_downloader(info_dict, protocol=None, *args, **kwargs):
+def get_suitable_downloader(info_dict, params={}, default=NO_DEFAULT, protocol=None, to_stdout=False):
+    info_dict['protocol'] = determine_protocol(info_dict)
     info_copy = info_dict.copy()
-    if protocol:
+    info_copy['to_stdout'] = to_stdout
-        info_copy['protocol'] = protocol
+
-    return get_suitable_downloader(info_copy, *args, **kwargs)
+    downloaders = [_get_suitable_downloader(info_copy, proto, params, default)
+                   for proto in (protocol or info_copy['protocol']).split('+')]
+    if set(downloaders) == {FFmpegFD} and FFmpegFD.can_merge_formats(info_copy, params):
+        return FFmpegFD
+    elif len(downloaders) == 1:
+        return downloaders[0]
+    return None
 
 
-# Some of these require _get_real_downloader
+# Some of these require get_suitable_downloader
 from .common import FileDownloader
 from .dash import DashSegmentsFD
 from .f4m import F4mFD
@@ -69,32 +77,39 @@ def shorten_protocol_name(proto, simplify=False):
     return short_protocol_names.get(proto, proto)
 
 
-def get_suitable_downloader(info_dict, params={}, default=HttpFD):
+def _get_suitable_downloader(info_dict, protocol, params, default):
     """Get the downloader class that can handle the info dict."""
-    protocol = determine_protocol(info_dict)
+    if default is NO_DEFAULT:
-    info_dict['protocol'] = protocol
+        default = HttpFD
 
     # if (info_dict.get('start_time') or info_dict.get('end_time')) and not info_dict.get('requested_formats') and FFmpegFD.can_download(info_dict):
     #     return FFmpegFD
 
+    info_dict['protocol'] = protocol
     downloaders = params.get('external_downloader')
     external_downloader = (
         downloaders if isinstance(downloaders, compat_str) or downloaders is None
         else downloaders.get(shorten_protocol_name(protocol, True), downloaders.get('default')))
-    if external_downloader and external_downloader.lower() == 'native':
-        external_downloader = 'native'
 
-    if external_downloader not in (None, 'native'):
+    if external_downloader is None:
+        if info_dict['to_stdout'] and FFmpegFD.can_merge_formats(info_dict, params):
+            return FFmpegFD
+    elif external_downloader.lower() != 'native':
         ed = get_external_downloader(external_downloader)
         if ed.can_download(info_dict, external_downloader):
             return ed
 
+    if protocol == 'http_dash_segments':
+        if info_dict.get('is_live') and (external_downloader or '').lower() != 'native':
+            return FFmpegFD
+
     if protocol in ('m3u8', 'm3u8_native'):
         if info_dict.get('is_live'):
             return FFmpegFD
-        elif external_downloader == 'native':
+        elif (external_downloader or '').lower() == 'native':
             return HlsFD
-        elif _get_real_downloader(info_dict, 'm3u8_frag_urls', params, None):
+        elif get_suitable_downloader(
+                info_dict, params, None, protocol='m3u8_frag_urls', to_stdout=info_dict['to_stdout']):
             return HlsFD
         elif params.get('hls_prefer_native') is True:
             return HlsFD

yt_dlp/downloader/common.py

@@ -2,11 +2,9 @@ from __future__ import division, unicode_literals
 
 import os
 import re
-import sys
 import time
 import random
 
-from ..compat import compat_os_name
 from ..utils import (
     decodeArgument,
     encodeFilename,
@@ -14,6 +12,13 @@ from ..utils import (
     format_bytes,
     shell_quote,
     timeconvert,
+    timetuple_from_msec,
+)
+from ..minicurses import (
+    MultilineLogger,
+    MultilinePrinter,
+    QuietMultilinePrinter,
+    BreaklineStatusPrinter
 )
 
 
@@ -38,20 +43,22 @@ class FileDownloader(object):
     noresizebuffer:     Do not automatically resize the download buffer.
     continuedl:         Try to continue downloads if possible.
     noprogress:         Do not print the progress bar.
-    logtostderr:        Log messages to stderr instead of stdout.
-    consoletitle:       Display progress in console window's titlebar.
     nopart:             Do not use temporary .part files.
     updatetime:         Use the Last-modified header to set output file timestamps.
     test:               Download only first bytes to test the downloader.
     min_filesize:       Skip files smaller than this size
     max_filesize:       Skip files larger than this size
     xattr_set_filesize: Set ytdl.filesize user xattribute with expected size.
-    external_downloader_args:  A list of additional command-line arguments for the
+    external_downloader_args:  A dictionary of downloader keys (in lower case)
-                        external downloader.
+                        and a list of additional command-line arguments for the
+                        executable. Use 'default' as the name for arguments to be
+                        passed to all downloaders. For compatibility with youtube-dl,
+                        a single list of args can also be used
     hls_use_mpegts:     Use the mpegts container for HLS videos.
     http_chunk_size:    Size of a chunk for chunk-based HTTP downloading. May be
                         useful for bypassing bandwidth throttling imposed by
                         a webserver (experimental)
+    progress_template:  See YoutubeDL.py
 
     Subclasses of this one must re-define the real_download method.
     """
@@ -64,18 +71,17 @@ class FileDownloader(object):
         self.ydl = ydl
         self._progress_hooks = []
         self.params = params
+        self._prepare_multiline_status()
         self.add_progress_hook(self.report_progress)
 
     @staticmethod
     def format_seconds(seconds):
-        (mins, secs) = divmod(seconds, 60)
+        time = timetuple_from_msec(seconds * 1000)
-        (hours, mins) = divmod(mins, 60)
+        if time.hours > 99:
-        if hours > 99:
             return '--:--:--'
-        if hours == 0:
+        if not time.hours:
-            return '%02d:%02d' % (mins, secs)
+            return '%02d:%02d' % time[1:-1]
-        else:
+        return '%02d:%02d:%02d' % time[:-1]
-            return '%02d:%02d:%02d' % (hours, mins, secs)
 
     @staticmethod
     def calc_percent(byte_counter, data_len):
@@ -200,12 +206,12 @@ class FileDownloader(object):
         return filename + '.ytdl'
 
     def try_rename(self, old_filename, new_filename):
+        if old_filename == new_filename:
+            return
         try:
-            if old_filename == new_filename:
+            os.replace(old_filename, new_filename)
-                return
-            os.rename(encodeFilename(old_filename), encodeFilename(new_filename))
         except (IOError, OSError) as err:
-            self.report_error('unable to rename file: %s' % error_to_compat_str(err))
+            self.report_error(f'unable to rename file: {err}')
 
     def try_utime(self, filename, last_modified_hdr):
         """Try to set the last-modified time of the given file."""
@@ -232,39 +238,46 @@ class FileDownloader(object):
         """Report destination filename."""
         self.to_screen('[download] Destination: ' + filename)
 
-    def _report_progress_status(self, msg, is_last_line=False):
+    def _prepare_multiline_status(self, lines=1):
-        fullmsg = '[download] ' + msg
+        if self.params.get('noprogress'):
-        if self.params.get('progress_with_newline', False):
+            self._multiline = QuietMultilinePrinter()
-            self.to_screen(fullmsg)
+        elif self.ydl.params.get('logger'):
+            self._multiline = MultilineLogger(self.ydl.params['logger'], lines)
+        elif self.params.get('progress_with_newline'):
+            self._multiline = BreaklineStatusPrinter(self.ydl._screen_file, lines)
         else:
-            if compat_os_name == 'nt':
+            self._multiline = MultilinePrinter(self.ydl._screen_file, lines, not self.params.get('quiet'))
-                prev_len = getattr(self, '_report_progress_prev_line_length',
+
-                                   0)
+    def _finish_multiline_status(self):
-                if prev_len > len(fullmsg):
+        self._multiline.end()
-                    fullmsg += ' ' * (prev_len - len(fullmsg))
+
-                self._report_progress_prev_line_length = len(fullmsg)
+    def _report_progress_status(self, s):
-                clear_line = '\r'
+        progress_dict = s.copy()
-            else:
+        progress_dict.pop('info_dict')
-                clear_line = ('\r\x1b[K' if sys.stderr.isatty() else '\r')
+        progress_dict = {'info': s['info_dict'], 'progress': progress_dict}
-            self.to_screen(clear_line + fullmsg, skip_eol=not is_last_line)
+
-        self.to_console_title('yt-dlp ' + msg)
+        progress_template = self.params.get('progress_template', {})
+        self._multiline.print_at_line(self.ydl.evaluate_outtmpl(
+            progress_template.get('download') or '[download] %(progress._default_template)s',
+            progress_dict), s.get('progress_idx') or 0)
+        self.to_console_title(self.ydl.evaluate_outtmpl(
+            progress_template.get('download-title') or 'yt-dlp %(progress._default_template)s',
+            progress_dict))
 
     def report_progress(self, s):
         if s['status'] == 'finished':
-            if self.params.get('noprogress', False):
+            if self.params.get('noprogress'):
                 self.to_screen('[download] Download completed')
-            else:
+            msg_template = '100%%'
-                msg_template = '100%%'
+            if s.get('total_bytes') is not None:
-                if s.get('total_bytes') is not None:
+                s['_total_bytes_str'] = format_bytes(s['total_bytes'])
-                    s['_total_bytes_str'] = format_bytes(s['total_bytes'])
+                msg_template += ' of %(_total_bytes_str)s'
-                    msg_template += ' of %(_total_bytes_str)s'
+            if s.get('elapsed') is not None:
-                if s.get('elapsed') is not None:
+                s['_elapsed_str'] = self.format_seconds(s['elapsed'])
-                    s['_elapsed_str'] = self.format_seconds(s['elapsed'])
+                msg_template += ' in %(_elapsed_str)s'
-                    msg_template += ' in %(_elapsed_str)s'
+            s['_percent_str'] = self.format_percent(100)
-                self._report_progress_status(
+            s['_default_template'] = msg_template % s
-                    msg_template % s, is_last_line=True)
+            self._report_progress_status(s)
-
-        if self.params.get('noprogress'):
             return
 
         if s['status'] != 'downloading':
@@ -306,8 +319,12 @@ class FileDownloader(object):
                     msg_template = '%(_downloaded_bytes_str)s at %(_speed_str)s'
             else:
                 msg_template = '%(_percent_str)s % at %(_speed_str)s ETA %(_eta_str)s'
-
+        if s.get('fragment_index') and s.get('fragment_count'):
-        self._report_progress_status(msg_template % s)
+            msg_template += ' (frag %(fragment_index)s/%(fragment_count)s)'
+        elif s.get('fragment_index'):
+            msg_template += ' (frag %(fragment_index)s)'
+        s['_default_template'] = msg_template % s
+        self._report_progress_status(s)
 
     def report_resuming_byte(self, resume_len):
         """Report attempt to resume at given byte."""
@@ -319,12 +336,9 @@ class FileDownloader(object):
             '[download] Got server HTTP error: %s. Retrying (attempt %d of %s) ...'
             % (error_to_compat_str(err), count, self.format_retries(retries)))
 
-    def report_file_already_downloaded(self, file_name):
+    def report_file_already_downloaded(self, *args, **kwargs):
         """Report file has already been fully downloaded."""
-        try:
+        return self.ydl.report_file_already_downloaded(*args, **kwargs)
-            self.to_screen('[download] %s has already been downloaded' % file_name)
-        except UnicodeEncodeError:
-            self.to_screen('[download] The file has already been downloaded')
 
     def report_unable_to_resume(self):
         """Report it was impossible to resume download."""
@@ -342,7 +356,7 @@ class FileDownloader(object):
         """
 
         nooverwrites_and_exists = (
-            not self.params.get('overwrites', subtitle)
+            not self.params.get('overwrites', True)
             and os.path.exists(encodeFilename(filename))
         )
 
@@ -360,7 +374,7 @@ class FileDownloader(object):
                     'filename': filename,
                     'status': 'finished',
                     'total_bytes': os.path.getsize(encodeFilename(filename)),
-                })
+                }, info_dict)
                 return True, False
 
         if subtitle is False:
@@ -382,13 +396,21 @@ class FileDownloader(object):
                     '[download] Sleeping %s seconds ...' % (
                         sleep_interval_sub))
                 time.sleep(sleep_interval_sub)
-        return self.real_download(filename, info_dict), True
+        ret = self.real_download(filename, info_dict)
+        self._finish_multiline_status()
+        return ret, True
 
     def real_download(self, filename, info_dict):
         """Real download process. Redefine in subclasses."""
         raise NotImplementedError('This method must be implemented by subclasses')
 
-    def _hook_progress(self, status):
+    def _hook_progress(self, status, info_dict):
+        if not self._progress_hooks:
+            return
+        status['info_dict'] = info_dict
+        # youtube-dl passes the same status object to all the hooks.
+        # Some third party scripts seems to be relying on this.
+        # So keep this behavior if possible
         for ph in self._progress_hooks:
             ph(status)
 

yt_dlp/downloader/dash.py

@@ -1,6 +1,6 @@
 from __future__ import unicode_literals
 
-from ..downloader import _get_real_downloader
+from ..downloader import get_suitable_downloader
 from .fragment import FragmentFD
 
 from ..utils import urljoin
@@ -15,11 +15,15 @@ class DashSegmentsFD(FragmentFD):
     FD_NAME = 'dashsegments'
 
     def real_download(self, filename, info_dict):
+        if info_dict.get('is_live'):
+            self.report_error('Live DASH videos are not supported')
+
         fragment_base_url = info_dict.get('fragment_base_url')
         fragments = info_dict['fragments'][:1] if self.params.get(
             'test', False) else info_dict['fragments']
 
-        real_downloader = _get_real_downloader(info_dict, 'dash_frag_urls', self.params, None)
+        real_downloader = get_suitable_downloader(
+            info_dict, self.params, None, protocol='dash_frag_urls', to_stdout=(filename == '-'))
 
         ctx = {
             'filename': filename,
@@ -29,7 +33,7 @@ class DashSegmentsFD(FragmentFD):
         if real_downloader:
             self._prepare_external_frag_download(ctx)
         else:
-            self._prepare_and_start_frag_download(ctx)
+            self._prepare_and_start_frag_download(ctx, info_dict)
 
         fragments_to_download = []
         frag_index = 0
@@ -51,15 +55,8 @@ class DashSegmentsFD(FragmentFD):
         if real_downloader:
             self.to_screen(
                 '[%s] Fragment downloads will be delegated to %s' % (self.FD_NAME, real_downloader.get_basename()))
-            info_copy = info_dict.copy()
+            info_dict['fragments'] = fragments_to_download
-            info_copy['fragments'] = fragments_to_download
             fd = real_downloader(self.ydl, self.params)
-            # TODO: Make progress updates work without hooking twice
+            return fd.real_download(filename, info_dict)
-            # for ph in self._progress_hooks:
+
-            #     fd.add_progress_hook(ph)
+        return self.download_and_append_fragments(ctx, fragments_to_download, info_dict)
-            success = fd.real_download(filename, info_copy)
-            if not success:
-                return False
-        else:
-            self.download_and_append_fragments(ctx, fragments_to_download, info_dict)
-        return True

yt_dlp/downloader/external.py

@@ -6,13 +6,7 @@ import subprocess
 import sys
 import time
 
-try:
+from .fragment import FragmentFD
-    from Crypto.Cipher import AES
-    can_decrypt_frag = True
-except ImportError:
-    can_decrypt_frag = False
-
-from .common import FileDownloader
 from ..compat import (
     compat_setenv,
     compat_str,
@@ -22,20 +16,19 @@ from ..utils import (
     cli_option,
     cli_valueless_option,
     cli_bool_option,
-    cli_configuration_args,
+    _configuration_args,
     encodeFilename,
     encodeArgument,
     handle_youtubedl_headers,
     check_executable,
-    is_outdated_version,
+    Popen,
-    process_communicate_or_kill,
-    sanitized_Request,
     sanitize_open,
 )
 
 
-class ExternalFD(FileDownloader):
+class ExternalFD(FragmentFD):
     SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps')
+    can_download_to_stdout = False
 
     def real_download(self, filename, info_dict):
         self.report_destination(filename)
@@ -67,7 +60,7 @@ class ExternalFD(FileDownloader):
                     'downloaded_bytes': fsize,
                     'total_bytes': fsize,
                 })
-            self._hook_progress(status)
+            self._hook_progress(status, info_dict)
             return True
         else:
             self.to_stderr('\n')
@@ -93,7 +86,9 @@ class ExternalFD(FileDownloader):
 
     @classmethod
     def supports(cls, info_dict):
-        return info_dict['protocol'] in cls.SUPPORTED_PROTOCOLS
+        return (
+            (cls.can_download_to_stdout or not info_dict.get('to_stdout'))
+            and info_dict['protocol'] in cls.SUPPORTED_PROTOCOLS)
 
     @classmethod
     def can_download(cls, info_dict, path=None):
@@ -108,11 +103,10 @@ class ExternalFD(FileDownloader):
     def _valueless_option(self, command_option, param, expected_value=True):
         return cli_valueless_option(self.params, command_option, param, expected_value)
 
-    def _configuration_args(self, *args, **kwargs):
+    def _configuration_args(self, keys=None, *args, **kwargs):
-        return cli_configuration_args(
+        return _configuration_args(
-            self.params.get('external_downloader_args'),
+            self.get_basename(), self.params.get('external_downloader_args'), self.get_basename(),
-            [self.get_basename(), 'default'],
+            keys, *args, **kwargs)
-            *args, **kwargs)
 
     def _call_downloader(self, tmpfilename, info_dict):
         """ Either overwrite this or implement _make_cmd """
@@ -120,73 +114,54 @@ class ExternalFD(FileDownloader):
 
         self._debug_cmd(cmd)
 
-        if 'fragments' in info_dict:
+        if 'fragments' not in info_dict:
-            fragment_retries = self.params.get('fragment_retries', 0)
+            p = Popen(cmd, stderr=subprocess.PIPE)
-            skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
+            _, stderr = p.communicate_or_kill()
-
-            count = 0
-            while count <= fragment_retries:
-                p = subprocess.Popen(
-                    cmd, stderr=subprocess.PIPE)
-                _, stderr = process_communicate_or_kill(p)
-                if p.returncode == 0:
-                    break
-                # TODO: Decide whether to retry based on error code
-                # https://aria2.github.io/manual/en/html/aria2c.html#exit-status
-                self.to_stderr(stderr.decode('utf-8', 'replace'))
-                count += 1
-                if count <= fragment_retries:
-                    self.to_screen(
-                        '[%s] Got error. Retrying fragments (attempt %d of %s)...'
-                        % (self.get_basename(), count, self.format_retries(fragment_retries)))
-            if count > fragment_retries:
-                if not skip_unavailable_fragments:
-                    self.report_error('Giving up after %s fragment retries' % fragment_retries)
-                    return -1
-
-            dest, _ = sanitize_open(tmpfilename, 'wb')
-            for frag_index, fragment in enumerate(info_dict['fragments']):
-                fragment_filename = '%s-Frag%d' % (tmpfilename, frag_index)
-                try:
-                    src, _ = sanitize_open(fragment_filename, 'rb')
-                except IOError:
-                    if skip_unavailable_fragments and frag_index > 1:
-                        self.to_screen('[%s] Skipping fragment %d ...' % (self.get_basename(), frag_index))
-                        continue
-                    self.report_error('Unable to open fragment %d' % frag_index)
-                    return -1
-                decrypt_info = fragment.get('decrypt_info')
-                if decrypt_info:
-                    if decrypt_info['METHOD'] == 'AES-128':
-                        iv = decrypt_info.get('IV')
-                        decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
-                            self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
-                        encrypted_data = src.read()
-                        decrypted_data = AES.new(
-                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(encrypted_data)
-                        dest.write(decrypted_data)
-                    else:
-                        fragment_data = src.read()
-                        dest.write(fragment_data)
-                else:
-                    fragment_data = src.read()
-                    dest.write(fragment_data)
-                src.close()
-                if not self.params.get('keep_fragments', False):
-                    os.remove(encodeFilename(fragment_filename))
-            dest.close()
-            os.remove(encodeFilename('%s.frag.urls' % tmpfilename))
-        else:
-            p = subprocess.Popen(
-                cmd, stderr=subprocess.PIPE)
-            _, stderr = process_communicate_or_kill(p)
             if p.returncode != 0:
                 self.to_stderr(stderr.decode('utf-8', 'replace'))
-        return p.returncode
+            return p.returncode
 
-    def _prepare_url(self, info_dict, url):
+        fragment_retries = self.params.get('fragment_retries', 0)
-        headers = info_dict.get('http_headers')
+        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
-        return sanitized_Request(url, None, headers) if headers else url
+
+        count = 0
+        while count <= fragment_retries:
+            p = Popen(cmd, stderr=subprocess.PIPE)
+            _, stderr = p.communicate_or_kill()
+            if p.returncode == 0:
+                break
+            # TODO: Decide whether to retry based on error code
+            # https://aria2.github.io/manual/en/html/aria2c.html#exit-status
+            self.to_stderr(stderr.decode('utf-8', 'replace'))
+            count += 1
+            if count <= fragment_retries:
+                self.to_screen(
+                    '[%s] Got error. Retrying fragments (attempt %d of %s)...'
+                    % (self.get_basename(), count, self.format_retries(fragment_retries)))
+        if count > fragment_retries:
+            if not skip_unavailable_fragments:
+                self.report_error('Giving up after %s fragment retries' % fragment_retries)
+                return -1
+
+        decrypt_fragment = self.decrypter(info_dict)
+        dest, _ = sanitize_open(tmpfilename, 'wb')
+        for frag_index, fragment in enumerate(info_dict['fragments']):
+            fragment_filename = '%s-Frag%d' % (tmpfilename, frag_index)
+            try:
+                src, _ = sanitize_open(fragment_filename, 'rb')
+            except IOError as err:
+                if skip_unavailable_fragments and frag_index > 1:
+                    self.report_skip_fragment(frag_index, err)
+                    continue
+                self.report_error(f'Unable to open fragment {frag_index}; {err}')
+                return -1
+            dest.write(decrypt_fragment(fragment, src.read()))
+            src.close()
+            if not self.params.get('keep_fragments', False):
+                os.remove(encodeFilename(fragment_filename))
+        dest.close()
+        os.remove(encodeFilename('%s.frag.urls' % tmpfilename))
+        return 0
 
 
 class CurlFD(ExternalFD):
@@ -221,8 +196,8 @@ class CurlFD(ExternalFD):
         self._debug_cmd(cmd)
 
         # curl writes the progress to stderr so don't capture it.
-        p = subprocess.Popen(cmd)
+        p = Popen(cmd)
-        process_communicate_or_kill(p)
+        p.communicate_or_kill()
         return p.returncode
 
 
@@ -286,6 +261,7 @@ class Aria2cFD(ExternalFD):
         if info_dict.get('http_headers') is not None:
             for key, val in info_dict['http_headers'].items():
                 cmd += ['--header', '%s: %s' % (key, val)]
+        cmd += self._option('--max-overall-download-limit', 'ratelimit')
         cmd += self._option('--interface', 'source_address')
         cmd += self._option('--all-proxy', 'proxy')
         cmd += self._bool_option('--check-certificate', 'nocheckcertificate', 'false', 'true', '=')
@@ -340,17 +316,32 @@ class HttpieFD(ExternalFD):
 
 
 class FFmpegFD(ExternalFD):
-    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'm3u8_native', 'rtsp', 'rtmp', 'rtmp_ffmpeg', 'mms')
+    SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'm3u8', 'm3u8_native', 'rtsp', 'rtmp', 'rtmp_ffmpeg', 'mms', 'http_dash_segments')
+    can_download_to_stdout = True
 
     @classmethod
     def available(cls, path=None):
         # TODO: Fix path for ffmpeg
+        # Fixme: This may be wrong when --ffmpeg-location is used
         return FFmpegPostProcessor().available
 
+    @classmethod
+    def supports(cls, info_dict):
+        return all(proto in cls.SUPPORTED_PROTOCOLS for proto in info_dict['protocol'].split('+'))
+
     def on_process_started(self, proc, stdin):
         """ Override this in subclasses  """
         pass
 
+    @classmethod
+    def can_merge_formats(cls, info_dict, params):
+        return (
+            info_dict.get('requested_formats')
+            and info_dict.get('protocol')
+            and not params.get('allow_unplayable_formats')
+            and 'no-direct-merge' not in params.get('compat_opts', [])
+            and cls.can_download(info_dict))
+
     def _call_downloader(self, tmpfilename, info_dict):
         urls = [f['url'] for f in info_dict.get('requested_formats', [])] or [info_dict['url']]
         ffpp = FFmpegPostProcessor(downloader=self)
@@ -368,6 +359,9 @@ class FFmpegFD(ExternalFD):
         if not self.params.get('verbose'):
             args += ['-hide_banner']
 
+        args += info_dict.get('_ffmpeg_args', [])
+
+        # This option exists only for compatibility. Extractors should use `_ffmpeg_args` instead
         seekable = info_dict.get('_seekable')
         if seekable is not None:
             # setting -seekable prevents ffmpeg from guessing if the server
@@ -377,8 +371,6 @@ class FFmpegFD(ExternalFD):
             # http://trac.ffmpeg.org/ticket/6125#comment:10
             args += ['-seekable', '1' if seekable else '0']
 
-        args += self._configuration_args()
-
         # start_time = info_dict.get('start_time') or 0
         # if start_time:
         #     args += ['-ss', compat_str(start_time)]
@@ -444,19 +436,20 @@ class FFmpegFD(ExternalFD):
             elif isinstance(conn, compat_str):
                 args += ['-rtmp_conn', conn]
 
-        for url in urls:
+        for i, url in enumerate(urls):
-            args += ['-i', url]
+            args += self._configuration_args((f'_i{i + 1}', '_i')) + ['-i', url]
+
         args += ['-c', 'copy']
-        if info_dict.get('requested_formats'):
+        if info_dict.get('requested_formats') or protocol == 'http_dash_segments':
-            for (i, fmt) in enumerate(info_dict['requested_formats']):
+            for (i, fmt) in enumerate(info_dict.get('requested_formats') or [info_dict]):
-                if fmt.get('acodec') != 'none':
+                stream_number = fmt.get('manifest_stream_number', 0)
-                    args.extend(['-map', '%d:a:0' % i])
+                a_or_v = 'a' if fmt.get('acodec') != 'none' else 'v'
-                if fmt.get('vcodec') != 'none':
+                args.extend(['-map', f'{i}:{a_or_v}:{stream_number}'])
-                    args.extend(['-map', '%d:v:0' % i])
 
         if self.params.get('test', False):
             args += ['-fs', compat_str(self._TEST_FILE_SIZE)]
 
+        ext = info_dict['ext']
         if protocol in ('m3u8', 'm3u8_native'):
             use_mpegts = (tmpfilename == '-') or self.params.get('hls_use_mpegts')
             if use_mpegts is None:
@@ -465,19 +458,22 @@ class FFmpegFD(ExternalFD):
                 args += ['-f', 'mpegts']
             else:
                 args += ['-f', 'mp4']
-                if (ffpp.basename == 'ffmpeg' and is_outdated_version(ffpp._versions['ffmpeg'], '3.2', False)) and (not info_dict.get('acodec') or info_dict['acodec'].split('.')[0] in ('aac', 'mp4a')):
+                if (ffpp.basename == 'ffmpeg' and ffpp._features.get('needs_adtstoasc')) and (not info_dict.get('acodec') or info_dict['acodec'].split('.')[0] in ('aac', 'mp4a')):
                     args += ['-bsf:a', 'aac_adtstoasc']
         elif protocol == 'rtmp':
             args += ['-f', 'flv']
+        elif ext == 'mp4' and tmpfilename == '-':
+            args += ['-f', 'mpegts']
         else:
-            args += ['-f', EXT_TO_OUT_FORMATS.get(info_dict['ext'], info_dict['ext'])]
+            args += ['-f', EXT_TO_OUT_FORMATS.get(ext, ext)]
+
+        args += self._configuration_args(('_o1', '_o', ''))
 
         args = [encodeArgument(opt) for opt in args]
         args.append(encodeFilename(ffpp._ffmpeg_filename_argument(tmpfilename), True))
-
         self._debug_cmd(args)
 
-        proc = subprocess.Popen(args, stdin=subprocess.PIPE, env=env)
+        proc = Popen(args, stdin=subprocess.PIPE, env=env)
         if url in ('-', 'pipe:'):
             self.on_process_started(proc, proc.stdin)
         try:
@@ -489,7 +485,7 @@ class FFmpegFD(ExternalFD):
             # streams). Note that Windows is not affected and produces playable
             # files (see https://github.com/ytdl-org/youtube-dl/issues/8300).
             if isinstance(e, KeyboardInterrupt) and sys.platform != 'win32' and url not in ('-', 'pipe:'):
-                process_communicate_or_kill(proc, b'q')
+                proc.communicate_or_kill(b'q')
             else:
                 proc.kill()
                 proc.wait()
@@ -504,7 +500,7 @@ class AVconvFD(FFmpegFD):
 _BY_NAME = dict(
     (klass.get_basename(), klass)
     for name, klass in globals().items()
-    if name.endswith('FD') and name != 'ExternalFD'
+    if name.endswith('FD') and name not in ('ExternalFD', 'FragmentFD')
 )
 
 

yt_dlp/downloader/f4m.py

@@ -380,7 +380,7 @@ class F4mFD(FragmentFD):
 
         base_url_parsed = compat_urllib_parse_urlparse(base_url)
 
-        self._start_frag_download(ctx)
+        self._start_frag_download(ctx, info_dict)
 
         frag_index = 0
         while fragments_list:
@@ -434,6 +434,6 @@ class F4mFD(FragmentFD):
                     msg = 'Missed %d fragments' % (fragments_list[0][1] - (frag_i + 1))
                     self.report_warning(msg)
 
-        self._finish_frag_download(ctx)
+        self._finish_frag_download(ctx, info_dict)
 
         return True

yt_dlp/downloader/fragment.py

@@ -3,12 +3,7 @@ from __future__ import division, unicode_literals
 import os
 import time
 import json
-
+from math import ceil
-try:
-    from Crypto.Cipher import AES
-    can_decrypt_frag = True
-except ImportError:
-    can_decrypt_frag = False
 
 try:
     import concurrent.futures
@@ -18,6 +13,7 @@ except ImportError:
 
 from .common import FileDownloader
 from .http import HttpFD
+from ..aes import aes_cbc_decrypt_bytes
 from ..compat import (
     compat_urllib_error,
     compat_struct_pack,
@@ -35,6 +31,10 @@ class HttpQuietDownloader(HttpFD):
     def to_screen(self, *args, **kargs):
         pass
 
+    def report_retry(self, err, count, retries):
+        super().to_screen(
+            f'[download] Got server HTTP error: {err}. Retrying (attempt {count} of {self.format_retries(retries)}) ...')
+
 
 class FragmentFD(FileDownloader):
     """
@@ -48,6 +48,7 @@ class FragmentFD(FileDownloader):
                         Skip unavailable fragments (DASH and hlsnative only)
     keep_fragments:     Keep downloaded fragments on disk after downloading is
                         finished
+    concurrent_fragment_downloads:  The number of threads to use for native hls and dash downloads
     _no_ytdl_file:      Don't use .ytdl file
 
     For each incomplete fragment download yt-dlp keeps on disk a special
@@ -76,16 +77,17 @@ class FragmentFD(FileDownloader):
             '\r[download] Got server HTTP error: %s. Retrying fragment %d (attempt %d of %s) ...'
             % (error_to_compat_str(err), frag_index, count, self.format_retries(retries)))
 
-    def report_skip_fragment(self, frag_index):
+    def report_skip_fragment(self, frag_index, err=None):
-        self.to_screen('[download] Skipping fragment %d ...' % frag_index)
+        err = f' {err};' if err else ''
+        self.to_screen(f'[download]{err} Skipping fragment {frag_index:d} ...')
 
     def _prepare_url(self, info_dict, url):
         headers = info_dict.get('http_headers')
         return sanitized_Request(url, None, headers) if headers else url
 
-    def _prepare_and_start_frag_download(self, ctx):
+    def _prepare_and_start_frag_download(self, ctx, info_dict):
         self._prepare_frag_download(ctx)
-        self._start_frag_download(ctx)
+        self._start_frag_download(ctx, info_dict)
 
     def __do_ytdl_file(self, ctx):
         return not ctx['live'] and not ctx['tmpfilename'] == '-' and not self.params.get('_no_ytdl_file')
@@ -105,17 +107,19 @@ class FragmentFD(FileDownloader):
 
     def _write_ytdl_file(self, ctx):
         frag_index_stream, _ = sanitize_open(self.ytdl_filename(ctx['filename']), 'w')
-        downloader = {
+        try:
-            'current_fragment': {
+            downloader = {
-                'index': ctx['fragment_index'],
+                'current_fragment': {
-            },
+                    'index': ctx['fragment_index'],
-        }
+                },
-        if 'extra_state' in ctx:
+            }
-            downloader['extra_state'] = ctx['extra_state']
+            if 'extra_state' in ctx:
-        if ctx.get('fragment_count') is not None:
+                downloader['extra_state'] = ctx['extra_state']
-            downloader['fragment_count'] = ctx['fragment_count']
+            if ctx.get('fragment_count') is not None:
-        frag_index_stream.write(json.dumps({'downloader': downloader}))
+                downloader['fragment_count'] = ctx['fragment_count']
-        frag_index_stream.close()
+            frag_index_stream.write(json.dumps({'downloader': downloader}))
+        finally:
+            frag_index_stream.close()
 
     def _download_fragment(self, ctx, frag_url, info_dict, headers=None, request_data=None):
         fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], ctx['fragment_index'])
@@ -123,6 +127,7 @@ class FragmentFD(FileDownloader):
             'url': frag_url,
             'http_headers': headers or info_dict.get('http_headers'),
             'request_data': request_data,
+            'ctx_id': ctx.get('ctx_id'),
         }
         success = ctx['dl'].download(fragment_filename, fragment_info_dict)
         if not success:
@@ -167,7 +172,7 @@ class FragmentFD(FileDownloader):
             self.ydl,
             {
                 'continuedl': True,
-                'quiet': True,
+                'quiet': self.params.get('quiet'),
                 'noprogress': True,
                 'ratelimit': self.params.get('ratelimit'),
                 'retries': self.params.get('retries', 0),
@@ -219,9 +224,10 @@ class FragmentFD(FileDownloader):
             'complete_frags_downloaded_bytes': resume_len,
         })
 
-    def _start_frag_download(self, ctx):
+    def _start_frag_download(self, ctx, info_dict):
         resume_len = ctx['complete_frags_downloaded_bytes']
         total_frags = ctx['total_frags']
+        ctx_id = ctx.get('ctx_id')
         # This dict stores the download progress, it's updated by the progress
         # hook
         state = {
@@ -236,6 +242,7 @@ class FragmentFD(FileDownloader):
         start = time.time()
         ctx.update({
             'started': start,
+            'fragment_started': start,
             # Amount of fragment's bytes downloaded by the time of the previous
             # frag progress hook invocation
             'prev_frag_downloaded_bytes': 0,
@@ -245,9 +252,16 @@ class FragmentFD(FileDownloader):
             if s['status'] not in ('downloading', 'finished'):
                 return
 
+            if ctx_id is not None and s.get('ctx_id') != ctx_id:
+                return
+
+            state['max_progress'] = ctx.get('max_progress')
+            state['progress_idx'] = ctx.get('progress_idx')
+
             time_now = time.time()
             state['elapsed'] = time_now - start
             frag_total_bytes = s.get('total_bytes') or 0
+            s['fragment_info_dict'] = s.pop('info_dict', {})
             if not ctx['live']:
                 estimated_size = (
                     (ctx['complete_frags_downloaded_bytes'] + frag_total_bytes)
@@ -259,6 +273,9 @@ class FragmentFD(FileDownloader):
                 ctx['fragment_index'] = state['fragment_index']
                 state['downloaded_bytes'] += frag_total_bytes - ctx['prev_frag_downloaded_bytes']
                 ctx['complete_frags_downloaded_bytes'] = state['downloaded_bytes']
+                ctx['speed'] = state['speed'] = self.calc_speed(
+                    ctx['fragment_started'], time_now, frag_total_bytes)
+                ctx['fragment_started'] = time.time()
                 ctx['prev_frag_downloaded_bytes'] = 0
             else:
                 frag_downloaded_bytes = s['downloaded_bytes']
@@ -267,16 +284,16 @@ class FragmentFD(FileDownloader):
                     state['eta'] = self.calc_eta(
                         start, time_now, estimated_size - resume_len,
                         state['downloaded_bytes'] - resume_len)
-                state['speed'] = s.get('speed') or ctx.get('speed')
+                ctx['speed'] = state['speed'] = self.calc_speed(
-                ctx['speed'] = state['speed']
+                    ctx['fragment_started'], time_now, frag_downloaded_bytes)
                 ctx['prev_frag_downloaded_bytes'] = frag_downloaded_bytes
-            self._hook_progress(state)
+            self._hook_progress(state, info_dict)
 
         ctx['dl'].add_progress_hook(frag_progress_hook)
 
         return start
 
-    def _finish_frag_download(self, ctx):
+    def _finish_frag_download(self, ctx, info_dict):
         ctx['dest_stream'].close()
         if self.__do_ytdl_file(ctx):
             ytdl_filename = encodeFilename(self.ytdl_filename(ctx['filename']))
@@ -303,7 +320,10 @@ class FragmentFD(FileDownloader):
             'filename': ctx['filename'],
             'status': 'finished',
             'elapsed': elapsed,
-        })
+            'ctx_id': ctx.get('ctx_id'),
+            'max_progress': ctx.get('max_progress'),
+            'progress_idx': ctx.get('progress_idx'),
+        }, info_dict)
 
     def _prepare_external_frag_download(self, ctx):
         if 'live' not in ctx:
@@ -326,22 +346,81 @@ class FragmentFD(FileDownloader):
             'fragment_index': 0,
         })
 
-    def download_and_append_fragments(self, ctx, fragments, info_dict, pack_func=None):
+    def decrypter(self, info_dict):
+        _key_cache = {}
+
+        def _get_key(url):
+            if url not in _key_cache:
+                _key_cache[url] = self.ydl.urlopen(self._prepare_url(info_dict, url)).read()
+            return _key_cache[url]
+
+        def decrypt_fragment(fragment, frag_content):
+            decrypt_info = fragment.get('decrypt_info')
+            if not decrypt_info or decrypt_info['METHOD'] != 'AES-128':
+                return frag_content
+            iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', fragment['media_sequence'])
+            decrypt_info['KEY'] = decrypt_info.get('KEY') or _get_key(info_dict.get('_decryption_key_url') or decrypt_info['URI'])
+            # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
+            # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
+            # not what it decrypts to.
+            if self.params.get('test', False):
+                return frag_content
+            decrypted_data = aes_cbc_decrypt_bytes(frag_content, decrypt_info['KEY'], iv)
+            return decrypted_data[:-decrypted_data[-1]]
+
+        return decrypt_fragment
+
+    def download_and_append_fragments_multiple(self, *args, pack_func=None, finish_func=None):
+        '''
+        @params (ctx1, fragments1, info_dict1), (ctx2, fragments2, info_dict2), ...
+                all args must be either tuple or list
+        '''
+        max_progress = len(args)
+        if max_progress == 1:
+            return self.download_and_append_fragments(*args[0], pack_func=pack_func, finish_func=finish_func)
+        max_workers = self.params.get('concurrent_fragment_downloads', max_progress)
+        if max_progress > 1:
+            self._prepare_multiline_status(max_progress)
+
+        def thread_func(idx, ctx, fragments, info_dict, tpe):
+            ctx['max_progress'] = max_progress
+            ctx['progress_idx'] = idx
+            return self.download_and_append_fragments(ctx, fragments, info_dict, pack_func=pack_func, finish_func=finish_func, tpe=tpe)
+
+        class FTPE(concurrent.futures.ThreadPoolExecutor):
+            # has to stop this or it's going to wait on the worker thread itself
+            def __exit__(self, exc_type, exc_val, exc_tb):
+                pass
+
+        spins = []
+        for idx, (ctx, fragments, info_dict) in enumerate(args):
+            tpe = FTPE(ceil(max_workers / max_progress))
+            job = tpe.submit(thread_func, idx, ctx, fragments, info_dict, tpe)
+            spins.append((tpe, job))
+
+        result = True
+        for tpe, job in spins:
+            try:
+                result = result and job.result()
+            finally:
+                tpe.shutdown(wait=True)
+        return result
+
+    def download_and_append_fragments(self, ctx, fragments, info_dict, *, pack_func=None, finish_func=None, tpe=None):
         fragment_retries = self.params.get('fragment_retries', 0)
-        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
+        is_fatal = (lambda idx: idx == 0) if self.params.get('skip_unavailable_fragments', True) else (lambda _: True)
-        test = self.params.get('test', False)
         if not pack_func:
             pack_func = lambda frag_content, _: frag_content
 
         def download_fragment(fragment, ctx):
             frag_index = ctx['fragment_index'] = fragment['frag_index']
-            headers = info_dict.get('http_headers', {})
+            headers = info_dict.get('http_headers', {}).copy()
             byte_range = fragment.get('byte_range')
             if byte_range:
                 headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'] - 1)
 
             # Never skip the first fragment
-            fatal = (fragment.get('index') or frag_index) == 0 or not skip_unavailable_fragments
+            fatal = is_fatal(fragment.get('index') or (frag_index - 1))
             count, frag_content = 0, None
             while count <= fragment_retries:
                 try:
@@ -372,25 +451,10 @@ class FragmentFD(FileDownloader):
                 return False, frag_index
             return frag_content, frag_index
 
-        def decrypt_fragment(fragment, frag_content):
-            decrypt_info = fragment.get('decrypt_info')
-            if not decrypt_info or decrypt_info['METHOD'] != 'AES-128':
-                return frag_content
-            iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', fragment['media_sequence'])
-            decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
-                self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
-            # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
-            # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
-            # not what it decrypts to.
-            if test:
-                return frag_content
-            return AES.new(decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
-
         def append_fragment(frag_content, frag_index, ctx):
             if not frag_content:
-                fatal = frag_index == 1 or not skip_unavailable_fragments
+                if not is_fatal(frag_index - 1):
-                if not fatal:
+                    self.report_skip_fragment(frag_index, 'fragment not found')
-                    self.report_skip_fragment(frag_index)
                     return True
                 else:
                     ctx['dest_stream'].close()
@@ -400,20 +464,18 @@ class FragmentFD(FileDownloader):
             self._append_fragment(ctx, pack_func(frag_content, frag_index))
             return True
 
+        decrypt_fragment = self.decrypter(info_dict)
+
         max_workers = self.params.get('concurrent_fragment_downloads', 1)
         if can_threaded_download and max_workers > 1:
 
             def _download_fragment(fragment):
-                try:
+                ctx_copy = ctx.copy()
-                    ctx_copy = ctx.copy()
+                frag_content, frag_index = download_fragment(fragment, ctx_copy)
-                    frag_content, frag_index = download_fragment(fragment, ctx_copy)
+                return fragment, frag_content, frag_index, ctx_copy.get('fragment_filename_sanitized')
-                    return fragment, frag_content, frag_index, ctx_copy.get('fragment_filename_sanitized')
-                except Exception:
-                    # Return immediately on exception so that it is raised in the main thread
-                    return
 
             self.report_warning('The download speed shown is only of one thread. This is a known issue and patches are welcome')
-            with concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
+            with tpe or concurrent.futures.ThreadPoolExecutor(max_workers) as pool:
                 for fragment, frag_content, frag_index, frag_filename in pool.map(_download_fragment, fragments):
                     ctx['fragment_filename_sanitized'] = frag_filename
                     ctx['fragment_index'] = frag_index
@@ -427,4 +489,8 @@ class FragmentFD(FileDownloader):
                 if not result:
                     return False
 
-        self._finish_frag_download(ctx)
+        if finish_func is not None:
+            ctx['dest_stream'].write(finish_func())
+            ctx['dest_stream'].flush()
+        self._finish_frag_download(ctx, info_dict)
+        return True

yt_dlp/downloader/hls.py

@@ -4,11 +4,12 @@ import re
 import io
 import binascii
 
-from ..downloader import _get_real_downloader
+from ..downloader import get_suitable_downloader
-from .fragment import FragmentFD, can_decrypt_frag
+from .fragment import FragmentFD
 from .external import FFmpegFD
 
 from ..compat import (
+    compat_pycrypto_AES,
     compat_urlparse,
 )
 from ..utils import (
@@ -29,7 +30,7 @@ class HlsFD(FragmentFD):
     FD_NAME = 'hlsnative'
 
     @staticmethod
-    def can_download(manifest, info_dict, allow_unplayable_formats=False, with_crypto=can_decrypt_frag):
+    def can_download(manifest, info_dict, allow_unplayable_formats=False):
         UNSUPPORTED_FEATURES = [
             # r'#EXT-X-BYTERANGE',  # playlists composed of byte ranges of media files [2]
 
@@ -56,9 +57,6 @@ class HlsFD(FragmentFD):
 
         def check_results():
             yield not info_dict.get('is_live')
-            is_aes128_enc = '#EXT-X-KEY:METHOD=AES-128' in manifest
-            yield with_crypto or not is_aes128_enc
-            yield not (is_aes128_enc and r'#EXT-X-BYTERANGE' in manifest)
             for feature in UNSUPPORTED_FEATURES:
                 yield not re.search(feature, manifest)
         return all(check_results())
@@ -71,25 +69,27 @@ class HlsFD(FragmentFD):
         man_url = urlh.geturl()
         s = urlh.read().decode('utf-8', 'ignore')
 
-        if not self.can_download(s, info_dict, self.params.get('allow_unplayable_formats')):
+        can_download, message = self.can_download(s, info_dict, self.params.get('allow_unplayable_formats')), None
-            if info_dict.get('extra_param_to_segment_url') or info_dict.get('_decryption_key_url'):
+        if can_download and not compat_pycrypto_AES and '#EXT-X-KEY:METHOD=AES-128' in s:
-                self.report_error('pycryptodome not found. Please install')
+            if FFmpegFD.available():
-                return False
+                can_download, message = False, 'The stream has AES-128 encryption and pycryptodomex is not available'
-            if self.can_download(s, info_dict, with_crypto=True):
+            else:
-                self.report_warning('pycryptodome is needed to download this file natively')
+                message = ('The stream has AES-128 encryption and neither ffmpeg nor pycryptodomex are available; '
+                           'Decryption will be performed natively, but will be extremely slow')
+        if not can_download:
+            message = message or 'Unsupported features have been detected'
             fd = FFmpegFD(self.ydl, self.params)
-            self.report_warning(
+            self.report_warning(f'{message}; extraction will be delegated to {fd.get_basename()}')
-                '%s detected unsupported features; extraction will be delegated to %s' % (self.FD_NAME, fd.get_basename()))
-            # TODO: Make progress updates work without hooking twice
-            # for ph in self._progress_hooks:
-            #     fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
+        elif message:
+            self.report_warning(message)
 
         is_webvtt = info_dict['ext'] == 'vtt'
         if is_webvtt:
             real_downloader = None  # Packing the fragments is not currently supported for external downloader
         else:
-            real_downloader = _get_real_downloader(info_dict, 'm3u8_frag_urls', self.params, None)
+            real_downloader = get_suitable_downloader(
+                info_dict, self.params, None, protocol='m3u8_frag_urls', to_stdout=(filename == '-'))
         if real_downloader and not real_downloader.supports_manifest(s):
             real_downloader = None
         if real_downloader:
@@ -133,7 +133,7 @@ class HlsFD(FragmentFD):
         if real_downloader:
             self._prepare_external_frag_download(ctx)
         else:
-            self._prepare_and_start_frag_download(ctx)
+            self._prepare_and_start_frag_download(ctx, info_dict)
 
         extra_state = ctx.setdefault('extra_state', {})
 
@@ -174,6 +174,7 @@ class HlsFD(FragmentFD):
                         'byte_range': byte_range,
                         'media_sequence': media_sequence,
                     })
+                    media_sequence += 1
 
                 elif line.startswith('#EXT-X-MAP'):
                     if format_index and discontinuity_count != format_index:
@@ -198,6 +199,7 @@ class HlsFD(FragmentFD):
                         'byte_range': byte_range,
                         'media_sequence': media_sequence
                     })
+                    media_sequence += 1
 
                     if map_info.get('BYTERANGE'):
                         splitted_byte_range = map_info.get('BYTERANGE').split('@')
@@ -237,91 +239,111 @@ class HlsFD(FragmentFD):
                 elif line.startswith('#EXT-X-DISCONTINUITY'):
                     discontinuity_count += 1
                 i += 1
-                media_sequence += 1
 
         # We only download the first fragment during the test
         if self.params.get('test', False):
             fragments = [fragments[0] if fragments else None]
 
         if real_downloader:
-            info_copy = info_dict.copy()
+            info_dict['fragments'] = fragments
-            info_copy['fragments'] = fragments
             fd = real_downloader(self.ydl, self.params)
             # TODO: Make progress updates work without hooking twice
             # for ph in self._progress_hooks:
             #     fd.add_progress_hook(ph)
-            success = fd.real_download(filename, info_copy)
+            return fd.real_download(filename, info_dict)
-            if not success:
+
-                return False
+        if is_webvtt:
+            def pack_fragment(frag_content, frag_index):
+                output = io.StringIO()
+                adjust = 0
+                overflow = False
+                mpegts_last = None
+                for block in webvtt.parse_fragment(frag_content):
+                    if isinstance(block, webvtt.CueBlock):
+                        extra_state['webvtt_mpegts_last'] = mpegts_last
+                        if overflow:
+                            extra_state['webvtt_mpegts_adjust'] += 1
+                            overflow = False
+                        block.start += adjust
+                        block.end += adjust
+
+                        dedup_window = extra_state.setdefault('webvtt_dedup_window', [])
+
+                        ready = []
+
+                        i = 0
+                        is_new = True
+                        while i < len(dedup_window):
+                            wcue = dedup_window[i]
+                            wblock = webvtt.CueBlock.from_json(wcue)
+                            i += 1
+                            if wblock.hinges(block):
+                                wcue['end'] = block.end
+                                is_new = False
+                                continue
+                            if wblock == block:
+                                is_new = False
+                                continue
+                            if wblock.end > block.start:
+                                continue
+                            ready.append(wblock)
+                            i -= 1
+                            del dedup_window[i]
+
+                        if is_new:
+                            dedup_window.append(block.as_json)
+                        for block in ready:
+                            block.write_into(output)
+
+                        # we only emit cues once they fall out of the duplicate window
+                        continue
+                    elif isinstance(block, webvtt.Magic):
+                        # take care of MPEG PES timestamp overflow
+                        if block.mpegts is None:
+                            block.mpegts = 0
+                        extra_state.setdefault('webvtt_mpegts_adjust', 0)
+                        block.mpegts += extra_state['webvtt_mpegts_adjust'] << 33
+                        if block.mpegts < extra_state.get('webvtt_mpegts_last', 0):
+                            overflow = True
+                            block.mpegts += 1 << 33
+                        mpegts_last = block.mpegts
+
+                        if frag_index == 1:
+                            extra_state['webvtt_mpegts'] = block.mpegts or 0
+                            extra_state['webvtt_local'] = block.local or 0
+                            # XXX: block.local = block.mpegts = None ?
+                        else:
+                            if block.mpegts is not None and block.local is not None:
+                                adjust = (
+                                    (block.mpegts - extra_state.get('webvtt_mpegts', 0))
+                                    - (block.local - extra_state.get('webvtt_local', 0))
+                                )
+                            continue
+                    elif isinstance(block, webvtt.HeaderBlock):
+                        if frag_index != 1:
+                            # XXX: this should probably be silent as well
+                            # or verify that all segments contain the same data
+                            self.report_warning(bug_reports_message(
+                                'Discarding a %s block found in the middle of the stream; '
+                                'if the subtitles display incorrectly,'
+                                % (type(block).__name__)))
+                            continue
+                    block.write_into(output)
+
+                return output.getvalue().encode('utf-8')
+
+            def fin_fragments():
+                dedup_window = extra_state.get('webvtt_dedup_window')
+                if not dedup_window:
+                    return b''
+
+                output = io.StringIO()
+                for cue in dedup_window:
+                    webvtt.CueBlock.from_json(cue).write_into(output)
+
+                return output.getvalue().encode('utf-8')
+
+            self.download_and_append_fragments(
+                ctx, fragments, info_dict, pack_func=pack_fragment, finish_func=fin_fragments)
         else:
-            if is_webvtt:
+            return self.download_and_append_fragments(ctx, fragments, info_dict)
-                def pack_fragment(frag_content, frag_index):
-                    output = io.StringIO()
-                    adjust = 0
-                    for block in webvtt.parse_fragment(frag_content):
-                        if isinstance(block, webvtt.CueBlock):
-                            block.start += adjust
-                            block.end += adjust
-
-                            dedup_window = extra_state.setdefault('webvtt_dedup_window', [])
-                            cue = block.as_json
-
-                            # skip the cue if an identical one appears
-                            # in the window of potential duplicates
-                            # and prune the window of unviable candidates
-                            i = 0
-                            skip = True
-                            while i < len(dedup_window):
-                                window_cue = dedup_window[i]
-                                if window_cue == cue:
-                                    break
-                                if window_cue['end'] >= cue['start']:
-                                    i += 1
-                                    continue
-                                del dedup_window[i]
-                            else:
-                                skip = False
-
-                            if skip:
-                                continue
-
-                            # add the cue to the window
-                            dedup_window.append(cue)
-                        elif isinstance(block, webvtt.Magic):
-                            # take care of MPEG PES timestamp overflow
-                            if block.mpegts is None:
-                                block.mpegts = 0
-                            extra_state.setdefault('webvtt_mpegts_adjust', 0)
-                            block.mpegts += extra_state['webvtt_mpegts_adjust'] << 33
-                            if block.mpegts < extra_state.get('webvtt_mpegts_last', 0):
-                                extra_state['webvtt_mpegts_adjust'] += 1
-                                block.mpegts += 1 << 33
-                            extra_state['webvtt_mpegts_last'] = block.mpegts
-
-                            if frag_index == 1:
-                                extra_state['webvtt_mpegts'] = block.mpegts or 0
-                                extra_state['webvtt_local'] = block.local or 0
-                                # XXX: block.local = block.mpegts = None ?
-                            else:
-                                if block.mpegts is not None and block.local is not None:
-                                    adjust = (
-                                        (block.mpegts - extra_state.get('webvtt_mpegts', 0))
-                                        - (block.local - extra_state.get('webvtt_local', 0))
-                                    )
-                                continue
-                        elif isinstance(block, webvtt.HeaderBlock):
-                            if frag_index != 1:
-                                # XXX: this should probably be silent as well
-                                # or verify that all segments contain the same data
-                                self.report_warning(bug_reports_message(
-                                    'Discarding a %s block found in the middle of the stream; '
-                                    'if the subtitles display incorrectly,'
-                                    % (type(block).__name__)))
-                                continue
-                        block.write_into(output)
-
-                    return output.getvalue().encode('utf-8')
-            else:
-                pack_fragment = None
-            self.download_and_append_fragments(ctx, fragments, info_dict, pack_fragment)
-        return True

yt_dlp/downloader/http.py

@@ -48,8 +48,9 @@ class HttpFD(FileDownloader):
 
         is_test = self.params.get('test', False)
         chunk_size = self._TEST_FILE_SIZE if is_test else (
-            info_dict.get('downloader_options', {}).get('http_chunk_size')
+            self.params.get('http_chunk_size')
-            or self.params.get('http_chunk_size') or 0)
+            or info_dict.get('downloader_options', {}).get('http_chunk_size')
+            or 0)
 
         ctx.open_mode = 'wb'
         ctx.resume_len = 0
@@ -57,6 +58,7 @@ class HttpFD(FileDownloader):
         ctx.block_size = self.params.get('buffersize', 1024)
         ctx.start_time = time.time()
         ctx.chunk_size = None
+        throttle_start = None
 
         if self.params.get('continuedl', True):
             # Establish possible resume length
@@ -177,7 +179,7 @@ class HttpFD(FileDownloader):
                                 'status': 'finished',
                                 'downloaded_bytes': ctx.resume_len,
                                 'total_bytes': ctx.resume_len,
-                            })
+                            }, info_dict)
                             raise SucceedDownload()
                         else:
                             # The length does not match, we start the download over
@@ -189,13 +191,16 @@ class HttpFD(FileDownloader):
                     # Unexpected HTTP error
                     raise
                 raise RetryDownload(err)
-            except socket.error as err:
+            except socket.timeout as err:
-                if err.errno != errno.ECONNRESET:
-                    # Connection reset is no problem, just retry
-                    raise
                 raise RetryDownload(err)
+            except socket.error as err:
+                if err.errno in (errno.ECONNRESET, errno.ETIMEDOUT):
+                    # Connection reset is no problem, just retry
+                    raise RetryDownload(err)
+                raise
 
         def download():
+            nonlocal throttle_start
             data_len = ctx.data.info().get('Content-length', None)
 
             # Range HTTP header may be ignored/unsupported by a webserver
@@ -224,7 +229,6 @@ class HttpFD(FileDownloader):
             # measure time over whole while-loop, so slow_down() and best_block_size() work together properly
             now = None  # needed for slow_down() in the first loop run
             before = start  # start measuring
-            throttle_start = None
 
             def retry(e):
                 to_stdout = ctx.tmpfilename == '-'
@@ -238,7 +242,7 @@ class HttpFD(FileDownloader):
             while True:
                 try:
                     # Download and write
-                    data_block = ctx.data.read(block_size if data_len is None else min(block_size, data_len - byte_counter))
+                    data_block = ctx.data.read(block_size if not is_test else min(block_size, data_len - byte_counter))
                 # socket.timeout is a subclass of socket.error but may not have
                 # errno set
                 except socket.timeout as e:
@@ -310,7 +314,8 @@ class HttpFD(FileDownloader):
                     'eta': eta,
                     'speed': speed,
                     'elapsed': now - ctx.start_time,
-                })
+                    'ctx_id': info_dict.get('ctx_id'),
+                }, info_dict)
 
                 if data_len is not None and byte_counter == data_len:
                     break
@@ -324,7 +329,7 @@ class HttpFD(FileDownloader):
                         if ctx.stream is not None and ctx.tmpfilename != '-':
                             ctx.stream.close()
                         raise ThrottledDownload()
-                else:
+                elif speed:
                     throttle_start = None
 
             if not is_test and ctx.chunk_size and ctx.data_len is not None and byte_counter < ctx.data_len:
@@ -357,7 +362,8 @@ class HttpFD(FileDownloader):
                 'filename': ctx.filename,
                 'status': 'finished',
                 'elapsed': time.time() - ctx.start_time,
-            })
+                'ctx_id': info_dict.get('ctx_id'),
+            }, info_dict)
 
             return True
 
@@ -369,6 +375,8 @@ class HttpFD(FileDownloader):
                 count += 1
                 if count <= retries:
                     self.report_retry(e.source_error, count, retries)
+                else:
+                    self.to_screen(f'[download] Got server HTTP error: {e.source_error}')
                 continue
             except NextFragment:
                 continue

yt_dlp/downloader/ism.py

@@ -246,7 +246,7 @@ class IsmFD(FragmentFD):
             'total_frags': len(segments),
         }
 
-        self._prepare_and_start_frag_download(ctx)
+        self._prepare_and_start_frag_download(ctx, info_dict)
 
         extra_state = ctx.setdefault('extra_state', {
             'ism_track_written': False,
@@ -284,6 +284,6 @@ class IsmFD(FragmentFD):
                 self.report_error('giving up after %s fragment retries' % fragment_retries)
                 return False
 
-        self._finish_frag_download(ctx)
+        self._finish_frag_download(ctx, info_dict)
 
         return True

yt_dlp/downloader/mhtml.py

@@ -122,7 +122,7 @@ body > figure > img {
             'total_frags': len(fragments),
         }
 
-        self._prepare_and_start_frag_download(ctx)
+        self._prepare_and_start_frag_download(ctx, info_dict)
 
         extra_state = ctx.setdefault('extra_state', {
             'header_written': False,
@@ -198,5 +198,5 @@ body > figure > img {
 
         ctx['dest_stream'].write(
             b'--%b--\r\n\r\n' % frag_boundary.encode('us-ascii'))
-        self._finish_frag_download(ctx)
+        self._finish_frag_download(ctx, info_dict)
         return True

yt_dlp/downloader/niconico.py

@@ -4,9 +4,9 @@ from __future__ import unicode_literals
 import threading
 
 from .common import FileDownloader
-from ..downloader import _get_real_downloader
+from ..downloader import get_suitable_downloader
 from ..extractor.niconico import NiconicoIE
-from ..compat import compat_urllib_request
+from ..utils import sanitized_Request
 
 
 class NiconicoDmcFD(FileDownloader):
@@ -20,7 +20,7 @@ class NiconicoDmcFD(FileDownloader):
         ie = NiconicoIE(self.ydl)
         info_dict, heartbeat_info_dict = ie._get_heartbeat_info(info_dict)
 
-        fd = _get_real_downloader(info_dict, params=self.params)(self.ydl, self.params)
+        fd = get_suitable_downloader(info_dict, params=self.params)(self.ydl, self.params)
 
         success = download_complete = False
         timer = [None]
@@ -29,9 +29,11 @@ class NiconicoDmcFD(FileDownloader):
         heartbeat_data = heartbeat_info_dict['data'].encode()
         heartbeat_interval = heartbeat_info_dict.get('interval', 30)
 
+        request = sanitized_Request(heartbeat_url, heartbeat_data)
+
         def heartbeat():
             try:
-                compat_urllib_request.urlopen(url=heartbeat_url, data=heartbeat_data)
+                self.ydl.urlopen(request).read()
             except Exception:
                 self.to_screen('[%s] Heartbeat failed' % self.FD_NAME)
 

yt_dlp/downloader/rtmp.py

@@ -12,6 +12,7 @@ from ..utils import (
     encodeFilename,
     encodeArgument,
     get_exe_version,
+    Popen,
 )
 
 
@@ -26,7 +27,7 @@ class RtmpFD(FileDownloader):
             start = time.time()
             resume_percent = None
             resume_downloaded_data_len = None
-            proc = subprocess.Popen(args, stderr=subprocess.PIPE)
+            proc = Popen(args, stderr=subprocess.PIPE)
             cursor_in_new_line = True
             proc_stderr_closed = False
             try:
@@ -66,7 +67,7 @@ class RtmpFD(FileDownloader):
                             'eta': eta,
                             'elapsed': time_now - start,
                             'speed': speed,
-                        })
+                        }, info_dict)
                         cursor_in_new_line = False
                     else:
                         # no percent for live streams
@@ -82,7 +83,7 @@ class RtmpFD(FileDownloader):
                                 'status': 'downloading',
                                 'elapsed': time_now - start,
                                 'speed': speed,
-                            })
+                            }, info_dict)
                             cursor_in_new_line = False
                         elif self.params.get('verbose', False):
                             if not cursor_in_new_line:
@@ -208,7 +209,7 @@ class RtmpFD(FileDownloader):
                 'filename': filename,
                 'status': 'finished',
                 'elapsed': time.time() - started,
-            })
+            }, info_dict)
             return True
         else:
             self.to_stderr('\n')

yt_dlp/downloader/rtsp.py

@@ -39,7 +39,7 @@ class RtspFD(FileDownloader):
                 'total_bytes': fsize,
                 'filename': filename,
                 'status': 'finished',
-            })
+            }, info_dict)
             return True
         else:
             self.to_stderr('\n')

yt_dlp/downloader/youtube_live_chat.py

@@ -44,7 +44,7 @@ class YoutubeLiveChatFD(FragmentFD):
             return self._download_fragment(ctx, url, info_dict, http_headers, data)
 
         def parse_actions_replay(live_chat_continuation):
-            offset = continuation_id = None
+            offset = continuation_id = click_tracking_params = None
             processed_fragment = bytearray()
             for action in live_chat_continuation.get('actions', []):
                 if 'replayChatItemAction' in action:
@@ -53,17 +53,34 @@ class YoutubeLiveChatFD(FragmentFD):
                 processed_fragment.extend(
                     json.dumps(action, ensure_ascii=False).encode('utf-8') + b'\n')
             if offset is not None:
-                continuation_id = try_get(
+                continuation = try_get(
                     live_chat_continuation,
-                    lambda x: x['continuations'][0]['liveChatReplayContinuationData']['continuation'])
+                    lambda x: x['continuations'][0]['liveChatReplayContinuationData'], dict)
+                if continuation:
+                    continuation_id = continuation.get('continuation')
+                    click_tracking_params = continuation.get('clickTrackingParams')
             self._append_fragment(ctx, processed_fragment)
-            return continuation_id, offset
+            return continuation_id, offset, click_tracking_params
+
+        def try_refresh_replay_beginning(live_chat_continuation):
+            # choose the second option that contains the unfiltered live chat replay
+            refresh_continuation = try_get(
+                live_chat_continuation,
+                lambda x: x['header']['liveChatHeaderRenderer']['viewSelector']['sortFilterSubMenuRenderer']['subMenuItems'][1]['continuation']['reloadContinuationData'], dict)
+            if refresh_continuation:
+                # no data yet but required to call _append_fragment
+                self._append_fragment(ctx, b'')
+                refresh_continuation_id = refresh_continuation.get('continuation')
+                offset = 0
+                click_tracking_params = refresh_continuation.get('trackingParams')
+                return refresh_continuation_id, offset, click_tracking_params
+            return parse_actions_replay(live_chat_continuation)
 
         live_offset = 0
 
         def parse_actions_live(live_chat_continuation):
             nonlocal live_offset
-            continuation_id = None
+            continuation_id = click_tracking_params = None
             processed_fragment = bytearray()
             for action in live_chat_continuation.get('actions', []):
                 timestamp = self.parse_live_timestamp(action)
@@ -84,45 +101,52 @@ class YoutubeLiveChatFD(FragmentFD):
             continuation_data = try_get(live_chat_continuation, continuation_data_getters, dict)
             if continuation_data:
                 continuation_id = continuation_data.get('continuation')
+                click_tracking_params = continuation_data.get('clickTrackingParams')
                 timeout_ms = int_or_none(continuation_data.get('timeoutMs'))
                 if timeout_ms is not None:
                     time.sleep(timeout_ms / 1000)
             self._append_fragment(ctx, processed_fragment)
-            return continuation_id, live_offset
+            return continuation_id, live_offset, click_tracking_params
 
-        if info_dict['protocol'] == 'youtube_live_chat_replay':
+        def download_and_parse_fragment(url, frag_index, request_data=None, headers=None):
-            parse_actions = parse_actions_replay
-        elif info_dict['protocol'] == 'youtube_live_chat':
-            parse_actions = parse_actions_live
-
-        def download_and_parse_fragment(url, frag_index, request_data, headers):
             count = 0
             while count <= fragment_retries:
                 try:
                     success, raw_fragment = dl_fragment(url, request_data, headers)
                     if not success:
-                        return False, None, None
+                        return False, None, None, None
-                    data = json.loads(raw_fragment)
+                    try:
+                        data = ie.extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
+                    except RegexNotFoundError:
+                        data = None
+                    if not data:
+                        data = json.loads(raw_fragment)
                     live_chat_continuation = try_get(
                         data,
                         lambda x: x['continuationContents']['liveChatContinuation'], dict) or {}
-                    continuation_id, offset = parse_actions(live_chat_continuation)
+                    if info_dict['protocol'] == 'youtube_live_chat_replay':
-                    return True, continuation_id, offset
+                        if frag_index == 1:
+                            continuation_id, offset, click_tracking_params = try_refresh_replay_beginning(live_chat_continuation)
+                        else:
+                            continuation_id, offset, click_tracking_params = parse_actions_replay(live_chat_continuation)
+                    elif info_dict['protocol'] == 'youtube_live_chat':
+                        continuation_id, offset, click_tracking_params = parse_actions_live(live_chat_continuation)
+                    return True, continuation_id, offset, click_tracking_params
                 except compat_urllib_error.HTTPError as err:
                     count += 1
                     if count <= fragment_retries:
                         self.report_retry_fragment(err, frag_index, count, fragment_retries)
             if count > fragment_retries:
                 self.report_error('giving up after %s fragment retries' % fragment_retries)
-                return False, None, None
+                return False, None, None, None
 
-        self._prepare_and_start_frag_download(ctx)
+        self._prepare_and_start_frag_download(ctx, info_dict)
 
         success, raw_fragment = dl_fragment(info_dict['url'])
         if not success:
             return False
         try:
-            data = ie._extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
+            data = ie.extract_yt_initial_data(video_id, raw_fragment.decode('utf-8', 'replace'))
         except RegexNotFoundError:
             return False
         continuation_id = try_get(
@@ -131,7 +155,7 @@ class YoutubeLiveChatFD(FragmentFD):
         # no data yet but required to call _append_fragment
         self._append_fragment(ctx, b'')
 
-        ytcfg = ie._extract_ytcfg(video_id, raw_fragment.decode('utf-8', 'replace'))
+        ytcfg = ie.extract_ytcfg(video_id, raw_fragment.decode('utf-8', 'replace'))
 
         if not ytcfg:
             return False
@@ -142,10 +166,13 @@ class YoutubeLiveChatFD(FragmentFD):
         visitor_data = try_get(innertube_context, lambda x: x['client']['visitorData'], str)
         if info_dict['protocol'] == 'youtube_live_chat_replay':
             url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat_replay?key=' + api_key
+            chat_page_url = 'https://www.youtube.com/live_chat_replay?continuation=' + continuation_id
         elif info_dict['protocol'] == 'youtube_live_chat':
             url = 'https://www.youtube.com/youtubei/v1/live_chat/get_live_chat?key=' + api_key
+            chat_page_url = 'https://www.youtube.com/live_chat?continuation=' + continuation_id
 
         frag_index = offset = 0
+        click_tracking_params = None
         while continuation_id is not None:
             frag_index += 1
             request_data = {
@@ -154,17 +181,22 @@ class YoutubeLiveChatFD(FragmentFD):
             }
             if frag_index > 1:
                 request_data['currentPlayerState'] = {'playerOffsetMs': str(max(offset - 5000, 0))}
-            headers = ie._generate_api_headers(ytcfg, visitor_data=visitor_data)
+                if click_tracking_params:
-            headers.update({'content-type': 'application/json'})
+                    request_data['context']['clickTracking'] = {'clickTrackingParams': click_tracking_params}
-            fragment_request_data = json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n'
+                headers = ie.generate_api_headers(ytcfg=ytcfg, visitor_data=visitor_data)
-            success, continuation_id, offset = download_and_parse_fragment(
+                headers.update({'content-type': 'application/json'})
-                url, frag_index, fragment_request_data, headers)
+                fragment_request_data = json.dumps(request_data, ensure_ascii=False).encode('utf-8') + b'\n'
+                success, continuation_id, offset, click_tracking_params = download_and_parse_fragment(
+                    url, frag_index, fragment_request_data, headers)
+            else:
+                success, continuation_id, offset, click_tracking_params = download_and_parse_fragment(
+                    chat_page_url, frag_index)
             if not success:
                 return False
             if test:
                 break
 
-        self._finish_frag_download(ctx)
+        self._finish_frag_download(ctx, info_dict)
         return True
 
     @staticmethod

yt_dlp/extractor/__init__.py

@@ -1,14 +1,15 @@
-from __future__ import unicode_literals
+import os
 
 from ..utils import load_plugins
 
-try:
+_LAZY_LOADER = False
-    from .lazy_extractors import *
+if not os.environ.get('YTDLP_NO_LAZY_EXTRACTORS'):
-    from .lazy_extractors import _ALL_CLASSES
+    try:
-    _LAZY_LOADER = True
+        from .lazy_extractors import *
-    _PLUGIN_CLASSES = []
+        from .lazy_extractors import _ALL_CLASSES
-except ImportError:
+        _LAZY_LOADER = True
-    _LAZY_LOADER = False
+    except ImportError:
+        pass
 
 if not _LAZY_LOADER:
     from .extractors import *
@@ -19,8 +20,8 @@ if not _LAZY_LOADER:
     ]
     _ALL_CLASSES.append(GenericIE)
 
-    _PLUGIN_CLASSES = load_plugins('extractor', 'IE', globals())
+_PLUGIN_CLASSES = load_plugins('extractor', 'IE', globals())
-    _ALL_CLASSES = _PLUGIN_CLASSES + _ALL_CLASSES
+_ALL_CLASSES = list(_PLUGIN_CLASSES.values()) + _ALL_CLASSES
 
 
 def gen_extractor_classes():

yt_dlp/extractor/abcnews.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .amp import AMPIE
 from .common import InfoExtractor
@@ -59,7 +58,7 @@ class AbcNewsVideoIE(AMPIE):
     }]
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        mobj = self._match_valid_url(url)
         display_id = mobj.group('display_id')
         video_id = mobj.group('id')
         info_dict = self._extract_feed_info(

yt_dlp/extractor/abcotvs.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..compat import compat_str
@@ -55,7 +54,7 @@ class ABCOTVSIE(InfoExtractor):
     }
 
     def _real_extract(self, url):
-        site, display_id, video_id = re.match(self._VALID_URL, url).groups()
+        site, display_id, video_id = self._match_valid_url(url).groups()
         display_id = display_id or video_id
         station = self._SITE_MAP[site]
 

yt_dlp/extractor/acast.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
 
 from .common import InfoExtractor
 from ..utils import (
@@ -80,7 +79,7 @@ class ACastIE(ACastBaseIE):
     }]
 
     def _real_extract(self, url):
-        channel, display_id = re.match(self._VALID_URL, url).groups()
+        channel, display_id = self._match_valid_url(url).groups()
         episode = self._call_api(
             '%s/episodes/%s' % (channel, display_id),
             display_id, {'showInfo': 'true'})

yt_dlp/extractor/adn.py

@@ -15,6 +15,7 @@ from ..compat import (
     compat_ord,
 )
 from ..utils import (
+    ass_subtitles_timecode,
     bytes_to_intlist,
     bytes_to_long,
     ExtractorError,
@@ -68,10 +69,6 @@ class ADNIE(InfoExtractor):
         'end': 4,
     }
 
-    @staticmethod
-    def _ass_subtitles_timecode(seconds):
-        return '%01d:%02d:%02d.%02d' % (seconds / 3600, (seconds % 3600) / 60, seconds % 60, (seconds % 1) * 100)
-
     def _get_subtitles(self, sub_url, video_id):
         if not sub_url:
             return None
@@ -117,8 +114,8 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
                     continue
                 alignment = self._POS_ALIGN_MAP.get(position_align, 2) + self._LINE_ALIGN_MAP.get(line_align, 0)
                 ssa += os.linesep + 'Dialogue: Marked=0,%s,%s,Default,,0,0,0,,%s%s' % (
-                    self._ass_subtitles_timecode(start),
+                    ass_subtitles_timecode(start),
-                    self._ass_subtitles_timecode(end),
+                    ass_subtitles_timecode(end),
                     '{\\a%d}' % alignment if alignment != 2 else '',
                     text.replace('\n', '\\N').replace('<i>', '{\\i1}').replace('</i>', '{\\i0}'))
 

yt_dlp/extractor/adobepass.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import json
 import re
 import time
 import xml.etree.ElementTree as etree
@@ -36,6 +37,11 @@ MSO_INFO = {
         'username_field': 'email',
         'password_field': 'loginpassword',
     },
+    'RCN': {
+        'name': 'RCN',
+        'username_field': 'username',
+        'password_field': 'password',
+    },
     'Rogers': {
         'name': 'Rogers',
         'username_field': 'UserName',
@@ -61,6 +67,11 @@ MSO_INFO = {
         'username_field': 'IDToken1',
         'password_field': 'IDToken2',
     },
+    'Spectrum': {
+        'name': 'Spectrum',
+        'username_field': 'IDToken1',
+        'password_field': 'IDToken2',
+    },
     'Philo': {
         'name': 'Philo',
         'username_field': 'ident'
@@ -70,6 +81,11 @@ MSO_INFO = {
         'username_field': 'IDToken1',
         'password_field': 'IDToken2',
     },
+    'Cablevision': {
+        'name': 'Optimum/Cablevision',
+        'username_field': 'j_username',
+        'password_field': 'j_password',
+    },
     'thr030': {
         'name': '3 Rivers Communications'
     },
@@ -1324,6 +1340,11 @@ MSO_INFO = {
     'cou060': {
         'name': 'Zito Media'
     },
+    'slingtv': {
+        'name': 'Sling TV',
+        'username_field': 'username',
+        'password_field': 'password',
+    },
 }
 
 
@@ -1492,7 +1513,8 @@ class AdobePassIE(InfoExtractor):
                     # In general, if you're connecting from a Verizon-assigned IP,
                     # you will not actually pass your credentials.
                     provider_redirect_page, urlh = provider_redirect_page_res
-                    if 'Please wait ...' in provider_redirect_page:
+                    # From non-Verizon IP, still gave 'Please wait', but noticed N==Y; will need to try on Verizon IP
+                    if 'Please wait ...' in provider_redirect_page and '\'N\'== "Y"' not in provider_redirect_page:
                         saml_redirect_url = self._html_search_regex(
                             r'self\.parent\.location=(["\'])(?P<url>.+?)\1',
                             provider_redirect_page,
@@ -1500,7 +1522,8 @@ class AdobePassIE(InfoExtractor):
                         saml_login_page = self._download_webpage(
                             saml_redirect_url, video_id,
                             'Downloading SAML Login Page')
-                    else:
+                    elif 'Verizon FiOS - sign in' in provider_redirect_page:
+                        # FXNetworks from non-Verizon IP
                         saml_login_page_res = post_form(
                             provider_redirect_page_res, 'Logging in', {
                                 mso_info['username_field']: username,
@@ -1510,6 +1533,26 @@ class AdobePassIE(InfoExtractor):
                         if 'Please try again.' in saml_login_page:
                             raise ExtractorError(
                                 'We\'re sorry, but either the User ID or Password entered is not correct.')
+                    else:
+                        # ABC from non-Verizon IP
+                        saml_redirect_url = self._html_search_regex(
+                            r'var\surl\s*=\s*(["\'])(?P<url>.+?)\1',
+                            provider_redirect_page,
+                            'SAML Redirect URL', group='url')
+                        saml_redirect_url = saml_redirect_url.replace(r'\/', '/')
+                        saml_redirect_url = saml_redirect_url.replace(r'\-', '-')
+                        saml_redirect_url = saml_redirect_url.replace(r'\x26', '&')
+                        saml_login_page = self._download_webpage(
+                            saml_redirect_url, video_id,
+                            'Downloading SAML Login Page')
+                        saml_login_page, urlh = post_form(
+                            [saml_login_page, saml_redirect_url], 'Logging in', {
+                                mso_info['username_field']: username,
+                                mso_info['password_field']: password,
+                            })
+                        if 'Please try again.' in saml_login_page:
+                            raise ExtractorError(
+                                'Failed to login, incorrect User ID or Password.')
                     saml_login_url = self._search_regex(
                         r'xmlHttp\.open\("POST"\s*,\s*(["\'])(?P<url>.+?)\1',
                         saml_login_page, 'SAML Login URL', group='url')
@@ -1524,6 +1567,75 @@ class AdobePassIE(InfoExtractor):
                         }), headers={
                             'Content-Type': 'application/x-www-form-urlencoded'
                         })
+                elif mso_id == 'Spectrum':
+                    # Spectrum's login for is dynamically loaded via JS so we need to hardcode the flow
+                    # as a one-off implementation.
+                    provider_redirect_page, urlh = provider_redirect_page_res
+                    provider_login_page_res = post_form(
+                        provider_redirect_page_res, self._DOWNLOADING_LOGIN_PAGE)
+                    saml_login_page, urlh = provider_login_page_res
+                    relay_state = self._search_regex(
+                        r'RelayState\s*=\s*"(?P<relay>.+?)";',
+                        saml_login_page, 'RelayState', group='relay')
+                    saml_request = self._search_regex(
+                        r'SAMLRequest\s*=\s*"(?P<saml_request>.+?)";',
+                        saml_login_page, 'SAMLRequest', group='saml_request')
+                    login_json = {
+                        mso_info['username_field']: username,
+                        mso_info['password_field']: password,
+                        'RelayState': relay_state,
+                        'SAMLRequest': saml_request,
+                    }
+                    saml_response_json = self._download_json(
+                        'https://tveauthn.spectrum.net/tveauthentication/api/v1/manualAuth', video_id,
+                        'Downloading SAML Response',
+                        data=json.dumps(login_json).encode(),
+                        headers={
+                            'Content-Type': 'application/json',
+                            'Accept': 'application/json',
+                        })
+                    self._download_webpage(
+                        saml_response_json['SAMLRedirectUri'], video_id,
+                        'Confirming Login', data=urlencode_postdata({
+                            'SAMLResponse': saml_response_json['SAMLResponse'],
+                            'RelayState': relay_state,
+                        }), headers={
+                            'Content-Type': 'application/x-www-form-urlencoded'
+                        })
+                elif mso_id == 'slingtv':
+                    # SlingTV has a meta-refresh based authentication, but also
+                    # looks at the tab history to count the number of times the
+                    # browser has been on a page
+
+                    first_bookend_page, urlh = provider_redirect_page_res
+
+                    hidden_data = self._hidden_inputs(first_bookend_page)
+                    hidden_data['history'] = 1
+
+                    provider_login_page_res = self._download_webpage_handle(
+                        urlh.geturl(), video_id, 'Sending first bookend',
+                        query=hidden_data)
+
+                    provider_association_redirect, urlh = post_form(
+                        provider_login_page_res, 'Logging in', {
+                            mso_info['username_field']: username,
+                            mso_info['password_field']: password
+                        })
+
+                    provider_refresh_redirect_url = extract_redirect_url(
+                        provider_association_redirect, url=urlh.geturl())
+
+                    last_bookend_page, urlh = self._download_webpage_handle(
+                        provider_refresh_redirect_url, video_id,
+                        'Downloading Auth Association Redirect Page')
+                    hidden_data = self._hidden_inputs(last_bookend_page)
+                    hidden_data['history'] = 3
+
+                    mvpd_confirm_page_res = self._download_webpage_handle(
+                        urlh.geturl(), video_id, 'Sending final bookend',
+                        query=hidden_data)
+
+                    post_form(mvpd_confirm_page_res, 'Confirming Login')
                 else:
                     # Some providers (e.g. DIRECTV NOW) have another meta refresh
                     # based redirect that should be followed.
@@ -1536,10 +1648,13 @@ class AdobePassIE(InfoExtractor):
                             'Downloading Provider Redirect Page (meta refresh)')
                     provider_login_page_res = post_form(
                         provider_redirect_page_res, self._DOWNLOADING_LOGIN_PAGE)
-                    mvpd_confirm_page_res = post_form(provider_login_page_res, 'Logging in', {
+                    form_data = {
                         mso_info.get('username_field', 'username'): username,
-                        mso_info.get('password_field', 'password'): password,
+                        mso_info.get('password_field', 'password'): password
-                    })
+                    }
+                    if mso_id == 'Cablevision':
+                        form_data['_eventId_proceed'] = ''
+                    mvpd_confirm_page_res = post_form(provider_login_page_res, 'Logging in', form_data)
                     if mso_id != 'Rogers':
                         post_form(mvpd_confirm_page_res, 'Confirming Login')
 

yt_dlp/extractor/adobetv.py

@@ -9,6 +9,7 @@ from ..utils import (
     float_or_none,
     int_or_none,
     ISO639Utils,
+    join_nonempty,
     OnDemandPagedList,
     parse_duration,
     str_or_none,
@@ -132,7 +133,7 @@ class AdobeTVIE(AdobeTVBaseIE):
     }
 
     def _real_extract(self, url):
-        language, show_urlname, urlname = re.match(self._VALID_URL, url).groups()
+        language, show_urlname, urlname = self._match_valid_url(url).groups()
         if not language:
             language = 'en'
 
@@ -178,7 +179,7 @@ class AdobeTVShowIE(AdobeTVPlaylistBaseIE):
     _process_data = AdobeTVBaseIE._parse_video_data
 
     def _real_extract(self, url):
-        language, show_urlname = re.match(self._VALID_URL, url).groups()
+        language, show_urlname = self._match_valid_url(url).groups()
         if not language:
             language = 'en'
         query = {
@@ -215,7 +216,7 @@ class AdobeTVChannelIE(AdobeTVPlaylistBaseIE):
             show_data['url'], 'AdobeTVShow', str_or_none(show_data.get('id')))
 
     def _real_extract(self, url):
-        language, channel_urlname, category_urlname = re.match(self._VALID_URL, url).groups()
+        language, channel_urlname, category_urlname = self._match_valid_url(url).groups()
         if not language:
             language = 'en'
         query = {
@@ -263,7 +264,7 @@ class AdobeTVVideoIE(AdobeTVBaseIE):
                 continue
             formats.append({
                 'filesize': int_or_none(source.get('kilobytes') or None, invscale=1000),
-                'format_id': '-'.join(filter(None, [source.get('format'), source.get('label')])),
+                'format_id': join_nonempty(source.get('format'), source.get('label')),
                 'height': int_or_none(source.get('height') or None),
                 'tbr': int_or_none(source.get('bitrate') or None),
                 'width': int_or_none(source.get('width') or None),