[version] update

:ci skip all

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -1,73 +0,0 @@
----
-name: Broken site support
-about: Report broken or misfunctioning site
-title: "[Broken] Website Name: A short description of the issue"
-labels: ['triage', 'extractor-bug']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.10.09. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped.
-- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, confirm all RELEVANT tasks from the following by putting x into all the boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.10.09**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar issues including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.10.09
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-
-```
-<!--
-Do not remove the above ```
--->
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/1_broken_site.yml

@@ -0,0 +1,63 @@
+name: Broken site support
+description: Report broken or misfunctioning site
+labels: [triage, extractor-bug]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a broken site
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.10.22**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your issue in an arbitrary form.
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version 2021.10.22 (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (2021.10.22)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -1,60 +0,0 @@
----
-name: Site support request
-about: Request support for a new site
-title: "[Site Request] Website Name"
-labels: ['triage', 'site-request']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.10.09. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, confirm all RELEVANT tasks from the following by putting x into all the boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.10.09**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that none of provided URLs violate any copyrights
-- [ ] The provided URLs do not contain any DRM to the best of my knowledge
-- [ ] I've searched the bugtracker for similar site support requests including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Example URLs
-
-<!--
-Provide all kinds of example URLs support for which should be included. Replace following example URLs by yours.
--->
-
-- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
-- Single video: https://youtu.be/BaW_jenozKc
-- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
-
-
-## Description
-
-<!--
-Provide any additional information.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/2_site_support_request.yml

@@ -0,0 +1,74 @@
+name: Site support request
+description: Request support for a new site
+labels: [triage, site-request]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a new site support request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.10.22**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and am willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: example-urls
+    attributes:
+      label: Example URLs
+      description: |
+        Provide all kinds of example URLs for which support should be added
+      value: |
+        - Single video: https://www.youtube.com/watch?v=BaW_jenozKc
+        - Single video: https://youtu.be/BaW_jenozKc
+        - Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
+    validations:
+      required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide any additional information
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output using one of the example URLs provided above.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version 2021.10.22 (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (2021.10.22)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -1,43 +0,0 @@
----
-name: Site feature request
-about: Request a new functionality for a site
-title: "[Site Feature] Website Name: A short description of the feature"
-labels: ['triage', 'site-enhancement']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.10.09. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, confirm all RELEVANT tasks from the following by putting x into all the boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.10.09**
-- [ ] I've searched the bugtracker for similar site feature requests including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Description
-
-<!--
-Provide an explanation of your site feature request in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/3_site_feature_request.yml

@@ -0,0 +1,49 @@
+name: Site feature request
+description: Request a new functionality for a site
+labels: [triage, site-enhancement]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a site feature request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.10.22**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: example-urls
+    attributes:
+      label: Example URLs
+      description: |
+        Example URLs that can be used to demonstrate the requested feature
+      value: |
+        https://www.youtube.com/watch?v=BaW_jenozKc
+    validations:
+      required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your site feature request in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -1,74 +0,0 @@
----
-name: Bug report
-about: Report a bug unrelated to any particular site or extractor
-title: '[Bug] A short description of the issue'
-labels: ['triage', 'bug']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.10.09. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped.
-- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, confirm all RELEVANT tasks from the following by putting x into all the boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a bug unrelated to a specific site
-- [ ] I've verified that I'm running yt-dlp version **2021.10.09**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] The provided URLs do not contain any DRM to the best of my knowledge
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar bug reports including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.10.09
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-
-```
-<!--
-Do not remove the above ```
--->
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/4_bug_report.yml

@@ -0,0 +1,57 @@
+name: Bug report
+description: Report a bug unrelated to any particular site or extractor
+labels: [triage,bug]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a bug unrelated to a specific site
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.10.22**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your issue in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version 2021.10.22 (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (2021.10.22)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -1,43 +0,0 @@
----
-name: Feature request
-about: Request a new functionality unrelated to any particular site or extractor
-title: "[Feature Request] A short description of your feature"
-labels: ['triage', 'enhancement']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.10.09. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.10.09**
-- [ ] I've searched the bugtracker for similar feature requests including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/5_feature_request.yml

@@ -0,0 +1,30 @@
+name: Feature request request
+description: Request a new functionality unrelated to any particular site or extractor
+labels: [triage, enhancement]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a feature request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **2021.10.22**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your site feature request in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/6_question.md

@@ -1,43 +0,0 @@
----
-name: Ask question
-about: Ask yt-dlp related question
-title: "[Question] A short description of your question"
-labels: question
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- Look through the README (https://github.com/yt-dlp/yt-dlp)
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Search the bugtracker for similar questions: https://github.com/yt-dlp/yt-dlp/issues
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm asking a question
-- [ ] I've looked through the README
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I've searched the bugtracker for similar questions including closed ones
-- [ ] I have given an appropriate title to the issue
-
-
-## Question
-
-<!--
-Ask your question in an arbitrary form. Please make sure it's worded well enough to be understood, see https://github.com/yt-dlp/yt-dlp.
--->
-
-WRITE QUESTION HERE

.github/ISSUE_TEMPLATE/6_question.yml

@@ -0,0 +1,30 @@
+name: Ask question
+description: Ask yt-dlp related question
+labels: [question]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm asking a question and not reporting a bug/feature request
+          required: true
+        - label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions including closed ones
+          required: true
+  - type: textarea
+    id: question
+    attributes:
+      label: Question
+      description: |
+        Ask your question in an arbitrary form.
+        Please make sure it's worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information and as much context and examples as possible
+      placeholder: WRITE QUESTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,5 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Get help from the community on Discord
+    url: https://discord.gg/H5MNcFW63r
+    about: Join the yt-dlp Discord for community-powered support!

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.md

@@ -1,73 +0,0 @@
----
-name: Broken site support
-about: Report broken or misfunctioning site
-title: "[Broken] Website Name: A short description of the issue"
-labels: ['triage', 'extractor-bug']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped.
-- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, confirm all RELEVANT tasks from the following by putting x into all the boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar issues including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version %(version)s
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-
-```
-<!--
-Do not remove the above ```
--->
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml

@@ -0,0 +1,63 @@
+name: Broken site support
+description: Report broken or misfunctioning site
+labels: [triage, extractor-bug]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a broken site
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your issue in an arbitrary form.
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version %(version)s (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (%(version)s)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md

@@ -1,60 +0,0 @@
----
-name: Site support request
-about: Request support for a new site
-title: "[Site Request] Website Name"
-labels: ['triage', 'site-request']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, confirm all RELEVANT tasks from the following by putting x into all the boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that none of provided URLs violate any copyrights
-- [ ] The provided URLs do not contain any DRM to the best of my knowledge
-- [ ] I've searched the bugtracker for similar site support requests including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Example URLs
-
-<!--
-Provide all kinds of example URLs support for which should be included. Replace following example URLs by yours.
--->
-
-- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
-- Single video: https://youtu.be/BaW_jenozKc
-- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
-
-
-## Description
-
-<!--
-Provide any additional information.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml

@@ -0,0 +1,74 @@
+name: Site support request
+description: Request support for a new site
+labels: [triage, site-request]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a new site support request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that none of provided URLs [violate any copyrights](https://github.com/ytdl-org/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free) or contain any [DRM](https://en.wikipedia.org/wiki/Digital_rights_management) to the best of my knowledge
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and am willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: example-urls
+    attributes:
+      label: Example URLs
+      description: |
+        Provide all kinds of example URLs for which support should be added
+      value: |
+        - Single video: https://www.youtube.com/watch?v=BaW_jenozKc
+        - Single video: https://youtu.be/BaW_jenozKc
+        - Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
+    validations:
+      required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide any additional information
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output using one of the example URLs provided above.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version %(version)s (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (%(version)s)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md

@@ -1,43 +0,0 @@
----
-name: Site feature request
-about: Request a new functionality for a site
-title: "[Site Feature] Website Name: A short description of the feature"
-labels: ['triage', 'site-enhancement']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, confirm all RELEVANT tasks from the following by putting x into all the boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've searched the bugtracker for similar site feature requests including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Description
-
-<!--
-Provide an explanation of your site feature request in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml

@@ -0,0 +1,49 @@
+name: Site feature request
+description: Request a new functionality for a site
+labels: [triage, site-enhancement]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a site feature request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required
+  - type: input
+    id: region
+    attributes:
+      label: Region
+      description: "Enter the region the site is accessible from"
+      placeholder: "India"
+  - type: textarea
+    id: example-urls
+    attributes:
+      label: Example URLs
+      description: |
+        Example URLs that can be used to demonstrate the requested feature
+      value: |
+        https://www.youtube.com/watch?v=BaW_jenozKc
+    validations:
+      required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your site feature request in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md

@@ -1,74 +0,0 @@
----
-name: Bug report
-about: Report a bug unrelated to any particular site or extractor
-title: '[Bug] A short description of the issue'
-labels: ['triage', 'bug']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped.
-- Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, confirm all RELEVANT tasks from the following by putting x into all the boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a bug unrelated to a specific site
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] The provided URLs do not contain any DRM to the best of my knowledge
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar bug reports including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version %(version)s
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-
-```
-<!--
-Do not remove the above ```
--->
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml

@@ -0,0 +1,57 @@
+name: Bug report
+description: Report a bug unrelated to any particular site or extractor
+labels: [triage,bug]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a bug unrelated to a specific site
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've checked that all provided URLs are alive and playable in a browser
+          required: true
+        - label: I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your issue in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true
+  - type: textarea
+    id: log
+    attributes:
+      label: Verbose log
+      description: |
+        Provide the complete verbose output of yt-dlp that clearly demonstrates the problem.
+        Add the `-Uv` flag to your command line you run yt-dlp with (`yt-dlp -Uv <your command line>`), copy the WHOLE output and insert it below.
+        It should look similar to this:
+      placeholder: |
+        [debug] Command-line config: ['-Uv', 'http://www.youtube.com/watch?v=BaW_jenozKc']
+        [debug] Portable config file: yt-dlp.conf
+        [debug] Portable config: ['-i']
+        [debug] Encodings: locale cp1252, fs utf-8, stdout utf-8, stderr utf-8, pref cp1252
+        [debug] yt-dlp version %(version)s (exe)
+        [debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
+        [debug] exe versions: ffmpeg 3.0.1, ffprobe 3.0.1
+        [debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
+        [debug] Proxy map: {}
+        yt-dlp is up to date (%(version)s)
+        <more lines>
+      render: shell
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/5_feature_request.md

@@ -1,43 +0,0 @@
----
-name: Feature request
-about: Request a new functionality unrelated to any particular site or extractor
-title: "[Feature Request] A short description of your feature"
-labels: ['triage', 'enhancement']
-assignees: ''
-
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is %(version)s. If it's not, see https://github.com/yt-dlp/yt-dlp#update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp/issues. DO NOT post duplicates.
-- Read "opening an issue" section in CONTRIBUTING.md: https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue
-- Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
--->
-
-- [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **%(version)s**
-- [ ] I've searched the bugtracker for similar feature requests including closed ones
-- [ ] I've read the opening an issue section in CONTRIBUTING.md
-- [ ] I have given an appropriate title to the issue
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml

@@ -0,0 +1,30 @@
+name: Feature request request
+description: Request a new functionality unrelated to any particular site or extractor
+labels: [triage, enhancement]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm reporting a feature request
+          required: true
+        - label: I've verified that I'm running yt-dlp version **%(version)s**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+  - type: textarea
+    id: description
+    attributes:
+      label: Description
+      description: |
+        Provide an explanation of your site feature request in an arbitrary form.
+        Please make sure the description is worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information, any suggested solutions, and as much context and examples as possible
+      placeholder: WRITE DESCRIPTION HERE
+    validations:
+      required: true

.github/ISSUE_TEMPLATE_tmpl/6_question.yml

@@ -0,0 +1,30 @@
+name: Ask question
+description: Ask yt-dlp related question
+labels: [question]
+body:
+  - type: checkboxes
+    id: checklist
+    attributes:
+      label: Checklist
+      description: |
+        Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
+      options:
+        - label: I'm asking a question and not reporting a bug/feature request
+          required: true
+        - label: I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
+          required: true
+        - label: I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
+          required: true
+        - label: I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions including closed ones
+          required: true
+  - type: textarea
+    id: question
+    attributes:
+      label: Question
+      description: |
+        Ask your question in an arbitrary form.
+        Please make sure it's worded well enough to be understood, see [is-the-description-of-the-issue-itself-sufficient](https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient).
+        Provide any additional information and as much context and examples as possible
+      placeholder: WRITE QUESTION HERE
+    validations:
+      required: true

.github/workflows/build.yml

@@ -8,7 +8,6 @@ on:
 jobs:
   build_unix:
     runs-on: ubuntu-latest
-
     outputs:
       ytdlp_version: ${{ steps.bump_version.outputs.ytdlp_version }}
       upload_url: ${{ steps.create_release.outputs.upload_url }}
@@ -47,10 +46,14 @@ jobs:
     - name: Get Changelog
       id: get_changelog
       run: |
-        changelog=$(cat Changelog.md | grep -oPz '(?s)(?<=### ${{ steps.bump_version.outputs.ytdlp_version }}\n{2}).+?(?=\n{2,3}###)')
+        changelog=$(cat Changelog.md | grep -oPz '(?s)(?<=### ${{ steps.bump_version.outputs.ytdlp_version }}\n{2}).+?(?=\n{2,3}###)') || true
         echo "changelog<<EOF" >> $GITHUB_ENV
         echo "$changelog" >> $GITHUB_ENV
         echo "EOF" >> $GITHUB_ENV
+
+    - name: Build lazy extractors
+      id: lazy_extractors
+      run: python devscripts/make_lazy_extractors.py
     - name: Run Make
       run: make all tar
     - name: Get SHA2-256SUMS for yt-dlp
@@ -65,6 +68,7 @@ jobs:
     - name: Get SHA2-512SUMS for yt-dlp.tar.gz
       id: sha512_tar
       run: echo "::set-output name=sha512_tar::$(sha512sum yt-dlp.tar.gz | awk '{print $1}')"
+
     - name: Install dependencies for pypi
       env:
         PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
@@ -81,6 +85,7 @@ jobs:
         rm -rf dist/*
         python setup.py sdist bdist_wheel
         twine upload dist/*
+
     - name: Install SSH private key
       env:
         BREW_TOKEN: ${{ secrets.BREW_TOKEN }}
@@ -99,6 +104,7 @@ jobs:
         git -C taps/ config user.email github-actions@example.com
         git -C taps/ commit -am 'yt-dlp: ${{ steps.bump_version.outputs.ytdlp_version }}'
         git -C taps/ push
+
     - name: Create Release
       id: create_release
       uses: actions/create-release@v1
@@ -109,8 +115,12 @@ jobs:
         release_name: yt-dlp ${{ steps.bump_version.outputs.ytdlp_version }}
         commitish: ${{ steps.push_update.outputs.head_sha }}
         body: |
-          Changelog:
+          ### Changelog:
           ${{ env.changelog }}
+
+          ---
+
+          ### See [this](https://github.com/yt-dlp/yt-dlp#release-files) for a description of the release files
         draft: false
         prerelease: false
     - name: Upload yt-dlp Unix binary
@@ -133,13 +143,78 @@ jobs:
         asset_name: yt-dlp.tar.gz
         asset_content_type: application/gzip
 
+  build_macos:
+    runs-on: macos-11
+    needs: build_unix
+    outputs:
+      sha256_macos: ${{ steps.sha256_macos.outputs.sha256_macos }}
+      sha512_macos: ${{ steps.sha512_macos.outputs.sha512_macos }}
+      sha256_macos_zip: ${{ steps.sha256_macos_zip.outputs.sha256_macos_zip }}
+      sha512_macos_zip: ${{ steps.sha512_macos_zip.outputs.sha512_macos_zip }}
+
+    steps:
+    - uses: actions/checkout@v2
+    # In order to create a universal2 application, the version of python3 in /usr/bin has to be used
+    - name: Install Requirements
+      run: |
+          brew install coreutils
+          /usr/bin/python3 -m pip install -U --user pip Pyinstaller mutagen pycryptodomex websockets
+    - name: Bump version
+      id: bump_version
+      run: /usr/bin/python3 devscripts/update-version.py
+    - name: Build lazy extractors
+      id: lazy_extractors
+      run: /usr/bin/python3 devscripts/make_lazy_extractors.py
+    - name: Run PyInstaller Script
+      run: /usr/bin/python3 pyinst.py --target-architecture universal2 --onefile
+    - name: Upload yt-dlp MacOS binary
+      id: upload-release-macos
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ needs.build_unix.outputs.upload_url }}
+        asset_path: ./dist/yt-dlp_macos
+        asset_name: yt-dlp_macos
+        asset_content_type: application/octet-stream
+    - name: Get SHA2-256SUMS for yt-dlp_macos
+      id: sha256_macos
+      run: echo "::set-output name=sha256_macos::$(sha256sum dist/yt-dlp_macos | awk '{print $1}')"
+    - name: Get SHA2-512SUMS for yt-dlp_macos
+      id: sha512_macos
+      run: echo "::set-output name=sha512_macos::$(sha512sum dist/yt-dlp_macos | awk '{print $1}')"
+
+    - name: Run PyInstaller Script with --onedir
+      run: /usr/bin/python3 pyinst.py --target-architecture universal2 --onedir
+    - uses: papeloto/action-zip@v1
+      with:
+        files: ./dist/yt-dlp_macos
+        dest: ./dist/yt-dlp_macos.zip
+    - name: Upload yt-dlp MacOS onedir
+      id: upload-release-macos-zip
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ needs.build_unix.outputs.upload_url }}
+        asset_path: ./dist/yt-dlp_macos.zip
+        asset_name: yt-dlp_macos.zip
+        asset_content_type: application/zip
+    - name: Get SHA2-256SUMS for yt-dlp_macos.zip
+      id: sha256_macos_zip
+      run: echo "::set-output name=sha256_macos_zip::$(sha256sum dist/yt-dlp_macos.zip | awk '{print $1}')"
+    - name: Get SHA2-512SUMS for yt-dlp_macos
+      id: sha512_macos_zip
+      run: echo "::set-output name=sha512_macos_zip::$(sha512sum dist/yt-dlp_macos.zip | awk '{print $1}')"
+
   build_windows:
     runs-on: windows-latest
     needs: build_unix
-
     outputs:
       sha256_win: ${{ steps.sha256_win.outputs.sha256_win }}
       sha512_win: ${{ steps.sha512_win.outputs.sha512_win }}
+      sha256_py2exe: ${{ steps.sha256_py2exe.outputs.sha256_py2exe }}
+      sha512_py2exe: ${{ steps.sha512_py2exe.outputs.sha512_py2exe }}
       sha256_win_zip: ${{ steps.sha256_win_zip.outputs.sha256_win_zip }}
       sha512_win_zip: ${{ steps.sha512_win_zip.outputs.sha512_win_zip }}
 
@@ -150,16 +225,17 @@ jobs:
       uses: actions/setup-python@v2
       with:
           python-version: '3.8'
-    - name: Upgrade pip and enable wheel support
-      run: python -m pip install --upgrade pip setuptools wheel
     - name: Install Requirements
       # Custom pyinstaller built with https://github.com/yt-dlp/pyinstaller-builds
-      run: pip install "https://yt-dlp.github.io/Pyinstaller-Builds/x86_64/pyinstaller-4.5.1-py3-none-any.whl" mutagen pycryptodomex websockets
+      run: |
+          python -m pip install --upgrade pip setuptools wheel py2exe
+          pip install "https://yt-dlp.github.io/Pyinstaller-Builds/x86_64/pyinstaller-4.5.1-py3-none-any.whl" mutagen pycryptodomex websockets
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
-    - name: Print version
-      run: echo "${{ steps.bump_version.outputs.ytdlp_version }}"
+    - name: Build lazy extractors
+      id: lazy_extractors
+      run: python devscripts/make_lazy_extractors.py
     - name: Run PyInstaller Script
       run: python pyinst.py
     - name: Upload yt-dlp.exe Windows binary
@@ -178,32 +254,52 @@ jobs:
     - name: Get SHA2-512SUMS for yt-dlp.exe
       id: sha512_win
       run: echo "::set-output name=sha512_win::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA512).Hash.ToLower())"
+
     - name: Run PyInstaller Script with --onedir
       run: python pyinst.py --onedir
     - uses: papeloto/action-zip@v1
       with:
         files: ./dist/yt-dlp
-        dest: ./dist/yt-dlp.zip
-    - name: Upload yt-dlp.zip Windows onedir
+        dest: ./dist/yt-dlp_win.zip
+    - name: Upload yt-dlp Windows onedir
       id: upload-release-windows-zip
       uses: actions/upload-release-asset@v1
       env:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
       with:
         upload_url: ${{ needs.build_unix.outputs.upload_url }}
-        asset_path: ./dist/yt-dlp.zip
-        asset_name: yt-dlp.zip
+        asset_path: ./dist/yt-dlp_win.zip
+        asset_name: yt-dlp_win.zip
         asset_content_type: application/zip
-    - name: Get SHA2-256SUMS for yt-dlp.zip
+    - name: Get SHA2-256SUMS for yt-dlp_win.zip
       id: sha256_win_zip
-      run: echo "::set-output name=sha256_win_zip::$((Get-FileHash dist\yt-dlp.zip -Algorithm SHA256).Hash.ToLower())"
-    - name: Get SHA2-512SUMS for yt-dlp.zip
+      run: echo "::set-output name=sha256_win_zip::$((Get-FileHash dist\yt-dlp_win.zip -Algorithm SHA256).Hash.ToLower())"
+    - name: Get SHA2-512SUMS for yt-dlp_win.zip
       id: sha512_win_zip
-      run: echo "::set-output name=sha512_win_zip::$((Get-FileHash dist\yt-dlp.zip -Algorithm SHA512).Hash.ToLower())"
+      run: echo "::set-output name=sha512_win_zip::$((Get-FileHash dist\yt-dlp_win.zip -Algorithm SHA512).Hash.ToLower())"
+
+    - name: Run py2exe Script
+      run: python setup.py py2exe
+    - name: Upload yt-dlp_min.exe Windows binary
+      id: upload-release-windows-py2exe
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ needs.build_unix.outputs.upload_url }}
+        asset_path: ./dist/yt-dlp.exe
+        asset_name: yt-dlp_min.exe
+        asset_content_type: application/vnd.microsoft.portable-executable
+    - name: Get SHA2-256SUMS for yt-dlp_min.exe
+      id: sha256_py2exe
+      run: echo "::set-output name=sha256_py2exe::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())"
+    - name: Get SHA2-512SUMS for yt-dlp_min.exe
+      id: sha512_py2exe
+      run: echo "::set-output name=sha512_py2exe::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA512).Hash.ToLower())"
 
   build_windows32:
     runs-on: windows-latest
-    needs: [build_unix, build_windows]
+    needs: build_unix
 
     outputs:
       sha256_win32: ${{ steps.sha256_win32.outputs.sha256_win32 }}
@@ -217,15 +313,16 @@ jobs:
       with:
           python-version: '3.7'
           architecture: 'x86'
-    - name: Upgrade pip and enable wheel support
-      run: python -m pip install --upgrade pip setuptools wheel
     - name: Install Requirements
-      run: pip install "https://yt-dlp.github.io/Pyinstaller-Builds/i686/pyinstaller-4.5.1-py3-none-any.whl" mutagen pycryptodomex websockets
+      run: |
+          python -m pip install --upgrade pip setuptools wheel
+          pip install "https://yt-dlp.github.io/Pyinstaller-Builds/i686/pyinstaller-4.5.1-py3-none-any.whl" mutagen pycryptodomex websockets
     - name: Bump version
       id: bump_version
       run: python devscripts/update-version.py
-    - name: Print version
-      run: echo "${{ steps.bump_version.outputs.ytdlp_version }}"
+    - name: Build lazy extractors
+      id: lazy_extractors
+      run: python devscripts/make_lazy_extractors.py
     - name: Run PyInstaller Script for 32 Bit
       run: python pyinst.py
     - name: Upload Executable yt-dlp_x86.exe
@@ -247,22 +344,28 @@ jobs:
 
   finish:
     runs-on: ubuntu-latest
-    needs: [build_unix, build_windows, build_windows32]
+    needs: [build_unix, build_windows, build_windows32, build_macos]
 
     steps:
     - name: Make SHA2-256SUMS file
       env:
-        SHA256_WIN: ${{ needs.build_windows.outputs.sha256_win }}
-        SHA256_WIN_ZIP: ${{ needs.build_windows.outputs.sha256_win_zip }}
-        SHA256_WIN32: ${{ needs.build_windows32.outputs.sha256_win32 }}
         SHA256_BIN: ${{ needs.build_unix.outputs.sha256_bin }}
         SHA256_TAR: ${{ needs.build_unix.outputs.sha256_tar }}
+        SHA256_WIN: ${{ needs.build_windows.outputs.sha256_win }}
+        SHA256_PY2EXE: ${{ needs.build_windows.outputs.sha256_py2exe }}
+        SHA256_WIN_ZIP: ${{ needs.build_windows.outputs.sha256_win_zip }}
+        SHA256_WIN32: ${{ needs.build_windows32.outputs.sha256_win32 }}
+        SHA256_MACOS: ${{ needs.build_macos.outputs.sha256_macos }}
+        SHA256_MACOS_ZIP: ${{ needs.build_macos.outputs.sha256_macos_zip }}
       run: |
-        echo "${{ env.SHA256_WIN }}  yt-dlp.exe" >> SHA2-256SUMS
-        echo "${{ env.SHA256_WIN32 }}  yt-dlp_x86.exe" >> SHA2-256SUMS
         echo "${{ env.SHA256_BIN }}  yt-dlp" >> SHA2-256SUMS
         echo "${{ env.SHA256_TAR }}  yt-dlp.tar.gz" >> SHA2-256SUMS
-        echo "${{ env.SHA256_WIN_ZIP }}  yt-dlp.zip" >> SHA2-256SUMS
+        echo "${{ env.SHA256_WIN }}  yt-dlp.exe" >> SHA2-256SUMS
+        echo "${{ env.SHA256_PY2EXE }}  yt-dlp_min.exe" >> SHA2-256SUMS
+        echo "${{ env.SHA256_WIN32 }}  yt-dlp_x86.exe" >> SHA2-256SUMS
+        echo "${{ env.SHA256_WIN_ZIP }}  yt-dlp_win.zip" >> SHA2-256SUMS
+        echo "${{ env.SHA256_MACOS }}  yt-dlp_macos" >> SHA2-256SUMS
+        echo "${{ env.SHA256_MACOS_ZIP }}  yt-dlp_macos.zip" >> SHA2-256SUMS
     - name: Upload 256SUMS file
       id: upload-sums
       uses: actions/upload-release-asset@v1
@@ -275,17 +378,23 @@ jobs:
         asset_content_type: text/plain
     - name: Make SHA2-512SUMS file
       env:
-        SHA512_WIN: ${{ needs.build_windows.outputs.sha512_win }}
-        SHA512_WIN_ZIP: ${{ needs.build_windows.outputs.sha512_win_zip }}
-        SHA512_WIN32: ${{ needs.build_windows32.outputs.sha512_win32 }}
         SHA512_BIN: ${{ needs.build_unix.outputs.sha512_bin }}
         SHA512_TAR: ${{ needs.build_unix.outputs.sha512_tar }}
+        SHA512_WIN: ${{ needs.build_windows.outputs.sha512_win }}
+        SHA512_PY2EXE: ${{ needs.build_windows.outputs.sha512_py2exe }}
+        SHA512_WIN_ZIP: ${{ needs.build_windows.outputs.sha512_win_zip }}
+        SHA512_WIN32: ${{ needs.build_windows32.outputs.sha512_win32 }}
+        SHA512_MACOS: ${{ needs.build_macos.outputs.sha512_macos }}
+        SHA512_MACOS_ZIP: ${{ needs.build_macos.outputs.sha512_macos_zip }}
       run: |
-        echo "${{ env.SHA512_WIN }}  yt-dlp.exe" >> SHA2-512SUMS
-        echo "${{ env.SHA512_WIN32 }}  yt-dlp_x86.exe" >> SHA2-512SUMS
         echo "${{ env.SHA512_BIN }}  yt-dlp" >> SHA2-512SUMS
         echo "${{ env.SHA512_TAR }}  yt-dlp.tar.gz" >> SHA2-512SUMS
-        echo "${{ env.SHA512_WIN_ZIP }}  yt-dlp.zip" >> SHA2-512SUMS
+        echo "${{ env.SHA512_WIN }}  yt-dlp.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_WIN_ZIP }}  yt-dlp_win.zip" >> SHA2-512SUMS
+        echo "${{ env.SHA512_PY2EXE }}  yt-dlp_min.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_WIN32 }}  yt-dlp_x86.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_MACOS }}  yt-dlp_macos" >> SHA2-512SUMS
+        echo "${{ env.SHA512_MACOS_ZIP }}  yt-dlp_macos.zip" >> SHA2-512SUMS
     - name: Upload 512SUMS file
       id: upload-512sums
       uses: actions/upload-release-asset@v1

.github/workflows/quick-test.yml

@@ -28,6 +28,6 @@ jobs:
     - name: Install flake8
       run: pip install flake8
     - name: Make lazy extractors
-      run: python devscripts/make_lazy_extractors.py yt_dlp/extractor/lazy_extractors.py
+      run: python devscripts/make_lazy_extractors.py
     - name: Run flake8
       run: flake8 .

CONTRIBUTING.md

@@ -109,6 +109,18 @@ Some bug reports are completely unrelated to yt-dlp and relate to a different, o
 
 If the issue is with `youtube-dl` (the upstream fork of yt-dlp) and not with yt-dlp, the issue should be raised in the youtube-dl project.
 
+### Are you willing to share account details if needed?
+
+The maintainers and potential contributors of the project often do not have an account for the website you are asking support for. So any developer interested in solving your issue may ask you for account details. It is your personal discression whether you are willing to share the account in order for the developer to try and solve your issue. However, if you are unwilling or unable to provide details, they obviously cannot work on the issue and it cannot be solved unless some developer who both has an account and is willing/able to contribute decides to solve it.
+
+By sharing an account with anyone, you agree to bear all risks associated with it. The maintainers and yt-dlp can't be held responsible for any misuse of the credentials.
+
+While these steps won't necessarily ensure that no misuse of the account takes place, these are still some good practices to follow.
+
+- Look for people with `Member` or `Contributor` tag on their messages.
+- Change the password before sharing the account to something random (use [this](https://passwordsgenerator.net/) if you don't have a random password generator).
+- Change the password after receiving the account back.
+
 
 
 

CONTRIBUTORS

@@ -123,3 +123,9 @@ ajj8
 jakubadamw
 jfogelman
 timethrow
+sarnoud
+Bojidarist
+18928172992817182/gustaf
+nixklai
+smplayer-dev
+Zirro

Changelog.md

@@ -14,6 +14,95 @@
 -->
 
 
+### 2021.10.22
+
+* [build] Improvements
+    * Build standalone MacOS packages by [smplayer-dev](https://github.com/smplayer-dev)
+    * Release windows exe built with `py2exe`
+    * Enable lazy-extractors in releases. 
+        * Set env var `YTDLP_NO_LAZY_EXTRACTORS` to forcefully disable this (experimental)
+    * Clean up error reporting in update
+    * Refactor `pyinst.py`, misc cleanup and improve docs
+* [docs] Migrate issues to use forms by [Ashish0804](https://github.com/Ashish0804)
+* [downloader] **Fix slow progress hooks**
+    * This was causing HLS/DASH downloads to be extremely slow in some situations
+* [downloader/ffmpeg] Improve simultaneous download and merge
+* [EmbedMetadata] Allow overwriting all default metadata with `meta_default` key
+* [ModifyChapters] Add ability for `--remove-chapters` to remove sections by timestamp
+* [utils] Allow duration strings in `--match-filter`
+* Add HDR information to formats
+* Add negative option `--no-batch-file` by [Zirro](https://github.com/Zirro)
+* Calculate more fields for merged formats
+* Do not verify thumbnail URLs unless `--check-formats` is specified
+* Don't create console for subprocesses on Windows
+* Fix `--restrict-filename` when used with default template
+* Fix `check_formats` output being written to stdout when `-qv`
+* Fix bug in storyboards
+* Fix conflict b/w id and ext in format selection
+* Fix verbose head not showing custom configs
+* Load archive only after printing verbose head
+* Make `duration_string` and `resolution` available in --match-filter
+* Re-implement deprecated option `--id`
+* Reduce default `--socket-timeout`
+* Write verbose header to logger
+* [outtmpl] Fix bug in expanding environment variables
+* [cookies] Local State should be opened as utf-8
+* [extractor,utils] Detect more codecs/mimetypes
+* [extractor] Detect `EXT-X-KEY` Apple FairPlay
+* [utils] Use `importlib` to load plugins by [sulyi](https://github.com/sulyi)
+* [http] Retry on socket timeout and show the last encountered error
+* [fragment] Print error message when skipping fragment
+* [aria2c] Fix `--skip-unavailable-fragment`
+* [SponsorBlock] Obey `extractor-retries` and `sleep-requests`
+* [Merger] Do not add `aac_adtstoasc` to non-hls audio
+* [ModifyChapters] Do not mutate original chapters by [nihil-admirari](https://github.com/nihil-admirari)
+* [devscripts/run_tests] Use markers to filter tests by [sulyi](https://github.com/sulyi)
+* [7plus] Add cookie based authentication by [nyuszika7h](https://github.com/nyuszika7h)
+* [AdobePass] Fix RCN MSO by [jfogelman](https://github.com/jfogelman)
+* [CBC] Fix Gem livestream by [makeworld-the-better-one](https://github.com/makeworld-the-better-one)
+* [CBC] Support CBC Gem member content by [makeworld-the-better-one](https://github.com/makeworld-the-better-one)
+* [crunchyroll] Add season to flat-playlist Closes #1319
+* [crunchyroll] Add support for `beta.crunchyroll` URLs and fix series URLs with language code
+* [EUScreen] Add Extractor by [Ashish0804](https://github.com/Ashish0804)
+* [Gronkh] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [hidive] Fix typo
+* [Hotstar] Mention Dynamic Range in `format_id` by [Ashish0804](https://github.com/Ashish0804)
+* [Hotstar] Raise appropriate error for DRM
+* [instagram] Add login by [u-spec-png](https://github.com/u-spec-png)
+* [instagram] Show appropriate error when login is needed
+* [microsoftstream] Add extractor by [damianoamatruda](https://github.com/damianoamatruda), [nixklai](https://github.com/nixklai)
+* [on24] Add extractor by [damianoamatruda](https://github.com/damianoamatruda)
+* [patreon] Fix vimeo player regex by [zenerdi0de](https://github.com/zenerdi0de)
+* [SkyNewsAU] Add extractor by [Ashish0804](https://github.com/Ashish0804)
+* [tagesschau] Fix extractor by [u-spec-png](https://github.com/u-spec-png)
+* [tbs] Add tbs live streams by [llacb47](https://github.com/llacb47)
+* [tiktok] Fix typo and update tests
+* [trovo] Support channel clips and VODs by [Ashish0804](https://github.com/Ashish0804)
+* [Viafree] Add support for Finland by [18928172992817182](https://github.com/18928172992817182)
+* [vimeo] Fix embedded `player.vimeo`
+* [vlive:channel] Fix extraction by [kikuyan](https://github.com/kikuyan), [pukkandan](https://github.com/pukkandan)
+* [youtube] Add auto-translated subtitles
+* [youtube] Expose different formats with same itag
+* [youtube:comments] Fix for new layout by [coletdjnz](https://github.com/coletdjnz)
+* [cleanup] Cleanup bilibili code by [pukkandan](https://github.com/pukkandan), [u-spec-png](https://github.com/u-spec-png)
+* [cleanup] Remove broken youtube login code
+* [cleanup] Standardize timestamp formatting code
+* [cleanup] Generalize `getcomments` implementation for extractors
+* [cleanup] Simplify search extractors code
+* [cleanup] misc
+
+
+### 2021.10.10
+
+* [downloader/ffmpeg] Fix bug in initializing `FFmpegPostProcessor`
+* [minicurses] Fix when printing to file
+* [downloader] Fix throttledratelimit
+* [francetv] Fix extractor by [fstirlitz](https://github.com/fstirlitz), [sarnoud](https://github.com/sarnoud)
+* [NovaPlay] Add extractor by [Bojidarist](https://github.com/Bojidarist)
+* [ffmpeg] Revert "Set max probesize" - No longer needed
+* [docs] Remove incorrect dependency on VC++10
+* [build] Allow to release without changelog
+
 ### 2021.10.09
 
 * Improved progress reporting

Makefile

@@ -1,4 +1,4 @@
-all: yt-dlp doc pypi-files
+all: lazy-extractors yt-dlp doc pypi-files
 clean: clean-test clean-dist clean-cache
 completions: completion-bash completion-fish completion-zsh
 doc: README.md CONTRIBUTING.md issuetemplates supportedsites
@@ -40,9 +40,9 @@ SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then ech
 # set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2
 MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi)
 
-install: yt-dlp yt-dlp.1 completions
-	install -Dm755 yt-dlp $(DESTDIR)$(BINDIR)
-	install -Dm644 yt-dlp.1 $(DESTDIR)$(MANDIR)/man1
+install: lazy-extractors yt-dlp yt-dlp.1 completions
+	install -Dm755 yt-dlp $(DESTDIR)$(BINDIR)/yt-dlp
+	install -Dm644 yt-dlp.1 $(DESTDIR)$(MANDIR)/man1/yt-dlp.1
 	install -Dm644 completions/bash/yt-dlp $(DESTDIR)$(SHAREDIR)/bash-completion/completions/yt-dlp
 	install -Dm644 completions/zsh/_yt-dlp $(DESTDIR)$(SHAREDIR)/zsh/site-functions/_yt-dlp
 	install -Dm644 completions/fish/yt-dlp.fish $(DESTDIR)$(SHAREDIR)/fish/vendor_completions.d/yt-dlp.fish
@@ -78,12 +78,13 @@ README.md: yt_dlp/*.py yt_dlp/*/*.py
 CONTRIBUTING.md: README.md
 	$(PYTHON) devscripts/make_contributing.py README.md CONTRIBUTING.md
 
-issuetemplates: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.md .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md .github/ISSUE_TEMPLATE_tmpl/4_bug_report.md .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md yt_dlp/version.py
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.md .github/ISSUE_TEMPLATE/1_broken_site.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md .github/ISSUE_TEMPLATE/2_site_support_request.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md .github/ISSUE_TEMPLATE/3_site_feature_request.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/4_bug_report.md .github/ISSUE_TEMPLATE/4_bug_report.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md .github/ISSUE_TEMPLATE/5_feature_request.md
+issuetemplates: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml .github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml .github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml yt_dlp/version.py
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.yml .github/ISSUE_TEMPLATE/1_broken_site.yml
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.yml .github/ISSUE_TEMPLATE/2_site_support_request.yml
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.yml .github/ISSUE_TEMPLATE/3_site_feature_request.yml
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/4_bug_report.yml .github/ISSUE_TEMPLATE/4_bug_report.yml
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.yml .github/ISSUE_TEMPLATE/5_feature_request.yml
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/6_question.yml .github/ISSUE_TEMPLATE/6_question.yml
 
 supportedsites:
 	$(PYTHON) devscripts/make_supportedsites.py supportedsites.md

README.md

@@ -22,6 +22,7 @@ yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on t
     * [Differences in default behavior](#differences-in-default-behavior)
 * [INSTALLATION](#installation)
     * [Update](#update)
+    * [Release Files](#release-files)
     * [Dependencies](#dependencies)
     * [Compile](#compile)
 * [USAGE AND OPTIONS](#usage-and-options)
@@ -92,9 +93,9 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **Aria2c with HLS/DASH**: You can use `aria2c` as the external downloader for DASH(mpd) and HLS(m3u8) formats
 
-* **New extractors**: AnimeLab, Philo MSO, Spectrum MSO, SlingTV MSO, Cablevision MSO, RCN MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries, VidioPremier, VidioLive, RCTIPlus, TBS Live, douyin, pornflip, ParamountPlusSeries, ScienceChannel, Utreon, OpenRec, BandcampMusic, blackboardcollaborate, eroprofile albums, mirrativ, BannedVideo, bilibili categories, Epicon, filmmodu, GabTV, HungamaAlbum, ManotoTV, Niconico search, Patreon User, peloton, ProjectVeritas, radiko, StarTV, tiktok user, Tokentube, voicy, TV2HuSeries, biliintl, 17live, NewgroundsUser, peertube channel/playlist, ZenYandex, CAM4, CGTN, damtomo, gotostage, Koo, Mediaite, Mediaklikk, MuseScore, nzherald, Olympics replay, radlive, SovietsCloset, Streamanity, Theta, Chingari, ciscowebex, Gettr, GoPro, N1, Theta, Veo, Vupload
+* **New extractors**: AnimeLab, Philo MSO, Spectrum MSO, SlingTV MSO, Cablevision MSO, RCN MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries, VidioPremier, VidioLive, RCTIPlus, TBS Live, douyin, pornflip, ParamountPlusSeries, ScienceChannel, Utreon, OpenRec, BandcampMusic, blackboardcollaborate, eroprofile albums, mirrativ, BannedVideo, bilibili categories, Epicon, filmmodu, GabTV, HungamaAlbum, ManotoTV, Niconico search, Patreon User, peloton, ProjectVeritas, radiko, StarTV, tiktok user, Tokentube, voicy, TV2HuSeries, biliintl, 17live, NewgroundsUser, peertube channel/playlist, ZenYandex, CAM4, CGTN, damtomo, gotostage, Koo, Mediaite, Mediaklikk, MuseScore, nzherald, Olympics replay, radlive, SovietsCloset, Streamanity, Theta, Chingari, ciscowebex, Gettr, GoPro, N1, Theta, Veo, Vupload, NovaPlay, SkyNewsAU, EUScreen, Gronkh, microsoftstream, on24, trovo channels
 
-* **Fixed/improved extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay, patreon, videa, yahoo, BravoTV, crunchyroll playlist, RTP, viki, Hotstar, vidio, vimeo, mediaset, Mxplayer, nbcolympics, ParamountPlus, Newgrounds, SAML Verizon login, Hungama, afreecatv, aljazeera, ATV, bitchute, camtube, CDA, eroprofile, facebook, HearThisAtIE, iwara, kakao, Motherless, Nova, peertube, pornhub, reddit, tiktok, TV2, TV2Hu, tv5mondeplus, VH1, Viafree, XHamster, 9Now, AnimalPlanet, Arte, CBC, Chingari, comedycentral, DIYNetwork, niconico, dw, funimation, globo, HiDive, NDR, Nuvid, Oreilly, pbs, plutotv, reddit, redtube, soundcloud, SpankBang, VrtNU, bbc, Bilibili, LinkedInLearning, parliamentlive, PolskieRadio, Streamable, vidme
+* **Fixed/improved extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay, patreon, videa, yahoo, BravoTV, crunchyroll, RTP, viki, Hotstar, vidio, vimeo, mediaset, Mxplayer, nbcolympics, ParamountPlus, Newgrounds, SAML Verizon login, Hungama, afreecatv, aljazeera, ATV, bitchute, camtube, CDA, eroprofile, facebook, HearThisAtIE, iwara, kakao, Motherless, Nova, peertube, pornhub, reddit, tiktok, TV2, TV2Hu, tv5mondeplus, VH1, Viafree, XHamster, 9Now, AnimalPlanet, Arte, CBC, Chingari, comedycentral, DIYNetwork, niconico, dw, funimation, globo, HiDive, NDR, Nuvid, Oreilly, pbs, plutotv, reddit, redtube, soundcloud, SpankBang, VrtNU, bbc, Bilibili, LinkedInLearning, parliamentlive, PolskieRadio, Streamable, vidme, francetv, 7plus, tagesschau
 
 * **Subtitle extraction from manifests**: Subtitles can be extracted from streaming media manifests. See [commit/be6202f](https://github.com/yt-dlp/yt-dlp/commit/be6202f12b97858b9d716e608394b51065d0419f) for details
 
@@ -154,11 +155,10 @@ For ease of use, a few more compat options are available:
 yt-dlp is not platform specific. So it should work on your Unix box, on Windows or on macOS
 
 You can install yt-dlp using one of the following methods:
-* Download the binary from the [latest release](https://github.com/yt-dlp/yt-dlp/releases/latest)
+* Download [the binary](#release-files) from the [latest release](https://github.com/yt-dlp/yt-dlp/releases/latest)
 * With Homebrew, `brew install yt-dlp/taps/yt-dlp`
 * Use [PyPI package](https://pypi.org/project/yt-dlp): `python3 -m pip install --upgrade yt-dlp`
-* Use pip+git: `python3 -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp.git@release`
-* Install master branch: `python3 -m pip install --upgrade git+https://github.com/yt-dlp/yt-dlp`
+* Install master branch: `python3 -m pip3 install -U https://github.com/yt-dlp/yt-dlp/archive/master.zip`
 
 Note that on some systems, you may need to use `py` or `python` instead of `python3`
 
@@ -190,11 +190,40 @@ You can use `yt-dlp -U` to update if you are using the provided release.
 If you are using `pip`, simply re-run the same command that was used to install the program.
 If you have installed using Homebrew, run `brew upgrade yt-dlp/taps/yt-dlp`
 
+### RELEASE FILES
+
+#### Recommended
+
+File|Description
+:---|:---
+[yt-dlp](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp)|Platform independant binary. Needs Python (Recommended for **UNIX-like systems**)
+[yt-dlp.exe](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.exe)|Windows standalone x64 binary (Recommended for **Windows**)
+
+#### Alternatives
+
+File|Description
+:---|:---
+[yt-dlp_macos](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_macos)|MacOS standalone executable
+[yt-dlp_x86.exe](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_x86.exe)|Windows standalone x86 (32bit) binary
+[yt-dlp_min.exe](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_min.exe)|Windows standalone x64 binary built with `py2exe`.<br/> Does not contain `pycryptodomex`, needs VC++14
+[yt-dlp_win.zip](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_win.zip)|Unpackaged windows executable (No auto-update)
+[yt-dlp_macos.zip](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp_macos.zip)|Unpackaged MacOS executable (No auto-update)
+
+#### Misc
+
+File|Description
+:---|:---
+[yt-dlp.tar.gz](https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp.tar.gz)|Source tarball. Also contains manpages, completions, etc
+[SHA2-512SUMS](https://github.com/yt-dlp/yt-dlp/releases/latest/download/SHA2-512SUMS)|GNU-style SHA512 sums
+[SHA2-256SUMS](https://github.com/yt-dlp/yt-dlp/releases/latest/download/SHA2-256SUMS)|GNU-style SHA256 sums
+
 ### DEPENDENCIES
 Python versions 3.6+ (CPython and PyPy) are supported. Other versions and implementations may or may not work correctly.
 
-<!-- https://www.microsoft.com/en-us/download/details.aspx?id=26999 -->
+<!-- Python 3.5+ uses VC++14 and it is already embedded in the binary created
+<!x-- https://www.microsoft.com/en-us/download/details.aspx?id=26999 --x>
 On windows, [Microsoft Visual C++ 2010 SP1 Redistributable Package (x86)](https://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe) is also necessary to run yt-dlp. You probably already have this, but if the executable throws an error due to missing `MSVCR100.dll` you need to install it manually.
+-->
 
 While all the other dependancies are optional, `ffmpeg` and `ffprobe` are highly recommended
 * [**ffmpeg** and **ffprobe**](https://www.ffmpeg.org) - Required for [merging seperate video and audio files](#format-selection) as well as for various [post-processing](#post-processing-options) tasks. Licence [depends on the build](https://www.ffmpeg.org/legal.html)
@@ -219,15 +248,11 @@ The windows releases are already built with the python interpreter, mutagen, pyc
 ### COMPILE
 
 **For Windows**:
-To build the Windows executable, you must have pyinstaller (and optionally mutagen, pycryptodomex, websockets)
+To build the Windows executable, you must have pyinstaller (and optionally mutagen, pycryptodomex, websockets). Once you have all the necessary dependencies installed, (optionally) build lazy extractors using `devscripts/make_lazy_extractors.py`, and then just run `pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it.
 
-    python3 -m pip install -U -r requirements.txt
-
-Once you have all the necessary dependencies installed, just run `py pyinst.py`. The executable will be built for the same architecture (32/64 bit) as the python used to build it.
-
-You can also build the executable without any version info or metadata by using:
-
-    pyinstaller.exe yt_dlp\__main__.py --onefile --name yt-dlp
+    py -m pip install -U pyinstaller -r requirements.txt
+    py devscripts/make_lazy_extractors.py
+    py pyinst.py
 
 Note that pyinstaller [does not support](https://github.com/pyinstaller/pyinstaller#requirements-and-tested-platforms) Python installed from the Windows store without using a virtual environment
 
@@ -235,7 +260,7 @@ Note that pyinstaller [does not support](https://github.com/pyinstaller/pyinstal
 You will need the required build tools: `python`, `make` (GNU), `pandoc`, `zip`, `pytest`  
 Then simply run `make`. You can also run `make yt-dlp` instead to compile only the binary without updating any of the additional files
 
-**Note**: In either platform, `devscripts\update-version.py` can be used to automatically update the version number
+**Note**: In either platform, `devscripts/update-version.py` can be used to automatically update the version number
 
 # USAGE AND OPTIONS
 
@@ -463,6 +488,7 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      stdin), one URL per line. Lines starting
                                      with '#', ';' or ']' are considered as
                                      comments and ignored
+    --no-batch-file                  Do not read URLs from batch file (default)
     -P, --paths [TYPES:]PATH         The paths where the files should be
                                      downloaded. Specify the type of file and
                                      the path separated by a colon ":". All the
@@ -845,7 +871,11 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
     --no-split-chapters              Do not split video based on chapters
                                      (default)
     --remove-chapters REGEX          Remove chapters whose title matches the
-                                     given regular expression. This option can
+                                     given regular expression. Time ranges
+                                     prefixed by a "*" can also be used in place
+                                     of chapters to remove the specified range.
+                                     Eg: --remove-chapters "*10:15-15:00"
+                                     --remove-chapters "intro". This option can
                                      be used multiple times
     --no-remove-chapters             Do not remove any chapters from the file
                                      (default)
@@ -1054,6 +1084,7 @@ The available fields are:
  - `asr` (numeric): Audio sampling rate in Hertz
  - `vbr` (numeric): Average video bitrate in KBit/s
  - `fps` (numeric): Frame rate
+ - `dynamic_range` (string): The dynamic range of the video
  - `vcodec` (string): Name of the video codec in use
  - `container` (string): Name of the container format
  - `filesize` (numeric): The number of bytes, if known in advance
@@ -1124,11 +1155,13 @@ Available only in `--sponsorblock-chapter-title`:
  - `category_names` (list): Friendly names of the categories
  - `name` (string): Friendly name of the smallest category
 
-Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default).
+Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `yt-dlp test video` and id `BaW_jenozKc`, this will result in a `yt-dlp test video-BaW_jenozKc.mp4` file created in the current directory.
 
-For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `yt-dlp test video` and id `BaW_jenozKc`, this will result in a `yt-dlp test video-BaW_jenozKc.mp4` file created in the current directory.
+Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default).
 
-For numeric sequences you can use numeric related formatting, for example, `%(view_count)05d` will result in a string with view count padded with zeros up to 5 characters, like in `00042`.
+**Tip**: Look at the `-j` output to identify which fields are available for the purticular URL
+
+For numeric sequences you can use [numeric related formatting](https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting), for example, `%(view_count)05d` will result in a string with view count padded with zeros up to 5 characters, like in `00042`.
 
 Output templates can also contain arbitrary hierarchical path, e.g. `-o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s'` which will result in downloading each video in a directory corresponding to this path template. Any missing directory will be automatically created for you.
 
@@ -1177,6 +1210,8 @@ $ yt-dlp -o - BaW_jenozKc
 By default, yt-dlp tries to download the best available quality if you **don't** pass any options.
 This is generally equivalent to using `-f bestvideo*+bestaudio/best`. However, if multiple audiostreams is enabled (`--audio-multistreams`), the default format changes to `-f bestvideo+bestaudio/best`. Similarly, if ffmpeg is unavailable, or if you use yt-dlp to stream to `stdout` (`-o -`), the default becomes `-f best/bestvideo+bestaudio`.
 
+**Deprecation warning**: Latest versions of yt-dlp can stream multiple formats to the stdout simultaneously using ffmpeg. So, in future versions, the default for this will be set to `-f bv*+ba/b` similar to normal downloads. If you want to preserve the `-f b/bv+ba` setting, it is recommended to explicitly specify it in the configuration options.
+
 The general syntax for format selection is `-f FORMAT` (or `--format FORMAT`) where `FORMAT` is a *selector expression*, i.e. an expression that describes format or formats you would like to download.
 
 **tl;dr:** [navigate me to examples](#format-selection-examples).
@@ -1275,6 +1310,7 @@ The available fields are:
  - `width`: Width of video
  - `res`: Video resolution, calculated as the smallest dimension.
  - `fps`: Framerate of video
+ - `hdr`: The dynamic range of the video (`DV` > `HDR12` > `HDR10+` > `HDR10` > `HLG` > `SDR`)
  - `tbr`: Total average bitrate in KBit/s
  - `vbr`: Average video bitrate in KBit/s
  - `abr`: Average audio bitrate in KBit/s
@@ -1285,9 +1321,9 @@ The available fields are:
 
 All fields, unless specified otherwise, are sorted in descending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a preferred value for the fields, separated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two preferred values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp9.2` > `av01` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
 
-The fields `hasvid` and `ie_pref` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--format-sort-force`. Apart from these, the default order used is: `lang,quality,res,fps,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. The extractors may override this default order, but they cannot override the user-provided order.
+The fields `hasvid` and `ie_pref` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--format-sort-force`. Apart from these, the default order used is: `lang,quality,res,fps,hdr:12,codec:vp9.2,size,br,asr,proto,ext,hasaud,source,id`. The extractors may override this default order, but they cannot override the user-provided order.
 
-Note that the default has `codec:vp9.2`; i.e. `av1` is not prefered
+Note that the default has `codec:vp9.2`; i.e. `av1` is not prefered. Similarly, the default for hdr is `hdr:12`; i.e. dolby vision is not prefered. These choices are made since DV and AV1 formats are not yet fully compatible with most devices. This may be changed in the future as more devices become capable of smoothly playing back these formats.
 
 If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all respects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
 
@@ -1429,7 +1465,7 @@ Note that any field created by this can be used in the [output template](#output
 
 This option also has a few special uses:
 * You can download an additional URL based on the metadata of the currently downloaded video. To do this, set the field `additional_urls` to the URL that you want to download. Eg: `--parse-metadata "description:(?P<additional_urls>https?://www\.vimeo\.com/\d+)` will download the first vimeo video found in the description
-* You can use this to change the metadata that is embedded in the media file. To do this, set the value of the corresponding field with a `meta_` prefix. For example, any value you set to `meta_description` field will be added to the `description` field in the file. For example, you can use this to set a different "description" and "synopsis"
+* You can use this to change the metadata that is embedded in the media file. To do this, set the value of the corresponding field with a `meta_` prefix. For example, any value you set to `meta_description` field will be added to the `description` field in the file. For example, you can use this to set a different "description" and "synopsis". Any value set to the `meta_` field will overwrite all default values.
 
 For reference, these are the fields yt-dlp adds by default to the file metadata:
 
@@ -1592,6 +1628,8 @@ with yt_dlp.YoutubeDL(ydl_opts) as ydl:
 
 See the public functions in [`yt_dlp/YoutubeDL.py`](yt_dlp/YoutubeDL.py) for other available functions. Eg: `ydl.download`, `ydl.download_with_info_file`
 
+**Tip**: If you are porting your code from youtube-dl to yt-dlp, one important point to look out for is that we do not guarantee the return value of `YoutubeDL.extract_info` to be json serializable, or even be a dictionary. It will be dictionary-like, but if you want to ensure it is a serializable dictionary, pass it through `YoutubeDL.sanitize_info` as shown in the example above
+
 
 # DEPRECATED OPTIONS
 
@@ -1623,6 +1661,7 @@ While these options still work, their use is not recommended since there are oth
     --print-json                     -j --no-simulate
     --autonumber-size NUMBER         Use string formatting. Eg: %(autonumber)03d
     --autonumber-start NUMBER        Use internal field formatting like %(autonumber+NUMBER)s
+    --id                             -o "%(id)s.%(ext)s"
     --metadata-from-title FORMAT     --parse-metadata "%(title)s:FORMAT"
     --hls-prefer-native              --downloader "m3u8:native"
     --hls-prefer-ffmpeg              --downloader "m3u8:ffmpeg"
@@ -1689,7 +1728,6 @@ These options may no longer work as intended
 #### Removed
 These options were deprecated since 2014 and have now been entirely removed
 
-    --id                             -o "%(id)s.%(ext)s"
     -A, --auto-number                -o "%(autonumber)s-%(id)s.%(ext)s"
     -t, --title                      -o "%(title)s-%(id)s.%(ext)s"
     -l, --literal                    -o accepts literal names

devscripts/make_lazy_extractors.py

@@ -9,7 +9,7 @@ import sys
 
 sys.path.insert(0, dirn(dirn((os.path.abspath(__file__)))))
 
-lazy_extractors_filename = sys.argv[1]
+lazy_extractors_filename = sys.argv[1] if len(sys.argv) > 1 else 'yt_dlp/extractor/lazy_extractors.py'
 if os.path.exists(lazy_extractors_filename):
     os.remove(lazy_extractors_filename)
 

devscripts/run_tests.bat

@@ -3,11 +3,11 @@
 cd /d %~dp0..
 
 if ["%~1"]==[""] (
-    set "test_set="
+    set "test_set="test""
 ) else if ["%~1"]==["core"] (
-    set "test_set=-k "not download""
+    set "test_set="-m not download""
 ) else if ["%~1"]==["download"] (
-    set "test_set=-k download"
+    set "test_set="-m "download""
 ) else (
     echo.Invalid test type "%~1". Use "core" ^| "download"
     exit /b 1

devscripts/run_tests.sh

@@ -3,12 +3,12 @@
 if [ -z $1 ]; then
     test_set='test'
 elif [ $1 = 'core' ]; then
-    test_set='not download'
+    test_set="-m not download"
 elif [ $1 = 'download' ]; then
-    test_set='download'
+    test_set="-m download"
 else
     echo 'Invalid test type "'$1'". Use "core" | "download"'
     exit 1
 fi
 
-python3 -m pytest -k "$test_set"
+python3 -m pytest "$test_set"

pyinst.py

@@ -1,75 +1,84 @@
 #!/usr/bin/env python3
 # coding: utf-8
-
-from __future__ import unicode_literals
-import sys
+import os
 import platform
-
+import sys
 from PyInstaller.utils.hooks import collect_submodules
-from PyInstaller.utils.win32.versioninfo import (
-    VarStruct, VarFileInfo, StringStruct, StringTable,
-    StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
-)
-import PyInstaller.__main__
 
-arch = platform.architecture()[0][:2]
-assert arch in ('32', '64')
-_x86 = '_x86' if arch == '32' else ''
 
-# Compatability with older arguments
-opts = sys.argv[1:]
-if opts[0:1] in (['32'], ['64']):
-    if arch != opts[0]:
-        raise Exception(f'{opts[0]}bit executable cannot be built on a {arch}bit system')
-    opts = opts[1:]
-opts = opts or ['--onefile']
+OS_NAME = platform.system()
+if OS_NAME == 'Windows':
+    from PyInstaller.utils.win32.versioninfo import (
+        VarStruct, VarFileInfo, StringStruct, StringTable,
+        StringFileInfo, FixedFileInfo, VSVersionInfo, SetVersion,
+    )
+elif OS_NAME == 'Darwin':
+    pass
+else:
+    raise Exception('{OS_NAME} is not supported')
 
-print(f'Building {arch}bit version with options {opts}')
+ARCH = platform.architecture()[0][:2]
 
-FILE_DESCRIPTION = 'yt-dlp%s' % (' (32 Bit)' if _x86 else '')
 
-exec(compile(open('yt_dlp/version.py').read(), 'yt_dlp/version.py', 'exec'))
-VERSION = locals()['__version__']
+def main():
+    opts = parse_options()
+    version = read_version()
 
-VERSION_LIST = VERSION.split('.')
-VERSION_LIST = list(map(int, VERSION_LIST)) + [0] * (4 - len(VERSION_LIST))
+    suffix = '_macos' if OS_NAME == 'Darwin' else '_x86' if ARCH == '32' else ''
+    final_file = 'dist/%syt-dlp%s%s' % (
+        'yt-dlp/' if '--onedir' in opts else '', suffix, '.exe' if OS_NAME == 'Windows' else '')
 
-print('Version: %s%s' % (VERSION, _x86))
-print('Remember to update the version using devscipts\\update-version.py')
+    print(f'Building yt-dlp v{version} {ARCH}bit for {OS_NAME} with options {opts}')
+    print('Remember to update the version using  "devscripts/update-version.py"')
+    if not os.path.isfile('yt_dlp/extractor/lazy_extractors.py'):
+        print('WARNING: Building without lazy_extractors. Run  '
+              '"devscripts/make_lazy_extractors.py"  to build lazy extractors', file=sys.stderr)
+    print(f'Destination: {final_file}\n')
 
-VERSION_FILE = VSVersionInfo(
-    ffi=FixedFileInfo(
-        filevers=VERSION_LIST,
-        prodvers=VERSION_LIST,
-        mask=0x3F,
-        flags=0x0,
-        OS=0x4,
-        fileType=0x1,
-        subtype=0x0,
-        date=(0, 0),
-    ),
-    kids=[
-        StringFileInfo([
-            StringTable(
-                '040904B0', [
-                    StringStruct('Comments', 'yt-dlp%s Command Line Interface.' % _x86),
-                    StringStruct('CompanyName', 'https://github.com/yt-dlp'),
-                    StringStruct('FileDescription', FILE_DESCRIPTION),
-                    StringStruct('FileVersion', VERSION),
-                    StringStruct('InternalName', 'yt-dlp%s' % _x86),
-                    StringStruct(
-                        'LegalCopyright',
-                        'pukkandan.ytdlp@gmail.com | UNLICENSE',
-                    ),
-                    StringStruct('OriginalFilename', 'yt-dlp%s.exe' % _x86),
-                    StringStruct('ProductName', 'yt-dlp%s' % _x86),
-                    StringStruct(
-                        'ProductVersion',
-                        '%s%s on Python %s' % (VERSION, _x86, platform.python_version())),
-                ])]),
-        VarFileInfo([VarStruct('Translation', [0, 1200])])
+    opts = [
+        f'--name=yt-dlp{suffix}',
+        '--icon=devscripts/logo.ico',
+        '--upx-exclude=vcruntime140.dll',
+        '--noconfirm',
+        *dependancy_options(),
+        *opts,
+        'yt_dlp/__main__.py',
     ]
-)
+    print(f'Running PyInstaller with {opts}')
+
+    import PyInstaller.__main__
+
+    PyInstaller.__main__.run(opts)
+
+    set_version_info(final_file, version)
+
+
+def parse_options():
+    # Compatability with older arguments
+    opts = sys.argv[1:]
+    if opts[0:1] in (['32'], ['64']):
+        if ARCH != opts[0]:
+            raise Exception(f'{opts[0]}bit executable cannot be built on a {ARCH}bit system')
+        opts = opts[1:]
+    return opts or ['--onefile']
+
+
+def read_version():
+    exec(compile(open('yt_dlp/version.py').read(), 'yt_dlp/version.py', 'exec'))
+    return locals()['__version__']
+
+
+def version_to_list(version):
+    version_list = version.split('.')
+    return list(map(int, version_list)) + [0] * (4 - len(version_list))
+
+
+def dependancy_options():
+    dependancies = [pycryptodome_module(), 'mutagen'] + collect_submodules('websockets')
+    excluded_modules = ['test', 'ytdlp_plugins', 'youtube-dl', 'youtube-dlc']
+
+    yield from (f'--hidden-import={module}' for module in dependancies)
+    yield from (f'--exclude-module={module}' for module in excluded_modules)
 
 
 def pycryptodome_module():
@@ -86,17 +95,41 @@ def pycryptodome_module():
     return 'Cryptodome'
 
 
-dependancies = [pycryptodome_module(), 'mutagen'] + collect_submodules('websockets')
-excluded_modules = ['test', 'ytdlp_plugins', 'youtube-dl', 'youtube-dlc']
+def set_version_info(exe, version):
+    if OS_NAME == 'Windows':
+        windows_set_version(exe, version)
 
-PyInstaller.__main__.run([
-    '--name=yt-dlp%s' % _x86,
-    '--icon=devscripts/logo.ico',
-    *[f'--exclude-module={module}' for module in excluded_modules],
-    *[f'--hidden-import={module}' for module in dependancies],
-    '--upx-exclude=vcruntime140.dll',
-    '--noconfirm',
-    *opts,
-    'yt_dlp/__main__.py',
-])
-SetVersion('dist/%syt-dlp%s.exe' % ('yt-dlp/' if '--onedir' in opts else '', _x86), VERSION_FILE)
+
+def windows_set_version(exe, version):
+    version_list = version_to_list(version)
+    suffix = '_x86' if ARCH == '32' else ''
+    SetVersion(exe, VSVersionInfo(
+        ffi=FixedFileInfo(
+            filevers=version_list,
+            prodvers=version_list,
+            mask=0x3F,
+            flags=0x0,
+            OS=0x4,
+            fileType=0x1,
+            subtype=0x0,
+            date=(0, 0),
+        ),
+        kids=[
+            StringFileInfo([StringTable('040904B0', [
+                StringStruct('Comments', 'yt-dlp%s Command Line Interface.' % suffix),
+                StringStruct('CompanyName', 'https://github.com/yt-dlp'),
+                StringStruct('FileDescription', 'yt-dlp%s' % (' (32 Bit)' if ARCH == '32' else '')),
+                StringStruct('FileVersion', version),
+                StringStruct('InternalName', f'yt-dlp{suffix}'),
+                StringStruct('LegalCopyright', 'pukkandan.ytdlp@gmail.com | UNLICENSE'),
+                StringStruct('OriginalFilename', f'yt-dlp{suffix}.exe'),
+                StringStruct('ProductName', f'yt-dlp{suffix}'),
+                StringStruct(
+                    'ProductVersion', f'{version}{suffix} on Python {platform.python_version()}'),
+            ])]), VarFileInfo([VarStruct('Translation', [0, 1200])])
+        ]
+    ))
+
+
+if __name__ == '__main__':
+    main()

setup.py

@@ -29,7 +29,7 @@ REQUIREMENTS = ['mutagen', 'pycryptodomex', 'websockets']
 if sys.argv[1:2] == ['py2exe']:
     import py2exe
     warnings.warn(
-        'Building with py2exe is not officially supported. '
+        'py2exe builds do not support pycryptodomex and needs VC++14 to run. '
         'The recommended way is to use "pyinst.py" to build using pyinstaller')
     params = {
         'console': [{

supportedsites.md

@@ -226,13 +226,14 @@
  - **Crackle**
  - **CrooksAndLiars**
  - **crunchyroll**
+ - **crunchyroll:beta**
  - **crunchyroll:playlist**
+ - **crunchyroll:playlist:beta**
  - **CSpan**: C-SPAN
  - **CtsNews**: 華視新聞
  - **CTV**
  - **CTVNews**
  - **cu.ntv.co.jp**: Nippon Television Network
- - **Culturebox**
  - **CultureUnplugged**
  - **curiositystream**
  - **curiositystream:collection**
@@ -316,6 +317,7 @@
  - **ESPNArticle**
  - **EsriVideo**
  - **Europa**
+ - **EUScreen**
  - **EWETV**
  - **ExpoTV**
  - **Expressen**
@@ -346,13 +348,10 @@
  - **foxnews**: Fox News and Fox Business Video
  - **foxnews:article**
  - **FoxSports**
- - **france2.fr:generation-what**
  - **FranceCulture**
  - **FranceInter**
  - **FranceTV**
- - **FranceTVEmbed**
  - **francetvinfo.fr**
- - **FranceTVJeunesse**
  - **FranceTVSite**
  - **Freesound**
  - **freespeech.org**
@@ -398,6 +397,7 @@
  - **Goshgay**
  - **GoToStage**
  - **GPUTechConf**
+ - **Gronkh**
  - **Groupon**
  - **hbo**
  - **HearThisAt**
@@ -574,6 +574,7 @@
  - **Mgoon**
  - **MGTV**: 芒果TV
  - **MiaoPai**
+ - **microsoftstream**: Microsoft Stream
  - **mildom**: Record ongoing live by specific user in Mildom
  - **mildom:user:vod**: Download all VODs from specific user in Mildom
  - **mildom:vod**: Download a VOD in Mildom
@@ -705,6 +706,7 @@
  - **NosVideo**
  - **Nova**: TN.cz, Prásk.tv, Nova.cz, Novaplus.cz, FANDA.tv, Krásná.cz and Doma.cz
  - **NovaEmbed**
+ - **NovaPlay**
  - **nowness**
  - **nowness:playlist**
  - **nowness:series**
@@ -737,6 +739,7 @@
  - **Odnoklassniki**
  - **OktoberfestTV**
  - **OlympicsReplay**
+ - **on24**: ON24
  - **OnDemandKorea**
  - **onet.pl**
  - **onet.tv**
@@ -964,6 +967,7 @@
  - **SkylineWebcams**
  - **skynewsarabia:article**
  - **skynewsarabia:video**
+ - **SkyNewsAU**
  - **Slideshare**
  - **SlidesLive**
  - **Slutload**
@@ -973,7 +977,7 @@
  - **SonyLIVSeries**
  - **soundcloud**
  - **soundcloud:playlist**
- - **soundcloud:search**: Soundcloud search
+ - **soundcloud:search**: Soundcloud search, "scsearch" keyword
  - **soundcloud:set**
  - **soundcloud:trackstation**
  - **soundcloud:user**
@@ -991,7 +995,6 @@
  - **SpankBangPlaylist**
  - **Spankwire**
  - **Spiegel**
- - **sport.francetvinfo.fr**
  - **Sport5**
  - **SportBox**
  - **SportDeutschland**
@@ -1033,7 +1036,6 @@
  - **SztvHu**
  - **t-online.de**
  - **Tagesschau**
- - **tagesschau:player**
  - **Tass**
  - **TBS**
  - **TDSLifeway**
@@ -1093,6 +1095,8 @@
  - **TrailerAddict** (Currently broken)
  - **Trilulilu**
  - **Trovo**
+ - **TrovoChannelClip**: All Clips of a trovo.live channel, "trovoclip" keyword
+ - **TrovoChannelVod**: All VODs of a trovo.live channel, "trovovod" keyword
  - **TrovoVod**
  - **TruNews**
  - **TruTV**
@@ -1197,7 +1201,7 @@
  - **Viddler**
  - **Videa**
  - **video.arnes.si**: Arnes Video
- - **video.google:search**: Google Video search
+ - **video.google:search**: Google Video search (Currently broken)
  - **video.sky.it**
  - **video.sky.it:live**
  - **VideoDetective**

test/parameters.json

@@ -44,6 +44,5 @@
     "writesubtitles": false,
     "allsubtitles": false,
     "listsubtitles": false,
-    "socket_timeout": 20,
     "fixup": "never"
 }

test/test_YoutubeDL.py

@@ -817,6 +817,12 @@ class TestYoutubeDL(unittest.TestCase):
         compat_setenv('__yt_dlp_var', 'expanded')
         envvar = '%__yt_dlp_var%' if compat_os_name == 'nt' else '$__yt_dlp_var'
         test(envvar, (envvar, 'expanded'))
+        if compat_os_name == 'nt':
+            test('%s%', ('%s%', '%s%'))
+            compat_setenv('s', 'expanded')
+            test('%s%', ('%s%', 'expanded'))  # %s% should be expanded before escaping %s
+            compat_setenv('(test)s', 'expanded')
+            test('%(test)s%', ('NA%', 'expanded'))  # Environment should take priority over template
 
         # Path expansion and escaping
         test('Hello %(title1)s', 'Hello $PATH')

test/test_utils.py

@@ -848,30 +848,52 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(parse_codecs('avc1.77.30, mp4a.40.2'), {
             'vcodec': 'avc1.77.30',
             'acodec': 'mp4a.40.2',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('mp4a.40.2'), {
             'vcodec': 'none',
             'acodec': 'mp4a.40.2',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('mp4a.40.5,avc1.42001e'), {
             'vcodec': 'avc1.42001e',
             'acodec': 'mp4a.40.5',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('avc3.640028'), {
             'vcodec': 'avc3.640028',
             'acodec': 'none',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs(', h264,,newcodec,aac'), {
             'vcodec': 'h264',
             'acodec': 'aac',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('av01.0.05M.08'), {
             'vcodec': 'av01.0.05M.08',
             'acodec': 'none',
+            'dynamic_range': None,
+        })
+        self.assertEqual(parse_codecs('vp9.2'), {
+            'vcodec': 'vp9.2',
+            'acodec': 'none',
+            'dynamic_range': 'HDR10',
+        })
+        self.assertEqual(parse_codecs('av01.0.12M.10.0.110.09.16.09.0'), {
+            'vcodec': 'av01.0.12M.10',
+            'acodec': 'none',
+            'dynamic_range': 'HDR10',
+        })
+        self.assertEqual(parse_codecs('dvhe'), {
+            'vcodec': 'dvhe',
+            'acodec': 'none',
+            'dynamic_range': 'DV',
         })
         self.assertEqual(parse_codecs('theora, vorbis'), {
             'vcodec': 'theora',
             'acodec': 'vorbis',
+            'dynamic_range': None,
         })
         self.assertEqual(parse_codecs('unknownvcodec, unknownacodec'), {
             'vcodec': 'unknownvcodec',
@@ -1141,12 +1163,15 @@ class TestUtil(unittest.TestCase):
     def test_parse_resolution(self):
         self.assertEqual(parse_resolution(None), {})
         self.assertEqual(parse_resolution(''), {})
-        self.assertEqual(parse_resolution('1920x1080'), {'width': 1920, 'height': 1080})
-        self.assertEqual(parse_resolution('1920×1080'), {'width': 1920, 'height': 1080})
+        self.assertEqual(parse_resolution(' 1920x1080'), {'width': 1920, 'height': 1080})
+        self.assertEqual(parse_resolution('1920×1080 '), {'width': 1920, 'height': 1080})
         self.assertEqual(parse_resolution('1920 x 1080'), {'width': 1920, 'height': 1080})
         self.assertEqual(parse_resolution('720p'), {'height': 720})
         self.assertEqual(parse_resolution('4k'), {'height': 2160})
         self.assertEqual(parse_resolution('8K'), {'height': 4320})
+        self.assertEqual(parse_resolution('pre_1920x1080_post'), {'width': 1920, 'height': 1080})
+        self.assertEqual(parse_resolution('ep1x2'), {})
+        self.assertEqual(parse_resolution('1920, 1080'), {'width': 1920, 'height': 1080})
 
     def test_parse_bitrate(self):
         self.assertEqual(parse_bitrate(None), None)
@@ -1231,6 +1256,7 @@ ffmpeg version 2.4.4 Copyright (c) 2000-2014 the FFmpeg ...'''), '2.4.4')
         self.assertFalse(match_str('x>2K', {'x': 1200}))
         self.assertTrue(match_str('x>=1200 & x < 1300', {'x': 1200}))
         self.assertFalse(match_str('x>=1100 & x < 1200', {'x': 1200}))
+        self.assertTrue(match_str('x > 1:0:0', {'x': 3700}))
 
         # String
         self.assertFalse(match_str('y=a212', {'y': 'foobar42'}))
@@ -1367,21 +1393,21 @@ The first line
   </body>
 </tt>'''.encode('utf-8')
         srt_data = '''1
-00:00:02,080 --> 00:00:05,839
+00:00:02,080 --> 00:00:05,840
 <font color="white" face="sansSerif" size="16">default style<font color="red">custom style</font></font>
 
 2
-00:00:02,080 --> 00:00:05,839
+00:00:02,080 --> 00:00:05,840
 <b><font color="cyan" face="sansSerif" size="16"><font color="lime">part 1
 </font>part 2</font></b>
 
 3
-00:00:05,839 --> 00:00:09,560
+00:00:05,840 --> 00:00:09,560
 <u><font color="lime">line 3
 part 3</font></u>
 
 4
-00:00:09,560 --> 00:00:12,359
+00:00:09,560 --> 00:00:12,360
 <i><u><font color="yellow"><font color="lime">inner
  </font>style</font></u></i>
 

yt_dlp/YoutubeDL.py

@@ -87,10 +87,10 @@ from .utils import (
     parse_filesize,
     PerRequestProxyHandler,
     platform_name,
+    Popen,
     PostProcessingError,
     preferredencoding,
     prepend_extension,
-    process_communicate_or_kill,
     register_socks_protocols,
     RejectedVideoReached,
     render_table,
@@ -307,7 +307,7 @@ class YoutubeDL(object):
     cookiefile:        File name where cookies should be read from and dumped to
     cookiesfrombrowser: A tuple containing the name of the browser and the profile
                        name/path from where cookies are loaded.
-                       Eg: ('chrome', ) or (vivaldi, 'default')
+                       Eg: ('chrome', ) or ('vivaldi', 'default')
     nocheckcertificate:Do not verify SSL certificates
     prefer_insecure:   Use HTTP instead of HTTPS to retrieve information.
                        At the moment, this is only supported by YouTube.
@@ -483,6 +483,12 @@ class YoutubeDL(object):
         'track_number', 'disc_number', 'release_year',
     ))
 
+    _format_selection_exts = {
+        'audio': {'m4a', 'mp3', 'ogg', 'aac'},
+        'video': {'mp4', 'flv', 'webm', '3gp'},
+        'storyboards': {'mhtml'},
+    }
+
     params = None
     _ies = {}
     _pps = {'pre_process': [], 'before_dl': [], 'after_move': [], 'post_process': []}
@@ -495,7 +501,10 @@ class YoutubeDL(object):
     _screen_file = None
 
     def __init__(self, params=None, auto_init=True):
-        """Create a FileDownloader object with the given options."""
+        """Create a FileDownloader object with the given options.
+        @param auto_init    Whether to load the default extractors and print header (if verbose).
+                            Set to 'no_verbose_header' to not print the header
+        """
         if params is None:
             params = {}
         self._ies = {}
@@ -514,6 +523,7 @@ class YoutubeDL(object):
         self.cache = Cache(self)
 
         windows_enable_vt_mode()
+        # FIXME: This will break if we ever print color to stdout
         self.params['no_color'] = self.params.get('no_color') or not supports_terminal_sequences(self._err_file)
 
         if sys.version_info < (3, 6):
@@ -541,7 +551,7 @@ class YoutubeDL(object):
         check_deprecated('usetitle', '--title', '-o "%(title)s-%(id)s.%(ext)s"')
         check_deprecated('useid', '--id', '-o "%(id)s.%(ext)s"')
 
-        for msg in self.params.get('warnings', []):
+        for msg in self.params.get('_warnings', []):
             self.report_warning(msg)
 
         if 'overwrites' not in self.params and self.params.get('nooverwrites') is not None:
@@ -568,16 +578,15 @@ class YoutubeDL(object):
                     stdout=slave,
                     stderr=self._err_file)
                 try:
-                    self._output_process = subprocess.Popen(
-                        ['bidiv'] + width_args, **sp_kwargs
-                    )
+                    self._output_process = Popen(['bidiv'] + width_args, **sp_kwargs)
                 except OSError:
-                    self._output_process = subprocess.Popen(
-                        ['fribidi', '-c', 'UTF-8'] + width_args, **sp_kwargs)
+                    self._output_process = Popen(['fribidi', '-c', 'UTF-8'] + width_args, **sp_kwargs)
                 self._output_channel = os.fdopen(master, 'rb')
             except OSError as ose:
                 if ose.errno == errno.ENOENT:
-                    self.report_warning('Could not find fribidi executable, ignoring --bidi-workaround . Make sure that  fribidi  is an executable file in one of the directories in your $PATH.')
+                    self.report_warning(
+                        'Could not find fribidi executable, ignoring --bidi-workaround. '
+                        'Make sure that  fribidi  is an executable file in one of the directories in your $PATH.')
                 else:
                     raise
 
@@ -600,26 +609,9 @@ class YoutubeDL(object):
 
         self._setup_opener()
 
-        def preload_download_archive(fn):
-            """Preload the archive, if any is specified"""
-            if fn is None:
-                return False
-            self.write_debug('Loading archive file %r\n' % fn)
-            try:
-                with locked_file(fn, 'r', encoding='utf-8') as archive_file:
-                    for line in archive_file:
-                        self.archive.add(line.strip())
-            except IOError as ioe:
-                if ioe.errno != errno.ENOENT:
-                    raise
-                return False
-            return True
-
-        self.archive = set()
-        preload_download_archive(self.params.get('download_archive'))
-
         if auto_init:
-            self.print_debug_header()
+            if auto_init != 'no_verbose_header':
+                self.print_debug_header()
             self.add_default_info_extractors()
 
         for pp_def_raw in self.params.get('postprocessors', []):
@@ -637,6 +629,24 @@ class YoutubeDL(object):
 
         register_socks_protocols()
 
+        def preload_download_archive(fn):
+            """Preload the archive, if any is specified"""
+            if fn is None:
+                return False
+            self.write_debug(f'Loading archive file {fn!r}')
+            try:
+                with locked_file(fn, 'r', encoding='utf-8') as archive_file:
+                    for line in archive_file:
+                        self.archive.add(line.strip())
+            except IOError as ioe:
+                if ioe.errno != errno.ENOENT:
+                    raise
+                return False
+            return True
+
+        self.archive = set()
+        preload_download_archive(self.params.get('download_archive'))
+
     def warn_if_short_id(self, argv):
         # short YouTube ID starting with dash?
         idxs = [
@@ -650,7 +660,7 @@ class YoutubeDL(object):
             )
             self.report_warning(
                 'Long argument string detected. '
-                'Use -- to separate parameters and URLs, like this:\n%s\n' %
+                'Use -- to separate parameters and URLs, like this:\n%s' %
                 args_to_str(correct_argv))
 
     def add_info_extractor(self, ie):
@@ -878,8 +888,13 @@ class YoutubeDL(object):
         outtmpl_dict = self.params.get('outtmpl', {})
         if not isinstance(outtmpl_dict, dict):
             outtmpl_dict = {'default': outtmpl_dict}
+        # Remove spaces in the default template
+        if self.params.get('restrictfilenames'):
+            sanitize = lambda x: x.replace(' - ', ' ').replace(' ', '-')
+        else:
+            sanitize = lambda x: x
         outtmpl_dict.update({
-            k: v for k, v in DEFAULT_OUTTMPL.items()
+            k: sanitize(v) for k, v in DEFAULT_OUTTMPL.items()
             if outtmpl_dict.get(k) is None})
         for key, val in outtmpl_dict.items():
             if isinstance(val, bytes):
@@ -939,13 +954,18 @@ class YoutubeDL(object):
         except ValueError as err:
             return err
 
+    @staticmethod
+    def _copy_infodict(info_dict):
+        info_dict = dict(info_dict)
+        for key in ('__original_infodict', '__postprocessors'):
+            info_dict.pop(key, None)
+        return info_dict
+
     def prepare_outtmpl(self, outtmpl, info_dict, sanitize=None):
         """ Make the outtmpl and info_dict suitable for substitution: ydl.escape_outtmpl(outtmpl) % info_dict """
         info_dict.setdefault('epoch', int(time.time()))  # keep epoch consistent once set
 
-        info_dict = dict(info_dict)  # Do not sanitize so as not to consume LazyList
-        for key in ('__original_infodict', '__postprocessors'):
-            info_dict.pop(key, None)
+        info_dict = self._copy_infodict(info_dict)
         info_dict['duration_string'] = (  # %(duration>%H-%M-%S)s is wrong if duration > 24hrs
             formatSeconds(info_dict['duration'], '-' if sanitize else ':')
             if info_dict.get('duration', None) is not None
@@ -1033,7 +1053,7 @@ class YoutubeDL(object):
 
         def create_key(outer_mobj):
             if not outer_mobj.group('has_key'):
-                return f'%{outer_mobj.group(0)}'
+                return outer_mobj.group(0)
             key = outer_mobj.group('key')
             mobj = re.match(INTERNAL_FORMAT_RE, key)
             initial_field = mobj.group('fields').split('.')[-1] if mobj else ''
@@ -1104,10 +1124,8 @@ class YoutubeDL(object):
                 compat_str(v),
                 restricted=self.params.get('restrictfilenames'),
                 is_id=(k == 'id' or k.endswith('_id')))
-            outtmpl = self.outtmpl_dict.get(tmpl_type, self.outtmpl_dict['default'])
-            outtmpl, template_dict = self.prepare_outtmpl(outtmpl, info_dict, sanitize)
-            outtmpl = self.escape_outtmpl(self._outtmpl_expandpath(outtmpl))
-            filename = outtmpl % template_dict
+            outtmpl = self._outtmpl_expandpath(self.outtmpl_dict.get(tmpl_type, self.outtmpl_dict['default']))
+            filename = self.evaluate_outtmpl(outtmpl, info_dict, sanitize)
 
             force_ext = OUTTMPL_TYPES.get(tmpl_type)
             if filename and force_ext is not None:
@@ -1534,7 +1552,7 @@ class YoutubeDL(object):
             playlistitems = list(range(playliststart, playliststart + n_entries))
         ie_result['requested_entries'] = playlistitems
 
-        if self.params.get('allow_playlist_files', True):
+        if not self.params.get('simulate') and self.params.get('allow_playlist_files', True):
             ie_copy = {
                 'playlist': playlist,
                 'playlist_id': ie_result.get('id'),
@@ -1542,6 +1560,7 @@ class YoutubeDL(object):
                 'playlist_uploader': ie_result.get('uploader'),
                 'playlist_uploader_id': ie_result.get('uploader_id'),
                 'playlist_index': 0,
+                'n_entries': n_entries,
             }
             ie_copy.update(dict(ie_result))
 
@@ -1847,11 +1866,18 @@ class YoutubeDL(object):
                 else:
                     output_ext = 'mkv'
 
+            filtered = lambda *keys: filter(None, (traverse_obj(fmt, *keys) for fmt in formats_info))
+
             new_dict = {
                 'requested_formats': formats_info,
-                'format': '+'.join(fmt_info.get('format') for fmt_info in formats_info),
-                'format_id': '+'.join(fmt_info.get('format_id') for fmt_info in formats_info),
+                'format': '+'.join(filtered('format')),
+                'format_id': '+'.join(filtered('format_id')),
                 'ext': output_ext,
+                'protocol': '+'.join(map(determine_protocol, formats_info)),
+                'language': '+'.join(orderedSet(filtered('language'))),
+                'format_note': '+'.join(orderedSet(filtered('format_note'))),
+                'filesize_approx': sum(filtered('filesize', 'filesize_approx')),
+                'tbr': sum(filtered('tbr', 'vbr', 'abr')),
             }
 
             if the_only_video:
@@ -1860,6 +1886,7 @@ class YoutubeDL(object):
                     'height': the_only_video.get('height'),
                     'resolution': the_only_video.get('resolution') or self.format_resolution(the_only_video),
                     'fps': the_only_video.get('fps'),
+                    'dynamic_range': the_only_video.get('dynamic_range'),
                     'vcodec': the_only_video.get('vcodec'),
                     'vbr': the_only_video.get('vbr'),
                     'stretched_ratio': the_only_video.get('stretched_ratio'),
@@ -1869,6 +1896,7 @@ class YoutubeDL(object):
                 new_dict.update({
                     'acodec': the_only_audio.get('acodec'),
                     'abr': the_only_audio.get('abr'),
+                    'asr': the_only_audio.get('asr'),
                 })
 
             return new_dict
@@ -1969,11 +1997,11 @@ class YoutubeDL(object):
                         filter_f = lambda f: _filter_f(f) and (
                             f.get('vcodec') != 'none' or f.get('acodec') != 'none')
                     else:
-                        if format_spec in ('m4a', 'mp3', 'ogg', 'aac'):  # audio extension
+                        if format_spec in self._format_selection_exts['audio']:
                             filter_f = lambda f: f.get('ext') == format_spec and f.get('acodec') != 'none'
-                        elif format_spec in ('mp4', 'flv', 'webm', '3gp'):  # video extension
+                        elif format_spec in self._format_selection_exts['video']:
                             filter_f = lambda f: f.get('ext') == format_spec and f.get('acodec') != 'none' and f.get('vcodec') != 'none'
-                        elif format_spec in ('mhtml', ):  # storyboards extension
+                        elif format_spec in self._format_selection_exts['storyboards']:
                             filter_f = lambda f: f.get('ext') == format_spec and f.get('acodec') == 'none' and f.get('vcodec') == 'none'
                         else:
                             filter_f = lambda f: f.get('format_id') == format_spec  # id
@@ -2068,25 +2096,14 @@ class YoutubeDL(object):
                 t.get('url')))
 
             def thumbnail_tester():
-                if self.params.get('check_formats'):
-                    test_all = True
-                    to_screen = lambda msg: self.to_screen(f'[info] {msg}')
-                else:
-                    test_all = False
-                    to_screen = self.write_debug
-
                 def test_thumbnail(t):
-                    if not test_all and not t.get('_test_url'):
-                        return True
-                    to_screen('Testing thumbnail %s' % t['id'])
+                    self.to_screen(f'[info] Testing thumbnail {t["id"]}')
                     try:
                         self.urlopen(HEADRequest(t['url']))
                     except network_exceptions as err:
-                        to_screen('Unable to connect to thumbnail %s URL "%s" - %s. Skipping...' % (
-                            t['id'], t['url'], error_to_compat_str(err)))
+                        self.to_screen(f'[info] Unable to connect to thumbnail {t["id"]} URL {t["url"]!r} - {err}. Skipping...')
                         return False
                     return True
-
                 return test_thumbnail
 
             for i, t in enumerate(thumbnails):
@@ -2096,7 +2113,7 @@ class YoutubeDL(object):
                     t['resolution'] = '%dx%d' % (t['width'], t['height'])
                 t['url'] = sanitize_url(t['url'])
 
-            if self.params.get('check_formats') is not False:
+            if self.params.get('check_formats'):
                 info_dict['thumbnails'] = LazyList(filter(thumbnail_tester(), thumbnails[::-1])).reverse()
             else:
                 info_dict['thumbnails'] = thumbnails
@@ -2150,6 +2167,9 @@ class YoutubeDL(object):
         if info_dict.get('display_id') is None and 'id' in info_dict:
             info_dict['display_id'] = info_dict['id']
 
+        if info_dict.get('duration') is not None:
+            info_dict['duration_string'] = formatSeconds(info_dict['duration'])
+
         for ts_key, date_key in (
                 ('timestamp', 'upload_date'),
                 ('release_timestamp', 'release_date'),
@@ -2248,10 +2268,18 @@ class YoutubeDL(object):
             formats_dict[format_id].append(format)
 
         # Make sure all formats have unique format_id
+        common_exts = set(itertools.chain(*self._format_selection_exts.values()))
         for format_id, ambiguous_formats in formats_dict.items():
-            if len(ambiguous_formats) > 1:
-                for i, format in enumerate(ambiguous_formats):
+            ambigious_id = len(ambiguous_formats) > 1
+            for i, format in enumerate(ambiguous_formats):
+                if ambigious_id:
                     format['format_id'] = '%s-%d' % (format_id, i)
+                if format.get('ext') is None:
+                    format['ext'] = determine_ext(format['url']).lower()
+                # Ensure there is no conflict between id and ext in format selection
+                # See https://github.com/yt-dlp/yt-dlp/issues/1282
+                if format['format_id'] != format['ext'] and format['format_id'] in common_exts:
+                    format['format_id'] = 'f%s' % format['format_id']
 
         for i, format in enumerate(formats):
             if format.get('format') is None:
@@ -2260,13 +2288,12 @@ class YoutubeDL(object):
                     res=self.format_resolution(format),
                     note=format_field(format, 'format_note', ' (%s)'),
                 )
-            # Automatically determine file extension if missing
-            if format.get('ext') is None:
-                format['ext'] = determine_ext(format['url']).lower()
-            # Automatically determine protocol if missing (useful for format
-            # selection purposes)
             if format.get('protocol') is None:
                 format['protocol'] = determine_protocol(format)
+            if format.get('resolution') is None:
+                format['resolution'] = self.format_resolution(format, default=None)
+            if format.get('dynamic_range') is None and format.get('vcodec') != 'none':
+                format['dynamic_range'] = 'SDR'
             # Add HTTP headers, so that external programs can use them from the
             # json output
             full_format_info = info_dict.copy()
@@ -2358,7 +2385,7 @@ class YoutubeDL(object):
                 new_info['__original_infodict'] = info_dict
                 new_info.update(fmt)
                 self.process_info(new_info)
-        # We update the info dict with the best quality format (backwards compatibility)
+        # We update the info dict with the selected best quality format (backwards compatibility)
         if formats_to_download:
             info_dict.update(formats_to_download[-1])
         return info_dict
@@ -2484,7 +2511,7 @@ class YoutubeDL(object):
             verbose = self.params.get('verbose')
             params = {
                 'test': True,
-                'quiet': not verbose,
+                'quiet': self.params.get('quiet') or not verbose,
                 'verbose': verbose,
                 'noprogress': not verbose,
                 'nopart': True,
@@ -2501,7 +2528,8 @@ class YoutubeDL(object):
                 fd.add_progress_hook(ph)
             urls = '", "'.join([f['url'] for f in info.get('requested_formats', [])] or [info['url']])
             self.write_debug('Invoking downloader on "%s"' % urls)
-        new_info = dict(info)
+
+        new_info = copy.deepcopy(self._copy_infodict(info))
         if new_info.get('http_headers') is None:
             new_info['http_headers'] = self._calc_headers(new_info)
         return fd.download(name, new_info, subtitle)
@@ -2735,14 +2763,9 @@ class YoutubeDL(object):
                     dl_filename = existing_file(full_filename, temp_filename)
                     info_dict['__real_download'] = False
 
-                    _protocols = set(determine_protocol(f) for f in requested_formats)
-                    if len(_protocols) == 1:  # All requested formats have same protocol
-                        info_dict['protocol'] = _protocols.pop()
-                    directly_mergable = FFmpegFD.can_merge_formats(info_dict, self.params)
                     if dl_filename is not None:
                         self.report_file_already_downloaded(dl_filename)
-                    elif (directly_mergable and get_suitable_downloader(
-                            info_dict, self.params, to_stdout=(temp_filename == '-')) == FFmpegFD):
+                    elif get_suitable_downloader(info_dict, self.params, to_stdout=temp_filename == '-'):
                         info_dict['url'] = '\n'.join(f['url'] for f in requested_formats)
                         success, real_download = self.dl(temp_filename, info_dict)
                         info_dict['__real_download'] = real_download
@@ -2760,7 +2783,7 @@ class YoutubeDL(object):
                                 'The formats won\'t be merged.')
 
                         if temp_filename == '-':
-                            reason = ('using a downloader other than ffmpeg' if directly_mergable
+                            reason = ('using a downloader other than ffmpeg' if FFmpegFD.can_merge_formats(info_dict)
                                       else 'but the formats are incompatible for simultaneous download' if merger.available
                                       else 'but ffmpeg is not installed')
                             self.report_warning(
@@ -2854,8 +2877,8 @@ class YoutubeDL(object):
                         'writing DASH m4a. Only some players support this container',
                         FFmpegFixupM4aPP)
 
-                    downloader = (get_suitable_downloader(info_dict, self.params).__name__
-                                  if 'protocol' in info_dict else None)
+                    downloader = get_suitable_downloader(info_dict, self.params) if 'protocol' in info_dict else None
+                    downloader = downloader.__name__ if downloader else None
                     ffmpeg_fixup(info_dict.get('requested_formats') is None and downloader == 'HlsFD',
                                  'malformed AAC bitstream detected', FFmpegFixupM3u8PP)
                     ffmpeg_fixup(downloader == 'WebSocketFragmentFD', 'malformed timestamps detected', FFmpegFixupTimestampPP)
@@ -3071,6 +3094,7 @@ class YoutubeDL(object):
 
     @staticmethod
     def format_resolution(format, default='unknown'):
+        is_images = format.get('vcodec') == 'none' and format.get('acodec') == 'none'
         if format.get('vcodec') == 'none' and format.get('acodec') != 'none':
             return 'audio only'
         if format.get('resolution') is not None:
@@ -3081,11 +3105,11 @@ class YoutubeDL(object):
             res = '%sp' % format['height']
         elif format.get('width'):
             res = '%dx?' % format['width']
+        elif is_images:
+            return 'images'
         else:
-            res = default
-        if format.get('vcodec') == 'none' and format.get('acodec') == 'none':
-            res += ' (images)'
-        return res
+            return default
+        return f'{res} images' if is_images else res
 
     def _format_note(self, fdict):
         res = ''
@@ -3155,6 +3179,7 @@ class YoutubeDL(object):
                     format_field(f, 'ext'),
                     self.format_resolution(f),
                     format_field(f, 'fps', '%d'),
+                    format_field(f, 'dynamic_range', '%s', ignore=(None, 'SDR')).replace('HDR', ''),
                     '|',
                     format_field(f, 'filesize', ' %s', func=format_bytes) + format_field(f, 'filesize_approx', '~%s', func=format_bytes),
                     format_field(f, 'tbr', '%4dk'),
@@ -3172,7 +3197,7 @@ class YoutubeDL(object):
                         format_field(f, 'container', ignore=(None, f.get('ext'))),
                     ))),
                 ] for f in formats if f.get('preference') is None or f['preference'] >= -1000]
-            header_line = ['ID', 'EXT', 'RESOLUTION', 'FPS', '|', ' FILESIZE', '  TBR', 'PROTO',
+            header_line = ['ID', 'EXT', 'RESOLUTION', 'FPS', 'HDR', '|', ' FILESIZE', '  TBR', 'PROTO',
                            '|', 'VCODEC', '  VBR', 'ACODEC', ' ABR', ' ASR', 'MORE INFO']
         else:
             table = [
@@ -3230,36 +3255,48 @@ class YoutubeDL(object):
         if not self.params.get('verbose'):
             return
 
-        stdout_encoding = getattr(
-            sys.stdout, 'encoding', 'missing (%s)' % type(sys.stdout).__name__)
-        encoding_str = (
-            '[debug] Encodings: locale %s, fs %s, out %s, pref %s\n' % (
-                locale.getpreferredencoding(),
-                sys.getfilesystemencoding(),
-                stdout_encoding,
-                self.get_encoding()))
-        write_string(encoding_str, encoding=None)
+        def get_encoding(stream):
+            ret = getattr(stream, 'encoding', 'missing (%s)' % type(stream).__name__)
+            if not supports_terminal_sequences(stream):
+                ret += ' (No ANSI)'
+            return ret
+
+        encoding_str = 'Encodings: locale %s, fs %s, out %s, err %s, pref %s' % (
+            locale.getpreferredencoding(),
+            sys.getfilesystemencoding(),
+            get_encoding(self._screen_file), get_encoding(self._err_file),
+            self.get_encoding())
+
+        logger = self.params.get('logger')
+        if logger:
+            write_debug = lambda msg: logger.debug(f'[debug] {msg}')
+            write_debug(encoding_str)
+        else:
+            write_string(f'[debug] {encoding_str}', encoding=None)
+            write_debug = lambda msg: self._write_string(f'[debug] {msg}\n')
 
         source = detect_variant()
-        self._write_string('[debug] yt-dlp version %s%s\n' % (__version__, '' if source == 'unknown' else f' ({source})'))
-        if _LAZY_LOADER:
-            self._write_string('[debug] Lazy loading extractors enabled\n')
+        write_debug('yt-dlp version %s%s' % (__version__, '' if source == 'unknown' else f' ({source})'))
+        if not _LAZY_LOADER:
+            if os.environ.get('YTDLP_NO_LAZY_EXTRACTORS'):
+                write_debug('Lazy loading extractors is forcibly disabled')
+            else:
+                write_debug('Lazy loading extractors is disabled')
         if plugin_extractors or plugin_postprocessors:
-            self._write_string('[debug] Plugins: %s\n' % [
+            write_debug('Plugins: %s' % [
                 '%s%s' % (klass.__name__, '' if klass.__name__ == name else f' as {name}')
                 for name, klass in itertools.chain(plugin_extractors.items(), plugin_postprocessors.items())])
         if self.params.get('compat_opts'):
-            self._write_string(
-                '[debug] Compatibility options: %s\n' % ', '.join(self.params.get('compat_opts')))
+            write_debug('Compatibility options: %s' % ', '.join(self.params.get('compat_opts')))
         try:
-            sp = subprocess.Popen(
+            sp = Popen(
                 ['git', 'rev-parse', '--short', 'HEAD'],
                 stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                 cwd=os.path.dirname(os.path.abspath(__file__)))
-            out, err = process_communicate_or_kill(sp)
+            out, err = sp.communicate_or_kill()
             out = out.decode().strip()
             if re.match('[0-9a-f]+', out):
-                self._write_string('[debug] Git HEAD: %s\n' % out)
+                write_debug('Git HEAD: %s' % out)
         except Exception:
             try:
                 sys.exc_clear()
@@ -3272,7 +3309,7 @@ class YoutubeDL(object):
                 return impl_name + ' version %d.%d.%d' % sys.pypy_version_info[:3]
             return impl_name
 
-        self._write_string('[debug] Python version %s (%s %s) - %s\n' % (
+        write_debug('Python version %s (%s %s) - %s' % (
             platform.python_version(),
             python_implementation(),
             platform.architecture()[0],
@@ -3284,7 +3321,7 @@ class YoutubeDL(object):
         exe_str = ', '.join(
             f'{exe} {v}' for exe, v in sorted(exe_versions.items()) if v
         ) or 'none'
-        self._write_string('[debug] exe versions: %s\n' % exe_str)
+        write_debug('exe versions: %s' % exe_str)
 
         from .downloader.websocket import has_websockets
         from .postprocessor.embedthumbnail import has_mutagen
@@ -3297,18 +3334,18 @@ class YoutubeDL(object):
             SQLITE_AVAILABLE and 'sqlite',
             KEYRING_AVAILABLE and 'keyring',
         )))) or 'none'
-        self._write_string('[debug] Optional libraries: %s\n' % lib_str)
+        write_debug('Optional libraries: %s' % lib_str)
 
         proxy_map = {}
         for handler in self._opener.handlers:
             if hasattr(handler, 'proxies'):
                 proxy_map.update(handler.proxies)
-        self._write_string('[debug] Proxy map: ' + compat_str(proxy_map) + '\n')
+        write_debug(f'Proxy map: {proxy_map}')
 
-        if self.params.get('call_home', False):
+        # Not implemented
+        if False and self.params.get('call_home'):
             ipaddr = self.urlopen('https://yt-dl.org/ip').read().decode('utf-8')
-            self._write_string('[debug] Public IP address: %s\n' % ipaddr)
-            return
+            write_debug('Public IP address: %s' % ipaddr)
             latest_version = self.urlopen(
                 'https://yt-dl.org/latest/version').read().decode('utf-8')
             if version_tuple(latest_version) > version_tuple(__version__):
@@ -3319,7 +3356,7 @@ class YoutubeDL(object):
 
     def _setup_opener(self):
         timeout_val = self.params.get('socket_timeout')
-        self._socket_timeout = 600 if timeout_val is None else float(timeout_val)
+        self._socket_timeout = 20 if timeout_val is None else float(timeout_val)
 
         opts_cookiesfrombrowser = self.params.get('cookiesfrombrowser')
         opts_cookiefile = self.params.get('cookiefile')

yt_dlp/__init__.py

@@ -31,6 +31,7 @@ from .utils import (
     expand_path,
     match_filter_func,
     MaxDownloadsReached,
+    parse_duration,
     preferredencoding,
     read_batch_urls,
     RejectedVideoReached,
@@ -258,6 +259,9 @@ def _real_main(argv=None):
 
     compat_opts = opts.compat_opts
 
+    def report_conflict(arg1, arg2):
+        warnings.append(f'{arg2} is ignored since {arg1} was given')
+
     def _unused_compat_opt(name):
         if name not in compat_opts:
             return False
@@ -289,10 +293,14 @@ def _real_main(argv=None):
     if _video_multistreams_set is False and _audio_multistreams_set is False:
         _unused_compat_opt('multistreams')
     outtmpl_default = opts.outtmpl.get('default')
+    if opts.useid:
+        if outtmpl_default is None:
+            outtmpl_default = opts.outtmpl['default'] = '%(id)s.%(ext)s'
+        else:
+            report_conflict('--output', '--id')
     if 'filename' in compat_opts:
         if outtmpl_default is None:
-            outtmpl_default = '%(title)s-%(id)s.%(ext)s'
-            opts.outtmpl.update({'default': outtmpl_default})
+            outtmpl_default = opts.outtmpl['default'] = '%(title)s-%(id)s.%(ext)s'
         else:
             _unused_compat_opt('filename')
 
@@ -365,9 +373,6 @@ def _real_main(argv=None):
         opts.addchapters = True
     opts.remove_chapters = opts.remove_chapters or []
 
-    def report_conflict(arg1, arg2):
-        warnings.append('%s is ignored since %s was given' % (arg2, arg1))
-
     if (opts.remove_chapters or sponsorblock_query) and opts.sponskrub is not False:
         if opts.sponskrub:
             if opts.remove_chapters:
@@ -490,8 +495,14 @@ def _real_main(argv=None):
     if opts.allsubtitles and not opts.writeautomaticsub:
         opts.writesubtitles = True
     # ModifyChapters must run before FFmpegMetadataPP
-    remove_chapters_patterns = []
+    remove_chapters_patterns, remove_ranges = [], []
     for regex in opts.remove_chapters:
+        if regex.startswith('*'):
+            dur = list(map(parse_duration, regex[1:].split('-')))
+            if len(dur) == 2 and all(t is not None for t in dur):
+                remove_ranges.append(tuple(dur))
+                continue
+            parser.error(f'invalid --remove-chapters time range {regex!r}. Must be of the form ?start-end')
         try:
             remove_chapters_patterns.append(re.compile(regex))
         except re.error as err:
@@ -501,6 +512,7 @@ def _real_main(argv=None):
             'key': 'ModifyChapters',
             'remove_chapters_patterns': remove_chapters_patterns,
             'remove_sponsor_segments': opts.sponsorblock_remove,
+            'remove_ranges': remove_ranges,
             'sponsorblock_chapter_title': opts.sponsorblock_chapter_title,
             'force_keyframes': opts.force_keyframes_at_cuts
         })
@@ -733,7 +745,7 @@ def _real_main(argv=None):
         'geo_bypass': opts.geo_bypass,
         'geo_bypass_country': opts.geo_bypass_country,
         'geo_bypass_ip_block': opts.geo_bypass_ip_block,
-        'warnings': warnings,
+        '_warnings': warnings,
         'compat_opts': compat_opts,
     }
 

yt_dlp/cookies.py

@@ -17,7 +17,7 @@ from .compat import (
 from .utils import (
     bug_reports_message,
     expand_path,
-    process_communicate_or_kill,
+    Popen,
     YoutubeDLCookieJar,
 )
 
@@ -599,14 +599,14 @@ def _get_mac_keyring_password(browser_keyring_name, logger):
         return password.encode('utf-8')
     else:
         logger.debug('using find-generic-password to obtain password')
-        proc = subprocess.Popen(['security', 'find-generic-password',
-                                 '-w',  # write password to stdout
-                                 '-a', browser_keyring_name,  # match 'account'
-                                 '-s', '{} Safe Storage'.format(browser_keyring_name)],  # match 'service'
-                                stdout=subprocess.PIPE,
-                                stderr=subprocess.DEVNULL)
+        proc = Popen(
+            ['security', 'find-generic-password',
+             '-w',  # write password to stdout
+             '-a', browser_keyring_name,  # match 'account'
+             '-s', '{} Safe Storage'.format(browser_keyring_name)],  # match 'service'
+            stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
         try:
-            stdout, stderr = process_communicate_or_kill(proc)
+            stdout, stderr = proc.communicate_or_kill()
             if stdout[-1:] == b'\n':
                 stdout = stdout[:-1]
             return stdout
@@ -620,7 +620,7 @@ def _get_windows_v10_key(browser_root, logger):
     if path is None:
         logger.error('could not find local state file')
         return None
-    with open(path, 'r') as f:
+    with open(path, 'r', encoding='utf8') as f:
         data = json.load(f)
     try:
         base64_key = data['os_crypt']['encrypted_key']

yt_dlp/downloader/__init__.py

@@ -10,10 +10,15 @@ from ..utils import (
 def get_suitable_downloader(info_dict, params={}, default=NO_DEFAULT, protocol=None, to_stdout=False):
     info_dict['protocol'] = determine_protocol(info_dict)
     info_copy = info_dict.copy()
-    if protocol:
-        info_copy['protocol'] = protocol
     info_copy['to_stdout'] = to_stdout
-    return _get_suitable_downloader(info_copy, params, default)
+
+    downloaders = [_get_suitable_downloader(info_copy, proto, params, default)
+                   for proto in (protocol or info_copy['protocol']).split('+')]
+    if set(downloaders) == {FFmpegFD} and FFmpegFD.can_merge_formats(info_copy, params):
+        return FFmpegFD
+    elif len(downloaders) == 1:
+        return downloaders[0]
+    return None
 
 
 # Some of these require get_suitable_downloader
@@ -72,7 +77,7 @@ def shorten_protocol_name(proto, simplify=False):
     return short_protocol_names.get(proto, proto)
 
 
-def _get_suitable_downloader(info_dict, params, default):
+def _get_suitable_downloader(info_dict, protocol, params, default):
     """Get the downloader class that can handle the info dict."""
     if default is NO_DEFAULT:
         default = HttpFD
@@ -80,7 +85,7 @@ def _get_suitable_downloader(info_dict, params, default):
     # if (info_dict.get('start_time') or info_dict.get('end_time')) and not info_dict.get('requested_formats') and FFmpegFD.can_download(info_dict):
     #     return FFmpegFD
 
-    protocol = info_dict['protocol']
+    info_dict['protocol'] = protocol
     downloaders = params.get('external_downloader')
     external_downloader = (
         downloaders if isinstance(downloaders, compat_str) or downloaders is None

yt_dlp/downloader/common.py

@@ -1,9 +1,7 @@
 from __future__ import division, unicode_literals
 
-import copy
 import os
 import re
-import sys
 import time
 import random
 
@@ -14,6 +12,7 @@ from ..utils import (
     format_bytes,
     shell_quote,
     timeconvert,
+    timetuple_from_msec,
 )
 from ..minicurses import (
     MultilineLogger,
@@ -77,14 +76,12 @@ class FileDownloader(object):
 
     @staticmethod
     def format_seconds(seconds):
-        (mins, secs) = divmod(seconds, 60)
-        (hours, mins) = divmod(mins, 60)
-        if hours > 99:
+        time = timetuple_from_msec(seconds * 1000)
+        if time.hours > 99:
             return '--:--:--'
-        if hours == 0:
-            return '%02d:%02d' % (mins, secs)
-        else:
-            return '%02d:%02d:%02d' % (hours, mins, secs)
+        if not time.hours:
+            return '%02d:%02d' % time[1:-1]
+        return '%02d:%02d:%02d' % time[:-1]
 
     @staticmethod
     def calc_percent(byte_counter, data_len):
@@ -247,9 +244,9 @@ class FileDownloader(object):
         elif self.ydl.params.get('logger'):
             self._multiline = MultilineLogger(self.ydl.params['logger'], lines)
         elif self.params.get('progress_with_newline'):
-            self._multiline = BreaklineStatusPrinter(sys.stderr, lines)
+            self._multiline = BreaklineStatusPrinter(self.ydl._screen_file, lines)
         else:
-            self._multiline = MultilinePrinter(sys.stderr, lines, not self.params.get('quiet'))
+            self._multiline = MultilinePrinter(self.ydl._screen_file, lines, not self.params.get('quiet'))
 
     def _finish_multiline_status(self):
         self._multiline.end()
@@ -406,13 +403,10 @@ class FileDownloader(object):
     def _hook_progress(self, status, info_dict):
         if not self._progress_hooks:
             return
-        info_dict = dict(info_dict)
-        for key in ('__original_infodict', '__postprocessors'):
-            info_dict.pop(key, None)
+        status['info_dict'] = info_dict
         # youtube-dl passes the same status object to all the hooks.
         # Some third party scripts seems to be relying on this.
         # So keep this behavior if possible
-        status['info_dict'] = copy.deepcopy(info_dict)
         for ph in self._progress_hooks:
             ph(status)
 

yt_dlp/downloader/dash.py

@@ -55,9 +55,8 @@ class DashSegmentsFD(FragmentFD):
         if real_downloader:
             self.to_screen(
                 '[%s] Fragment downloads will be delegated to %s' % (self.FD_NAME, real_downloader.get_basename()))
-            info_copy = info_dict.copy()
-            info_copy['fragments'] = fragments_to_download
+            info_dict['fragments'] = fragments_to_download
             fd = real_downloader(self.ydl, self.params)
-            return fd.real_download(filename, info_copy)
+            return fd.real_download(filename, info_dict)
 
         return self.download_and_append_fragments(ctx, fragments_to_download, info_dict)

yt_dlp/downloader/external.py

@@ -22,7 +22,7 @@ from ..utils import (
     handle_youtubedl_headers,
     check_executable,
     is_outdated_version,
-    process_communicate_or_kill,
+    Popen,
     sanitize_open,
 )
 
@@ -115,55 +115,54 @@ class ExternalFD(FragmentFD):
 
         self._debug_cmd(cmd)
 
-        if 'fragments' in info_dict:
-            fragment_retries = self.params.get('fragment_retries', 0)
-            skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
-
-            count = 0
-            while count <= fragment_retries:
-                p = subprocess.Popen(
-                    cmd, stderr=subprocess.PIPE)
-                _, stderr = process_communicate_or_kill(p)
-                if p.returncode == 0:
-                    break
-                # TODO: Decide whether to retry based on error code
-                # https://aria2.github.io/manual/en/html/aria2c.html#exit-status
-                self.to_stderr(stderr.decode('utf-8', 'replace'))
-                count += 1
-                if count <= fragment_retries:
-                    self.to_screen(
-                        '[%s] Got error. Retrying fragments (attempt %d of %s)...'
-                        % (self.get_basename(), count, self.format_retries(fragment_retries)))
-            if count > fragment_retries:
-                if not skip_unavailable_fragments:
-                    self.report_error('Giving up after %s fragment retries' % fragment_retries)
-                    return -1
-
-            decrypt_fragment = self.decrypter(info_dict)
-            dest, _ = sanitize_open(tmpfilename, 'wb')
-            for frag_index, fragment in enumerate(info_dict['fragments']):
-                fragment_filename = '%s-Frag%d' % (tmpfilename, frag_index)
-                try:
-                    src, _ = sanitize_open(fragment_filename, 'rb')
-                except IOError:
-                    if skip_unavailable_fragments and frag_index > 1:
-                        self.to_screen('[%s] Skipping fragment %d ...' % (self.get_basename(), frag_index))
-                        continue
-                    self.report_error('Unable to open fragment %d' % frag_index)
-                    return -1
-                dest.write(decrypt_fragment(fragment, src.read()))
-                src.close()
-                if not self.params.get('keep_fragments', False):
-                    os.remove(encodeFilename(fragment_filename))
-            dest.close()
-            os.remove(encodeFilename('%s.frag.urls' % tmpfilename))
-        else:
-            p = subprocess.Popen(
-                cmd, stderr=subprocess.PIPE)
-            _, stderr = process_communicate_or_kill(p)
+        if 'fragments' not in info_dict:
+            p = Popen(cmd, stderr=subprocess.PIPE)
+            _, stderr = p.communicate_or_kill()
             if p.returncode != 0:
                 self.to_stderr(stderr.decode('utf-8', 'replace'))
-        return p.returncode
+            return p.returncode
+
+        fragment_retries = self.params.get('fragment_retries', 0)
+        skip_unavailable_fragments = self.params.get('skip_unavailable_fragments', True)
+
+        count = 0
+        while count <= fragment_retries:
+            p = Popen(cmd, stderr=subprocess.PIPE)
+            _, stderr = p.communicate_or_kill()
+            if p.returncode == 0:
+                break
+            # TODO: Decide whether to retry based on error code
+            # https://aria2.github.io/manual/en/html/aria2c.html#exit-status
+            self.to_stderr(stderr.decode('utf-8', 'replace'))
+            count += 1
+            if count <= fragment_retries:
+                self.to_screen(
+                    '[%s] Got error. Retrying fragments (attempt %d of %s)...'
+                    % (self.get_basename(), count, self.format_retries(fragment_retries)))
+        if count > fragment_retries:
+            if not skip_unavailable_fragments:
+                self.report_error('Giving up after %s fragment retries' % fragment_retries)
+                return -1
+
+        decrypt_fragment = self.decrypter(info_dict)
+        dest, _ = sanitize_open(tmpfilename, 'wb')
+        for frag_index, fragment in enumerate(info_dict['fragments']):
+            fragment_filename = '%s-Frag%d' % (tmpfilename, frag_index)
+            try:
+                src, _ = sanitize_open(fragment_filename, 'rb')
+            except IOError as err:
+                if skip_unavailable_fragments and frag_index > 1:
+                    self.report_skip_fragment(frag_index, err)
+                    continue
+                self.report_error(f'Unable to open fragment {frag_index}; {err}')
+                return -1
+            dest.write(decrypt_fragment(fragment, src.read()))
+            src.close()
+            if not self.params.get('keep_fragments', False):
+                os.remove(encodeFilename(fragment_filename))
+        dest.close()
+        os.remove(encodeFilename('%s.frag.urls' % tmpfilename))
+        return 0
 
 
 class CurlFD(ExternalFD):
@@ -198,8 +197,8 @@ class CurlFD(ExternalFD):
         self._debug_cmd(cmd)
 
         # curl writes the progress to stderr so don't capture it.
-        p = subprocess.Popen(cmd)
-        process_communicate_or_kill(p)
+        p = Popen(cmd)
+        p.communicate_or_kill()
         return p.returncode
 
 
@@ -327,6 +326,10 @@ class FFmpegFD(ExternalFD):
         # Fixme: This may be wrong when --ffmpeg-location is used
         return FFmpegPostProcessor().available
 
+    @classmethod
+    def supports(cls, info_dict):
+        return all(proto in cls.SUPPORTED_PROTOCOLS for proto in info_dict['protocol'].split('+'))
+
     def on_process_started(self, proc, stdin):
         """ Override this in subclasses  """
         pass
@@ -471,7 +474,7 @@ class FFmpegFD(ExternalFD):
         args.append(encodeFilename(ffpp._ffmpeg_filename_argument(tmpfilename), True))
         self._debug_cmd(args)
 
-        proc = subprocess.Popen(args, stdin=subprocess.PIPE, env=env)
+        proc = Popen(args, stdin=subprocess.PIPE, env=env)
         if url in ('-', 'pipe:'):
             self.on_process_started(proc, proc.stdin)
         try:
@@ -483,7 +486,7 @@ class FFmpegFD(ExternalFD):
             # streams). Note that Windows is not affected and produces playable
             # files (see https://github.com/ytdl-org/youtube-dl/issues/8300).
             if isinstance(e, KeyboardInterrupt) and sys.platform != 'win32' and url not in ('-', 'pipe:'):
-                process_communicate_or_kill(proc, b'q')
+                proc.communicate_or_kill(b'q')
             else:
                 proc.kill()
                 proc.wait()

yt_dlp/downloader/fragment.py

@@ -72,8 +72,9 @@ class FragmentFD(FileDownloader):
             '\r[download] Got server HTTP error: %s. Retrying fragment %d (attempt %d of %s) ...'
             % (error_to_compat_str(err), frag_index, count, self.format_retries(retries)))
 
-    def report_skip_fragment(self, frag_index):
-        self.to_screen('[download] Skipping fragment %d ...' % frag_index)
+    def report_skip_fragment(self, frag_index, err=None):
+        err = f' {err};' if err else ''
+        self.to_screen(f'[download]{err} Skipping fragment {frag_index:d} ...')
 
     def _prepare_url(self, info_dict, url):
         headers = info_dict.get('http_headers')
@@ -369,7 +370,8 @@ class FragmentFD(FileDownloader):
         if max_progress == 1:
             return self.download_and_append_fragments(*args[0], pack_func=pack_func, finish_func=finish_func)
         max_workers = self.params.get('concurrent_fragment_downloads', max_progress)
-        self._prepare_multiline_status(max_progress)
+        if max_progress > 1:
+            self._prepare_multiline_status(max_progress)
 
         def thread_func(idx, ctx, fragments, info_dict, tpe):
             ctx['max_progress'] = max_progress
@@ -443,7 +445,7 @@ class FragmentFD(FileDownloader):
         def append_fragment(frag_content, frag_index, ctx):
             if not frag_content:
                 if not is_fatal(frag_index - 1):
-                    self.report_skip_fragment(frag_index)
+                    self.report_skip_fragment(frag_index, 'fragment not found')
                     return True
                 else:
                     ctx['dest_stream'].close()

yt_dlp/downloader/hls.py

@@ -245,13 +245,12 @@ class HlsFD(FragmentFD):
             fragments = [fragments[0] if fragments else None]
 
         if real_downloader:
-            info_copy = info_dict.copy()
-            info_copy['fragments'] = fragments
+            info_dict['fragments'] = fragments
             fd = real_downloader(self.ydl, self.params)
             # TODO: Make progress updates work without hooking twice
             # for ph in self._progress_hooks:
             #     fd.add_progress_hook(ph)
-            return fd.real_download(filename, info_copy)
+            return fd.real_download(filename, info_dict)
 
         if is_webvtt:
             def pack_fragment(frag_content, frag_index):

yt_dlp/downloader/http.py

@@ -58,6 +58,7 @@ class HttpFD(FileDownloader):
         ctx.block_size = self.params.get('buffersize', 1024)
         ctx.start_time = time.time()
         ctx.chunk_size = None
+        throttle_start = None
 
         if self.params.get('continuedl', True):
             # Establish possible resume length
@@ -190,13 +191,16 @@ class HttpFD(FileDownloader):
                     # Unexpected HTTP error
                     raise
                 raise RetryDownload(err)
-            except socket.error as err:
-                if err.errno != errno.ECONNRESET:
-                    # Connection reset is no problem, just retry
-                    raise
+            except socket.timeout as err:
                 raise RetryDownload(err)
+            except socket.error as err:
+                if err.errno in (errno.ECONNRESET, errno.ETIMEDOUT):
+                    # Connection reset is no problem, just retry
+                    raise RetryDownload(err)
+                raise
 
         def download():
+            nonlocal throttle_start
             data_len = ctx.data.info().get('Content-length', None)
 
             # Range HTTP header may be ignored/unsupported by a webserver
@@ -225,7 +229,6 @@ class HttpFD(FileDownloader):
             # measure time over whole while-loop, so slow_down() and best_block_size() work together properly
             now = None  # needed for slow_down() in the first loop run
             before = start  # start measuring
-            throttle_start = None
 
             def retry(e):
                 to_stdout = ctx.tmpfilename == '-'
@@ -326,7 +329,7 @@ class HttpFD(FileDownloader):
                         if ctx.stream is not None and ctx.tmpfilename != '-':
                             ctx.stream.close()
                         raise ThrottledDownload()
-                else:
+                elif speed:
                     throttle_start = None
 
             if not is_test and ctx.chunk_size and ctx.data_len is not None and byte_counter < ctx.data_len:
@@ -372,6 +375,8 @@ class HttpFD(FileDownloader):
                 count += 1
                 if count <= retries:
                     self.report_retry(e.source_error, count, retries)
+                else:
+                    self.to_screen(f'[download] Got server HTTP error: {e.source_error}')
                 continue
             except NextFragment:
                 continue

yt_dlp/downloader/rtmp.py

@@ -12,6 +12,7 @@ from ..utils import (
     encodeFilename,
     encodeArgument,
     get_exe_version,
+    Popen,
 )
 
 
@@ -26,7 +27,7 @@ class RtmpFD(FileDownloader):
             start = time.time()
             resume_percent = None
             resume_downloaded_data_len = None
-            proc = subprocess.Popen(args, stderr=subprocess.PIPE)
+            proc = Popen(args, stderr=subprocess.PIPE)
             cursor_in_new_line = True
             proc_stderr_closed = False
             try:

yt_dlp/extractor/__init__.py

@@ -1,14 +1,15 @@
-from __future__ import unicode_literals
+import os
 
 from ..utils import load_plugins
 
-try:
-    from .lazy_extractors import *
-    from .lazy_extractors import _ALL_CLASSES
-    _LAZY_LOADER = True
-    _PLUGIN_CLASSES = {}
-except ImportError:
-    _LAZY_LOADER = False
+_LAZY_LOADER = False
+if not os.environ.get('YTDLP_NO_LAZY_EXTRACTORS'):
+    try:
+        from .lazy_extractors import *
+        from .lazy_extractors import _ALL_CLASSES
+        _LAZY_LOADER = True
+    except ImportError:
+        pass
 
 if not _LAZY_LOADER:
     from .extractors import *
@@ -19,8 +20,8 @@ if not _LAZY_LOADER:
     ]
     _ALL_CLASSES.append(GenericIE)
 
-    _PLUGIN_CLASSES = load_plugins('extractor', 'IE', globals())
-    _ALL_CLASSES = list(_PLUGIN_CLASSES.values()) + _ALL_CLASSES
+_PLUGIN_CLASSES = load_plugins('extractor', 'IE', globals())
+_ALL_CLASSES = list(_PLUGIN_CLASSES.values()) + _ALL_CLASSES
 
 
 def gen_extractor_classes():

yt_dlp/extractor/adn.py

@@ -15,6 +15,7 @@ from ..compat import (
     compat_ord,
 )
 from ..utils import (
+    ass_subtitles_timecode,
     bytes_to_intlist,
     bytes_to_long,
     ExtractorError,
@@ -68,10 +69,6 @@ class ADNIE(InfoExtractor):
         'end': 4,
     }
 
-    @staticmethod
-    def _ass_subtitles_timecode(seconds):
-        return '%01d:%02d:%02d.%02d' % (seconds / 3600, (seconds % 3600) / 60, seconds % 60, (seconds % 1) * 100)
-
     def _get_subtitles(self, sub_url, video_id):
         if not sub_url:
             return None
@@ -117,8 +114,8 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
                     continue
                 alignment = self._POS_ALIGN_MAP.get(position_align, 2) + self._LINE_ALIGN_MAP.get(line_align, 0)
                 ssa += os.linesep + 'Dialogue: Marked=0,%s,%s,Default,,0,0,0,,%s%s' % (
-                    self._ass_subtitles_timecode(start),
-                    self._ass_subtitles_timecode(end),
+                    ass_subtitles_timecode(start),
+                    ass_subtitles_timecode(end),
                     '{\\a%d}' % alignment if alignment != 2 else '',
                     text.replace('\n', '\\N').replace('<i>', '{\\i1}').replace('</i>', '{\\i0}'))
 

yt_dlp/extractor/adobepass.py

@@ -39,8 +39,8 @@ MSO_INFO = {
     },
     'RCN': {
         'name': 'RCN',
-        'username_field': 'UserName',
-        'password_field': 'UserPassword',
+        'username_field': 'username',
+        'password_field': 'password',
     },
     'Rogers': {
         'name': 'Rogers',

yt_dlp/extractor/bannedvideo.py

@@ -97,21 +97,16 @@ query GetCommentReplies($id: String!) {
                 'query': self._GRAPHQL_QUERIES[operation]
             }).encode('utf8')).get('data')
 
-    def _extract_comments(self, video_id, comments, comment_data):
+    def _get_comments(self, video_id, comments, comment_data):
+        yield from comments
         for comment in comment_data.copy():
             comment_id = comment.get('_id')
             if comment.get('replyCount') > 0:
                 reply_json = self._call_api(
                     video_id, comment_id, 'GetCommentReplies',
                     f'Downloading replies for comment {comment_id}')
-                comments.extend(
-                    self._parse_comment(reply, comment_id)
-                    for reply in reply_json.get('getCommentReplies'))
-
-        return {
-            'comments': comments,
-            'comment_count': len(comments),
-        }
+                for reply in reply_json.get('getCommentReplies'):
+                    yield self._parse_comment(reply, comment_id)
 
     @staticmethod
     def _parse_comment(comment_data, parent):
@@ -159,7 +154,5 @@ query GetCommentReplies($id: String!) {
             'tags': [tag.get('name') for tag in video_info.get('tags')],
             'availability': self._availability(is_unlisted=video_info.get('unlisted')),
             'comments': comments,
-            '__post_extractor': (
-                (lambda: self._extract_comments(video_id, comments, video_json.get('getVideoComments')))
-                if self.get_param('getcomments') else None)
+            '__post_extractor': self.extract_comments(video_id, comments, video_json.get('getVideoComments'))
         }

yt_dlp/extractor/bilibili.py

@@ -1,16 +1,13 @@
 # coding: utf-8
-from __future__ import unicode_literals
 
 import hashlib
 import itertools
-import json
 import functools
 import re
 import math
 
 from .common import InfoExtractor, SearchInfoExtractor
 from ..compat import (
-    compat_str,
     compat_parse_qs,
     compat_urlparse,
     compat_urllib_parse_urlparse
@@ -20,6 +17,7 @@ from ..utils import (
     int_or_none,
     float_or_none,
     parse_iso8601,
+    traverse_obj,
     try_get,
     smuggle_url,
     srt_subtitles_timecode,
@@ -101,7 +99,7 @@ class BiliBiliIE(InfoExtractor):
                 'upload_date': '20170301',
             },
             'params': {
-                'skip_download': True,  # Test metadata only
+                'skip_download': True,
             },
         }, {
             'info_dict': {
@@ -115,7 +113,7 @@ class BiliBiliIE(InfoExtractor):
                 'upload_date': '20170301',
             },
             'params': {
-                'skip_download': True,  # Test metadata only
+                'skip_download': True,
             },
         }]
     }, {
@@ -169,7 +167,7 @@ class BiliBiliIE(InfoExtractor):
 
         if 'anime/' not in url:
             cid = self._search_regex(
-                r'\bcid(?:["\']:|=)(\d+),["\']page(?:["\']:|=)' + compat_str(page_id), webpage, 'cid',
+                r'\bcid(?:["\']:|=)(\d+),["\']page(?:["\']:|=)' + str(page_id), webpage, 'cid',
                 default=None
             ) or self._search_regex(
                 r'\bcid(?:["\']:|=)(\d+)', webpage, 'cid',
@@ -259,7 +257,7 @@ class BiliBiliIE(InfoExtractor):
             # TODO: The json is already downloaded by _extract_anthology_entries. Don't redownload for each video
             part_title = try_get(
                 self._download_json(
-                    "https://api.bilibili.com/x/player/pagelist?bvid=%s&jsonp=jsonp" % bv_id,
+                    f'https://api.bilibili.com/x/player/pagelist?bvid={bv_id}&jsonp=jsonp',
                     video_id, note='Extracting videos in anthology'),
                 lambda x: x['data'][int(page_id) - 1]['part'])
             title = part_title or title
@@ -273,7 +271,7 @@ class BiliBiliIE(InfoExtractor):
 
         # TODO 'view_count' requires deobfuscating Javascript
         info = {
-            'id': compat_str(video_id) if page_id is None else '%s_p%s' % (video_id, page_id),
+            'id': str(video_id) if page_id is None else '%s_part%s' % (video_id, page_id),
             'cid': cid,
             'title': title,
             'description': description,
@@ -295,29 +293,25 @@ class BiliBiliIE(InfoExtractor):
             info['uploader'] = self._html_search_meta(
                 'author', webpage, 'uploader', default=None)
 
-        raw_danmaku = self._get_raw_danmaku(video_id, cid)
-
-        raw_tags = self._get_tags(video_id)
-        tags = list(map(lambda x: x['tag_name'], raw_tags))
-
         top_level_info = {
-            'raw_danmaku': raw_danmaku,
-            'tags': tags,
-            'raw_tags': raw_tags,
+            'tags': traverse_obj(self._download_json(
+                f'https://api.bilibili.com/x/tag/archive/tags?aid={video_id}',
+                video_id, fatal=False, note='Downloading tags'), ('data', ..., 'tag_name')),
         }
-        if self.get_param('getcomments', False):
-            def get_comments():
-                comments = self._get_all_comment_pages(video_id)
-                return {
-                    'comments': comments,
-                    'comment_count': len(comments)
-                }
 
-            top_level_info['__post_extractor'] = get_comments
+        entries[0]['subtitles'] = {
+            'danmaku': [{
+                'ext': 'xml',
+                'url': f'https://comment.bilibili.com/{cid}.xml',
+            }]
+        }
 
-        '''
+        r'''
         # Requires https://github.com/m13253/danmaku2ass which is licenced under GPL3
         # See https://github.com/animelover1984/youtube-dl
+
+        raw_danmaku = self._download_webpage(
+            f'https://comment.bilibili.com/{cid}.xml', video_id, fatal=False, note='Downloading danmaku comments')
         danmaku = NiconicoIE.CreateDanmaku(raw_danmaku, commentType='Bilibili', x=1024, y=576)
         entries[0]['subtitles'] = {
             'danmaku': [{
@@ -327,29 +321,27 @@ class BiliBiliIE(InfoExtractor):
         }
         '''
 
+        top_level_info['__post_extractor'] = self.extract_comments(video_id)
+
         for entry in entries:
             entry.update(info)
 
         if len(entries) == 1:
             entries[0].update(top_level_info)
             return entries[0]
-        else:
-            for idx, entry in enumerate(entries):
-                entry['id'] = '%s_part%d' % (video_id, (idx + 1))
 
-            global_info = {
-                '_type': 'multi_video',
-                'id': compat_str(video_id),
-                'bv_id': bv_id,
-                'title': title,
-                'description': description,
-                'entries': entries,
-            }
+        for idx, entry in enumerate(entries):
+            entry['id'] = '%s_part%d' % (video_id, (idx + 1))
 
-            global_info.update(info)
-            global_info.update(top_level_info)
-
-            return global_info
+        return {
+            '_type': 'multi_video',
+            'id': str(video_id),
+            'bv_id': bv_id,
+            'title': title,
+            'description': description,
+            'entries': entries,
+            **info, **top_level_info
+        }
 
     def _extract_anthology_entries(self, bv_id, video_id, webpage):
         title = self._html_search_regex(
@@ -357,10 +349,10 @@ class BiliBiliIE(InfoExtractor):
              r'(?s)<h1[^>]*>(?P<title>.+?)</h1>'), webpage, 'title',
             group='title')
         json_data = self._download_json(
-            "https://api.bilibili.com/x/player/pagelist?bvid=%s&jsonp=jsonp" % bv_id,
+            f'https://api.bilibili.com/x/player/pagelist?bvid={bv_id}&jsonp=jsonp',
             video_id, note='Extracting videos in anthology')
 
-        if len(json_data['data']) > 1:
+        if json_data['data']:
             return self.playlist_from_matches(
                 json_data['data'], bv_id, title, ie=BiliBiliIE.ie_key(),
                 getter=lambda entry: 'https://www.bilibili.com/video/%s?p=%d' % (bv_id, entry['page']))
@@ -375,65 +367,31 @@ class BiliBiliIE(InfoExtractor):
         if response['code'] == -400:
             raise ExtractorError('Video ID does not exist', expected=True, video_id=id)
         elif response['code'] != 0:
-            raise ExtractorError('Unknown error occurred during API check (code %s)' % response['code'], expected=True, video_id=id)
-        return (response['data']['aid'], response['data']['bvid'])
+            raise ExtractorError(f'Unknown error occurred during API check (code {response["code"]})',
+                                 expected=True, video_id=id)
+        return response['data']['aid'], response['data']['bvid']
 
-    # recursive solution to getting every page of comments for the video
-    # we can stop when we reach a page without any comments
-    def _get_all_comment_pages(self, video_id, commentPageNumber=0):
-        comment_url = "https://api.bilibili.com/x/v2/reply?jsonp=jsonp&pn=%s&type=1&oid=%s&sort=2&_=1567227301685" % (commentPageNumber, video_id)
-        json_str = self._download_webpage(
-            comment_url, video_id,
-            note='Extracting comments from page %s' % (commentPageNumber))
-        replies = json.loads(json_str)['data']['replies']
-        if replies is None:
-            return []
-        return self._get_all_children(replies) + self._get_all_comment_pages(video_id, commentPageNumber + 1)
+    def _get_comments(self, video_id, commentPageNumber=0):
+        for idx in itertools.count(1):
+            replies = traverse_obj(
+                self._download_json(
+                    f'https://api.bilibili.com/x/v2/reply?pn={idx}&oid={video_id}&type=1&jsonp=jsonp&sort=2&_=1567227301685',
+                    video_id, note=f'Extracting comments from page {idx}'),
+                ('data', 'replies')) or []
+            for children in map(self._get_all_children, replies):
+                yield from children
 
-    # extracts all comments in the tree
-    def _get_all_children(self, replies):
-        if replies is None:
-            return []
-
-        ret = []
-        for reply in replies:
-            author = reply['member']['uname']
-            author_id = reply['member']['mid']
-            id = reply['rpid']
-            text = reply['content']['message']
-            timestamp = reply['ctime']
-            parent = reply['parent'] if reply['parent'] != 0 else 'root'
-
-            comment = {
-                "author": author,
-                "author_id": author_id,
-                "id": id,
-                "text": text,
-                "timestamp": timestamp,
-                "parent": parent,
-            }
-            ret.append(comment)
-
-            # from the JSON, the comment structure seems arbitrarily deep, but I could be wrong.
-            # Regardless, this should work.
-            ret += self._get_all_children(reply['replies'])
-
-        return ret
-
-    def _get_raw_danmaku(self, video_id, cid):
-        # This will be useful if I decide to scrape all pages instead of doing them individually
-        # cid_url = "https://www.bilibili.com/widget/getPageList?aid=%s" % (video_id)
-        # cid_str = self._download_webpage(cid_url, video_id, note=False)
-        # cid = json.loads(cid_str)[0]['cid']
-
-        danmaku_url = "https://comment.bilibili.com/%s.xml" % (cid)
-        danmaku = self._download_webpage(danmaku_url, video_id, note='Downloading danmaku comments')
-        return danmaku
-
-    def _get_tags(self, video_id):
-        tags_url = "https://api.bilibili.com/x/tag/archive/tags?aid=%s" % (video_id)
-        tags_json = self._download_json(tags_url, video_id, note='Downloading tags')
-        return tags_json['data']
+    def _get_all_children(self, reply):
+        yield {
+            'author': traverse_obj(reply, ('member', 'uname')),
+            'author_id': traverse_obj(reply, ('member', 'mid')),
+            'id': reply.get('rpid'),
+            'text': traverse_obj(reply, ('content', 'message')),
+            'timestamp': reply.get('ctime'),
+            'parent': reply.get('parent') or 'root',
+        }
+        for children in map(self._get_all_children, reply.get('replies') or []):
+            yield from children
 
 
 class BiliBiliBangumiIE(InfoExtractor):
@@ -516,11 +474,8 @@ class BilibiliChannelIE(InfoExtractor):
         count, max_count = 0, None
 
         for page_num in itertools.count(1):
-            data = self._parse_json(
-                self._download_webpage(
-                    self._API_URL % (list_id, page_num), list_id,
-                    note='Downloading page %d' % page_num),
-                list_id)['data']
+            data = self._download_json(
+                self._API_URL % (list_id, page_num), list_id, note=f'Downloading page {page_num}')['data']
 
             max_count = max_count or try_get(data, lambda x: x['page']['count'])
 
@@ -583,11 +538,11 @@ class BilibiliCategoryIE(InfoExtractor):
         }
 
         if category not in rid_map:
-            raise ExtractorError('The supplied category, %s, is not supported. List of supported categories: %s' % (category, list(rid_map.keys())))
-
+            raise ExtractorError(
+                f'The category {category} isn\'t supported. Supported categories: {list(rid_map.keys())}')
         if subcategory not in rid_map[category]:
-            raise ExtractorError('The subcategory, %s, isn\'t supported for this category. Supported subcategories: %s' % (subcategory, list(rid_map[category].keys())))
-
+            raise ExtractorError(
+                f'The subcategory {subcategory} isn\'t supported for this category. Supported subcategories: {list(rid_map[category].keys())}')
         rid_value = rid_map[category][subcategory]
 
         api_url = 'https://api.bilibili.com/x/web-interface/newlist?rid=%d&type=1&ps=20&jsonp=jsonp' % rid_value
@@ -614,41 +569,26 @@ class BiliBiliSearchIE(SearchInfoExtractor):
     IE_DESC = 'Bilibili video search, "bilisearch" keyword'
     _MAX_RESULTS = 100000
     _SEARCH_KEY = 'bilisearch'
-    MAX_NUMBER_OF_RESULTS = 1000
 
-    def _get_n_results(self, query, n):
-        """Get a specified number of results for a query"""
-
-        entries = []
-        pageNumber = 0
-        while True:
-            pageNumber += 1
-            # FIXME
-            api_url = 'https://api.bilibili.com/x/web-interface/search/type?context=&page=%s&order=pubdate&keyword=%s&duration=0&tids_2=&__refresh__=true&search_type=video&tids=0&highlight=1' % (pageNumber, query)
-            json_str = self._download_webpage(
-                api_url, "None", query={"Search_key": query},
-                note='Extracting results from page %s' % pageNumber)
-            data = json.loads(json_str)['data']
-
-            # FIXME: this is hideous
-            if "result" not in data:
-                return {
-                    '_type': 'playlist',
-                    'id': query,
-                    'entries': entries[:n]
-                }
-
-            videos = data['result']
+    def _search_results(self, query):
+        for page_num in itertools.count(1):
+            videos = self._download_json(
+                'https://api.bilibili.com/x/web-interface/search/type', query,
+                note=f'Extracting results from page {page_num}', query={
+                    'Search_key': query,
+                    'keyword': query,
+                    'page': page_num,
+                    'context': '',
+                    'order': 'pubdate',
+                    'duration': 0,
+                    'tids_2': '',
+                    '__refresh__': 'true',
+                    'search_type': 'video',
+                    'tids': 0,
+                    'highlight': 1,
+                })['data'].get('result') or []
             for video in videos:
-                e = self.url_result(video['arcurl'], 'BiliBili', compat_str(video['aid']))
-                entries.append(e)
-
-            if(len(entries) >= n or len(videos) >= BiliBiliSearchIE.MAX_NUMBER_OF_RESULTS):
-                return {
-                    '_type': 'playlist',
-                    'id': query,
-                    'entries': entries[:n]
-                }
+                yield self.url_result(video['arcurl'], 'BiliBili', str(video['aid']))
 
 
 class BilibiliAudioBaseIE(InfoExtractor):

yt_dlp/extractor/cbc.py

@@ -2,6 +2,9 @@
 from __future__ import unicode_literals
 
 import re
+import json
+import base64
+import time
 
 from .common import InfoExtractor
 from ..compat import (
@@ -244,37 +247,96 @@ class CBCGemIE(InfoExtractor):
         'params': {'format': 'bv'},
         'skip': 'Geo-restricted to Canada',
     }]
-    _API_BASE = 'https://services.radio-canada.ca/ott/cbc-api/v2/assets/'
+
+    _GEO_COUNTRIES = ['CA']
+    _TOKEN_API_KEY = '3f4beddd-2061-49b0-ae80-6f1f2ed65b37'
+    _NETRC_MACHINE = 'cbcgem'
+    _claims_token = None
+
+    def _new_claims_token(self, email, password):
+        data = json.dumps({
+            'email': email,
+            'password': password,
+        }).encode()
+        headers = {'content-type': 'application/json'}
+        query = {'apikey': self._TOKEN_API_KEY}
+        resp = self._download_json('https://api.loginradius.com/identity/v2/auth/login',
+                                   None, data=data, headers=headers, query=query)
+        access_token = resp['access_token']
+
+        query = {
+            'access_token': access_token,
+            'apikey': self._TOKEN_API_KEY,
+            'jwtapp': 'jwt',
+        }
+        resp = self._download_json('https://cloud-api.loginradius.com/sso/jwt/api/token',
+                                   None, headers=headers, query=query)
+        sig = resp['signature']
+
+        data = json.dumps({'jwt': sig}).encode()
+        headers = {'content-type': 'application/json', 'ott-device-type': 'web'}
+        resp = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/token',
+                                   None, data=data, headers=headers)
+        cbc_access_token = resp['accessToken']
+
+        headers = {'content-type': 'application/json', 'ott-device-type': 'web', 'ott-access-token': cbc_access_token}
+        resp = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/profile',
+                                   None, headers=headers)
+        return resp['claimsToken']
+
+    def _get_claims_token_expiry(self):
+        # Token is a JWT
+        # JWT is decoded here and 'exp' field is extracted
+        # It is a Unix timestamp for when the token expires
+        b64_data = self._claims_token.split('.')[1]
+        data = base64.urlsafe_b64decode(b64_data + "==")
+        return json.loads(data)['exp']
+
+    def claims_token_expired(self):
+        exp = self._get_claims_token_expiry()
+        if exp - time.time() < 10:
+            # It will expire in less than 10 seconds, or has already expired
+            return True
+        return False
+
+    def claims_token_valid(self):
+        return self._claims_token is not None and not self.claims_token_expired()
+
+    def _get_claims_token(self, email, password):
+        if not self.claims_token_valid():
+            self._claims_token = self._new_claims_token(email, password)
+            self._downloader.cache.store(self._NETRC_MACHINE, 'claims_token', self._claims_token)
+        return self._claims_token
+
+    def _real_initialize(self):
+        if self.claims_token_valid():
+            return
+        self._claims_token = self._downloader.cache.load(self._NETRC_MACHINE, 'claims_token')
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        video_info = self._download_json(self._API_BASE + video_id, video_id)
+        video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)
 
-        last_error = None
-        attempt = -1
-        retries = self.get_param('extractor_retries', 15)
-        while attempt < retries:
-            attempt += 1
-            if last_error:
-                self.report_warning('%s. Retrying ...' % last_error)
-            m3u8_info = self._download_json(
-                video_info['playSession']['url'], video_id,
-                note='Downloading JSON metadata%s' % f' (attempt {attempt})')
-            m3u8_url = m3u8_info.get('url')
-            if m3u8_url:
-                break
-            elif m3u8_info.get('errorCode') == 1:
-                self.raise_geo_restricted(countries=['CA'])
-            else:
-                last_error = f'{self.IE_NAME} said: {m3u8_info.get("errorCode")} - {m3u8_info.get("message")}'
-                # 35 means media unavailable, but retries work
-                if m3u8_info.get('errorCode') != 35 or attempt >= retries:
-                    raise ExtractorError(last_error)
+        email, password = self._get_login_info()
+        if email and password:
+            claims_token = self._get_claims_token(email, password)
+            headers = {'x-claims-token': claims_token}
+        else:
+            headers = {}
+        m3u8_info = self._download_json(video_info['playSession']['url'], video_id, headers=headers)
+        m3u8_url = m3u8_info.get('url')
+
+        if m3u8_info.get('errorCode') == 1:
+            self.raise_geo_restricted(countries=['CA'])
+        elif m3u8_info.get('errorCode') == 35:
+            self.raise_login_required(method='password')
+        elif m3u8_info.get('errorCode') != 0:
+            raise ExtractorError(f'{self.IE_NAME} said: {m3u8_info.get("errorCode")} - {m3u8_info.get("message")}')
 
         formats = self._extract_m3u8_formats(m3u8_url, video_id, m3u8_id='hls')
         self._remove_duplicate_formats(formats)
 
-        for i, format in enumerate(formats):
+        for format in formats:
             if format.get('vcodec') == 'none':
                 if format.get('ext') is None:
                     format['ext'] = 'm4a'
@@ -377,7 +439,7 @@ class CBCGemPlaylistIE(InfoExtractor):
 
 class CBCGemLiveIE(InfoExtractor):
     IE_NAME = 'gem.cbc.ca:live'
-    _VALID_URL = r'https?://gem\.cbc\.ca/live/(?P<id>[0-9]{12})'
+    _VALID_URL = r'https?://gem\.cbc\.ca/live/(?P<id>\d+)'
     _TEST = {
         'url': 'https://gem.cbc.ca/live/920604739687',
         'info_dict': {
@@ -396,21 +458,21 @@ class CBCGemLiveIE(InfoExtractor):
 
     # It's unclear where the chars at the end come from, but they appear to be
     # constant. Might need updating in the future.
-    _API = 'https://tpfeed.cbc.ca/f/ExhSPC/t_t3UKJR6MAT'
+    # There are two URLs, some livestreams are in one, and some
+    # in the other. The JSON schema is the same for both.
+    _API_URLS = ['https://tpfeed.cbc.ca/f/ExhSPC/t_t3UKJR6MAT', 'https://tpfeed.cbc.ca/f/ExhSPC/FNiv9xQx_BnT']
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        live_info = self._download_json(self._API, video_id)['entries']
 
-        video_info = None
-        for stream in live_info:
-            if stream.get('guid') == video_id:
-                video_info = stream
-
-        if video_info is None:
-            raise ExtractorError(
-                'Couldn\'t find video metadata, maybe this livestream is now offline',
-                expected=True)
+        for api_url in self._API_URLS:
+            video_info = next((
+                stream for stream in self._download_json(api_url, video_id)['entries']
+                if stream.get('guid') == video_id), None)
+            if video_info:
+                break
+        else:
+            raise ExtractorError('Couldn\'t find video metadata, maybe this livestream is now offline', expected=True)
 
         return {
             '_type': 'url_transparent',

yt_dlp/extractor/common.py

@@ -4,6 +4,7 @@ from __future__ import unicode_literals
 import base64
 import datetime
 import hashlib
+import itertools
 import json
 import netrc
 import os
@@ -146,6 +147,8 @@ class InfoExtractor(object):
                     * width      Width of the video, if known
                     * height     Height of the video, if known
                     * resolution Textual description of width and height
+                    * dynamic_range The dynamic range of the video. One of:
+                                 "SDR" (None), "HDR10", "HDR10+, "HDR12", "HLG, "DV"
                     * tbr        Average bitrate of audio and video in KBit/s
                     * abr        Average audio bitrate in KBit/s
                     * acodec     Name of the audio codec in use
@@ -232,7 +235,6 @@ class InfoExtractor(object):
                         * "resolution" (optional, string "{width}x{height}",
                                         deprecated)
                         * "filesize" (optional, int)
-                        * "_test_url" (optional, bool) - If true, test the URL
     thumbnail:      Full URL to a video thumbnail image.
     description:    Full video description.
     uploader:       Full name of the video uploader.
@@ -440,13 +442,15 @@ class InfoExtractor(object):
     _LOGIN_HINTS = {
         'any': 'Use --cookies, --username and --password or --netrc to provide account credentials',
         'cookies': (
-            'Use --cookies for the authentication. '
-            'See  https://github.com/ytdl-org/youtube-dl#how-do-i-pass-cookies-to-youtube-dl  for how to pass cookies'),
+            'Use --cookies-from-browser or --cookies for the authentication. '
+            'See  https://github.com/ytdl-org/youtube-dl#how-do-i-pass-cookies-to-youtube-dl  for how to manually pass cookies'),
         'password': 'Use --username and --password or --netrc to provide account credentials',
     }
 
     def __init__(self, downloader=None):
-        """Constructor. Receives an optional downloader."""
+        """Constructor. Receives an optional downloader (a YoutubeDL instance).
+        If a downloader is not passed during initialization,
+        it must be set using "set_downloader()" before "extract()" is called"""
         self._ready = False
         self._x_forwarded_for_ip = None
         self._printed_messages = set()
@@ -662,7 +666,7 @@ class InfoExtractor(object):
         See _download_webpage docstring for arguments specification.
         """
         if not self._downloader._first_webpage_request:
-            sleep_interval = float_or_none(self.get_param('sleep_interval_requests')) or 0
+            sleep_interval = self.get_param('sleep_interval_requests') or 0
             if sleep_interval > 0:
                 self.to_screen('Sleeping %s seconds ...' % sleep_interval)
                 time.sleep(sleep_interval)
@@ -1086,12 +1090,13 @@ class InfoExtractor(object):
 
     # Methods for following #608
     @staticmethod
-    def url_result(url, ie=None, video_id=None, video_title=None):
+    def url_result(url, ie=None, video_id=None, video_title=None, **kwargs):
         """Returns a URL that points to a page that should be processed"""
         # TODO: ie should be the class used for getting the info
         video_info = {'_type': 'url',
                       'url': url,
                       'ie_key': ie}
+        video_info.update(kwargs)
         if video_id is not None:
             video_info['id'] = video_id
         if video_title is not None:
@@ -1506,7 +1511,7 @@ class InfoExtractor(object):
         regex = r' *((?P<reverse>\+)?(?P<field>[a-zA-Z0-9_]+)((?P<separator>[~:])(?P<limit>.*?))?)? *$'
 
         default = ('hidden', 'aud_or_vid', 'hasvid', 'ie_pref', 'lang', 'quality',
-                   'res', 'fps', 'codec:vp9.2', 'size', 'br', 'asr',
+                   'res', 'fps', 'hdr:12', 'codec:vp9.2', 'size', 'br', 'asr',
                    'proto', 'ext', 'hasaud', 'source', 'format_id')  # These must not be aliases
         ytdl_default = ('hasaud', 'lang', 'quality', 'tbr', 'filesize', 'vbr',
                         'height', 'width', 'proto', 'vext', 'abr', 'aext',
@@ -1517,6 +1522,8 @@ class InfoExtractor(object):
                        'order': ['av0?1', 'vp0?9.2', 'vp0?9', '[hx]265|he?vc?', '[hx]264|avc', 'vp0?8', 'mp4v|h263', 'theora', '', None, 'none']},
             'acodec': {'type': 'ordered', 'regex': True,
                        'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
+            'hdr': {'type': 'ordered', 'regex': True, 'field': 'dynamic_range',
+                    'order': ['dv', '(hdr)?12', r'(hdr)?10\+', '(hdr)?10', 'hlg', '', 'sdr', None]},
             'proto': {'type': 'ordered', 'regex': True, 'field': 'protocol',
                       'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', '.*dash', 'ws|websocket', '', 'mms|rtsp', 'none', 'f4']},
             'vext': {'type': 'ordered', 'field': 'video_ext',
@@ -2012,7 +2019,7 @@ class InfoExtractor(object):
         if '#EXT-X-FAXS-CM:' in m3u8_doc:  # Adobe Flash Access
             return formats, subtitles
 
-        has_drm = re.search(r'#EXT-X-SESSION-KEY:.*?URI="skd://', m3u8_doc)
+        has_drm = re.search(r'#EXT-X-(?:SESSION-)?KEY:.*?URI="skd://', m3u8_doc)
 
         def format_url(url):
             return url if re.match(r'^https?://', url) else compat_urlparse.urljoin(m3u8_url, url)
@@ -2645,6 +2652,8 @@ class InfoExtractor(object):
                             content_type = mime_type
                         elif codecs.split('.')[0] == 'stpp':
                             content_type = 'text'
+                        elif mimetype2ext(mime_type) in ('tt', 'dfxp', 'ttml', 'xml', 'json'):
+                            content_type = 'text'
                         else:
                             self.report_warning('Unknown MIME type %s in DASH manifest' % mime_type)
                             continue
@@ -3501,6 +3510,32 @@ class InfoExtractor(object):
     def _get_subtitles(self, *args, **kwargs):
         raise NotImplementedError('This method must be implemented by subclasses')
 
+    def extract_comments(self, *args, **kwargs):
+        if not self.get_param('getcomments'):
+            return None
+        generator = self._get_comments(*args, **kwargs)
+
+        def extractor():
+            comments = []
+            try:
+                while True:
+                    comments.append(next(generator))
+            except KeyboardInterrupt:
+                interrupted = True
+                self.to_screen('Interrupted by user')
+            except StopIteration:
+                interrupted = False
+            comment_count = len(comments)
+            self.to_screen(f'Extracted {comment_count} comments')
+            return {
+                'comments': comments,
+                'comment_count': None if interrupted else comment_count
+            }
+        return extractor
+
+    def _get_comments(self, *args, **kwargs):
+        raise NotImplementedError('This method must be implemented by subclasses')
+
     @staticmethod
     def _merge_subtitle_items(subtitle_list1, subtitle_list2):
         """ Merge subtitle items for one language. Items with duplicated URLs
@@ -3617,7 +3652,14 @@ class SearchInfoExtractor(InfoExtractor):
             return self._get_n_results(query, n)
 
     def _get_n_results(self, query, n):
-        """Get a specified number of results for a query"""
+        """Get a specified number of results for a query.
+        Either this function or _search_results must be overridden by subclasses """
+        return self.playlist_result(
+            itertools.islice(self._search_results(query), 0, None if n == float('inf') else n),
+            query, query)
+
+    def _search_results(self, query):
+        """Returns an iterator of search results"""
         raise NotImplementedError('This method must be implemented by subclasses')
 
     @property

yt_dlp/extractor/crunchyroll.py

@@ -650,7 +650,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
 
 class CrunchyrollShowPlaylistIE(CrunchyrollBaseIE):
     IE_NAME = 'crunchyroll:playlist'
-    _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.com/(?!(?:news|anime-news|library|forum|launchcalendar|lineup|store|comics|freetrial|login|media-\d+))(?P<id>[\w\-]+))/?(?:\?|$)'
+    _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.com/(?:\w{1,2}/)?(?!(?:news|anime-news|library|forum|launchcalendar|lineup|store|comics|freetrial|login|media-\d+))(?P<id>[\w\-]+))/?(?:\?|$)'
 
     _TESTS = [{
         'url': 'https://www.crunchyroll.com/a-bridge-to-the-starry-skies-hoshizora-e-kakaru-hashi',
@@ -672,6 +672,9 @@ class CrunchyrollShowPlaylistIE(CrunchyrollBaseIE):
         # geo-restricted (US), 18+ maturity wall, non-premium will be available since 2015.11.14
         'url': 'http://www.crunchyroll.com/ladies-versus-butlers?skip_wall=1',
         'only_matching': True,
+    }, {
+        'url': 'http://www.crunchyroll.com/fr/ladies-versus-butlers',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -683,18 +686,72 @@ class CrunchyrollShowPlaylistIE(CrunchyrollBaseIE):
             headers=self.geo_verification_headers())
         title = self._html_search_meta('name', webpage, default=None)
 
-        episode_paths = re.findall(
-            r'(?s)<li id="showview_videos_media_(\d+)"[^>]+>.*?<a href="([^"]+)"',
-            webpage)
-        entries = [
-            self.url_result('http://www.crunchyroll.com' + ep, 'Crunchyroll', ep_id)
-            for ep_id, ep in episode_paths
-        ]
-        entries.reverse()
+        episode_re = r'<li id="showview_videos_media_(\d+)"[^>]+>.*?<a href="([^"]+)"'
+        season_re = r'<a [^>]+season-dropdown[^>]+>([^<]+)'
+        paths = re.findall(f'(?s){episode_re}|{season_re}', webpage)
+
+        entries, current_season = [], None
+        for ep_id, ep, season in paths:
+            if season:
+                current_season = season
+                continue
+            entries.append(self.url_result(
+                f'http://www.crunchyroll.com{ep}', CrunchyrollIE.ie_key(), ep_id, season=current_season))
 
         return {
             '_type': 'playlist',
             'id': show_id,
             'title': title,
-            'entries': entries,
+            'entries': reversed(entries),
         }
+
+
+class CrunchyrollBetaIE(CrunchyrollBaseIE):
+    IE_NAME = 'crunchyroll:beta'
+    _VALID_URL = r'https?://beta\.crunchyroll\.com/(?P<lang>(?:\w{1,2}/)?)watch/(?P<internal_id>\w+)/(?P<id>[\w\-]+)/?(?:\?|$)'
+    _TESTS = [{
+        'url': 'https://beta.crunchyroll.com/watch/GY2P1Q98Y/to-the-future',
+        'info_dict': {
+            'id': '696363',
+            'ext': 'mp4',
+            'timestamp': 1459610100,
+            'description': 'md5:a022fbec4fbb023d43631032c91ed64b',
+            'uploader': 'Toei Animation',
+            'title': 'World Trigger Episode 73 – To the Future',
+            'upload_date': '20160402',
+        },
+        'params': {'skip_download': 'm3u8'},
+        'expected_warnings': ['Unable to download XML']
+    }]
+
+    def _real_extract(self, url):
+        lang, internal_id, display_id = self._match_valid_url(url).group('lang', 'internal_id', 'id')
+        webpage = self._download_webpage(url, display_id)
+        episode_data = self._parse_json(
+            self._search_regex(r'__INITIAL_STATE__\s*=\s*({.+?})\s*;', webpage, 'episode data'),
+            display_id)['content']['byId'][internal_id]
+        video_id = episode_data['external_id'].split('.')[1]
+        series_id = episode_data['episode_metadata']['series_slug_title']
+        return self.url_result(f'https://www.crunchyroll.com/{lang}{series_id}/{display_id}-{video_id}',
+                               CrunchyrollIE.ie_key(), video_id)
+
+
+class CrunchyrollBetaShowIE(CrunchyrollBaseIE):
+    IE_NAME = 'crunchyroll:playlist:beta'
+    _VALID_URL = r'https?://beta\.crunchyroll\.com/(?P<lang>(?:\w{1,2}/)?)series/\w+/(?P<id>[\w\-]+)/?(?:\?|$)'
+    _TESTS = [{
+        'url': 'https://beta.crunchyroll.com/series/GY19NQ2QR/Girl-Friend-BETA',
+        'info_dict': {
+            'id': 'girl-friend-beta',
+            'title': 'Girl Friend BETA',
+        },
+        'playlist_mincount': 10,
+    }, {
+        'url': 'https://beta.crunchyroll.com/it/series/GY19NQ2QR/Girl-Friend-BETA',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        lang, series_id = self._match_valid_url(url).group('lang', 'id')
+        return self.url_result(f'https://www.crunchyroll.com/{lang}{series_id.lower()}',
+                               CrunchyrollShowPlaylistIE.ie_key(), series_id)

yt_dlp/extractor/euscreen.py

@@ -0,0 +1,64 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+from ..utils import (
+    parse_duration,
+    js_to_json,
+)
+
+
+class EUScreenIE(InfoExtractor):
+    _VALID_URL = r'(?:https?://)(?:www\.)?euscreen\.eu/item.html\?id=(?P<id>[^&?$/]+)'
+
+    _TESTS = [{
+        'url': 'https://euscreen.eu/item.html?id=EUS_0EBCBF356BFC4E12A014023BA41BD98C',
+        'info_dict': {
+            'id': 'EUS_0EBCBF356BFC4E12A014023BA41BD98C',
+            'ext': 'mp4',
+            'title': "L'effondrement du stade du Heysel",
+            'alt_title': 'Collapse of the Heysel Stadium',
+            'duration': 318.0,
+            'description': 'md5:f0ffffdfce6821139357a1b8359d6152',
+            'series': 'JA2 DERNIERE',
+            'episode': '-',
+            'uploader': 'INA /  France',
+            'thumbnail': 'http://images3.noterik.com/domain/euscreenxl/user/eu_ina/video/EUS_0EBCBF356BFC4E12A014023BA41BD98C/image.jpg'
+        },
+        'params': {'skip_download': True}
+    }]
+
+    _payload = b'<fsxml><screen><properties><screenId>-1</screenId></properties><capabilities id="1"><properties><platform>Win32</platform><appcodename>Mozilla</appcodename><appname>Netscape</appname><appversion>5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36</appversion><useragent>Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36</useragent><cookiesenabled>true</cookiesenabled><screenwidth>784</screenwidth><screenheight>758</screenheight><orientation>undefined</orientation><smt_browserid>Sat, 07 Oct 2021 08:56:50 GMT</smt_browserid><smt_sessionid>1633769810758</smt_sessionid></properties></capabilities></screen></fsxml>'
+
+    def _real_extract(self, url):
+        id = self._match_id(url)
+        args_for_js_request = self._download_webpage(
+            'https://euscreen.eu/lou/LouServlet/domain/euscreenxl/html5application/euscreenxlitem',
+            id, data=self._payload, query={'actionlist': 'itempage', 'id': id})
+        info_js = self._download_webpage(
+            'https://euscreen.eu/lou/LouServlet/domain/euscreenxl/html5application/euscreenxlitem',
+            id, data=args_for_js_request.replace('screenid', 'screenId').encode())
+        video_json = self._parse_json(
+            self._search_regex(r'setVideo\(({.+})\)\(\$end\$\)put', info_js, 'Video JSON'),
+            id, transform_source=js_to_json)
+        meta_json = self._parse_json(
+            self._search_regex(r'setData\(({.+})\)\(\$end\$\)', info_js, 'Metadata JSON'),
+            id, transform_source=js_to_json)
+        formats = [{
+            'url': source['src'],
+        } for source in video_json.get('sources', [])]
+        self._sort_formats(formats)
+
+        return {
+            'id': id,
+            'title': meta_json.get('originalTitle'),
+            'alt_title': meta_json.get('title'),
+            'duration': parse_duration(meta_json.get('duration')),
+            'description': '%s\n%s' % (meta_json.get('summaryOriginal', ''), meta_json.get('summaryEnglish', '')),
+            'series': meta_json.get('series') or meta_json.get('seriesEnglish'),
+            'episode': meta_json.get('episodeNumber'),
+            'uploader': meta_json.get('provider'),
+            'thumbnail': meta_json.get('screenshot') or video_json.get('screenshot'),
+            'formats': formats,
+        }

yt_dlp/extractor/extractors.py

@@ -298,7 +298,9 @@ from .crackle import CrackleIE
 from .crooksandliars import CrooksAndLiarsIE
 from .crunchyroll import (
     CrunchyrollIE,
-    CrunchyrollShowPlaylistIE
+    CrunchyrollShowPlaylistIE,
+    CrunchyrollBetaIE,
+    CrunchyrollBetaShowIE,
 )
 from .cspan import CSpanIE
 from .ctsnews import CtsNewsIE
@@ -420,6 +422,7 @@ from .espn import (
 )
 from .esri import EsriVideoIE
 from .europa import EuropaIE
+from .euscreen import EUScreenIE
 from .expotv import ExpoTVIE
 from .expressen import ExpressenIE
 from .extremetube import ExtremeTubeIE
@@ -474,12 +477,7 @@ from .franceinter import FranceInterIE
 from .francetv import (
     FranceTVIE,
     FranceTVSiteIE,
-    FranceTVEmbedIE,
     FranceTVInfoIE,
-    FranceTVInfoSportIE,
-    FranceTVJeunesseIE,
-    GenerationWhatIE,
-    CultureboxIE,
 )
 from .freesound import FreesoundIE
 from .freespeech import FreespeechIE
@@ -529,6 +527,7 @@ from .gopro import GoProIE
 from .goshgay import GoshgayIE
 from .gotostage import GoToStageIE
 from .gputechconf import GPUTechConfIE
+from .gronkh import GronkhIE
 from .groupon import GrouponIE
 from .hbo import HBOIE
 from .hearthisat import HearThisAtIE
@@ -761,6 +760,7 @@ from .metacritic import MetacriticIE
 from .mgoon import MgoonIE
 from .mgtv import MGTVIE
 from .miaopai import MiaoPaiIE
+from .microsoftstream import MicrosoftStreamIE
 from .microsoftvirtualacademy import (
     MicrosoftVirtualAcademyIE,
     MicrosoftVirtualAcademyCourseIE,
@@ -938,6 +938,7 @@ from .nova import (
     NovaEmbedIE,
     NovaIE,
 )
+from .novaplay import NovaPlayIE
 from .nowness import (
     NownessIE,
     NownessPlaylistIE,
@@ -984,6 +985,7 @@ from .odatv import OdaTVIE
 from .odnoklassniki import OdnoklassnikiIE
 from .oktoberfesttv import OktoberfestTVIE
 from .olympics import OlympicsReplayIE
+from .on24 import On24IE
 from .ondemandkorea import OnDemandKoreaIE
 from .onet import (
     OnetIE,
@@ -1284,6 +1286,7 @@ from .skynewsarabia import (
     SkyNewsArabiaIE,
     SkyNewsArabiaArticleIE,
 )
+from .skynewsau import SkyNewsAUIE
 from .sky import (
     SkyNewsIE,
     SkySportsIE,
@@ -1385,10 +1388,7 @@ from .svt import (
 from .swrmediathek import SWRMediathekIE
 from .syfy import SyfyIE
 from .sztvhu import SztvHuIE
-from .tagesschau import (
-    TagesschauPlayerIE,
-    TagesschauIE,
-)
+from .tagesschau import TagesschauIE
 from .tass import TassIE
 from .tbs import TBSIE
 from .tdslifeway import TDSLifewayIE
@@ -1473,6 +1473,8 @@ from .trilulilu import TriluliluIE
 from .trovo import (
     TrovoIE,
     TrovoVodIE,
+    TrovoChannelVodIE,
+    TrovoChannelClipIE,
 )
 from .trunews import TruNewsIE
 from .trutv import TruTVIE

yt_dlp/extractor/francetv.py

@@ -4,19 +4,12 @@ from __future__ import unicode_literals
 
 
 from .common import InfoExtractor
-from ..compat import (
-    compat_str,
-)
 from ..utils import (
-    clean_html,
     determine_ext,
     ExtractorError,
-    int_or_none,
-    parse_duration,
+    format_field,
+    parse_iso8601,
     parse_qs,
-    try_get,
-    url_or_none,
-    urljoin,
 )
 from .dailymotion import DailymotionIE
 
@@ -89,97 +82,81 @@ class FranceTVIE(InfoExtractor):
         # Videos are identified by idDiffusion so catalogue part is optional.
         # However when provided, some extra formats may be returned so we pass
         # it if available.
-        info = self._download_json(
-            'https://sivideo.webservices.francetelevisions.fr/tools/getInfosOeuvre/v2/',
-            video_id, 'Downloading video JSON', query={
-                'idDiffusion': video_id,
-                'catalogue': catalogue or '',
-            })
-
-        if info.get('status') == 'NOK':
-            raise ExtractorError(
-                '%s returned error: %s' % (self.IE_NAME, info['message']),
-                expected=True)
-        allowed_countries = info['videos'][0].get('geoblocage')
-        if allowed_countries:
-            georestricted = True
-            geo_info = self._download_json(
-                'http://geo.francetv.fr/ws/edgescape.json', video_id,
-                'Downloading geo restriction info')
-            country = geo_info['reponse']['geo_info']['country_code']
-            if country not in allowed_countries:
-                raise ExtractorError(
-                    'The video is not available from your location',
-                    expected=True)
-        else:
-            georestricted = False
-
-        def sign(manifest_url, manifest_id):
-            for host in ('hdfauthftv-a.akamaihd.net', 'hdfauth.francetv.fr'):
-                signed_url = url_or_none(self._download_webpage(
-                    'https://%s/esi/TA' % host, video_id,
-                    'Downloading signed %s manifest URL' % manifest_id,
-                    fatal=False, query={
-                        'url': manifest_url,
-                    }))
-                if signed_url:
-                    return signed_url
-            return manifest_url
-
         is_live = None
-
         videos = []
+        title = None
+        subtitle = None
+        image = None
+        duration = None
+        timestamp = None
+        spritesheets = None
 
-        for video in (info.get('videos') or []):
-            if video.get('statut') != 'ONLINE':
+        for device_type in ('desktop', 'mobile'):
+            dinfo = self._download_json(
+                'https://player.webservices.francetelevisions.fr/v1/videos/%s' % video_id,
+                video_id, 'Downloading %s video JSON' % device_type, query={
+                    'device_type': device_type,
+                    'browser': 'chrome',
+                }, fatal=False)
+
+            if not dinfo:
                 continue
-            if not video.get('url'):
-                continue
-            videos.append(video)
 
-        if not videos:
-            for device_type in ['desktop', 'mobile']:
-                fallback_info = self._download_json(
-                    'https://player.webservices.francetelevisions.fr/v1/videos/%s' % video_id,
-                    video_id, 'Downloading fallback %s video JSON' % device_type, query={
-                        'device_type': device_type,
-                        'browser': 'chrome',
-                    }, fatal=False)
+            video = dinfo.get('video')
+            if video:
+                videos.append(video)
+                if duration is None:
+                    duration = video.get('duration')
+                if is_live is None:
+                    is_live = video.get('is_live')
+                if spritesheets is None:
+                    spritesheets = video.get('spritesheets')
 
-                if fallback_info and fallback_info.get('video'):
-                    videos.append(fallback_info['video'])
+            meta = dinfo.get('meta')
+            if meta:
+                if title is None:
+                    title = meta.get('title')
+                # XXX: what is meta['pre_title']?
+                if subtitle is None:
+                    subtitle = meta.get('additional_title')
+                if image is None:
+                    image = meta.get('image_url')
+                if timestamp is None:
+                    timestamp = parse_iso8601(meta.get('broadcasted_at'))
 
         formats = []
         subtitles = {}
         for video in videos:
-            video_url = video.get('url')
-            if not video_url:
-                continue
-            if is_live is None:
-                is_live = (try_get(
-                    video, lambda x: x['plages_ouverture'][0]['direct'], bool) is True
-                    or video.get('is_live') is True
-                    or '/live.francetv.fr/' in video_url)
             format_id = video.get('format')
+
+            video_url = None
+            if video.get('workflow') == 'token-akamai':
+                token_url = video.get('token')
+                if token_url:
+                    token_json = self._download_json(
+                        token_url, video_id,
+                        'Downloading signed %s manifest URL' % format_id)
+                    if token_json:
+                        video_url = token_json.get('url')
+            if not video_url:
+                video_url = video.get('url')
+
             ext = determine_ext(video_url)
             if ext == 'f4m':
-                if georestricted:
-                    # See https://github.com/ytdl-org/youtube-dl/issues/3963
-                    # m3u8 urls work fine
-                    continue
                 formats.extend(self._extract_f4m_formats(
-                    sign(video_url, format_id) + '&hdcore=3.7.0&plugin=aasp-3.7.0.39.44',
-                    video_id, f4m_id=format_id, fatal=False))
+                    video_url, video_id, f4m_id=format_id, fatal=False))
             elif ext == 'm3u8':
-                m3u8_fmts, m3u8_subs = self._extract_m3u8_formats_and_subtitles(
-                    sign(video_url, format_id), video_id, 'mp4',
+                fmts, subs = self._extract_m3u8_formats_and_subtitles(
+                    video_url, video_id, 'mp4',
                     entry_protocol='m3u8_native', m3u8_id=format_id,
                     fatal=False)
-                formats.extend(m3u8_fmts)
-                subtitles = self._merge_subtitles(subtitles, m3u8_subs)
+                formats.extend(fmts)
+                self._merge_subtitles(subs, target=subtitles)
             elif ext == 'mpd':
-                formats.extend(self._extract_mpd_formats(
-                    sign(video_url, format_id), video_id, mpd_id=format_id, fatal=False))
+                fmts, subs = self._extract_mpd_formats_and_subtitles(
+                    video_url, video_id, mpd_id=format_id, fatal=False)
+                formats.extend(fmts)
+                self._merge_subtitles(subs, target=subtitles)
             elif video_url.startswith('rtmp'):
                 formats.append({
                     'url': video_url,
@@ -193,28 +170,43 @@ class FranceTVIE(InfoExtractor):
                         'format_id': format_id,
                     })
 
+            # XXX: what is video['captions']?
+
+        for f in formats:
+            if f.get('acodec') != 'none' and f.get('language') in ('qtz', 'qad'):
+                f['language_preference'] = -10
+                f['format_note'] = 'audio description%s' % format_field(f, 'format_note', ', %s')
+
+        if spritesheets:
+            formats.append({
+                'format_id': 'spritesheets',
+                'format_note': 'storyboard',
+                'acodec': 'none',
+                'vcodec': 'none',
+                'ext': 'mhtml',
+                'protocol': 'mhtml',
+                'url': 'about:dummy',
+                'fragments': [{
+                    'path': sheet,
+                    # XXX: not entirely accurate; each spritesheet seems to be
+                    # a 10×10 grid of thumbnails corresponding to approximately
+                    # 2 seconds of the video; the last spritesheet may be shorter
+                    'duration': 200,
+                } for sheet in spritesheets]
+            })
+
         self._sort_formats(formats)
 
-        title = info['titre']
-        subtitle = info.get('sous_titre')
         if subtitle:
             title += ' - %s' % subtitle
         title = title.strip()
 
-        subtitles.setdefault('fr', []).extend(
-            [{
-                'url': subformat['url'],
-                'ext': subformat.get('format'),
-            } for subformat in info.get('subtitles', []) if subformat.get('url')]
-        )
-
         return {
             'id': video_id,
             'title': self._live_title(title) if is_live else title,
-            'description': clean_html(info.get('synopsis')),
-            'thumbnail': urljoin('https://sivideo.webservices.francetelevisions.fr', info.get('image')),
-            'duration': int_or_none(info.get('real_duration')) or parse_duration(info.get('duree')),
-            'timestamp': int_or_none(try_get(info, lambda x: x['diffusion']['timestamp'])),
+            'thumbnail': image,
+            'duration': duration,
+            'timestamp': timestamp,
             'is_live': is_live,
             'formats': formats,
             'subtitles': subtitles,
@@ -308,35 +300,6 @@ class FranceTVSiteIE(FranceTVBaseInfoExtractor):
         return self._make_url_result(video_id, catalogue)
 
 
-class FranceTVEmbedIE(FranceTVBaseInfoExtractor):
-    _VALID_URL = r'https?://embed\.francetv\.fr/*\?.*?\bue=(?P<id>[^&]+)'
-
-    _TESTS = [{
-        'url': 'http://embed.francetv.fr/?ue=7fd581a2ccf59d2fc5719c5c13cf6961',
-        'info_dict': {
-            'id': 'NI_983319',
-            'ext': 'mp4',
-            'title': 'Le Pen Reims',
-            'upload_date': '20170505',
-            'timestamp': 1493981780,
-            'duration': 16,
-        },
-        'params': {
-            'skip_download': True,
-        },
-        'add_ie': [FranceTVIE.ie_key()],
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        video = self._download_json(
-            'http://api-embed.webservices.francetelevisions.fr/key/%s' % video_id,
-            video_id)
-
-        return self._make_url_result(video['video_id'], video.get('catalog'))
-
-
 class FranceTVInfoIE(FranceTVBaseInfoExtractor):
     IE_NAME = 'francetvinfo.fr'
     _VALID_URL = r'https?://(?:www|mobile|france3-regions)\.francetvinfo\.fr/(?:[^/]+/)*(?P<id>[^/?#&.]+)'
@@ -426,139 +389,3 @@ class FranceTVInfoIE(FranceTVBaseInfoExtractor):
             webpage, 'video id')
 
         return self._make_url_result(video_id)
-
-
-class FranceTVInfoSportIE(FranceTVBaseInfoExtractor):
-    IE_NAME = 'sport.francetvinfo.fr'
-    _VALID_URL = r'https?://sport\.francetvinfo\.fr/(?:[^/]+/)*(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'https://sport.francetvinfo.fr/les-jeux-olympiques/retour-sur-les-meilleurs-moments-de-pyeongchang-2018',
-        'info_dict': {
-            'id': '6e49080e-3f45-11e8-b459-000d3a2439ea',
-            'ext': 'mp4',
-            'title': 'Retour sur les meilleurs moments de Pyeongchang 2018',
-            'timestamp': 1523639962,
-            'upload_date': '20180413',
-        },
-        'params': {
-            'skip_download': True,
-        },
-        'add_ie': [FranceTVIE.ie_key()],
-    }]
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        video_id = self._search_regex(r'data-video="([^"]+)"', webpage, 'video_id')
-        return self._make_url_result(video_id, 'Sport-web')
-
-
-class GenerationWhatIE(InfoExtractor):
-    IE_NAME = 'france2.fr:generation-what'
-    _VALID_URL = r'https?://generation-what\.francetv\.fr/[^/]+/video/(?P<id>[^/?#&]+)'
-
-    _TESTS = [{
-        'url': 'http://generation-what.francetv.fr/portrait/video/present-arms',
-        'info_dict': {
-            'id': 'wtvKYUG45iw',
-            'ext': 'mp4',
-            'title': 'Generation What - Garde à vous - FRA',
-            'uploader': 'Generation What',
-            'uploader_id': 'UCHH9p1eetWCgt4kXBYCb3_w',
-            'upload_date': '20160411',
-        },
-        'params': {
-            'skip_download': True,
-        },
-        'add_ie': ['Youtube'],
-    }, {
-        'url': 'http://generation-what.francetv.fr/europe/video/present-arms',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-
-        webpage = self._download_webpage(url, display_id)
-
-        youtube_id = self._search_regex(
-            r"window\.videoURL\s*=\s*'([0-9A-Za-z_-]{11})';",
-            webpage, 'youtube id')
-
-        return self.url_result(youtube_id, ie='Youtube', video_id=youtube_id)
-
-
-class CultureboxIE(FranceTVBaseInfoExtractor):
-    _VALID_URL = r'https?://(?:m\.)?culturebox\.francetvinfo\.fr/(?:[^/]+/)*(?P<id>[^/?#&]+)'
-
-    _TESTS = [{
-        'url': 'https://culturebox.francetvinfo.fr/opera-classique/musique-classique/c-est-baroque/concerts/cantates-bwv-4-106-et-131-de-bach-par-raphael-pichon-57-268689',
-        'info_dict': {
-            'id': 'EV_134885',
-            'ext': 'mp4',
-            'title': 'Cantates BWV 4, 106 et 131 de Bach par Raphaël Pichon 5/7',
-            'description': 'md5:19c44af004b88219f4daa50fa9a351d4',
-            'upload_date': '20180206',
-            'timestamp': 1517945220,
-            'duration': 5981,
-        },
-        'params': {
-            'skip_download': True,
-        },
-        'add_ie': [FranceTVIE.ie_key()],
-    }]
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-
-        webpage = self._download_webpage(url, display_id)
-
-        if ">Ce live n'est plus disponible en replay<" in webpage:
-            raise ExtractorError(
-                'Video %s is not available' % display_id, expected=True)
-
-        video_id, catalogue = self._search_regex(
-            r'["\'>]https?://videos\.francetv\.fr/video/([^@]+@.+?)["\'<]',
-            webpage, 'video id').split('@')
-
-        return self._make_url_result(video_id, catalogue)
-
-
-class FranceTVJeunesseIE(FranceTVBaseInfoExtractor):
-    _VALID_URL = r'(?P<url>https?://(?:www\.)?(?:zouzous|ludo)\.fr/heros/(?P<id>[^/?#&]+))'
-
-    _TESTS = [{
-        'url': 'https://www.zouzous.fr/heros/simon',
-        'info_dict': {
-            'id': 'simon',
-        },
-        'playlist_count': 9,
-    }, {
-        'url': 'https://www.ludo.fr/heros/ninjago',
-        'info_dict': {
-            'id': 'ninjago',
-        },
-        'playlist_count': 10,
-    }, {
-        'url': 'https://www.zouzous.fr/heros/simon?abc',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        mobj = self._match_valid_url(url)
-        playlist_id = mobj.group('id')
-
-        playlist = self._download_json(
-            '%s/%s' % (mobj.group('url'), 'playlist'), playlist_id)
-
-        if not playlist.get('count'):
-            raise ExtractorError(
-                '%s is not available' % playlist_id, expected=True)
-
-        entries = []
-        for item in playlist['items']:
-            identity = item.get('identity')
-            if identity and isinstance(identity, compat_str):
-                entries.append(self._make_url_result(identity))
-
-        return self.playlist_result(entries, playlist_id)

yt_dlp/extractor/googlesearch.py

@@ -11,6 +11,7 @@ class GoogleSearchIE(SearchInfoExtractor):
     _MAX_RESULTS = 1000
     IE_NAME = 'video.google:search'
     _SEARCH_KEY = 'gvsearch'
+    _WORKING = False
     _TEST = {
         'url': 'gvsearch15:python language',
         'info_dict': {
@@ -20,16 +21,7 @@ class GoogleSearchIE(SearchInfoExtractor):
         'playlist_count': 15,
     }
 
-    def _get_n_results(self, query, n):
-        """Get a specified number of results for a query"""
-
-        entries = []
-        res = {
-            '_type': 'playlist',
-            'id': query,
-            'title': query,
-        }
-
+    def _search_results(self, query):
         for pagenum in itertools.count():
             webpage = self._download_webpage(
                 'http://www.google.com/search',
@@ -44,16 +36,8 @@ class GoogleSearchIE(SearchInfoExtractor):
 
             for hit_idx, mobj in enumerate(re.finditer(
                     r'<h3 class="r"><a href="([^"]+)"', webpage)):
+                if re.search(f'id="vidthumb{hit_idx + 1}"', webpage):
+                    yield self.url_result(mobj.group(1))
 
-                # Skip playlists
-                if not re.search(r'id="vidthumb%d"' % (hit_idx + 1), webpage):
-                    continue
-
-                entries.append({
-                    '_type': 'url',
-                    'url': mobj.group(1)
-                })
-
-            if (len(entries) >= n) or not re.search(r'id="pnnext"', webpage):
-                res['entries'] = entries[:n]
-                return res
+            if not re.search(r'id="pnnext"', webpage):
+                return

yt_dlp/extractor/gronkh.py

@@ -0,0 +1,43 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import unified_strdate
+
+
+class GronkhIE(InfoExtractor):
+    _VALID_URL = r'(?:https?://)(?:www\.)?gronkh\.tv/stream/(?P<id>\d+)'
+
+    _TESTS = [{
+        'url': 'https://gronkh.tv/stream/536',
+        'info_dict': {
+            'id': '536',
+            'ext': 'mp4',
+            'title': 'GTV0536, 2021-10-01 - MARTHA IS DEAD  #FREiAB1830  !FF7 !horde !archiv',
+            'view_count': 19491,
+            'thumbnail': 'https://01.cdn.vod.farm/preview/6436746cce14e25f751260a692872b9b.jpg',
+            'upload_date': '20211001'
+        },
+        'params': {'skip_download': True}
+    }]
+
+    def _real_extract(self, url):
+        id = self._match_id(url)
+        data_json = self._download_json(f'https://api.gronkh.tv/v1/video/info?episode={id}', id)
+        m3u8_url = self._download_json(f'https://api.gronkh.tv/v1/video/playlist?episode={id}', id)['playlist_url']
+        formats, subtitles = self._extract_m3u8_formats_and_subtitles(m3u8_url, id)
+        if data_json.get('vtt_url'):
+            subtitles.setdefault('en', []).append({
+                'url': data_json['vtt_url'],
+                'ext': 'vtt',
+            })
+        self._sort_formats(formats)
+        return {
+            'id': id,
+            'title': data_json.get('title'),
+            'view_count': data_json.get('views'),
+            'thumbnail': data_json.get('preview_url'),
+            'upload_date': unified_strdate(data_json.get('created_at')),
+            'formats': formats,
+            'subtitles': subtitles,
+        }

yt_dlp/extractor/hidive.py

@@ -72,8 +72,9 @@ class HiDiveIE(InfoExtractor):
                 parsed_urls.add(cc_url)
                 subtitles.setdefault(cc_lang, []).append({'url': cc_url})
 
-    def _get_subtitles(self, url, video_id, title, key, subtitles, parsed_urls):
+    def _get_subtitles(self, url, video_id, title, key, parsed_urls):
         webpage = self._download_webpage(url, video_id, fatal=False) or ''
+        subtitles = {}
         for caption in set(re.findall(r'data-captions=\"([^\"]+)\"', webpage)):
             renditions = self._call_api(
                 video_id, title, key, {'Captions': caption}, fatal=False,
@@ -93,7 +94,7 @@ class HiDiveIE(InfoExtractor):
             raise ExtractorError(
                 '%s said: %s' % (self.IE_NAME, restriction), expected=True)
 
-        formats, parsed_urls = [], {}, {None}
+        formats, parsed_urls = [], {None}
         for rendition_id, rendition in settings['renditions'].items():
             audio, version, extra = rendition_id.split('_')
             m3u8_url = url_or_none(try_get(rendition, lambda x: x['bitrates']['hls']))

yt_dlp/extractor/hotstar.py

@@ -70,7 +70,7 @@ class HotStarBaseIE(InfoExtractor):
     def _call_api_v2(self, path, video_id, st=None, cookies=None):
         return self._call_api_impl(
             '%s/content/%s' % (path, video_id), video_id, st=st, cookies=cookies, query={
-                'desired-config': 'audio_channel:stereo|dynamic_range:sdr|encryption:plain|ladder:tv|package:dash|resolution:hd|subs-tag:HotstarVIP|video_codec:vp9',
+                'desired-config': 'audio_channel:stereo|container:fmp4|dynamic_range:hdr|encryption:plain|ladder:tv|package:dash|resolution:fhd|subs-tag:HotstarVIP|video_codec:h265',
                 'device-id': cookies.get('device_id').value if cookies.get('device_id') else compat_str(uuid.uuid4()),
                 'os-name': 'Windows',
                 'os-version': '10',
@@ -196,41 +196,42 @@ class HotStarIE(HotStarBaseIE):
         for playback_set in playback_sets:
             if not isinstance(playback_set, dict):
                 continue
+            dr = re.search(r'dynamic_range:(?P<dr>[a-z]+)', playback_set.get('tagsCombination')).group('dr')
             format_url = url_or_none(playback_set.get('playbackUrl'))
             if not format_url:
                 continue
             format_url = re.sub(
                 r'(?<=//staragvod)(\d)', r'web\1', format_url)
             tags = str_or_none(playback_set.get('tagsCombination')) or ''
-            if tags and 'encryption:plain' not in tags:
-                continue
             ext = determine_ext(format_url)
+            current_formats, current_subs = [], {}
             try:
                 if 'package:hls' in tags or ext == 'm3u8':
-                    hls_formats, hls_subs = self._extract_m3u8_formats_and_subtitles(
+                    current_formats, current_subs = self._extract_m3u8_formats_and_subtitles(
                         format_url, video_id, 'mp4',
                         entry_protocol='m3u8_native',
-                        m3u8_id='hls', headers=headers)
-                    formats.extend(hls_formats)
-                    subs = self._merge_subtitles(subs, hls_subs)
+                        m3u8_id=f'{dr}-hls', headers=headers)
                 elif 'package:dash' in tags or ext == 'mpd':
-                    dash_formats, dash_subs = self._extract_mpd_formats_and_subtitles(
-                        format_url, video_id, mpd_id='dash', headers=headers)
-                    formats.extend(dash_formats)
-                    subs = self._merge_subtitles(subs, dash_subs)
+                    current_formats, current_subs = self._extract_mpd_formats_and_subtitles(
+                        format_url, video_id, mpd_id=f'{dr}-dash', headers=headers)
                 elif ext == 'f4m':
                     # produce broken files
                     pass
                 else:
-                    formats.append({
+                    current_formats = [{
                         'url': format_url,
                         'width': int_or_none(playback_set.get('width')),
                         'height': int_or_none(playback_set.get('height')),
-                    })
+                    }]
             except ExtractorError as e:
                 if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
                     geo_restricted = True
                 continue
+            if tags and 'encryption:plain' not in tags:
+                for f in current_formats:
+                    f['has_drm'] = True
+            formats.extend(current_formats)
+            subs = self._merge_subtitles(subs, current_subs)
         if not formats and geo_restricted:
             self.raise_geo_restricted(countries=['IN'], metadata_available=True)
         self._sort_formats(formats)

yt_dlp/extractor/instagram.py

@@ -4,6 +4,7 @@ import itertools
 import hashlib
 import json
 import re
+import time
 
 from .common import InfoExtractor
 from ..compat import (
@@ -20,11 +21,13 @@ from ..utils import (
     try_get,
     url_or_none,
     variadic,
+    urlencode_postdata,
 )
 
 
 class InstagramIE(InfoExtractor):
     _VALID_URL = r'(?P<url>https?://(?:www\.)?instagram\.com/(?:p|tv|reel)/(?P<id>[^/?#&]+))'
+    _NETRC_MACHINE = 'instagram'
     _TESTS = [{
         'url': 'https://instagram.com/p/aye83DjauH/?foo=bar#abc',
         'md5': '0d2da106a9d2631273e192b372806516',
@@ -140,12 +143,53 @@ class InstagramIE(InfoExtractor):
         if mobj:
             return mobj.group('link')
 
+    def _login(self):
+        username, password = self._get_login_info()
+
+        login_webpage = self._download_webpage(
+            'https://www.instagram.com/accounts/login/', None,
+            note='Downloading login webpage', errnote='Failed to download login webpage')
+
+        shared_data = self._parse_json(
+            self._search_regex(
+                r'window\._sharedData\s*=\s*({.+?});',
+                login_webpage, 'shared data', default='{}'),
+            None)
+
+        login = self._download_json('https://www.instagram.com/accounts/login/ajax/', None, note='Logging in', headers={
+            'Accept': '*/*',
+            'X-IG-App-ID': '936619743392459',
+            'X-ASBD-ID': '198387',
+            'X-IG-WWW-Claim': '0',
+            'X-Requested-With': 'XMLHttpRequest',
+            'X-CSRFToken': shared_data['config']['csrf_token'],
+            'X-Instagram-AJAX': shared_data['rollout_hash'],
+            'Referer': 'https://www.instagram.com/',
+        }, data=urlencode_postdata({
+            'enc_password': f'#PWD_INSTAGRAM_BROWSER:0:{int(time.time())}:{password}',
+            'username': username,
+            'queryParams': '{}',
+            'optIntoOneTap': 'false',
+            'stopDeletionNonce': '',
+            'trustedDeviceRecords': '{}',
+        }))
+
+        if not login.get('authenticated'):
+            if login.get('message'):
+                raise ExtractorError(f'Unable to login: {login["message"]}')
+            raise ExtractorError('Unable to login')
+
+    def _real_initialize(self):
+        self._login()
+
     def _real_extract(self, url):
         mobj = self._match_valid_url(url)
         video_id = mobj.group('id')
         url = mobj.group('url')
 
-        webpage = self._download_webpage(url, video_id)
+        webpage, urlh = self._download_webpage_handle(url, video_id)
+        if 'www.instagram.com/accounts/login' in urlh.geturl().rstrip('/'):
+            self.raise_login_required('You need to log in to access this content')
 
         (media, video_url, description, thumbnail, timestamp, uploader,
          uploader_id, like_count, comment_count, comments, height,

yt_dlp/extractor/microsoftstream.py

@@ -0,0 +1,125 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from base64 import b64decode
+
+from .common import InfoExtractor
+from ..utils import (
+    merge_dicts,
+    parse_iso8601,
+    parse_duration,
+    parse_resolution,
+    try_get,
+    url_basename,
+)
+
+
+class MicrosoftStreamIE(InfoExtractor):
+    IE_NAME = 'microsoftstream'
+    IE_DESC = 'Microsoft Stream'
+    _VALID_URL = r'https?://(?:web|www|msit)\.microsoftstream\.com/video/(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
+
+    _TESTS = [{
+        'url': 'https://web.microsoftstream.com/video/6e51d928-4f46-4f1c-b141-369925e37b62?list=user&userId=f5491e02-e8fe-4e34-b67c-ec2e79a6ecc0',
+        'only_matching': True,
+    }, {
+        'url': 'https://msit.microsoftstream.com/video/b60f5987-aabd-4e1c-a42f-c559d138f2ca',
+        'only_matching': True,
+    }]
+
+    def _get_all_subtitles(self, api_url, video_id, headers):
+        subtitles = {}
+        automatic_captions = {}
+        text_tracks = self._download_json(
+            f'{api_url}/videos/{video_id}/texttracks', video_id,
+            note='Downloading subtitles JSON', fatal=False, headers=headers,
+            query={'api-version': '1.4-private'}).get('value') or []
+        for track in text_tracks:
+            if not track.get('language') or not track.get('url'):
+                continue
+            sub_dict = automatic_captions if track.get('autoGenerated') else subtitles
+            sub_dict.setdefault(track['language'], []).append({
+                'ext': 'vtt',
+                'url': track.get('url')
+            })
+        return {
+            'subtitles': subtitles,
+            'automatic_captions': automatic_captions
+        }
+
+    def extract_all_subtitles(self, *args, **kwargs):
+        if (self.get_param('writesubtitles', False)
+                or self.get_param('writeautomaticsub', False)
+                or self.get_param('listsubtitles')):
+            return self._get_all_subtitles(*args, **kwargs)
+        return {}
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+        if '<title>Microsoft Stream</title>' not in webpage:
+            self.raise_login_required(method='cookies')
+
+        access_token = self._html_search_regex(r'"AccessToken":"(.+?)"', webpage, 'access token')
+        api_url = self._html_search_regex(r'"ApiGatewayUri":"(.+?)"', webpage, 'api url')
+
+        headers = {'Authorization': f'Bearer {access_token}'}
+
+        video_data = self._download_json(
+            f'{api_url}/videos/{video_id}', video_id,
+            headers=headers, query={
+                '$expand': 'creator,tokens,status,liveEvent,extensions',
+                'api-version': '1.4-private'
+            })
+        video_id = video_data.get('id') or video_id
+        language = video_data.get('language')
+
+        thumbnails = []
+        for thumbnail_id in ('extraSmall', 'small', 'medium', 'large'):
+            thumbnail_url = try_get(video_data, lambda x: x['posterImage'][thumbnail_id]['url'], str)
+            if not thumbnail_url:
+                continue
+            thumb = {
+                'id': thumbnail_id,
+                'url': thumbnail_url,
+            }
+            thumb_name = url_basename(thumbnail_url)
+            thumb_name = str(b64decode(thumb_name + '=' * (-len(thumb_name) % 4)))
+            thumb.update(parse_resolution(thumb_name))
+            thumbnails.append(thumb)
+
+        formats = []
+        for playlist in video_data['playbackUrls']:
+            if playlist['mimeType'] == 'application/vnd.apple.mpegurl':
+                formats.extend(self._extract_m3u8_formats(
+                    playlist['playbackUrl'], video_id,
+                    ext='mp4', entry_protocol='m3u8_native', m3u8_id='hls',
+                    fatal=False, headers=headers))
+            elif playlist['mimeType'] == 'application/dash+xml':
+                formats.extend(self._extract_mpd_formats(
+                    playlist['playbackUrl'], video_id, mpd_id='dash',
+                    fatal=False, headers=headers))
+            elif playlist['mimeType'] == 'application/vnd.ms-sstr+xml':
+                formats.extend(self._extract_ism_formats(
+                    playlist['playbackUrl'], video_id, ism_id='mss',
+                    fatal=False, headers=headers))
+        formats = [merge_dicts(f, {'language': language}) for f in formats]
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': video_data['name'],
+            'description': video_data.get('description'),
+            'uploader': try_get(video_data, lambda x: x['creator']['name'], str),
+            'uploader_id': try_get(video_data, (lambda x: x['creator']['mail'],
+                                                lambda x: x['creator']['id']), str),
+            'thumbnails': thumbnails,
+            **self.extract_all_subtitles(api_url, video_id, headers),
+            'timestamp': parse_iso8601(video_data.get('created')),
+            'duration': parse_duration(try_get(video_data, lambda x: x['media']['duration'])),
+            'webpage_url': f'https://web.microsoftstream.com/video/{video_id}',
+            'view_count': try_get(video_data, lambda x: x['metrics']['views'], int),
+            'like_count': try_get(video_data, lambda x: x['metrics']['likes'], int),
+            'comment_count': try_get(video_data, lambda x: x['metrics']['comments'], int),
+            'formats': formats,
+        }

yt_dlp/extractor/niconico.py

@@ -709,11 +709,9 @@ class NicovideoSearchIE(SearchInfoExtractor, NicovideoSearchURLIE):
     _SEARCH_KEY = 'nicosearch'
     _TESTS = []
 
-    def _get_n_results(self, query, n):
-        entries = self._entries(self._proto_relative_url(f'//www.nicovideo.jp/search/{query}'), query)
-        if n < float('inf'):
-            entries = itertools.islice(entries, 0, n)
-        return self.playlist_result(entries, query, query)
+    def _search_results(self, query):
+        return self._entries(
+            self._proto_relative_url(f'//www.nicovideo.jp/search/{query}'), query)
 
 
 class NicovideoSearchDateIE(NicovideoSearchIE):

yt_dlp/extractor/novaplay.py

@@ -0,0 +1,63 @@
+# coding: utf-8
+from .common import InfoExtractor
+from ..utils import int_or_none, parse_duration, parse_iso8601
+
+
+class NovaPlayIE(InfoExtractor):
+    _VALID_URL = r'https://play.nova\.bg/video/.*/(?P<id>\d+)'
+    _TESTS = [
+        {
+            'url': 'https://play.nova.bg/video/bratya/season-3/bratq-2021-10-08/548677',
+            'md5': 'b1127a84e61bed1632b7c2ca9cbb4153',
+            'info_dict': {
+                'id': '548677',
+                'ext': 'mp4',
+                'title': 'Братя',
+                'alt_title': 'bratya/season-3/bratq-2021-10-08',
+                'duration': 1603.0,
+                'timestamp': 1633724150,
+                'upload_date': '20211008',
+                'thumbnail': 'https://nbg-img.fite.tv/img/548677_460x260.jpg',
+                'description': 'Сезон 3 Епизод 25'
+            },
+        },
+        {
+            'url': 'https://play.nova.bg/video/igri-na-volqta/season-3/igri-na-volqta-2021-09-20-1/548227',
+            'md5': '5fd61b8ecbe582fc021019d570965d58',
+            'info_dict': {
+                'id': '548227',
+                'ext': 'mp4',
+                'title': 'Игри на волята: България (20.09.2021) - част 1',
+                'alt_title': 'gri-na-volqta/season-3/igri-na-volqta-2021-09-20-1',
+                'duration': 4060.0,
+                'timestamp': 1632167564,
+                'upload_date': '20210920',
+                'thumbnail': 'https://nbg-img.fite.tv/img/548227_460x260.jpg',
+                'description': 'Сезон 3 Епизод 13'
+            },
+        }
+    ]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        webpage = self._download_webpage(url, video_id)
+        video_props = self._parse_json(self._search_regex(
+            r'<script\s?id=\"__NEXT_DATA__\"\s?type=\"application/json\">({.+})</script>',
+            webpage, 'video_props'), video_id)['props']['pageProps']['video']
+        m3u8_url = self._download_json(
+            f'https://nbg-api.fite.tv/api/v2/videos/{video_id}/streams',
+            video_id, headers={'x-flipps-user-agent': 'Flipps/75/9.7'})[0]['url']
+        formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4', m3u8_id='hls')
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': video_props['title'],
+            'alt_title': video_props.get('slug'),
+            'thumbnail': self._og_search_thumbnail(webpage),
+            'description': self._og_search_description(webpage),
+            'formats': formats,
+            'duration': parse_duration(video_props['duration']),
+            'timestamp': parse_iso8601(video_props['published_at']),
+            'view_count': int_or_none(video_props['view_count']),
+        }

yt_dlp/extractor/on24.py

@@ -0,0 +1,91 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+    int_or_none,
+    strip_or_none,
+    try_get,
+    urljoin,
+)
+
+
+class On24IE(InfoExtractor):
+    IE_NAME = 'on24'
+    IE_DESC = 'ON24'
+
+    _VALID_URL = r'''(?x)
+                    https?://event\.on24\.com/(?:
+                        wcc/r/(?P<id_1>\d{7})/(?P<key_1>[0-9A-F]{32})|
+                        eventRegistration/(?:console/EventConsoleApollo|EventLobbyServlet\?target=lobby30)
+                            \.jsp\?(?:[^/#?]*&)?eventid=(?P<id_2>\d{7})[^/#?]*&key=(?P<key_2>[0-9A-F]{32})
+                    )'''
+
+    _TESTS = [{
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?uimode=nextgeneration&eventid=2197467&sessionid=1&key=5DF57BE53237F36A43B478DD36277A84&contenttype=A&eventuserid=305999&playerwidth=1000&playerheight=650&caller=previewLobby&text_language_id=en&format=fhaudio&newConsole=false',
+        'info_dict': {
+            'id': '2197467',
+            'ext': 'wav',
+            'title': 'Pearson Test of English General/Pearson English International Certificate Teacher Training Guide',
+            'upload_date': '20200219',
+            'timestamp': 1582149600.0,
+            'view_count': int,
+        }
+    }, {
+        'url': 'https://event.on24.com/wcc/r/2639291/82829018E813065A122363877975752E?mode=login&email=johnsmith@gmail.com',
+        'only_matching': True,
+    }, {
+        'url': 'https://event.on24.com/eventRegistration/console/EventConsoleApollo.jsp?&eventid=2639291&sessionid=1&username=&partnerref=&format=fhvideo1&mobile=&flashsupportedmobiledevice=&helpcenter=&key=82829018E813065A122363877975752E&newConsole=true&nxChe=true&newTabCon=true&text_language_id=en&playerwidth=748&playerheight=526&eventuserid=338788762&contenttype=A&mediametricsessionid=384764716&mediametricid=3558192&usercd=369267058&mode=launch',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        mobj = self._match_valid_url(url)
+        event_id = mobj.group('id_1') or mobj.group('id_2')
+        event_key = mobj.group('key_1') or mobj.group('key_2')
+
+        event_data = self._download_json(
+            'https://event.on24.com/apic/utilApp/EventConsoleCachedServlet',
+            event_id, query={
+                'eventId': event_id,
+                'displayProfile': 'player',
+                'key': event_key,
+                'contentType': 'A'
+            })
+        event_id = str(try_get(event_data, lambda x: x['presentationLogInfo']['eventid'])) or event_id
+        language = event_data.get('localelanguagecode')
+
+        formats = []
+        for media in event_data.get('mediaUrlInfo', []):
+            media_url = urljoin('https://event.on24.com/media/news/corporatevideo/events/', str(media.get('url')))
+            if not media_url:
+                continue
+            media_type = media.get('code')
+            if media_type == 'fhvideo1':
+                formats.append({
+                    'format_id': 'video',
+                    'url': media_url,
+                    'language': language,
+                    'ext': 'mp4',
+                    'vcodec': 'avc1.640020',
+                    'acodec': 'mp4a.40.2',
+                })
+            elif media_type == 'audio':
+                formats.append({
+                    'format_id': 'audio',
+                    'url': media_url,
+                    'language': language,
+                    'ext': 'wav',
+                    'vcodec': 'none',
+                    'acodec': 'wav'
+                })
+        self._sort_formats(formats)
+
+        return {
+            'id': event_id,
+            'title': strip_or_none(event_data.get('description')),
+            'timestamp': int_or_none(try_get(event_data, lambda x: x['session']['startdate']), 1000),
+            'webpage_url': f'https://event.on24.com/wcc/r/{event_id}/{event_key}',
+            'view_count': event_data.get('registrantcount'),
+            'formats': formats,
+        }

yt_dlp/extractor/openload.py

@@ -17,7 +17,7 @@ from ..utils import (
     get_exe_version,
     is_outdated_version,
     std_headers,
-    process_communicate_or_kill,
+    Popen,
 )
 
 
@@ -223,11 +223,10 @@ class PhantomJSwrapper(object):
         else:
             self.extractor.to_screen('%s: %s' % (video_id, note2))
 
-        p = subprocess.Popen([
-            self.exe, '--ssl-protocol=any',
-            self._TMP_FILES['script'].name
-        ], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        out, err = process_communicate_or_kill(p)
+        p = Popen(
+            [self.exe, '--ssl-protocol=any', self._TMP_FILES['script'].name],
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        out, err = p.communicate_or_kill()
         if p.returncode != 0:
             raise ExtractorError(
                 'Executing JS failed\n:' + encodeArgument(err))

yt_dlp/extractor/patreon.py

@@ -161,7 +161,7 @@ class PatreonIE(InfoExtractor):
             if try_get(attributes, lambda x: x['embed']['provider']) == 'Vimeo':
                 embed_html = try_get(attributes, lambda x: x['embed']['html'])
                 v_url = url_or_none(compat_urllib_parse_unquote(
-                    self._search_regex(r'src=(https%3A%2F%2Fplayer\.vimeo\.com.+)%3F', embed_html, 'vimeo url', fatal=False)))
+                    self._search_regex(r'(https(?:%3A%2F%2F|://)player\.vimeo\.com.+app_id(?:=|%3D)+\d+)', embed_html, 'vimeo url', fatal=False)))
                 if v_url:
                     info.update({
                         '_type': 'url_transparent',

yt_dlp/extractor/sevenplus.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import json
 import re
 
 from .brightcove import BrightcoveNewIE
@@ -42,9 +43,52 @@ class SevenPlusIE(BrightcoveNewIE):
         'only_matching': True,
     }]
 
+    def _real_initialize(self):
+        self.token = None
+
+        cookies = self._get_cookies('https://7plus.com.au')
+        api_key = next((x for x in cookies if x.startswith('glt_')), '')[4:]
+        if not api_key:  # Cookies are signed out, skip login
+            return
+
+        login_resp = self._download_json(
+            'https://login.7plus.com.au/accounts.getJWT', None, 'Logging in', fatal=False,
+            query={
+                'APIKey': api_key,
+                'sdk': 'js_latest',
+                'login_token': cookies[f'glt_{api_key}'].value,
+                'authMode': 'cookie',
+                'pageURL': 'https://7plus.com.au/',
+                'sdkBuild': '12471',
+                'format': 'json',
+            }) or {}
+
+        if 'errorMessage' in login_resp:
+            self.report_warning(f'Unable to login: 7plus said: {login_resp["errorMessage"]}')
+            return
+        id_token = login_resp.get('id_token')
+        if not id_token:
+            self.report_warning('Unable to login: Could not extract id token')
+            return
+
+        token_resp = self._download_json(
+            'https://7plus.com.au/auth/token', None, 'Getting auth token', fatal=False,
+            headers={'Content-Type': 'application/json'}, data=json.dumps({
+                'idToken': id_token,
+                'platformId': 'web',
+                'regSource': '7plus',
+            }).encode('utf-8')) or {}
+        self.token = token_resp.get('token')
+        if not self.token:
+            self.report_warning('Unable to log in: Could not extract auth token')
+
     def _real_extract(self, url):
         path, episode_id = self._match_valid_url(url).groups()
 
+        headers = {}
+        if self.token:
+            headers['Authorization'] = f'Bearer {self.token}'
+
         try:
             media = self._download_json(
                 'https://videoservice.swm.digital/playback', episode_id, query={
@@ -55,7 +99,7 @@ class SevenPlusIE(BrightcoveNewIE):
                     'referenceId': 'ref:' + episode_id,
                     'deliveryId': 'csai',
                     'videoType': 'vod',
-                })['media']
+                }, headers=headers)['media']
         except ExtractorError as e:
             if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
                 raise ExtractorError(self._parse_json(

yt_dlp/extractor/skynewsau.py

@@ -0,0 +1,46 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+    try_get,
+    unified_strdate,
+)
+
+
+class SkyNewsAUIE(InfoExtractor):
+    _VALID_URL = r'(?:https?://)(?:www\.)?skynews\.com\.au/[^/]+/[^/]+/[^/]+/video/(?P<id>[a-z0-9]+)'
+
+    _TESTS = [{
+        'url': 'https://www.skynews.com.au/world-news/united-states/incredible-vision-shows-lava-overflowing-from-spains-la-palma-volcano/video/0f4c6243d6903502c01251f228b91a71',
+        'info_dict': {
+            'id': '6277184925001',
+            'ext': 'mp4',
+            'title': 'md5:60594f1ea6d5ae93e292900f4d34e9ae',
+            'description': 'md5:60594f1ea6d5ae93e292900f4d34e9ae',
+            'thumbnail': r're:^https?://.*\.jpg',
+            'duration': 76.394,
+            'timestamp': 1634271300,
+            'uploader_id': '5348771529001',
+            'tags': ['fblink', 'msn', 'usa', 'world', 'yt'],
+            'upload_date': '20211015',
+        },
+        'params': {'skip_download': True, 'format': 'bv'}
+    }]
+
+    _API_KEY = '6krsj3w249nk779d8fukqx9f'
+
+    def _real_extract(self, url):
+        id = self._match_id(url)
+        webpage = self._download_webpage(url, id)
+        embedcode = self._search_regex(r'embedcode\s?=\s?\"([^\"]+)\"', webpage, 'embedcode')
+        data_json = self._download_json(
+            f'https://content.api.news/v3/videos/brightcove/{embedcode}?api_key={self._API_KEY}', id)['content']
+        return {
+            'id': id,
+            '_type': 'url_transparent',
+            'url': 'https://players.brightcove.net/%s/default_default/index.html?videoId=%s' % tuple(embedcode.split('-')),
+            'ie_key': 'BrightcoveNew',
+            'title': data_json.get('caption'),
+            'upload_date': unified_strdate(try_get(data_json, lambda x: x['date']['created'])),
+        }

yt_dlp/extractor/soundcloud.py

@@ -855,7 +855,7 @@ class SoundcloudPlaylistIE(SoundcloudPlaylistBaseIE):
 
 class SoundcloudSearchIE(SearchInfoExtractor, SoundcloudIE):
     IE_NAME = 'soundcloud:search'
-    IE_DESC = 'Soundcloud search'
+    IE_DESC = 'Soundcloud search, "scsearch" keyword'
     _MAX_RESULTS = float('inf')
     _TESTS = [{
         'url': 'scsearch15:post-avant jazzcore',
@@ -880,25 +880,14 @@ class SoundcloudSearchIE(SearchInfoExtractor, SoundcloudIE):
         })
         next_url = update_url_query(self._API_V2_BASE + endpoint, query)
 
-        collected_results = 0
-
         for i in itertools.count(1):
             response = self._download_json(
-                next_url, collection_id, 'Downloading page {0}'.format(i),
+                next_url, collection_id, f'Downloading page {i}',
                 'Unable to download API page', headers=self._HEADERS)
 
-            collection = response.get('collection', [])
-            if not collection:
-                break
-
-            collection = list(filter(bool, collection))
-            collected_results += len(collection)
-
-            for item in collection:
-                yield self.url_result(item['uri'], SoundcloudIE.ie_key())
-
-            if not collection or collected_results >= limit:
-                break
+            for item in response.get('collection') or []:
+                if item:
+                    yield self.url_result(item['uri'], SoundcloudIE.ie_key())
 
             next_url = response.get('next_href')
             if not next_url:
@@ -906,4 +895,4 @@ class SoundcloudSearchIE(SearchInfoExtractor, SoundcloudIE):
 
     def _get_n_results(self, query, n):
         tracks = self._get_collection('search/tracks', query, limit=n, q=query)
-        return self.playlist_result(tracks, playlist_title=query)
+        return self.playlist_result(tracks, query, query)

yt_dlp/extractor/tagesschau.py

@@ -5,177 +5,63 @@ import re
 
 from .common import InfoExtractor
 from ..utils import (
-    determine_ext,
     js_to_json,
-    parse_iso8601,
-    parse_filesize,
+    extract_attributes,
+    try_get,
+    int_or_none,
 )
 
 
-class TagesschauPlayerIE(InfoExtractor):
-    IE_NAME = 'tagesschau:player'
-    _VALID_URL = r'https?://(?:www\.)?tagesschau\.de/multimedia/(?P<kind>audio|video)/(?P=kind)-(?P<id>\d+)~player(?:_[^/?#&]+)?\.html'
-
-    _TESTS = [{
-        'url': 'http://www.tagesschau.de/multimedia/video/video-179517~player.html',
-        'md5': '8d09548d5c15debad38bee3a4d15ca21',
-        'info_dict': {
-            'id': '179517',
-            'ext': 'mp4',
-            'title': 'Marie Kristin Boese, ARD Berlin, über den zukünftigen Kurs der AfD',
-            'thumbnail': r're:^https?:.*\.jpg$',
-            'formats': 'mincount:6',
-        },
-    }, {
-        'url': 'https://www.tagesschau.de/multimedia/audio/audio-29417~player.html',
-        'md5': '76e6eec6ebd40740671cf0a2c88617e5',
-        'info_dict': {
-            'id': '29417',
-            'ext': 'mp3',
-            'title': 'Trabi - Bye, bye Rennpappe',
-            'thumbnail': r're:^https?:.*\.jpg$',
-            'formats': 'mincount:2',
-        },
-    }, {
-        'url': 'http://www.tagesschau.de/multimedia/audio/audio-29417~player_autoplay-true.html',
-        'only_matching': True,
-    }]
-
-    _FORMATS = {
-        'xs': {'quality': 0},
-        's': {'width': 320, 'height': 180, 'quality': 1},
-        'm': {'width': 512, 'height': 288, 'quality': 2},
-        'l': {'width': 960, 'height': 540, 'quality': 3},
-        'xl': {'width': 1280, 'height': 720, 'quality': 4},
-        'xxl': {'quality': 5},
-    }
-
-    def _extract_via_api(self, kind, video_id):
-        info = self._download_json(
-            'https://www.tagesschau.de/api/multimedia/{0}/{0}-{1}.json'.format(kind, video_id),
-            video_id)
-        title = info['headline']
-        formats = []
-        for media in info['mediadata']:
-            for format_id, format_url in media.items():
-                if determine_ext(format_url) == 'm3u8':
-                    formats.extend(self._extract_m3u8_formats(
-                        format_url, video_id, 'mp4',
-                        entry_protocol='m3u8_native', m3u8_id='hls'))
-                else:
-                    formats.append({
-                        'url': format_url,
-                        'format_id': format_id,
-                        'vcodec': 'none' if kind == 'audio' else None,
-                    })
-        self._sort_formats(formats)
-        timestamp = parse_iso8601(info.get('date'))
-        return {
-            'id': video_id,
-            'title': title,
-            'timestamp': timestamp,
-            'formats': formats,
-        }
-
-    def _real_extract(self, url):
-        mobj = self._match_valid_url(url)
-        video_id = mobj.group('id')
-
-        # kind = mobj.group('kind').lower()
-        # if kind == 'video':
-        #     return self._extract_via_api(kind, video_id)
-
-        # JSON api does not provide some audio formats (e.g. ogg) thus
-        # extracting audio via webpage
-
-        webpage = self._download_webpage(url, video_id)
-
-        title = self._og_search_title(webpage).strip()
-        formats = []
-
-        for media_json in re.findall(r'({src\s*:\s*["\']http[^}]+type\s*:[^}]+})', webpage):
-            media = self._parse_json(js_to_json(media_json), video_id, fatal=False)
-            if not media:
-                continue
-            src = media.get('src')
-            if not src:
-                return
-            quality = media.get('quality')
-            kind = media.get('type', '').split('/')[0]
-            ext = determine_ext(src)
-            f = {
-                'url': src,
-                'format_id': '%s_%s' % (quality, ext) if quality else ext,
-                'ext': ext,
-                'vcodec': 'none' if kind == 'audio' else None,
-            }
-            f.update(self._FORMATS.get(quality, {}))
-            formats.append(f)
-
-        self._sort_formats(formats)
-
-        thumbnail = self._og_search_thumbnail(webpage)
-
-        return {
-            'id': video_id,
-            'title': title,
-            'thumbnail': thumbnail,
-            'formats': formats,
-        }
-
-
 class TagesschauIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?tagesschau\.de/(?P<path>[^/]+/(?:[^/]+/)*?(?P<id>[^/#?]+?(?:-?[0-9]+)?))(?:~_?[^/#?]+?)?\.html'
 
     _TESTS = [{
         'url': 'http://www.tagesschau.de/multimedia/video/video-102143.html',
-        'md5': 'f7c27a0eff3bfe8c7727e65f8fe1b1e6',
+        'md5': '7a7287612fa881a1ae1d087df45c2fd6',
         'info_dict': {
-            'id': 'video-102143',
+            'id': 'video-102143-1',
             'ext': 'mp4',
             'title': 'Regierungsumbildung in Athen: Neue Minister in Griechenland vereidigt',
-            'description': '18.07.2015 20:10 Uhr',
-            'thumbnail': r're:^https?:.*\.jpg$',
         },
     }, {
         'url': 'http://www.tagesschau.de/multimedia/sendung/ts-5727.html',
         'md5': '3c54c1f6243d279b706bde660ceec633',
         'info_dict': {
-            'id': 'ts-5727',
+            'id': 'ts-5727-1',
             'ext': 'mp4',
-            'title': 'Sendung: tagesschau \t04.12.2014 20:00 Uhr',
-            'description': 'md5:695c01bfd98b7e313c501386327aea59',
-            'thumbnail': r're:^https?:.*\.jpg$',
+            'title': 'Ganze Sendung',
         },
     }, {
         # exclusive audio
         'url': 'http://www.tagesschau.de/multimedia/audio/audio-29417.html',
-        'md5': '76e6eec6ebd40740671cf0a2c88617e5',
+        'md5': '4cf22023c285f35e99c24d290ba58cc9',
         'info_dict': {
-            'id': 'audio-29417',
+            'id': 'audio-29417-1',
             'ext': 'mp3',
-            'title': 'Trabi - Bye, bye Rennpappe',
-            'description': 'md5:8687dda862cbbe2cfb2df09b56341317',
-            'thumbnail': r're:^https?:.*\.jpg$',
+            'title': 'Brasilianischer Präsident Bolsonaro unter Druck: Corona-Bericht wird vorgestellt',
         },
     }, {
-        # audio in article
         'url': 'http://www.tagesschau.de/inland/bnd-303.html',
-        'md5': 'e0916c623e85fc1d2b26b78f299d3958',
+        'md5': '12cfb212d9325b5ba0d52b625f1aa61c',
         'info_dict': {
-            'id': 'bnd-303',
-            'ext': 'mp3',
-            'title': 'Viele Baustellen für neuen BND-Chef',
-            'description': 'md5:1e69a54be3e1255b2b07cdbce5bcd8b4',
-            'thumbnail': r're:^https?:.*\.jpg$',
+            'id': 'bnd-303-1',
+            'ext': 'mp4',
+            'title': 'SPD-Gruppenbild mit Bärbel Bas nach der Fraktionssitzung | dpa',
         },
     }, {
         'url': 'http://www.tagesschau.de/inland/afd-parteitag-135.html',
         'info_dict': {
             'id': 'afd-parteitag-135',
-            'title': 'Möchtegern-Underdog mit Machtanspruch',
+            'title': 'AfD',
+        },
+        'playlist_count': 20,
+    }, {
+        'url': 'https://www.tagesschau.de/multimedia/audio/audio-29417~player.html',
+        'info_dict': {
+            'id': 'audio-29417-1',
+            'ext': 'mp3',
+            'title': 'Brasilianischer Präsident Bolsonaro unter Druck: Corona-Bericht wird vorgestellt',
         },
-        'playlist_count': 2,
     }, {
         'url': 'http://www.tagesschau.de/multimedia/sendung/tsg-3771.html',
         'only_matching': True,
@@ -206,62 +92,6 @@ class TagesschauIE(InfoExtractor):
         'only_matching': True,
     }]
 
-    @classmethod
-    def suitable(cls, url):
-        return False if TagesschauPlayerIE.suitable(url) else super(TagesschauIE, cls).suitable(url)
-
-    def _extract_formats(self, download_text, media_kind):
-        links = re.finditer(
-            r'<div class="button" title="(?P<title>[^"]*)"><a href="(?P<url>[^"]+)">(?P<name>.+?)</a></div>',
-            download_text)
-        formats = []
-        for l in links:
-            link_url = l.group('url')
-            if not link_url:
-                continue
-            format_id = self._search_regex(
-                r'.*/[^/.]+\.([^/]+)\.[^/.]+$', link_url, 'format ID',
-                default=determine_ext(link_url))
-            format = {
-                'format_id': format_id,
-                'url': l.group('url'),
-                'format_name': l.group('name'),
-            }
-            title = l.group('title')
-            if title:
-                if media_kind.lower() == 'video':
-                    m = re.match(
-                        r'''(?x)
-                            Video:\s*(?P<vcodec>[a-zA-Z0-9/._-]+)\s*&\#10;
-                            (?P<width>[0-9]+)x(?P<height>[0-9]+)px&\#10;
-                            (?P<vbr>[0-9]+)kbps&\#10;
-                            Audio:\s*(?P<abr>[0-9]+)kbps,\s*(?P<audio_desc>[A-Za-z\.0-9]+)&\#10;
-                            Gr&ouml;&szlig;e:\s*(?P<filesize_approx>[0-9.,]+\s+[a-zA-Z]*B)''',
-                        title)
-                    if m:
-                        format.update({
-                            'format_note': m.group('audio_desc'),
-                            'vcodec': m.group('vcodec'),
-                            'width': int(m.group('width')),
-                            'height': int(m.group('height')),
-                            'abr': int(m.group('abr')),
-                            'vbr': int(m.group('vbr')),
-                            'filesize_approx': parse_filesize(m.group('filesize_approx')),
-                        })
-                else:
-                    m = re.match(
-                        r'(?P<format>.+?)-Format\s*:\s*(?P<abr>\d+)kbps\s*,\s*(?P<note>.+)',
-                        title)
-                    if m:
-                        format.update({
-                            'format_note': '%s, %s' % (m.group('format'), m.group('note')),
-                            'vcodec': 'none',
-                            'abr': int(m.group('abr')),
-                        })
-            formats.append(format)
-        self._sort_formats(formats)
-        return formats
-
     def _real_extract(self, url):
         mobj = self._match_valid_url(url)
         video_id = mobj.group('id') or mobj.group('path')
@@ -271,34 +101,46 @@ class TagesschauIE(InfoExtractor):
 
         title = self._html_search_regex(
             r'<span[^>]*class="headline"[^>]*>(.+?)</span>',
-            webpage, 'title', default=None) or self._og_search_title(webpage)
+            webpage, 'title', default=None) or self._og_search_title(webpage, fatal=False)
 
-        DOWNLOAD_REGEX = r'(?s)<p>Wir bieten dieses (?P<kind>Video|Audio) in folgenden Formaten zum Download an:</p>\s*<div class="controls">(?P<links>.*?)</div>\s*<p>'
-
-        webpage_type = self._og_search_property('type', webpage, default=None)
-        if webpage_type == 'website':  # Article
-            entries = []
-            for num, (entry_title, media_kind, download_text) in enumerate(re.findall(
-                    r'(?s)<p[^>]+class="infotext"[^>]*>\s*(?:<a[^>]+>)?\s*<strong>(.+?)</strong>.*?</p>.*?%s' % DOWNLOAD_REGEX,
-                    webpage), 1):
+        entries = []
+        videos = re.findall(r'<div[^>]+>', webpage)
+        num = 0
+        for video in videos:
+            video = extract_attributes(video).get('data-config')
+            if not video:
+                continue
+            video = self._parse_json(video, video_id, transform_source=js_to_json, fatal=False)
+            video_formats = try_get(video, lambda x: x['mc']['_mediaArray'][0]['_mediaStreamArray'])
+            if not video_formats:
+                continue
+            num += 1
+            for video_format in video_formats:
+                media_url = video_format.get('_stream') or ''
+                formats = []
+                if media_url.endswith('master.m3u8'):
+                    formats = self._extract_m3u8_formats(media_url, video_id, 'mp4', m3u8_id='hls')
+                elif media_url.endswith('.hi.mp3') and media_url.startswith('https://download'):
+                    formats = [{
+                        'url': media_url,
+                        'vcodec': 'none',
+                    }]
+                if not formats:
+                    continue
                 entries.append({
                     'id': '%s-%d' % (display_id, num),
-                    'title': '%s' % entry_title,
-                    'formats': self._extract_formats(download_text, media_kind),
+                    'title': try_get(video, lambda x: x['mc']['_title']),
+                    'duration': int_or_none(try_get(video, lambda x: x['mc']['_duration'])),
+                    'formats': formats
                 })
-            if len(entries) > 1:
-                return self.playlist_result(entries, display_id, title)
-            formats = entries[0]['formats']
-        else:  # Assume single video
-            download_text = self._search_regex(
-                DOWNLOAD_REGEX, webpage, 'download links', group='links')
-            media_kind = self._search_regex(
-                DOWNLOAD_REGEX, webpage, 'media kind', default='Video', group='kind')
-            formats = self._extract_formats(download_text, media_kind)
-        thumbnail = self._og_search_thumbnail(webpage)
-        description = self._html_search_regex(
-            r'(?s)<p class="teasertext">(.*?)</p>',
-            webpage, 'description', default=None)
+        if len(entries) > 1:
+            return self.playlist_result(entries, display_id, title)
+        formats = entries[0]['formats']
+        video_info = self._search_json_ld(webpage, video_id)
+        description = video_info.get('description')
+        thumbnail = self._og_search_thumbnail(webpage) or video_info.get('thumbnail')
+        timestamp = video_info.get('timestamp')
+        title = title or video_info.get('description')
 
         self._sort_formats(formats)
 
@@ -307,5 +149,6 @@ class TagesschauIE(InfoExtractor):
             'title': title,
             'thumbnail': thumbnail,
             'formats': formats,
+            'timestamp': timestamp,
             'description': description,
         }

yt_dlp/extractor/tbs.py

@@ -16,7 +16,7 @@ from ..utils import (
 
 
 class TBSIE(TurnerBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?(?P<site>tbs|tntdrama)\.com(?P<path>/(?:movies|watchtnt|shows/[^/]+/(?:clips|season-\d+/episode-\d+))/(?P<id>[^/?#]+))'
+    _VALID_URL = r'https?://(?:www\.)?(?P<site>tbs|tntdrama)\.com(?P<path>/(?:movies|watchtnt|watchtbs|shows/[^/]+/(?:clips|season-\d+/episode-\d+))/(?P<id>[^/?#]+))'
     _TESTS = [{
         'url': 'http://www.tntdrama.com/shows/the-alienist/clips/monster',
         'info_dict': {
@@ -45,7 +45,7 @@ class TBSIE(TurnerBaseIE):
         drupal_settings = self._parse_json(self._search_regex(
             r'<script[^>]+?data-drupal-selector="drupal-settings-json"[^>]*?>({.+?})</script>',
             webpage, 'drupal setting'), display_id)
-        isLive = 'watchtnt' in path
+        isLive = 'watchtnt' in path or 'watchtbs' in path
         video_data = next(v for v in drupal_settings['turner_playlist'] if isLive or v.get('url') == path)
 
         media_id = video_data['mediaID']

yt_dlp/extractor/tiktok.py

@@ -208,7 +208,7 @@ class TikTokBaseIE(InfoExtractor):
             'duration': int_or_none(traverse_obj(video_info, 'duration', ('download_addr', 'duration')), scale=1000)
         }
 
-    def _parse_aweme_video_web(self, aweme_detail, webpage, url):
+    def _parse_aweme_video_web(self, aweme_detail, webpage_url):
         video_info = aweme_detail['video']
         author_info = traverse_obj(aweme_detail, 'author', 'authorInfo', default={})
         music_info = aweme_detail.get('music') or {}
@@ -277,7 +277,7 @@ class TikTokBaseIE(InfoExtractor):
             'thumbnails': thumbnails,
             'description': str_or_none(aweme_detail.get('desc')),
             'http_headers': {
-                'Referer': url
+                'Referer': webpage_url
             }
         }
 
@@ -287,18 +287,18 @@ class TikTokIE(TikTokBaseIE):
 
     _TESTS = [{
         'url': 'https://www.tiktok.com/@leenabhushan/video/6748451240264420610',
-        'md5': '34a7543afd5a151b0840ba6736fb633b',
+        'md5': '736bb7a466c6f0a6afeb597da1e6f5b7',
         'info_dict': {
             'id': '6748451240264420610',
             'ext': 'mp4',
             'title': '#jassmanak #lehanga #leenabhushan',
             'description': '#jassmanak #lehanga #leenabhushan',
             'duration': 13,
-            'height': 1280,
-            'width': 720,
+            'height': 1024,
+            'width': 576,
             'uploader': 'leenabhushan',
             'uploader_id': '6691488002098119685',
-            'uploader_url': 'https://www.tiktok.com/@leenabhushan',
+            'uploader_url': 'https://www.tiktok.com/@MS4wLjABAAAA_Eb4t1vodM1IuTy_cvp9CY22RAb59xqrO0Xtz9CYQJvgXaDvZxYnZYRzDWhhgJmy',
             'creator': 'facestoriesbyleenabh',
             'thumbnail': r're:^https?://[\w\/\.\-]+(~[\w\-]+\.image)?',
             'upload_date': '20191016',
@@ -310,7 +310,7 @@ class TikTokIE(TikTokBaseIE):
         }
     }, {
         'url': 'https://www.tiktok.com/@patroxofficial/video/6742501081818877190?langCountry=en',
-        'md5': '06b9800d47d5fe51a19e322dd86e61c9',
+        'md5': '6f3cf8cdd9b28cb8363fe0a9a160695b',
         'info_dict': {
             'id': '6742501081818877190',
             'ext': 'mp4',
@@ -321,7 +321,7 @@ class TikTokIE(TikTokBaseIE):
             'width': 540,
             'uploader': 'patrox',
             'uploader_id': '18702747',
-            'uploader_url': 'https://www.tiktok.com/@patrox',
+            'uploader_url': 'https://www.tiktok.com/@MS4wLjABAAAAiFnldaILebi5heDoVU6bn4jBWWycX6-9U3xuNPqZ8Ws',
             'creator': 'patroX',
             'thumbnail': r're:^https?://[\w\/\.\-]+(~[\w\-]+\.image)?',
             'upload_date': '20190930',
@@ -362,7 +362,7 @@ class TikTokIE(TikTokBaseIE):
         # Chech statusCode for success
         status = props_data.get('pageProps').get('statusCode')
         if status == 0:
-            return self._parse_aweme_video_web(props_data['pageProps']['itemInfo']['itemStruct'], webpage, url)
+            return self._parse_aweme_video_web(props_data['pageProps']['itemInfo']['itemStruct'], url)
         elif status == 10216:
             raise ExtractorError('This video is private', expected=True)
 
@@ -377,13 +377,17 @@ class TikTokUserIE(TikTokBaseIE):
         'playlist_mincount': 45,
         'info_dict': {
             'id': '6935371178089399301',
+            'title': 'corgibobaa',
         },
+        'expected_warnings': ['Retrying']
     }, {
         'url': 'https://www.tiktok.com/@meme',
         'playlist_mincount': 593,
         'info_dict': {
             'id': '79005827461758976',
+            'title': 'meme',
         },
+        'expected_warnings': ['Retrying']
     }]
 
     r'''  # TODO: Fix by adding _signature to api_url
@@ -430,7 +434,7 @@ class TikTokUserIE(TikTokBaseIE):
                 break
             for video in post_list.get('aweme_list', []):
                 yield {
-                    **self._parse_aweme_video(video),
+                    **self._parse_aweme_video_app(video),
                     'ie_key': TikTokIE.ie_key(),
                     'extractor': 'TikTok',
                 }
@@ -439,12 +443,12 @@ class TikTokUserIE(TikTokBaseIE):
             query['max_cursor'] = post_list['max_cursor']
 
     def _real_extract(self, url):
-        user_id = self._match_id(url)
-        webpage = self._download_webpage(url, user_id, headers={
+        user_name = self._match_id(url)
+        webpage = self._download_webpage(url, user_name, headers={
             'User-Agent': 'facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)'
         })
-        own_id = self._html_search_regex(r'snssdk\d*://user/profile/(\d+)', webpage, 'user ID')
-        return self.playlist_result(self._entries_api(webpage, own_id, user_id), user_id)
+        user_id = self._html_search_regex(r'snssdk\d*://user/profile/(\d+)', webpage, 'user ID')
+        return self.playlist_result(self._entries_api(webpage, user_id, user_name), user_id, user_name)
 
 
 class DouyinIE(TikTokIE):
@@ -556,4 +560,4 @@ class DouyinIE(TikTokIE):
         render_data = self._parse_json(
             render_data_json, video_id, transform_source=compat_urllib_parse_unquote)
         return self._parse_aweme_video_web(
-            traverse_obj(render_data, (..., 'aweme', 'detail'), get_all=False), webpage, url)
+            traverse_obj(render_data, (..., 'aweme', 'detail'), get_all=False), url)

yt_dlp/extractor/trovo.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import itertools
 import json
 
 from .common import InfoExtractor
@@ -194,3 +195,69 @@ class TrovoVodIE(TrovoBaseIE):
         }
         info.update(self._extract_streamer_info(vod_detail_info))
         return info
+
+
+class TrovoChannelBaseIE(InfoExtractor):
+    def _get_vod_json(self, page, uid):
+        raise NotImplementedError('This method must be implemented by subclasses')
+
+    def _entries(self, uid):
+        for page in itertools.count(1):
+            vod_json = self._get_vod_json(page, uid)
+            vods = vod_json.get('vodInfos', [])
+            for vod in vods:
+                yield self.url_result(
+                    'https://trovo.live/%s/%s' % (self._TYPE, vod.get('vid')),
+                    ie=TrovoVodIE.ie_key())
+            has_more = vod_json['hasMore']
+            if not has_more:
+                break
+
+    def _real_extract(self, url):
+        id = self._match_id(url)
+        uid = str(self._download_json('https://gql.trovo.live/', id, query={
+            'query': '{getLiveInfo(params:{userName:"%s"}){streamerInfo{uid}}}' % id
+        })['data']['getLiveInfo']['streamerInfo']['uid'])
+        return self.playlist_result(self._entries(uid), playlist_id=uid)
+
+
+class TrovoChannelVodIE(TrovoChannelBaseIE):
+    _VALID_URL = r'trovovod:(?P<id>[^\s]+)'
+    IE_DESC = 'All VODs of a trovo.live channel, "trovovod" keyword'
+
+    _TESTS = [{
+        'url': 'trovovod:OneTappedYou',
+        'playlist_mincount': 24,
+        'info_dict': {
+            'id': '100719456',
+        },
+    }]
+
+    _QUERY = '{getChannelLtvVideoInfos(params:{pageSize:99,currPage:%d,channelID:%s}){hasMore,vodInfos{vid}}}'
+    _TYPE = 'video'
+
+    def _get_vod_json(self, page, uid):
+        return self._download_json('https://gql.trovo.live/', uid, query={
+            'query': self._QUERY % (page, uid)
+        })['data']['getChannelLtvVideoInfos']
+
+
+class TrovoChannelClipIE(TrovoChannelBaseIE):
+    _VALID_URL = r'trovoclip:(?P<id>[^\s]+)'
+    IE_DESC = 'All Clips of a trovo.live channel, "trovoclip" keyword'
+
+    _TESTS = [{
+        'url': 'trovoclip:OneTappedYou',
+        'playlist_mincount': 29,
+        'info_dict': {
+            'id': '100719456',
+        },
+    }]
+
+    _QUERY = '{getChannelClipVideoInfos(params:{pageSize:99,currPage:%d,channelID:%s,albumType:VOD_CLIP_ALBUM_TYPE_LATEST}){hasMore,vodInfos{vid}}}'
+    _TYPE = 'clip'
+
+    def _get_vod_json(self, page, uid):
+        return self._download_json('https://gql.trovo.live/', uid, query={
+            'query': self._QUERY % (page, uid)
+        })['data']['getChannelClipVideoInfos']

yt_dlp/extractor/tvplay.py

@@ -336,8 +336,8 @@ class ViafreeIE(InfoExtractor):
     _VALID_URL = r'''(?x)
                     https?://
                         (?:www\.)?
-                        viafree\.(?P<country>dk|no|se)
-                        /(?P<id>program(?:mer)?/(?:[^/]+/)+[^/?#&]+)
+                        viafree\.(?P<country>dk|no|se|fi)
+                        /(?P<id>(?:program(?:mer)?|ohjelmat)?/(?:[^/]+/)+[^/?#&]+)
                     '''
     _TESTS = [{
         'url': 'http://www.viafree.no/programmer/underholdning/det-beste-vorspielet/sesong-2/episode-1',
@@ -389,6 +389,9 @@ class ViafreeIE(InfoExtractor):
     }, {
         'url': 'http://www.viafree.se/program/underhallning/i-like-radio-live/sasong-1/676869',
         'only_matching': True,
+    }, {
+        'url': 'https://www.viafree.fi/ohjelmat/entertainment/amazing-makeovers/kausi-7/jakso-2',
+        'only_matching': True,
     }]
     _GEO_BYPASS = False
 

yt_dlp/extractor/vimeo.py

@@ -3,7 +3,6 @@ from __future__ import unicode_literals
 
 import base64
 import functools
-import json
 import re
 import itertools
 
@@ -17,8 +16,8 @@ from ..compat import (
 from ..utils import (
     clean_html,
     determine_ext,
-    dict_get,
     ExtractorError,
+    get_element_by_class,
     js_to_json,
     int_or_none,
     merge_dicts,
@@ -26,7 +25,6 @@ from ..utils import (
     parse_filesize,
     parse_iso8601,
     parse_qs,
-    RegexNotFoundError,
     sanitized_Request,
     smuggle_url,
     std_headers,
@@ -129,10 +127,11 @@ class VimeoBaseInfoExtractor(InfoExtractor):
         video_title = video_data['title']
         live_event = video_data.get('live_event') or {}
         is_live = live_event.get('status') == 'started'
+        request = config.get('request') or {}
 
         formats = []
-        config_files = video_data.get('files') or config['request'].get('files', {})
-        for f in config_files.get('progressive', []):
+        config_files = video_data.get('files') or request.get('files') or {}
+        for f in (config_files.get('progressive') or []):
             video_url = f.get('url')
             if not video_url:
                 continue
@@ -148,7 +147,7 @@ class VimeoBaseInfoExtractor(InfoExtractor):
         # TODO: fix handling of 308 status code returned for live archive manifest requests
         sep_pattern = r'/sep/video/'
         for files_type in ('hls', 'dash'):
-            for cdn_name, cdn_data in config_files.get(files_type, {}).get('cdns', {}).items():
+            for cdn_name, cdn_data in (try_get(config_files, lambda x: x[files_type]['cdns']) or {}).items():
                 manifest_url = cdn_data.get('url')
                 if not manifest_url:
                     continue
@@ -188,17 +187,15 @@ class VimeoBaseInfoExtractor(InfoExtractor):
             })
 
         subtitles = {}
-        text_tracks = config['request'].get('text_tracks')
-        if text_tracks:
-            for tt in text_tracks:
-                subtitles[tt['lang']] = [{
-                    'ext': 'vtt',
-                    'url': urljoin('https://vimeo.com', tt['url']),
-                }]
+        for tt in (request.get('text_tracks') or []):
+            subtitles[tt['lang']] = [{
+                'ext': 'vtt',
+                'url': urljoin('https://vimeo.com', tt['url']),
+            }]
 
         thumbnails = []
         if not is_live:
-            for key, thumb in video_data.get('thumbs', {}).items():
+            for key, thumb in (video_data.get('thumbs') or {}).items():
                 thumbnails.append({
                     'id': key,
                     'width': int_or_none(key),
@@ -342,6 +339,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
                 'duration': 1595,
                 'upload_date': '20130610',
                 'timestamp': 1370893156,
+                'license': 'by',
             },
             'params': {
                 'format': 'best[protocol=https]',
@@ -420,6 +418,12 @@ class VimeoIE(VimeoBaseInfoExtractor):
                 'uploader_id': 'staff',
                 'uploader': 'Vimeo Staff',
                 'duration': 62,
+                'subtitles': {
+                    'de': [{'ext': 'vtt'}],
+                    'en': [{'ext': 'vtt'}],
+                    'es': [{'ext': 'vtt'}],
+                    'fr': [{'ext': 'vtt'}],
+                },
             }
         },
         {
@@ -626,6 +630,37 @@ class VimeoIE(VimeoBaseInfoExtractor):
     def _real_initialize(self):
         self._login()
 
+    def _extract_from_api(self, video_id, unlisted_hash=None):
+        token = self._download_json(
+            'https://vimeo.com/_rv/jwt', video_id, headers={
+                'X-Requested-With': 'XMLHttpRequest'
+            })['token']
+        api_url = 'https://api.vimeo.com/videos/' + video_id
+        if unlisted_hash:
+            api_url += ':' + unlisted_hash
+        video = self._download_json(
+            api_url, video_id, headers={
+                'Authorization': 'jwt ' + token,
+            }, query={
+                'fields': 'config_url,created_time,description,license,metadata.connections.comments.total,metadata.connections.likes.total,release_time,stats.plays',
+            })
+        info = self._parse_config(self._download_json(
+            video['config_url'], video_id), video_id)
+        self._vimeo_sort_formats(info['formats'])
+        get_timestamp = lambda x: parse_iso8601(video.get(x + '_time'))
+        info.update({
+            'description': video.get('description'),
+            'license': video.get('license'),
+            'release_timestamp': get_timestamp('release'),
+            'timestamp': get_timestamp('created'),
+            'view_count': int_or_none(try_get(video, lambda x: x['stats']['plays'])),
+        })
+        connections = try_get(
+            video, lambda x: x['metadata']['connections'], dict) or {}
+        for k in ('comment', 'like'):
+            info[k + '_count'] = int_or_none(try_get(connections, lambda x: x[k + 's']['total']))
+        return info
+
     def _try_album_password(self, url):
         album_id = self._search_regex(
             r'vimeo\.com/(?:album|showcase)/([^/]+)', url, 'album id', default=None)
@@ -675,45 +710,16 @@ class VimeoIE(VimeoBaseInfoExtractor):
         # Extract ID from URL
         video_id, unlisted_hash = self._match_valid_url(url).groups()
         if unlisted_hash:
-            token = self._download_json(
-                'https://vimeo.com/_rv/jwt', video_id, headers={
-                    'X-Requested-With': 'XMLHttpRequest'
-                })['token']
-            video = self._download_json(
-                'https://api.vimeo.com/videos/%s:%s' % (video_id, unlisted_hash),
-                video_id, headers={
-                    'Authorization': 'jwt ' + token,
-                }, query={
-                    'fields': 'config_url,created_time,description,license,metadata.connections.comments.total,metadata.connections.likes.total,release_time,stats.plays',
-                })
-            info = self._parse_config(self._download_json(
-                video['config_url'], video_id), video_id)
-            self._vimeo_sort_formats(info['formats'])
-            get_timestamp = lambda x: parse_iso8601(video.get(x + '_time'))
-            info.update({
-                'description': video.get('description'),
-                'license': video.get('license'),
-                'release_timestamp': get_timestamp('release'),
-                'timestamp': get_timestamp('created'),
-                'view_count': int_or_none(try_get(video, lambda x: x['stats']['plays'])),
-            })
-            connections = try_get(
-                video, lambda x: x['metadata']['connections'], dict) or {}
-            for k in ('comment', 'like'):
-                info[k + '_count'] = int_or_none(try_get(connections, lambda x: x[k + 's']['total']))
-            return info
+            return self._extract_from_api(video_id, unlisted_hash)
 
         orig_url = url
         is_pro = 'vimeopro.com/' in url
-        is_player = '://player.vimeo.com/video/' in url
         if is_pro:
             # some videos require portfolio_id to be present in player url
             # https://github.com/ytdl-org/youtube-dl/issues/20070
             url = self._extract_url(url, self._download_webpage(url, video_id))
             if not url:
                 url = 'https://vimeo.com/' + video_id
-        elif is_player:
-            url = 'https://player.vimeo.com/video/' + video_id
         elif any(p in url for p in ('play_redirect_hls', 'moogaloop.swf')):
             url = 'https://vimeo.com/' + video_id
 
@@ -734,14 +740,25 @@ class VimeoIE(VimeoBaseInfoExtractor):
                         expected=True)
             raise
 
-        # Now we begin extracting as much information as we can from what we
-        # retrieved. First we extract the information common to all extractors,
-        # and latter we extract those that are Vimeo specific.
-        self.report_extraction(video_id)
+        if '://player.vimeo.com/video/' in url:
+            config = self._parse_json(self._search_regex(
+                r'\bconfig\s*=\s*({.+?})\s*;', webpage, 'info section'), video_id)
+            if config.get('view') == 4:
+                config = self._verify_player_video_password(
+                    redirect_url, video_id, headers)
+            info = self._parse_config(config, video_id)
+            self._vimeo_sort_formats(info['formats'])
+            return info
+
+        if re.search(r'<form[^>]+?id="pw_form"', webpage):
+            video_password = self._get_video_password()
+            token, vuid = self._extract_xsrft_and_vuid(webpage)
+            webpage = self._verify_video_password(
+                redirect_url, video_id, video_password, token, vuid)
 
         vimeo_config = self._extract_vimeo_config(webpage, video_id, default=None)
         if vimeo_config:
-            seed_status = vimeo_config.get('seed_status', {})
+            seed_status = vimeo_config.get('seed_status') or {}
             if seed_status.get('state') == 'failed':
                 raise ExtractorError(
                     '%s said: %s' % (self.IE_NAME, seed_status['title']),
@@ -750,70 +767,40 @@ class VimeoIE(VimeoBaseInfoExtractor):
         cc_license = None
         timestamp = None
         video_description = None
+        info_dict = {}
 
-        # Extract the config JSON
-        try:
-            try:
-                config_url = self._html_search_regex(
-                    r' data-config-url="(.+?)"', webpage,
-                    'config URL', default=None)
-                if not config_url:
-                    # Sometimes new react-based page is served instead of old one that require
-                    # different config URL extraction approach (see
-                    # https://github.com/ytdl-org/youtube-dl/pull/7209)
-                    page_config = self._parse_json(self._search_regex(
-                        r'vimeo\.(?:clip|vod_title)_page_config\s*=\s*({.+?});',
-                        webpage, 'page config'), video_id)
-                    config_url = page_config['player']['config_url']
-                    cc_license = page_config.get('cc_license')
-                    timestamp = try_get(
-                        page_config, lambda x: x['clip']['uploaded_on'],
-                        compat_str)
-                    video_description = clean_html(dict_get(
-                        page_config, ('description', 'description_html_escaped')))
-                config = self._download_json(config_url, video_id)
-            except RegexNotFoundError:
-                # For pro videos or player.vimeo.com urls
-                # We try to find out to which variable is assigned the config dic
-                m_variable_name = re.search(r'(\w)\.video\.id', webpage)
-                if m_variable_name is not None:
-                    config_re = [r'%s=({[^}].+?});' % re.escape(m_variable_name.group(1))]
-                else:
-                    config_re = [r' = {config:({.+?}),assets:', r'(?:[abc])=({.+?});']
-                config_re.append(r'\bvar\s+r\s*=\s*({.+?})\s*;')
-                config_re.append(r'\bconfig\s*=\s*({.+?})\s*;')
-                config = self._search_regex(config_re, webpage, 'info section',
-                                            flags=re.DOTALL)
-                config = json.loads(config)
-        except Exception as e:
-            if re.search('The creator of this video has not given you permission to embed it on this domain.', webpage):
-                raise ExtractorError('The author has restricted the access to this video, try with the "--referer" option')
-
-            if re.search(r'<form[^>]+?id="pw_form"', webpage) is not None:
-                if '_video_password_verified' in data:
-                    raise ExtractorError('video password verification failed!')
-                video_password = self._get_video_password()
-                token, vuid = self._extract_xsrft_and_vuid(webpage)
-                self._verify_video_password(
-                    redirect_url, video_id, video_password, token, vuid)
-                return self._real_extract(
-                    smuggle_url(redirect_url, {'_video_password_verified': 'verified'}))
-            else:
-                raise ExtractorError('Unable to extract info section',
-                                     cause=e)
+        channel_id = self._search_regex(
+            r'vimeo\.com/channels/([^/]+)', url, 'channel id', default=None)
+        if channel_id:
+            config_url = self._html_search_regex(
+                r'\bdata-config-url="([^"]+)"', webpage, 'config URL')
+            video_description = clean_html(get_element_by_class('description', webpage))
+            info_dict.update({
+                'channel_id': channel_id,
+                'channel_url': 'https://vimeo.com/channels/' + channel_id,
+            })
         else:
-            if config.get('view') == 4:
-                config = self._verify_player_video_password(redirect_url, video_id, headers)
-
+            page_config = self._parse_json(self._search_regex(
+                r'vimeo\.(?:clip|vod_title)_page_config\s*=\s*({.+?});',
+                webpage, 'page config', default='{}'), video_id, fatal=False)
+            if not page_config:
+                return self._extract_from_api(video_id)
+            config_url = page_config['player']['config_url']
+            cc_license = page_config.get('cc_license')
+            clip = page_config.get('clip') or {}
+            timestamp = clip.get('uploaded_on')
+            video_description = clean_html(
+                clip.get('description') or page_config.get('description_html_escaped'))
+        config = self._download_json(config_url, video_id)
         video = config.get('video') or {}
         vod = video.get('vod') or {}
 
         def is_rented():
             if '>You rented this title.<' in webpage:
                 return True
-            if config.get('user', {}).get('purchased'):
+            if try_get(config, lambda x: x['user']['purchased']):
                 return True
-            for purchase_option in vod.get('purchase_options', []):
+            for purchase_option in (vod.get('purchase_options') or []):
                 if purchase_option.get('purchased'):
                     return True
                 label = purchase_option.get('label_string')
@@ -828,14 +815,14 @@ class VimeoIE(VimeoBaseInfoExtractor):
                     'https://player.vimeo.com/player/%s' % feature_id,
                     {'force_feature_id': True}), 'Vimeo')
 
-        # Extract video description
         if not video_description:
             video_description = self._html_search_regex(
                 r'(?s)<div\s+class="[^"]*description[^"]*"[^>]*>(.*?)</div>',
                 webpage, 'description', default=None)
         if not video_description:
             video_description = self._html_search_meta(
-                'description', webpage, default=None)
+                ['description', 'og:description', 'twitter:description'],
+                webpage, default=None)
         if not video_description and is_pro:
             orig_webpage = self._download_webpage(
                 orig_url, video_id,
@@ -844,24 +831,17 @@ class VimeoIE(VimeoBaseInfoExtractor):
             if orig_webpage:
                 video_description = self._html_search_meta(
                     'description', orig_webpage, default=None)
-        if not video_description and not is_player:
+        if not video_description:
             self.report_warning('Cannot find video description')
 
-        # Extract upload date
         if not timestamp:
             timestamp = self._search_regex(
                 r'<time[^>]+datetime="([^"]+)"', webpage,
                 'timestamp', default=None)
 
-        try:
-            view_count = int(self._search_regex(r'UserPlays:(\d+)', webpage, 'view count'))
-            like_count = int(self._search_regex(r'UserLikes:(\d+)', webpage, 'like count'))
-            comment_count = int(self._search_regex(r'UserComments:(\d+)', webpage, 'comment count'))
-        except RegexNotFoundError:
-            # This info is only available in vimeo.com/{id} urls
-            view_count = None
-            like_count = None
-            comment_count = None
+        view_count = int_or_none(self._search_regex(r'UserPlays:(\d+)', webpage, 'view count', default=None))
+        like_count = int_or_none(self._search_regex(r'UserLikes:(\d+)', webpage, 'like count', default=None))
+        comment_count = int_or_none(self._search_regex(r'UserComments:(\d+)', webpage, 'comment count', default=None))
 
         formats = []
 
@@ -881,11 +861,7 @@ class VimeoIE(VimeoBaseInfoExtractor):
                 r'<link[^>]+rel=["\']license["\'][^>]+href=(["\'])(?P<license>(?:(?!\1).)+)\1',
                 webpage, 'license', default=None, group='license')
 
-        channel_id = self._search_regex(
-            r'vimeo\.com/channels/([^/]+)', url, 'channel id', default=None)
-        channel_url = 'https://vimeo.com/channels/%s' % channel_id if channel_id else None
-
-        info_dict = {
+        info_dict.update({
             'formats': formats,
             'timestamp': unified_timestamp(timestamp),
             'description': video_description,
@@ -894,18 +870,14 @@ class VimeoIE(VimeoBaseInfoExtractor):
             'like_count': like_count,
             'comment_count': comment_count,
             'license': cc_license,
-            'channel_id': channel_id,
-            'channel_url': channel_url,
-        }
+        })
 
-        info_dict = merge_dicts(info_dict, info_dict_config, json_ld)
-
-        return info_dict
+        return merge_dicts(info_dict, info_dict_config, json_ld)
 
 
 class VimeoOndemandIE(VimeoIE):
     IE_NAME = 'vimeo:ondemand'
-    _VALID_URL = r'https?://(?:www\.)?vimeo\.com/ondemand/([^/]+/)?(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?vimeo\.com/ondemand/(?:[^/]+/)?(?P<id>[^/?#&]+)'
     _TESTS = [{
         # ondemand video not available via https://vimeo.com/id
         'url': 'https://vimeo.com/ondemand/20704',

yt_dlp/extractor/vlive.py

@@ -17,17 +17,65 @@ from ..utils import (
     strip_or_none,
     try_get,
     urlencode_postdata,
+    url_or_none,
 )
 
 
 class VLiveBaseIE(NaverBaseIE):
-    _APP_ID = '8c6cc7b45d2568fb668be6e05b6e5a3b'
+    _NETRC_MACHINE = 'vlive'
+    _logged_in = False
+
+    def _real_initialize(self):
+        if not self._logged_in:
+            VLiveBaseIE._logged_in = self._login()
+
+    def _login(self):
+        email, password = self._get_login_info()
+        if email is None:
+            return False
+
+        LOGIN_URL = 'https://www.vlive.tv/auth/email/login'
+        self._request_webpage(
+            LOGIN_URL, None, note='Downloading login cookies')
+
+        self._download_webpage(
+            LOGIN_URL, None, note='Logging in',
+            data=urlencode_postdata({'email': email, 'pwd': password}),
+            headers={
+                'Referer': LOGIN_URL,
+                'Content-Type': 'application/x-www-form-urlencoded'
+            })
+
+        login_info = self._download_json(
+            'https://www.vlive.tv/auth/loginInfo', None,
+            note='Checking login status',
+            headers={'Referer': 'https://www.vlive.tv/home'})
+
+        if not try_get(login_info, lambda x: x['message']['login'], bool):
+            raise ExtractorError('Unable to log in', expected=True)
+        return True
+
+    def _call_api(self, path_template, video_id, fields=None, query_add={}, note=None):
+        if note is None:
+            note = 'Downloading %s JSON metadata' % path_template.split('/')[-1].split('-')[0]
+        query = {'appId': '8c6cc7b45d2568fb668be6e05b6e5a3b', 'gcc': 'KR', 'platformType': 'PC'}
+        if fields:
+            query['fields'] = fields
+        if query_add:
+            query.update(query_add)
+        try:
+            return self._download_json(
+                'https://www.vlive.tv/globalv-web/vam-web/' + path_template % video_id, video_id,
+                note, headers={'Referer': 'https://www.vlive.tv/'}, query=query)
+        except ExtractorError as e:
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
+                self.raise_login_required(json.loads(e.cause.read().decode('utf-8'))['message'])
+            raise
 
 
 class VLiveIE(VLiveBaseIE):
     IE_NAME = 'vlive'
     _VALID_URL = r'https?://(?:(?:www|m)\.)?vlive\.tv/(?:video|embed)/(?P<id>[0-9]+)'
-    _NETRC_MACHINE = 'vlive'
     _TESTS = [{
         'url': 'http://www.vlive.tv/video/1326',
         'md5': 'cc7314812855ce56de70a06a27314983',
@@ -81,53 +129,6 @@ class VLiveIE(VLiveBaseIE):
         'playlist_mincount': 120
     }]
 
-    def _real_initialize(self):
-        self._login()
-
-    def _login(self):
-        email, password = self._get_login_info()
-        if None in (email, password):
-            return
-
-        def is_logged_in():
-            login_info = self._download_json(
-                'https://www.vlive.tv/auth/loginInfo', None,
-                note='Downloading login info',
-                headers={'Referer': 'https://www.vlive.tv/home'})
-            return try_get(
-                login_info, lambda x: x['message']['login'], bool) or False
-
-        LOGIN_URL = 'https://www.vlive.tv/auth/email/login'
-        self._request_webpage(
-            LOGIN_URL, None, note='Downloading login cookies')
-
-        self._download_webpage(
-            LOGIN_URL, None, note='Logging in',
-            data=urlencode_postdata({'email': email, 'pwd': password}),
-            headers={
-                'Referer': LOGIN_URL,
-                'Content-Type': 'application/x-www-form-urlencoded'
-            })
-
-        if not is_logged_in():
-            raise ExtractorError('Unable to log in', expected=True)
-
-    def _call_api(self, path_template, video_id, fields=None, limit=None):
-        query = {'appId': self._APP_ID, 'gcc': 'KR', 'platformType': 'PC'}
-        if fields:
-            query['fields'] = fields
-        if limit:
-            query['limit'] = limit
-        try:
-            return self._download_json(
-                'https://www.vlive.tv/globalv-web/vam-web/' + path_template % video_id, video_id,
-                'Downloading %s JSON metadata' % path_template.split('/')[-1].split('-')[0],
-                headers={'Referer': 'https://www.vlive.tv/'}, query=query)
-        except ExtractorError as e:
-            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
-                self.raise_login_required(json.loads(e.cause.read().decode('utf-8'))['message'])
-            raise
-
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
@@ -150,7 +151,7 @@ class VLiveIE(VLiveBaseIE):
             playlist_count = str_or_none(playlist.get('totalCount'))
 
             playlist = self._call_api(
-                'playlist/v1.0/playlist-%s/posts', playlist_id, 'data', limit=playlist_count)
+                'playlist/v1.0/playlist-%s/posts', playlist_id, 'data', {'limit': playlist_count})
 
             entries = []
             for video_data in playlist['data']:
@@ -216,7 +217,7 @@ class VLiveIE(VLiveBaseIE):
                 raise ExtractorError('Unknown status ' + status)
 
 
-class VLivePostIE(VLiveIE):
+class VLivePostIE(VLiveBaseIE):
     IE_NAME = 'vlive:post'
     _VALID_URL = r'https?://(?:(?:www|m)\.)?vlive\.tv/post/(?P<id>\d-\d+)'
     _TESTS = [{
@@ -238,8 +239,6 @@ class VLivePostIE(VLiveIE):
         'playlist_count': 1,
     }]
     _FVIDEO_TMPL = 'fvideo/v1.0/fvideo-%%s/%s'
-    _SOS_TMPL = _FVIDEO_TMPL % 'sosPlayInfo'
-    _INKEY_TMPL = _FVIDEO_TMPL % 'inKey'
 
     def _real_extract(self, url):
         post_id = self._match_id(url)
@@ -266,7 +265,7 @@ class VLivePostIE(VLiveIE):
             entry = None
             if upload_type == 'SOS':
                 download = self._call_api(
-                    self._SOS_TMPL, video_id)['videoUrl']['download']
+                    self._FVIDEO_TMPL % 'sosPlayInfo', video_id)['videoUrl']['download']
                 formats = []
                 for f_id, f_url in download.items():
                     formats.append({
@@ -284,7 +283,7 @@ class VLivePostIE(VLiveIE):
                 vod_id = upload_info.get('videoId')
                 if not vod_id:
                     continue
-                inkey = self._call_api(self._INKEY_TMPL, video_id)['inKey']
+                inkey = self._call_api(self._FVIDEO_TMPL % 'inKey', video_id)['inKey']
                 entry = self._extract_video_info(video_id, vod_id, inkey)
             if entry:
                 entry['title'] = '%s_part%s' % (title, idx)
@@ -295,7 +294,7 @@ class VLivePostIE(VLiveIE):
 
 class VLiveChannelIE(VLiveBaseIE):
     IE_NAME = 'vlive:channel'
-    _VALID_URL = r'https?://(?:channels\.vlive\.tv|(?:(?:www|m)\.)?vlive\.tv/channel)/(?P<id>[0-9A-Z]+)'
+    _VALID_URL = r'https?://(?:channels\.vlive\.tv|(?:(?:www|m)\.)?vlive\.tv/channel)/(?P<channel_id>[0-9A-Z]+)(?:/board/(?P<posts_id>\d+))?'
     _TESTS = [{
         'url': 'http://channels.vlive.tv/FCD4B',
         'info_dict': {
@@ -306,78 +305,58 @@ class VLiveChannelIE(VLiveBaseIE):
     }, {
         'url': 'https://www.vlive.tv/channel/FCD4B',
         'only_matching': True,
+    }, {
+        'url': 'https://www.vlive.tv/channel/FCD4B/board/3546',
+        'info_dict': {
+            'id': 'FCD4B-3546',
+            'title': 'MAMAMOO - Star Board',
+        },
+        'playlist_mincount': 880
     }]
 
-    def _call_api(self, path, channel_key_suffix, channel_value, note, query):
-        q = {
-            'app_id': self._APP_ID,
-            'channel' + channel_key_suffix: channel_value,
-        }
-        q.update(query)
-        return self._download_json(
-            'http://api.vfan.vlive.tv/vproxy/channelplus/' + path,
-            channel_value, note='Downloading ' + note, query=q)['result']
-
-    def _real_extract(self, url):
-        channel_code = self._match_id(url)
-
-        channel_seq = self._call_api(
-            'decodeChannelCode', 'Code', channel_code,
-            'decode channel code', {})['channelSeq']
-
-        channel_name = None
-        entries = []
+    def _entries(self, posts_id, board_name):
+        if board_name:
+            posts_path = 'post/v1.0/board-%s/posts'
+            query_add = {'limit': 100, 'sortType': 'LATEST'}
+        else:
+            posts_path = 'post/v1.0/channel-%s/starPosts'
+            query_add = {'limit': 100}
 
         for page_num in itertools.count(1):
             video_list = self._call_api(
-                'getChannelVideoList', 'Seq', channel_seq,
-                'channel list page #%d' % page_num, {
-                    # Large values of maxNumOfRows (~300 or above) may cause
-                    # empty responses (see [1]), e.g. this happens for [2] that
-                    # has more than 300 videos.
-                    # 1. https://github.com/ytdl-org/youtube-dl/issues/13830
-                    # 2. http://channels.vlive.tv/EDBF.
-                    'maxNumOfRows': 100,
-                    'pageNo': page_num
-                }
-            )
+                posts_path, posts_id, 'channel{channelName},contentType,postId,title,url', query_add,
+                note=f'Downloading playlist page {page_num}')
 
-            if not channel_name:
-                channel_name = try_get(
-                    video_list,
-                    lambda x: x['channelInfo']['channelName'],
-                    compat_str)
-
-            videos = try_get(
-                video_list, lambda x: x['videoList'], list)
-            if not videos:
-                break
-
-            for video in videos:
-                video_id = video.get('videoSeq')
-                video_type = video.get('videoType')
-
-                if not video_id or not video_type:
+            for video in try_get(video_list, lambda x: x['data'], list) or []:
+                video_id = str(video.get('postId'))
+                video_title = str_or_none(video.get('title'))
+                video_url = url_or_none(video.get('url'))
+                if not all((video_id, video_title, video_url)) or video.get('contentType') != 'VIDEO':
                     continue
-                video_id = compat_str(video_id)
+                channel_name = try_get(video, lambda x: x['channel']['channelName'], compat_str)
+                yield self.url_result(video_url, VLivePostIE.ie_key(), video_id, video_title, channel=channel_name)
 
-                if video_type in ('PLAYLIST'):
-                    first_video_id = try_get(
-                        video,
-                        lambda x: x['videoPlaylist']['videoList'][0]['videoSeq'], int)
+            after = try_get(video_list, lambda x: x['paging']['nextParams']['after'], compat_str)
+            if not after:
+                break
+            query_add['after'] = after
 
-                    if not first_video_id:
-                        continue
+    def _real_extract(self, url):
+        channel_id, posts_id = self._match_valid_url(url).groups()
 
-                    entries.append(
-                        self.url_result(
-                            'http://www.vlive.tv/video/%s' % first_video_id,
-                            ie=VLiveIE.ie_key(), video_id=first_video_id))
-                else:
-                    entries.append(
-                        self.url_result(
-                            'http://www.vlive.tv/video/%s' % video_id,
-                            ie=VLiveIE.ie_key(), video_id=video_id))
+        board_name = None
+        if posts_id:
+            board = self._call_api(
+                'board/v1.0/board-%s', posts_id, 'title,boardType')
+            board_name = board.get('title') or 'Unknown'
+            if board.get('boardType') not in ('STAR', 'VLIVE_PLUS'):
+                raise ExtractorError(f'Board {board_name!r} is not supported', expected=True)
+
+        entries = self._entries(posts_id or channel_id, board_name)
+        first_video = next(entries)
+        channel_name = first_video['channel']
 
         return self.playlist_result(
-            entries, channel_code, channel_name)
+            itertools.chain([first_video], entries),
+            f'{channel_id}-{posts_id}' if posts_id else channel_id,
+            f'{channel_name} - {board_name}' if channel_name and board_name else channel_name)

yt_dlp/extractor/yahoo.py

@@ -334,31 +334,15 @@ class YahooSearchIE(SearchInfoExtractor):
     IE_NAME = 'screen.yahoo:search'
     _SEARCH_KEY = 'yvsearch'
 
-    def _get_n_results(self, query, n):
-        """Get a specified number of results for a query"""
-        entries = []
+    def _search_results(self, query):
         for pagenum in itertools.count(0):
             result_url = 'http://video.search.yahoo.com/search/?p=%s&fr=screen&o=js&gs=0&b=%d' % (compat_urllib_parse.quote_plus(query), pagenum * 30)
             info = self._download_json(result_url, query,
                                        note='Downloading results page ' + str(pagenum + 1))
-            m = info['m']
-            results = info['results']
-
-            for (i, r) in enumerate(results):
-                if (pagenum * 30) + i >= n:
-                    break
-                mobj = re.search(r'(?P<url>screen\.yahoo\.com/.*?-\d*?\.html)"', r)
-                e = self.url_result('http://' + mobj.group('url'), 'Yahoo')
-                entries.append(e)
-            if (pagenum * 30 + i >= n) or (m['last'] >= (m['total'] - 1)):
+            yield from (self.url_result(result['rurl']) for result in info['results'])
+            if info['m']['last'] >= info['m']['total'] - 1:
                 break
 
-        return {
-            '_type': 'playlist',
-            'id': query,
-            'entries': entries,
-        }
-
 
 class YahooGyaOPlayerIE(InfoExtractor):
     IE_NAME = 'yahoo:gyao:player'

yt_dlp/extractor/youtube.py

@@ -258,28 +258,12 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
     # If True it will raise an error if no login info is provided
     _LOGIN_REQUIRED = False
 
-    r'''  # Unused since login is broken
-    _LOGIN_URL = 'https://accounts.google.com/ServiceLogin'
-    _TWOFACTOR_URL = 'https://accounts.google.com/signin/challenge'
-
-    _LOOKUP_URL = 'https://accounts.google.com/_/signin/sl/lookup'
-    _CHALLENGE_URL = 'https://accounts.google.com/_/signin/sl/challenge'
-    _TFA_URL = 'https://accounts.google.com/_/signin/challenge?hl=en&TL={0}'
-    '''
-
     def _login(self):
         """
         Attempt to log in to YouTube.
-        True is returned if successful or skipped.
-        False is returned if login failed.
-
         If _LOGIN_REQUIRED is set and no authentication was provided, an error is raised.
         """
 
-        def warn(message):
-            self.report_warning(message)
-
-        # username+password login is broken
         if (self._LOGIN_REQUIRED
                 and self.get_param('cookiefile') is None
                 and self.get_param('cookiesfrombrowser') is None):
@@ -287,184 +271,7 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
                 'Login details are needed to download this content', method='cookies')
         username, password = self._get_login_info()
         if username:
-            warn('Logging in using username and password is broken. %s' % self._LOGIN_HINTS['cookies'])
-        return
-
-        # Everything below this is broken!
-        r'''
-        # No authentication to be performed
-        if username is None:
-            if self._LOGIN_REQUIRED and self.get_param('cookiefile') is None:
-                raise ExtractorError('No login info available, needed for using %s.' % self.IE_NAME, expected=True)
-            # if self.get_param('cookiefile'):  # TODO remove 'and False' later - too many people using outdated cookies and open issues, remind them.
-            #     self.to_screen('[Cookies] Reminder - Make sure to always use up to date cookies!')
-            return True
-
-        login_page = self._download_webpage(
-            self._LOGIN_URL, None,
-            note='Downloading login page',
-            errnote='unable to fetch login page', fatal=False)
-        if login_page is False:
-            return
-
-        login_form = self._hidden_inputs(login_page)
-
-        def req(url, f_req, note, errnote):
-            data = login_form.copy()
-            data.update({
-                'pstMsg': 1,
-                'checkConnection': 'youtube',
-                'checkedDomains': 'youtube',
-                'hl': 'en',
-                'deviceinfo': '[null,null,null,[],null,"US",null,null,[],"GlifWebSignIn",null,[null,null,[]]]',
-                'f.req': json.dumps(f_req),
-                'flowName': 'GlifWebSignIn',
-                'flowEntry': 'ServiceLogin',
-                # TODO: reverse actual botguard identifier generation algo
-                'bgRequest': '["identifier",""]',
-            })
-            return self._download_json(
-                url, None, note=note, errnote=errnote,
-                transform_source=lambda s: re.sub(r'^[^[]*', '', s),
-                fatal=False,
-                data=urlencode_postdata(data), headers={
-                    'Content-Type': 'application/x-www-form-urlencoded;charset=utf-8',
-                    'Google-Accounts-XSRF': 1,
-                })
-
-        lookup_req = [
-            username,
-            None, [], None, 'US', None, None, 2, False, True,
-            [
-                None, None,
-                [2, 1, None, 1,
-                 'https://accounts.google.com/ServiceLogin?passive=true&continue=https%3A%2F%2Fwww.youtube.com%2Fsignin%3Fnext%3D%252F%26action_handle_signin%3Dtrue%26hl%3Den%26app%3Ddesktop%26feature%3Dsign_in_button&hl=en&service=youtube&uilel=3&requestPath=%2FServiceLogin&Page=PasswordSeparationSignIn',
-                 None, [], 4],
-                1, [None, None, []], None, None, None, True
-            ],
-            username,
-        ]
-
-        lookup_results = req(
-            self._LOOKUP_URL, lookup_req,
-            'Looking up account info', 'Unable to look up account info')
-
-        if lookup_results is False:
-            return False
-
-        user_hash = try_get(lookup_results, lambda x: x[0][2], compat_str)
-        if not user_hash:
-            warn('Unable to extract user hash')
-            return False
-
-        challenge_req = [
-            user_hash,
-            None, 1, None, [1, None, None, None, [password, None, True]],
-            [
-                None, None, [2, 1, None, 1, 'https://accounts.google.com/ServiceLogin?passive=true&continue=https%3A%2F%2Fwww.youtube.com%2Fsignin%3Fnext%3D%252F%26action_handle_signin%3Dtrue%26hl%3Den%26app%3Ddesktop%26feature%3Dsign_in_button&hl=en&service=youtube&uilel=3&requestPath=%2FServiceLogin&Page=PasswordSeparationSignIn', None, [], 4],
-                1, [None, None, []], None, None, None, True
-            ]]
-
-        challenge_results = req(
-            self._CHALLENGE_URL, challenge_req,
-            'Logging in', 'Unable to log in')
-
-        if challenge_results is False:
-            return
-
-        login_res = try_get(challenge_results, lambda x: x[0][5], list)
-        if login_res:
-            login_msg = try_get(login_res, lambda x: x[5], compat_str)
-            warn(
-                'Unable to login: %s' % 'Invalid password'
-                if login_msg == 'INCORRECT_ANSWER_ENTERED' else login_msg)
-            return False
-
-        res = try_get(challenge_results, lambda x: x[0][-1], list)
-        if not res:
-            warn('Unable to extract result entry')
-            return False
-
-        login_challenge = try_get(res, lambda x: x[0][0], list)
-        if login_challenge:
-            challenge_str = try_get(login_challenge, lambda x: x[2], compat_str)
-            if challenge_str == 'TWO_STEP_VERIFICATION':
-                # SEND_SUCCESS - TFA code has been successfully sent to phone
-                # QUOTA_EXCEEDED - reached the limit of TFA codes
-                status = try_get(login_challenge, lambda x: x[5], compat_str)
-                if status == 'QUOTA_EXCEEDED':
-                    warn('Exceeded the limit of TFA codes, try later')
-                    return False
-
-                tl = try_get(challenge_results, lambda x: x[1][2], compat_str)
-                if not tl:
-                    warn('Unable to extract TL')
-                    return False
-
-                tfa_code = self._get_tfa_info('2-step verification code')
-
-                if not tfa_code:
-                    warn(
-                        'Two-factor authentication required. Provide it either interactively or with --twofactor <code>'
-                        '(Note that only TOTP (Google Authenticator App) codes work at this time.)')
-                    return False
-
-                tfa_code = remove_start(tfa_code, 'G-')
-
-                tfa_req = [
-                    user_hash, None, 2, None,
-                    [
-                        9, None, None, None, None, None, None, None,
-                        [None, tfa_code, True, 2]
-                    ]]
-
-                tfa_results = req(
-                    self._TFA_URL.format(tl), tfa_req,
-                    'Submitting TFA code', 'Unable to submit TFA code')
-
-                if tfa_results is False:
-                    return False
-
-                tfa_res = try_get(tfa_results, lambda x: x[0][5], list)
-                if tfa_res:
-                    tfa_msg = try_get(tfa_res, lambda x: x[5], compat_str)
-                    warn(
-                        'Unable to finish TFA: %s' % 'Invalid TFA code'
-                        if tfa_msg == 'INCORRECT_ANSWER_ENTERED' else tfa_msg)
-                    return False
-
-                check_cookie_url = try_get(
-                    tfa_results, lambda x: x[0][-1][2], compat_str)
-            else:
-                CHALLENGES = {
-                    'LOGIN_CHALLENGE': "This device isn't recognized. For your security, Google wants to make sure it's really you.",
-                    'USERNAME_RECOVERY': 'Please provide additional information to aid in the recovery process.',
-                    'REAUTH': "There is something unusual about your activity. For your security, Google wants to make sure it's really you.",
-                }
-                challenge = CHALLENGES.get(
-                    challenge_str,
-                    '%s returned error %s.' % (self.IE_NAME, challenge_str))
-                warn('%s\nGo to https://accounts.google.com/, login and solve a challenge.' % challenge)
-                return False
-        else:
-            check_cookie_url = try_get(res, lambda x: x[2], compat_str)
-
-        if not check_cookie_url:
-            warn('Unable to extract CheckCookie URL')
-            return False
-
-        check_cookie_results = self._download_webpage(
-            check_cookie_url, None, 'Checking cookie', fatal=False)
-
-        if check_cookie_results is False:
-            return False
-
-        if 'https://myaccount.google.com/' not in check_cookie_results:
-            warn('Unable to log in')
-            return False
-
-        return True
-        '''
+            self.report_warning(f'Cannot login to YouTube using username and password. {self._LOGIN_HINTS["cookies"]}')
 
     def _initialize_consent(self):
         cookies = self._get_cookies('https://www.youtube.com/')
@@ -483,10 +290,7 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
 
     def _real_initialize(self):
         self._initialize_consent()
-        if self._downloader is None:
-            return
-        if not self._login():
-            return
+        self._login()
 
     _YT_INITIAL_DATA_RE = r'(?:window\s*\[\s*["\']ytInitialData["\']\s*\]|ytInitialData)\s*=\s*({.+?})\s*;'
     _YT_INITIAL_PLAYER_RESPONSE_RE = r'ytInitialPlayerResponse\s*=\s*({.+?})\s*;'
@@ -2241,7 +2045,6 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
     def _comment_entries(self, root_continuation_data, ytcfg, video_id, parent=None, comment_counts=None):
 
         def extract_header(contents):
-            _total_comments = 0
             _continuation = None
             for content in contents:
                 comments_header_renderer = try_get(content, lambda x: x['commentsHeaderRenderer'])
@@ -2251,7 +2054,6 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 if expected_comment_count:
                     comment_counts[1] = expected_comment_count
                     self.to_screen('Downloading ~%d comments' % expected_comment_count)
-                    _total_comments = comment_counts[1]
                 sort_mode_str = self._configuration_arg('comment_sort', [''])[0]
                 comment_sort_index = int(sort_mode_str != 'top')  # 1 = new, 0 = top
 
@@ -2271,7 +2073,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     sort_text = 'top comments' if comment_sort_index == 0 else 'newest first'
                 self.to_screen('Sorting comments by %s' % sort_text)
                 break
-            return _total_comments, _continuation
+            return _continuation
 
         def extract_thread(contents):
             if not parent:
@@ -2316,6 +2118,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             continuation_token = self._generate_comment_continuation(video_id)
             continuation = self._build_api_continuation_query(continuation_token, None)
 
+        message = self._get_text(root_continuation_data, ('contents', ..., 'messageRenderer', 'text'), max_runs=1)
+        if message and not parent:
+            self.report_warning(message, video_id=video_id)
+
         visitor_data = None
         is_first_continuation = parent is None
 
@@ -2359,9 +2165,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                          lambda x: x['appendContinuationItemsAction']['continuationItems']),
                         list) or []
                     if is_first_continuation:
-                        total_comments, continuation = extract_header(continuation_items)
-                        if total_comments:
-                            yield total_comments
+                        continuation = extract_header(continuation_items)
                         is_first_continuation = False
                         if continuation:
                             break
@@ -2389,9 +2193,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         continue
                     if is_first_continuation:
                         header_continuation_items = [continuation_renderer.get('header') or {}]
-                        total_comments, continuation = extract_header(header_continuation_items)
-                        if total_comments:
-                            yield total_comments
+                        continuation = extract_header(header_continuation_items)
                         is_first_continuation = False
                         if continuation:
                             break
@@ -2419,35 +2221,21 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             [bytes_to_intlist(base64.b64decode(part)) for part in parts]))
         return base64.b64encode(intlist_to_bytes(new_continuation_intlist)).decode('utf-8')
 
-    def _extract_comments(self, ytcfg, video_id, contents, webpage):
+    def _get_comments(self, ytcfg, video_id, contents, webpage):
         """Entry for comment extraction"""
         def _real_comment_extract(contents):
-            yield from self._comment_entries(
-                traverse_obj(contents, (..., 'itemSectionRenderer'), get_all=False), ytcfg, video_id)
+            renderer = next((
+                item for item in traverse_obj(contents, (..., 'itemSectionRenderer'), default={})
+                if item.get('sectionIdentifier') == 'comment-item-section'), None)
+            yield from self._comment_entries(renderer, ytcfg, video_id)
 
-        comments = []
-        estimated_total = 0
-        max_comments = int_or_none(self._configuration_arg('max_comments', [''])[0]) or float('inf')
+        max_comments = int_or_none(self._configuration_arg('max_comments', [''])[0])
         # Force English regardless of account setting to prevent parsing issues
         # See: https://github.com/yt-dlp/yt-dlp/issues/532
         ytcfg = copy.deepcopy(ytcfg)
         traverse_obj(
             ytcfg, ('INNERTUBE_CONTEXT', 'client'), expected_type=dict, default={})['hl'] = 'en'
-        try:
-            for comment in _real_comment_extract(contents):
-                if len(comments) >= max_comments:
-                    break
-                if isinstance(comment, int):
-                    estimated_total = comment
-                    continue
-                comments.append(comment)
-        except KeyboardInterrupt:
-            self.to_screen('Interrupted by user')
-        self.to_screen('Downloaded %d/%d comments' % (len(comments), estimated_total))
-        return {
-            'comments': comments,
-            'comment_count': len(comments),
-        }
+        return itertools.islice(_real_comment_extract(contents), 0, max_comments)
 
     @staticmethod
     def _get_checkok_params():
@@ -2714,7 +2502,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     itag = self._search_regex(
                         r'/itag/(\d+)', f['url'], 'itag', default=None)
                     if itag in itags:
-                        continue
+                        itag += '-hls'
+                        if itag in itags:
+                            continue
                     if itag:
                         f['format_id'] = itag
                         itags.append(itag)
@@ -2726,8 +2516,11 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 for f in self._extract_mpd_formats(dash_manifest_url, video_id, fatal=False):
                     itag = f['format_id']
                     if itag in itags:
-                        continue
+                        itag += '-dash'
+                        if itag in itags:
+                            continue
                     if itag:
+                        f['format_id'] = itag
                         itags.append(itag)
                     f['quality'] = guess_quality(f)
                     filesize = int_or_none(self._search_regex(
@@ -2860,7 +2653,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
 
         # Source is given priority since formats that throttle are given lower source_preference
         # When throttling issue is fully fixed, remove this
-        self._sort_formats(formats, ('quality', 'res', 'fps', 'source', 'codec:vp9.2', 'lang'))
+        self._sort_formats(formats, ('quality', 'res', 'fps', 'hdr:12', 'source', 'codec:vp9.2', 'lang'))
 
         keywords = get_first(video_details, 'keywords', expected_type=list) or []
         if not keywords and webpage:
@@ -2906,21 +2699,18 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         # The best resolution thumbnails sometimes does not appear in the webpage
         # See: https://github.com/ytdl-org/youtube-dl/issues/29049, https://github.com/yt-dlp/yt-dlp/issues/340
         # List of possible thumbnails - Ref: <https://stackoverflow.com/a/20542029>
-        hq_thumbnail_names = ['maxresdefault', 'hq720', 'sddefault', 'sd1', 'sd2', 'sd3']
-        # TODO: Test them also? - For some videos, even these don't exist
-        guaranteed_thumbnail_names = [
+        thumbnail_names = [
+            'maxresdefault', 'hq720', 'sddefault', 'sd1', 'sd2', 'sd3',
             'hqdefault', 'hq1', 'hq2', 'hq3', '0',
             'mqdefault', 'mq1', 'mq2', 'mq3',
             'default', '1', '2', '3'
         ]
-        thumbnail_names = hq_thumbnail_names + guaranteed_thumbnail_names
         n_thumbnail_names = len(thumbnail_names)
 
         thumbnails.extend({
             'url': 'https://i.ytimg.com/vi{webp}/{video_id}/{name}{live}.{ext}'.format(
                 video_id=video_id, name=name, ext=ext,
                 webp='_webp' if ext == 'webp' else '', live='_live' if is_live else ''),
-            '_test_url': name in hq_thumbnail_names,
         } for name in thumbnail_names for ext in ('webp', 'jpg'))
         for thumb in thumbnails:
             i = next((i for i, t in enumerate(thumbnail_names) if f'/{video_id}/{t}' in thumb['url']), n_thumbnail_names)
@@ -2986,15 +2776,19 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         }
 
         pctr = traverse_obj(player_responses, (..., 'captions', 'playerCaptionsTracklistRenderer'), expected_type=dict)
-        # Converted into dicts to remove duplicates
-        captions = {
-            sub.get('baseUrl'): sub
-            for sub in traverse_obj(pctr, (..., 'captionTracks', ...), default=[])}
-        translation_languages = {
-            lang.get('languageCode'): lang.get('languageName')
-            for lang in traverse_obj(pctr, (..., 'translationLanguages', ...), default=[])}
-        subtitles = {}
         if pctr:
+            def get_lang_code(track):
+                return (remove_start(track.get('vssId') or '', '.').replace('.', '-')
+                        or track.get('languageCode'))
+
+            # Converted into dicts to remove duplicates
+            captions = {
+                get_lang_code(sub): sub
+                for sub in traverse_obj(pctr, (..., 'captionTracks', ...), default=[])}
+            translation_languages = {
+                lang.get('languageCode'): self._get_text(lang.get('languageName'), max_runs=1)
+                for lang in traverse_obj(pctr, (..., 'translationLanguages', ...), default=[])}
+
             def process_language(container, base_url, lang_code, sub_name, query):
                 lang_subs = container.setdefault(lang_code, [])
                 for fmt in self._SUBTITLE_FORMATS:
@@ -3007,30 +2801,29 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         'name': sub_name,
                     })
 
-            for base_url, caption_track in captions.items():
+            subtitles, automatic_captions = {}, {}
+            for lang_code, caption_track in captions.items():
+                base_url = caption_track.get('baseUrl')
                 if not base_url:
                     continue
+                lang_name = self._get_text(caption_track, 'name', max_runs=1)
                 if caption_track.get('kind') != 'asr':
-                    lang_code = (
-                        remove_start(caption_track.get('vssId') or '', '.').replace('.', '-')
-                        or caption_track.get('languageCode'))
                     if not lang_code:
                         continue
                     process_language(
-                        subtitles, base_url, lang_code,
-                        traverse_obj(caption_track, ('name', 'simpleText'), ('name', 'runs', ..., 'text'), get_all=False),
-                        {})
-                    continue
-                automatic_captions = {}
+                        subtitles, base_url, lang_code, lang_name, {})
+                    if not caption_track.get('isTranslatable'):
+                        continue
                 for trans_code, trans_name in translation_languages.items():
                     if not trans_code:
                         continue
+                    if caption_track.get('kind') != 'asr':
+                        trans_code += f'-{lang_code}'
+                        trans_name += format_field(lang_name, template=' from %s')
                     process_language(
-                        automatic_captions, base_url, trans_code,
-                        self._get_text(trans_name, max_runs=1),
-                        {'tlang': trans_code})
-                info['automatic_captions'] = automatic_captions
-        info['subtitles'] = subtitles
+                        automatic_captions, base_url, trans_code, trans_name, {'tlang': trans_code})
+            info['automatic_captions'] = automatic_captions
+            info['subtitles'] = subtitles
 
         parsed_url = compat_urllib_parse_urlparse(url)
         for component in [parsed_url.fragment, parsed_url.query]:
@@ -3076,7 +2869,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         try:
             # This will error if there is no livechat
             initial_data['contents']['twoColumnWatchNextResults']['conversationBar']['liveChatRenderer']['continuations'][0]['reloadContinuationData']['continuation']
-            info['subtitles']['live_chat'] = [{
+            info.setdefault('subtitles', {})['live_chat'] = [{
                 'url': 'https://www.youtube.com/watch?v=%s' % video_id,  # url is needed to set cookies
                 'video_id': video_id,
                 'ext': 'json',
@@ -3209,8 +3002,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             needs_auth=info['age_limit'] >= 18,
             is_unlisted=None if is_private is None else is_unlisted)
 
-        if self.get_param('getcomments', False):
-            info['__post_extractor'] = lambda: self._extract_comments(master_ytcfg, video_id, contents, webpage)
+        info['__post_extractor'] = self.extract_comments(master_ytcfg, video_id, contents, webpage)
 
         self.mark_watched(video_id, player_responses)
 
@@ -4512,9 +4304,7 @@ class YoutubePlaylistIE(InfoExtractor):
     def suitable(cls, url):
         if YoutubeTabIE.suitable(url):
             return False
-        # Hack for lazy extractors until more generic solution is implemented
-        # (see #28780)
-        from .youtube import parse_qs
+        from ..utils import parse_qs
         qs = parse_qs(url)
         if qs.get('v', [None])[0]:
             return False
@@ -4615,11 +4405,10 @@ class YoutubeSearchIE(SearchInfoExtractor, YoutubeTabIE):
     _SEARCH_PARAMS = None
     _TESTS = []
 
-    def _entries(self, query, n):
+    def _search_results(self, query):
         data = {'query': query}
         if self._SEARCH_PARAMS:
             data['params'] = self._SEARCH_PARAMS
-        total = 0
         continuation = {}
         for page_num in itertools.count(1):
             data.update(continuation)
@@ -4662,17 +4451,10 @@ class YoutubeSearchIE(SearchInfoExtractor, YoutubeTabIE):
                         continue
 
                     yield self._extract_video(video)
-                    total += 1
-                    if total == n:
-                        return
 
             if not continuation:
                 break
 
-    def _get_n_results(self, query, n):
-        """Get a specified number of results for a query"""
-        return self.playlist_result(self._entries(query, n), query, query)
-
 
 class YoutubeSearchDateIE(YoutubeSearchIE):
     IE_NAME = YoutubeSearchIE.IE_NAME + ':date'

yt_dlp/minicurses.py

@@ -1,6 +1,6 @@
 import functools
 from threading import Lock
-from .utils import supports_terminal_sequences, TERMINAL_SEQUENCES
+from .utils import supports_terminal_sequences, TERMINAL_SEQUENCES, write_string
 
 
 class MultilinePrinterBase:
@@ -25,20 +25,26 @@ class MultilinePrinterBase:
             return f'{line + 1}: {text}'
         return text
 
+    def write(self, *text):
+        write_string(''.join(text), self.stream)
+
 
 class QuietMultilinePrinter(MultilinePrinterBase):
     pass
 
 
 class MultilineLogger(MultilinePrinterBase):
+    def write(self, *text):
+        self.stream.debug(''.join(text))
+
     def print_at_line(self, text, pos):
         # stream is the logger object, not an actual stream
-        self.stream.debug(self._add_line_number(text, pos))
+        self.write(self._add_line_number(text, pos))
 
 
 class BreaklineStatusPrinter(MultilinePrinterBase):
     def print_at_line(self, text, pos):
-        self.stream.write(self._add_line_number(text, pos) + '\n')
+        self.write(self._add_line_number(text, pos), '\n')
 
 
 class MultilinePrinter(MultilinePrinterBase):
@@ -58,50 +64,46 @@ class MultilinePrinter(MultilinePrinterBase):
 
     def _move_cursor(self, dest):
         current = min(self._lastline, self.maximum)
-        self.stream.write('\r')
+        yield '\r'
         distance = dest - current
         if distance < 0:
-            self.stream.write(TERMINAL_SEQUENCES['UP'] * -distance)
+            yield TERMINAL_SEQUENCES['UP'] * -distance
         elif distance > 0:
-            self.stream.write(TERMINAL_SEQUENCES['DOWN'] * distance)
+            yield TERMINAL_SEQUENCES['DOWN'] * distance
         self._lastline = dest
 
     @lock
     def print_at_line(self, text, pos):
         if self._HAVE_FULLCAP:
-            self._move_cursor(pos)
-            self.stream.write(TERMINAL_SEQUENCES['ERASE_LINE'])
-            self.stream.write(text)
-            return
+            self.write(*self._move_cursor(pos), TERMINAL_SEQUENCES['ERASE_LINE'], text)
 
         text = self._add_line_number(text, pos)
         textlen = len(text)
         if self._lastline == pos:
             # move cursor at the start of progress when writing to same line
-            self.stream.write('\r')
+            prefix = '\r'
             if self._lastlength > textlen:
                 text += ' ' * (self._lastlength - textlen)
             self._lastlength = textlen
         else:
             # otherwise, break the line
-            self.stream.write('\n')
+            prefix = '\n'
             self._lastlength = textlen
-        self.stream.write(text)
+        self.write(prefix, text)
         self._lastline = pos
 
     @lock
     def end(self):
         # move cursor to the end of the last line, and write line break
         # so that other to_screen calls can precede
-        if self._HAVE_FULLCAP:
-            self._move_cursor(self.maximum)
+        text = self._move_cursor(self.maximum) if self._HAVE_FULLCAP else []
         if self.preserve_output:
-            self.stream.write('\n')
+            self.write(*text, '\n')
             return
 
         if self._HAVE_FULLCAP:
-            self.stream.write(
-                TERMINAL_SEQUENCES['ERASE_LINE']
-                + f'{TERMINAL_SEQUENCES["UP"]}{TERMINAL_SEQUENCES["ERASE_LINE"]}' * self.maximum)
+            self.write(
+                *text, TERMINAL_SEQUENCES['ERASE_LINE'],
+                f'{TERMINAL_SEQUENCES["UP"]}{TERMINAL_SEQUENCES["ERASE_LINE"]}' * self.maximum)
         else:
-            self.stream.write(' ' * self._lastlength)
+            self.write(*text, ' ' * self._lastlength)

yt_dlp/options.py

@@ -971,6 +971,13 @@ def parseOpts(overrideArguments=None):
         dest='batchfile', metavar='FILE',
         help="File containing URLs to download ('-' for stdin), one URL per line. "
              "Lines starting with '#', ';' or ']' are considered as comments and ignored")
+    filesystem.add_option(
+        '--no-batch-file',
+        dest='batchfile', action='store_const', const=None,
+        help='Do not read URLs from batch file (default)')
+    filesystem.add_option(
+        '--id', default=False,
+        action='store_true', dest='useid', help=optparse.SUPPRESS_HELP)
     filesystem.add_option(
         '-P', '--paths',
         metavar='[TYPES:]PATH', dest='paths', default={}, type='str',
@@ -1378,7 +1385,11 @@ def parseOpts(overrideArguments=None):
     postproc.add_option(
         '--remove-chapters',
         metavar='REGEX', dest='remove_chapters', action='append',
-        help='Remove chapters whose title matches the given regular expression. This option can be used multiple times')
+        help=(
+            'Remove chapters whose title matches the given regular expression. '
+            'Time ranges prefixed by a "*" can also be used in place of chapters to remove the specified range. '
+            'Eg: --remove-chapters "*10:15-15:00" --remove-chapters "intro". '
+            'This option can be used multiple times'))
     postproc.add_option(
         '--no-remove-chapters', dest='remove_chapters', action='store_const', const=None,
         help='Do not remove any chapters from the file (default)')
@@ -1590,7 +1601,7 @@ def parseOpts(overrideArguments=None):
                     parser.error('config-location %s does not exist.' % location)
                 config = _readOptions(location, default=None)
                 if config:
-                    configs['custom'], paths['config'] = config, location
+                    configs['custom'], paths['custom'] = config, location
 
             if opts.ignoreconfig:
                 return

yt_dlp/postprocessor/common.py

@@ -17,11 +17,12 @@ class PostProcessorMetaClass(type):
     def run_wrapper(func):
         @functools.wraps(func)
         def run(self, info, *args, **kwargs):
-            self._hook_progress({'status': 'started'}, info)
+            info_copy = copy.deepcopy(self._copy_infodict(info))
+            self._hook_progress({'status': 'started'}, info_copy)
             ret = func(self, info, *args, **kwargs)
             if ret is not None:
                 _, info = ret
-            self._hook_progress({'status': 'finished'}, info)
+            self._hook_progress({'status': 'finished'}, info_copy)
             return ret
         return run
 
@@ -90,11 +91,12 @@ class PostProcessor(metaclass=PostProcessorMetaClass):
     def set_downloader(self, downloader):
         """Sets the downloader for this PP."""
         self._downloader = downloader
-        if not downloader:
-            return
-        for ph in downloader._postprocessor_hooks:
+        for ph in getattr(downloader, '_postprocessor_hooks', []):
             self.add_progress_hook(ph)
 
+    def _copy_infodict(self, info_dict):
+        return getattr(self._downloader, '_copy_infodict', dict)(info_dict)
+
     @staticmethod
     def _restrict_to(*, video=True, audio=True, images=True):
         allowed = {'video': video, 'audio': audio, 'images': images}
@@ -144,11 +146,8 @@ class PostProcessor(metaclass=PostProcessorMetaClass):
     def _hook_progress(self, status, info_dict):
         if not self._progress_hooks:
             return
-        info_dict = dict(info_dict)
-        for key in ('__original_infodict', '__postprocessors'):
-            info_dict.pop(key, None)
         status.update({
-            'info_dict': copy.deepcopy(info_dict),
+            'info_dict': info_dict,
             'postprocessor': self.pp_key(),
         })
         for ph in self._progress_hooks:

yt_dlp/postprocessor/embedthumbnail.py

@@ -26,9 +26,9 @@ from ..utils import (
     encodeArgument,
     encodeFilename,
     error_to_compat_str,
+    Popen,
     PostProcessingError,
     prepend_extension,
-    process_communicate_or_kill,
     shell_quote,
 )
 
@@ -183,8 +183,8 @@ class EmbedThumbnailPP(FFmpegPostProcessor):
 
                 self._report_run('atomicparsley', filename)
                 self.write_debug('AtomicParsley command line: %s' % shell_quote(cmd))
-                p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-                stdout, stderr = process_communicate_or_kill(p)
+                p = Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                stdout, stderr = p.communicate_or_kill()
                 if p.returncode != 0:
                     msg = stderr.decode('utf-8', 'replace').strip()
                     raise EmbedThumbnailPPError(msg)

yt_dlp/postprocessor/ffmpeg.py

@@ -10,7 +10,7 @@ import json
 
 from .common import AudioConversionError, PostProcessor
 
-from ..compat import compat_str, compat_numeric_types
+from ..compat import compat_str
 from ..utils import (
     dfxp2srt,
     encodeArgument,
@@ -20,9 +20,9 @@ from ..utils import (
     is_outdated_version,
     ISO639Utils,
     orderedSet,
+    Popen,
     PostProcessingError,
     prepend_extension,
-    process_communicate_or_kill,
     replace_extension,
     shell_quote,
     traverse_obj,
@@ -178,10 +178,8 @@ class FFmpegPostProcessor(PostProcessor):
                     encodeArgument('-i')]
             cmd.append(encodeFilename(self._ffmpeg_filename_argument(path), True))
             self.write_debug('%s command line: %s' % (self.basename, shell_quote(cmd)))
-            handle = subprocess.Popen(
-                cmd, stderr=subprocess.PIPE,
-                stdout=subprocess.PIPE, stdin=subprocess.PIPE)
-            stdout_data, stderr_data = process_communicate_or_kill(handle)
+            handle = Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            stdout_data, stderr_data = handle.communicate_or_kill()
             expected_ret = 0 if self.probe_available else 1
             if handle.wait() != expected_ret:
                 return None
@@ -223,7 +221,7 @@ class FFmpegPostProcessor(PostProcessor):
         cmd += opts
         cmd.append(encodeFilename(self._ffmpeg_filename_argument(path), True))
         self.write_debug('ffprobe command line: %s' % shell_quote(cmd))
-        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
+        p = Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
         stdout, stderr = p.communicate()
         return json.loads(stdout.decode('utf-8', 'replace'))
 
@@ -262,7 +260,7 @@ class FFmpegPostProcessor(PostProcessor):
         oldest_mtime = min(
             os.stat(encodeFilename(path)).st_mtime for path, _ in input_path_opts if path)
 
-        cmd = [encodeFilename(self.executable, True), encodeArgument('-y'), encodeArgument('-probesize'), encodeArgument('max')]
+        cmd = [encodeFilename(self.executable, True), encodeArgument('-y')]
         # avconv does not have repeat option
         if self.basename == 'ffmpeg':
             cmd += [encodeArgument('-loglevel'), encodeArgument('repeat+info')]
@@ -284,8 +282,8 @@ class FFmpegPostProcessor(PostProcessor):
                 for i, (path, opts) in enumerate(path_opts) if path)
 
         self.write_debug('ffmpeg command line: %s' % shell_quote(cmd))
-        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
-        stdout, stderr = process_communicate_or_kill(p)
+        p = Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
+        stdout, stderr = p.communicate_or_kill()
         if p.returncode not in variadic(expected_retcodes):
             stderr = stderr.decode('utf-8', 'replace').strip()
             self.write_debug(stderr)
@@ -664,15 +662,14 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
 
     def _get_metadata_opts(self, info):
         metadata = {}
+        meta_prefix = 'meta_'
 
         def add(meta_list, info_list=None):
-            if not meta_list:
-                return
-            for info_f in variadic(info_list or meta_list):
-                if isinstance(info.get(info_f), (compat_str, compat_numeric_types)):
-                    for meta_f in variadic(meta_list):
-                        metadata[meta_f] = info[info_f]
-                    break
+            value = next((
+                str(info[key]) for key in [meta_prefix] + list(variadic(info_list or meta_list))
+                if info.get(key) is not None), None)
+            if value not in ('', None):
+                metadata.update({meta_f: value for meta_f in variadic(meta_list)})
 
         # See [1-4] for some info on media metadata/metadata supported
         # by ffmpeg.
@@ -695,9 +692,9 @@ class FFmpegMetadataPP(FFmpegPostProcessor):
         add('episode_id', ('episode', 'episode_id'))
         add('episode_sort', 'episode_number')
 
-        prefix = 'meta_'
-        for key in filter(lambda k: k.startswith(prefix), info.keys()):
-            add(key[len(prefix):], key)
+        for key, value in info.items():
+            if value is not None and key != meta_prefix and key.startswith(meta_prefix):
+                metadata[key[len(meta_prefix):]] = value
 
         for name, value in metadata.items():
             yield ('-metadata', f'{name}={value}')
@@ -732,7 +729,8 @@ class FFmpegMergerPP(FFmpegPostProcessor):
         for (i, fmt) in enumerate(info['requested_formats']):
             if fmt.get('acodec') != 'none':
                 args.extend(['-map', f'{i}:a:0'])
-                if self.get_audio_codec(fmt['filepath']) == 'aac':
+                aac_fixup = fmt['protocol'].startswith('m3u8') and self.get_audio_codec(fmt['filepath']) == 'aac'
+                if aac_fixup:
                     args.extend([f'-bsf:a:{audio_streams}', 'aac_adtstoasc'])
                 audio_streams += 1
             if fmt.get('vcodec') != 'none':

yt_dlp/postprocessor/modify_chapters.py

@@ -20,18 +20,21 @@ DEFAULT_SPONSORBLOCK_CHAPTER_TITLE = '[SponsorBlock]: %(category_names)l'
 
 
 class ModifyChaptersPP(FFmpegPostProcessor):
-    def __init__(self, downloader, remove_chapters_patterns=None, remove_sponsor_segments=None,
-                 sponsorblock_chapter_title=DEFAULT_SPONSORBLOCK_CHAPTER_TITLE, force_keyframes=False):
+    def __init__(self, downloader, remove_chapters_patterns=None, remove_sponsor_segments=None, remove_ranges=None,
+                 *, sponsorblock_chapter_title=DEFAULT_SPONSORBLOCK_CHAPTER_TITLE, force_keyframes=False):
         FFmpegPostProcessor.__init__(self, downloader)
         self._remove_chapters_patterns = set(remove_chapters_patterns or [])
         self._remove_sponsor_segments = set(remove_sponsor_segments or [])
+        self._ranges_to_remove = set(remove_ranges or [])
         self._sponsorblock_chapter_title = sponsorblock_chapter_title
         self._force_keyframes = force_keyframes
 
     @PostProcessor._restrict_to(images=False)
     def run(self, info):
+        # Chapters must be preserved intact when downloading multiple formats of the same video.
         chapters, sponsor_chapters = self._mark_chapters_to_remove(
-            info.get('chapters') or [], info.get('sponsorblock_chapters') or [])
+            copy.deepcopy(info.get('chapters')) or [],
+            copy.deepcopy(info.get('sponsorblock_chapters')) or [])
         if not chapters and not sponsor_chapters:
             return [], info
 
@@ -97,6 +100,14 @@ class ModifyChaptersPP(FFmpegPostProcessor):
             if warn_no_chapter_to_remove:
                 self.to_screen('There are no matching SponsorBlock chapters')
 
+        sponsor_chapters.extend({
+            'start_time': start,
+            'end_time': end,
+            'category': 'manually_removed',
+            '_categories': [('manually_removed', start, end)],
+            'remove': True,
+        } for start, end in self._ranges_to_remove)
+
         return chapters, sponsor_chapters
 
     def _get_supported_subs(self, info):
@@ -117,7 +128,7 @@ class ModifyChaptersPP(FFmpegPostProcessor):
         cuts = []
 
         def append_cut(c):
-            assert 'remove' in c
+            assert 'remove' in c, 'Not a cut is appended to cuts'
             last_to_cut = cuts[-1] if cuts else None
             if last_to_cut and last_to_cut['end_time'] >= c['start_time']:
                 last_to_cut['end_time'] = max(last_to_cut['end_time'], c['end_time'])
@@ -145,7 +156,7 @@ class ModifyChaptersPP(FFmpegPostProcessor):
         new_chapters = []
 
         def append_chapter(c):
-            assert 'remove' not in c
+            assert 'remove' not in c, 'Cut is appended to chapters'
             length = c['end_time'] - c['start_time'] - excess_duration(c)
             # Chapter is completely covered by cuts or sponsors.
             if length <= 0:
@@ -228,7 +239,7 @@ class ModifyChaptersPP(FFmpegPostProcessor):
                     heapq.heappush(chapters, (c['start_time'], i, c))
             # (normal, sponsor) and (sponsor, sponsor)
             else:
-                assert '_categories' in c
+                assert '_categories' in c, 'Normal chapters overlap'
                 cur_chapter['_was_cut'] = True
                 c['_was_cut'] = True
                 # Push the part after the sponsor to PQ.

yt_dlp/postprocessor/sponskrub.py

@@ -11,9 +11,9 @@ from ..utils import (
     encodeFilename,
     shell_quote,
     str_or_none,
+    Popen,
     PostProcessingError,
     prepend_extension,
-    process_communicate_or_kill,
 )
 
 
@@ -81,8 +81,8 @@ class SponSkrubPP(PostProcessor):
 
         self.write_debug('sponskrub command line: %s' % shell_quote(cmd))
         pipe = None if self.get_param('verbose') else subprocess.PIPE
-        p = subprocess.Popen(cmd, stdout=pipe)
-        stdout = process_communicate_or_kill(p)[0]
+        p = Popen(cmd, stdout=pipe)
+        stdout = p.communicate_or_kill()[0]
 
         if p.returncode == 0:
             os.replace(temp_filename, filename)

yt_dlp/postprocessor/sponsorblock.py

@@ -1,6 +1,8 @@
+from hashlib import sha256
+import itertools
 import json
 import re
-from hashlib import sha256
+import time
 
 from .ffmpeg import FFmpegPostProcessor
 from ..compat import compat_urllib_parse_urlencode, compat_HTTPError
@@ -33,6 +35,7 @@ class SponsorBlockPP(FFmpegPostProcessor):
             self.to_screen(f'SponsorBlock is not supported for {extractor}')
             return [], info
 
+        self.to_screen('Fetching SponsorBlock segments')
         info['sponsorblock_chapters'] = self._get_sponsor_chapters(info, info['duration'])
         return [], info
 
@@ -79,18 +82,28 @@ class SponsorBlockPP(FFmpegPostProcessor):
             'service': service,
             'categories': json.dumps(self._categories),
         })
+        self.write_debug(f'SponsorBlock query: {url}')
         for d in self._get_json(url):
             if d['videoID'] == video_id:
                 return d['segments']
         return []
 
     def _get_json(self, url):
-        self.write_debug(f'SponsorBlock query: {url}')
-        try:
-            rsp = self._downloader.urlopen(sanitized_Request(url))
-        except network_exceptions as e:
-            if isinstance(e, compat_HTTPError) and e.code == 404:
-                return []
-            raise PostProcessingError(f'Unable to communicate with SponsorBlock API - {e}')
-
-        return json.loads(rsp.read().decode(rsp.info().get_param('charset') or 'utf-8'))
+        # While this is not an extractor, it behaves similar to one and
+        # so obey extractor_retries and sleep_interval_requests
+        max_retries = self.get_param('extractor_retries', 3)
+        sleep_interval = self.get_param('sleep_interval_requests') or 0
+        for retries in itertools.count():
+            try:
+                rsp = self._downloader.urlopen(sanitized_Request(url))
+                return json.loads(rsp.read().decode(rsp.info().get_param('charset') or 'utf-8'))
+            except network_exceptions as e:
+                if isinstance(e, compat_HTTPError) and e.code == 404:
+                    return []
+                if retries < max_retries:
+                    self.report_warning(f'{e}. Retrying...')
+                    if sleep_interval > 0:
+                        self.to_screen(f'Sleeping {sleep_interval} seconds ...')
+                        time.sleep(sleep_interval)
+                    continue
+                raise PostProcessingError(f'Unable to communicate with SponsorBlock API: {e}')

yt_dlp/update.py

@@ -10,7 +10,7 @@ import traceback
 from zipimport import zipimporter
 
 from .compat import compat_realpath
-from .utils import encode_compat_str
+from .utils import encode_compat_str, Popen
 
 from .version import __version__
 
@@ -33,10 +33,11 @@ def rsa_verify(message, signature, key):
 
 def detect_variant():
     if hasattr(sys, 'frozen'):
+        prefix = 'mac' if sys.platform == 'darwin' else 'win'
         if getattr(sys, '_MEIPASS', None):
             if sys._MEIPASS == os.path.dirname(sys.executable):
-                return 'dir'
-            return 'exe'
+                return f'{prefix}_dir'
+            return f'{prefix}_exe'
         return 'py2exe'
     elif isinstance(globals().get('__loader__'), zipimporter):
         return 'zip'
@@ -46,12 +47,14 @@ def detect_variant():
 
 
 _NON_UPDATEABLE_REASONS = {
-    'exe': None,
+    'win_exe': None,
     'zip': None,
-    'dir': 'Auto-update is not supported for unpackaged windows executable. Re-download the latest release',
-    'py2exe': 'There is no official release for py2exe executable. Build it again with the latest source code',
-    'source': 'You cannot update when running from source code',
-    'unknown': 'It looks like you installed yt-dlp with a package manager, pip, setup.py or a tarball. Use that to update',
+    'mac_exe': None,
+    'py2exe': None,
+    'win_dir': 'Auto-update is not supported for unpackaged windows executable; Re-download the latest release',
+    'mac_dir': 'Auto-update is not supported for unpackaged MacOS executable; Re-download the latest release',
+    'source': 'You cannot update when running from source code; Use git to pull the latest changes',
+    'unknown': 'It looks like you installed yt-dlp with a package manager, pip, setup.py or a tarball; Use that to update',
 }
 
 
@@ -59,11 +62,203 @@ def is_non_updateable():
     return _NON_UPDATEABLE_REASONS.get(detect_variant(), _NON_UPDATEABLE_REASONS['unknown'])
 
 
+def run_update(ydl):
+    """
+    Update the program file with the latest version from the repository
+    Returns whether the program should terminate
+    """
+
+    JSON_URL = 'https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest'
+
+    def report_error(msg, expected=False):
+        ydl.report_error(msg, tb='' if expected else None)
+
+    def report_unable(action, expected=False):
+        report_error(f'Unable to {action}', expected)
+
+    def report_permission_error(file):
+        report_unable(f'write to {file}; Try running as administrator', True)
+
+    def report_network_error(action, delim=';'):
+        report_unable(f'{action}{delim} Visit  https://github.com/yt-dlp/yt-dlp/releases/latest', True)
+
+    def calc_sha256sum(path):
+        h = hashlib.sha256()
+        b = bytearray(128 * 1024)
+        mv = memoryview(b)
+        with open(os.path.realpath(path), 'rb', buffering=0) as f:
+            for n in iter(lambda: f.readinto(mv), 0):
+                h.update(mv[:n])
+        return h.hexdigest()
+
+    # Download and check versions info
+    try:
+        version_info = ydl._opener.open(JSON_URL).read().decode('utf-8')
+        version_info = json.loads(version_info)
+    except Exception:
+        return report_network_error('obtain version info', delim='; Please try again later or')
+
+    def version_tuple(version_str):
+        return tuple(map(int, version_str.split('.')))
+
+    version_id = version_info['tag_name']
+    if version_tuple(__version__) >= version_tuple(version_id):
+        ydl.to_screen(f'yt-dlp is up to date ({__version__})')
+        return
+
+    err = is_non_updateable()
+    if err:
+        ydl.to_screen(f'Latest version: {version_id}, Current version: {__version__}')
+        return report_error(err, True)
+
+    # sys.executable is set to the full pathname of the exe-file for py2exe
+    # though symlinks are not followed so that we need to do this manually
+    # with help of realpath
+    filename = compat_realpath(sys.executable if hasattr(sys, 'frozen') else sys.argv[0])
+    ydl.to_screen(f'Current version {__version__}; Build Hash {calc_sha256sum(filename)}')
+    ydl.to_screen(f'Updating to version {version_id} ...')
+
+    version_labels = {
+        'zip_3': '',
+        'win_exe_64': '.exe',
+        'py2exe_64': '_min.exe',
+        'win_exe_32': '_x86.exe',
+        'mac_exe_64': '_macos',
+    }
+
+    def get_bin_info(bin_or_exe, version):
+        label = version_labels['%s_%s' % (bin_or_exe, version)]
+        return next((i for i in version_info['assets'] if i['name'] == 'yt-dlp%s' % label), {})
+
+    def get_sha256sum(bin_or_exe, version):
+        filename = 'yt-dlp%s' % version_labels['%s_%s' % (bin_or_exe, version)]
+        urlh = next(
+            (i for i in version_info['assets'] if i['name'] in ('SHA2-256SUMS')),
+            {}).get('browser_download_url')
+        if not urlh:
+            return None
+        hash_data = ydl._opener.open(urlh).read().decode('utf-8')
+        return dict(ln.split()[::-1] for ln in hash_data.splitlines()).get(filename)
+
+    if not os.access(filename, os.W_OK):
+        return report_permission_error(filename)
+
+    # PyInstaller
+    variant = detect_variant()
+    if variant in ('win_exe', 'py2exe'):
+        directory = os.path.dirname(filename)
+        if not os.access(directory, os.W_OK):
+            return report_permission_error(directory)
+        try:
+            if os.path.exists(filename + '.old'):
+                os.remove(filename + '.old')
+        except (IOError, OSError):
+            return report_unable('remove the old version')
+
+        try:
+            arch = platform.architecture()[0][:2]
+            url = get_bin_info(variant, arch).get('browser_download_url')
+            if not url:
+                return report_network_error('fetch updates')
+            urlh = ydl._opener.open(url)
+            newcontent = urlh.read()
+            urlh.close()
+        except (IOError, OSError):
+            return report_network_error('download latest version')
+
+        try:
+            with open(filename + '.new', 'wb') as outf:
+                outf.write(newcontent)
+        except (IOError, OSError):
+            return report_permission_error(f'{filename}.new')
+
+        expected_sum = get_sha256sum(variant, arch)
+        if not expected_sum:
+            ydl.report_warning('no hash information found for the release')
+        elif calc_sha256sum(filename + '.new') != expected_sum:
+            report_network_error('verify the new executable')
+            try:
+                os.remove(filename + '.new')
+            except OSError:
+                return report_unable('remove corrupt download')
+
+        try:
+            os.rename(filename, filename + '.old')
+        except (IOError, OSError):
+            return report_unable('move current version')
+        try:
+            os.rename(filename + '.new', filename)
+        except (IOError, OSError):
+            report_unable('overwrite current version')
+            os.rename(filename + '.old', filename)
+            return
+        try:
+            # Continues to run in the background
+            Popen(
+                'ping 127.0.0.1 -n 5 -w 1000 & del /F "%s.old"' % filename,
+                shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+            ydl.to_screen('Updated yt-dlp to version %s' % version_id)
+            return True  # Exit app
+        except OSError:
+            report_unable('delete the old version')
+
+    elif variant in ('zip', 'mac_exe'):
+        pack_type = '3' if variant == 'zip' else '64'
+        try:
+            url = get_bin_info(variant, pack_type).get('browser_download_url')
+            if not url:
+                return report_network_error('fetch updates')
+            urlh = ydl._opener.open(url)
+            newcontent = urlh.read()
+            urlh.close()
+        except (IOError, OSError):
+            return report_network_error('download the latest version')
+
+        expected_sum = get_sha256sum(variant, pack_type)
+        if not expected_sum:
+            ydl.report_warning('no hash information found for the release')
+        elif hashlib.sha256(newcontent).hexdigest() != expected_sum:
+            return report_network_error('verify the new package')
+
+        try:
+            with open(filename, 'wb') as outf:
+                outf.write(newcontent)
+        except (IOError, OSError):
+            return report_unable('overwrite current version')
+
+        ydl.to_screen('Updated yt-dlp to version %s; Restart yt-dlp to use the new version' % version_id)
+        return
+
+    assert False, f'Unhandled variant: {variant}'
+
+
+'''  # UNUSED
+def get_notes(versions, fromVersion):
+    notes = []
+    for v, vdata in sorted(versions.items()):
+        if v > fromVersion:
+            notes.extend(vdata.get('notes', []))
+    return notes
+
+
+def print_notes(to_screen, versions, fromVersion=__version__):
+    notes = get_notes(versions, fromVersion)
+    if notes:
+        to_screen('PLEASE NOTE:')
+        for note in notes:
+            to_screen(note)
+'''
+
+
 def update_self(to_screen, verbose, opener):
-    ''' Exists for backward compatibility. Use run_update(ydl) instead '''
+    ''' Exists for backward compatibility '''
 
     printfn = to_screen
 
+    printfn(
+        'WARNING: "yt_dlp.update.update_self" is deprecated and may be removed in a future version. '
+        'Use "yt_dlp.update.run_update(ydl)" instead')
+
     class FakeYDL():
         _opener = opener
         to_screen = printfn
@@ -91,179 +286,3 @@ def update_self(to_screen, verbose, opener):
                 printfn(tb)
 
     return run_update(FakeYDL())
-
-
-def run_update(ydl):
-    """
-    Update the program file with the latest version from the repository
-    Returns whether the program should terminate
-    """
-
-    JSON_URL = 'https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest'
-
-    def report_error(msg, network=False, expected=False, delim=';'):
-        if network:
-            msg += '%s Visit  https://github.com/yt-dlp/yt-dlp/releases/latest' % delim
-        ydl.report_error(msg, tb='' if network or expected else None)
-
-    def calc_sha256sum(path):
-        h = hashlib.sha256()
-        b = bytearray(128 * 1024)
-        mv = memoryview(b)
-        with open(os.path.realpath(path), 'rb', buffering=0) as f:
-            for n in iter(lambda: f.readinto(mv), 0):
-                h.update(mv[:n])
-        return h.hexdigest()
-
-    # Download and check versions info
-    try:
-        version_info = ydl._opener.open(JSON_URL).read().decode('utf-8')
-        version_info = json.loads(version_info)
-    except Exception:
-        return report_error('can\'t obtain versions info. Please try again later ', True, delim='or')
-
-    def version_tuple(version_str):
-        return tuple(map(int, version_str.split('.')))
-
-    version_id = version_info['tag_name']
-    if version_tuple(__version__) >= version_tuple(version_id):
-        ydl.to_screen(f'yt-dlp is up to date ({__version__})')
-        return
-
-    err = is_non_updateable()
-    if err:
-        ydl.to_screen(f'Latest version: {version_id}, Current version: {__version__}')
-        return report_error(err, expected=True)
-
-    # sys.executable is set to the full pathname of the exe-file for py2exe
-    # though symlinks are not followed so that we need to do this manually
-    # with help of realpath
-    filename = compat_realpath(sys.executable if hasattr(sys, 'frozen') else sys.argv[0])
-    ydl.to_screen(f'Current version {__version__}; Build Hash {calc_sha256sum(filename)}')
-    ydl.to_screen(f'Updating to version {version_id} ...')
-
-    version_labels = {
-        'zip_3': '',
-        'exe_64': '.exe',
-        'exe_32': '_x86.exe',
-    }
-
-    def get_bin_info(bin_or_exe, version):
-        label = version_labels['%s_%s' % (bin_or_exe, version)]
-        return next((i for i in version_info['assets'] if i['name'] == 'yt-dlp%s' % label), {})
-
-    def get_sha256sum(bin_or_exe, version):
-        filename = 'yt-dlp%s' % version_labels['%s_%s' % (bin_or_exe, version)]
-        urlh = next(
-            (i for i in version_info['assets'] if i['name'] in ('SHA2-256SUMS')),
-            {}).get('browser_download_url')
-        if not urlh:
-            return None
-        hash_data = ydl._opener.open(urlh).read().decode('utf-8')
-        return dict(ln.split()[::-1] for ln in hash_data.splitlines()).get(filename)
-
-    if not os.access(filename, os.W_OK):
-        return report_error('no write permissions on %s' % filename, expected=True)
-
-    # PyInstaller
-    if hasattr(sys, 'frozen'):
-        exe = filename
-        directory = os.path.dirname(exe)
-        if not os.access(directory, os.W_OK):
-            return report_error('no write permissions on %s' % directory, expected=True)
-        try:
-            if os.path.exists(filename + '.old'):
-                os.remove(filename + '.old')
-        except (IOError, OSError):
-            return report_error('unable to remove the old version')
-
-        try:
-            arch = platform.architecture()[0][:2]
-            url = get_bin_info('exe', arch).get('browser_download_url')
-            if not url:
-                return report_error('unable to fetch updates', True)
-            urlh = ydl._opener.open(url)
-            newcontent = urlh.read()
-            urlh.close()
-        except (IOError, OSError, StopIteration):
-            return report_error('unable to download latest version', True)
-
-        try:
-            with open(exe + '.new', 'wb') as outf:
-                outf.write(newcontent)
-        except (IOError, OSError):
-            return report_error('unable to write the new version')
-
-        expected_sum = get_sha256sum('exe', arch)
-        if not expected_sum:
-            ydl.report_warning('no hash information found for the release')
-        elif calc_sha256sum(exe + '.new') != expected_sum:
-            report_error('unable to verify the new executable', True)
-            try:
-                os.remove(exe + '.new')
-            except OSError:
-                return report_error('unable to remove corrupt download')
-
-        try:
-            os.rename(exe, exe + '.old')
-        except (IOError, OSError):
-            return report_error('unable to move current version')
-        try:
-            os.rename(exe + '.new', exe)
-        except (IOError, OSError):
-            report_error('unable to overwrite current version')
-            os.rename(exe + '.old', exe)
-            return
-        try:
-            # Continues to run in the background
-            subprocess.Popen(
-                'ping 127.0.0.1 -n 5 -w 1000 & del /F "%s.old"' % exe,
-                shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
-            ydl.to_screen('Updated yt-dlp to version %s' % version_id)
-            return True  # Exit app
-        except OSError:
-            report_error('unable to delete old version')
-
-    # Zip unix package
-    elif isinstance(globals().get('__loader__'), zipimporter):
-        try:
-            url = get_bin_info('zip', '3').get('browser_download_url')
-            if not url:
-                return report_error('unable to fetch updates', True)
-            urlh = ydl._opener.open(url)
-            newcontent = urlh.read()
-            urlh.close()
-        except (IOError, OSError, StopIteration):
-            return report_error('unable to download latest version', True)
-
-        expected_sum = get_sha256sum('zip', '3')
-        if not expected_sum:
-            ydl.report_warning('no hash information found for the release')
-        elif hashlib.sha256(newcontent).hexdigest() != expected_sum:
-            return report_error('unable to verify the new zip', True)
-
-        try:
-            with open(filename, 'wb') as outf:
-                outf.write(newcontent)
-        except (IOError, OSError):
-            return report_error('unable to overwrite current version')
-
-    ydl.to_screen('Updated yt-dlp to version %s; Restart yt-dlp to use the new version' % version_id)
-
-
-'''  # UNUSED
-def get_notes(versions, fromVersion):
-    notes = []
-    for v, vdata in sorted(versions.items()):
-        if v > fromVersion:
-            notes.extend(vdata.get('notes', []))
-    return notes
-
-
-def print_notes(to_screen, versions, fromVersion=__version__):
-    notes = get_notes(versions, fromVersion)
-    if notes:
-        to_screen('PLEASE NOTE:')
-        for note in notes:
-            to_screen(note)
-'''

yt_dlp/utils.py

@@ -18,7 +18,7 @@ import functools
 import gzip
 import hashlib
 import hmac
-import imp
+import importlib.util
 import io
 import itertools
 import json
@@ -2272,6 +2272,20 @@ def process_communicate_or_kill(p, *args, **kwargs):
         raise
 
 
+class Popen(subprocess.Popen):
+    if sys.platform == 'win32':
+        _startupinfo = subprocess.STARTUPINFO()
+        _startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
+    else:
+        _startupinfo = None
+
+    def __init__(self, *args, **kwargs):
+        super(Popen, self).__init__(*args, **kwargs, startupinfo=self._startupinfo)
+
+    def communicate_or_kill(self, *args, **kwargs):
+        return process_communicate_or_kill(self, *args, **kwargs)
+
+
 def get_subprocess_encoding():
     if sys.platform == 'win32' and sys.getwindowsversion()[0] >= 5:
         # For subprocess calls, encode with locale encoding
@@ -2342,14 +2356,25 @@ def decodeOption(optval):
     return optval
 
 
+_timetuple = collections.namedtuple('Time', ('hours', 'minutes', 'seconds', 'milliseconds'))
+
+
+def timetuple_from_msec(msec):
+    secs, msec = divmod(msec, 1000)
+    mins, secs = divmod(secs, 60)
+    hrs, mins = divmod(mins, 60)
+    return _timetuple(hrs, mins, secs, msec)
+
+
 def formatSeconds(secs, delim=':', msec=False):
-    if secs > 3600:
-        ret = '%d%s%02d%s%02d' % (secs // 3600, delim, (secs % 3600) // 60, delim, secs % 60)
-    elif secs > 60:
-        ret = '%d%s%02d' % (secs // 60, delim, secs % 60)
+    time = timetuple_from_msec(secs * 1000)
+    if time.hours:
+        ret = '%d%s%02d%s%02d' % (time.hours, delim, time.minutes, delim, time.seconds)
+    elif time.minutes:
+        ret = '%d%s%02d' % (time.minutes, delim, time.seconds)
     else:
-        ret = '%d' % secs
-    return '%s.%03d' % (ret, secs % 1) if msec else ret
+        ret = '%d' % time.seconds
+    return '%s.%03d' % (ret, time.milliseconds) if msec else ret
 
 
 def _ssl_load_windows_store_certs(ssl_context, storename):
@@ -3689,14 +3714,14 @@ def parse_resolution(s):
     if s is None:
         return {}
 
-    mobj = re.search(r'\b(?P<w>\d+)\s*[xX×]\s*(?P<h>\d+)\b', s)
+    mobj = re.search(r'(?<![a-zA-Z0-9])(?P<w>\d+)\s*[xX×,]\s*(?P<h>\d+)(?![a-zA-Z0-9])', s)
     if mobj:
         return {
             'width': int(mobj.group('w')),
             'height': int(mobj.group('h')),
         }
 
-    mobj = re.search(r'\b(\d+)[pPiI]\b', s)
+    mobj = re.search(r'(?<![a-zA-Z0-9])(\d+)[pPiI](?![a-zA-Z0-9])', s)
     if mobj:
         return {'height': int(mobj.group(1))}
 
@@ -3966,8 +3991,7 @@ def check_executable(exe, args=[]):
     """ Checks if the given binary is installed somewhere in PATH, and returns its name.
     args can be a list of arguments for a short output (like -version) """
     try:
-        process_communicate_or_kill(subprocess.Popen(
-            [exe] + args, stdout=subprocess.PIPE, stderr=subprocess.PIPE))
+        Popen([exe] + args, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate_or_kill()
     except OSError:
         return False
     return exe
@@ -3981,10 +4005,9 @@ def get_exe_version(exe, args=['--version'],
         # STDIN should be redirected too. On UNIX-like systems, ffmpeg triggers
         # SIGTTOU if yt-dlp is run in the background.
         # See https://github.com/ytdl-org/youtube-dl/issues/955#issuecomment-209789656
-        out, _ = process_communicate_or_kill(subprocess.Popen(
-            [encodeArgument(exe)] + args,
-            stdin=subprocess.PIPE,
-            stdout=subprocess.PIPE, stderr=subprocess.STDOUT))
+        out, _ = Popen(
+            [encodeArgument(exe)] + args, stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate_or_kill()
     except OSError:
         return False
     if isinstance(out, bytes):  # Python 2.x
@@ -4618,12 +4641,21 @@ def parse_codecs(codecs_str):
         return {}
     split_codecs = list(filter(None, map(
         str.strip, codecs_str.strip().strip(',').split(','))))
-    vcodec, acodec = None, None
+    vcodec, acodec, hdr = None, None, None
     for full_codec in split_codecs:
         codec = full_codec.split('.')[0]
-        if codec in ('avc1', 'avc2', 'avc3', 'avc4', 'vp9', 'vp8', 'hev1', 'hev2', 'h263', 'h264', 'mp4v', 'hvc1', 'av01', 'theora'):
+        if codec in ('avc1', 'avc2', 'avc3', 'avc4', 'vp9', 'vp8', 'hev1', 'hev2', 'h263', 'h264', 'mp4v', 'hvc1', 'av01', 'theora', 'dvh1', 'dvhe'):
             if not vcodec:
                 vcodec = full_codec
+                if codec in ('dvh1', 'dvhe'):
+                    hdr = 'DV'
+                elif codec == 'vp9' and vcodec.startswith('vp9.2'):
+                    hdr = 'HDR10'
+                elif codec == 'av01':
+                    parts = full_codec.split('.')
+                    if len(parts) > 3 and parts[3] == '10':
+                        hdr = 'HDR10'
+                        vcodec = '.'.join(parts[:4])
         elif codec in ('mp4a', 'opus', 'vorbis', 'mp3', 'aac', 'ac-3', 'ec-3', 'eac3', 'dtsc', 'dtse', 'dtsh', 'dtsl'):
             if not acodec:
                 acodec = full_codec
@@ -4639,6 +4671,7 @@ def parse_codecs(codecs_str):
         return {
             'vcodec': vcodec or 'none',
             'acodec': acodec or 'none',
+            'dynamic_range': hdr,
         }
     return {}
 
@@ -4756,7 +4789,6 @@ def _match_one(filter_part, dct, incomplete):
         (?P<key>[a-z_]+)
         \s*(?P<negation>!\s*)?(?P<op>%s)(?P<none_inclusive>\s*\?)?\s*
         (?:
-            (?P<intval>[0-9.]+(?:[kKmMgGtTpPeEzZyY]i?[Bb]?)?)|
             (?P<quote>["\'])(?P<quotedstrval>.+?)(?P=quote)|
             (?P<strval>.+?)
         )
@@ -4764,40 +4796,35 @@ def _match_one(filter_part, dct, incomplete):
         ''' % '|'.join(map(re.escape, COMPARISON_OPERATORS.keys())))
     m = operator_rex.search(filter_part)
     if m:
-        unnegated_op = COMPARISON_OPERATORS[m.group('op')]
-        if m.group('negation'):
+        m = m.groupdict()
+        unnegated_op = COMPARISON_OPERATORS[m['op']]
+        if m['negation']:
             op = lambda attr, value: not unnegated_op(attr, value)
         else:
             op = unnegated_op
-        actual_value = dct.get(m.group('key'))
-        if (m.group('quotedstrval') is not None
-            or m.group('strval') is not None
+        comparison_value = m['quotedstrval'] or m['strval'] or m['intval']
+        if m['quote']:
+            comparison_value = comparison_value.replace(r'\%s' % m['quote'], m['quote'])
+        actual_value = dct.get(m['key'])
+        numeric_comparison = None
+        if isinstance(actual_value, compat_numeric_types):
             # If the original field is a string and matching comparisonvalue is
             # a number we should respect the origin of the original field
             # and process comparison value as a string (see
-            # https://github.com/ytdl-org/youtube-dl/issues/11082).
-            or actual_value is not None and m.group('intval') is not None
-                and isinstance(actual_value, compat_str)):
-            comparison_value = m.group('quotedstrval') or m.group('strval') or m.group('intval')
-            quote = m.group('quote')
-            if quote is not None:
-                comparison_value = comparison_value.replace(r'\%s' % quote, quote)
-        else:
-            if m.group('op') in STRING_OPERATORS:
-                raise ValueError('Operator %s only supports string values!' % m.group('op'))
+            # https://github.com/ytdl-org/youtube-dl/issues/11082)
             try:
-                comparison_value = int(m.group('intval'))
+                numeric_comparison = int(comparison_value)
             except ValueError:
-                comparison_value = parse_filesize(m.group('intval'))
-                if comparison_value is None:
-                    comparison_value = parse_filesize(m.group('intval') + 'B')
-                if comparison_value is None:
-                    raise ValueError(
-                        'Invalid integer value %r in filter part %r' % (
-                            m.group('intval'), filter_part))
+                numeric_comparison = parse_filesize(comparison_value)
+                if numeric_comparison is None:
+                    numeric_comparison = parse_filesize(f'{comparison_value}B')
+                if numeric_comparison is None:
+                    numeric_comparison = parse_duration(comparison_value)
+        if numeric_comparison is not None and m['op'] in STRING_OPERATORS:
+            raise ValueError('Operator %s only supports string values!' % m['op'])
         if actual_value is None:
-            return incomplete or m.group('none_inclusive')
-        return op(actual_value, comparison_value)
+            return incomplete or m['none_inclusive']
+        return op(actual_value, comparison_value if numeric_comparison is None else numeric_comparison)
 
     UNARY_OPERATORS = {
         '': lambda v: (v is True) if isinstance(v, bool) else (v is not None),
@@ -4851,7 +4878,12 @@ def parse_dfxp_time_expr(time_expr):
 
 
 def srt_subtitles_timecode(seconds):
-    return '%02d:%02d:%02d,%03d' % (seconds / 3600, (seconds % 3600) / 60, seconds % 60, (seconds % 1) * 1000)
+    return '%02d:%02d:%02d,%03d' % timetuple_from_msec(seconds * 1000)
+
+
+def ass_subtitles_timecode(seconds):
+    time = timetuple_from_msec(seconds * 1000)
+    return '%01d:%02d:%02d.%02d' % (*time[:-1], time.milliseconds / 10)
 
 
 def dfxp2srt(dfxp_data):
@@ -6135,11 +6167,11 @@ def write_xattr(path, key, value):
                        + [encodeFilename(path, True)])
 
                 try:
-                    p = subprocess.Popen(
+                    p = Popen(
                         cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
                 except EnvironmentError as e:
                     raise XAttrMetadataError(e.errno, e.strerror)
-                stdout, stderr = process_communicate_or_kill(p)
+                stdout, stderr = p.communicate_or_kill()
                 stderr = stderr.decode('utf-8', 'replace')
                 if p.returncode != 0:
                     raise XAttrMetadataError(p.returncode, stderr)
@@ -6308,12 +6340,13 @@ def get_executable_path():
 
 
 def load_plugins(name, suffix, namespace):
-    plugin_info = [None]
     classes = {}
     try:
-        plugin_info = imp.find_module(
-            name, [os.path.join(get_executable_path(), 'ytdlp_plugins')])
-        plugins = imp.load_module(name, *plugin_info)
+        plugins_spec = importlib.util.spec_from_file_location(
+            name, os.path.join(get_executable_path(), 'ytdlp_plugins', name, '__init__.py'))
+        plugins = importlib.util.module_from_spec(plugins_spec)
+        sys.modules[plugins_spec.name] = plugins
+        plugins_spec.loader.exec_module(plugins)
         for name in dir(plugins):
             if name in namespace:
                 continue
@@ -6321,11 +6354,8 @@ def load_plugins(name, suffix, namespace):
                 continue
             klass = getattr(plugins, name)
             classes[name] = namespace[name] = klass
-    except ImportError:
+    except FileNotFoundError:
         pass
-    finally:
-        if plugin_info[0] is not None:
-            plugin_info[0].close()
     return classes
 
 
@@ -6458,7 +6488,7 @@ def jwt_encode_hs256(payload_data, key, headers={}):
 
 def supports_terminal_sequences(stream):
     if compat_os_name == 'nt':
-        if get_windows_version() < (10, ):
+        if get_windows_version() < (10, 0, 10586):
             return False
     elif not os.getenv('TERM'):
         return False

yt_dlp/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.10.09'
+__version__ = '2021.10.22'

yt_dlp/webvtt.py

@@ -13,7 +13,7 @@ in RFC 8216 §3.5 <https://tools.ietf.org/html/rfc8216#section-3.5>.
 
 import re
 import io
-from .utils import int_or_none
+from .utils import int_or_none, timetuple_from_msec
 from .compat import (
     compat_str as str,
     compat_Pattern,
@@ -124,11 +124,7 @@ def _format_ts(ts):
     Convert an MPEG PES timestamp into a WebVTT timestamp.
     This will lose sub-millisecond precision.
     """
-    msec = int((ts + 45) // 90)
-    secs, msec = divmod(msec, 1000)
-    mins, secs = divmod(secs, 60)
-    hrs, mins = divmod(mins, 60)
-    return '%02u:%02u:%02u.%03u' % (hrs, mins, secs, msec)
+    return '%02u:%02u:%02u.%03u' % timetuple_from_msec(int((ts + 45) // 90))
 
 
 class Block(object):