Release 2021.06.08

Closes #386

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.05.20
+ [debug] yt-dlp version 2021.06.01
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.05.20
+ [debug] yt-dlp version 2021.06.01
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.05.20. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.06.01. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.05.20**
+- [ ] I've verified that I'm running yt-dlp version **2021.06.01**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/build.yml

@@ -7,13 +7,13 @@ on:
 
 jobs:
   build_unix:
-
     runs-on: ubuntu-latest
 
     outputs:
       ytdlp_version: ${{ steps.bump_version.outputs.ytdlp_version }}
       upload_url: ${{ steps.create_release.outputs.upload_url }}
-      sha2_unix: ${{ steps.sha2_file.outputs.sha2_unix }}
+      sha256_unix: ${{ steps.sha256_file.outputs.sha256_unix }}
+      sha512_unix: ${{ steps.sha512_file.outputs.sha512_unix }}
 
     steps:
     - uses: actions/checkout@v2
@@ -29,7 +29,7 @@ jobs:
     - name: Print version
       run: echo "${{ steps.bump_version.outputs.ytdlp_version }}"
     - name: Run Make
-      run: make
+      run: make all tar
     - name: Create Release
       id: create_release
       uses: actions/create-release@v1
@@ -44,7 +44,7 @@ jobs:
         draft: false
         prerelease: false
     - name: Upload yt-dlp Unix binary
-      id: upload-release-asset 
+      id: upload-release-asset
       uses: actions/upload-release-asset@v1
       env:
         GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -53,9 +53,21 @@ jobs:
         asset_path: ./yt-dlp
         asset_name: yt-dlp
         asset_content_type: application/octet-stream
+    - name: Upload Source tar
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ steps.create_release.outputs.upload_url }}
+        asset_path: ./yt-dlp.tar.gz
+        asset_name: yt-dlp.tar.gz
+        asset_content_type: application/gzip
     - name: Get SHA2-256SUMS for yt-dlp
-      id: sha2_file
-      run: echo "::set-output name=sha2_unix::$(sha256sum yt-dlp | awk '{print $1}')"
+      id: sha256_file
+      run: echo "::set-output name=sha256_unix::$(sha256sum yt-dlp | awk '{print $1}')"
+    - name: Get SHA2-512SUMS for yt-dlp
+      id: sha512_file
+      run: echo "::set-output name=sha512_unix::$(sha512sum yt-dlp | awk '{print $1}')"
     - name: Install dependencies for pypi
       env:
         PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
@@ -74,13 +86,12 @@ jobs:
         twine upload dist/*
 
   build_windows:
-
     runs-on: windows-latest
+    needs: build_unix
 
     outputs:
-      sha2_windows: ${{ steps.sha2_file_win.outputs.sha2_windows }}
-
-    needs: build_unix
+      sha256_windows: ${{ steps.sha256_file_win.outputs.sha256_windows }}
+      sha512_windows: ${{ steps.sha512_file_win.outputs.sha512_windows }}
 
     steps:
     - uses: actions/checkout@v2
@@ -110,17 +121,19 @@ jobs:
         asset_name: yt-dlp.exe
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for yt-dlp.exe
-      id: sha2_file_win
-      run: echo "::set-output name=sha2_windows::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())"
+      id: sha256_file_win
+      run: echo "::set-output name=sha256_windows::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA256).Hash.ToLower())"
+    - name: Get SHA2-512SUMS for yt-dlp.exe
+      id: sha512_file_win
+      run: echo "::set-output name=sha512_windows::$((Get-FileHash dist\yt-dlp.exe -Algorithm SHA512).Hash.ToLower())"
 
   build_windows32:
-
     runs-on: windows-latest
+    needs: [build_unix, build_windows]
 
     outputs:
-      sha2_windows32: ${{ steps.sha2_file_win32.outputs.sha2_windows32 }}
-
-    needs: [build_unix, build_windows]
+      sha256_windows32: ${{ steps.sha256_file_win32.outputs.sha256_windows32 }}
+      sha512_windows32: ${{ steps.sha512_file_win32.outputs.sha512_windows32 }}
 
     steps:
     - uses: actions/checkout@v2
@@ -151,20 +164,28 @@ jobs:
         asset_name: yt-dlp_x86.exe
         asset_content_type: application/vnd.microsoft.portable-executable
     - name: Get SHA2-256SUMS for yt-dlp_x86.exe
-      id: sha2_file_win32
-      run: echo "::set-output name=sha2_windows32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA256).Hash.ToLower())"
+      id: sha256_file_win32
+      run: echo "::set-output name=sha256_windows32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA256).Hash.ToLower())"
+    - name: Get SHA2-512SUMS for yt-dlp_x86.exe
+      id: sha512_file_win32
+      run: echo "::set-output name=sha512_windows32::$((Get-FileHash dist\yt-dlp_x86.exe -Algorithm SHA512).Hash.ToLower())"
+
+  finish:
+    runs-on: ubuntu-latest
+    needs: [build_unix, build_windows, build_windows32]
+
+    steps:
     - name: Make SHA2-256SUMS file
       env:
-        SHA2_WINDOWS: ${{ needs.build_windows.outputs.sha2_windows }}
-        SHA2_WINDOWS32: ${{ steps.sha2_file_win32.outputs.sha2_windows32 }}
-        SHA2_UNIX: ${{ needs.build_unix.outputs.sha2_unix }}
+        SHA256_WINDOWS: ${{ needs.build_windows.outputs.sha256_windows }}
+        SHA256_WINDOWS32: ${{ needs.build_windows32.outputs.sha256_windows32 }}
+        SHA256_UNIX: ${{ needs.build_unix.outputs.sha256_unix }}
         YTDLP_VERSION: ${{ needs.build_unix.outputs.ytdlp_version }}
       run: |
-        echo "version:${env:YTDLP_VERSION}" >> SHA2-256SUMS
-        echo "yt-dlp.exe:${env:SHA2_WINDOWS}" >> SHA2-256SUMS
-        echo "yt-dlp_x86.exe:${env:SHA2_WINDOWS32}" >> SHA2-256SUMS
-        echo "yt-dlp:${env:SHA2_UNIX}" >> SHA2-256SUMS
-
+        echo "version:${{ env.YTDLP_VERSION }}" >> SHA2-256SUMS
+        echo "yt-dlp.exe:${{ env.SHA256_WINDOWS }}" >> SHA2-256SUMS
+        echo "yt-dlp_x86.exe:${{ env.SHA256_WINDOWS32 }}" >> SHA2-256SUMS
+        echo "yt-dlp:${{ env.SHA256_UNIX }}" >> SHA2-256SUMS
     - name: Upload 256SUMS file
       id: upload-sums
       uses: actions/upload-release-asset@v1
@@ -175,3 +196,22 @@ jobs:
         asset_path: ./SHA2-256SUMS
         asset_name: SHA2-256SUMS
         asset_content_type: text/plain
+    - name: Make SHA2-512SUMS file
+      env:
+        SHA512_WINDOWS: ${{ needs.build_windows.outputs.sha512_windows }}
+        SHA512_WINDOWS32: ${{ needs.build_windows32.outputs.sha512_windows32 }}
+        SHA512_UNIX: ${{ needs.build_unix.outputs.sha512_unix }}
+      run: |
+        echo "${{ env.SHA512_WINDOWS }}  yt-dlp.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_WINDOWS32 }}  yt-dlp_x86.exe" >> SHA2-512SUMS
+        echo "${{ env.SHA512_UNIX }}  yt-dlp" >> SHA2-512SUMS
+    - name: Upload 512SUMS file
+      id: upload-512sums
+      uses: actions/upload-release-asset@v1
+      env:
+        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      with:
+        upload_url: ${{ needs.build_unix.outputs.upload_url }}
+        asset_path: ./SHA2-512SUMS
+        asset_name: SHA2-512SUMS
+        asset_content_type: text/plain

.github/workflows/core.yml

@@ -9,53 +9,23 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-18.04]
-        # TODO: python 2.6
-        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
-        python-impl: [cpython]
-        ytdl-test-set: [core]
+        python-version: [3.6, 3.7, 3.8, 3.9, pypy-3.6, pypy-3.7]
         run-tests-ext: [sh]
         include:
-        # python 3.2 is only available on windows via setup-python
         - os: windows-latest
-          python-version: 3.2
-          python-impl: cpython
-          ytdl-test-set: core
+          python-version: 3.4  # Windows x86 build is still in 3.4
           run-tests-ext: bat
-        # jython
-        - os: ubuntu-latest
-          python-impl: jython
-          ytdl-test-set: core
-          run-tests-ext: sh
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v2
-      if: ${{ matrix.python-impl == 'cpython' }}
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Set up Java 8
-      if: ${{ matrix.python-impl == 'jython' }}
-      uses: actions/setup-java@v1
-      with:
-        java-version: 8
-    - name: Install Jython
-      if: ${{ matrix.python-impl == 'jython' }}
-      run: |
-        wget https://repo1.maven.org/maven2/org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
-        java -jar jython-installer.jar -s -d "$HOME/jython"
-        echo "$HOME/jython/bin" >> $GITHUB_PATH
     - name: Install nose
-      if: ${{ matrix.python-impl != 'jython' }}
       run: pip install nose
-    - name: Install nose (Jython)
-      if: ${{ matrix.python-impl == 'jython' }}
-      # Working around deprecation of support for non-SNI clients at PyPI CDN (see https://status.python.org/incidents/hzmjhqsdjqgb)
-      run: |
-        wget https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl
-        pip install nose-1.3.7-py2-none-any.whl
     - name: Run tests
-      continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
+      continue-on-error: False
       env:
-        YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
+        YTDL_TEST_SET: core
       run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}
   # Linter is in quick-test

.github/workflows/download.yml

@@ -9,52 +9,22 @@ jobs:
       fail-fast: true
       matrix:
         os: [ubuntu-18.04]
-        # TODO: python 2.6
-        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
-        python-impl: [cpython]
-        ytdl-test-set: [download]
+        python-version: [3.6, 3.7, 3.8, 3.9, pypy-3.6, pypy-3.7]
         run-tests-ext: [sh]
         include:
-        # python 3.2 is only available on windows via setup-python
         - os: windows-latest
-          python-version: 3.2
-          python-impl: cpython
-          ytdl-test-set: download
+          python-version: 3.4  # Windows x86 build is still in 3.4
           run-tests-ext: bat
-        # jython - disable for now since it takes too long to complete
-        # - os: ubuntu-latest
-        #   python-impl: jython
-        #   ytdl-test-set: download
-        #   run-tests-ext: sh
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}
       uses: actions/setup-python@v2
-      if: ${{ matrix.python-impl == 'cpython' }}
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Set up Java 8
-      if: ${{ matrix.python-impl == 'jython' }}
-      uses: actions/setup-java@v1
-      with:
-        java-version: 8
-    - name: Install Jython
-      if: ${{ matrix.python-impl == 'jython' }}
-      run: |
-        wget https://repo1.maven.org/maven2/org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
-        java -jar jython-installer.jar -s -d "$HOME/jython"
-        echo "$HOME/jython/bin" >> $GITHUB_PATH
     - name: Install nose
-      if: ${{ matrix.python-impl != 'jython' }}
       run: pip install nose
-    - name: Install nose (Jython)
-      if: ${{ matrix.python-impl == 'jython' }}
-      # Working around deprecation of support for non-SNI clients at PyPI CDN (see https://status.python.org/incidents/hzmjhqsdjqgb)
-      run: |
-        wget https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl
-        pip install nose-1.3.7-py2-none-any.whl
     - name: Run tests
-      continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
+      continue-on-error: true
       env:
-        YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
+        YTDL_TEST_SET: download
       run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}

CONTRIBUTORS

@@ -1,6 +1,7 @@
 pukkandan (owner)
 shirt-dev (collaborator)
 colethedj (collaborator)
+Ashish0804 (collaborator)
 h-h-h-h
 pauldubois98
 nixxo
@@ -20,11 +21,9 @@ FelixFrog
 Zocker1999NET
 nao20010128nao
 kurumigi
-tsukumi
 bbepis
 animelover1984
 Pccode66
-Ashish0804
 RobinD42
 hseg
 DennyDai
@@ -48,3 +47,10 @@ craftingmod
 tpikonen
 tripulse
 king-millez
+alex-gedeon
+hhirtz
+louie-github
+MinePlayersPE
+olifre
+rhsmachine
+nihil-admirari

Changelog.md

@@ -19,13 +19,39 @@
 -->
 
 
+### 2021.06.08
+
+* Remove support for obsolete Python versions: Only 3.6+ is now supported
+* Merge youtube-dl: Upto [commit/c2350ca](https://github.com/ytdl-org/youtube-dl/commit/c2350cac243ba1ec1586fe85b0d62d1b700047a2)
+* [hls] Fix decryption for multithreaded downloader
+* [extractor] Fix pre-checking archive for some extractors
+* [extractor] Fix FourCC fallback when parsing ISM [fstirlitz](https://github.com/fstirlitz)
+* [twitcasting] Add TwitCastingUserIE, TwitCastingLiveIE [pukkandan](https://github.com/pukkandan), [nao20010128nao](https://github.com/nao20010128nao)
+* [vidio] Add VidioPremierIE and VidioLiveIE [minEplaYerspe](Https://github.com/MinePlayersPE)
+* [viki] Fix extraction from [ytdl-org/youtube-dl@59e583f](https://github.com/ytdl-org/youtube-dl/commit/59e583f7e8530ca92776c866897d895c072e2a82)
+* [youtube] Support shorts URL
+* [zoom] Extract transcripts as subtitles
+* Add field `original_url` with the user-inputted URL
+* Fix and refactor `prepare_outtmpl`
+* Make more fields available for `--print` when used with `--flat-playlist`
+* [utils] Generalize `traverse_dict` to `traverse_obj`
+* [downloader/ffmpeg] Hide FFmpeg banner unless in verbose mode [fstirlitz](https://github.com/fstirlitz)
+* [build] Release `yt-dlp.tar.gz`
+* [build,update] Add GNU-style SHA512 and prepare updater for simlar SHA256 [nihil-admirari](https://github.com/nihil-admirari)
+* [pyinst] Show Python version in exe metadata [nihil-admirari](https://github.com/nihil-admirari)
+* [docs] Improve documentation of dependencies
+* [cleanup] Mark unused files
+* [cleanup] Point all shebang to `python3` [fstirlitz](https://github.com/fstirlitz)
+* [cleanup] Remove duplicate file `trovolive.py`
+
+
 ### 2021.06.01
 
 * Merge youtube-dl: Upto [commit/d495292](https://github.com/ytdl-org/youtube-dl/commit/d495292852b6c2f1bd58bc2141ff2b0265c952cf)
 * Pre-check archive and filters during playlist extraction
 * Handle Basic Auth `user:pass` in URLs by [hhirtz](https://github.com/hhirtz) and [pukkandan](https://github.com/pukkandan)
 * [archiveorg] Add YoutubeWebArchiveIE by [colethedj](https://github.com/colethedj) and [alex-gedeon](https://github.com/alex-gedeon)
-* [fancode] Add extractor by [rmsmachine](https://github.com/rmsmachine)
+* [fancode] Add extractor by [rhsmachine](https://github.com/rhsmachine)
 * [patreon] Support vimeo embeds by [rhsmachine](https://github.com/rhsmachine)
 * [Saitosan] Add new extractor by [llacb47](https://github.com/llacb47)
 * [ShemarooMe] Add extractor by [Ashish0804](https://github.com/Ashish0804) and [pukkandan](https://github.com/pukkandan)
@@ -400,7 +426,7 @@
 
 ### 2021.02.15
 * Merge youtube-dl: Upto [2021.02.10](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.02.10) (except archive.org)
-* [niconico] Improved extraction and support encrypted/SMILE movies by [kurumigi](https://github.com/kurumigi), [tsukumi](https://github.com/tsukumi), [bbepis](https://github.com/bbepis), [pukkandan](https://github.com/pukkandan)
+* [niconico] Improved extraction and support encrypted/SMILE movies by [kurumigi](https://github.com/kurumigi), [tsukumijima](https://github.com/tsukumijima), [bbepis](https://github.com/bbepis), [pukkandan](https://github.com/pukkandan)
 * Fix HLS AES-128 with multiple keys in external downloaders by [shirt](https://github.com/shirt-dev)
 * [youtube_live_chat] Fix by using POST API by [siikamiika](https://github.com/siikamiika)
 * [rumble] Add support for video page

Makefile

@@ -25,6 +25,7 @@ completion-zsh: completions/zsh/_yt-dlp
 lazy-extractors: yt_dlp/extractor/lazy_extractors.py
 
 PREFIX ?= /usr/local
+DESTDIR ?= .
 BINDIR ?= $(PREFIX)/bin
 MANDIR ?= $(PREFIX)/man
 SHAREDIR ?= $(PREFIX)/share

README.md

@@ -22,8 +22,8 @@ yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on t
 * [NEW FEATURES](#new-features)
     * [Differences in default behavior](#differences-in-default-behavior)
 * [INSTALLATION](#installation)
-    * [Dependencies](#dependencies)
     * [Update](#update)
+    * [Dependencies](#dependencies)
     * [Compile](#compile)
 * [USAGE AND OPTIONS](#usage-and-options)
     * [General Options](#general-options)
@@ -66,7 +66,7 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
 
-* **Merged with youtube-dl [commit/d495292](https://github.com/ytdl-org/youtube-dl/commit/d495292852b6c2f1bd58bc2141ff2b0265c952cf)**: (v2021.05.16) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl [commit/c2350ca](https://github.com/ytdl-org/youtube-dl/commit/c2350cac243ba1ec1586fe85b0d62d1b700047a2)**: (v2021.06.06) You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
 
 * **Merged with animelover1984/youtube-dl**: You get most of the features and improvements from [animelover1984/youtube-dl](https://github.com/animelover1984/youtube-dl) including `--write-comments`, `BiliBiliSearch`, `BilibiliChannel`, Embedding thumbnail in mp4/ogg/opus, playlist infojson etc. Note that the NicoNico improvements are not available. See [#31](https://github.com/yt-dlp/yt-dlp/pull/31) for details.
 
@@ -84,7 +84,7 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **Aria2c with HLS/DASH**: You can use `aria2c` as the external downloader for DASH(mpd) and HLS(m3u8) formats
 
-* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries
+* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom, audius, zee5, mtv.it, wimtv, pluto.tv, niconico users, discoveryplus.in, mediathek, NFHSNetwork, nebula, ukcolumn, whowatch, MxplayerShow, parlview (au), YoutubeWebArchive, fancode, Saitosan, ShemarooMe, telemundo, VootSeries, SonyLIVSeries, HotstarSeries, VidioPremier, VidioLive
 
 * **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, akamai, ina, rumble, tennistv, amcnetworks, la7 podcasts, linuxacadamy, nitter, twitcasting, viu, crackle, curiositystream, mediasite, rmcdecouverte, sonyliv, tubi, tenplay, patreon
 
@@ -166,17 +166,31 @@ sudo aria2c https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -o
 sudo chmod a+rx /usr/local/bin/yt-dlp
 ```
 
-### DEPENDENCIES
-Python versions 3.6+ (CPython and PyPy) are officially supported. Other versions and implementations may or maynot work correctly.
-
-On windows, [Microsoft Visual C++ 2010 Redistributable Package (x86)](https://www.microsoft.com/en-us/download/details.aspx?id=26999) is also necessary to run yt-dlp. You probably already have this, but if the executable throws an error due to missing `MSVCR100.dll` you need to install it.
-
-Although there are no other required dependencies, `ffmpeg` and `ffprobe` are highly recommended. Other optional dependencies are `sponskrub`, `AtomicParsley`, `mutagen`, `pycryptodome`, `phantomjs` and any of the supported external downloaders. Note that the windows releases are already built with the python interpreter, mutagen and pycryptodome included.
-
 ### UPDATE
 You can use `yt-dlp -U` to update if you are using the provided release.
 If you are using `pip`, simply re-run the same command that was used to install the program.
 
+### DEPENDENCIES
+Python versions 3.6+ (CPython and PyPy) are supported. Other versions and implementations may or may not work correctly.
+
+<!-- https://www.microsoft.com/en-us/download/details.aspx?id=26999 -->
+On windows, [Microsoft Visual C++ 2010 SP1 Redistributable Package (x86)](https://download.microsoft.com/download/1/6/5/165255E7-1014-4D0A-B094-B6A430A6BFFC/vcredist_x86.exe) is also necessary to run yt-dlp. You probably already have this, but if the executable throws an error due to missing `MSVCR100.dll` you need to install it manually.
+
+While all the other dependancies are optional, `ffmpeg` and `ffprobe` are highly recommended
+* [**ffmpeg** and **ffprobe**](https://www.ffmpeg.org) - Required for [merging seperate video and audio files](#format-selection) as well as for various [post-processing](#post-processing-options) tasks. Licence [depends on the build](https://www.ffmpeg.org/legal.html)
+* [**sponskrub**](https://github.com/faissaloo/SponSkrub) - For using the [sponskrub options](#sponskrub-sponsorblock-options). Licenced under [GPLv3+](https://github.com/faissaloo/SponSkrub/blob/master/LICENCE.md)
+* [**mutagen**](https://github.com/quodlibet/mutagen) - For embedding thumbnail in certain formats. Licenced under [GPLv2+](https://github.com/quodlibet/mutagen/blob/master/COPYING)
+* [**pycryptodome**](https://github.com/Legrandin/pycryptodome) - For decrypting various data. Licenced under [BSD2](https://github.com/Legrandin/pycryptodome/blob/master/LICENSE.rst)
+* [**AtomicParsley**](https://github.com/wez/atomicparsley) - For embedding thumbnail in mp4/m4a if mutagen is not present. Licenced under [GPLv2+](https://github.com/wez/atomicparsley/blob/master/COPYING)
+* [**rtmpdump**](http://rtmpdump.mplayerhq.hu) - For downloading `rtmp` streams. ffmpeg will be used as a fallback. Licenced under [GPLv2+](http://rtmpdump.mplayerhq.hu)
+* [**mplayer**](http://mplayerhq.hu/design7/info.html) or [**mpv**](https://mpv.io) - For downloading `rstp` streams. ffmpeg will be used as a fallback. Licenced under [GPLv2+](https://github.com/mpv-player/mpv/blob/master/Copyright)
+* [**phantomjs**](https://github.com/ariya/phantomjs) - Used in extractors where javascript needs to be run. Licenced under [BSD3](https://github.com/ariya/phantomjs/blob/master/LICENSE.BSD)
+* Any external downloader that you want to use with `--downloader`
+
+To use or redistribute the dependencies, you must agree to their respective licensing terms.
+
+Note that the windows releases are already built with the python interpreter, mutagen and pycryptodome included.
+
 ### COMPILE
 
 **For Windows**:
@@ -500,13 +514,10 @@ Then simply run `make`. You can also run `make yt-dlp` instead to compile only t
                                      jar in
     --no-cookies                     Do not read/dump cookies (default)
     --cache-dir DIR                  Location in the filesystem where youtube-dl
-                                     can store some downloaded information
-                                     permanently. By default
-                                     $XDG_CACHE_HOME/youtube-dl or
-                                     ~/.cache/youtube-dl . At the moment, only
-                                     YouTube player files (for videos with
-                                     obfuscated signatures) are cached, but that
-                                     may change
+                                     can store some downloaded information (such
+                                     as client ids and signatures) permanently.
+                                     By default $XDG_CACHE_HOME/youtube-dl or
+                                     ~/.cache/youtube-dl
     --no-cache-dir                   Disable filesystem caching
     --rm-cache-dir                   Delete all filesystem cache files
 
@@ -954,6 +965,8 @@ The available fields are:
  - `playlist_title` (string): Playlist title
  - `playlist_uploader` (string): Full name of the playlist uploader
  - `playlist_uploader_id` (string): Nickname or id of the playlist uploader
+ - `webpage_url` (string): A URL to the video webpage which if given to yt-dlp should allow to get the same result again
+ - `original_url` (string): The URL given by the user (or same as `webpage_url` for playlist entries)
 
 Available for the video that belongs to some logical chapter or section:
 

devscripts/bash-completion.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import os

devscripts/buildserver.py

@@ -1,3 +1,5 @@
+# UNUSED
+
 #!/usr/bin/python3
 
 import argparse

devscripts/check-porn.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 """

devscripts/create-github-release.py

@@ -1,6 +1,6 @@
 # Unused
 
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import io

devscripts/fish-completion.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import optparse

devscripts/generate_aes_testdata.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import codecs

devscripts/gh-pages.unused/update-copyright.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import with_statement, unicode_literals

devscripts/lazy_load_template.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

devscripts/make_contributing.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # import io

devscripts/make_issue_template.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import io

devscripts/make_lazy_extractors.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 from __future__ import unicode_literals, print_function
 
 from inspect import getsource

devscripts/make_readme.py

@@ -1,3 +1,8 @@
+#!/usr/bin/env python3
+
+# yt-dlp --help | make_readme.py
+# This must be run in a console of correct width
+
 from __future__ import unicode_literals
 
 import io

devscripts/make_supportedsites.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import io

devscripts/prepare_manpage.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import io

devscripts/release.sh

@@ -1,4 +1,5 @@
 # Unused
+
 #!/bin/bash
 
 # IMPORTANT: the following assumptions are made

devscripts/show-downloads-statistics.py

@@ -1,6 +1,6 @@
 # Unused
 
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import itertools

devscripts/update-version.py

@@ -1,4 +1,6 @@
+#!/usr/bin/env python3
 from __future__ import unicode_literals
+
 from datetime import datetime
 # import urllib.request
 

devscripts/wine-py2exe.sh

@@ -1,3 +1,5 @@
+# UNUSED
+
 #!/bin/bash
 
 # Run with as parameter a setup.py that works in the current directory

devscripts/zsh-completion.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import os

pyinst.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals
@@ -58,7 +58,9 @@ VERSION_FILE = VSVersionInfo(
                     ),
                     StringStruct('OriginalFilename', 'yt-dlp%s.exe' % _x86),
                     StringStruct('ProductName', 'yt-dlp%s' % _x86),
-                    StringStruct('ProductVersion', '%s%s' % (VERSION, _x86)),
+                    StringStruct(
+                        'ProductVersion',
+                        '%s%s on Python %s' % (VERSION, _x86, platform.python_version())),
                 ])]),
         VarFileInfo([VarStruct('Translation', [0, 1200])])
     ]

setup.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from setuptools import setup, Command, find_packages
@@ -88,26 +88,16 @@ setup(
         'Development Status :: 5 - Production/Stable',
         'Environment :: Console',
         'Programming Language :: Python',
-        'Programming Language :: Python :: 2',
-        'Programming Language :: Python :: 2.6',
-        'Programming Language :: Python :: 2.7',
-        'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.2',
-        'Programming Language :: Python :: 3.3',
-        'Programming Language :: Python :: 3.4',
-        'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
         'Programming Language :: Python :: Implementation',
         'Programming Language :: Python :: Implementation :: CPython',
-        'Programming Language :: Python :: Implementation :: IronPython',
-        'Programming Language :: Python :: Implementation :: Jython',
         'Programming Language :: Python :: Implementation :: PyPy',
         'License :: Public Domain',
         'Operating System :: OS Independent',
     ],
-    python_requires='>=2.6',
+    python_requires='>=3.6',
 
     cmdclass={'build_lazy_extractors': build_lazy_extractors},
     **params

supportedsites.md

@@ -1069,6 +1069,8 @@
  - **TVPlayHome**
  - **Tweakers**
  - **TwitCasting**
+ - **TwitCastingLive**
+ - **TwitCastingUser**
  - **twitch:clips**
  - **twitch:stream**
  - **twitch:vod**
@@ -1130,6 +1132,8 @@
  - **videomore:video**
  - **VideoPress**
  - **Vidio**
+ - **VidioLive**
+ - **VidioPremier**
  - **VidLii**
  - **vidme**
  - **vidme:user**

test/test_InfoExtractor.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_YoutubeDL.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals
@@ -17,7 +17,7 @@ from yt_dlp.compat import compat_str, compat_urllib_error
 from yt_dlp.extractor import YoutubeIE
 from yt_dlp.extractor.common import InfoExtractor
 from yt_dlp.postprocessor.common import PostProcessor
-from yt_dlp.utils import ExtractorError, match_filter_func
+from yt_dlp.utils import ExtractorError, float_or_none, match_filter_func
 
 TEST_URL = 'http://localhost/sample.mp4'
 
@@ -648,56 +648,108 @@ class TestYoutubeDL(unittest.TestCase):
         self.assertEqual(test_dict['extractor'], 'Foo')
         self.assertEqual(test_dict['playlist'], 'funny videos')
 
-    def test_prepare_filename(self):
-        info = {
-            'id': '1234',
-            'ext': 'mp4',
-            'width': None,
-            'height': 1080,
-            'title1': '$PATH',
-            'title2': '%PATH%',
-            'timestamp': 1618488000,
-            'formats': [{'id': 'id1'}, {'id': 'id2'}]
-        }
+    outtmpl_info = {
+        'id': '1234',
+        'ext': 'mp4',
+        'width': None,
+        'height': 1080,
+        'title1': '$PATH',
+        'title2': '%PATH%',
+        'title3': 'foo/bar\\test',
+        'timestamp': 1618488000,
+        'duration': 100000,
+        'playlist_index': 1,
+        '_last_playlist_index': 100,
+        'n_entries': 10,
+        'formats': [{'id': 'id1'}, {'id': 'id2'}, {'id': 'id3'}]
+    }
 
-        def fname(templ, na_placeholder='NA'):
-            params = {'outtmpl': templ}
-            if na_placeholder != 'NA':
-                params['outtmpl_na_placeholder'] = na_placeholder
+    def test_prepare_outtmpl(self):
+        def out(tmpl, **params):
+            params['outtmpl'] = tmpl
             ydl = YoutubeDL(params)
-            return ydl.prepare_filename(info)
-        self.assertEqual(fname('%(id)s.%(ext)s'), '1234.mp4')
-        self.assertEqual(fname('%(id)s-%(width)s.%(ext)s'), '1234-NA.mp4')
-        NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(id)s.%(ext)s'
-        # Replace missing fields with 'NA' by default
-        self.assertEqual(fname(NA_TEST_OUTTMPL), 'NA-NA-1234.mp4')
-        # Or by provided placeholder
-        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder='none'), 'none-none-1234.mp4')
-        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder=''), '--1234.mp4')
-        self.assertEqual(fname('%(height)s.%(ext)s'), '1080.mp4')
-        self.assertEqual(fname('%(height)d.%(ext)s'), '1080.mp4')
-        self.assertEqual(fname('%(height)6d.%(ext)s'), '  1080.mp4')
-        self.assertEqual(fname('%(height)-6d.%(ext)s'), '1080  .mp4')
-        self.assertEqual(fname('%(height)06d.%(ext)s'), '001080.mp4')
-        self.assertEqual(fname('%(height) 06d.%(ext)s'), ' 01080.mp4')
-        self.assertEqual(fname('%(height)   06d.%(ext)s'), ' 01080.mp4')
-        self.assertEqual(fname('%(height)0 6d.%(ext)s'), ' 01080.mp4')
-        self.assertEqual(fname('%(height)0   6d.%(ext)s'), ' 01080.mp4')
-        self.assertEqual(fname('%(height)   0   6d.%(ext)s'), ' 01080.mp4')
+            ydl._num_downloads = 1
+            err = ydl.validate_outtmpl(tmpl)
+            if err:
+                raise err
+            outtmpl, tmpl_dict = ydl.prepare_outtmpl(tmpl, self.outtmpl_info)
+            return outtmpl % tmpl_dict
+
+        self.assertEqual(out('%(id)s.%(ext)s'), '1234.mp4')
+        self.assertEqual(out('%(duration_string)s'), '27:46:40')
+        self.assertTrue(float_or_none(out('%(epoch)d')))
+        self.assertEqual(out('%(resolution)s'), '1080p')
+        self.assertEqual(out('%(playlist_index)s'), '001')
+        self.assertEqual(out('%(autonumber)s'), '00001')
+        self.assertEqual(out('%(autonumber+2)03d', autonumber_start=3), '005')
+        self.assertEqual(out('%(autonumber)s', autonumber_size=3), '001')
+
+        self.assertEqual(out('%%'), '%')
+        self.assertEqual(out('%%%%'), '%%')
+        self.assertEqual(out('%(invalid@tmpl|def)s', outtmpl_na_placeholder='none'), 'none')
+        self.assertEqual(out('%()s'), 'NA')
+        self.assertEqual(out('%s'), '%s')
+        self.assertEqual(out('%d'), '%d')
+        self.assertRaises(ValueError, out, '%')
+        self.assertRaises(ValueError, out, '%(title)')
+
+        NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(x|def)s-%(id)s.%(ext)s'
+        self.assertEqual(out(NA_TEST_OUTTMPL), 'NA-NA-def-1234.mp4')
+        self.assertEqual(out(NA_TEST_OUTTMPL, outtmpl_na_placeholder='none'), 'none-none-def-1234.mp4')
+        self.assertEqual(out(NA_TEST_OUTTMPL, outtmpl_na_placeholder=''), '--def-1234.mp4')
+
+        FMT_TEST_OUTTMPL = '%%(height)%s.%%(ext)s'
+        self.assertEqual(out(FMT_TEST_OUTTMPL % 's'), '1080.mp4')
+        self.assertEqual(out(FMT_TEST_OUTTMPL % 'd'), '1080.mp4')
+        self.assertEqual(out(FMT_TEST_OUTTMPL % '6d'), '  1080.mp4')
+        self.assertEqual(out(FMT_TEST_OUTTMPL % '-6d'), '1080  .mp4')
+        self.assertEqual(out(FMT_TEST_OUTTMPL % '06d'), '001080.mp4')
+        self.assertEqual(out(FMT_TEST_OUTTMPL % ' 06d'), ' 01080.mp4')
+        self.assertEqual(out(FMT_TEST_OUTTMPL % '   06d'), ' 01080.mp4')
+        self.assertEqual(out(FMT_TEST_OUTTMPL % '0 6d'), ' 01080.mp4')
+        self.assertEqual(out(FMT_TEST_OUTTMPL % '0   6d'), ' 01080.mp4')
+        self.assertEqual(out(FMT_TEST_OUTTMPL % '   0   6d'), ' 01080.mp4')
+
+        self.assertEqual(out('%(id)d'), '1234')
+        self.assertEqual(out('%(height)c'), '1')
+        self.assertEqual(out('%(ext)c'), 'm')
+        self.assertEqual(out('%(id)d %(id)r'), "1234 '1234'")
+        self.assertEqual(out('%(ext)s-%(ext|def)d'), 'mp4-def')
+        self.assertEqual(out('%(width|0)04d'), '0000')
+        self.assertEqual(out('%(width|)d', outtmpl_na_placeholder='none'), '')
+
+        FORMATS = self.outtmpl_info['formats']
+        self.assertEqual(out('%(timestamp+-1000>%H-%M-%S)s'), '11-43-20')
+        self.assertEqual(out('%(id+1-height+3)05d'), '00158')
+        self.assertEqual(out('%(width+100)05d'), 'NA')
+        self.assertEqual(out('%(formats.0)s'), str(FORMATS[0]))
+        self.assertEqual(out('%(height.0)03d'), '001')
+        self.assertEqual(out('%(formats.-1.id)s'), str(FORMATS[-1]['id']))
+        self.assertEqual(out('%(formats.3)s'), 'NA')
+        self.assertEqual(out('%(formats.:2:-1)r'), repr(FORMATS[:2:-1]))
+        self.assertEqual(out('%(formats.0.id.-1+id)f'), '1235.000000')
+
+    def test_prepare_filename(self):
+        def fname(templ):
+            params = {'outtmpl': templ}
+            ydl = YoutubeDL(params)
+            return ydl.prepare_filename(self.outtmpl_info)
+
         self.assertEqual(fname('%%'), '%')
         self.assertEqual(fname('%%%%'), '%%')
-        self.assertEqual(fname('%%(height)06d.%(ext)s'), '%(height)06d.mp4')
+        self.assertEqual(fname('%%(width)06d.%(ext)s'), '%(width)06d.mp4')
         self.assertEqual(fname('%(width)06d.%(ext)s'), 'NA.mp4')
         self.assertEqual(fname('%(width)06d.%%(ext)s'), 'NA.%(ext)s')
         self.assertEqual(fname('%%(width)06d.%(ext)s'), '%(width)06d.mp4')
+
         self.assertEqual(fname('Hello %(title1)s'), 'Hello $PATH')
         self.assertEqual(fname('Hello %(title2)s'), 'Hello %PATH%')
-        self.assertEqual(fname('%(timestamp+-1000>%H-%M-%S)s'), '11-43-20')
-        self.assertEqual(fname('%(id+1)05d'), '01235')
-        self.assertEqual(fname('%(width+100)05d'), 'NA')
-        self.assertEqual(fname('%(formats.0)s').replace("u", ""), "{'id' - 'id1'}")
-        self.assertEqual(fname('%(formats.-1.id)s'), 'id2')
-        self.assertEqual(fname('%(formats.2)s'), 'NA')
+
+        self.assertEqual(fname('%(title3)s'), 'foo_bar_test')
+        self.assertEqual(fname('%(formats.0)s'), "{'id' - 'id1'}")
+
+        self.assertEqual(fname('%(id)r %(height)r'), "'1234' 1080")
+        self.assertEqual(fname('%(formats.0)r'), "{'id' - 'id1'}")
 
     def test_format_note(self):
         ydl = YoutubeDL()

test/test_YoutubeDLCookieJar.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_aes.py

@@ -1,5 +1,4 @@
-#!/usr/bin/env python
-
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_age_restriction.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_all_urls.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_cache.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_compat.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_download.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_downloader_http.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

test/test_execution.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_http.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

test/test_iqiyi_sdk_interpreter.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_jsinterp.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_overwrites.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 import os

test/test_post_hooks.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_postprocessors.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 
@@ -8,7 +8,10 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
+from yt_dlp import YoutubeDL
+from yt_dlp.compat import compat_shlex_quote
 from yt_dlp.postprocessor import (
+    ExecAfterDownloadPP,
     FFmpegThumbnailsConvertorPP,
     MetadataFromFieldPP,
     MetadataFromTitlePP,
@@ -55,3 +58,14 @@ class TestConvertThumbnail(unittest.TestCase):
 
         for _, out in tests:
             os.remove(file.format(out))
+
+
+class TestExecAfterDownload(unittest.TestCase):
+    def test_parse_cmd(self):
+        pp = ExecAfterDownloadPP(YoutubeDL(), '')
+        info = {'filepath': 'file name'}
+        quoted_filepath = compat_shlex_quote(info['filepath'])
+
+        self.assertEqual(pp.parse_cmd('echo', info), 'echo %s' % quoted_filepath)
+        self.assertEqual(pp.parse_cmd('echo.{}', info), 'echo.%s' % quoted_filepath)
+        self.assertEqual(pp.parse_cmd('echo "%(filepath)s"', info), 'echo "%s"' % info['filepath'])

test/test_socks.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

test/test_subtitles.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_swfinterp.py.disabled

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_update.py.disabled

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

test/test_utils.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_verbose_output.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals

test/test_write_annotations.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 from __future__ import unicode_literals
 

test/test_youtube_lists.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_youtube_misc.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Allow direct execution

test/test_youtube_signature.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 from __future__ import unicode_literals
 

yt_dlp/YoutubeDL.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import absolute_import, unicode_literals
@@ -64,7 +64,7 @@ from .utils import (
     float_or_none,
     format_bytes,
     format_field,
-    FORMAT_RE,
+    STR_FORMAT_RE,
     formatSeconds,
     GeoRestrictedError,
     int_or_none,
@@ -101,7 +101,7 @@ from .utils import (
     strftime_or_none,
     subtitles_filename,
     to_high_limit_path,
-    traverse_dict,
+    traverse_obj,
     UnavailableVideoError,
     url_basename,
     version_tuple,
@@ -813,54 +813,41 @@ class YoutubeDL(object):
                     'Put  from __future__ import unicode_literals  at the top of your code file or consider switching to Python 3.x.')
         return outtmpl_dict
 
+    @staticmethod
+    def validate_outtmpl(tmpl):
+        ''' @return None or Exception object '''
+        try:
+            re.sub(
+                STR_FORMAT_RE.format(''),
+                lambda mobj: ('%' if not mobj.group('has_key') else '') + mobj.group(0),
+                tmpl
+            ) % collections.defaultdict(int)
+            return None
+        except ValueError as err:
+            return err
+
     def prepare_outtmpl(self, outtmpl, info_dict, sanitize=None):
         """ Make the template and info_dict suitable for substitution (outtmpl % info_dict)"""
-        template_dict = dict(info_dict)
+        info_dict = dict(info_dict)
         na = self.params.get('outtmpl_na_placeholder', 'NA')
 
-        # duration_string
-        template_dict['duration_string'] = (  # %(duration>%H-%M-%S)s is wrong if duration > 24hrs
+        info_dict['duration_string'] = (  # %(duration>%H-%M-%S)s is wrong if duration > 24hrs
             formatSeconds(info_dict['duration'], '-' if sanitize else ':')
             if info_dict.get('duration', None) is not None
             else None)
-
-        # epoch
-        template_dict['epoch'] = int(time.time())
-
-        # autonumber
-        autonumber_size = self.params.get('autonumber_size')
-        if autonumber_size is None:
-            autonumber_size = 5
-        template_dict['autonumber'] = self.params.get('autonumber_start', 1) - 1 + self._num_downloads
-
-        # resolution if not defined
-        if template_dict.get('resolution') is None:
-            if template_dict.get('width') and template_dict.get('height'):
-                template_dict['resolution'] = '%dx%d' % (template_dict['width'], template_dict['height'])
-            elif template_dict.get('height'):
-                template_dict['resolution'] = '%sp' % template_dict['height']
-            elif template_dict.get('width'):
-                template_dict['resolution'] = '%dx?' % template_dict['width']
+        info_dict['epoch'] = int(time.time())
+        info_dict['autonumber'] = self.params.get('autonumber_start', 1) - 1 + self._num_downloads
+        if info_dict.get('resolution') is None:
+            info_dict['resolution'] = self.format_resolution(info_dict, default=None)
 
         # For fields playlist_index and autonumber convert all occurrences
         # of %(field)s to %(field)0Nd for backward compatibility
         field_size_compat_map = {
-            'playlist_index': len(str(template_dict.get('_last_playlist_index') or '')),
-            'autonumber': autonumber_size,
+            'playlist_index': len(str(info_dict.get('_last_playlist_index') or '')),
+            'autonumber': self.params.get('autonumber_size') or 5,
         }
-        FIELD_SIZE_COMPAT_RE = r'(?<!%)%\((?P<field>autonumber|playlist_index)\)s'
-        mobj = re.search(FIELD_SIZE_COMPAT_RE, outtmpl)
-        if mobj:
-            outtmpl = re.sub(
-                FIELD_SIZE_COMPAT_RE,
-                r'%%(\1)0%dd' % field_size_compat_map[mobj.group('field')],
-                outtmpl)
 
-        numeric_fields = list(self._NUMERIC_FIELDS)
-        if sanitize is None:
-            sanitize = lambda k, v: v
-
-        EXTERNAL_FORMAT_RE = FORMAT_RE.format('(?P<key>[^)]*)')
+        EXTERNAL_FORMAT_RE = STR_FORMAT_RE.format('[^)]*')
         # Field is of the form key1.key2...
         # where keys (except first) can be string, int or slice
         FIELD_RE = r'\w+(?:\.(?:\w+|[-\d]*(?::[-\d]*){0,2}))*'
@@ -876,71 +863,84 @@ class YoutubeDL(object):
             '+': float.__add__,
             '-': float.__sub__,
         }
-        for outer_mobj in re.finditer(EXTERNAL_FORMAT_RE, outtmpl):
-            final_key = outer_mobj.group('key')
-            str_type = outer_mobj.group('type')
-            value = None
-            mobj = re.match(INTERNAL_FORMAT_RE, final_key)
-            if mobj is not None:
-                mobj = mobj.groupdict()
-                # Object traversal
-                fields = mobj['fields'].split('.')
-                value = traverse_dict(template_dict, fields)
-                # Negative
-                if mobj['negate']:
-                    value = float_or_none(value)
-                    if value is not None:
-                        value *= -1
-                # Do maths
-                if mobj['maths']:
-                    value = float_or_none(value)
-                    operator = None
-                    for item in MATH_OPERATORS_RE.split(mobj['maths'])[1:]:
-                        if item == '':
-                            value = None
-                        if value is None:
-                            break
-                        if operator:
-                            item, multiplier = (item[1:], -1) if item[0] == '-' else (item, 1)
-                            offset = float_or_none(item)
-                            if offset is None:
-                                offset = float_or_none(traverse_dict(template_dict, item.split('.')))
-                            try:
-                                value = operator(value, multiplier * offset)
-                            except (TypeError, ZeroDivisionError):
-                                value = None
-                            operator = None
-                        else:
-                            operator = MATH_FUNCTIONS[item]
-                # Datetime formatting
-                if mobj['strf_format']:
-                    value = strftime_or_none(value, mobj['strf_format'])
-                # Set default
-                if value is None and mobj['default'] is not None:
-                    value = mobj['default']
-            # Sanitize
-            if str_type in 'crs' and value is not None:  # string
-                value = sanitize('%{}'.format(str_type) % fields[-1], value)
-            else:  # numeric
-                numeric_fields.append(final_key)
+        tmpl_dict = {}
+
+        get_key = lambda k: traverse_obj(
+            info_dict, k.split('.'), is_user_input=True, traverse_string=True)
+
+        def get_value(mdict):
+            # Object traversal
+            value = get_key(mdict['fields'])
+            # Negative
+            if mdict['negate']:
                 value = float_or_none(value)
-            if value is not None:
-                template_dict[final_key] = value
+                if value is not None:
+                    value *= -1
+            # Do maths
+            if mdict['maths']:
+                value = float_or_none(value)
+                operator = None
+                for item in MATH_OPERATORS_RE.split(mdict['maths'])[1:]:
+                    if item == '' or value is None:
+                        return None
+                    if operator:
+                        item, multiplier = (item[1:], -1) if item[0] == '-' else (item, 1)
+                        offset = float_or_none(item)
+                        if offset is None:
+                            offset = float_or_none(get_key(item))
+                        try:
+                            value = operator(value, multiplier * offset)
+                        except (TypeError, ZeroDivisionError):
+                            return None
+                        operator = None
+                    else:
+                        operator = MATH_FUNCTIONS[item]
+            # Datetime formatting
+            if mdict['strf_format']:
+                value = strftime_or_none(value, mdict['strf_format'])
 
-        # Missing numeric fields used together with integer presentation types
-        # in format specification will break the argument substitution since
-        # string NA placeholder is returned for missing fields. We will patch
-        # output template for missing fields to meet string presentation type.
-        for numeric_field in numeric_fields:
-            if template_dict.get(numeric_field) is None:
-                outtmpl = re.sub(
-                    FORMAT_RE.format(re.escape(numeric_field)),
-                    r'%({0})s'.format(numeric_field), outtmpl)
+            return value
 
-        template_dict = collections.defaultdict(lambda: na, (
-            (k, v if isinstance(v, compat_numeric_types) else sanitize(k, v))
-            for k, v in template_dict.items() if v is not None))
-        return outtmpl, template_dict
+        def create_key(outer_mobj):
+            if not outer_mobj.group('has_key'):
+                return '%{}'.format(outer_mobj.group(0))
+
+            key = outer_mobj.group('key')
+            fmt = outer_mobj.group('format')
+            mobj = re.match(INTERNAL_FORMAT_RE, key)
+            if mobj is None:
+                value, default = None, na
+            else:
+                mobj = mobj.groupdict()
+                default = mobj['default'] if mobj['default'] is not None else na
+                value = get_value(mobj)
+
+            if fmt == 's' and value is not None and key in field_size_compat_map.keys():
+                fmt = '0{:d}d'.format(field_size_compat_map[key])
+
+            value = default if value is None else value
+            key += '\0%s' % fmt
+
+            if fmt == 'c':
+                value = compat_str(value)
+                if value is None:
+                    value, fmt = default, 's'
+                else:
+                    value = value[0]
+            elif fmt[-1] not in 'rs':  # numeric
+                value = float_or_none(value)
+                if value is None:
+                    value, fmt = default, 's'
+            if sanitize:
+                if fmt[-1] == 'r':
+                    # If value is an object, sanitize might convert it to a string
+                    # So we convert it to repr first
+                    value, fmt = repr(value), '%ss' % fmt[:-1]
+                value = sanitize(key, value)
+            tmpl_dict[key] = value
+            return '%({key}){fmt}'.format(key=key, fmt=fmt)
+
+        return re.sub(EXTERNAL_FORMAT_RE, create_key, outtmpl), tmpl_dict
 
     def _prepare_filename(self, info_dict, tmpl_type='default'):
         try:
@@ -966,7 +966,7 @@ class YoutubeDL(object):
 
             force_ext = OUTTMPL_TYPES.get(tmpl_type)
             if force_ext is not None:
-                filename = replace_extension(filename, force_ext, template_dict.get('ext'))
+                filename = replace_extension(filename, force_ext, info_dict.get('ext'))
 
             # https://github.com/blackjack4494/youtube-dlc/issues/85
             trim_file_name = self.params.get('trim_file_name', False)
@@ -1165,6 +1165,7 @@ class YoutubeDL(object):
         self.add_extra_info(ie_result, {
             'extractor': ie.IE_NAME,
             'webpage_url': url,
+            'original_url': url,
             'webpage_url_basename': url_basename(url),
             'extractor_key': ie.ie_key(),
         })
@@ -1184,7 +1185,11 @@ class YoutubeDL(object):
             extract_flat = self.params.get('extract_flat', False)
             if ((extract_flat == 'in_playlist' and 'playlist' in extra_info)
                     or extract_flat is True):
-                self.__forced_printings(ie_result, self.prepare_filename(ie_result), incomplete=True)
+                info_copy = ie_result.copy()
+                self.add_extra_info(info_copy, extra_info)
+                self.add_default_extra_info(
+                    info_copy, self.get_info_extractor(ie_result.get('ie_key')), ie_result['url'])
+                self.__forced_printings(info_copy, self.prepare_filename(info_copy), incomplete=True)
                 return ie_result
 
         if result_type == 'video':
@@ -2780,7 +2785,7 @@ class YoutubeDL(object):
         remove_keys = ['__original_infodict']  # Always remove this since this may contain a copy of the entire dict
         keep_keys = ['_type'],  # Always keep this to facilitate load-info-json
         if actually_filter:
-            remove_keys += ('requested_formats', 'requested_subtitles', 'requested_entries', 'filepath', 'entries')
+            remove_keys += ('requested_formats', 'requested_subtitles', 'requested_entries', 'filepath', 'entries', 'original_url')
             empty_values = (None, {}, [], set(), tuple())
             reject = lambda k, v: k not in keep_keys and (
                 k.startswith('_') or k in remove_keys or v in empty_values)

yt_dlp/__init__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # coding: utf-8
 
 from __future__ import unicode_literals
@@ -24,6 +24,7 @@ from .utils import (
     DateRange,
     decodeOption,
     DownloadError,
+    error_to_compat_str,
     ExistingVideoReached,
     expand_path,
     match_filter_func,
@@ -307,6 +308,16 @@ def _real_main(argv=None):
         else:
             _unused_compat_opt('filename')
 
+    def validate_outtmpl(tmpl, msg):
+        err = YoutubeDL.validate_outtmpl(tmpl)
+        if err:
+            parser.error('invalid %s %r: %s' % (msg, tmpl, error_to_compat_str(err)))
+
+    for k, tmpl in opts.outtmpl.items():
+        validate_outtmpl(tmpl, '%s output template' % k)
+    for tmpl in opts.forceprint:
+        validate_outtmpl(tmpl, 'print template')
+
     if opts.extractaudio and not opts.keepvideo and opts.format is None:
         opts.format = 'bestaudio/best'
 

yt_dlp/__main__.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from __future__ import unicode_literals
 
 # Execute with

yt_dlp/downloader/external.py

@@ -359,6 +359,8 @@ class FFmpegFD(ExternalFD):
             if self.params.get(log_level, False):
                 args += ['-loglevel', log_level]
                 break
+        if not self.params.get('verbose'):
+            args += ['-hide_banner']
 
         seekable = info_dict.get('_seekable')
         if seekable is not None:

yt_dlp/downloader/hls.py

@@ -272,12 +272,24 @@ class HlsFD(FragmentFD):
             if not success:
                 return False
         else:
+            def decrypt_fragment(fragment, frag_content):
+                decrypt_info = fragment['decrypt_info']
+                if decrypt_info['METHOD'] != 'AES-128':
+                    return frag_content
+                iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', fragment['media_sequence'])
+                decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
+                    self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
+                # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
+                # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
+                # not what it decrypts to.
+                if test:
+                    return frag_content
+                return AES.new(decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
+
             def download_fragment(fragment):
                 frag_index = fragment['frag_index']
                 frag_url = fragment['url']
-                decrypt_info = fragment['decrypt_info']
                 byte_range = fragment['byte_range']
-                media_sequence = fragment['media_sequence']
 
                 ctx['fragment_index'] = frag_index
 
@@ -305,18 +317,7 @@ class HlsFD(FragmentFD):
                     self.report_error('Giving up after %s fragment retries' % fragment_retries)
                     return False, frag_index
 
-                if decrypt_info['METHOD'] == 'AES-128':
-                    iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
-                    decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
-                        self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
-                    # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
-                    # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
-                    # not what it decrypts to.
-                    if not test:
-                        frag_content = AES.new(
-                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
-
-                return frag_content, frag_index
+                return decrypt_fragment(fragment, frag_content), frag_index
 
             pack_fragment = lambda frag_content, _: frag_content
 
@@ -447,7 +448,7 @@ class HlsFD(FragmentFD):
                     fragment['fragment_filename_sanitized'] = frag_sanitized
                     frag_content = down.read()
                     down.close()
-                    result = append_fragment(frag_content, frag_index)
+                    result = append_fragment(decrypt_fragment(fragment, frag_content), frag_index)
                     if not result:
                         return False
             else:

yt_dlp/extractor/awaan.py

@@ -19,7 +19,7 @@ from ..utils import (
 
 
 class AWAANIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?(?:awaan|dcndigital)\.ae/(?:#/)?show/(?P<show_id>\d+)/[^/]+(?:/(?P<video_id>\d+)/(?P<season_id>\d+))?'
+    _VALID_URL = r'https?://(?:www\.)?(?:awaan|dcndigital)\.ae/(?:#/)?show/(?P<show_id>\d+)/[^/]+(?:/(?P<id>\d+)/(?P<season_id>\d+))?'
 
     def _real_extract(self, url):
         show_id, video_id, season_id = re.match(self._VALID_URL, url).groups()

yt_dlp/extractor/common.py

@@ -203,6 +203,9 @@ class InfoExtractor(object):
                                  (HTTP or RTMP) download. Boolean.
                     * downloader_options  A dictionary of downloader options as
                                  described in FileDownloader
+                    RTMP formats can also have the additional fields: page_url,
+                    app, play_path, tc_url, flash_version, rtmp_live, rtmp_conn,
+                    rtmp_protocol, rtmp_real_time
 
     url:            Final video URL.
     ext:            Video filename extension.
@@ -2876,7 +2879,7 @@ class InfoExtractor(object):
             stream_name = stream.get('Name')
             stream_language = stream.get('Language', 'und')
             for track in stream.findall('QualityLevel'):
-                fourcc = track.get('FourCC', 'AACL' if track.get('AudioTag') == '255' else None)
+                fourcc = track.get('FourCC') or ('AACL' if track.get('AudioTag') == '255' else None)
                 # TODO: add support for WVC1 and WMAP
                 if fourcc not in ('H264', 'AVC1', 'AACL', 'TTML'):
                     self.report_warning('%s is not a supported codec' % fourcc)

yt_dlp/extractor/crunchyroll.py

@@ -120,7 +120,7 @@ class CrunchyrollBaseIE(InfoExtractor):
 
 class CrunchyrollIE(CrunchyrollBaseIE, VRVIE):
     IE_NAME = 'crunchyroll'
-    _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.(?:com|fr)/(?:media(?:-|/\?id=)|(?:[^/]*/){1,2}[^/?&]*?)(?P<video_id>[0-9]+))(?:[/?&]|$)'
+    _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.(?:com|fr)/(?:media(?:-|/\?id=)|(?:[^/]*/){1,2}[^/?&]*?)(?P<id>[0-9]+))(?:[/?&]|$)'
     _TESTS = [{
         'url': 'http://www.crunchyroll.com/wanna-be-the-strongest-in-the-world/episode-1-an-idol-wrestler-is-born-645513',
         'info_dict': {
@@ -413,7 +413,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
 
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
-        video_id = mobj.group('video_id')
+        video_id = mobj.group('id')
 
         if mobj.group('prefix') == 'm':
             mobile_webpage = self._download_webpage(url, video_id, 'Downloading mobile webpage')

yt_dlp/extractor/extractors.py

@@ -1429,7 +1429,11 @@ from .tweakers import TweakersIE
 from .twentyfourvideo import TwentyFourVideoIE
 from .twentymin import TwentyMinutenIE
 from .twentythreevideo import TwentyThreeVideoIE
-from .twitcasting import TwitCastingIE
+from .twitcasting import (
+    TwitCastingIE,
+    TwitCastingLiveIE,
+    TwitCastingUserIE,
+)
 from .twitch import (
     TwitchVodIE,
     TwitchCollectionIE,
@@ -1510,7 +1514,11 @@ from .videomore import (
     VideomoreSeasonIE,
 )
 from .videopress import VideoPressIE
-from .vidio import VidioIE
+from .vidio import (
+    VidioIE,
+    VidioPremierIE,
+    VidioLiveIE
+)
 from .vidlii import VidLiiIE
 from .vidme import (
     VidmeIE,

yt_dlp/extractor/facebook.py

@@ -519,7 +519,10 @@ class FacebookIE(InfoExtractor):
                 raise ExtractorError(
                     'The video is not available, Facebook said: "%s"' % m_msg.group(1),
                     expected=True)
-            elif '>You must log in to continue' in webpage:
+            elif any(p in webpage for p in (
+                    '>You must log in to continue',
+                    'id="login_form"',
+                    'id="loginbutton"')):
                 self.raise_login_required()
 
         if not video_data and '/watchparty/' in url:

yt_dlp/extractor/formula1.py

@@ -5,29 +5,23 @@ from .common import InfoExtractor
 
 
 class Formula1IE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?formula1\.com/(?:content/fom-website/)?en/video/\d{4}/\d{1,2}/(?P<id>.+?)\.html'
-    _TESTS = [{
-        'url': 'http://www.formula1.com/content/fom-website/en/video/2016/5/Race_highlights_-_Spain_2016.html',
-        'md5': '8c79e54be72078b26b89e0e111c0502b',
+    _VALID_URL = r'https?://(?:www\.)?formula1\.com/en/latest/video\.[^.]+\.(?P<id>\d+)\.html'
+    _TEST = {
+        'url': 'https://www.formula1.com/en/latest/video.race-highlights-spain-2016.6060988138001.html',
+        'md5': 'be7d3a8c2f804eb2ab2aa5d941c359f8',
         'info_dict': {
-            'id': 'JvYXJpMzE6pArfHWm5ARp5AiUmD-gibV',
+            'id': '6060988138001',
             'ext': 'mp4',
             'title': 'Race highlights - Spain 2016',
+            'timestamp': 1463332814,
+            'upload_date': '20160515',
+            'uploader_id': '6057949432001',
         },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-        'add_ie': ['Ooyala'],
-    }, {
-        'url': 'http://www.formula1.com/en/video/2016/5/Race_highlights_-_Spain_2016.html',
-        'only_matching': True,
-    }]
+        'add_ie': ['BrightcoveNew'],
+    }
+    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/6057949432001/S1WMrhjlh_default/index.html?videoId=%s'
 
     def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        ooyala_embed_code = self._search_regex(
-            r'data-videoid="([^"]+)"', webpage, 'ooyala embed code')
+        bc_id = self._match_id(url)
         return self.url_result(
-            'ooyala:%s' % ooyala_embed_code, 'Ooyala', ooyala_embed_code)
+            self.BRIGHTCOVE_URL_TEMPLATE % bc_id, 'BrightcoveNew', bc_id)

yt_dlp/extractor/metacafe.py

@@ -19,7 +19,7 @@ from ..utils import (
 
 
 class MetacafeIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?metacafe\.com/watch/(?P<video_id>[^/]+)/(?P<display_id>[^/?#]+)'
+    _VALID_URL = r'https?://(?:www\.)?metacafe\.com/watch/(?P<id>[^/]+)/(?P<display_id>[^/?#]+)'
     _DISCLAIMER = 'http://www.metacafe.com/family_filter/'
     _FILTER_POST = 'http://www.metacafe.com/f/index.php?inputType=filter&controllerGroup=user'
     IE_NAME = 'metacafe'

yt_dlp/extractor/orf.py

@@ -140,6 +140,25 @@ class ORFTVthekIE(InfoExtractor):
                 })
 
             upload_date = unified_strdate(sd.get('created_date'))
+
+            thumbnails = []
+            preview = sd.get('preview_image_url')
+            if preview:
+                thumbnails.append({
+                    'id': 'preview',
+                    'url': preview,
+                    'preference': 0,
+                })
+            image = sd.get('image_full_url')
+            if not image and len(data_jsb) == 1:
+                image = self._og_search_thumbnail(webpage)
+            if image:
+                thumbnails.append({
+                    'id': 'full',
+                    'url': image,
+                    'preference': 1,
+                })
+
             entries.append({
                 '_type': 'video',
                 'id': video_id,
@@ -149,7 +168,7 @@ class ORFTVthekIE(InfoExtractor):
                 'description': sd.get('description'),
                 'duration': int_or_none(sd.get('duration_in_seconds')),
                 'upload_date': upload_date,
-                'thumbnail': sd.get('image_full_url'),
+                'thumbnails': thumbnails,
             })
 
         return {

yt_dlp/extractor/sina.py

@@ -18,7 +18,7 @@ from ..utils import (
 class SinaIE(InfoExtractor):
     _VALID_URL = r'''(?x)https?://(?:.*?\.)?video\.sina\.com\.cn/
                         (?:
-                            (?:view/|.*\#)(?P<video_id>\d+)|
+                            (?:view/|.*\#)(?P<id>\d+)|
                             .+?/(?P<pseudo_id>[^/?#]+)(?:\.s?html)|
                             # This is used by external sites like Weibo
                             api/sinawebApi/outplay.php/(?P<token>.+?)\.swf
@@ -58,7 +58,7 @@ class SinaIE(InfoExtractor):
     def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
 
-        video_id = mobj.group('video_id')
+        video_id = mobj.group('id')
         if not video_id:
             if mobj.group('token') is not None:
                 # The video id is in the redirected url

yt_dlp/extractor/trovolive.py

@@ -1,111 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-
-from ..utils import (
-    js_to_json,
-    try_get,
-    int_or_none,
-    str_or_none,
-    url_or_none,
-)
-from ..compat import compat_str
-
-
-class TrovoLiveIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?trovo\.live/video/(?P<id>[\w-]+)'
-    _TEST = {
-        'url': 'https://trovo.live/video/ltv-100759829_100759829_1610625308',
-        'md5': 'ea7b58427910e9af66a462d895201a30',
-        'info_dict': {
-            'id': 'ltv-100759829_100759829_1610625308',
-            'ext': 'ts',
-            'title': 'GTA RP ASTERIX doa najjaca',
-            'uploader': 'Peroo42',
-            'duration': 5872,
-            'view_count': int,
-            'like_count': int,
-            'comment_count': int,
-            'categories': list,
-            'is_live': False,
-            'thumbnail': r're:^https?://.*\.jpg$',
-            'uploader_id': '100759829',
-        }
-    }
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        nuxt = self._search_regex(r'\bwindow\.__NUXT__\s*=\s*(.+?);?\s*</script>', webpage, 'nuxt', default='')
-        mobj = re.search(r'\((?P<arg_names>[^(]+)\)\s*{\s*return\s+(?P<json>{.+})\s*\((?P<args>.+?)\)\s*\)$', nuxt)
-
-        vod_details = vod_info = {}
-        if mobj:
-            vod_details = self._parse_json(
-                js_to_json(
-                    self._search_regex(r'VodDetailInfos\s*:({.+?}),\s*_', webpage, 'VodDetailInfos'),
-                    dict(zip(
-                        (i.strip() for i in mobj.group('arg_names').split(',')),
-                        (i.strip() for i in mobj.group('args').split(','))))),
-                video_id)
-            vod_info = try_get(vod_details, lambda x: x['json'][video_id]['vodInfo'], dict) or {}
-
-        player_info = self._parse_json(
-            self._search_regex(
-                r'_playerInfo\s*=\s*({.+?})\s*</script>', webpage, 'player info'),
-            video_id)
-
-        title = (
-            vod_info.get('title')
-            or self._html_search_regex(r'<h3>(.+?)</h3>', webpage, 'title', fatal=False)
-            or self._og_search_title(webpage))
-        uploader = (
-            try_get(vod_details, lambda x: x['json'][video_id]['streamerInfo']['userName'], compat_str)
-            or self._search_regex(r'<div[^>]+userName\s=\s[\'"](.+?)[\'"]', webpage, 'uploader', fatal=False))
-
-        format_dicts = vod_info.get('playInfos') or player_info.get('urlArray') or []
-
-        def _extract_format_data(format_dict):
-            res = format_dict.get('desc')
-            enc = str_or_none(format_dict.get('encodeType'))
-            if enc:
-                notes = [enc.replace('VOD_ENCODE_TYPE_', '')]
-            level = str_or_none(format_dict.get('levelType'))
-            if level:
-                notes.append('level %s' % level)
-            height = int_or_none(res[:-1]) if res else None
-            bitrate = format_dict.get('bitrate')
-            fid = res or ('%sk' % str_or_none(bitrate) if bitrate else None) or ' '.join(notes)
-
-            return {
-                'url': format_dict['playUrl'],
-                'format_id': fid,
-                'format_note': ' '.join(notes),
-                'height': height,
-                'resolution': str_or_none(res),
-                'tbr': int_or_none(bitrate),
-                'filesize': int_or_none(format_dict.get('fileSize')),
-                'vcodec': 'avc3',
-                'acodec': 'aac',
-                'ext': 'ts'
-            }
-
-        formats = [_extract_format_data(f) for f in format_dicts]
-        self._sort_formats(formats)
-        return {
-            'id': video_id,
-            'title': title,
-            'uploader': uploader,
-            'duration': int_or_none(vod_info.get('duration')),
-            'formats': formats,
-            'view_count': int_or_none(vod_info.get('watchNum')),
-            'like_count': int_or_none(vod_info.get('likeNum')),
-            'comment_count': int_or_none(vod_info.get('commentNum')),
-            'categories': [str_or_none(vod_info.get('categoryName'))],
-            'is_live': try_get(player_info, lambda x: x['isLive'], bool),
-            'thumbnail': url_or_none(vod_info.get('coverUrl')),
-            'uploader_id': str_or_none(try_get(vod_details, lambda x: x['json'][video_id]['streamerInfo']['uid'])),
-        }

yt_dlp/extractor/twitcasting.py

@@ -1,6 +1,7 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import itertools
 import re
 
 from .common import InfoExtractor
@@ -11,13 +12,16 @@ from ..utils import (
     get_element_by_id,
     parse_duration,
     str_to_int,
+    try_get,
     unified_timestamp,
     urlencode_postdata,
+    urljoin,
+    ExtractorError,
 )
 
 
 class TwitCastingIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:[^/]+\.)?twitcasting\.tv/(?P<uploader_id>[^/]+)/movie/(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:[^/]+\.)?twitcasting\.tv/(?P<uploader_id>[^/]+)/(?:movie|twplayer)/(?P<id>\d+)'
     _TESTS = [{
         'url': 'https://twitcasting.tv/ivetesangalo/movie/2357609',
         'md5': '745243cad58c4681dc752490f7540d7f',
@@ -69,9 +73,8 @@ class TwitCastingIE(InfoExtractor):
             url, video_id, data=request_data,
             headers={'Origin': 'https://twitcasting.tv'})
 
-        title = clean_html(get_element_by_id(
-            'movietitle', webpage)) or self._html_search_meta(
-            ['og:title', 'twitter:title'], webpage, fatal=True)
+        title = (clean_html(get_element_by_id('movietitle', webpage))
+                 or self._html_search_meta(['og:title', 'twitter:title'], webpage, fatal=True))
 
         video_js_data = {}
         m3u8_url = self._search_regex(
@@ -80,14 +83,16 @@ class TwitCastingIE(InfoExtractor):
         if not m3u8_url:
             video_js_data = self._parse_json(self._search_regex(
                 r'data-movie-playlist=(["\'])(?P<url>(?:(?!\1).)+)',
-                webpage, 'movie playlist', group='url'), video_id)
+                webpage, 'movie playlist', group='url', default='[{}]'), video_id)
             if isinstance(video_js_data, dict):
                 video_js_data = list(video_js_data.values())[0]
             video_js_data = video_js_data[0]
-            m3u8_url = video_js_data['source']['url']
+            m3u8_url = try_get(video_js_data, lambda x: x['source']['url'])
+
+        is_live = 'data-status="online"' in webpage
+        if is_live and not m3u8_url:
+            m3u8_url = 'https://twitcasting.tv/%s/metastream.m3u8' % uploader_id
 
-        formats = self._extract_m3u8_formats(
-            m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls')
         thumbnail = video_js_data.get('thumbnailUrl') or self._og_search_thumbnail(webpage)
         description = clean_html(get_element_by_id(
             'authorcomment', webpage)) or self._html_search_meta(
@@ -101,6 +106,12 @@ class TwitCastingIE(InfoExtractor):
             r'data-toggle="true"[^>]+datetime="([^"]+)"',
             webpage, 'datetime', None))
 
+        formats = None
+        if m3u8_url:
+            formats = self._extract_m3u8_formats(
+                m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', live=is_live)
+        self._sort_formats(formats)
+
         return {
             'id': video_id,
             'title': title,
@@ -111,4 +122,59 @@ class TwitCastingIE(InfoExtractor):
             'duration': duration,
             'view_count': view_count,
             'formats': formats,
+            'is_live': is_live,
         }
+
+
+class TwitCastingLiveIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:[^/]+\.)?twitcasting\.tv/(?P<id>[^/]+)/?(?:[#?]|$)'
+    _TESTS = [{
+        'url': 'https://twitcasting.tv/ivetesangalo',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        uploader_id = self._match_id(url)
+        self.to_screen(
+            'Downloading live video of user {0}. '
+            'Pass "https://twitcasting.tv/{0}/show" to download the history'.format(uploader_id))
+
+        webpage = self._download_webpage(url, uploader_id)
+        current_live = self._search_regex(
+            (r'data-type="movie" data-id="(\d+)">',
+             r'tw-sound-flag-open-link" data-id="(\d+)" style=',),
+            webpage, 'current live ID', default=None)
+        if not current_live:
+            raise ExtractorError('The user is not currently live')
+        return self.url_result('https://twitcasting.tv/%s/movie/%s' % (uploader_id, current_live))
+
+
+class TwitCastingUserIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:[^/]+\.)?twitcasting\.tv/(?P<id>[^/]+)/show/?(?:[#?]|$)'
+    _TESTS = [{
+        'url': 'https://twitcasting.tv/noriyukicas/show',
+        'only_matching': True,
+    }]
+
+    def _entries(self, uploader_id):
+        base_url = next_url = 'https://twitcasting.tv/%s/show' % uploader_id
+        for page_num in itertools.count(1):
+            webpage = self._download_webpage(
+                next_url, uploader_id, query={'filter': 'watchable'}, note='Downloading page %d' % page_num)
+            matches = re.finditer(
+                r'''(?isx)<a\s+class="tw-movie-thumbnail"\s*href="(?P<url>/[^/]+/movie/\d+)"\s*>.+?</a>''',
+                webpage)
+            for mobj in matches:
+                yield self.url_result(urljoin(base_url, mobj.group('url')))
+
+            next_url = self._search_regex(
+                r'<a href="(/%s/show/%d-\d+)[?"]' % (re.escape(uploader_id), page_num),
+                webpage, 'next url', default=None)
+            next_url = urljoin(base_url, next_url)
+            if not next_url:
+                return
+
+    def _real_extract(self, url):
+        uploader_id = self._match_id(url)
+        return self.playlist_result(
+            self._entries(uploader_id), uploader_id, '%s - Live History' % uploader_id)