Release 2021.01.24

Extractor plugins are loaded from <root-dir>/ytdlp_plugins/extractor/__init__.py

Inspired by https://github.com/un-def/dl-plus

:ci skip dl

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.20. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.01.16**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.20**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.16
+ [debug] yt-dlp version 2021.01.20
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.20. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/pukkandan/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.16**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.20**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.20. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.16**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.20**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.20. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/pukkandan/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.01.16**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.20**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run youtube-dlc with (`youtube-dlc -v
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.01.16
+ [debug] yt-dlp version 2021.01.20
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dlc:
-- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.16. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `youtube-dlc --version` and ensure your version is 2021.01.20. If it's not, see https://github.com/pukkandan/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/pukkandan/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.01.16**
+- [ ] I've verified that I'm running yt-dlp version **2021.01.20**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/core.yml

@@ -1,9 +1,9 @@
-name: Full Test
+name: Core Tests
 on: [push, pull_request]
 jobs:
   tests:
-    name: Tests
+    name: Core Tests
-    if: "!contains(github.event.head_commit.message, 'skip ci')"
+    if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: true
@@ -12,7 +12,7 @@ jobs:
         # TODO: python 2.6
         python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
         python-impl: [cpython]
-        ytdl-test-set: [core, download]
+        ytdl-test-set: [core]
         run-tests-ext: [sh]
         include:
         # python 3.2 is only available on windows via setup-python
@@ -21,20 +21,11 @@ jobs:
           python-impl: cpython
           ytdl-test-set: core
           run-tests-ext: bat
-        - os: windows-latest
-          python-version: 3.2
-          python-impl: cpython
-          ytdl-test-set: download
-          run-tests-ext: bat
         # jython
         - os: ubuntu-latest
           python-impl: jython
           ytdl-test-set: core
           run-tests-ext: sh
-        - os: ubuntu-latest
-          python-impl: jython
-          ytdl-test-set: download
-          run-tests-ext: sh
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python ${{ matrix.python-version }}
@@ -60,4 +51,4 @@ jobs:
       env:
         YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
       run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}
-  # flake8 has been moved to quick-test
+  # Linter is in quick-test

.github/workflows/download.yml

@@ -0,0 +1,53 @@
+name: Download Tests
+on: [push, pull_request]
+jobs:
+  tests:
+    name: Download Tests
+    if: "!contains(github.event.head_commit.message, 'ci skip dl') && !contains(github.event.head_commit.message, 'ci skip all')"
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: true
+      matrix:
+        os: [ubuntu-18.04]
+        # TODO: python 2.6
+        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
+        python-impl: [cpython]
+        ytdl-test-set: [download]
+        run-tests-ext: [sh]
+        include:
+        # python 3.2 is only available on windows via setup-python
+        - os: windows-latest
+          python-version: 3.2
+          python-impl: cpython
+          ytdl-test-set: download
+          run-tests-ext: bat
+        # jython - disable for now since it takes too long to complete
+        # - os: ubuntu-latest
+        #   python-impl: jython
+        #   ytdl-test-set: download
+        #   run-tests-ext: sh
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      if: ${{ matrix.python-impl == 'cpython' }}
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Set up Java 8
+      if: ${{ matrix.python-impl == 'jython' }}
+      uses: actions/setup-java@v1
+      with:
+        java-version: 8
+    - name: Install Jython
+      if: ${{ matrix.python-impl == 'jython' }}
+      run: |
+        wget http://search.maven.org/remotecontent?filepath=org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
+        java -jar jython-installer.jar -s -d "$HOME/jython"
+        echo "$HOME/jython/bin" >> $GITHUB_PATH
+    - name: Install nose
+      run: pip install nose
+    - name: Run tests
+      continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
+      env:
+        YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
+      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}

.github/workflows/quick-test.yml

@@ -1,13 +1,13 @@
-name: Core Test
+name: Quick Test
 on: [push, pull_request]
 jobs:
   tests:
     name: Core Tests
-    if: "!contains(github.event.head_commit.message, 'skip ci all')"
+    if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
-    - name: Set up Python 3.9
+    - name: Set up Python
       uses: actions/setup-python@v2
       with:
         python-version: 3.9
@@ -19,7 +19,7 @@ jobs:
       run: ./devscripts/run_tests.sh
   flake8:
     name: Linter
-    if: "!contains(github.event.head_commit.message, 'skip ci all')"
+    if: "!contains(github.event.head_commit.message, 'ci skip all')"
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2

.gitignore

@@ -65,6 +65,14 @@ venv/
 # VS Code related files
 .vscode
 
+# SublimeText files
+*.sublime-workspace
+
+# Cookies
+cookies
 cookies.txt
 
-*.sublime-workspace
+# Plugins
+ytdlp_plugins/extractor/*
+!ytdlp_plugins/extractor/__init__.py
+!ytdlp_plugins/extractor/sample.py

AUTHORS

@@ -1,248 +0,0 @@
-Ricardo Garcia Gonzalez
-Danny Colligan
-Benjamin Johnson
-Vasyl' Vavrychuk
-Witold Baryluk
-Paweł Paprota
-Gergely Imreh
-Rogério Brito
-Philipp Hagemeister
-Sören Schulze
-Kevin Ngo
-Ori Avtalion
-shizeeg
-Filippo Valsorda
-Christian Albrecht
-Dave Vasilevsky
-Jaime Marquínez Ferrándiz
-Jeff Crouse
-Osama Khalid
-Michael Walter
-M. Yasoob Ullah Khalid
-Julien Fraichard
-Johny Mo Swag
-Axel Noack
-Albert Kim
-Pierre Rudloff
-Huarong Huo
-Ismael Mejía
-Steffan Donal
-Andras Elso
-Jelle van der Waa
-Marcin Cieślak
-Anton Larionov
-Takuya Tsuchida
-Sergey M.
-Michael Orlitzky
-Chris Gahan
-Saimadhav Heblikar
-Mike Col
-Oleg Prutz
-pulpe
-Andreas Schmitz
-Michael Kaiser
-Niklas Laxström
-David Triendl
-Anthony Weems
-David Wagner
-Juan C. Olivares
-Mattias Harrysson
-phaer
-Sainyam Kapoor
-Nicolas Évrard
-Jason Normore
-Hoje Lee
-Adam Thalhammer
-Georg Jähnig
-Ralf Haring
-Koki Takahashi
-Ariset Llerena
-Adam Malcontenti-Wilson
-Tobias Bell
-Naglis Jonaitis
-Charles Chen
-Hassaan Ali
-Dobrosław Żybort
-David Fabijan
-Sebastian Haas
-Alexander Kirk
-Erik Johnson
-Keith Beckman
-Ole Ernst
-Aaron McDaniel (mcd1992)
-Magnus Kolstad
-Hari Padmanaban
-Carlos Ramos
-5moufl
-lenaten
-Dennis Scheiba
-Damon Timm
-winwon
-Xavier Beynon
-Gabriel Schubiner
-xantares
-Jan Matějka
-Mauroy Sébastien
-William Sewell
-Dao Hoang Son
-Oskar Jauch
-Matthew Rayfield
-t0mm0
-Tithen-Firion
-Zack Fernandes
-cryptonaut
-Adrian Kretz
-Mathias Rav
-Petr Kutalek
-Will Glynn
-Max Reimann
-Cédric Luthi
-Thijs Vermeir
-Joel Leclerc
-Christopher Krooss
-Ondřej Caletka
-Dinesh S
-Johan K. Jensen
-Yen Chi Hsuan
-Enam Mijbah Noor
-David Luhmer
-Shaya Goldberg
-Paul Hartmann
-Frans de Jonge
-Robin de Rooij
-Ryan Schmidt
-Leslie P. Polzer
-Duncan Keall
-Alexander Mamay
-Devin J. Pohly
-Eduardo Ferro Aldama
-Jeff Buchbinder
-Amish Bhadeshia
-Joram Schrijver
-Will W.
-Mohammad Teimori Pabandi
-Roman Le Négrate
-Matthias Küch
-Julian Richen
-Ping O.
-Mister Hat
-Peter Ding
-jackyzy823
-George Brighton
-Remita Amine
-Aurélio A. Heckert
-Bernhard Minks
-sceext
-Zach Bruggeman
-Tjark Saul
-slangangular
-Behrouz Abbasi
-ngld
-nyuszika7h
-Shaun Walbridge
-Lee Jenkins
-Anssi Hannula
-Lukáš Lalinský
-Qijiang Fan
-Rémy Léone
-Marco Ferragina
-reiv
-Muratcan Simsek
-Evan Lu
-flatgreen
-Brian Foley
-Vignesh Venkat
-Tom Gijselinck
-Founder Fang
-Andrew Alexeyew
-Saso Bezlaj
-Erwin de Haan
-Jens Wille
-Robin Houtevelts
-Patrick Griffis
-Aidan Rowe
-mutantmonkey
-Ben Congdon
-Kacper Michajłow
-José Joaquín Atria
-Viťas Strádal
-Kagami Hiiragi
-Philip Huppert
-blahgeek
-Kevin Deldycke
-inondle
-Tomáš Čech
-Déstin Reed
-Roman Tsiupa
-Artur Krysiak
-Jakub Adam Wieczorek
-Aleksandar Topuzović
-Nehal Patel
-Rob van Bekkum
-Petr Zvoníček
-Pratyush Singh
-Aleksander Nitecki
-Sebastian Blunt
-Matěj Cepl
-Xie Yanbo
-Philip Xu
-John Hawkinson
-Rich Leeper
-Zhong Jianxin
-Thor77
-Mattias Wadman
-Arjan Verwer
-Costy Petrisor
-Logan B
-Alex Seiler
-Vijay Singh
-Paul Hartmann
-Stephen Chen
-Fabian Stahl
-Bagira
-Odd Stråbø
-Philip Herzog
-Thomas Christlieb
-Marek Rusinowski
-Tobias Gruetzmacher
-Olivier Bilodeau
-Lars Vierbergen
-Juanjo Benages
-Xiao Di Guan
-Thomas Winant
-Daniel Twardowski
-Jeremie Jarosh
-Gerard Rovira
-Marvin Ewald
-Frédéric Bournival
-Timendum
-gritstub
-Adam Voss
-Mike Fährmann
-Jan Kundrát
-Giuseppe Fabiano
-Örn Guðjónsson
-Parmjit Virk
-Genki Sky
-Ľuboš Katrinec
-Corey Nicholson
-Ashutosh Chaudhary
-John Dong
-Tatsuyuki Ishi
-Daniel Weber
-Kay Bouché
-Yang Hongbo
-Lei Wang
-Petr Novák
-Leonardo Taccari
-Martin Weinelt
-Surya Oktafendri
-TingPing
-Alexandre Macabies
-Bastian de Groot
-Niklas Haas
-András Veres-Szentkirályi
-Enes Solak
-Nathan Rossi
-Thomas van der Berg
-Luca Cherubin

CONTRIBUTORS

@@ -15,4 +15,5 @@ ohnonot
 samiksome
 alxnull
 FelixFrog
-Zocker1999NET
+Zocker1999NET
+nao20010128nao

Changelog.md

@@ -4,25 +4,47 @@
 # Instuctions for creating release
 
 * Run `make doc`
-* Update Changelog.md and Authors-Fork
+* Update Changelog.md and CONTRIBUTORS
+* Change "Merged with youtube-dl" version in Readme.md if needed
 * Commit to master as `Release <version>`
 * Push to origin/release - build task will now run
 * Update version.py and run `make issuetemplates`
-* Commit to master as `[version] update`
+* Commit to master as `[version] update :skip ci all`
 * Push to origin/master
 * Update changelog in /releases
 
 -->
 
 
+### 2021.01.24
+* **Merge youtube-dl:** Upto [2021.01.24](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
+* Plugin support ([documentation](https://github.com/pukkandan/yt-dlp#plugins))
+* **Multiple paths**: New option `-P`/`--paths` to give different paths for different types of files
+    * The syntax is `-P "type:path" -P "type:path"` ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=-P,%20--paths%20TYPE:PATH))
+    * Valid types are: home, temp, description, annotation, subtitle, infojson, thumbnail
+    * Additionally, configuration file is taken from home directory or current directory ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=Home%20Configuration))
+* Allow passing different arguments to different external downloaders ([documentation](https://github.com/pukkandan/yt-dlp#:~:text=--downloader-args%20NAME:ARGS))
+* [mildom] Add extractor by @nao20010128nao
+* Warn when using old style `--external-downloader-args` and `--post-processor-args`
+* Fix `--no-overwrite` when using `--write-link`
+* [sponskrub] Output `unrecognized argument` error message correctly
+* [cbs] Make failure to extract title non-fatal
+* Fix typecasting when pre-checking archive
+* Fix issue with setting title on unix
+* Deprecate redundant aliases in `formatSort`. The aliases remain functional for backward compatability, but will be left undocumented
+* [tests] Fix test_post_hooks
+* [tests] Split core and download tests
+
+
 ### 2021.01.20
 * [TrovoLive] Add extractor (only VODs)
-* [pokemon] Add `/#/player` URLs (Closes #24)
+* [pokemon] Add `/#/player` URLs
 * Improved parsing of multiple postprocessor-args, add `--ppa` as alias
 * [EmbedThumbnail] Simplify embedding in mkv
 * [sponskrub] Encode filenames correctly, better debug output and error message
 * [readme] Cleanup options
 
+
 ### 2021.01.16
 * **Merge youtube-dl:** Upto [2021.01.16](https://github.com/ytdl-org/youtube-dl/releases/tag/2021.01.16)
 * **Configuration files:**

README.md

@@ -2,9 +2,8 @@
 
 <!-- See: https://github.com/marketplace/actions/dynamic-badges -->
 [![Release Version](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/pukkandan/c69cb23c3c5b3316248e52022790aa57/raw/version.json&color=brightgreen)](https://github.com/pukkandan/yt-dlp/releases/latest)
-[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](https://github.com/pukkandan/yt-dlp/blob/master/LICENSE)
+[![License: Unlicense](https://img.shields.io/badge/License-Unlicense-blue.svg)](LICENSE)
-[![Core Status](https://github.com/pukkandan/yt-dlp/workflows/Core%20Test/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions?query=workflow%3ACore)
+[![CI Status](https://github.com/pukkandan/yt-dlp/workflows/Core%20Tests/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions)
-[![CI Status](https://github.com/pukkandan/yt-dlp/workflows/Full%20Test/badge.svg?branch=master)](https://github.com/pukkandan/yt-dlp/actions?query=workflow%3AFull)
 
 A command-line program to download videos from youtube.com and many other [video platforms](docs/supportedsites.md)
 
@@ -41,6 +40,7 @@ This is a fork of [youtube-dlc](https://github.com/blackjack4494/yt-dlc) which i
     * [Filtering Formats](#filtering-formats)
     * [Sorting Formats](#sorting-formats)
     * [Format Selection examples](#format-selection-examples)
+* [PLUGINS](#plugins)
 * [MORE](#more)
 
 
@@ -51,20 +51,26 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
 
 * **[Format Sorting](#sorting-formats)**: The default format sorting options have been changed so that higher resolution and better codecs will be now preferred instead of simply using larger bitrate. Furthermore, you can now specify the sort order using `-S`. This allows for much easier format selection that what is possible by simply using `--format` ([examples](#format-selection-examples))
 
-* **Merged with youtube-dl v2021.01.16**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
+* **Merged with youtube-dl v2021.01.24.1**: You get all the latest features and patches of [youtube-dl](https://github.com/ytdl-org/youtube-dl) in addition to all the features of [youtube-dlc](https://github.com/blackjack4494/yt-dlc)
 
 * **Youtube improvements**:
     * All Youtube Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) works correctly and support downloading multiple pages of content
     * Youtube search works correctly (`ytsearch:`, `ytsearchdate:`) along with Search URLs
     * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
 
-* **New extractors**: Trovo.live, AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv
+* **New extractors**: AnimeLab, Philo MSO, Rcs, Gedi, bitwave.tv, mildom
 
 * **Fixed extractors**: archive.org, roosterteeth.com, skyit, instagram, itv, SouthparkDe, spreaker, Vlive, tiktok, akamai, ina
 
-* **New options**: `--list-formats-as-table`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc
+* **Plugin support**: Extractors can be loaded from an external file. See [plugins](#plugins) for details
 
-* **Improvements**: Multiple `--postprocessor-args`, `%(duration_string)s` in `-o`, faster archive checking, more [format selection options](#format-selection) etc
+* **Multiple paths**: You can give different paths for different types of files. You can also set a temporary path where intermediary files are downloaded to. See [--paths](#:~:text=-P,%20--paths%20TYPE:PATH) for details
+
+* **Portable Configuration**: Configuration files are automatically loaded from the home and root directories. See [configuration](#configuration) for details
+
+* **Other new options**: `--list-formats-as-table`, `--write-link`, `--force-download-archive`, `--force-overwrites`, `--break-on-reject` etc
+
+* **Improvements**: Multiple `--postprocessor-args` and `--external-downloader-args`, `%(duration_string)s` in `-o`, faster archive checking, more [format selection options](#format-selection) etc
 
 See [changelog](Changelog.md) or [commits](https://github.com/pukkandan/yt-dlp/commits) for the full list of changes
 
@@ -151,9 +157,9 @@ Then simply type this
                                      compatibility) if this option is found
                                      inside the system configuration file, the
                                      user configuration is not loaded
-    --config-location PATH           Location of the configuration file; either
+    --config-location PATH           Location of the main configuration file;
-                                     the path to the config or its containing
+                                     either the path to the config or its
-                                     directory
+                                     containing directory
     --flat-playlist                  Do not extract the videos of a playlist,
                                      only list them
     --flat-videos                    Do not resolve the video urls
@@ -303,19 +309,36 @@ Then simply type this
                                      allowing to play the video while
                                      downloading (some players may not be able
                                      to play it)
-    --external-downloader COMMAND    Use the specified external downloader.
+    --external-downloader NAME       Use the specified external downloader.
-                                     Currently supports
+                                     Currently supports aria2c, avconv, axel,
-                                     aria2c,avconv,axel,curl,ffmpeg,httpie,wget
+                                     curl, ffmpeg, httpie, wget
-    --external-downloader-args ARGS  Give these arguments to the external
+    --downloader-args NAME:ARGS      Give these arguments to the external
-                                     downloader
+                                     downloader. Specify the downloader name and
+                                     the arguments separated by a colon ":". You
+                                     can use this option multiple times (Alias:
+                                     --external-downloader-args)
 
 ## Filesystem Options:
     -a, --batch-file FILE            File containing URLs to download ('-' for
                                      stdin), one URL per line. Lines starting
                                      with '#', ';' or ']' are considered as
                                      comments and ignored
+    -P, --paths TYPE:PATH            The paths where the files should be
+                                     downloaded. Specify the type of file and
+                                     the path separated by a colon ":"
+                                     (supported: description|annotation|subtitle
+                                     |infojson|thumbnail). Additionally, you can
+                                     also provide "home" and "temp" paths. All
+                                     intermediary files are first downloaded to
+                                     the temp path and then the final files are
+                                     moved over to the home path after download
+                                     is finished. Note that this option is
+                                     ignored if --output is an absolute path
     -o, --output TEMPLATE            Output filename template, see "OUTPUT
                                      TEMPLATE" for details
+    --output-na-placeholder TEXT     Placeholder value for unavailable meta
+                                     fields in output filename template
+                                     (default: "NA")
     --autonumber-start NUMBER        Specify the start value for %(autonumber)s
                                      (default is 1)
     --restrict-filenames             Restrict filenames to only ASCII
@@ -435,7 +458,7 @@ Then simply type this
     --referer URL                    Specify a custom referer, use if the video
                                      access is restricted to one domain
     --add-header FIELD:VALUE         Specify a custom HTTP header and its value,
-                                     separated by a colon ':'. You can use this
+                                     separated by a colon ":". You can use this
                                      option multiple times
     --bidi-workaround                Work around terminals that lack
                                      bidirectional text support. Requires bidiv
@@ -554,8 +577,8 @@ Then simply type this
                                      supported: mp4|flv|ogg|webm|mkv|avi)
     --postprocessor-args NAME:ARGS   Give these arguments to the postprocessors.
                                      Specify the postprocessor/executable name
-                                     and the arguments separated by a colon ':'
+                                     and the arguments separated by a colon ":"
-                                     to give the argument to only the specified
+                                     to give the argument to the specified
                                      postprocessor/executable. Supported
                                      postprocessors are: SponSkrub,
                                      ExtractAudio, VideoRemuxer, VideoConvertor,
@@ -569,7 +592,8 @@ Then simply type this
                                      to different postprocessors. You can also
                                      specify "PP+EXE:ARGS" to give the arguments
                                      to the specified executable only when being
-                                     used by the specified postprocessor (Alias:
+                                     used by the specified postprocessor. You
+                                     can use this option multiple times (Alias:
                                      --ppa)
     -k, --keep-video                 Keep the intermediate video file on disk
                                      after post-processing
@@ -648,8 +672,9 @@ Then simply type this
 
 You can configure youtube-dlc by placing any supported command line option to a configuration file. The configuration is loaded from the following locations:
 
-1. The file given by `--config-location`
+1. **Main Configuration**: The file given by `--config-location`
 1. **Portable Configuration**: `yt-dlp.conf` or `youtube-dlc.conf` in the same directory as the bundled binary. If you are running from source-code (`<root dir>/youtube_dlc/__main__.py`), the root directory is used instead.
+1. **Home Configuration**: `yt-dlp.conf` or `youtube-dlc.conf` in the home path given by `-P "home:<path>"`, or in the current directory if no such path is given
 1. **User Configuration**:
     * `%XDG_CONFIG_HOME%/yt-dlp/config` (recommended on Linux/macOS)
     * `%XDG_CONFIG_HOME%/yt-dlp.conf`
@@ -707,7 +732,7 @@ set HOME=%USERPROFILE%
 
 # OUTPUT TEMPLATE
 
-The `-o` option allows users to indicate a template for the output file names.
+The `-o` option is used to indicate a template for the output file names while `-P` option is used to specify the path each type of file should be saved to.
 
 **tl;dr:** [navigate me to examples](#output-template-examples).
 
@@ -798,7 +823,7 @@ Available for the media that is a track or a part of a music album:
  - `disc_number` (numeric): Number of the disc or other physical medium the track belongs to
  - `release_year` (numeric): Year (YYYY) when the album was released
 
-Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with `NA`.
+Each aforementioned sequence when referenced in an output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by a particular extractor. Such sequences will be replaced with placeholder value provided with `--output-na-placeholder` (`NA` by default).
 
 For example for `-o %(title)s-%(id)s.%(ext)s` and an mp4 video with title `youtube-dlc test video` and id `BaW_jenozKcj`, this will result in a `youtube-dlc test video-BaW_jenozKcj.mp4` file created in the current directory.
 
@@ -916,35 +941,35 @@ Format selectors can also be grouped using parentheses, for example if you want
 
 You can change the criteria for being considered the `best` by using `-S` (`--format-sort`). The general format for this is `--format-sort field1,field2...`. The available fields are:
 
- - `video`, `has_video`: Gives priority to formats that has a video stream
+ - `hasvid`: Gives priority to formats that has a video stream
- - `audio`, `has_audio`: Gives priority to formats that has a audio stream
+ - `hasaud`: Gives priority to formats that has a audio stream
- - `extractor`, `preference`, `extractor_preference`: The format preference as given by the extractor
+ - `ie_pref`: The format preference as given by the extractor
- - `lang`, `language_preference`: Language preference as given by the extractor
+ - `lang`: Language preference as given by the extractor
  - `quality`: The quality of the format. This is a metadata field available in some websites
- - `source`, `source_preference`: Preference of the source as given by the extractor
+ - `source`: Preference of the source as given by the extractor
- - `proto`, `protocol`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
+ - `proto`: Protocol used for download (`https`/`ftps` > `http`/`ftp` > `m3u8-native` > `m3u8` > `http-dash-segments` > other > `mms`/`rtsp` > unknown > `f4f`/`f4m`)
- - `vcodec`, `video_codec`: Video Codec (`vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
+ - `vcodec`: Video Codec (`vp9` > `h265` > `h264` > `vp8` > `h263` > `theora` > other > unknown)
- - `acodec`, `audio_codec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
+ - `acodec`: Audio Codec (`opus` > `vorbis` > `aac` > `mp4a` > `mp3` > `ac3` > `dts` > other > unknown)
  - `codec`: Equivalent to `vcodec,acodec`
- - `vext`, `video_ext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
+ - `vext`: Video Extension (`mp4` > `webm` > `flv` > other > unknown). If `--prefer-free-formats` is used, `webm` is prefered.
- - `aext`, `audio_ext`: Audio Extension (`m4a` > `aac` > `mp3` > `ogg` > `opus` > `webm` > other > unknown). If `--prefer-free-formats` is used, the order changes to `opus` > `ogg` > `webm` > `m4a` > `mp3` > `aac`.
+ - `aext`: Audio Extension (`m4a` > `aac` > `mp3` > `ogg` > `opus` > `webm` > other > unknown). If `--prefer-free-formats` is used, the order changes to `opus` > `ogg` > `webm` > `m4a` > `mp3` > `aac`.
- - `ext`, `extension`: Equivalent to `vext,aext`
+ - `ext`: Equivalent to `vext,aext`
  - `filesize`: Exact filesize, if know in advance. This will be unavailable for mu38 and DASH formats.
- - `filesize_approx`: Approximate filesize calculated from the manifests
+ - `fs_approx`: Approximate filesize calculated from the manifests
- - `size`, `filesize_estimate`: Exact filesize if available, otherwise approximate filesize
+ - `size`: Exact filesize if available, otherwise approximate filesize
  - `height`: Height of video
  - `width`: Width of video
- - `res`, `dimension`: Video resolution, calculated as the smallest dimension.
+ - `res`: Video resolution, calculated as the smallest dimension.
- - `fps`, `framerate`: Framerate of video
+ - `fps`: Framerate of video
- - `tbr`, `total_bitrate`: Total average bitrate in KBit/s
+ - `tbr`: Total average bitrate in KBit/s
- - `vbr`, `video_bitrate`: Average video bitrate in KBit/s
+ - `vbr`: Average video bitrate in KBit/s
- - `abr`, `audio_bitrate`: Average audio bitrate in KBit/s
+ - `abr`: Average audio bitrate in KBit/s
- - `br`, `bitrate`: Equivalent to using `tbr,vbr,abr`
+ - `br`: Equivalent to using `tbr,vbr,abr`
- - `samplerate`, `asr`: Audio sample rate in Hz
+ - `asr`: Audio sample rate in Hz
 
 Note that any other **numerical** field made available by the extractor can also be used. All fields, unless specified otherwise, are sorted in decending order. To reverse this, prefix the field with a `+`. Eg: `+res` prefers format with the smallest resolution. Additionally, you can suffix a prefered value for the fields, seperated by a `:`. Eg: `res:720` prefers larger videos, but no larger than 720p and the smallest video if there are no videos less than 720p. For `codec` and `ext`, you can provide two prefered values, the first for video and the second for audio. Eg: `+codec:avc:m4a` (equivalent to `+vcodec:avc,+acodec:m4a`) sets the video codec preference to `h264` > `h265` > `vp9` > `vp8` > `h263` > `theora` and audio codec preference to `mp4a` > `aac` > `vorbis` > `opus` > `mp3` > `ac3` > `dts`. You can also make the sorting prefer the nearest values to the provided by using `~` as the delimiter. Eg: `filesize~1G` prefers the format with filesize closest to 1 GiB.
 
-The fields `has_video`, `extractor`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec,size,br,asr,proto,ext,has_audio,source,format_id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
+The fields `hasvid`, `ie_pref`, `lang`, `quality` are always given highest priority in sorting, irrespective of the user-defined order. This behaviour can be changed by using `--force-format-sort`. Apart from these, the default order used is: `res,fps,codec,size,br,asr,proto,ext,hasaud,source,id`. Note that the extractors may override this default order, but they cannot override the user-provided order.
 
 If your format selector is `worst`, the last item is selected after sorting. This means it will select the format that is worst in all repects. Most of the time, what you actually want is the video with the smallest filesize instead. So it is generally better to use `-f best -S +size,+br,+res,+fps`.
 
@@ -983,7 +1008,7 @@ $ youtube-dlc -f 'wv*+wa/w'
 $ youtube-dlc -S '+res'
 
 # Download the smallest video available
-$ youtube-dlc -S '+size,+bitrate'
+$ youtube-dlc -S '+size,+br'
 
 
 
@@ -1031,7 +1056,7 @@ $ youtube-dlc -f '(bv*+ba/b)[protocol^=http][protocol!*=dash] / (bv*+ba/b)'
 
 # Download best video available via the best protocol
 # (https/ftps > http/ftp > m3u8_native > m3u8 > http_dash_segments ...)
-$ youtube-dlc -S 'protocol'
+$ youtube-dlc -S 'proto'
 
 
 
@@ -1067,9 +1092,11 @@ $ youtube-dlc -S 'res:720,fps'
 $ youtube-dlc -S '+res:480,codec,br'
 ```
 
+# PLUGINS
 
+Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`. Currently only `extractor` plugins are supported. Support for `downloader` and `postprocessor` plugins may be added in the future. See [ytdlp_plugins](ytdlp_plugins) for example.
 
-
+**Note**: `<root-dir>` is the directory of the binary (`<root-dir>/youtube-dlc`), or the root directory of the module if you are running directly from source-code ((`<root dir>/youtube_dlc/__main__.py`)
 
 # MORE
 For FAQ, Developer Instructions etc., see the [original README](https://github.com/ytdl-org/youtube-dl)

docs/supportedsites.md

@@ -47,12 +47,13 @@
  - **Amara**
  - **AMCNetworks**
  - **AmericasTestKitchen**
+ - **AmericasTestKitchenSeason**
  - **anderetijden**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
  - **AnimeLab**
  - **AnimeLabShows**
  - **AnimeOnDemand**
  - **Anvato**
- - **aol.com**
+ - **aol.com**: Yahoo screen and movies
  - **APA**
  - **Aparat**
  - **AppleConnect**
@@ -197,8 +198,6 @@
  - **CNNArticle**
  - **CNNBlogs**
  - **ComedyCentral**
- - **ComedyCentralFullEpisodes**
- - **ComedyCentralShortname**
  - **ComedyCentralTV**
  - **CondeNast**: Condé Nast media group: Allure, Architectural Digest, Ars Technica, Bon Appétit, Brides, Condé Nast, Condé Nast Traveler, Details, Epicurious, GQ, Glamour, Golf Digest, SELF, Teen Vogue, The New Yorker, Vanity Fair, Vogue, W Magazine, WIRED
  - **CONtv**
@@ -520,6 +519,12 @@
  - **Mgoon**
  - **MGTV**: 芒果TV
  - **MiaoPai**
+ - **mildom**: Record ongoing live by specific user in Mildom
+ - **mildom:user:vod**: Download all VODs from specific user in Mildom
+ - **mildom:vod**: Download a VOD in Mildom
+ - **minds**
+ - **minds:channel**
+ - **minds:group**
  - **MinistryGrid**
  - **Minoto**
  - **miomio.tv**
@@ -880,6 +885,8 @@
  - **Sport5**
  - **SportBox**
  - **SportDeutschland**
+ - **spotify**
+ - **spotify:show**
  - **Spreaker**
  - **SpreakerPage**
  - **SpreakerShow**
@@ -962,13 +969,13 @@
  - **TNAFlixNetworkEmbed**
  - **toggle**
  - **ToonGoggles**
- - **Tosh**: Tosh.0
  - **tou.tv**
  - **Toypics**: Toypics video
  - **ToypicsUser**: Toypics user profile
  - **TrailerAddict** (Currently broken)
  - **Trilulilu**
- - **TrovoLive**
+ - **Trovo**
+ - **TrovoVod**
  - **TruNews**
  - **TruTV**
  - **Tube8**

make_win.bat

@@ -1 +1 @@
-py -m PyInstaller youtube_dlc\__main__.py --onefile --name youtube-dlc --version-file win\ver.txt --icon win\icon\cloud.ico --upx-exclude=vcruntime140.dll
+py -m PyInstaller youtube_dlc\__main__.py --onefile --name youtube-dlc --version-file win\ver.txt --icon win\icon\cloud.ico --upx-exclude=vcruntime140.dll --exclude-module ytdlp_plugins

test/test_YoutubeDL.py

@@ -637,13 +637,20 @@ class TestYoutubeDL(unittest.TestCase):
             'title2': '%PATH%',
         }
 
-        def fname(templ):
+        def fname(templ, na_placeholder='NA'):
-            ydl = YoutubeDL({'outtmpl': templ})
+            params = {'outtmpl': templ}
+            if na_placeholder != 'NA':
+                params['outtmpl_na_placeholder'] = na_placeholder
+            ydl = YoutubeDL(params)
             return ydl.prepare_filename(info)
         self.assertEqual(fname('%(id)s.%(ext)s'), '1234.mp4')
         self.assertEqual(fname('%(id)s-%(width)s.%(ext)s'), '1234-NA.mp4')
-        # Replace missing fields with 'NA'
+        NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(id)s.%(ext)s'
-        self.assertEqual(fname('%(uploader_date)s-%(id)s.%(ext)s'), 'NA-1234.mp4')
+        # Replace missing fields with 'NA' by default
+        self.assertEqual(fname(NA_TEST_OUTTMPL), 'NA-NA-1234.mp4')
+        # Or by provided placeholder
+        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder='none'), 'none-none-1234.mp4')
+        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder=''), '--1234.mp4')
         self.assertEqual(fname('%(height)d.%(ext)s'), '1080.mp4')
         self.assertEqual(fname('%(height)6d.%(ext)s'), '  1080.mp4')
         self.assertEqual(fname('%(height)-6d.%(ext)s'), '1080  .mp4')

test/test_post_hooks.py

@@ -8,11 +8,11 @@ import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from test.helper import get_params, try_rm
-import youtube_dl.YoutubeDL
+import youtube_dlc.YoutubeDL
-from youtube_dl.utils import DownloadError
+from youtube_dlc.utils import DownloadError
 
 
-class YoutubeDL(youtube_dl.YoutubeDL):
+class YoutubeDL(youtube_dlc.YoutubeDL):
     def __init__(self, *args, **kwargs):
         super(YoutubeDL, self).__init__(*args, **kwargs)
         self.to_stderr = self.to_screen

youtube_dlc/YoutubeDL.py

@@ -69,6 +69,7 @@ from .utils import (
     iri_to_uri,
     ISO3166Utils,
     locked_file,
+    make_dir,
     make_HTTPS_handler,
     MaxDownloadsReached,
     orderedSet,
@@ -104,7 +105,7 @@ from .utils import (
     process_communicate_or_kill,
 )
 from .cache import Cache
-from .extractor import get_info_extractor, gen_extractor_classes, _LAZY_LOADER
+from .extractor import get_info_extractor, gen_extractor_classes, _LAZY_LOADER, _PLUGIN_CLASSES
 from .extractor.openload import PhantomJSwrapper
 from .downloader import get_suitable_downloader
 from .downloader.rtmp import rtmpdump_version
@@ -114,8 +115,9 @@ from .postprocessor import (
     FFmpegFixupStretchedPP,
     FFmpegMergerPP,
     FFmpegPostProcessor,
-    FFmpegSubtitlesConvertorPP,
+    # FFmpegSubtitlesConvertorPP,
     get_postprocessor,
+    MoveFilesAfterDownloadPP,
 )
 from .version import __version__
 
@@ -179,9 +181,12 @@ class YoutubeDL(object):
     allow_multiple_video_streams:   Allow multiple video streams to be merged into a single file
     allow_multiple_audio_streams:   Allow multiple audio streams to be merged into a single file
     outtmpl:           Template for output names.
-    restrictfilenames: Do not allow "&" and spaces in file names.
+    outtmpl_na_placeholder: Placeholder for unavailable meta fields.
-    trim_file_name:    Limit length of filename (extension excluded).
+    restrictfilenames: Do not allow "&" and spaces in file names
-    ignoreerrors:      Do not stop on download errors. (Default True when running youtube-dlc, but False when directly accessing YoutubeDL class)
+    trim_file_name:    Limit length of filename (extension excluded)
+    ignoreerrors:      Do not stop on download errors
+                       (Default True when running youtube-dlc,
+                       but False when directly accessing YoutubeDL class)
     force_generic_extractor: Force downloader to use the generic extractor
     overwrites:        Overwrite all video and metadata files if True,
                        overwrite only non-video files if None
@@ -257,6 +262,8 @@ class YoutubeDL(object):
     postprocessors:    A list of dictionaries, each with an entry
                        * key:  The name of the postprocessor. See
                                youtube_dlc/postprocessor/__init__.py for a list.
+                       * _after_move: Optional. If True, run this post_processor
+                               after 'MoveFilesAfterDownload'
                        as well as any further keyword arguments for the
                        postprocessor.
     post_hooks:        A list of functions that get called as the final step
@@ -369,6 +376,8 @@ class YoutubeDL(object):
     params = None
     _ies = []
     _pps = []
+    _pps_end = []
+    __prepare_filename_warned = False
     _download_retcode = None
     _num_downloads = None
     _playlist_level = 0
@@ -382,6 +391,8 @@ class YoutubeDL(object):
         self._ies = []
         self._ies_instances = {}
         self._pps = []
+        self._pps_end = []
+        self.__prepare_filename_warned = False
         self._post_hooks = []
         self._progress_hooks = []
         self._download_retcode = 0
@@ -483,8 +494,11 @@ class YoutubeDL(object):
             pp_class = get_postprocessor(pp_def_raw['key'])
             pp_def = dict(pp_def_raw)
             del pp_def['key']
+            after_move = pp_def.get('_after_move', False)
+            if '_after_move' in pp_def:
+                del pp_def['_after_move']
             pp = pp_class(self, **compat_kwargs(pp_def))
-            self.add_post_processor(pp)
+            self.add_post_processor(pp, after_move=after_move)
 
         for ph in self.params.get('post_hooks', []):
             self.add_post_hook(ph)
@@ -536,9 +550,12 @@ class YoutubeDL(object):
         for ie in gen_extractor_classes():
             self.add_info_extractor(ie)
 
-    def add_post_processor(self, pp):
+    def add_post_processor(self, pp, after_move=False):
         """Add a PostProcessor object to the end of the chain."""
-        self._pps.append(pp)
+        if after_move:
+            self._pps_end.append(pp)
+        else:
+            self._pps.append(pp)
         pp.set_downloader(self)
 
     def add_post_hook(self, ph):
@@ -599,7 +616,7 @@ class YoutubeDL(object):
                 # already of type unicode()
                 ctypes.windll.kernel32.SetConsoleTitleW(ctypes.c_wchar_p(message))
         elif 'TERM' in os.environ:
-            self._write_string('\033[0;%s\007' % message, self._screen_file)
+            self._write_string('\033]0;%s\007' % message, self._screen_file)
 
     def save_console_title(self):
         if not self.params.get('consoletitle', False):
@@ -702,7 +719,7 @@ class YoutubeDL(object):
         except UnicodeEncodeError:
             self.to_screen('Deleting already existent file')
 
-    def prepare_filename(self, info_dict):
+    def prepare_filename(self, info_dict, warn=False):
         """Generate the output filename."""
         try:
             template_dict = dict(info_dict)
@@ -727,7 +744,7 @@ class YoutubeDL(object):
             template_dict = dict((k, v if isinstance(v, compat_numeric_types) else sanitize(k, v))
                                  for k, v in template_dict.items()
                                  if v is not None and not isinstance(v, (list, tuple, dict)))
-            template_dict = collections.defaultdict(lambda: 'NA', template_dict)
+            template_dict = collections.defaultdict(lambda: self.params.get('outtmpl_na_placeholder', 'NA'), template_dict)
 
             outtmpl = self.params.get('outtmpl', DEFAULT_OUTTMPL)
 
@@ -747,8 +764,8 @@ class YoutubeDL(object):
 
             # Missing numeric fields used together with integer presentation types
             # in format specification will break the argument substitution since
-            # string 'NA' is returned for missing fields. We will patch output
+            # string NA placeholder is returned for missing fields. We will patch
-            # template for missing fields to meet string presentation type.
+            # output template for missing fields to meet string presentation type.
             for numeric_field in self._NUMERIC_FIELDS:
                 if numeric_field not in template_dict:
                     # As of [1] format syntax is:
@@ -796,11 +813,33 @@ class YoutubeDL(object):
             # to workaround encoding issues with subprocess on python2 @ Windows
             if sys.version_info < (3, 0) and sys.platform == 'win32':
                 filename = encodeFilename(filename, True).decode(preferredencoding())
-            return sanitize_path(filename)
+            filename = sanitize_path(filename)
+
+            if warn and not self.__prepare_filename_warned:
+                if not self.params.get('paths'):
+                    pass
+                elif filename == '-':
+                    self.report_warning('--paths is ignored when an outputting to stdout')
+                elif os.path.isabs(filename):
+                    self.report_warning('--paths is ignored since an absolute path is given in output template')
+                self.__prepare_filename_warned = True
+
+            return filename
         except ValueError as err:
             self.report_error('Error in output template: ' + str(err) + ' (encoding: ' + repr(preferredencoding()) + ')')
             return None
 
+    def prepare_filepath(self, filename, dir_type=''):
+        if filename == '-':
+            return filename
+        paths = self.params.get('paths', {})
+        assert isinstance(paths, dict)
+        homepath = expand_path(paths.get('home', '').strip())
+        assert isinstance(homepath, compat_str)
+        subdir = expand_path(paths.get(dir_type, '').strip()) if dir_type else ''
+        assert isinstance(subdir, compat_str)
+        return sanitize_path(os.path.join(homepath, subdir, filename))
+
     def _match_entry(self, info_dict, incomplete):
         """ Returns None if the file should be downloaded """
 
@@ -885,7 +924,9 @@ class YoutubeDL(object):
                                     'and will probably not work.')
 
             try:
-                temp_id = ie.extract_id(url) if callable(getattr(ie, 'extract_id', None)) else ie._match_id(url)
+                temp_id = str_or_none(
+                    ie.extract_id(url) if callable(getattr(ie, 'extract_id', None))
+                    else ie._match_id(url))
             except (AssertionError, IndexError, AttributeError):
                 temp_id = None
             if temp_id is not None and self.in_download_archive({'id': temp_id, 'ie_key': ie_key}):
@@ -970,7 +1011,8 @@ class YoutubeDL(object):
             if ((extract_flat == 'in_playlist' and 'playlist' in extra_info)
                     or extract_flat is True):
                 self.__forced_printings(
-                    ie_result, self.prepare_filename(ie_result),
+                    ie_result,
+                    self.prepare_filepath(self.prepare_filename(ie_result)),
                     incomplete=True)
                 return ie_result
 
@@ -1888,6 +1930,8 @@ class YoutubeDL(object):
 
         assert info_dict.get('_type', 'video') == 'video'
 
+        info_dict.setdefault('__postprocessors', [])
+
         max_downloads = self.params.get('max_downloads')
         if max_downloads is not None:
             if self._num_downloads >= int(max_downloads):
@@ -1904,10 +1948,13 @@ class YoutubeDL(object):
 
         self._num_downloads += 1
 
-        info_dict['_filename'] = filename = self.prepare_filename(info_dict)
+        filename = self.prepare_filename(info_dict, warn=True)
+        info_dict['_filename'] = full_filename = self.prepare_filepath(filename)
+        temp_filename = self.prepare_filepath(filename, 'temp')
+        files_to_move = {}
 
         # Forced printings
-        self.__forced_printings(info_dict, filename, incomplete=False)
+        self.__forced_printings(info_dict, full_filename, incomplete=False)
 
         if self.params.get('simulate', False):
             if self.params.get('force_write_download_archive', False):
@@ -1920,20 +1967,19 @@ class YoutubeDL(object):
             return
 
         def ensure_dir_exists(path):
-            try:
+            return make_dir(path, self.report_error)
-                dn = os.path.dirname(path)
-                if dn and not os.path.exists(dn):
-                    os.makedirs(dn)
-                return True
-            except (OSError, IOError) as err:
-                self.report_error('unable to create directory ' + error_to_compat_str(err))
-                return False
 
-        if not ensure_dir_exists(sanitize_path(encodeFilename(filename))):
+        if not ensure_dir_exists(encodeFilename(full_filename)):
+            return
+        if not ensure_dir_exists(encodeFilename(temp_filename)):
             return
 
         if self.params.get('writedescription', False):
-            descfn = replace_extension(filename, 'description', info_dict.get('ext'))
+            descfn = replace_extension(
+                self.prepare_filepath(filename, 'description'),
+                'description', info_dict.get('ext'))
+            if not ensure_dir_exists(encodeFilename(descfn)):
+                return
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(descfn)):
                 self.to_screen('[info] Video description is already present')
             elif info_dict.get('description') is None:
@@ -1948,7 +1994,11 @@ class YoutubeDL(object):
                     return
 
         if self.params.get('writeannotations', False):
-            annofn = replace_extension(filename, 'annotations.xml', info_dict.get('ext'))
+            annofn = replace_extension(
+                self.prepare_filepath(filename, 'annotation'),
+                'annotations.xml', info_dict.get('ext'))
+            if not ensure_dir_exists(encodeFilename(annofn)):
+                return
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(annofn)):
                 self.to_screen('[info] Video annotations are already present')
             elif not info_dict.get('annotations'):
@@ -1982,9 +2032,13 @@ class YoutubeDL(object):
             # ie = self.get_info_extractor(info_dict['extractor_key'])
             for sub_lang, sub_info in subtitles.items():
                 sub_format = sub_info['ext']
-                sub_filename = subtitles_filename(filename, sub_lang, sub_format, info_dict.get('ext'))
+                sub_filename = subtitles_filename(temp_filename, sub_lang, sub_format, info_dict.get('ext'))
+                sub_filename_final = subtitles_filename(
+                    self.prepare_filepath(filename, 'subtitle'),
+                    sub_lang, sub_format, info_dict.get('ext'))
                 if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(sub_filename)):
                     self.to_screen('[info] Video subtitle %s.%s is already present' % (sub_lang, sub_format))
+                    files_to_move[sub_filename] = sub_filename_final
                 else:
                     self.to_screen('[info] Writing video subtitles to: ' + sub_filename)
                     if sub_info.get('data') is not None:
@@ -1993,6 +2047,7 @@ class YoutubeDL(object):
                             # See https://github.com/ytdl-org/youtube-dl/issues/10268
                             with io.open(encodeFilename(sub_filename), 'w', encoding='utf-8', newline='') as subfile:
                                 subfile.write(sub_info['data'])
+                            files_to_move[sub_filename] = sub_filename_final
                         except (OSError, IOError):
                             self.report_error('Cannot write subtitles file ' + sub_filename)
                             return
@@ -2008,6 +2063,7 @@ class YoutubeDL(object):
                                 with io.open(encodeFilename(sub_filename), 'wb') as subfile:
                                     subfile.write(sub_data)
                             '''
+                            files_to_move[sub_filename] = sub_filename_final
                         except (ExtractorError, IOError, OSError, ValueError, compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
                             self.report_warning('Unable to download subtitle for "%s": %s' %
                                                 (sub_lang, error_to_compat_str(err)))
@@ -2015,29 +2071,32 @@ class YoutubeDL(object):
 
         if self.params.get('skip_download', False):
             if self.params.get('convertsubtitles', False):
-                subconv = FFmpegSubtitlesConvertorPP(self, format=self.params.get('convertsubtitles'))
+                # subconv = FFmpegSubtitlesConvertorPP(self, format=self.params.get('convertsubtitles'))
                 filename_real_ext = os.path.splitext(filename)[1][1:]
                 filename_wo_ext = (
-                    os.path.splitext(filename)[0]
+                    os.path.splitext(full_filename)[0]
                     if filename_real_ext == info_dict['ext']
-                    else filename)
+                    else full_filename)
                 afilename = '%s.%s' % (filename_wo_ext, self.params.get('convertsubtitles'))
-                if subconv.available:
+                # if subconv.available:
-                    info_dict.setdefault('__postprocessors', [])
+                #     info_dict['__postprocessors'].append(subconv)
-                    # info_dict['__postprocessors'].append(subconv)
                 if os.path.exists(encodeFilename(afilename)):
                     self.to_screen(
                         '[download] %s has already been downloaded and '
                         'converted' % afilename)
                 else:
                     try:
-                        self.post_process(filename, info_dict)
+                        self.post_process(full_filename, info_dict, files_to_move)
                     except (PostProcessingError) as err:
                         self.report_error('postprocessing: %s' % str(err))
                         return
 
         if self.params.get('writeinfojson', False):
-            infofn = replace_extension(filename, 'info.json', info_dict.get('ext'))
+            infofn = replace_extension(
+                self.prepare_filepath(filename, 'infojson'),
+                'info.json', info_dict.get('ext'))
+            if not ensure_dir_exists(encodeFilename(infofn)):
+                return
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(infofn)):
                 self.to_screen('[info] Video description metadata is already present')
             else:
@@ -2048,7 +2107,9 @@ class YoutubeDL(object):
                     self.report_error('Cannot write metadata to JSON file ' + infofn)
                     return
 
-        self._write_thumbnails(info_dict, filename)
+        thumbdir = os.path.dirname(self.prepare_filepath(filename, 'thumbnail'))
+        for thumbfn in self._write_thumbnails(info_dict, temp_filename):
+            files_to_move[thumbfn] = os.path.join(thumbdir, os.path.basename(thumbfn))
 
         # Write internet shortcut files
         url_link = webloc_link = desktop_link = False
@@ -2073,8 +2134,8 @@ class YoutubeDL(object):
             ascii_url = iri_to_uri(info_dict['webpage_url'])
 
         def _write_link_file(extension, template, newline, embed_filename):
-            linkfn = replace_extension(filename, extension, info_dict.get('ext'))
+            linkfn = replace_extension(full_filename, extension, info_dict.get('ext'))
-            if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(linkfn)):
+            if self.params.get('overwrites', True) and os.path.exists(encodeFilename(linkfn)):
                 self.to_screen('[info] Internet shortcut is already present')
             else:
                 try:
@@ -2103,9 +2164,27 @@ class YoutubeDL(object):
         must_record_download_archive = False
         if not self.params.get('skip_download', False):
             try:
+
+                def existing_file(filename, temp_filename):
+                    file_exists = os.path.exists(encodeFilename(filename))
+                    tempfile_exists = (
+                        False if temp_filename == filename
+                        else os.path.exists(encodeFilename(temp_filename)))
+                    if not self.params.get('overwrites', False) and (file_exists or tempfile_exists):
+                        existing_filename = temp_filename if tempfile_exists else filename
+                        self.to_screen('[download] %s has already been downloaded and merged' % existing_filename)
+                        return existing_filename
+                    if tempfile_exists:
+                        self.report_file_delete(temp_filename)
+                        os.remove(encodeFilename(temp_filename))
+                    if file_exists:
+                        self.report_file_delete(filename)
+                        os.remove(encodeFilename(filename))
+                    return None
+
+                success = True
                 if info_dict.get('requested_formats') is not None:
                     downloaded = []
-                    success = True
                     merger = FFmpegMergerPP(self)
                     if not merger.available:
                         postprocessors = []
@@ -2134,32 +2213,31 @@ class YoutubeDL(object):
                         # TODO: Check acodec/vcodec
                         return False
 
-                    filename_real_ext = os.path.splitext(filename)[1][1:]
-                    filename_wo_ext = (
-                        os.path.splitext(filename)[0]
-                        if filename_real_ext == info_dict['ext']
-                        else filename)
                     requested_formats = info_dict['requested_formats']
+                    old_ext = info_dict['ext']
                     if self.params.get('merge_output_format') is None and not compatible_formats(requested_formats):
                         info_dict['ext'] = 'mkv'
                         self.report_warning(
                             'Requested formats are incompatible for merge and will be merged into mkv.')
+
+                    def correct_ext(filename):
+                        filename_real_ext = os.path.splitext(filename)[1][1:]
+                        filename_wo_ext = (
+                            os.path.splitext(filename)[0]
+                            if filename_real_ext == old_ext
+                            else filename)
+                        return '%s.%s' % (filename_wo_ext, info_dict['ext'])
+
                     # Ensure filename always has a correct extension for successful merge
-                    filename = '%s.%s' % (filename_wo_ext, info_dict['ext'])
+                    full_filename = correct_ext(full_filename)
-                    file_exists = os.path.exists(encodeFilename(filename))
+                    temp_filename = correct_ext(temp_filename)
-                    if not self.params.get('overwrites', False) and file_exists:
+                    dl_filename = existing_file(full_filename, temp_filename)
-                        self.to_screen(
+                    if dl_filename is None:
-                            '[download] %s has already been downloaded and '
-                            'merged' % filename)
-                    else:
-                        if file_exists:
-                            self.report_file_delete(filename)
-                            os.remove(encodeFilename(filename))
                         for f in requested_formats:
                             new_info = dict(info_dict)
                             new_info.update(f)
                             fname = prepend_extension(
-                                self.prepare_filename(new_info),
+                                self.prepare_filepath(self.prepare_filename(new_info), 'temp'),
                                 'f%s' % f['format_id'], new_info['ext'])
                             if not ensure_dir_exists(fname):
                                 return
@@ -2171,14 +2249,15 @@ class YoutubeDL(object):
                         # Even if there were no downloads, it is being merged only now
                         info_dict['__real_download'] = True
                 else:
-                    # Delete existing file with --yes-overwrites
-                    if self.params.get('overwrites', False):
-                        if os.path.exists(encodeFilename(filename)):
-                            self.report_file_delete(filename)
-                            os.remove(encodeFilename(filename))
                     # Just a single file
-                    success, real_download = dl(filename, info_dict)
+                    dl_filename = existing_file(full_filename, temp_filename)
-                    info_dict['__real_download'] = real_download
+                    if dl_filename is None:
+                        success, real_download = dl(temp_filename, info_dict)
+                        info_dict['__real_download'] = real_download
+
+                dl_filename = dl_filename or temp_filename
+                info_dict['__finaldir'] = os.path.dirname(os.path.abspath(encodeFilename(full_filename)))
+
             except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
                 self.report_error('unable to download video data: %s' % error_to_compat_str(err))
                 return
@@ -2204,7 +2283,6 @@ class YoutubeDL(object):
                     elif fixup_policy == 'detect_or_warn':
                         stretched_pp = FFmpegFixupStretchedPP(self)
                         if stretched_pp.available:
-                            info_dict.setdefault('__postprocessors', [])
                             info_dict['__postprocessors'].append(stretched_pp)
                         else:
                             self.report_warning(
@@ -2223,7 +2301,6 @@ class YoutubeDL(object):
                     elif fixup_policy == 'detect_or_warn':
                         fixup_pp = FFmpegFixupM4aPP(self)
                         if fixup_pp.available:
-                            info_dict.setdefault('__postprocessors', [])
                             info_dict['__postprocessors'].append(fixup_pp)
                         else:
                             self.report_warning(
@@ -2242,7 +2319,6 @@ class YoutubeDL(object):
                     elif fixup_policy == 'detect_or_warn':
                         fixup_pp = FFmpegFixupM3u8PP(self)
                         if fixup_pp.available:
-                            info_dict.setdefault('__postprocessors', [])
                             info_dict['__postprocessors'].append(fixup_pp)
                         else:
                             self.report_warning(
@@ -2252,13 +2328,13 @@ class YoutubeDL(object):
                         assert fixup_policy in ('ignore', 'never')
 
                 try:
-                    self.post_process(filename, info_dict)
+                    self.post_process(dl_filename, info_dict, files_to_move)
                 except (PostProcessingError) as err:
                     self.report_error('postprocessing: %s' % str(err))
                     return
                 try:
                     for ph in self._post_hooks:
-                        ph(filename)
+                        ph(full_filename)
                 except Exception as err:
                     self.report_error('post hooks: %s' % str(err))
                     return
@@ -2324,27 +2400,41 @@ class YoutubeDL(object):
             (k, v) for k, v in info_dict.items()
             if k not in ['requested_formats', 'requested_subtitles'])
 
-    def post_process(self, filename, ie_info):
+    def post_process(self, filename, ie_info, files_to_move={}):
         """Run all the postprocessors on the given file."""
         info = dict(ie_info)
         info['filepath'] = filename
-        pps_chain = []
+
-        if ie_info.get('__postprocessors') is not None:
+        def run_pp(pp):
-            pps_chain.extend(ie_info['__postprocessors'])
-        pps_chain.extend(self._pps)
-        for pp in pps_chain:
             files_to_delete = []
+            infodict = info
             try:
-                files_to_delete, info = pp.run(info)
+                files_to_delete, infodict = pp.run(infodict)
             except PostProcessingError as e:
                 self.report_error(e.msg)
-            if files_to_delete and not self.params.get('keepvideo', False):
+            if not files_to_delete:
+                return infodict
+
+            if self.params.get('keepvideo', False):
+                for f in files_to_delete:
+                    files_to_move.setdefault(f, '')
+            else:
                 for old_filename in set(files_to_delete):
                     self.to_screen('Deleting original file %s (pass -k to keep)' % old_filename)
                     try:
                         os.remove(encodeFilename(old_filename))
                     except (IOError, OSError):
                         self.report_warning('Unable to remove downloaded original file')
+                    if old_filename in files_to_move:
+                        del files_to_move[old_filename]
+            return infodict
+
+        for pp in ie_info.get('__postprocessors', []) + self._pps:
+            info = run_pp(pp)
+        info = run_pp(MoveFilesAfterDownloadPP(self, files_to_move))
+        files_to_move = {}
+        for pp in self._pps_end:
+            info = run_pp(pp)
 
     def _make_archive_id(self, info_dict):
         video_id = info_dict.get('id')
@@ -2364,7 +2454,7 @@ class YoutubeDL(object):
                     break
             else:
                 return
-        return extractor.lower() + ' ' + video_id
+        return '%s %s' % (extractor.lower(), video_id)
 
     def in_download_archive(self, info_dict):
         fn = self.params.get('download_archive')
@@ -2565,9 +2655,12 @@ class YoutubeDL(object):
                 self.get_encoding()))
         write_string(encoding_str, encoding=None)
 
-        self._write_string('[debug] yt-dlp version ' + __version__ + '\n')
+        self._write_string('[debug] yt-dlp version %s\n' % __version__)
         if _LAZY_LOADER:
-            self._write_string('[debug] Lazy loading extractors enabled' + '\n')
+            self._write_string('[debug] Lazy loading extractors enabled\n')
+        if _PLUGIN_CLASSES:
+            self._write_string(
+                '[debug] Plugin Extractors: %s\n' % [ie.ie_key() for ie in _PLUGIN_CLASSES])
         try:
             sp = subprocess.Popen(
                 ['git', 'rev-parse', '--short', 'HEAD'],
@@ -2576,7 +2669,7 @@ class YoutubeDL(object):
             out, err = process_communicate_or_kill(sp)
             out = out.decode().strip()
             if re.match('[0-9a-f]+', out):
-                self._write_string('[debug] Git HEAD: ' + out + '\n')
+                self._write_string('[debug] Git HEAD: %s\n' % out)
         except Exception:
             try:
                 sys.exc_clear()
@@ -2698,14 +2791,11 @@ class YoutubeDL(object):
             if thumbnails:
                 thumbnails = [thumbnails[-1]]
         elif self.params.get('write_all_thumbnails', False):
-            thumbnails = info_dict.get('thumbnails')
+            thumbnails = info_dict.get('thumbnails') or []
         else:
-            return
+            thumbnails = []
-
-        if not thumbnails:
-            # No thumbnails present, so return immediately
-            return
 
+        ret = []
         for t in thumbnails:
             thumb_ext = determine_ext(t['url'], 'jpg')
             suffix = '_%s' % t['id'] if len(thumbnails) > 1 else ''
@@ -2713,6 +2803,7 @@ class YoutubeDL(object):
             t['filename'] = thumb_filename = replace_extension(filename + suffix, thumb_ext, info_dict.get('ext'))
 
             if not self.params.get('overwrites', True) and os.path.exists(encodeFilename(thumb_filename)):
+                ret.append(thumb_filename)
                 self.to_screen('[%s] %s: Thumbnail %sis already present' %
                                (info_dict['extractor'], info_dict['id'], thumb_display_id))
             else:
@@ -2722,8 +2813,10 @@ class YoutubeDL(object):
                     uf = self.urlopen(t['url'])
                     with open(encodeFilename(thumb_filename), 'wb') as thumbf:
                         shutil.copyfileobj(uf, thumbf)
+                    ret.append(thumb_filename)
                     self.to_screen('[%s] %s: Writing thumbnail %sto: %s' %
                                    (info_dict['extractor'], info_dict['id'], thumb_display_id, thumb_filename))
                 except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
                     self.report_warning('Unable to download thumbnail "%s": %s' %
                                         (t['url'], error_to_compat_str(err)))
+        return ret

youtube_dlc/__init__.py

@@ -18,7 +18,6 @@ from .options import (
 )
 from .compat import (
     compat_getpass,
-    compat_shlex_split,
     workaround_optparse_bug9161,
 )
 from .utils import (
@@ -70,14 +69,7 @@ def _real_main(argv=None):
         std_headers['Referer'] = opts.referer
 
     # Custom HTTP headers
-    if opts.headers is not None:
+    std_headers.update(opts.headers)
-        for h in opts.headers:
-            if ':' not in h:
-                parser.error('wrong header formatting, it should be key:value, not "%s"' % h)
-            key, value = h.split(':', 1)
-            if opts.verbose:
-                write_string('[debug] Adding header from command line option %s:%s\n' % (key, value))
-            std_headers[key] = value
 
     # Dump user agent
     if opts.dump_user_agent:
@@ -252,6 +244,7 @@ def _real_main(argv=None):
         parser.error('Cannot download a video and extract audio into the same'
                      ' file! Use "{0}.%(ext)s" instead of "{0}" as the output'
                      ' template'.format(outtmpl))
+
     for f in opts.format_sort:
         if re.match(InfoExtractor.FormatSort.regex, f) is None:
             parser.error('invalid format sort string "%s" specified' % f)
@@ -326,32 +319,22 @@ def _real_main(argv=None):
             'force': opts.sponskrub_force,
             'ignoreerror': opts.sponskrub is None,
         })
-    # Please keep ExecAfterDownload towards the bottom as it allows the user to modify the final file in any way.
+    # ExecAfterDownload must be the last PP
-    # So if the user is able to remove the file before your postprocessor runs it might cause a few problems.
     if opts.exec_cmd:
         postprocessors.append({
             'key': 'ExecAfterDownload',
             'exec_cmd': opts.exec_cmd,
+            '_after_move': True
         })
-    external_downloader_args = None
-    if opts.external_downloader_args:
-        external_downloader_args = compat_shlex_split(opts.external_downloader_args)
 
-    postprocessor_args = {}
+    _args_compat_warning = 'WARNING: %s given without specifying name. The arguments will be given to all %s\n'
-    if opts.postprocessor_args is not None:
+    if 'default' in opts.external_downloader_args:
-        for string in opts.postprocessor_args:
+        write_string(_args_compat_warning % ('--external-downloader-args', 'external downloaders'), out=sys.stderr),
-            mobj = re.match(r'(?P<pp>\w+(?:\+\w+)?):(?P<args>.*)$', string)
+
-            if mobj is None:
+    if 'default-compat' in opts.postprocessor_args and 'default' not in opts.postprocessor_args:
-                if 'sponskrub' not in postprocessor_args:  # for backward compatibility
+        write_string(_args_compat_warning % ('--post-processor-args', 'post-processors'), out=sys.stderr),
-                    postprocessor_args['sponskrub'] = []
+        opts.postprocessor_args.setdefault('sponskrub', [])
-                    if opts.verbose:
+        opts.postprocessor_args['default'] = opts.postprocessor_args['default-compat']
-                        write_string('[debug] Adding postprocessor args from command line option sponskrub: \n')
-                pp_key, pp_args = 'default', string
-            else:
-                pp_key, pp_args = mobj.group('pp').lower(), mobj.group('args')
-            if opts.verbose:
-                write_string('[debug] Adding postprocessor args from command line option %s: %s\n' % (pp_key, pp_args))
-            postprocessor_args[pp_key] = compat_shlex_split(pp_args)
 
     match_filter = (
         None if opts.match_filter is None
@@ -390,6 +373,8 @@ def _real_main(argv=None):
         'listformats': opts.listformats,
         'listformats_table': opts.listformats_table,
         'outtmpl': outtmpl,
+        'outtmpl_na_placeholder': opts.outtmpl_na_placeholder,
+        'paths': opts.paths,
         'autonumber_size': opts.autonumber_size,
         'autonumber_start': opts.autonumber_start,
         'restrictfilenames': opts.restrictfilenames,
@@ -485,8 +470,8 @@ def _real_main(argv=None):
         'ffmpeg_location': opts.ffmpeg_location,
         'hls_prefer_native': opts.hls_prefer_native,
         'hls_use_mpegts': opts.hls_use_mpegts,
-        'external_downloader_args': external_downloader_args,
+        'external_downloader_args': opts.external_downloader_args,
-        'postprocessor_args': postprocessor_args,
+        'postprocessor_args': opts.postprocessor_args,
         'cn_verification_proxy': opts.cn_verification_proxy,
         'geo_verification_proxy': opts.geo_verification_proxy,
         'config_location': opts.config_location,

youtube_dlc/downloader/external.py

@@ -95,7 +95,8 @@ class ExternalFD(FileDownloader):
         return cli_valueless_option(self.params, command_option, param, expected_value)
 
     def _configuration_args(self, default=[]):
-        return cli_configuration_args(self.params, 'external_downloader_args', default)
+        return cli_configuration_args(
+            self.params, 'external_downloader_args', self.get_basename(), default)[0]
 
     def _call_downloader(self, tmpfilename, info_dict):
         """ Either overwrite this or implement _make_cmd """

youtube_dlc/extractor/__init__.py

@@ -1,13 +1,19 @@
 from __future__ import unicode_literals
 
+from ..utils import load_plugins
+
 try:
     from .lazy_extractors import *
     from .lazy_extractors import _ALL_CLASSES
     _LAZY_LOADER = True
+    _PLUGIN_CLASSES = []
+
 except ImportError:
     _LAZY_LOADER = False
     from .extractors import *
 
+    _PLUGIN_CLASSES = load_plugins('extractor', 'IE', globals())
+
     _ALL_CLASSES = [
         klass
         for name, klass in globals().items()

youtube_dlc/extractor/aenetworks.py

@@ -256,7 +256,7 @@ class AENetworksShowIE(AENetworksListBaseIE):
             'title': 'Ancient Aliens',
             'description': 'md5:3f6d74daf2672ff3ae29ed732e37ea7f',
         },
-        'playlist_mincount': 168,
+        'playlist_mincount': 150,
     }]
     _RESOURCE = 'series'
     _ITEMS_KEY = 'episodes'

youtube_dlc/extractor/aljazeera.py

@@ -1,13 +1,16 @@
 from __future__ import unicode_literals
 
+import json
+import re
+
 from .common import InfoExtractor
 
 
 class AlJazeeraIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?aljazeera\.com/(?:programmes|video)/.*?/(?P<id>[^/]+)\.html'
+    _VALID_URL = r'https?://(?:www\.)?aljazeera\.com/(?P<type>program/[^/]+|(?:feature|video)s)/\d{4}/\d{1,2}/\d{1,2}/(?P<id>[^/?&#]+)'
 
     _TESTS = [{
-        'url': 'http://www.aljazeera.com/programmes/the-slum/2014/08/deliverance-201482883754237240.html',
+        'url': 'https://www.aljazeera.com/program/episode/2014/9/19/deliverance',
         'info_dict': {
             'id': '3792260579001',
             'ext': 'mp4',
@@ -20,14 +23,34 @@ class AlJazeeraIE(InfoExtractor):
         'add_ie': ['BrightcoveNew'],
         'skip': 'Not accessible from Travis CI server',
     }, {
-        'url': 'http://www.aljazeera.com/video/news/2017/05/sierra-leone-709-carat-diamond-auctioned-170511100111930.html',
+        'url': 'https://www.aljazeera.com/videos/2017/5/11/sierra-leone-709-carat-diamond-to-be-auctioned-off',
+        'only_matching': True,
+    }, {
+        'url': 'https://www.aljazeera.com/features/2017/8/21/transforming-pakistans-buses-into-art',
         'only_matching': True,
     }]
-    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/665003303001/default_default/index.html?videoId=%s'
+    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_default/index.html?videoId=%s'
 
     def _real_extract(self, url):
-        program_name = self._match_id(url)
+        post_type, name = re.match(self._VALID_URL, url).groups()
-        webpage = self._download_webpage(url, program_name)
+        post_type = {
-        brightcove_id = self._search_regex(
+            'features': 'post',
-            r'RenderPagesVideo\(\'(.+?)\'', webpage, 'brightcove id')
+            'program': 'episode',
-        return self.url_result(self.BRIGHTCOVE_URL_TEMPLATE % brightcove_id, 'BrightcoveNew', brightcove_id)
+            'videos': 'video',
+        }[post_type.split('/')[0]]
+        video = self._download_json(
+            'https://www.aljazeera.com/graphql', name, query={
+                'operationName': 'SingleArticleQuery',
+                'variables': json.dumps({
+                    'name': name,
+                    'postType': post_type,
+                }),
+            }, headers={
+                'wp-site': 'aje',
+            })['data']['article']['video']
+        video_id = video['id']
+        account_id = video.get('accountId') or '665003303001'
+        player_id = video.get('playerId') or 'BkeSH5BDb'
+        return self.url_result(
+            self.BRIGHTCOVE_URL_TEMPLATE % (account_id, player_id, video_id),
+            'BrightcoveNew', video_id)

youtube_dlc/extractor/americastestkitchen.py

@@ -1,13 +1,16 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import json
 import re
 
 from .common import InfoExtractor
 from ..utils import (
     clean_html,
+    int_or_none,
     try_get,
     unified_strdate,
+    unified_timestamp,
 )
 
 
@@ -22,8 +25,8 @@ class AmericasTestKitchenIE(InfoExtractor):
             'ext': 'mp4',
             'description': 'md5:64e606bfee910627efc4b5f050de92b3',
             'thumbnail': r're:^https?://',
-            'timestamp': 1523664000,
+            'timestamp': 1523318400,
-            'upload_date': '20180414',
+            'upload_date': '20180410',
             'release_date': '20180410',
             'series': "America's Test Kitchen",
             'season_number': 18,
@@ -33,6 +36,27 @@ class AmericasTestKitchenIE(InfoExtractor):
         'params': {
             'skip_download': True,
         },
+    }, {
+        # Metadata parsing behaves differently for newer episodes (705) as opposed to older episodes (582 above)
+        'url': 'https://www.americastestkitchen.com/episode/705-simple-chicken-dinner',
+        'md5': '06451608c57651e985a498e69cec17e5',
+        'info_dict': {
+            'id': '5fbe8c61bda2010001c6763b',
+            'title': 'Simple Chicken Dinner',
+            'ext': 'mp4',
+            'description': 'md5:eb68737cc2fd4c26ca7db30139d109e7',
+            'thumbnail': r're:^https?://',
+            'timestamp': 1610755200,
+            'upload_date': '20210116',
+            'release_date': '20210116',
+            'series': "America's Test Kitchen",
+            'season_number': 21,
+            'episode': 'Simple Chicken Dinner',
+            'episode_number': 3,
+        },
+        'params': {
+            'skip_download': True,
+        },
     }, {
         'url': 'https://www.americastestkitchen.com/videos/3420-pan-seared-salmon',
         'only_matching': True,
@@ -60,7 +84,76 @@ class AmericasTestKitchenIE(InfoExtractor):
             'url': 'https://player.zype.com/embed/%s.js?api_key=jZ9GUhRmxcPvX7M3SlfejB6Hle9jyHTdk2jVxG7wOHPLODgncEKVdPYBhuz9iWXQ' % video['zypeId'],
             'ie_key': 'Zype',
             'description': clean_html(video.get('description')),
+            'timestamp': unified_timestamp(video.get('publishDate')),
             'release_date': unified_strdate(video.get('publishDate')),
+            'episode_number': int_or_none(episode.get('number')),
+            'season_number': int_or_none(episode.get('season')),
             'series': try_get(episode, lambda x: x['show']['title']),
             'episode': episode.get('title'),
         }
+
+
+class AmericasTestKitchenSeasonIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?(?P<show>americastestkitchen|cookscountry)\.com/episodes/browse/season_(?P<id>\d+)'
+    _TESTS = [{
+        # ATK Season
+        'url': 'https://www.americastestkitchen.com/episodes/browse/season_1',
+        'info_dict': {
+            'id': 'season_1',
+            'title': 'Season 1',
+        },
+        'playlist_count': 13,
+    }, {
+        # Cooks Country Season
+        'url': 'https://www.cookscountry.com/episodes/browse/season_12',
+        'info_dict': {
+            'id': 'season_12',
+            'title': 'Season 12',
+        },
+        'playlist_count': 13,
+    }]
+
+    def _real_extract(self, url):
+        show_name, season_number = re.match(self._VALID_URL, url).groups()
+        season_number = int(season_number)
+
+        slug = 'atk' if show_name == 'americastestkitchen' else 'cco'
+
+        season = 'Season %d' % season_number
+
+        season_search = self._download_json(
+            'https://y1fnzxui30-dsn.algolia.net/1/indexes/everest_search_%s_season_desc_production' % slug,
+            season, headers={
+                'Origin': 'https://www.%s.com' % show_name,
+                'X-Algolia-API-Key': '8d504d0099ed27c1b73708d22871d805',
+                'X-Algolia-Application-Id': 'Y1FNZXUI30',
+            }, query={
+                'facetFilters': json.dumps([
+                    'search_season_list:' + season,
+                    'search_document_klass:episode',
+                    'search_show_slug:' + slug,
+                ]),
+                'attributesToRetrieve': 'description,search_%s_episode_number,search_document_date,search_url,title' % slug,
+                'attributesToHighlight': '',
+                'hitsPerPage': 1000,
+            })
+
+        def entries():
+            for episode in (season_search.get('hits') or []):
+                search_url = episode.get('search_url')
+                if not search_url:
+                    continue
+                yield {
+                    '_type': 'url',
+                    'url': 'https://www.%s.com%s' % (show_name, search_url),
+                    'id': try_get(episode, lambda e: e['objectID'].split('_')[-1]),
+                    'title': episode.get('title'),
+                    'description': episode.get('description'),
+                    'timestamp': unified_timestamp(episode.get('search_document_date')),
+                    'season_number': season_number,
+                    'episode_number': int_or_none(episode.get('search_%s_episode_number' % slug)),
+                    'ie_key': AmericasTestKitchenIE.ie_key(),
+                }
+
+        return self.playlist_result(
+            entries(), 'season_%d' % season_number, season)

youtube_dlc/extractor/aol.py

@@ -3,7 +3,7 @@ from __future__ import unicode_literals
 
 import re
 
-from .common import InfoExtractor
+from .yahoo import YahooIE
 from ..compat import (
     compat_parse_qs,
     compat_urllib_parse_urlparse,
@@ -15,9 +15,9 @@ from ..utils import (
 )
 
 
-class AolIE(InfoExtractor):
+class AolIE(YahooIE):
     IE_NAME = 'aol.com'
-    _VALID_URL = r'(?:aol-video:|https?://(?:www\.)?aol\.(?:com|ca|co\.uk|de|jp)/video/(?:[^/]+/)*)(?P<id>[0-9a-f]+)'
+    _VALID_URL = r'(?:aol-video:|https?://(?:www\.)?aol\.(?:com|ca|co\.uk|de|jp)/video/(?:[^/]+/)*)(?P<id>\d{9}|[0-9a-f]{24}|[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12})'
 
     _TESTS = [{
         # video with 5min ID
@@ -76,10 +76,16 @@ class AolIE(InfoExtractor):
     }, {
         'url': 'https://www.aol.jp/video/playlist/5a28e936a1334d000137da0c/5a28f3151e642219fde19831/',
         'only_matching': True,
+    }, {
+        # Yahoo video
+        'url': 'https://www.aol.com/video/play/991e6700-ac02-11ea-99ff-357400036f61/24bbc846-3e30-3c46-915e-fe8ccd7fcc46/',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
+        if '-' in video_id:
+            return self._extract_yahoo_video(video_id, 'us')
 
         response = self._download_json(
             'https://feedapi.b2c.on.aol.com/v1.0/app/videos/aolon/%s/details' % video_id,

youtube_dlc/extractor/ard.py

@@ -226,13 +226,13 @@ class ARDMediathekIE(ARDMediathekBaseIE):
             if doc.tag == 'rss':
                 return GenericIE()._extract_rss(url, video_id, doc)
 
-        title = self._html_search_regex(
+        title = self._og_search_title(webpage, default=None) or self._html_search_regex(
             [r'<h1(?:\s+class="boxTopHeadline")?>(.*?)</h1>',
              r'<meta name="dcterms\.title" content="(.*?)"/>',
              r'<h4 class="headline">(.*?)</h4>',
              r'<title[^>]*>(.*?)</title>'],
             webpage, 'title')
-        description = self._html_search_meta(
+        description = self._og_search_description(webpage, default=None) or self._html_search_meta(
             'dcterms.abstract', webpage, 'description', default=None)
         if description is None:
             description = self._html_search_meta(
@@ -289,18 +289,18 @@ class ARDMediathekIE(ARDMediathekBaseIE):
 
 
 class ARDIE(InfoExtractor):
-    _VALID_URL = r'(?P<mainurl>https?://(www\.)?daserste\.de/[^?#]+/videos(?:extern)?/(?P<display_id>[^/?#]+)-(?P<id>[0-9]+))\.html'
+    _VALID_URL = r'(?P<mainurl>https?://(?:www\.)?daserste\.de/[^?#]+/videos(?:extern)?/(?P<display_id>[^/?#]+)-(?:video-?)?(?P<id>[0-9]+))\.html'
     _TESTS = [{
-        # available till 14.02.2019
+        # available till 7.01.2022
-        'url': 'http://www.daserste.de/information/talk/maischberger/videos/das-groko-drama-zerlegen-sich-die-volksparteien-video-102.html',
+        'url': 'https://www.daserste.de/information/talk/maischberger/videos/maischberger-die-woche-video100.html',
-        'md5': '8e4ec85f31be7c7fc08a26cdbc5a1f49',
+        'md5': '867d8aa39eeaf6d76407c5ad1bb0d4c1',
         'info_dict': {
-            'display_id': 'das-groko-drama-zerlegen-sich-die-volksparteien-video',
+            'display_id': 'maischberger-die-woche',
-            'id': '102',
+            'id': '100',
             'ext': 'mp4',
-            'duration': 4435.0,
+            'duration': 3687.0,
-            'title': 'Das GroKo-Drama: Zerlegen sich die Volksparteien?',
+            'title': 'maischberger. die woche vom 7. Januar 2021',
-            'upload_date': '20180214',
+            'upload_date': '20210107',
             'thumbnail': r're:^https?://.*\.jpg$',
         },
     }, {
@@ -355,17 +355,17 @@ class ARDIE(InfoExtractor):
 class ARDBetaMediathekIE(ARDMediathekBaseIE):
     _VALID_URL = r'https://(?:(?:beta|www)\.)?ardmediathek\.de/(?P<client>[^/]+)/(?P<mode>player|live|video|sendung|sammlung)/(?P<display_id>(?:[^/]+/)*)(?P<video_id>[a-zA-Z0-9]+)'
     _TESTS = [{
-        'url': 'https://ardmediathek.de/ard/video/die-robuste-roswita/Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE',
+        'url': 'https://www.ardmediathek.de/mdr/video/die-robuste-roswita/Y3JpZDovL21kci5kZS9iZWl0cmFnL2Ntcy84MWMxN2MzZC0wMjkxLTRmMzUtODk4ZS0wYzhlOWQxODE2NGI/',
-        'md5': 'dfdc87d2e7e09d073d5a80770a9ce88f',
+        'md5': 'a1dc75a39c61601b980648f7c9f9f71d',
         'info_dict': {
             'display_id': 'die-robuste-roswita',
-            'id': '70153354',
+            'id': '78566716',
             'title': 'Die robuste Roswita',
-            'description': r're:^Der Mord.*trüber ist als die Ilm.',
+            'description': r're:^Der Mord.*totgeglaubte Ehefrau Roswita',
             'duration': 5316,
-            'thumbnail': 'https://img.ardmediathek.de/standard/00/70/15/33/90/-1852531467/16x9/960?mandant=ard',
+            'thumbnail': 'https://img.ardmediathek.de/standard/00/78/56/67/84/575672121/16x9/960?mandant=ard',
-            'timestamp': 1577047500,
+            'timestamp': 1596658200,
-            'upload_date': '20191222',
+            'upload_date': '20200805',
             'ext': 'mp4',
         },
     }, {

youtube_dlc/extractor/cbs.py

@@ -59,7 +59,7 @@ class CBSIE(CBSBaseIE):
             'http://can.cbs.com/thunder/player/videoPlayerService.php',
             content_id, query={'partner': site, 'contentId': content_id})
         video_data = xpath_element(items_data, './/item')
-        title = xpath_text(video_data, 'videoTitle', 'title', True)
+        title = xpath_text(video_data, 'videoTitle', 'title') or xpath_text(video_data, 'videotitle', 'title')
         tp_path = 'dJ5BDC/media/guid/%d/%s' % (mpx_acc, content_id)
         tp_release_url = 'http://link.theplatform.com/s/' + tp_path
 

youtube_dlc/extractor/comedycentral.py

@@ -1,142 +1,51 @@
 from __future__ import unicode_literals
 
 from .mtv import MTVServicesInfoExtractor
-from .common import InfoExtractor
 
 
 class ComedyCentralIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'''(?x)https?://(?:www\.)?cc\.com/
+    _VALID_URL = r'https?://(?:www\.)?cc\.com/(?:episodes|video(?:-clips)?)/(?P<id>[0-9a-z]{6})'
-        (video-clips|episodes|cc-studios|video-collections|shows(?=/[^/]+/(?!full-episodes)))
-        /(?P<title>.*)'''
     _FEED_URL = 'http://comedycentral.com/feeds/mrss/'
 
     _TESTS = [{
-        'url': 'http://www.cc.com/video-clips/kllhuv/stand-up-greg-fitzsimmons--uncensored---too-good-of-a-mother',
+        'url': 'http://www.cc.com/video-clips/5ke9v2/the-daily-show-with-trevor-noah-doc-rivers-and-steve-ballmer---the-nba-player-strike',
-        'md5': 'c4f48e9eda1b16dd10add0744344b6d8',
+        'md5': 'b8acb347177c680ff18a292aa2166f80',
         'info_dict': {
-            'id': 'cef0cbb3-e776-4bc9-b62e-8016deccb354',
+            'id': '89ccc86e-1b02-4f83-b0c9-1d9592ecd025',
             'ext': 'mp4',
-            'title': 'CC:Stand-Up|August 18, 2013|1|0101|Uncensored - Too Good of a Mother',
+            'title': 'The Daily Show with Trevor Noah|August 28, 2020|25|25149|Doc Rivers and Steve Ballmer - The NBA Player Strike',
-            'description': 'After a certain point, breastfeeding becomes c**kblocking.',
+            'description': 'md5:5334307c433892b85f4f5e5ac9ef7498',
-            'timestamp': 1376798400,
+            'timestamp': 1598670000,
-            'upload_date': '20130818',
+            'upload_date': '20200829',
         },
     }, {
-        'url': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/interviews/6yx39d/exclusive-rand-paul-extended-interview',
+        'url': 'http://www.cc.com/episodes/pnzzci/drawn-together--american-idol--parody-clip-show-season-3-ep-314',
         'only_matching': True,
-    }]
-
-
-class ComedyCentralFullEpisodesIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'''(?x)https?://(?:www\.)?cc\.com/
-        (?:full-episodes|shows(?=/[^/]+/full-episodes))
-        /(?P<id>[^?]+)'''
-    _FEED_URL = 'http://comedycentral.com/feeds/mrss/'
-
-    _TESTS = [{
-        'url': 'http://www.cc.com/full-episodes/pv391a/the-daily-show-with-trevor-noah-november-28--2016---ryan-speedo-green-season-22-ep-22028',
-        'info_dict': {
-            'description': 'Donald Trump is accused of exploiting his president-elect status for personal gain, Cuban leader Fidel Castro dies, and Ryan Speedo Green discusses "Sing for Your Life."',
-            'title': 'November 28, 2016 - Ryan Speedo Green',
-        },
-        'playlist_count': 4,
     }, {
-        'url': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
+        'url': 'https://www.cc.com/video/k3sdvm/the-daily-show-with-jon-stewart-exclusive-the-fourth-estate',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        playlist_id = self._match_id(url)
-        webpage = self._download_webpage(url, playlist_id)
-        mgid = self._extract_mgid(webpage, url, data_zone='t2_lc_promo1')
-        videos_info = self._get_videos_info(mgid)
-        return videos_info
-
-
-class ToshIE(MTVServicesInfoExtractor):
-    IE_DESC = 'Tosh.0'
-    _VALID_URL = r'^https?://tosh\.cc\.com/video-(?:clips|collections)/[^/]+/(?P<videotitle>[^/?#]+)'
-    _FEED_URL = 'http://tosh.cc.com/feeds/mrss'
-
-    _TESTS = [{
-        'url': 'http://tosh.cc.com/video-clips/68g93d/twitter-users-share-summer-plans',
-        'info_dict': {
-            'description': 'Tosh asked fans to share their summer plans.',
-            'title': 'Twitter Users Share Summer Plans',
-        },
-        'playlist': [{
-            'md5': 'f269e88114c1805bb6d7653fecea9e06',
-            'info_dict': {
-                'id': '90498ec2-ed00-11e0-aca6-0026b9414f30',
-                'ext': 'mp4',
-                'title': 'Tosh.0|June 9, 2077|2|211|Twitter Users Share Summer Plans',
-                'description': 'Tosh asked fans to share their summer plans.',
-                'thumbnail': r're:^https?://.*\.jpg',
-                # It's really reported to be published on year 2077
-                'upload_date': '20770610',
-                'timestamp': 3390510600,
-                'subtitles': {
-                    'en': 'mincount:3',
-                },
-            },
-        }]
-    }, {
-        'url': 'http://tosh.cc.com/video-collections/x2iz7k/just-plain-foul/m5q4fp',
         'only_matching': True,
     }]
 
 
 class ComedyCentralTVIE(MTVServicesInfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?comedycentral\.tv/(?:staffeln|shows)/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?comedycentral\.tv/folgen/(?P<id>[0-9a-z]{6})'
     _TESTS = [{
-        'url': 'http://www.comedycentral.tv/staffeln/7436-the-mindy-project-staffel-4',
+        'url': 'https://www.comedycentral.tv/folgen/pxdpec/josh-investigates-klimawandel-staffel-1-ep-1',
         'info_dict': {
-            'id': 'local_playlist-f99b626bdfe13568579a',
+            'id': '15907dc3-ec3c-11e8-a442-0e40cf2fc285',
-            'ext': 'flv',
+            'ext': 'mp4',
-            'title': 'Episode_the-mindy-project_shows_season-4_episode-3_full-episode_part1',
+            'title': 'Josh Investigates',
+            'description': 'Steht uns das Ende der Welt bevor?',
         },
-        'params': {
-            # rtmp download
-            'skip_download': True,
-        },
-    }, {
-        'url': 'http://www.comedycentral.tv/shows/1074-workaholics',
-        'only_matching': True,
-    }, {
-        'url': 'http://www.comedycentral.tv/shows/1727-the-mindy-project/bonus',
-        'only_matching': True,
     }]
+    _FEED_URL = 'http://feeds.mtvnservices.com/od/feed/intl-mrss-player-feed'
+    _GEO_COUNTRIES = ['DE']
 
-    def _real_extract(self, url):
+    def _get_feed_query(self, uri):
-        video_id = self._match_id(url)
+        return {
-
+            'accountOverride': 'intl.mtvi.com',
-        webpage = self._download_webpage(url, video_id)
+            'arcEp': 'web.cc.tv',
-
+            'ep': 'b9032c3a',
-        mrss_url = self._search_regex(
+            'imageEp': 'web.cc.tv',
-            r'data-mrss=(["\'])(?P<url>(?:(?!\1).)+)\1',
+            'mgid': uri,
-            webpage, 'mrss url', group='url')
-
-        return self._get_videos_info_from_url(mrss_url, video_id)
-
-
-class ComedyCentralShortnameIE(InfoExtractor):
-    _VALID_URL = r'^:(?P<id>tds|thedailyshow|theopposition)$'
-    _TESTS = [{
-        'url': ':tds',
-        'only_matching': True,
-    }, {
-        'url': ':thedailyshow',
-        'only_matching': True,
-    }, {
-        'url': ':theopposition',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        shortcut_map = {
-            'tds': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
-            'thedailyshow': 'http://www.cc.com/shows/the-daily-show-with-trevor-noah/full-episodes',
-            'theopposition': 'http://www.cc.com/shows/the-opposition-with-jordan-klepper/full-episodes',
         }
-        return self.url_result(shortcut_map[video_id])

youtube_dlc/extractor/common.py

@@ -1375,8 +1375,8 @@ class InfoExtractor(object):
                        'order': ['vp9', '(h265|he?vc?)', '(h264|avc)', 'vp8', '(mp4v|h263)', 'theora', '', None, 'none']},
             'acodec': {'type': 'ordered', 'regex': True,
                        'order': ['opus', 'vorbis', 'aac', 'mp?4a?', 'mp3', 'e?a?c-?3', 'dts', '', None, 'none']},
-            'protocol': {'type': 'ordered', 'regex': True,
+            'proto': {'type': 'ordered', 'regex': True,
-                         'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', 'm3u8', '.*dash', '', 'mms|rtsp', 'none', 'f4']},
+                      'order': ['(ht|f)tps', '(ht|f)tp$', 'm3u8.+', 'm3u8', '.*dash', '', 'mms|rtsp', 'none', 'f4']},
             'vext': {'type': 'ordered', 'field': 'video_ext',
                      'order': ('mp4', 'webm', 'flv', '', 'none'),
                      'order_free': ('webm', 'mp4', 'flv', '', 'none')},
@@ -1384,14 +1384,14 @@ class InfoExtractor(object):
                      'order': ('m4a', 'aac', 'mp3', 'ogg', 'opus', 'webm', '', 'none'),
                      'order_free': ('opus', 'ogg', 'webm', 'm4a', 'mp3', 'aac', '', 'none')},
             'hidden': {'visible': False, 'forced': True, 'type': 'extractor', 'max': -1000},
-            'extractor_preference': {'priority': True, 'type': 'extractor'},
+            'ie_pref': {'priority': True, 'type': 'extractor'},
-            'has_video': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
+            'hasvid': {'priority': True, 'field': 'vcodec', 'type': 'boolean', 'not_in_list': ('none',)},
-            'has_audio': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
+            'hasaud': {'field': 'acodec', 'type': 'boolean', 'not_in_list': ('none',)},
-            'language_preference': {'priority': True, 'convert': 'ignore'},
+            'lang': {'priority': True, 'convert': 'ignore'},
             'quality': {'priority': True, 'convert': 'float_none'},
             'filesize': {'convert': 'bytes'},
-            'filesize_approx': {'convert': 'bytes'},
+            'fs_approx': {'convert': 'bytes'},
-            'format_id': {'convert': 'string'},
+            'id': {'convert': 'string'},
             'height': {'convert': 'float_none'},
             'width': {'convert': 'float_none'},
             'fps': {'convert': 'float_none'},
@@ -1399,32 +1399,42 @@ class InfoExtractor(object):
             'vbr': {'convert': 'float_none'},
             'abr': {'convert': 'float_none'},
             'asr': {'convert': 'float_none'},
-            'source_preference': {'convert': 'ignore'},
+            'source': {'convert': 'ignore'},
+
             'codec': {'type': 'combined', 'field': ('vcodec', 'acodec')},
-            'bitrate': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
+            'br': {'type': 'combined', 'field': ('tbr', 'vbr', 'abr'), 'same_limit': True},
-            'filesize_estimate': {'type': 'combined', 'same_limit': True, 'field': ('filesize', 'filesize_approx')},
+            'size': {'type': 'combined', 'same_limit': True, 'field': ('filesize', 'fs_approx')},
-            'extension': {'type': 'combined', 'field': ('vext', 'aext')},
+            'ext': {'type': 'combined', 'field': ('vext', 'aext')},
-            'dimension': {'type': 'multiple', 'field': ('height', 'width'), 'function': min},  # not named as 'resolution' because such a field exists
+            'res': {'type': 'multiple', 'field': ('height', 'width'), 'function': min},
-            'res': {'type': 'alias', 'field': 'dimension'},
+
-            'ext': {'type': 'alias', 'field': 'extension'},
+            # Most of these exist only for compatibility reasons
-            'br': {'type': 'alias', 'field': 'bitrate'},
+            'dimension': {'type': 'alias', 'field': 'res'},
+            'resolution': {'type': 'alias', 'field': 'res'},
+            'extension': {'type': 'alias', 'field': 'ext'},
+            'bitrate': {'type': 'alias', 'field': 'br'},
             'total_bitrate': {'type': 'alias', 'field': 'tbr'},
             'video_bitrate': {'type': 'alias', 'field': 'vbr'},
             'audio_bitrate': {'type': 'alias', 'field': 'abr'},
             'framerate': {'type': 'alias', 'field': 'fps'},
-            'lang': {'type': 'alias', 'field': 'language_preference'},  # not named as 'language' because such a field exists
+            'language_preference': {'type': 'alias', 'field': 'lang'},  # not named as 'language' because such a field exists
-            'proto': {'type': 'alias', 'field': 'protocol'},
+            'protocol': {'type': 'alias', 'field': 'proto'},
-            'source': {'type': 'alias', 'field': 'source_preference'},
+            'source_preference': {'type': 'alias', 'field': 'source'},
-            'size': {'type': 'alias', 'field': 'filesize_estimate'},
+            'filesize_approx': {'type': 'alias', 'field': 'fs_approx'},
+            'filesize_estimate': {'type': 'alias', 'field': 'size'},
             'samplerate': {'type': 'alias', 'field': 'asr'},
             'video_ext': {'type': 'alias', 'field': 'vext'},
             'audio_ext': {'type': 'alias', 'field': 'aext'},
             'video_codec': {'type': 'alias', 'field': 'vcodec'},
             'audio_codec': {'type': 'alias', 'field': 'acodec'},
-            'video': {'type': 'alias', 'field': 'has_video'},
+            'video': {'type': 'alias', 'field': 'hasvid'},
-            'audio': {'type': 'alias', 'field': 'has_audio'},
+            'has_video': {'type': 'alias', 'field': 'hasvid'},
-            'extractor': {'type': 'alias', 'field': 'extractor_preference'},
+            'audio': {'type': 'alias', 'field': 'hasaud'},
-            'preference': {'type': 'alias', 'field': 'extractor_preference'}}
+            'has_audio': {'type': 'alias', 'field': 'hasaud'},
+            'extractor': {'type': 'alias', 'field': 'ie_pref'},
+            'preference': {'type': 'alias', 'field': 'ie_pref'},
+            'extractor_preference': {'type': 'alias', 'field': 'ie_pref'},
+            'format_id': {'type': 'alias', 'field': 'id'},
+        }
 
         _order = []
 

youtube_dlc/extractor/extractors.py

@@ -50,7 +50,10 @@ from .animelab import (
     AnimeLabIE,
     AnimeLabShowsIE,
 )
-from .americastestkitchen import AmericasTestKitchenIE
+from .americastestkitchen import (
+    AmericasTestKitchenIE,
+    AmericasTestKitchenSeasonIE,
+)
 from .animeondemand import AnimeOnDemandIE
 from .anvato import AnvatoIE
 from .aol import AolIE
@@ -244,11 +247,8 @@ from .cnn import (
 )
 from .coub import CoubIE
 from .comedycentral import (
-    ComedyCentralFullEpisodesIE,
     ComedyCentralIE,
-    ComedyCentralShortnameIE,
     ComedyCentralTVIE,
-    ToshIE,
 )
 from .commonmistakes import CommonMistakesIE, UnicodeBOMIE
 from .commonprotocols import (
@@ -677,6 +677,16 @@ from .microsoftvirtualacademy import (
     MicrosoftVirtualAcademyIE,
     MicrosoftVirtualAcademyCourseIE,
 )
+from .mildom import (
+    MildomIE,
+    MildomVodIE,
+    MildomUserVodIE,
+)
+from .minds import (
+    MindsIE,
+    MindsChannelIE,
+    MindsGroupIE,
+)
 from .ministrygrid import MinistryGridIE
 from .minoto import MinotoIE
 from .miomio import MioMioIE
@@ -1157,6 +1167,10 @@ from .stitcher import StitcherIE
 from .sport5 import Sport5IE
 from .sportbox import SportBoxIE
 from .sportdeutschland import SportDeutschlandIE
+from .spotify import (
+    SpotifyIE,
+    SpotifyShowIE,
+)
 from .spreaker import (
     SpreakerIE,
     SpreakerPageIE,
@@ -1265,7 +1279,10 @@ from .toutv import TouTvIE
 from .toypics import ToypicsUserIE, ToypicsIE
 from .traileraddict import TrailerAddictIE
 from .trilulilu import TriluliluIE
-from .trovolive import TrovoLiveIE
+from .trovo import (
+    TrovoIE,
+    TrovoVodIE,
+)
 from .trunews import TruNewsIE
 from .trutv import TruTVIE
 from .tube8 import Tube8IE

youtube_dlc/extractor/franceculture.py

@@ -11,7 +11,7 @@ from ..utils import (
 
 class FranceCultureIE(InfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?franceculture\.fr/emissions/(?:[^/]+/)*(?P<id>[^/?#&]+)'
-    _TEST = {
+    _TESTS = [{
         'url': 'http://www.franceculture.fr/emissions/carnet-nomade/rendez-vous-au-pays-des-geeks',
         'info_dict': {
             'id': 'rendez-vous-au-pays-des-geeks',
@@ -20,10 +20,14 @@ class FranceCultureIE(InfoExtractor):
             'title': 'Rendez-vous au pays des geeks',
             'thumbnail': r're:^https?://.*\.jpg$',
             'upload_date': '20140301',
-            'timestamp': 1393642916,
+            'timestamp': 1393700400,
             'vcodec': 'none',
         }
-    }
+    }, {
+        # no thumbnail
+        'url': 'https://www.franceculture.fr/emissions/la-recherche-montre-en-main/la-recherche-montre-en-main-du-mercredi-10-octobre-2018',
+        'only_matching': True,
+    }]
 
     def _real_extract(self, url):
         display_id = self._match_id(url)
@@ -36,19 +40,19 @@ class FranceCultureIE(InfoExtractor):
                     </h1>|
                     <div[^>]+class="[^"]*?(?:title-zone-diffusion|heading-zone-(?:wrapper|player-button))[^"]*?"[^>]*>
                 ).*?
-                (<button[^>]+data-asset-source="[^"]+"[^>]+>)
+                (<button[^>]+data-(?:url|asset-source)="[^"]+"[^>]+>)
             ''',
             webpage, 'video data'))
 
-        video_url = video_data['data-asset-source']
+        video_url = video_data.get('data-url') or video_data['data-asset-source']
-        title = video_data.get('data-asset-title') or self._og_search_title(webpage)
+        title = video_data.get('data-asset-title') or video_data.get('data-diffusion-title') or self._og_search_title(webpage)
 
         description = self._html_search_regex(
             r'(?s)<div[^>]+class="intro"[^>]*>.*?<h2>(.+?)</h2>',
             webpage, 'description', default=None)
         thumbnail = self._search_regex(
             r'(?s)<figure[^>]+itemtype="https://schema.org/ImageObject"[^>]*>.*?<img[^>]+(?:data-dejavu-)?src="([^"]+)"',
-            webpage, 'thumbnail', fatal=False)
+            webpage, 'thumbnail', default=None)
         uploader = self._html_search_regex(
             r'(?s)<span class="author">(.*?)</span>',
             webpage, 'uploader', default=None)
@@ -64,6 +68,6 @@ class FranceCultureIE(InfoExtractor):
             'ext': ext,
             'vcodec': 'none' if ext == 'mp3' else None,
             'uploader': uploader,
-            'timestamp': int_or_none(video_data.get('data-asset-created-date')),
+            'timestamp': int_or_none(video_data.get('data-start-time')) or int_or_none(video_data.get('data-asset-created-date')),
             'duration': int_or_none(video_data.get('data-duration')),
         }

youtube_dlc/extractor/lbry.py

@@ -5,7 +5,10 @@ import functools
 import json
 
 from .common import InfoExtractor
-from ..compat import compat_str
+from ..compat import (
+    compat_str,
+    compat_urllib_parse_unquote,
+)
 from ..utils import (
     determine_ext,
     ExtractorError,
@@ -131,6 +134,9 @@ class LBRYIE(LBRYBaseIE):
     }, {
         'url': 'https://lbry.tv/$/download/Episode-1/e7d93d772bd87e2b62d5ab993c1c3ced86ebb396',
         'only_matching': True,
+    }, {
+        'url': 'https://lbry.tv/@lacajadepandora:a/TRUMP-EST%C3%81-BIEN-PUESTO-con-Pilar-Baselga,-Carlos-Senra,-Luis-Palacios-(720p_30fps_H264-192kbit_AAC):1',
+        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -139,6 +145,7 @@ class LBRYIE(LBRYBaseIE):
             display_id = display_id.split('/', 2)[-1].replace('/', ':')
         else:
             display_id = display_id.replace(':', '#')
+        display_id = compat_urllib_parse_unquote(display_id)
         uri = 'lbry://' + display_id
         result = self._resolve_url(uri, display_id, 'stream')
         result_value = result['value']

youtube_dlc/extractor/mildom.py

@@ -0,0 +1,284 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from datetime import datetime
+import itertools
+import json
+import base64
+
+from .common import InfoExtractor
+from ..utils import (
+    ExtractorError, std_headers,
+    update_url_query,
+    random_uuidv4,
+    try_get,
+)
+from ..compat import (
+    compat_urlparse,
+    compat_urllib_parse_urlencode,
+    compat_str,
+)
+
+
+class MildomBaseIE(InfoExtractor):
+    _GUEST_ID = None
+    _DISPATCHER_CONFIG = None
+
+    def _call_api(self, url, video_id, query={}, note='Downloading JSON metadata', init=False):
+        url = update_url_query(url, self._common_queries(query, init=init))
+        return self._download_json(url, video_id, note=note)['body']
+
+    def _common_queries(self, query={}, init=False):
+        dc = self._fetch_dispatcher_config()
+        r = {
+            'timestamp': self.iso_timestamp(),
+            '__guest_id': '' if init else self.guest_id(),
+            '__location': dc['location'],
+            '__country': dc['country'],
+            '__cluster': dc['cluster'],
+            '__platform': 'web',
+            '__la': self.lang_code(),
+            '__pcv': 'v2.9.44',
+            'sfr': 'pc',
+            'accessToken': '',
+        }
+        r.update(query)
+        return r
+
+    def _fetch_dispatcher_config(self):
+        if not self._DISPATCHER_CONFIG:
+            try:
+                tmp = self._download_json(
+                    'https://disp.mildom.com/serverListV2', 'initialization',
+                    note='Downloading dispatcher_config', data=json.dumps({
+                        'protover': 0,
+                        'data': base64.b64encode(json.dumps({
+                            'fr': 'web',
+                            'sfr': 'pc',
+                            'devi': 'Windows',
+                            'la': 'ja',
+                            'gid': None,
+                            'loc': '',
+                            'clu': '',
+                            'wh': '1919*810',
+                            'rtm': self.iso_timestamp(),
+                            'ua': std_headers['User-Agent'],
+                        }).encode('utf8')).decode('utf8').replace('\n', ''),
+                    }).encode('utf8'))
+                self._DISPATCHER_CONFIG = self._parse_json(base64.b64decode(tmp['data']), 'initialization')
+            except ExtractorError:
+                self._DISPATCHER_CONFIG = self._download_json(
+                    'https://bookish-octo-barnacle.vercel.app/api/dispatcher_config', 'initialization',
+                    note='Downloading dispatcher_config fallback')
+        return self._DISPATCHER_CONFIG
+
+    @staticmethod
+    def iso_timestamp():
+        'new Date().toISOString()'
+        return datetime.utcnow().isoformat()[0:-3] + 'Z'
+
+    def guest_id(self):
+        'getGuestId'
+        if self._GUEST_ID:
+            return self._GUEST_ID
+        self._GUEST_ID = try_get(
+            self, (
+                lambda x: x._call_api(
+                    'https://cloudac.mildom.com/nonolive/gappserv/guest/h5init', 'initialization',
+                    note='Downloading guest token', init=True)['guest_id'] or None,
+                lambda x: x._get_cookies('https://www.mildom.com').get('gid').value,
+                lambda x: x._get_cookies('https://m.mildom.com').get('gid').value,
+            ), compat_str) or ''
+        return self._GUEST_ID
+
+    def lang_code(self):
+        'getCurrentLangCode'
+        return 'ja'
+
+
+class MildomIE(MildomBaseIE):
+    IE_NAME = 'mildom'
+    IE_DESC = 'Record ongoing live by specific user in Mildom'
+    _VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/(?P<id>\d+)'
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        url = 'https://www.mildom.com/%s' % video_id
+
+        webpage = self._download_webpage(url, video_id)
+
+        enterstudio = self._call_api(
+            'https://cloudac.mildom.com/nonolive/gappserv/live/enterstudio', video_id,
+            note='Downloading live metadata', query={'user_id': video_id})
+
+        title = try_get(
+            enterstudio, (
+                lambda x: self._html_search_meta('twitter:description', webpage),
+                lambda x: x['anchor_intro'],
+            ), compat_str)
+        description = try_get(
+            enterstudio, (
+                lambda x: x['intro'],
+                lambda x: x['live_intro'],
+            ), compat_str)
+        uploader = try_get(
+            enterstudio, (
+                lambda x: self._html_search_meta('twitter:title', webpage),
+                lambda x: x['loginname'],
+            ), compat_str)
+
+        servers = self._call_api(
+            'https://cloudac.mildom.com/nonolive/gappserv/live/liveserver', video_id,
+            note='Downloading live server list', query={
+                'user_id': video_id,
+                'live_server_type': 'hls',
+            })
+
+        stream_query = self._common_queries({
+            'streamReqId': random_uuidv4(),
+            'is_lhls': '0',
+        })
+        m3u8_url = update_url_query(servers['stream_server'] + '/%s_master.m3u8' % video_id, stream_query)
+        formats = self._extract_m3u8_formats(m3u8_url, video_id, 'mp4', headers={
+            'Referer': 'https://www.mildom.com/',
+            'Origin': 'https://www.mildom.com',
+        }, note='Downloading m3u8 information')
+        del stream_query['streamReqId'], stream_query['timestamp']
+        for fmt in formats:
+            # Uses https://github.com/nao20010128nao/bookish-octo-barnacle by @nao20010128nao as a proxy
+            parsed = compat_urlparse.urlparse(fmt['url'])
+            parsed = parsed._replace(
+                netloc='bookish-octo-barnacle.vercel.app',
+                query=compat_urllib_parse_urlencode(stream_query, True),
+                path='/api' + parsed.path)
+            fmt['url'] = compat_urlparse.urlunparse(parsed)
+
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': description,
+            'uploader': uploader,
+            'uploader_id': video_id,
+            'formats': formats,
+            'is_live': True,
+        }
+
+
+class MildomVodIE(MildomBaseIE):
+    IE_NAME = 'mildom:vod'
+    IE_DESC = 'Download a VOD in Mildom'
+    _VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/playback/(?P<user_id>\d+)/(?P<id>(?P=user_id)-[a-zA-Z0-9]+)'
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        m = self._VALID_URL_RE.match(url)
+        user_id = m.group('user_id')
+        url = 'https://www.mildom.com/playback/%s/%s' % (user_id, video_id)
+
+        webpage = self._download_webpage(url, video_id)
+
+        autoplay = self._call_api(
+            'https://cloudac.mildom.com/nonolive/videocontent/playback/getPlaybackDetail', video_id,
+            note='Downloading playback metadata', query={
+                'v_id': video_id,
+            })['playback']
+
+        title = try_get(
+            autoplay, (
+                lambda x: self._html_search_meta('og:description', webpage),
+                lambda x: x['title'],
+            ), compat_str)
+        description = try_get(
+            autoplay, (
+                lambda x: x['video_intro'],
+            ), compat_str)
+        uploader = try_get(
+            autoplay, (
+                lambda x: x['author_info']['login_name'],
+            ), compat_str)
+
+        audio_formats = [{
+            'url': autoplay['audio_url'],
+            'format_id': 'audio',
+            'protocol': 'm3u8_native',
+            'vcodec': 'none',
+            'acodec': 'aac',
+        }]
+        video_formats = []
+        for fmt in autoplay['video_link']:
+            video_formats.append({
+                'format_id': 'video-%s' % fmt['name'],
+                'url': fmt['url'],
+                'protocol': 'm3u8_native',
+                'width': fmt['level'] * autoplay['video_width'] // autoplay['video_height'],
+                'height': fmt['level'],
+                'vcodec': 'h264',
+                'acodec': 'aac',
+            })
+
+        stream_query = self._common_queries({
+            'is_lhls': '0',
+        })
+        del stream_query['timestamp']
+        formats = audio_formats + video_formats
+        for fmt in formats:
+            fmt['ext'] = 'mp4'
+            parsed = compat_urlparse.urlparse(fmt['url'])
+            stream_query['path'] = parsed.path[5:]
+            parsed = parsed._replace(
+                netloc='bookish-octo-barnacle.vercel.app',
+                query=compat_urllib_parse_urlencode(stream_query, True),
+                path='/api/vod2/proxy')
+            fmt['url'] = compat_urlparse.urlunparse(parsed)
+
+        self._sort_formats(formats)
+
+        return {
+            'id': video_id,
+            'title': title,
+            'description': description,
+            'uploader': uploader,
+            'uploader_id': user_id,
+            'formats': formats,
+        }
+
+
+class MildomUserVodIE(MildomBaseIE):
+    IE_NAME = 'mildom:user:vod'
+    IE_DESC = 'Download all VODs from specific user in Mildom'
+    _VALID_URL = r'https?://(?:(?:www|m)\.)mildom\.com/profile/(?P<id>\d+)'
+    _TESTS = [{
+        'url': 'https://www.mildom.com/profile/10093333',
+        'info_dict': {
+            'id': '10093333',
+            'title': 'Uploads from ねこばたけ',
+        },
+        'playlist_mincount': 351,
+    }]
+
+    def _real_extract(self, url):
+        user_id = self._match_id(url)
+
+        self._downloader.report_warning('To download ongoing live, please use "https://www.mildom.com/%s" instead. This will list up VODs belonging to user.' % user_id)
+
+        profile = self._call_api(
+            'https://cloudac.mildom.com/nonolive/gappserv/user/profileV2', user_id,
+            query={'user_id': user_id}, note='Downloading user profile')['user_info']
+
+        results = []
+        for page in itertools.count(1):
+            reply = self._call_api(
+                'https://cloudac.mildom.com/nonolive/videocontent/profile/playbackList',
+                user_id, note='Downloading page %d' % page, query={
+                    'user_id': user_id,
+                    'page': page,
+                    'limit': '30',
+                })
+            if not reply:
+                break
+            results.extend('https://www.mildom.com/playback/%s/%s' % (user_id, x['v_id']) for x in reply)
+        return self.playlist_result([
+            self.url_result(u, ie=MildomVodIE.ie_key()) for u in results
+        ], user_id, 'Uploads from %s' % profile['loginname'])

youtube_dlc/extractor/minds.py

@@ -0,0 +1,196 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..compat import compat_str
+from ..utils import (
+    clean_html,
+    int_or_none,
+    str_or_none,
+    strip_or_none,
+)
+
+
+class MindsBaseIE(InfoExtractor):
+    _VALID_URL_BASE = r'https?://(?:www\.)?minds\.com/'
+
+    def _call_api(self, path, video_id, resource, query=None):
+        api_url = 'https://www.minds.com/api/' + path
+        token = self._get_cookies(api_url).get('XSRF-TOKEN')
+        return self._download_json(
+            api_url, video_id, 'Downloading %s JSON metadata' % resource, headers={
+                'Referer': 'https://www.minds.com/',
+                'X-XSRF-TOKEN': token.value if token else '',
+            }, query=query)
+
+
+class MindsIE(MindsBaseIE):
+    IE_NAME = 'minds'
+    _VALID_URL = MindsBaseIE._VALID_URL_BASE + r'(?:media|newsfeed|archive/view)/(?P<id>[0-9]+)'
+    _TESTS = [{
+        'url': 'https://www.minds.com/media/100000000000086822',
+        'md5': '215a658184a419764852239d4970b045',
+        'info_dict': {
+            'id': '100000000000086822',
+            'ext': 'mp4',
+            'title': 'Minds intro sequence',
+            'thumbnail': r're:https?://.+\.png',
+            'uploader_id': 'ottman',
+            'upload_date': '20130524',
+            'timestamp': 1369404826,
+            'uploader': 'Bill Ottman',
+            'view_count': int,
+            'like_count': int,
+            'dislike_count': int,
+            'tags': ['animation'],
+            'comment_count': int,
+            'license': 'attribution-cc',
+        },
+    }, {
+        # entity.type == 'activity' and empty title
+        'url': 'https://www.minds.com/newsfeed/798025111988506624',
+        'md5': 'b2733a74af78d7fd3f541c4cbbaa5950',
+        'info_dict': {
+            'id': '798022190320226304',
+            'ext': 'mp4',
+            'title': '798022190320226304',
+            'uploader': 'ColinFlaherty',
+            'upload_date': '20180111',
+            'timestamp': 1515639316,
+            'uploader_id': 'ColinFlaherty',
+        },
+    }, {
+        'url': 'https://www.minds.com/archive/view/715172106794442752',
+        'only_matching': True,
+    }, {
+        # youtube perma_url
+        'url': 'https://www.minds.com/newsfeed/1197131838022602752',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        entity_id = self._match_id(url)
+        entity = self._call_api(
+            'v1/entities/entity/' + entity_id, entity_id, 'entity')['entity']
+        if entity.get('type') == 'activity':
+            if entity.get('custom_type') == 'video':
+                video_id = entity['entity_guid']
+            else:
+                return self.url_result(entity['perma_url'])
+        else:
+            assert(entity['subtype'] == 'video')
+            video_id = entity_id
+        # 1080p and webm formats available only on the sources array
+        video = self._call_api(
+            'v2/media/video/' + video_id, video_id, 'video')
+
+        formats = []
+        for source in (video.get('sources') or []):
+            src = source.get('src')
+            if not src:
+                continue
+            formats.append({
+                'format_id': source.get('label'),
+                'height': int_or_none(source.get('size')),
+                'url': src,
+            })
+        self._sort_formats(formats)
+
+        entity = video.get('entity') or entity
+        owner = entity.get('ownerObj') or {}
+        uploader_id = owner.get('username')
+
+        tags = entity.get('tags')
+        if tags and isinstance(tags, compat_str):
+            tags = [tags]
+
+        thumbnail = None
+        poster = video.get('poster') or entity.get('thumbnail_src')
+        if poster:
+            urlh = self._request_webpage(poster, video_id, fatal=False)
+            if urlh:
+                thumbnail = urlh.geturl()
+
+        return {
+            'id': video_id,
+            'title': entity.get('title') or video_id,
+            'formats': formats,
+            'description': clean_html(entity.get('description')) or None,
+            'license': str_or_none(entity.get('license')),
+            'timestamp': int_or_none(entity.get('time_created')),
+            'uploader': strip_or_none(owner.get('name')),
+            'uploader_id': uploader_id,
+            'uploader_url': 'https://www.minds.com/' + uploader_id if uploader_id else None,
+            'view_count': int_or_none(entity.get('play:count')),
+            'like_count': int_or_none(entity.get('thumbs:up:count')),
+            'dislike_count': int_or_none(entity.get('thumbs:down:count')),
+            'tags': tags,
+            'comment_count': int_or_none(entity.get('comments:count')),
+            'thumbnail': thumbnail,
+        }
+
+
+class MindsFeedBaseIE(MindsBaseIE):
+    _PAGE_SIZE = 150
+
+    def _entries(self, feed_id):
+        query = {'limit': self._PAGE_SIZE, 'sync': 1}
+        i = 1
+        while True:
+            data = self._call_api(
+                'v2/feeds/container/%s/videos' % feed_id,
+                feed_id, 'page %s' % i, query)
+            entities = data.get('entities') or []
+            for entity in entities:
+                guid = entity.get('guid')
+                if not guid:
+                    continue
+                yield self.url_result(
+                    'https://www.minds.com/newsfeed/' + guid,
+                    MindsIE.ie_key(), guid)
+            query['from_timestamp'] = data['load-next']
+            if not (query['from_timestamp'] and len(entities) == self._PAGE_SIZE):
+                break
+            i += 1
+
+    def _real_extract(self, url):
+        feed_id = self._match_id(url)
+        feed = self._call_api(
+            'v1/%s/%s' % (self._FEED_PATH, feed_id),
+            feed_id, self._FEED_TYPE)[self._FEED_TYPE]
+
+        return self.playlist_result(
+            self._entries(feed['guid']), feed_id,
+            strip_or_none(feed.get('name')),
+            feed.get('briefdescription'))
+
+
+class MindsChannelIE(MindsFeedBaseIE):
+    _FEED_TYPE = 'channel'
+    IE_NAME = 'minds:' + _FEED_TYPE
+    _VALID_URL = MindsBaseIE._VALID_URL_BASE + r'(?!(?:newsfeed|media|api|archive|groups)/)(?P<id>[^/?&#]+)'
+    _FEED_PATH = 'channel'
+    _TEST = {
+        'url': 'https://www.minds.com/ottman',
+        'info_dict': {
+            'id': 'ottman',
+            'title': 'Bill Ottman',
+            'description': 'Co-creator & CEO @minds',
+        },
+        'playlist_mincount': 54,
+    }
+
+
+class MindsGroupIE(MindsFeedBaseIE):
+    _FEED_TYPE = 'group'
+    IE_NAME = 'minds:' + _FEED_TYPE
+    _VALID_URL = MindsBaseIE._VALID_URL_BASE + r'groups/profile/(?P<id>[0-9]+)'
+    _FEED_PATH = 'groups/group'
+    _TEST = {
+        'url': 'https://www.minds.com/groups/profile/785582576369672204/feed/videos',
+        'info_dict': {
+            'id': '785582576369672204',
+            'title': 'Cooking Videos',
+        },
+        'playlist_mincount': 1,
+    }

youtube_dlc/extractor/mtv.py

@@ -255,6 +255,10 @@ class MTVServicesInfoExtractor(InfoExtractor):
 
         return try_get(feed, lambda x: x['result']['data']['id'], compat_str)
 
+    @staticmethod
+    def _extract_child_with_type(parent, t):
+        return next(c for c in parent['children'] if c.get('type') == t)
+
     def _extract_new_triforce_mgid(self, webpage, url='', video_id=None):
         if url == '':
             return
@@ -332,6 +336,13 @@ class MTVServicesInfoExtractor(InfoExtractor):
         if not mgid:
             mgid = self._extract_triforce_mgid(webpage, data_zone)
 
+        if not mgid:
+            data = self._parse_json(self._search_regex(
+                r'__DATA__\s*=\s*({.+?});', webpage, 'data'), None)
+            main_container = self._extract_child_with_type(data, 'MainContainer')
+            video_player = self._extract_child_with_type(main_container, 'VideoPlayer')
+            mgid = video_player['props']['media']['video']['config']['uri']
+
         return mgid
 
     def _real_extract(self, url):
@@ -403,18 +414,6 @@ class MTVIE(MTVServicesInfoExtractor):
         'only_matching': True,
     }]
 
-    @staticmethod
-    def extract_child_with_type(parent, t):
-        children = parent['children']
-        return next(c for c in children if c.get('type') == t)
-
-    def _extract_mgid(self, webpage):
-        data = self._parse_json(self._search_regex(
-            r'__DATA__\s*=\s*({.+?});', webpage, 'data'), None)
-        main_container = self.extract_child_with_type(data, 'MainContainer')
-        video_player = self.extract_child_with_type(main_container, 'VideoPlayer')
-        return video_player['props']['media']['video']['config']['uri']
-
 
 class MTVJapanIE(MTVServicesInfoExtractor):
     IE_NAME = 'mtvjapan'

youtube_dlc/extractor/ninegag.py

@@ -1,104 +1,125 @@
 from __future__ import unicode_literals
 
-import re
-
 from .common import InfoExtractor
-from ..utils import str_to_int
+from ..utils import (
+    determine_ext,
+    ExtractorError,
+    int_or_none,
+    try_get,
+    url_or_none,
+)
 
 
 class NineGagIE(InfoExtractor):
     IE_NAME = '9gag'
-    _VALID_URL = r'https?://(?:www\.)?9gag(?:\.com/tv|\.tv)/(?:p|embed)/(?P<id>[a-zA-Z0-9]+)(?:/(?P<display_id>[^?#/]+))?'
+    _VALID_URL = r'https?://(?:www\.)?9gag\.com/gag/(?P<id>[^/?&#]+)'
 
-    _TESTS = [{
+    _TEST = {
-        'url': 'http://9gag.com/tv/p/Kk2X5/people-are-awesome-2013-is-absolutely-awesome',
+        'url': 'https://9gag.com/gag/ae5Ag7B',
         'info_dict': {
-            'id': 'kXzwOKyGlSA',
+            'id': 'ae5Ag7B',
             'ext': 'mp4',
-            'description': 'This 3-minute video will make you smile and then make you feel untalented and insignificant. Anyway, you should share this awesomeness. (Thanks, Dino!)',
+            'title': 'Capybara Agility Training',
-            'title': '\"People Are Awesome 2013\" Is Absolutely Awesome',
+            'upload_date': '20191108',
-            'uploader_id': 'UCdEH6EjDKwtTe-sO2f0_1XA',
+            'timestamp': 1573237208,
-            'uploader': 'CompilationChannel',
+            'categories': ['Awesome'],
-            'upload_date': '20131110',
+            'tags': ['Weimaraner', 'American Pit Bull Terrier'],
-            'view_count': int,
+            'duration': 44,
-        },
+            'like_count': int,
-        'add_ie': ['Youtube'],
+            'dislike_count': int,
-    }, {
+            'comment_count': int,
-        'url': 'http://9gag.com/tv/p/aKolP3',
+        }
-        'info_dict': {
-            'id': 'aKolP3',
-            'ext': 'mp4',
-            'title': 'This Guy Travelled 11 countries In 44 days Just To Make This Amazing Video',
-            'description': "I just saw more in 1 minute than I've seen in 1 year. This guy's video is epic!!",
-            'uploader_id': 'rickmereki',
-            'uploader': 'Rick Mereki',
-            'upload_date': '20110803',
-            'view_count': int,
-        },
-        'add_ie': ['Vimeo'],
-    }, {
-        'url': 'http://9gag.com/tv/p/KklwM',
-        'only_matching': True,
-    }, {
-        'url': 'http://9gag.tv/p/Kk2X5',
-        'only_matching': True,
-    }, {
-        'url': 'http://9gag.com/tv/embed/a5Dmvl',
-        'only_matching': True,
-    }]
-
-    _EXTERNAL_VIDEO_PROVIDER = {
-        '1': {
-            'url': '%s',
-            'ie_key': 'Youtube',
-        },
-        '2': {
-            'url': 'http://player.vimeo.com/video/%s',
-            'ie_key': 'Vimeo',
-        },
-        '3': {
-            'url': 'http://instagram.com/p/%s',
-            'ie_key': 'Instagram',
-        },
-        '4': {
-            'url': 'http://vine.co/v/%s',
-            'ie_key': 'Vine',
-        },
     }
 
     def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        post_id = self._match_id(url)
-        video_id = mobj.group('id')
+        post = self._download_json(
-        display_id = mobj.group('display_id') or video_id
+            'https://9gag.com/v1/post', post_id, query={
+                'id': post_id
+            })['data']['post']
 
-        webpage = self._download_webpage(url, display_id)
+        if post.get('type') != 'Animated':
+            raise ExtractorError(
+                'The given url does not contain a video',
+                expected=True)
 
-        post_view = self._parse_json(
+        title = post['title']
-            self._search_regex(
-                r'var\s+postView\s*=\s*new\s+app\.PostView\({\s*post:\s*({.+?})\s*,\s*posts:\s*prefetchedCurrentPost',
-                webpage, 'post view'),
-            display_id)
 
-        ie_key = None
+        duration = None
-        source_url = post_view.get('sourceUrl')
+        formats = []
-        if not source_url:
+        thumbnails = []
-            external_video_id = post_view['videoExternalId']
+        for key, image in (post.get('images') or {}).items():
-            external_video_provider = post_view['videoExternalProvider']
+            image_url = url_or_none(image.get('url'))
-            source_url = self._EXTERNAL_VIDEO_PROVIDER[external_video_provider]['url'] % external_video_id
+            if not image_url:
-            ie_key = self._EXTERNAL_VIDEO_PROVIDER[external_video_provider]['ie_key']
+                continue
-        title = post_view['title']
+            ext = determine_ext(image_url)
-        description = post_view.get('description')
+            image_id = key.strip('image')
-        view_count = str_to_int(post_view.get('externalView'))
+            common = {
-        thumbnail = post_view.get('thumbnail_700w') or post_view.get('ogImageUrl') or post_view.get('thumbnail_300w')
+                'url': image_url,
+                'width': int_or_none(image.get('width')),
+                'height': int_or_none(image.get('height')),
+            }
+            if ext in ('jpg', 'png'):
+                webp_url = image.get('webpUrl')
+                if webp_url:
+                    t = common.copy()
+                    t.update({
+                        'id': image_id + '-webp',
+                        'url': webp_url,
+                    })
+                    thumbnails.append(t)
+                common.update({
+                    'id': image_id,
+                    'ext': ext,
+                })
+                thumbnails.append(common)
+            elif ext in ('webm', 'mp4'):
+                if not duration:
+                    duration = int_or_none(image.get('duration'))
+                common['acodec'] = 'none' if image.get('hasAudio') == 0 else None
+                for vcodec in ('vp8', 'vp9', 'h265'):
+                    c_url = image.get(vcodec + 'Url')
+                    if not c_url:
+                        continue
+                    c_f = common.copy()
+                    c_f.update({
+                        'format_id': image_id + '-' + vcodec,
+                        'url': c_url,
+                        'vcodec': vcodec,
+                    })
+                    formats.append(c_f)
+                common.update({
+                    'ext': ext,
+                    'format_id': image_id,
+                })
+                formats.append(common)
+        self._sort_formats(formats)
+
+        section = try_get(post, lambda x: x['postSection']['name'])
+
+        tags = None
+        post_tags = post.get('tags')
+        if post_tags:
+            tags = []
+            for tag in post_tags:
+                tag_key = tag.get('key')
+                if not tag_key:
+                    continue
+                tags.append(tag_key)
+
+        get_count = lambda x: int_or_none(post.get(x + 'Count'))
 
         return {
-            '_type': 'url_transparent',
+            'id': post_id,
-            'url': source_url,
-            'ie_key': ie_key,
-            'id': video_id,
-            'display_id': display_id,
             'title': title,
-            'description': description,
+            'timestamp': int_or_none(post.get('creationTs')),
-            'view_count': view_count,
+            'duration': duration,
-            'thumbnail': thumbnail,
+            'formats': formats,
+            'thumbnails': thumbnails,
+            'like_count': get_count('upVote'),
+            'dislike_count': get_count('downVote'),
+            'comment_count': get_count('comments'),
+            'age_limit': 18 if post.get('nsfw') == 1 else None,
+            'categories': [section] if section else None,
+            'tags': tags,
         }

youtube_dlc/extractor/njpwworld.py

@@ -6,30 +6,40 @@ import re
 from .common import InfoExtractor
 from ..compat import compat_urlparse
 from ..utils import (
-    extract_attributes,
     get_element_by_class,
     urlencode_postdata,
 )
 
 
 class NJPWWorldIE(InfoExtractor):
-    _VALID_URL = r'https?://njpwworld\.com/p/(?P<id>[a-z0-9_]+)'
+    _VALID_URL = r'https?://(front\.)?njpwworld\.com/p/(?P<id>[a-z0-9_]+)'
     IE_DESC = '新日本プロレスワールド'
     _NETRC_MACHINE = 'njpwworld'
 
-    _TEST = {
+    _TESTS = [{
         'url': 'http://njpwworld.com/p/s_series_00155_1_9/',
         'info_dict': {
             'id': 's_series_00155_1_9',
             'ext': 'mp4',
-            'title': '第9試合　ランディ・サベージ　vs　リック・スタイナー',
+            'title': '闘強導夢2000 2000年1月4日 東京ドーム 第9試合 ランディ・サベージ VS リック・スタイナー',
             'tags': list,
         },
         'params': {
             'skip_download': True,  # AES-encrypted m3u8
         },
         'skip': 'Requires login',
-    }
+    }, {
+        'url': 'https://front.njpwworld.com/p/s_series_00563_16_bs',
+        'info_dict': {
+            'id': 's_series_00563_16_bs',
+            'ext': 'mp4',
+            'title': 'WORLD TAG LEAGUE 2020 & BEST OF THE SUPER Jr.27 2020年12月6日 福岡・福岡国際センター バックステージコメント（字幕あり）',
+            'tags': ["福岡・福岡国際センター", "バックステージコメント", "2020", "20年代"],
+        },
+        'params': {
+            'skip_download': True,
+        },
+    }]
 
     _LOGIN_URL = 'https://front.njpwworld.com/auth/login'
 
@@ -64,35 +74,27 @@ class NJPWWorldIE(InfoExtractor):
         webpage = self._download_webpage(url, video_id)
 
         formats = []
-        for mobj in re.finditer(r'<a[^>]+\bhref=(["\'])/player.+?[^>]*>', webpage):
+        for kind, vid in re.findall(r'if\s+\(\s*imageQualityType\s*==\s*\'([^\']+)\'\s*\)\s*{\s*video_id\s*=\s*"(\d+)"', webpage):
-            player = extract_attributes(mobj.group(0))
+            player_path = '/intent?id=%s&type=url' % vid
-            player_path = player.get('href')
-            if not player_path:
-                continue
-            kind = self._search_regex(
-                r'(low|high)$', player.get('class') or '', 'kind',
-                default='low')
             player_url = compat_urlparse.urljoin(url, player_path)
-            player_page = self._download_webpage(
+            formats.append({
-                player_url, video_id, note='Downloading player page')
+                'url': player_url,
-            entries = self._parse_html5_media_entries(
+                'format_id': kind,
-                player_url, player_page, video_id, m3u8_id='hls-%s' % kind,
+                'ext': 'mp4',
-                m3u8_entry_protocol='m3u8_native')
+                'protocol': 'm3u8',
-            kind_formats = entries[0]['formats']
+                'quality': 2 if kind == 'high' else 1,
-            for f in kind_formats:
+            })
-                f['quality'] = 2 if kind == 'high' else 1
-            formats.extend(kind_formats)
 
         self._sort_formats(formats)
 
-        post_content = get_element_by_class('post-content', webpage)
+        tag_block = get_element_by_class('tag-block', webpage)
         tags = re.findall(
-            r'<li[^>]+class="tag-[^"]+"><a[^>]*>([^<]+)</a></li>', post_content
+            r'<a[^>]+class="tag-[^"]+"[^>]*>([^<]+)</a>', tag_block
-        ) if post_content else None
+        ) if tag_block else None
 
         return {
             'id': video_id,
-            'title': self._og_search_title(webpage),
+            'title': get_element_by_class('article-title', webpage) or self._og_search_title(webpage),
             'formats': formats,
             'tags': tags,
         }

youtube_dlc/extractor/spike.py

@@ -20,19 +20,6 @@ class BellatorIE(MTVServicesInfoExtractor):
     _FEED_URL = 'http://www.bellator.com/feeds/mrss/'
     _GEO_COUNTRIES = ['US']
 
-    def _extract_mgid(self, webpage, url):
-        mgid = None
-
-        if not mgid:
-            mgid = self._extract_triforce_mgid(webpage)
-
-        if not mgid:
-            mgid = self._extract_new_triforce_mgid(webpage, url)
-
-        return mgid
-
-# TODO Remove - Reason: Outdated Site
-
 
 class ParamountNetworkIE(MTVServicesInfoExtractor):
     _VALID_URL = r'https?://(?:www\.)?paramountnetwork\.com/[^/]+/[\da-z]{6}(?:[/?#&]|$)'
@@ -56,16 +43,6 @@ class ParamountNetworkIE(MTVServicesInfoExtractor):
     def _get_feed_query(self, uri):
         return {
             'arcEp': 'paramountnetwork.com',
+            'imageEp': 'paramountnetwork.com',
             'mgid': uri,
         }
-
-    def _extract_mgid(self, webpage, url):
-        root_data = self._parse_json(self._search_regex(
-            r'window\.__DATA__\s*=\s*({.+})',
-            webpage, 'data'), None)
-
-        def find_sub_data(data, data_type):
-            return next(c for c in data['children'] if c.get('type') == data_type)
-
-        c = find_sub_data(find_sub_data(root_data, 'MainContainer'), 'VideoPlayer')
-        return c['props']['media']['video']['config']['uri']

youtube_dlc/extractor/spotify.py

@@ -0,0 +1,156 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    clean_podcast_url,
+    float_or_none,
+    int_or_none,
+    strip_or_none,
+    try_get,
+    unified_strdate,
+)
+
+
+class SpotifyBaseIE(InfoExtractor):
+    _ACCESS_TOKEN = None
+    _OPERATION_HASHES = {
+        'Episode': '8276d4423d709ae9b68ec1b74cc047ba0f7479059a37820be730f125189ac2bf',
+        'MinimalShow': '13ee079672fad3f858ea45a55eb109553b4fb0969ed793185b2e34cbb6ee7cc0',
+        'ShowEpisodes': 'e0e5ce27bd7748d2c59b4d44ba245a8992a05be75d6fabc3b20753fc8857444d',
+    }
+    _VALID_URL_TEMPL = r'https?://open\.spotify\.com/%s/(?P<id>[^/?&#]+)'
+
+    def _real_initialize(self):
+        self._ACCESS_TOKEN = self._download_json(
+            'https://open.spotify.com/get_access_token', None)['accessToken']
+
+    def _call_api(self, operation, video_id, variables):
+        return self._download_json(
+            'https://api-partner.spotify.com/pathfinder/v1/query', video_id, query={
+                'operationName': 'query' + operation,
+                'variables': json.dumps(variables),
+                'extensions': json.dumps({
+                    'persistedQuery': {
+                        'sha256Hash': self._OPERATION_HASHES[operation],
+                    },
+                })
+            }, headers={'authorization': 'Bearer ' + self._ACCESS_TOKEN})['data']
+
+    def _extract_episode(self, episode, series):
+        episode_id = episode['id']
+        title = episode['name'].strip()
+
+        formats = []
+        audio_preview = episode.get('audioPreview') or {}
+        audio_preview_url = audio_preview.get('url')
+        if audio_preview_url:
+            f = {
+                'url': audio_preview_url.replace('://p.scdn.co/mp3-preview/', '://anon-podcast.scdn.co/'),
+                'vcodec': 'none',
+            }
+            audio_preview_format = audio_preview.get('format')
+            if audio_preview_format:
+                f['format_id'] = audio_preview_format
+                mobj = re.match(r'([0-9A-Z]{3})_(?:[A-Z]+_)?(\d+)', audio_preview_format)
+                if mobj:
+                    f.update({
+                        'abr': int(mobj.group(2)),
+                        'ext': mobj.group(1).lower(),
+                    })
+            formats.append(f)
+
+        for item in (try_get(episode, lambda x: x['audio']['items']) or []):
+            item_url = item.get('url')
+            if not (item_url and item.get('externallyHosted')):
+                continue
+            formats.append({
+                'url': clean_podcast_url(item_url),
+                'vcodec': 'none',
+            })
+
+        thumbnails = []
+        for source in (try_get(episode, lambda x: x['coverArt']['sources']) or []):
+            source_url = source.get('url')
+            if not source_url:
+                continue
+            thumbnails.append({
+                'url': source_url,
+                'width': int_or_none(source.get('width')),
+                'height': int_or_none(source.get('height')),
+            })
+
+        return {
+            'id': episode_id,
+            'title': title,
+            'formats': formats,
+            'thumbnails': thumbnails,
+            'description': strip_or_none(episode.get('description')),
+            'duration': float_or_none(try_get(
+                episode, lambda x: x['duration']['totalMilliseconds']), 1000),
+            'release_date': unified_strdate(try_get(
+                episode, lambda x: x['releaseDate']['isoString'])),
+            'series': series,
+        }
+
+
+class SpotifyIE(SpotifyBaseIE):
+    IE_NAME = 'spotify'
+    _VALID_URL = SpotifyBaseIE._VALID_URL_TEMPL % 'episode'
+    _TEST = {
+        'url': 'https://open.spotify.com/episode/4Z7GAJ50bgctf6uclHlWKo',
+        'md5': '74010a1e3fa4d9e1ab3aa7ad14e42d3b',
+        'info_dict': {
+            'id': '4Z7GAJ50bgctf6uclHlWKo',
+            'ext': 'mp3',
+            'title': 'From the archive: Why time management is ruining our lives',
+            'description': 'md5:b120d9c4ff4135b42aa9b6d9cde86935',
+            'duration': 2083.605,
+            'release_date': '20201217',
+            'series': "The Guardian's Audio Long Reads",
+        }
+    }
+
+    def _real_extract(self, url):
+        episode_id = self._match_id(url)
+        episode = self._call_api('Episode', episode_id, {
+            'uri': 'spotify:episode:' + episode_id
+        })['episode']
+        return self._extract_episode(
+            episode, try_get(episode, lambda x: x['podcast']['name']))
+
+
+class SpotifyShowIE(SpotifyBaseIE):
+    IE_NAME = 'spotify:show'
+    _VALID_URL = SpotifyBaseIE._VALID_URL_TEMPL % 'show'
+    _TEST = {
+        'url': 'https://open.spotify.com/show/4PM9Ke6l66IRNpottHKV9M',
+        'info_dict': {
+            'id': '4PM9Ke6l66IRNpottHKV9M',
+            'title': 'The Story from the Guardian',
+            'description': 'The Story podcast is dedicated to our finest audio documentaries, investigations and long form stories',
+        },
+        'playlist_mincount': 36,
+    }
+
+    def _real_extract(self, url):
+        show_id = self._match_id(url)
+        podcast = self._call_api('ShowEpisodes', show_id, {
+            'limit': 1000000000,
+            'offset': 0,
+            'uri': 'spotify:show:' + show_id,
+        })['podcast']
+        podcast_name = podcast.get('name')
+
+        entries = []
+        for item in (try_get(podcast, lambda x: x['episodes']['items']) or []):
+            episode = item.get('episode')
+            if not episode:
+                continue
+            entries.append(self._extract_episode(episode, podcast_name))
+
+        return self.playlist_result(
+            entries, show_id, podcast_name, podcast.get('description'))

youtube_dlc/extractor/trovo.py

@@ -0,0 +1,193 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+
+from .common import InfoExtractor
+from ..utils import (
+    ExtractorError,
+    int_or_none,
+    str_or_none,
+    try_get,
+)
+
+
+class TrovoBaseIE(InfoExtractor):
+    _VALID_URL_BASE = r'https?://(?:www\.)?trovo\.live/'
+
+    def _extract_streamer_info(self, data):
+        streamer_info = data.get('streamerInfo') or {}
+        username = streamer_info.get('userName')
+        return {
+            'uploader': streamer_info.get('nickName'),
+            'uploader_id': str_or_none(streamer_info.get('uid')),
+            'uploader_url': 'https://trovo.live/' + username if username else None,
+        }
+
+
+class TrovoIE(TrovoBaseIE):
+    _VALID_URL = TrovoBaseIE._VALID_URL_BASE + r'(?!(?:clip|video)/)(?P<id>[^/?&#]+)'
+
+    def _real_extract(self, url):
+        username = self._match_id(url)
+        live_info = self._download_json(
+            'https://gql.trovo.live/', username, query={
+                'query': '''{
+  getLiveInfo(params: {userName: "%s"}) {
+    isLive
+    programInfo	{
+      coverUrl
+      id
+      streamInfo {
+        desc
+        playUrl
+      }
+      title
+    }
+    streamerInfo {
+        nickName
+        uid
+        userName
+    }
+  }
+}''' % username,
+            })['data']['getLiveInfo']
+        if live_info.get('isLive') == 0:
+            raise ExtractorError('%s is offline' % username, expected=True)
+        program_info = live_info['programInfo']
+        program_id = program_info['id']
+        title = self._live_title(program_info['title'])
+
+        formats = []
+        for stream_info in (program_info.get('streamInfo') or []):
+            play_url = stream_info.get('playUrl')
+            if not play_url:
+                continue
+            format_id = stream_info.get('desc')
+            formats.append({
+                'format_id': format_id,
+                'height': int_or_none(format_id[:-1]) if format_id else None,
+                'url': play_url,
+            })
+        self._sort_formats(formats)
+
+        info = {
+            'id': program_id,
+            'title': title,
+            'formats': formats,
+            'thumbnail': program_info.get('coverUrl'),
+            'is_live': True,
+        }
+        info.update(self._extract_streamer_info(live_info))
+        return info
+
+
+class TrovoVodIE(TrovoBaseIE):
+    _VALID_URL = TrovoBaseIE._VALID_URL_BASE + r'(?:clip|video)/(?P<id>[^/?&#]+)'
+    _TESTS = [{
+        'url': 'https://trovo.live/video/ltv-100095501_100095501_1609596043',
+        'info_dict': {
+            'id': 'ltv-100095501_100095501_1609596043',
+            'ext': 'mp4',
+            'title': 'Spontaner 12 Stunden Stream! - Ok Boomer!',
+            'uploader': 'Exsl',
+            'timestamp': 1609640305,
+            'upload_date': '20210103',
+            'uploader_id': '100095501',
+            'duration': 43977,
+            'view_count': int,
+            'like_count': int,
+            'comment_count': int,
+            'comments': 'mincount:8',
+            'categories': ['Grand Theft Auto V'],
+        },
+    }, {
+        'url': 'https://trovo.live/clip/lc-5285890810184026005',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        vid = self._match_id(url)
+        resp = self._download_json(
+            'https://gql.trovo.live/', vid, data=json.dumps([{
+                'query': '''{
+  batchGetVodDetailInfo(params: {vids: ["%s"]}) {
+    VodDetailInfos
+  }
+}''' % vid,
+            }, {
+                'query': '''{
+  getCommentList(params: {appInfo: {postID: "%s"}, pageSize: 1000000000, preview: {}}) {
+    commentList {
+      author {
+        nickName
+        uid
+      }
+      commentID
+      content
+      createdAt
+      parentID
+    }
+  }
+}''' % vid,
+            }]).encode(), headers={
+                'Content-Type': 'application/json',
+            })
+        vod_detail_info = resp[0]['data']['batchGetVodDetailInfo']['VodDetailInfos'][vid]
+        vod_info = vod_detail_info['vodInfo']
+        title = vod_info['title']
+
+        language = vod_info.get('languageName')
+        formats = []
+        for play_info in (vod_info.get('playInfos') or []):
+            play_url = play_info.get('playUrl')
+            if not play_url:
+                continue
+            format_id = play_info.get('desc')
+            formats.append({
+                'ext': 'mp4',
+                'filesize': int_or_none(play_info.get('fileSize')),
+                'format_id': format_id,
+                'height': int_or_none(format_id[:-1]) if format_id else None,
+                'language': language,
+                'protocol': 'm3u8_native',
+                'tbr': int_or_none(play_info.get('bitrate')),
+                'url': play_url,
+            })
+        self._sort_formats(formats)
+
+        category = vod_info.get('categoryName')
+        get_count = lambda x: int_or_none(vod_info.get(x + 'Num'))
+
+        comment_list = try_get(resp, lambda x: x[1]['data']['getCommentList']['commentList'], list) or []
+        comments = []
+        for comment in comment_list:
+            content = comment.get('content')
+            if not content:
+                continue
+            author = comment.get('author') or {}
+            parent = comment.get('parentID')
+            comments.append({
+                'author': author.get('nickName'),
+                'author_id': str_or_none(author.get('uid')),
+                'id': str_or_none(comment.get('commentID')),
+                'text': content,
+                'timestamp': int_or_none(comment.get('createdAt')),
+                'parent': 'root' if parent == 0 else str_or_none(parent),
+            })
+
+        info = {
+            'id': vid,
+            'title': title,
+            'formats': formats,
+            'thumbnail': vod_info.get('coverUrl'),
+            'timestamp': int_or_none(vod_info.get('publishTs')),
+            'duration': int_or_none(vod_info.get('duration')),
+            'view_count': get_count('watch'),
+            'like_count': get_count('like'),
+            'comment_count': get_count('comment'),
+            'comments': comments,
+            'categories': [category] if category else None,
+        }
+        info.update(self._extract_streamer_info(vod_detail_info))
+        return info

youtube_dlc/extractor/wat.py

@@ -1,12 +1,9 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
-
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
-    ExtractorError,
     unified_strdate,
     HEADRequest,
     int_or_none,
@@ -46,15 +43,6 @@ class WatIE(InfoExtractor):
         },
     ]
 
-    _FORMATS = (
-        (200, 416, 234),
-        (400, 480, 270),
-        (600, 640, 360),
-        (1200, 640, 360),
-        (1800, 960, 540),
-        (2500, 1280, 720),
-    )
-
     def _real_extract(self, url):
         video_id = self._match_id(url)
         video_id = video_id if video_id.isdigit() and len(video_id) > 6 else compat_str(int(video_id, 36))
@@ -97,46 +85,20 @@ class WatIE(InfoExtractor):
                     return red_url
             return None
 
-        def remove_bitrate_limit(manifest_url):
-            return re.sub(r'(?:max|min)_bitrate=\d+&?', '', manifest_url)
-
         formats = []
-        try:
+        manifest_urls = self._download_json(
-            alt_urls = lambda manifest_url: [re.sub(r'(?:wdv|ssm)?\.ism/', repl + '.ism/', manifest_url) for repl in ('', 'ssm')]
+            'http://www.wat.tv/get/webhtml/' + video_id, video_id)
-            manifest_urls = self._download_json(
+        m3u8_url = manifest_urls.get('hls')
-                'http://www.wat.tv/get/webhtml/' + video_id, video_id)
+        if m3u8_url:
-            m3u8_url = manifest_urls.get('hls')
+            formats.extend(self._extract_m3u8_formats(
-            if m3u8_url:
+                m3u8_url, video_id, 'mp4',
-                m3u8_url = remove_bitrate_limit(m3u8_url)
+                'm3u8_native', m3u8_id='hls', fatal=False))
-                for m3u8_alt_url in alt_urls(m3u8_url):
+        mpd_url = manifest_urls.get('mpd')
-                    formats.extend(self._extract_m3u8_formats(
+        if mpd_url:
-                        m3u8_alt_url, video_id, 'mp4',
+            formats.extend(self._extract_mpd_formats(
-                        'm3u8_native', m3u8_id='hls', fatal=False))
+                mpd_url.replace('://das-q1.tf1.fr/', '://das-q1-ssl.tf1.fr/'),
-                    formats.extend(self._extract_f4m_formats(
+                video_id, mpd_id='dash', fatal=False))
-                        m3u8_alt_url.replace('ios', 'web').replace('.m3u8', '.f4m'),
+        self._sort_formats(formats)
-                        video_id, f4m_id='hds', fatal=False))
-            mpd_url = manifest_urls.get('mpd')
-            if mpd_url:
-                mpd_url = remove_bitrate_limit(mpd_url)
-                for mpd_alt_url in alt_urls(mpd_url):
-                    formats.extend(self._extract_mpd_formats(
-                        mpd_alt_url, video_id, mpd_id='dash', fatal=False))
-            self._sort_formats(formats)
-        except ExtractorError:
-            abr = 64
-            for vbr, width, height in self._FORMATS:
-                tbr = vbr + abr
-                format_id = 'http-%s' % tbr
-                fmt_url = 'http://dnl.adv.tf1.fr/2/USP-0x0/%s/%s/%s/ssm/%s-%s-64k.mp4' % (video_id[-4:-2], video_id[-2:], video_id, video_id, vbr)
-                if self._is_valid_url(fmt_url, video_id, format_id):
-                    formats.append({
-                        'format_id': format_id,
-                        'url': fmt_url,
-                        'vbr': vbr,
-                        'abr': abr,
-                        'width': width,
-                        'height': height,
-                    })
 
         date_diffusion = first_chapter.get('date_diffusion') or video_data.get('configv4', {}).get('estatS4')
         upload_date = unified_strdate(date_diffusion) if date_diffusion else None

youtube_dlc/extractor/yahoo.py

@@ -177,46 +177,9 @@ class YahooIE(InfoExtractor):
         'only_matching': True,
     }]
 
-    def _real_extract(self, url):
+    def _extract_yahoo_video(self, video_id, country):
-        url, country, display_id = re.match(self._VALID_URL, url).groups()
-        if not country:
-            country = 'us'
-        else:
-            country = country.split('-')[0]
-        api_base = 'https://%s.yahoo.com/_td/api/resource/' % country
-
-        for i, uuid in enumerate(['url=' + url, 'ymedia-alias=' + display_id]):
-            content = self._download_json(
-                api_base + 'content;getDetailView=true;uuids=["%s"]' % uuid,
-                display_id, 'Downloading content JSON metadata', fatal=i == 1)
-            if content:
-                item = content['items'][0]
-                break
-
-        if item.get('type') != 'video':
-            entries = []
-
-            cover = item.get('cover') or {}
-            if cover.get('type') == 'yvideo':
-                cover_url = cover.get('url')
-                if cover_url:
-                    entries.append(self.url_result(
-                        cover_url, 'Yahoo', cover.get('uuid')))
-
-            for e in item.get('body', []):
-                if e.get('type') == 'videoIframe':
-                    iframe_url = e.get('url')
-                    if not iframe_url:
-                        continue
-                    entries.append(self.url_result(iframe_url))
-
-            return self.playlist_result(
-                entries, item.get('uuid'),
-                item.get('title'), item.get('summary'))
-
-        video_id = item['uuid']
         video = self._download_json(
-            api_base + 'VideoService.videos;view=full;video_ids=["%s"]' % video_id,
+            'https://%s.yahoo.com/_td/api/resource/VideoService.videos;view=full;video_ids=["%s"]' % (country, video_id),
             video_id, 'Downloading video JSON metadata')[0]
         title = video['title']
 
@@ -298,7 +261,6 @@ class YahooIE(InfoExtractor):
             'id': video_id,
             'title': self._live_title(title) if is_live else title,
             'formats': formats,
-            'display_id': display_id,
             'thumbnails': thumbnails,
             'description': clean_html(video.get('description')),
             'timestamp': parse_iso8601(video.get('publish_time')),
@@ -311,6 +273,44 @@ class YahooIE(InfoExtractor):
             'episode_number': int_or_none(series_info.get('episode_number')),
         }
 
+    def _real_extract(self, url):
+        url, country, display_id = re.match(self._VALID_URL, url).groups()
+        if not country:
+            country = 'us'
+        else:
+            country = country.split('-')[0]
+
+        item = self._download_json(
+            'https://%s.yahoo.com/caas/content/article' % country, display_id,
+            'Downloading content JSON metadata', query={
+                'url': url
+            })['items'][0]['data']['partnerData']
+
+        if item.get('type') != 'video':
+            entries = []
+
+            cover = item.get('cover') or {}
+            if cover.get('type') == 'yvideo':
+                cover_url = cover.get('url')
+                if cover_url:
+                    entries.append(self.url_result(
+                        cover_url, 'Yahoo', cover.get('uuid')))
+
+            for e in (item.get('body') or []):
+                if e.get('type') == 'videoIframe':
+                    iframe_url = e.get('url')
+                    if not iframe_url:
+                        continue
+                    entries.append(self.url_result(iframe_url))
+
+            return self.playlist_result(
+                entries, item.get('uuid'),
+                item.get('title'), item.get('summary'))
+
+        info = self._extract_yahoo_video(item['uuid'], country)
+        info['display_id'] = display_id
+        return info
+
 
 class YahooSearchIE(SearchInfoExtractor):
     IE_DESC = 'Yahoo screen search'

youtube_dlc/options.py

@@ -14,6 +14,8 @@ from .compat import (
     compat_shlex_split,
 )
 from .utils import (
+    expand_path,
+    get_executable_path,
     preferredencoding,
     write_string,
 )
@@ -62,7 +64,7 @@ def parseOpts(overrideArguments=None):
             userConfFile = os.path.join(xdg_config_home, '%s.conf' % package_name)
         userConf = _readOptions(userConfFile, default=None)
         if userConf is not None:
-            return userConf
+            return userConf, userConfFile
 
         # appdata
         appdata_dir = compat_getenv('appdata')
@@ -70,19 +72,21 @@ def parseOpts(overrideArguments=None):
             userConfFile = os.path.join(appdata_dir, package_name, 'config')
             userConf = _readOptions(userConfFile, default=None)
             if userConf is None:
-                userConf = _readOptions('%s.txt' % userConfFile, default=None)
+                userConfFile += '.txt'
+                userConf = _readOptions(userConfFile, default=None)
         if userConf is not None:
-            return userConf
+            return userConf, userConfFile
 
         # home
         userConfFile = os.path.join(compat_expanduser('~'), '%s.conf' % package_name)
         userConf = _readOptions(userConfFile, default=None)
         if userConf is None:
-            userConf = _readOptions('%s.txt' % userConfFile, default=None)
+            userConfFile += '.txt'
+            userConf = _readOptions(userConfFile, default=None)
         if userConf is not None:
-            return userConf
+            return userConf, userConfFile
 
-        return default
+        return default, None
 
     def _format_option_string(option):
         ''' ('-o', '--option') -> -o, --format METAVAR'''
@@ -104,6 +108,20 @@ def parseOpts(overrideArguments=None):
     def _comma_separated_values_options_callback(option, opt_str, value, parser):
         setattr(parser.values, option.dest, value.split(','))
 
+    def _dict_from_multiple_values_options_callback(
+            option, opt_str, value, parser, allowed_keys=r'[\w-]+', delimiter=':', default_key=None, process=None):
+
+        out_dict = getattr(parser.values, option.dest)
+        mobj = re.match(r'(?i)(?P<key>%s)%s(?P<val>.*)$' % (allowed_keys, delimiter), value)
+        if mobj is not None:
+            key, val = mobj.group('key').lower(), mobj.group('val')
+        elif default_key is not None:
+            key, val = default_key, value
+        else:
+            raise optparse.OptionValueError(
+                'wrong %s formatting; it should be %s, not "%s"' % (opt_str, option.metavar, value))
+        out_dict[key] = process(val) if callable(process) else val
+
     # No need to wrap help messages if we're on a wide console
     columns = compat_get_terminal_size().columns
     max_width = columns if columns else 80
@@ -173,7 +191,7 @@ def parseOpts(overrideArguments=None):
     general.add_option(
         '--config-location',
         dest='config_location', metavar='PATH',
-        help='Location of the configuration file; either the path to the config or its containing directory')
+        help='Location of the main configuration file; either the path to the config or its containing directory')
     general.add_option(
         '--flat-playlist',
         action='store_const', dest='extract_flat', const='in_playlist', default=False,
@@ -618,14 +636,21 @@ def parseOpts(overrideArguments=None):
             'video while downloading (some players may not be able to play it)'))
     downloader.add_option(
         '--external-downloader',
-        dest='external_downloader', metavar='COMMAND',
+        dest='external_downloader', metavar='NAME',
         help=(
             'Use the specified external downloader. '
-            'Currently supports %s' % ','.join(list_external_downloaders())))
+            'Currently supports %s' % ', '.join(list_external_downloaders())))
     downloader.add_option(
-        '--external-downloader-args',
+        '--downloader-args', '--external-downloader-args',
-        dest='external_downloader_args', metavar='ARGS',
+        metavar='NAME:ARGS', dest='external_downloader_args', default={}, type='str',
-        help='Give these arguments to the external downloader')
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        callback_kwargs={
+            'allowed_keys': '|'.join(list_external_downloaders()),
+            'default_key': 'default', 'process': compat_shlex_split},
+        help=(
+            'Give these arguments to the external downloader. '
+            'Specify the downloader name and the arguments separated by a colon ":". '
+            'You can use this option multiple times (Alias: --external-downloader-args)'))
 
     workarounds = optparse.OptionGroup(parser, 'Workarounds')
     workarounds.add_option(
@@ -651,8 +676,9 @@ def parseOpts(overrideArguments=None):
     )
     workarounds.add_option(
         '--add-header',
-        metavar='FIELD:VALUE', dest='headers', action='append',
+        metavar='FIELD:VALUE', dest='headers', default={}, type='str',
-        help='Specify a custom HTTP header and its value, separated by a colon \':\'. You can use this option multiple times',
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        help='Specify a custom HTTP header and its value, separated by a colon ":". You can use this option multiple times',
     )
     workarounds.add_option(
         '--bidi-workaround',
@@ -797,10 +823,29 @@ def parseOpts(overrideArguments=None):
     filesystem.add_option(
         '--id', default=False,
         action='store_true', dest='useid', help=optparse.SUPPRESS_HELP)
+    filesystem.add_option(
+        '-P', '--paths',
+        metavar='TYPE:PATH', dest='paths', default={}, type='str',
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        callback_kwargs={
+            'allowed_keys': 'home|temp|config|description|annotation|subtitle|infojson|thumbnail',
+            'process': lambda x: x.strip()},
+        help=(
+            'The paths where the files should be downloaded. '
+            'Specify the type of file and the path separated by a colon ":" '
+            '(supported: description|annotation|subtitle|infojson|thumbnail). '
+            'Additionally, you can also provide "home" and "temp" paths. '
+            'All intermediary files are first downloaded to the temp path and '
+            'then the final files are moved over to the home path after download is finished. '
+            'Note that this option is ignored if --output is an absolute path'))
     filesystem.add_option(
         '-o', '--output',
         dest='outtmpl', metavar='TEMPLATE',
         help='Output filename template, see "OUTPUT TEMPLATE" for details')
+    filesystem.add_option(
+        '--output-na-placeholder',
+        dest='outtmpl_na_placeholder', metavar='TEXT', default='NA',
+        help=('Placeholder value for unavailable meta fields in output filename template (default: "%default")'))
     filesystem.add_option(
         '--autonumber-size',
         dest='autonumber_size', metavar='NUMBER', type=int,
@@ -956,7 +1001,7 @@ def parseOpts(overrideArguments=None):
     postproc.add_option(
         '-x', '--extract-audio',
         action='store_true', dest='extractaudio', default=False,
-        help='Convert video files to audio-only files (requires ffmpeg or avconv and ffprobe or avprobe)')
+        help='Convert video files to audio-only files (requires ffmpeg/avconv and ffprobe/avprobe)')
     postproc.add_option(
         '--audio-format', metavar='FORMAT', dest='audioformat', default='best',
         help='Specify audio format: "best", "aac", "flac", "mp3", "m4a", "opus", "vorbis", or "wav"; "%default" by default; No effect without -x')
@@ -975,18 +1020,21 @@ def parseOpts(overrideArguments=None):
         metavar='FORMAT', dest='recodevideo', default=None,
         help='Re-encode the video into another format if re-encoding is necessary (currently supported: mp4|flv|ogg|webm|mkv|avi)')
     postproc.add_option(
-        '--postprocessor-args', '--ppa', metavar='NAME:ARGS',
+        '--postprocessor-args', '--ppa',
-        dest='postprocessor_args', action='append',
+        metavar='NAME:ARGS', dest='postprocessor_args', default={}, type='str',
+        action='callback', callback=_dict_from_multiple_values_options_callback,
+        callback_kwargs={'default_key': 'default-compat', 'allowed_keys': r'\w+(?:\+\w+)?', 'process': compat_shlex_split},
         help=(
             'Give these arguments to the postprocessors. '
             'Specify the postprocessor/executable name and the arguments separated by a colon ":" '
-            'to give the argument to only the specified postprocessor/executable. Supported postprocessors are: '
+            'to give the argument to the specified postprocessor/executable. Supported postprocessors are: '
             'SponSkrub, ExtractAudio, VideoRemuxer, VideoConvertor, EmbedSubtitle, Metadata, Merger, '
             'FixupStretched, FixupM4a, FixupM3u8, SubtitlesConvertor and EmbedThumbnail. '
             'The supported executables are: SponSkrub, FFmpeg, FFprobe, avconf, avprobe and AtomicParsley. '
             'You can use this option multiple times to give different arguments to different postprocessors. '
             'You can also specify "PP+EXE:ARGS" to give the arguments to the specified executable '
-            'only when being used by the specified postprocessor (Alias: --ppa)'))
+            'only when being used by the specified postprocessor. '
+            'You can use this option multiple times (Alias: --ppa)'))
     postproc.add_option(
         '-k', '--keep-video',
         action='store_true', dest='keepvideo', default=False,
@@ -1146,59 +1194,73 @@ def parseOpts(overrideArguments=None):
             return conf
 
         configs = {
-            'command_line': compat_conf(sys.argv[1:]),
+            'command-line': compat_conf(sys.argv[1:]),
-            'custom': [], 'portable': [], 'user': [], 'system': []}
+            'custom': [], 'home': [], 'portable': [], 'user': [], 'system': []}
-        opts, args = parser.parse_args(configs['command_line'])
+        paths = {'command-line': False}
+        opts, args = parser.parse_args(configs['command-line'])
 
         def get_configs():
-            if '--config-location' in configs['command_line']:
+            if '--config-location' in configs['command-line']:
                 location = compat_expanduser(opts.config_location)
                 if os.path.isdir(location):
                     location = os.path.join(location, 'youtube-dlc.conf')
                 if not os.path.exists(location):
                     parser.error('config-location %s does not exist.' % location)
-                configs['custom'] = _readOptions(location)
+                configs['custom'] = _readOptions(location, default=None)
-
+                if configs['custom'] is None:
-            if '--ignore-config' in configs['command_line']:
+                    configs['custom'] = []
+                else:
+                    paths['custom'] = location
+            if '--ignore-config' in configs['command-line']:
                 return
             if '--ignore-config' in configs['custom']:
                 return
 
-            def get_portable_path():
+            def read_options(path, user=False):
-                path = os.path.dirname(sys.argv[0])
+                func = _readUserConf if user else _readOptions
-                if os.path.abspath(sys.argv[0]) != os.path.abspath(sys.executable):  # Not packaged
+                current_path = os.path.join(path, 'yt-dlp.conf')
-                    path = os.path.join(path, '..')
+                config = func(current_path, default=None)
-                return os.path.abspath(path)
+                if user:
-
+                    config, current_path = config
-            run_path = get_portable_path()
+                if config is None:
-            configs['portable'] = _readOptions(os.path.join(run_path, 'yt-dlp.conf'), default=None)
+                    current_path = os.path.join(path, 'youtube-dlc.conf')
-            if configs['portable'] is None:
+                    config = func(current_path, default=None)
-                configs['portable'] = _readOptions(os.path.join(run_path, 'youtube-dlc.conf'))
+                    if user:
+                        config, current_path = config
+                if config is None:
+                    return [], None
+                return config, current_path
 
+            configs['portable'], paths['portable'] = read_options(get_executable_path())
             if '--ignore-config' in configs['portable']:
                 return
-            configs['system'] = _readOptions('/etc/yt-dlp.conf', default=None)
-            if configs['system'] is None:
-                configs['system'] = _readOptions('/etc/youtube-dlc.conf')
 
+            def get_home_path():
+                opts = parser.parse_args(configs['portable'] + configs['custom'] + configs['command-line'])[0]
+                return expand_path(opts.paths.get('home', '')).strip()
+
+            configs['home'], paths['home'] = read_options(get_home_path())
+            if '--ignore-config' in configs['home']:
+                return
+
+            configs['system'], paths['system'] = read_options('/etc')
             if '--ignore-config' in configs['system']:
                 return
-            configs['user'] = _readUserConf('yt-dlp', default=None)
+
-            if configs['user'] is None:
+            configs['user'], paths['user'] = read_options('', True)
-                configs['user'] = _readUserConf('youtube-dlc')
             if '--ignore-config' in configs['user']:
-                configs['system'] = []
+                configs['system'], paths['system'] = [], None
 
         get_configs()
-        argv = configs['system'] + configs['user'] + configs['portable'] + configs['custom'] + configs['command_line']
+        argv = configs['system'] + configs['user'] + configs['home'] + configs['portable'] + configs['custom'] + configs['command-line']
         opts, args = parser.parse_args(argv)
         if opts.verbose:
-            for conf_label, conf in (
+            for label in ('System', 'User', 'Portable', 'Home', 'Custom', 'Command-line'):
-                    ('System config', configs['system']),
+                key = label.lower()
-                    ('User config', configs['user']),
+                if paths.get(key) is None:
-                    ('Portable config', configs['portable']),
+                    continue
-                    ('Custom config', configs['custom']),
+                if paths[key]:
-                    ('Command-line args', configs['command_line'])):
+                    write_string('[debug] %s config file: %s\n' % (label, paths[key]))
-                write_string('[debug] %s: %s\n' % (conf_label, repr(_hide_login_info(conf))))
+                write_string('[debug] %s config: %s\n' % (label, repr(_hide_login_info(configs[key]))))
 
     return parser, opts, args

youtube_dlc/postprocessor/__init__.py

@@ -17,6 +17,7 @@ from .ffmpeg import (
 from .xattrpp import XAttrMetadataPP
 from .execafterdownload import ExecAfterDownloadPP
 from .metadatafromtitle import MetadataFromTitlePP
+from .movefilesafterdownload import MoveFilesAfterDownloadPP
 from .sponskrub import SponSkrubPP
 
 
@@ -39,6 +40,7 @@ __all__ = [
     'FFmpegVideoConvertorPP',
     'FFmpegVideoRemuxerPP',
     'MetadataFromTitlePP',
+    'MoveFilesAfterDownloadPP',
     'SponSkrubPP',
     'XAttrMetadataPP',
 ]

youtube_dlc/postprocessor/common.py

@@ -4,8 +4,9 @@ import os
 
 from ..compat import compat_str
 from ..utils import (
-    PostProcessingError,
+    cli_configuration_args,
     encodeFilename,
+    PostProcessingError,
 )
 
 
@@ -91,39 +92,10 @@ class PostProcessor(object):
             self.report_warning(errnote)
 
     def _configuration_args(self, default=[], exe=None):
-        args = self.get_param('postprocessor_args', {})
+        key = self.pp_key().lower()
-        pp_key = self.pp_key().lower()
+        args, is_compat = cli_configuration_args(
-
+            self._downloader.params, 'postprocessor_args', key, default, exe)
-        if isinstance(args, (list, tuple)):  # for backward compatibility
+        return args if not is_compat or key != 'sponskrub' else default
-            return default if pp_key == 'sponskrub' else args
-        if args is None:
-            return default
-        assert isinstance(args, dict)
-
-        exe_args = None
-        if exe is not None:
-            assert isinstance(exe, compat_str)
-            exe = exe.lower()
-            specific_args = args.get('%s+%s' % (pp_key, exe))
-            if specific_args is not None:
-                assert isinstance(specific_args, (list, tuple))
-                return specific_args
-            exe_args = args.get(exe)
-
-        pp_args = args.get(pp_key) if pp_key != exe else None
-        if pp_args is None and exe_args is None:
-            default = args.get('default', default)
-            assert isinstance(default, (list, tuple))
-            return default
-
-        if pp_args is None:
-            pp_args = []
-        elif exe_args is None:
-            exe_args = []
-
-        assert isinstance(pp_args, (list, tuple))
-        assert isinstance(exe_args, (list, tuple))
-        return pp_args + exe_args
 
 
 class AudioConversionError(PostProcessingError):

youtube_dlc/postprocessor/movefilesafterdownload.py

@@ -0,0 +1,53 @@
+from __future__ import unicode_literals
+import os
+import shutil
+
+from .common import PostProcessor
+from ..utils import (
+    encodeFilename,
+    make_dir,
+    PostProcessingError,
+)
+from ..compat import compat_str
+
+
+class MoveFilesAfterDownloadPP(PostProcessor):
+
+    def __init__(self, downloader, files_to_move):
+        PostProcessor.__init__(self, downloader)
+        self.files_to_move = files_to_move
+
+    @classmethod
+    def pp_key(cls):
+        return 'MoveFiles'
+
+    def run(self, info):
+        dl_path, dl_name = os.path.split(encodeFilename(info['filepath']))
+        finaldir = info.get('__finaldir', dl_path)
+        finalpath = os.path.join(finaldir, dl_name)
+        self.files_to_move[info['filepath']] = finalpath
+
+        for oldfile, newfile in self.files_to_move.items():
+            if not os.path.exists(encodeFilename(oldfile)):
+                self.report_warning('File "%s" cannot be found' % oldfile)
+                continue
+            if not newfile:
+                newfile = os.path.join(finaldir, os.path.basename(encodeFilename(oldfile)))
+            oldfile, newfile = compat_str(oldfile), compat_str(newfile)
+            if os.path.abspath(encodeFilename(oldfile)) == os.path.abspath(encodeFilename(newfile)):
+                continue
+            if os.path.exists(encodeFilename(newfile)):
+                if self.get_param('overwrites', True):
+                    self.report_warning('Replacing existing file "%s"' % newfile)
+                    os.path.remove(encodeFilename(newfile))
+                else:
+                    self.report_warning(
+                        'Cannot move file "%s" out of temporary directory since "%s" already exists. '
+                        % (oldfile, newfile))
+                    continue
+            make_dir(newfile, PostProcessingError)
+            self.to_screen('Moving file "%s" to "%s"' % (oldfile, newfile))
+            shutil.move(oldfile, newfile)  # os.rename cannot move between volumes
+
+        info['filepath'] = compat_str(finalpath)
+        return [], info

youtube_dlc/postprocessor/sponskrub.py

@@ -84,6 +84,7 @@ class SponSkrubPP(PostProcessor):
         else:
             msg = stderr.decode('utf-8', 'replace').strip() or stdout.decode('utf-8', 'replace').strip()
             self.write_debug(msg, prefix=False)
-            msg = msg.split('\n')[-1]
+            line = 0 if msg[:12].lower() == 'unrecognised' else -1
+            msg = msg.split('\n')[line]
             raise PostProcessingError(msg if msg else 'sponskrub failed with error code %s' % p.returncode)
         return [], information

youtube_dlc/utils.py

@@ -16,6 +16,7 @@ import email.header
 import errno
 import functools
 import gzip
+import imp
 import io
 import itertools
 import json
@@ -4656,12 +4657,35 @@ def cli_valueless_option(params, command_option, param, expected_value=True):
     return [command_option] if param == expected_value else []
 
 
-def cli_configuration_args(params, param, default=[]):
+def cli_configuration_args(params, arg_name, key, default=[], exe=None):  # returns arg, for_compat
-    ex_args = params.get(param)
+    argdict = params.get(arg_name, {})
-    if ex_args is None:
+    if isinstance(argdict, (list, tuple)):  # for backward compatibility
-        return default
+        return argdict, True
-    assert isinstance(ex_args, list)
+
-    return ex_args
+    if argdict is None:
+        return default, False
+    assert isinstance(argdict, dict)
+
+    assert isinstance(key, compat_str)
+    key = key.lower()
+
+    args = exe_args = None
+    if exe is not None:
+        assert isinstance(exe, compat_str)
+        exe = exe.lower()
+        args = argdict.get('%s+%s' % (key, exe))
+        if args is None:
+            exe_args = argdict.get(exe)
+
+    if args is None:
+        args = argdict.get(key) if key != exe else None
+    if args is None and exe_args is None:
+        args = argdict.get('default', default)
+
+    args, exe_args = args or [], exe_args or []
+    assert isinstance(args, (list, tuple))
+    assert isinstance(exe_args, (list, tuple))
+    return args + exe_args, False
 
 
 class ISO639Utils(object):
@@ -5863,3 +5887,50 @@ def clean_podcast_url(url):
                 st\.fm # https://podsights.com/docs/
             )/e
         )/''', '', url)
+
+
+_HEX_TABLE = '0123456789abcdef'
+
+
+def random_uuidv4():
+    return re.sub(r'[xy]', lambda x: _HEX_TABLE[random.randint(0, 15)], 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx')
+
+
+def make_dir(path, to_screen=None):
+    try:
+        dn = os.path.dirname(path)
+        if dn and not os.path.exists(dn):
+            os.makedirs(dn)
+        return True
+    except (OSError, IOError) as err:
+        if callable(to_screen) is not None:
+            to_screen('unable to create directory ' + error_to_compat_str(err))
+        return False
+
+
+def get_executable_path():
+    path = os.path.dirname(sys.argv[0])
+    if os.path.abspath(sys.argv[0]) != os.path.abspath(sys.executable):  # Not packaged
+        path = os.path.join(path, '..')
+    return os.path.abspath(path)
+
+
+def load_plugins(name, type, namespace):
+    plugin_info = [None]
+    classes = []
+    try:
+        plugin_info = imp.find_module(
+            name, [os.path.join(get_executable_path(), 'ytdlp_plugins')])
+        plugins = imp.load_module(name, *plugin_info)
+        for name in dir(plugins):
+            if not name.endswith(type):
+                continue
+            klass = getattr(plugins, name)
+            classes.append(klass)
+            namespace[name] = klass
+    except ImportError:
+        pass
+    finally:
+        if plugin_info[0] is not None:
+            plugin_info[0].close()
+    return classes

youtube_dlc/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.01.16'
+__version__ = '2021.01.20'

ytdlp_plugins/extractor/__init__.py

@@ -0,0 +1,2 @@
+# flake8: noqa
+from .sample import SamplePluginIE

ytdlp_plugins/extractor/sample.py

@@ -0,0 +1,12 @@
+from __future__ import unicode_literals
+
+from youtube_dlc.extractor.common import InfoExtractor
+
+
+class SamplePluginIE(InfoExtractor):
+    _WORKING = False
+    IE_DESC = False
+    _VALID_URL = r'^sampleplugin:'
+
+    def _real_extract(self, url):
+        self.to_screen('URL "%s" sucessfully captured' % url)