Archives
/
wikiteam
mirror of https://github.com/WikiTeam/wikiteam
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #171 from mediawiki-client-tools/fix-pre-commit

Browse Source 
Fix Pre-Commit
pull/475/head
Elsie Hupp 9 months ago committed by GitHub
parent 6e4362acf6 eeab93a488
commit 9c65c5a719
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
66 changed files with 3130 additions and 2551 deletions
Show Stats Download Patch File Download Diff File

32
.github/ISSUE_TEMPLATE/bug_report.md vendored
Unescape Escape View File

@ -1,5 +1,5 @@
---
name: Bug report
name: Bug Report
about: Create a report to help us improve
title: ''
labels: ''
@ -30,7 +30,7 @@ please fill out as much of the following as possible. -->
(Please copy and paste within the code block below.) -->
```bash
$
```
## Output
@ -44,7 +44,7 @@ $
```bash
```
</details>
<details>
@ -53,10 +53,10 @@ $
<!-- The errors.log file from the dump folder, if there is one
(Please copy and paste within the code block below.) -->
```
```text
```
</details>
## Platform Details
@ -66,19 +66,19 @@ able to and remove whichever section is inapplicable -->
### Desktop
- OS and version: <!-- e.g. Kubuntu 23.04, Windows 10, macOS 14.2 -->
- File system: <!-- e.g. EXT4, NTFS, APFS -->
- Python version: <!-- `$ python --version` -->
- Command line shell: <!-- `$ $SHELL --version` -->
- `dumpgenerator` version: <!-- `$ dumpgenerator -v` -->
- OS and version: <!-- e.g. Kubuntu 23.04, Windows 10, macOS 14.2 -->
- File system: <!-- e.g. EXT4, NTFS, APFS -->
- Python version: <!-- `$ python --version` -->
- Command line shell: <!-- `$ $SHELL --version` -->
- `dumpgenerator` version: <!-- `$ dumpgenerator -v` -->
### Smartphone or Tablet
- OS: <!-- e.g. iOS 16.1, Android 11 -->
- Python version: <!-- `$ python --version` -->
- Command line shell: <!-- `$ $SHELL --version` -->
- Terminal application used: <!-- e.g. Termux, Termius -->
- `dumpgenerator` version: <!-- `$ dumpgenerator -v` -->
- OS: <!-- e.g. iOS 16.1, Android 11 -->
- Python version: <!-- `$ python --version` -->
- Command line shell: <!-- `$ $SHELL --version` -->
- Terminal application used: <!-- e.g. Termux, Termius -->
- `dumpgenerator` version: <!-- `$ dumpgenerator -v` -->
## Additional Context

16
.markdownlint.jsonc
Unescape Escape View File

@ -0,0 +1,16 @@
// If you change any options here,
// please change them in .pymarkdown.jsonc
// as well!
{
"line-length": false,
"no-inline-html": {
"allowed_elements": [
"details",
"summary",
"code"
]
},
"first-line-heading": {
"front_matter_title" : "name"
}
}

27
.pre-commit-config.yaml
Unescape Escape View File

@ -6,14 +6,14 @@ default_language_version:
python: python3.8
repos:
- repo: https://github.com/python-poetry/poetry
rev: 1.2.0b1
rev: 1.6.0
hooks:
- id: poetry-check
# - id: poetry-lock
- id: poetry-export
args: ["-f", "requirements.txt", "-o", "requirements.txt"]
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.1.0
rev: v4.4.0
hooks:
- id: check-ast
- id: fix-byte-order-marker
@ -34,27 +34,32 @@ repos:
# - id: mypy
# args: [--ignore-missing-imports]
- repo: https://github.com/PyCQA/isort
rev: 5.10.1
rev: 5.12.0
hooks:
- id: isort
args: ["--profile", "black", "--filter-files"]
- repo: https://github.com/psf/black
rev: 22.3.0
rev: 23.7.0
hooks:
- id: black
- repo: https://github.com/asottile/pyupgrade
rev: v2.31.1
rev: v3.10.1
hooks:
- id: pyupgrade
args: [--py38-plus]
- repo: https://github.com/asottile/blacken-docs
rev: v1.12.1
rev: 1.16.0
hooks:
- id: blacken-docs
additional_dependencies: [black==20.8b1]
# additional_dependencies: [black==20.8b1]
### Needs argument for diasabling line_length
### https://github.com/jackdewinter/pymarkdown/blob/main/docs/rules/rule_md013.md
# - repo: https://github.com/jackdewinter/pymarkdown
# rev: v0.9.5
# hooks:
# - id: pymarkdown
- repo: https://github.com/jackdewinter/pymarkdown
rev: v0.9.12
hooks:
- id: pymarkdown
args:
- --config=.pymarkdown.json
# - --disable-rules
# - line-length,no-inline-html
- scan

14
.pymarkdown.json
Unescape Escape View File

@ -0,0 +1,14 @@
{
"plugins": {
"line-length": {
"enabled": false
},
"no-inline-html": {
"allowed_elements": "details,summary,code,!--"
},
"first-line-heading": {
"enabled": false,
"front_matter_title" : "name"
}
}
}

13
README.md
Unescape Escape View File

@ -61,9 +61,9 @@ MediaWiki Scraper is a set of tools for archiving wikis. The main general-purpos
<details>
<summary>Windows Dependencies</summary>
The latest version of Python is available from [python.org](https://www.python.org/downloads/). Python will then be available from any Command Prompt or PowerShell session. Optionally, adding C:\Program Files\Git\usr\bin to the PATH environment variable will add some some useful Linux commands and utilities to Command Prompt.
If you are already using the [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/about), you can follow the Linux instructions above. If you don't want to install a full WSL distribution, [Git for Windows](https://gitforwindows.org/) provides Bash emulation, so you can use it as a more lightweight option instead. Git Bash also provides some useful Linux commands and utilities.
> When installing [Python 3.8](https://www.python.org/downloads/release/python-380/) (from python.org), be sure to check "Add Python to PATH" so that installed Python scripts are accessible from any location. If for some reason installed Python scripts, e.g. `pip`, are not available from any location, you can add Python to the `PATH` environment variable using the instructions [here](https://datatofish.com/add-python-to-windows-path/).
@ -185,6 +185,7 @@ test-dumpgenerator
```
### 7. Switching branches
```bash
git checkout --track origin/python3
```
@ -242,6 +243,7 @@ In the above example, `--path` is only necessary if the download path is not the
`launcher` is a way to download a large list of wikis with a single invocation.
Usage:
```bash
launcher path-to-apis.txt [--7z-path path-to-7z] [--generator-arg=--arg] ...
```
@ -252,7 +254,7 @@ launcher path-to-apis.txt [--7z-path path-to-7z] [--generator-arg=--arg] ...
Each wiki will be stored into files contiaining a stripped version of the url and the date the dump was started.
`path-to-apis.txt` is a path to a file that contains a list of URLs to `api.php`s of wikis, one on each line.
`path-to-apis.txt` is a path to a file that contains a list of URLs to `api.php`s of wikis, one on each line.
By default, a `7z` executable is found on `PATH`. The `--7z-path` argument can be used to use a specific executable instead.
@ -263,6 +265,7 @@ The `--generator-arg` argument can be used to pass through arguments to the `gen
`uploader` is a way to upload a large set of already-generated wiki dumps to the Internet Archive with a single invocation.
Usage:
```bash
uploader [-pd] [-pw] [-a] [-c COLLECTION] [-wd WIKIDUMP_DIR] [-u] [-kf KEYSFILE] [-lf LOGFILE] listfile
```
@ -272,6 +275,7 @@ For the positional parameter `listfile`, `uploader` expects a path to a file tha
`uploader` will search a configurable directory for files with the names generated by `launcher` and upload any that it finds to an Internet Archive item. The item will be created if it does not already exist.
Named arguments (short and long versions):
* `-pd`, `--prune_directories`: After uploading, remove the raw directory generated by `launcher`
* `-pw`, `--prune_wikidump`: After uploading, remove the `wikidump.7z` file generated by `launcher`
* `-c`, `--collection`: Assign the Internet Archive items to the specified collection
@ -289,7 +293,7 @@ If you want to check the XML dump integrity, type this into your command line to
grep -E '<title(.*?)>' *.xml -c;grep -E '<page(.*?)>' *.xml -c;grep \
"</page>" *.xml -c;grep -E '<revision(.*?)>' *.xml -c;grep "</revision>" *.xml -c
```
You should see something similar to this (not the actual numbers) - the first three numbers should be the same and the last two should be the same as each other:
```bash
@ -311,6 +315,7 @@ If your first three numbers or your last two numbers are different, then, your X
Please report any issues at [MediaWiki Scraper/issues](https://github.com/mediawiki-client-tools/mediawiki-scraper/issues)
Include the following in your report:
* The commandline you used, with full URL
* Dumpgenerator version, -v option
* Operating system version, e.g. Kubuntu 23.04 / Windows 10

549
poetry.lock generated
Unescape Escape View File

@ -1,9 +1,10 @@
# This file is automatically @generated by Poetry 1.5.1 and should not be changed by hand.
# This file is automatically @generated by Poetry and should not be changed by hand.
[[package]]
name = "atomicwrites"
version = "1.4.1"
description = "Atomic file writes."
category = "dev"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
files = [
@ -14,6 +15,7 @@ files = [
name = "attrs"
version = "23.1.0"
description = "Classes Without Boilerplate"
category = "dev"
optional = false
python-versions = ">=3.7"
files = [
@ -32,6 +34,7 @@ tests-no-zope = ["cloudpickle", "hypothesis", "mypy (>=1.1.1)", "pympler", "pyte
name = "certifi"
version = "2023.7.22"
description = "Python package for providing Mozilla's CA Bundle."
category = "main"
optional = false
python-versions = ">=3.6"
files = [
@ -41,103 +44,106 @@ files = [
[[package]]
name = "cfgv"
version = "3.3.1"
version = "3.4.0"
description = "Validate configuration and produce human readable error messages."
category = "dev"
optional = false
python-versions = ">=3.6.1"
python-versions = ">=3.8"
files = [
{file = "cfgv-3.3.1-py2.py3-none-any.whl", hash = "sha256:c6a0883f3917a037485059700b9e75da2464e6c27051014ad85ba6aaa5884426"},
{file = "cfgv-3.3.1.tar.gz", hash = "sha256:f5a830efb9ce7a445376bb66ec94c638a9787422f96264c98edc6bdeed8ab736"},
{file = "cfgv-3.4.0-py2.py3-none-any.whl", hash = "sha256:b7265b1f29fd3316bfcd2b330d63d024f2bfd8bcb8b0272f8e19a504856c48f9"},
{file = "cfgv-3.4.0.tar.gz", hash = "sha256:e52591d4c5f5dead8e0f673fb16db7949d2cfb3f7da4582893288f0ded8fe560"},
]
[[package]]
name = "charset-normalizer"
version = "3.1.0"
version = "3.2.0"
description = "The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet."
category = "main"
optional = false
python-versions = ">=3.7.0"
files = [
{file = "charset-normalizer-3.1.0.tar.gz", hash = "sha256:34e0a2f9c370eb95597aae63bf85eb5e96826d81e3dcf88b8886012906f509b5"},
{file = "charset_normalizer-3.1.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:e0ac8959c929593fee38da1c2b64ee9778733cdf03c482c9ff1d508b6b593b2b"},
{file = "charset_normalizer-3.1.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d7fc3fca01da18fbabe4625d64bb612b533533ed10045a2ac3dd194bfa656b60"},
{file = "charset_normalizer-3.1.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:04eefcee095f58eaabe6dc3cc2262f3bcd776d2c67005880894f447b3f2cb9c1"},
{file = "charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:20064ead0717cf9a73a6d1e779b23d149b53daf971169289ed2ed43a71e8d3b0"},
{file = "charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1435ae15108b1cb6fffbcea2af3d468683b7afed0169ad718451f8db5d1aff6f"},
{file = "charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c84132a54c750fda57729d1e2599bb598f5fa0344085dbde5003ba429a4798c0"},
{file = "charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:75f2568b4189dda1c567339b48cba4ac7384accb9c2a7ed655cd86b04055c795"},
{file = "charset_normalizer-3.1.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:11d3bcb7be35e7b1bba2c23beedac81ee893ac9871d0ba79effc7fc01167db6c"},
{file = "charset_normalizer-3.1.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:891cf9b48776b5c61c700b55a598621fdb7b1e301a550365571e9624f270c203"},
{file = "charset_normalizer-3.1.0-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:5f008525e02908b20e04707a4f704cd286d94718f48bb33edddc7d7b584dddc1"},
{file = "charset_normalizer-3.1.0-cp310-cp310-musllinux_1_1_ppc64le.whl", hash = "sha256:b06f0d3bf045158d2fb8837c5785fe9ff9b8c93358be64461a1089f5da983137"},
{file = "charset_normalizer-3.1.0-cp310-cp310-musllinux_1_1_s390x.whl", hash = "sha256:49919f8400b5e49e961f320c735388ee686a62327e773fa5b3ce6721f7e785ce"},
{file = "charset_normalizer-3.1.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:22908891a380d50738e1f978667536f6c6b526a2064156203d418f4856d6e86a"},
{file = "charset_normalizer-3.1.0-cp310-cp310-win32.whl", hash = "sha256:12d1a39aa6b8c6f6248bb54550efcc1c38ce0d8096a146638fd4738e42284448"},
{file = "charset_normalizer-3.1.0-cp310-cp310-win_amd64.whl", hash = "sha256:65ed923f84a6844de5fd29726b888e58c62820e0769b76565480e1fdc3d062f8"},
{file = "charset_normalizer-3.1.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:9a3267620866c9d17b959a84dd0bd2d45719b817245e49371ead79ed4f710d19"},
{file = "charset_normalizer-3.1.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6734e606355834f13445b6adc38b53c0fd45f1a56a9ba06c2058f86893ae8017"},
{file = "charset_normalizer-3.1.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f8303414c7b03f794347ad062c0516cee0e15f7a612abd0ce1e25caf6ceb47df"},
{file = "charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:aaf53a6cebad0eae578f062c7d462155eada9c172bd8c4d250b8c1d8eb7f916a"},
{file = "charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3dc5b6a8ecfdc5748a7e429782598e4f17ef378e3e272eeb1340ea57c9109f41"},
{file = "charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e1b25e3ad6c909f398df8921780d6a3d120d8c09466720226fc621605b6f92b1"},
{file = "charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0ca564606d2caafb0abe6d1b5311c2649e8071eb241b2d64e75a0d0065107e62"},
{file = "charset_normalizer-3.1.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b82fab78e0b1329e183a65260581de4375f619167478dddab510c6c6fb04d9b6"},
{file = "charset_normalizer-3.1.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:bd7163182133c0c7701b25e604cf1611c0d87712e56e88e7ee5d72deab3e76b5"},
{file = "charset_normalizer-3.1.0-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:11d117e6c63e8f495412d37e7dc2e2fff09c34b2d09dbe2bee3c6229577818be"},
{file = "charset_normalizer-3.1.0-cp311-cp311-musllinux_1_1_ppc64le.whl", hash = "sha256:cf6511efa4801b9b38dc5546d7547d5b5c6ef4b081c60b23e4d941d0eba9cbeb"},
{file = "charset_normalizer-3.1.0-cp311-cp311-musllinux_1_1_s390x.whl", hash = "sha256:abc1185d79f47c0a7aaf7e2412a0eb2c03b724581139193d2d82b3ad8cbb00ac"},
{file = "charset_normalizer-3.1.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:cb7b2ab0188829593b9de646545175547a70d9a6e2b63bf2cd87a0a391599324"},
{file = "charset_normalizer-3.1.0-cp311-cp311-win32.whl", hash = "sha256:c36bcbc0d5174a80d6cccf43a0ecaca44e81d25be4b7f90f0ed7bcfbb5a00909"},
{file = "charset_normalizer-3.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:cca4def576f47a09a943666b8f829606bcb17e2bc2d5911a46c8f8da45f56755"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:0c95f12b74681e9ae127728f7e5409cbbef9cd914d5896ef238cc779b8152373"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fca62a8301b605b954ad2e9c3666f9d97f63872aa4efcae5492baca2056b74ab"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:ac0aa6cd53ab9a31d397f8303f92c42f534693528fafbdb997c82bae6e477ad9"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c3af8e0f07399d3176b179f2e2634c3ce9c1301379a6b8c9c9aeecd481da494f"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3a5fc78f9e3f501a1614a98f7c54d3969f3ad9bba8ba3d9b438c3bc5d047dd28"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:628c985afb2c7d27a4800bfb609e03985aaecb42f955049957814e0491d4006d"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:74db0052d985cf37fa111828d0dd230776ac99c740e1a758ad99094be4f1803d"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:1e8fcdd8f672a1c4fc8d0bd3a2b576b152d2a349782d1eb0f6b8e52e9954731d"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-musllinux_1_1_ppc64le.whl", hash = "sha256:04afa6387e2b282cf78ff3dbce20f0cc071c12dc8f685bd40960cc68644cfea6"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-musllinux_1_1_s390x.whl", hash = "sha256:dd5653e67b149503c68c4018bf07e42eeed6b4e956b24c00ccdf93ac79cdff84"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:d2686f91611f9e17f4548dbf050e75b079bbc2a82be565832bc8ea9047b61c8c"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-win32.whl", hash = "sha256:4155b51ae05ed47199dc5b2a4e62abccb274cee6b01da5b895099b61b1982974"},
{file = "charset_normalizer-3.1.0-cp37-cp37m-win_amd64.whl", hash = "sha256:322102cdf1ab682ecc7d9b1c5eed4ec59657a65e1c146a0da342b78f4112db23"},
{file = "charset_normalizer-3.1.0-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:e633940f28c1e913615fd624fcdd72fdba807bf53ea6925d6a588e84e1151531"},
{file = "charset_normalizer-3.1.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:3a06f32c9634a8705f4ca9946d667609f52cf130d5548881401f1eb2c39b1e2c"},
{file = "charset_normalizer-3.1.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:7381c66e0561c5757ffe616af869b916c8b4e42b367ab29fedc98481d1e74e14"},
{file = "charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3573d376454d956553c356df45bb824262c397c6e26ce43e8203c4c540ee0acb"},
{file = "charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e89df2958e5159b811af9ff0f92614dabf4ff617c03a4c1c6ff53bf1c399e0e1"},
{file = "charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:78cacd03e79d009d95635e7d6ff12c21eb89b894c354bd2b2ed0b4763373693b"},
{file = "charset_normalizer-3.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:de5695a6f1d8340b12a5d6d4484290ee74d61e467c39ff03b39e30df62cf83a0"},
{file = "charset_normalizer-3.1.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1c60b9c202d00052183c9be85e5eaf18a4ada0a47d188a83c8f5c5b23252f649"},
{file = "charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:f645caaf0008bacf349875a974220f1f1da349c5dbe7c4ec93048cdc785a3326"},
{file = "charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:ea9f9c6034ea2d93d9147818f17c2a0860d41b71c38b9ce4d55f21b6f9165a11"},
{file = "charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_ppc64le.whl", hash = "sha256:80d1543d58bd3d6c271b66abf454d437a438dff01c3e62fdbcd68f2a11310d4b"},
{file = "charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_s390x.whl", hash = "sha256:73dc03a6a7e30b7edc5b01b601e53e7fc924b04e1835e8e407c12c037e81adbd"},
{file = "charset_normalizer-3.1.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:6f5c2e7bc8a4bf7c426599765b1bd33217ec84023033672c1e9a8b35eaeaaaf8"},
{file = "charset_normalizer-3.1.0-cp38-cp38-win32.whl", hash = "sha256:12a2b561af122e3d94cdb97fe6fb2bb2b82cef0cdca131646fdb940a1eda04f0"},
{file = "charset_normalizer-3.1.0-cp38-cp38-win_amd64.whl", hash = "sha256:3160a0fd9754aab7d47f95a6b63ab355388d890163eb03b2d2b87ab0a30cfa59"},
{file = "charset_normalizer-3.1.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:38e812a197bf8e71a59fe55b757a84c1f946d0ac114acafaafaf21667a7e169e"},
{file = "charset_normalizer-3.1.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:6baf0baf0d5d265fa7944feb9f7451cc316bfe30e8df1a61b1bb08577c554f31"},
{file = "charset_normalizer-3.1.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:8f25e17ab3039b05f762b0a55ae0b3632b2e073d9c8fc88e89aca31a6198e88f"},
{file = "charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3747443b6a904001473370d7810aa19c3a180ccd52a7157aacc264a5ac79265e"},
{file = "charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b116502087ce8a6b7a5f1814568ccbd0e9f6cfd99948aa59b0e241dc57cf739f"},
{file = "charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d16fd5252f883eb074ca55cb622bc0bee49b979ae4e8639fff6ca3ff44f9f854"},
{file = "charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:21fa558996782fc226b529fdd2ed7866c2c6ec91cee82735c98a197fae39f706"},
{file = "charset_normalizer-3.1.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6f6c7a8a57e9405cad7485f4c9d3172ae486cfef1344b5ddd8e5239582d7355e"},
{file = "charset_normalizer-3.1.0-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:ac3775e3311661d4adace3697a52ac0bab17edd166087d493b52d4f4f553f9f0"},
{file = "charset_normalizer-3.1.0-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:10c93628d7497c81686e8e5e557aafa78f230cd9e77dd0c40032ef90c18f2230"},
{file = "charset_normalizer-3.1.0-cp39-cp39-musllinux_1_1_ppc64le.whl", hash = "sha256:6f4f4668e1831850ebcc2fd0b1cd11721947b6dc7c00bf1c6bd3c929ae14f2c7"},
{file = "charset_normalizer-3.1.0-cp39-cp39-musllinux_1_1_s390x.whl", hash = "sha256:0be65ccf618c1e7ac9b849c315cc2e8a8751d9cfdaa43027d4f6624bd587ab7e"},
{file = "charset_normalizer-3.1.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:53d0a3fa5f8af98a1e261de6a3943ca631c526635eb5817a87a59d9a57ebf48f"},
{file = "charset_normalizer-3.1.0-cp39-cp39-win32.whl", hash = "sha256:a04f86f41a8916fe45ac5024ec477f41f886b3c435da2d4e3d2709b22ab02af1"},
{file = "charset_normalizer-3.1.0-cp39-cp39-win_amd64.whl", hash = "sha256:830d2948a5ec37c386d3170c483063798d7879037492540f10a475e3fd6f244b"},
{file = "charset_normalizer-3.1.0-py3-none-any.whl", hash = "sha256:3d9098b479e78c85080c98e1e35ff40b4a31d8953102bb0fd7d1b6f8a2111a3d"},
{file = "charset-normalizer-3.2.0.tar.gz", hash = "sha256:3bb3d25a8e6c0aedd251753a79ae98a093c7e7b471faa3aa9a93a81431987ace"},
{file = "charset_normalizer-3.2.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:0b87549028f680ca955556e3bd57013ab47474c3124dc069faa0b6545b6c9710"},
{file = "charset_normalizer-3.2.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:7c70087bfee18a42b4040bb9ec1ca15a08242cf5867c58726530bdf3945672ed"},
{file = "charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a103b3a7069b62f5d4890ae1b8f0597618f628b286b03d4bc9195230b154bfa9"},
{file = "charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:94aea8eff76ee6d1cdacb07dd2123a68283cb5569e0250feab1240058f53b623"},
{file = "charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:db901e2ac34c931d73054d9797383d0f8009991e723dab15109740a63e7f902a"},
{file = "charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b0dac0ff919ba34d4df1b6131f59ce95b08b9065233446be7e459f95554c0dc8"},
{file = "charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:193cbc708ea3aca45e7221ae58f0fd63f933753a9bfb498a3b474878f12caaad"},
{file = "charset_normalizer-3.2.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:09393e1b2a9461950b1c9a45d5fd251dc7c6f228acab64da1c9c0165d9c7765c"},
{file = "charset_normalizer-3.2.0-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:baacc6aee0b2ef6f3d308e197b5d7a81c0e70b06beae1f1fcacffdbd124fe0e3"},
{file = "charset_normalizer-3.2.0-cp310-cp310-musllinux_1_1_i686.whl", hash = "sha256:bf420121d4c8dce6b889f0e8e4ec0ca34b7f40186203f06a946fa0276ba54029"},
{file = "charset_normalizer-3.2.0-cp310-cp310-musllinux_1_1_ppc64le.whl", hash = "sha256:c04a46716adde8d927adb9457bbe39cf473e1e2c2f5d0a16ceb837e5d841ad4f"},
{file = "charset_normalizer-3.2.0-cp310-cp310-musllinux_1_1_s390x.whl", hash = "sha256:aaf63899c94de41fe3cf934601b0f7ccb6b428c6e4eeb80da72c58eab077b19a"},
{file = "charset_normalizer-3.2.0-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:d62e51710986674142526ab9f78663ca2b0726066ae26b78b22e0f5e571238dd"},
{file = "charset_normalizer-3.2.0-cp310-cp310-win32.whl", hash = "sha256:04e57ab9fbf9607b77f7d057974694b4f6b142da9ed4a199859d9d4d5c63fe96"},
{file = "charset_normalizer-3.2.0-cp310-cp310-win_amd64.whl", hash = "sha256:48021783bdf96e3d6de03a6e39a1171ed5bd7e8bb93fc84cc649d11490f87cea"},
{file = "charset_normalizer-3.2.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:4957669ef390f0e6719db3613ab3a7631e68424604a7b448f079bee145da6e09"},
{file = "charset_normalizer-3.2.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:46fb8c61d794b78ec7134a715a3e564aafc8f6b5e338417cb19fe9f57a5a9bf2"},
{file = "charset_normalizer-3.2.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f779d3ad205f108d14e99bb3859aa7dd8e9c68874617c72354d7ecaec2a054ac"},
{file = "charset_normalizer-3.2.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f25c229a6ba38a35ae6e25ca1264621cc25d4d38dca2942a7fce0b67a4efe918"},
{file = "charset_normalizer-3.2.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2efb1bd13885392adfda4614c33d3b68dee4921fd0ac1d3988f8cbb7d589e72a"},
{file = "charset_normalizer-3.2.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1f30b48dd7fa1474554b0b0f3fdfdd4c13b5c737a3c6284d3cdc424ec0ffff3a"},
{file = "charset_normalizer-3.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:246de67b99b6851627d945db38147d1b209a899311b1305dd84916f2b88526c6"},
{file = "charset_normalizer-3.2.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:9bd9b3b31adcb054116447ea22caa61a285d92e94d710aa5ec97992ff5eb7cf3"},
{file = "charset_normalizer-3.2.0-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:8c2f5e83493748286002f9369f3e6607c565a6a90425a3a1fef5ae32a36d749d"},
{file = "charset_normalizer-3.2.0-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:3170c9399da12c9dc66366e9d14da8bf7147e1e9d9ea566067bbce7bb74bd9c2"},
{file = "charset_normalizer-3.2.0-cp311-cp311-musllinux_1_1_ppc64le.whl", hash = "sha256:7a4826ad2bd6b07ca615c74ab91f32f6c96d08f6fcc3902ceeedaec8cdc3bcd6"},
{file = "charset_normalizer-3.2.0-cp311-cp311-musllinux_1_1_s390x.whl", hash = "sha256:3b1613dd5aee995ec6d4c69f00378bbd07614702a315a2cf6c1d21461fe17c23"},
{file = "charset_normalizer-3.2.0-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:9e608aafdb55eb9f255034709e20d5a83b6d60c054df0802fa9c9883d0a937aa"},
{file = "charset_normalizer-3.2.0-cp311-cp311-win32.whl", hash = "sha256:f2a1d0fd4242bd8643ce6f98927cf9c04540af6efa92323e9d3124f57727bfc1"},
{file = "charset_normalizer-3.2.0-cp311-cp311-win_amd64.whl", hash = "sha256:681eb3d7e02e3c3655d1b16059fbfb605ac464c834a0c629048a30fad2b27489"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:c57921cda3a80d0f2b8aec7e25c8aa14479ea92b5b51b6876d975d925a2ea346"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:41b25eaa7d15909cf3ac4c96088c1f266a9a93ec44f87f1d13d4a0e86c81b982"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f058f6963fd82eb143c692cecdc89e075fa0828db2e5b291070485390b2f1c9c"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a7647ebdfb9682b7bb97e2a5e7cb6ae735b1c25008a70b906aecca294ee96cf4"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eef9df1eefada2c09a5e7a40991b9fc6ac6ef20b1372abd48d2794a316dc0449"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e03b8895a6990c9ab2cdcd0f2fe44088ca1c65ae592b8f795c3294af00a461c3"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:ee4006268ed33370957f55bf2e6f4d263eaf4dc3cfc473d1d90baff6ed36ce4a"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-musllinux_1_1_i686.whl", hash = "sha256:c4983bf937209c57240cff65906b18bb35e64ae872da6a0db937d7b4af845dd7"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-musllinux_1_1_ppc64le.whl", hash = "sha256:3bb7fda7260735efe66d5107fb7e6af6a7c04c7fce9b2514e04b7a74b06bf5dd"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-musllinux_1_1_s390x.whl", hash = "sha256:72814c01533f51d68702802d74f77ea026b5ec52793c791e2da806a3844a46c3"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:70c610f6cbe4b9fce272c407dd9d07e33e6bf7b4aa1b7ffb6f6ded8e634e3592"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-win32.whl", hash = "sha256:a401b4598e5d3f4a9a811f3daf42ee2291790c7f9d74b18d75d6e21dda98a1a1"},
{file = "charset_normalizer-3.2.0-cp37-cp37m-win_amd64.whl", hash = "sha256:c0b21078a4b56965e2b12f247467b234734491897e99c1d51cee628da9786959"},
{file = "charset_normalizer-3.2.0-cp38-cp38-macosx_10_9_universal2.whl", hash = "sha256:95eb302ff792e12aba9a8b8f8474ab229a83c103d74a750ec0bd1c1eea32e669"},
{file = "charset_normalizer-3.2.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:1a100c6d595a7f316f1b6f01d20815d916e75ff98c27a01ae817439ea7726329"},
{file = "charset_normalizer-3.2.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:6339d047dab2780cc6220f46306628e04d9750f02f983ddb37439ca47ced7149"},
{file = "charset_normalizer-3.2.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e4b749b9cc6ee664a3300bb3a273c1ca8068c46be705b6c31cf5d276f8628a94"},
{file = "charset_normalizer-3.2.0-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a38856a971c602f98472050165cea2cdc97709240373041b69030be15047691f"},
{file = "charset_normalizer-3.2.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f87f746ee241d30d6ed93969de31e5ffd09a2961a051e60ae6bddde9ec3583aa"},
{file = "charset_normalizer-3.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:89f1b185a01fe560bc8ae5f619e924407efca2191b56ce749ec84982fc59a32a"},
{file = "charset_normalizer-3.2.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e1c8a2f4c69e08e89632defbfabec2feb8a8d99edc9f89ce33c4b9e36ab63037"},
{file = "charset_normalizer-3.2.0-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:2f4ac36d8e2b4cc1aa71df3dd84ff8efbe3bfb97ac41242fbcfc053c67434f46"},
{file = "charset_normalizer-3.2.0-cp38-cp38-musllinux_1_1_i686.whl", hash = "sha256:a386ebe437176aab38c041de1260cd3ea459c6ce5263594399880bbc398225b2"},
{file = "charset_normalizer-3.2.0-cp38-cp38-musllinux_1_1_ppc64le.whl", hash = "sha256:ccd16eb18a849fd8dcb23e23380e2f0a354e8daa0c984b8a732d9cfaba3a776d"},
{file = "charset_normalizer-3.2.0-cp38-cp38-musllinux_1_1_s390x.whl", hash = "sha256:e6a5bf2cba5ae1bb80b154ed68a3cfa2fa00fde979a7f50d6598d3e17d9ac20c"},
{file = "charset_normalizer-3.2.0-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:45de3f87179c1823e6d9e32156fb14c1927fcc9aba21433f088fdfb555b77c10"},
{file = "charset_normalizer-3.2.0-cp38-cp38-win32.whl", hash = "sha256:1000fba1057b92a65daec275aec30586c3de2401ccdcd41f8a5c1e2c87078706"},
{file = "charset_normalizer-3.2.0-cp38-cp38-win_amd64.whl", hash = "sha256:8b2c760cfc7042b27ebdb4a43a4453bd829a5742503599144d54a032c5dc7e9e"},
{file = "charset_normalizer-3.2.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:855eafa5d5a2034b4621c74925d89c5efef61418570e5ef9b37717d9c796419c"},
{file = "charset_normalizer-3.2.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:203f0c8871d5a7987be20c72442488a0b8cfd0f43b7973771640fc593f56321f"},
{file = "charset_normalizer-3.2.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e857a2232ba53ae940d3456f7533ce6ca98b81917d47adc3c7fd55dad8fab858"},
{file = "charset_normalizer-3.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5e86d77b090dbddbe78867a0275cb4df08ea195e660f1f7f13435a4649e954e5"},
{file = "charset_normalizer-3.2.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c4fb39a81950ec280984b3a44f5bd12819953dc5fa3a7e6fa7a80db5ee853952"},
{file = "charset_normalizer-3.2.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2dee8e57f052ef5353cf608e0b4c871aee320dd1b87d351c28764fc0ca55f9f4"},
{file = "charset_normalizer-3.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8700f06d0ce6f128de3ccdbc1acaea1ee264d2caa9ca05daaf492fde7c2a7200"},
{file = "charset_normalizer-3.2.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1920d4ff15ce893210c1f0c0e9d19bfbecb7983c76b33f046c13a8ffbd570252"},
{file = "charset_normalizer-3.2.0-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:c1c76a1743432b4b60ab3358c937a3fe1341c828ae6194108a94c69028247f22"},
{file = "charset_normalizer-3.2.0-cp39-cp39-musllinux_1_1_i686.whl", hash = "sha256:f7560358a6811e52e9c4d142d497f1a6e10103d3a6881f18d04dbce3729c0e2c"},
{file = "charset_normalizer-3.2.0-cp39-cp39-musllinux_1_1_ppc64le.whl", hash = "sha256:c8063cf17b19661471ecbdb3df1c84f24ad2e389e326ccaf89e3fb2484d8dd7e"},
{file = "charset_normalizer-3.2.0-cp39-cp39-musllinux_1_1_s390x.whl", hash = "sha256:cd6dbe0238f7743d0efe563ab46294f54f9bc8f4b9bcf57c3c666cc5bc9d1299"},
{file = "charset_normalizer-3.2.0-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:1249cbbf3d3b04902ff081ffbb33ce3377fa6e4c7356f759f3cd076cc138d020"},
{file = "charset_normalizer-3.2.0-cp39-cp39-win32.whl", hash = "sha256:6c409c0deba34f147f77efaa67b8e4bb83d2f11c8806405f76397ae5b8c0d1c9"},
{file = "charset_normalizer-3.2.0-cp39-cp39-win_amd64.whl", hash = "sha256:7095f6fbfaa55defb6b733cfeb14efaae7a29f0b59d8cf213be4e7ca0b857b80"},
{file = "charset_normalizer-3.2.0-py3-none-any.whl", hash = "sha256:8e098148dd37b4ce3baca71fb394c81dc5d9c7728c95df695d2dca218edf40e6"},
]
[[package]]
name = "colorama"
version = "0.4.6"
description = "Cross-platform colored terminal text."
category = "main"
optional = false
python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7"
files = [
@ -149,6 +155,7 @@ files = [
name = "contextlib2"
version = "21.6.0"
description = "Backports and enhancements for the contextlib module"
category = "main"
optional = false
python-versions = ">=3.6"
files = [
@ -158,19 +165,21 @@ files = [
[[package]]
name = "distlib"
version = "0.3.6"
version = "0.3.7"
description = "Distribution utilities"
category = "dev"
optional = false
python-versions = "*"
files = [
{file = "distlib-0.3.6-py2.py3-none-any.whl", hash = "sha256:f35c4b692542ca110de7ef0bea44d73981caeb34ca0b9b6b2e6d7790dda8f80e"},
{file = "distlib-0.3.6.tar.gz", hash = "sha256:14bad2d9b04d3a36127ac97f30b12a19268f211063d8f8ee4f47108896e11b46"},
{file = "distlib-0.3.7-py2.py3-none-any.whl", hash = "sha256:2e24928bc811348f0feb63014e97aaae3037f2cf48712d51ae61df7fd6075057"},
{file = "distlib-0.3.7.tar.gz", hash = "sha256:9dafe54b34a028eafd95039d5e5d4851a13734540f1331060d31c9916e7147a8"},
]
[[package]]
name = "docopt"
version = "0.6.2"
description = "Pythonic argument parser, that will make you smile"
category = "main"
optional = false
python-versions = "*"
files = [
@ -181,6 +190,7 @@ files = [
name = "file-read-backwards"
version = "2.0.0"
description = "Memory efficient way of reading files line-by-line from the end of file"
category = "main"
optional = false
python-versions = "*"
files = [
@ -192,6 +202,7 @@ files = [
name = "filelock"
version = "3.12.2"
description = "A platform independent file lock."
category = "dev"
optional = false
python-versions = ">=3.7"
files = [
@ -207,6 +218,7 @@ testing = ["covdefaults (>=2.3)", "coverage (>=7.2.7)", "diff-cover (>=7.5)", "p
name = "flake8"
version = "3.9.2"
description = "the modular source code checker: pep8 pyflakes and co"
category = "dev"
optional = false
python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7"
files = [
@ -221,13 +233,14 @@ pyflakes = ">=2.3.0,<2.4.0"
[[package]]
name = "identify"
version = "2.5.24"
version = "2.5.26"
description = "File identification library for Python"
category = "dev"
optional = false
python-versions = ">=3.7"
python-versions = ">=3.8"
files = [
{file = "identify-2.5.24-py2.py3-none-any.whl", hash = "sha256:986dbfb38b1140e763e413e6feb44cd731faf72d1909543178aa79b0e258265d"},
{file = "identify-2.5.24.tar.gz", hash = "sha256:0aac67d5b4812498056d28a9a512a483f5085cc28640b02b258a59dac34301d4"},
{file = "identify-2.5.26-py2.py3-none-any.whl", hash = "sha256:c22a8ead0d4ca11f1edd6c9418c3220669b3b7533ada0a0ffa6cc0ef85cf9b54"},
{file = "identify-2.5.26.tar.gz", hash = "sha256:7243800bce2f58404ed41b7c002e53d4d22bcf3ae1b7900c2d7aefd95394bf7f"},
]
[package.extras]
@ -237,6 +250,7 @@ license = ["ukkonen"]
name = "idna"
version = "3.4"
description = "Internationalized Domain Names in Applications (IDNA)"
category = "main"
optional = false
python-versions = ">=3.5"
files = [
@ -248,6 +262,7 @@ files = [
name = "iniconfig"
version = "2.0.0"
description = "brain-dead simple config-ini parsing"
category = "dev"
optional = false
python-versions = ">=3.7"
files = [
@ -259,6 +274,7 @@ files = [
name = "internetarchive"
version = "3.5.0"
description = "A Python interface to archive.org."
category = "main"
optional = false
python-versions = ">=3.7"
files = [
@ -284,6 +300,7 @@ types = ["tqdm-stubs (>=0.2.0)", "types-colorama", "types-docopt (>=0.6.10,<0.7.
name = "jsonpatch"
version = "1.33"
description = "Apply JSON-Patches (RFC 6902)"
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*"
files = [
@ -298,6 +315,7 @@ jsonpointer = ">=1.9"
name = "jsonpointer"
version = "2.4"
description = "Identify specific nodes in a JSON document (RFC 6901)"
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*, !=3.6.*"
files = [
@ -307,100 +325,117 @@ files = [
[[package]]
name = "lxml"
version = "4.9.2"
version = "4.9.3"
description = "Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API."
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, != 3.4.*"
files = [
{file = "lxml-4.9.2-cp27-cp27m-macosx_10_15_x86_64.whl", hash = "sha256:76cf573e5a365e790396a5cc2b909812633409306c6531a6877c59061e42c4f2"},
{file = "lxml-4.9.2-cp27-cp27m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:b1f42b6921d0e81b1bcb5e395bc091a70f41c4d4e55ba99c6da2b31626c44892"},
{file = "lxml-4.9.2-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:9f102706d0ca011de571de32c3247c6476b55bb6bc65a20f682f000b07a4852a"},
{file = "lxml-4.9.2-cp27-cp27m-win32.whl", hash = "sha256:8d0b4612b66ff5d62d03bcaa043bb018f74dfea51184e53f067e6fdcba4bd8de"},
{file = "lxml-4.9.2-cp27-cp27m-win_amd64.whl", hash = "sha256:4c8f293f14abc8fd3e8e01c5bd86e6ed0b6ef71936ded5bf10fe7a5efefbaca3"},
{file = "lxml-4.9.2-cp27-cp27mu-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2899456259589aa38bfb018c364d6ae7b53c5c22d8e27d0ec7609c2a1ff78b50"},
{file = "lxml-4.9.2-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:6749649eecd6a9871cae297bffa4ee76f90b4504a2a2ab528d9ebe912b101975"},
{file = "lxml-4.9.2-cp310-cp310-macosx_10_15_x86_64.whl", hash = "sha256:a08cff61517ee26cb56f1e949cca38caabe9ea9fbb4b1e10a805dc39844b7d5c"},
{file = "lxml-4.9.2-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:85cabf64adec449132e55616e7ca3e1000ab449d1d0f9d7f83146ed5bdcb6d8a"},
{file = "lxml-4.9.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:8340225bd5e7a701c0fa98284c849c9b9fc9238abf53a0ebd90900f25d39a4e4"},
{file = "lxml-4.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:1ab8f1f932e8f82355e75dda5413a57612c6ea448069d4fb2e217e9a4bed13d4"},
{file = "lxml-4.9.2-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:699a9af7dffaf67deeae27b2112aa06b41c370d5e7633e0ee0aea2e0b6c211f7"},
{file = "lxml-4.9.2-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:b9cc34af337a97d470040f99ba4282f6e6bac88407d021688a5d585e44a23184"},
{file = "lxml-4.9.2-cp310-cp310-win32.whl", hash = "sha256:d02a5399126a53492415d4906ab0ad0375a5456cc05c3fc0fc4ca11771745cda"},
{file = "lxml-4.9.2-cp310-cp310-win_amd64.whl", hash = "sha256:a38486985ca49cfa574a507e7a2215c0c780fd1778bb6290c21193b7211702ab"},
{file = "lxml-4.9.2-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:c83203addf554215463b59f6399835201999b5e48019dc17f182ed5ad87205c9"},
{file = "lxml-4.9.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:2a87fa548561d2f4643c99cd13131acb607ddabb70682dcf1dff5f71f781a4bf"},
{file = "lxml-4.9.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:d6b430a9938a5a5d85fc107d852262ddcd48602c120e3dbb02137c83d212b380"},
{file = "lxml-4.9.2-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:3efea981d956a6f7173b4659849f55081867cf897e719f57383698af6f618a92"},
{file = "lxml-4.9.2-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:df0623dcf9668ad0445e0558a21211d4e9a149ea8f5666917c8eeec515f0a6d1"},
{file = "lxml-4.9.2-cp311-cp311-win32.whl", hash = "sha256:da248f93f0418a9e9d94b0080d7ebc407a9a5e6d0b57bb30db9b5cc28de1ad33"},
{file = "lxml-4.9.2-cp311-cp311-win_amd64.whl", hash = "sha256:3818b8e2c4b5148567e1b09ce739006acfaa44ce3156f8cbbc11062994b8e8dd"},
{file = "lxml-4.9.2-cp35-cp35m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ca989b91cf3a3ba28930a9fc1e9aeafc2a395448641df1f387a2d394638943b0"},
{file = "lxml-4.9.2-cp35-cp35m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:822068f85e12a6e292803e112ab876bc03ed1f03dddb80154c395f891ca6b31e"},
{file = "lxml-4.9.2-cp35-cp35m-win32.whl", hash = "sha256:be7292c55101e22f2a3d4d8913944cbea71eea90792bf914add27454a13905df"},
{file = "lxml-4.9.2-cp35-cp35m-win_amd64.whl", hash = "sha256:998c7c41910666d2976928c38ea96a70d1aa43be6fe502f21a651e17483a43c5"},
{file = "lxml-4.9.2-cp36-cp36m-macosx_10_15_x86_64.whl", hash = "sha256:b26a29f0b7fc6f0897f043ca366142d2b609dc60756ee6e4e90b5f762c6adc53"},
{file = "lxml-4.9.2-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:ab323679b8b3030000f2be63e22cdeea5b47ee0abd2d6a1dc0c8103ddaa56cd7"},
{file = "lxml-4.9.2-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:689bb688a1db722485e4610a503e3e9210dcc20c520b45ac8f7533c837be76fe"},
{file = "lxml-4.9.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:f49e52d174375a7def9915c9f06ec4e569d235ad428f70751765f48d5926678c"},
{file = "lxml-4.9.2-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:36c3c175d34652a35475a73762b545f4527aec044910a651d2bf50de9c3352b1"},
{file = "lxml-4.9.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:a35f8b7fa99f90dd2f5dc5a9fa12332642f087a7641289ca6c40d6e1a2637d8e"},
{file = "lxml-4.9.2-cp36-cp36m-musllinux_1_1_aarch64.whl", hash = "sha256:58bfa3aa19ca4c0f28c5dde0ff56c520fbac6f0daf4fac66ed4c8d2fb7f22e74"},
{file = "lxml-4.9.2-cp36-cp36m-musllinux_1_1_x86_64.whl", hash = "sha256:bc718cd47b765e790eecb74d044cc8d37d58562f6c314ee9484df26276d36a38"},
{file = "lxml-4.9.2-cp36-cp36m-win32.whl", hash = "sha256:d5bf6545cd27aaa8a13033ce56354ed9e25ab0e4ac3b5392b763d8d04b08e0c5"},
{file = "lxml-4.9.2-cp36-cp36m-win_amd64.whl", hash = "sha256:3ab9fa9d6dc2a7f29d7affdf3edebf6ece6fb28a6d80b14c3b2fb9d39b9322c3"},
{file = "lxml-4.9.2-cp37-cp37m-macosx_10_15_x86_64.whl", hash = "sha256:05ca3f6abf5cf78fe053da9b1166e062ade3fa5d4f92b4ed688127ea7d7b1d03"},
{file = "lxml-4.9.2-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:a5da296eb617d18e497bcf0a5c528f5d3b18dadb3619fbdadf4ed2356ef8d941"},
{file = "lxml-4.9.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:04876580c050a8c5341d706dd464ff04fd597095cc8c023252566a8826505726"},
{file = "lxml-4.9.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:c9ec3eaf616d67db0764b3bb983962b4f385a1f08304fd30c7283954e6a7869b"},
{file = "lxml-4.9.2-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2a29ba94d065945944016b6b74e538bdb1751a1db6ffb80c9d3c2e40d6fa9894"},
{file = "lxml-4.9.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:a82d05da00a58b8e4c0008edbc8a4b6ec5a4bc1e2ee0fb6ed157cf634ed7fa45"},
{file = "lxml-4.9.2-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:223f4232855ade399bd409331e6ca70fb5578efef22cf4069a6090acc0f53c0e"},
{file = "lxml-4.9.2-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:d17bc7c2ccf49c478c5bdd447594e82692c74222698cfc9b5daae7ae7e90743b"},
{file = "lxml-4.9.2-cp37-cp37m-win32.whl", hash = "sha256:b64d891da92e232c36976c80ed7ebb383e3f148489796d8d31a5b6a677825efe"},
{file = "lxml-4.9.2-cp37-cp37m-win_amd64.whl", hash = "sha256:a0a336d6d3e8b234a3aae3c674873d8f0e720b76bc1d9416866c41cd9500ffb9"},
{file = "lxml-4.9.2-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:da4dd7c9c50c059aba52b3524f84d7de956f7fef88f0bafcf4ad7dde94a064e8"},
{file = "lxml-4.9.2-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:821b7f59b99551c69c85a6039c65b75f5683bdc63270fec660f75da67469ca24"},
{file = "lxml-4.9.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:e5168986b90a8d1f2f9dc1b841467c74221bd752537b99761a93d2d981e04889"},
{file = "lxml-4.9.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:8e20cb5a47247e383cf4ff523205060991021233ebd6f924bca927fcf25cf86f"},
{file = "lxml-4.9.2-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:13598ecfbd2e86ea7ae45ec28a2a54fb87ee9b9fdb0f6d343297d8e548392c03"},
{file = "lxml-4.9.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:880bbbcbe2fca64e2f4d8e04db47bcdf504936fa2b33933efd945e1b429bea8c"},
{file = "lxml-4.9.2-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:7d2278d59425777cfcb19735018d897ca8303abe67cc735f9f97177ceff8027f"},
{file = "lxml-4.9.2-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:5344a43228767f53a9df6e5b253f8cdca7dfc7b7aeae52551958192f56d98457"},
{file = "lxml-4.9.2-cp38-cp38-win32.whl", hash = "sha256:925073b2fe14ab9b87e73f9a5fde6ce6392da430f3004d8b72cc86f746f5163b"},
{file = "lxml-4.9.2-cp38-cp38-win_amd64.whl", hash = "sha256:9b22c5c66f67ae00c0199f6055705bc3eb3fcb08d03d2ec4059a2b1b25ed48d7"},
{file = "lxml-4.9.2-cp39-cp39-macosx_10_15_x86_64.whl", hash = "sha256:5f50a1c177e2fa3ee0667a5ab79fdc6b23086bc8b589d90b93b4bd17eb0e64d1"},
{file = "lxml-4.9.2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:090c6543d3696cbe15b4ac6e175e576bcc3f1ccfbba970061b7300b0c15a2140"},
{file = "lxml-4.9.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:63da2ccc0857c311d764e7d3d90f429c252e83b52d1f8f1d1fe55be26827d1f4"},
{file = "lxml-4.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:5b4545b8a40478183ac06c073e81a5ce4cf01bf1734962577cf2bb569a5b3bbf"},
{file = "lxml-4.9.2-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2e430cd2824f05f2d4f687701144556646bae8f249fd60aa1e4c768ba7018947"},
{file = "lxml-4.9.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:6804daeb7ef69e7b36f76caddb85cccd63d0c56dedb47555d2fc969e2af6a1a5"},
{file = "lxml-4.9.2-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:a6e441a86553c310258aca15d1c05903aaf4965b23f3bc2d55f200804e005ee5"},
{file = "lxml-4.9.2-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:ca34efc80a29351897e18888c71c6aca4a359247c87e0b1c7ada14f0ab0c0fb2"},
{file = "lxml-4.9.2-cp39-cp39-win32.whl", hash = "sha256:6b418afe5df18233fc6b6093deb82a32895b6bb0b1155c2cdb05203f583053f1"},
{file = "lxml-4.9.2-cp39-cp39-win_amd64.whl", hash = "sha256:f1496ea22ca2c830cbcbd473de8f114a320da308438ae65abad6bab7867fe38f"},
{file = "lxml-4.9.2-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:b264171e3143d842ded311b7dccd46ff9ef34247129ff5bf5066123c55c2431c"},
{file = "lxml-4.9.2-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:0dc313ef231edf866912e9d8f5a042ddab56c752619e92dfd3a2c277e6a7299a"},
{file = "lxml-4.9.2-pp38-pypy38_pp73-macosx_10_15_x86_64.whl", hash = "sha256:16efd54337136e8cd72fb9485c368d91d77a47ee2d42b057564aae201257d419"},
{file = "lxml-4.9.2-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:0f2b1e0d79180f344ff9f321327b005ca043a50ece8713de61d1cb383fb8ac05"},
{file = "lxml-4.9.2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:7b770ed79542ed52c519119473898198761d78beb24b107acf3ad65deae61f1f"},
{file = "lxml-4.9.2-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:efa29c2fe6b4fdd32e8ef81c1528506895eca86e1d8c4657fda04c9b3786ddf9"},
{file = "lxml-4.9.2-pp39-pypy39_pp73-macosx_10_15_x86_64.whl", hash = "sha256:7e91ee82f4199af8c43d8158024cbdff3d931df350252288f0d4ce656df7f3b5"},
{file = "lxml-4.9.2-pp39-pypy39_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:b23e19989c355ca854276178a0463951a653309fb8e57ce674497f2d9f208746"},
{file = "lxml-4.9.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:01d36c05f4afb8f7c20fd9ed5badca32a2029b93b1750f571ccc0b142531caf7"},
{file = "lxml-4.9.2-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:7b515674acfdcadb0eb5d00d8a709868173acece5cb0be3dd165950cbfdf5409"},
{file = "lxml-4.9.2.tar.gz", hash = "sha256:2455cfaeb7ac70338b3257f41e21f0724f4b5b0c0e7702da67ee6c3640835b67"},
{file = "lxml-4.9.3-cp27-cp27m-macosx_11_0_x86_64.whl", hash = "sha256:b0a545b46b526d418eb91754565ba5b63b1c0b12f9bd2f808c852d9b4b2f9b5c"},
{file = "lxml-4.9.3-cp27-cp27m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:075b731ddd9e7f68ad24c635374211376aa05a281673ede86cbe1d1b3455279d"},
{file = "lxml-4.9.3-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:1e224d5755dba2f4a9498e150c43792392ac9b5380aa1b845f98a1618c94eeef"},
{file = "lxml-4.9.3-cp27-cp27m-win32.whl", hash = "sha256:2c74524e179f2ad6d2a4f7caf70e2d96639c0954c943ad601a9e146c76408ed7"},
{file = "lxml-4.9.3-cp27-cp27m-win_amd64.whl", hash = "sha256:4f1026bc732b6a7f96369f7bfe1a4f2290fb34dce00d8644bc3036fb351a4ca1"},
{file = "lxml-4.9.3-cp27-cp27mu-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c0781a98ff5e6586926293e59480b64ddd46282953203c76ae15dbbbf302e8bb"},
{file = "lxml-4.9.3-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:cef2502e7e8a96fe5ad686d60b49e1ab03e438bd9123987994528febd569868e"},
{file = "lxml-4.9.3-cp310-cp310-macosx_11_0_x86_64.whl", hash = "sha256:b86164d2cff4d3aaa1f04a14685cbc072efd0b4f99ca5708b2ad1b9b5988a991"},
{file = "lxml-4.9.3-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:42871176e7896d5d45138f6d28751053c711ed4d48d8e30b498da155af39aebd"},
{file = "lxml-4.9.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:ae8b9c6deb1e634ba4f1930eb67ef6e6bf6a44b6eb5ad605642b2d6d5ed9ce3c"},
{file = "lxml-4.9.3-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:411007c0d88188d9f621b11d252cce90c4a2d1a49db6c068e3c16422f306eab8"},
{file = "lxml-4.9.3-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:cd47b4a0d41d2afa3e58e5bf1f62069255aa2fd6ff5ee41604418ca925911d76"},
{file = "lxml-4.9.3-cp310-cp310-musllinux_1_1_aarch64.whl", hash = "sha256:0e2cb47860da1f7e9a5256254b74ae331687b9672dfa780eed355c4c9c3dbd23"},
{file = "lxml-4.9.3-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:1247694b26342a7bf47c02e513d32225ededd18045264d40758abeb3c838a51f"},
{file = "lxml-4.9.3-cp310-cp310-win32.whl", hash = "sha256:cdb650fc86227eba20de1a29d4b2c1bfe139dc75a0669270033cb2ea3d391b85"},
{file = "lxml-4.9.3-cp310-cp310-win_amd64.whl", hash = "sha256:97047f0d25cd4bcae81f9ec9dc290ca3e15927c192df17331b53bebe0e3ff96d"},
{file = "lxml-4.9.3-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:1f447ea5429b54f9582d4b955f5f1985f278ce5cf169f72eea8afd9502973dd5"},
{file = "lxml-4.9.3-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:57d6ba0ca2b0c462f339640d22882acc711de224d769edf29962b09f77129cbf"},
{file = "lxml-4.9.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:9767e79108424fb6c3edf8f81e6730666a50feb01a328f4a016464a5893f835a"},
{file = "lxml-4.9.3-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:71c52db65e4b56b8ddc5bb89fb2e66c558ed9d1a74a45ceb7dcb20c191c3df2f"},
{file = "lxml-4.9.3-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:d73d8ecf8ecf10a3bd007f2192725a34bd62898e8da27eb9d32a58084f93962b"},
{file = "lxml-4.9.3-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:0a3d3487f07c1d7f150894c238299934a2a074ef590b583103a45002035be120"},
{file = "lxml-4.9.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:9e28c51fa0ce5674be9f560c6761c1b441631901993f76700b1b30ca6c8378d6"},
{file = "lxml-4.9.3-cp311-cp311-win32.whl", hash = "sha256:0bfd0767c5c1de2551a120673b72e5d4b628737cb05414f03c3277bf9bed3305"},
{file = "lxml-4.9.3-cp311-cp311-win_amd64.whl", hash = "sha256:25f32acefac14ef7bd53e4218fe93b804ef6f6b92ffdb4322bb6d49d94cad2bc"},
{file = "lxml-4.9.3-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:d3ff32724f98fbbbfa9f49d82852b159e9784d6094983d9a8b7f2ddaebb063d4"},
{file = "lxml-4.9.3-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:48d6ed886b343d11493129e019da91d4039826794a3e3027321c56d9e71505be"},
{file = "lxml-4.9.3-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:9a92d3faef50658dd2c5470af249985782bf754c4e18e15afb67d3ab06233f13"},
{file = "lxml-4.9.3-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:b4e4bc18382088514ebde9328da057775055940a1f2e18f6ad2d78aa0f3ec5b9"},
{file = "lxml-4.9.3-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:fc9b106a1bf918db68619fdcd6d5ad4f972fdd19c01d19bdb6bf63f3589a9ec5"},
{file = "lxml-4.9.3-cp312-cp312-win_amd64.whl", hash = "sha256:d37017287a7adb6ab77e1c5bee9bcf9660f90ff445042b790402a654d2ad81d8"},
{file = "lxml-4.9.3-cp35-cp35m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:56dc1f1ebccc656d1b3ed288f11e27172a01503fc016bcabdcbc0978b19352b7"},
{file = "lxml-4.9.3-cp35-cp35m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:578695735c5a3f51569810dfebd05dd6f888147a34f0f98d4bb27e92b76e05c2"},
{file = "lxml-4.9.3-cp35-cp35m-win32.whl", hash = "sha256:704f61ba8c1283c71b16135caf697557f5ecf3e74d9e453233e4771d68a1f42d"},
{file = "lxml-4.9.3-cp35-cp35m-win_amd64.whl", hash = "sha256:c41bfca0bd3532d53d16fd34d20806d5c2b1ace22a2f2e4c0008570bf2c58833"},
{file = "lxml-4.9.3-cp36-cp36m-macosx_11_0_x86_64.whl", hash = "sha256:64f479d719dc9f4c813ad9bb6b28f8390360660b73b2e4beb4cb0ae7104f1c12"},
{file = "lxml-4.9.3-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:dd708cf4ee4408cf46a48b108fb9427bfa00b9b85812a9262b5c668af2533ea5"},
{file = "lxml-4.9.3-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5c31c7462abdf8f2ac0577d9f05279727e698f97ecbb02f17939ea99ae8daa98"},
{file = "lxml-4.9.3-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:e3cd95e10c2610c360154afdc2f1480aea394f4a4f1ea0a5eacce49640c9b190"},
{file = "lxml-4.9.3-cp36-cp36m-manylinux_2_28_x86_64.whl", hash = "sha256:4930be26af26ac545c3dffb662521d4e6268352866956672231887d18f0eaab2"},
{file = "lxml-4.9.3-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:4aec80cde9197340bc353d2768e2a75f5f60bacda2bab72ab1dc499589b3878c"},
{file = "lxml-4.9.3-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:14e019fd83b831b2e61baed40cab76222139926b1fb5ed0e79225bc0cae14584"},
{file = "lxml-4.9.3-cp36-cp36m-musllinux_1_1_aarch64.whl", hash = "sha256:0c0850c8b02c298d3c7006b23e98249515ac57430e16a166873fc47a5d549287"},
{file = "lxml-4.9.3-cp36-cp36m-musllinux_1_1_x86_64.whl", hash = "sha256:aca086dc5f9ef98c512bac8efea4483eb84abbf926eaeedf7b91479feb092458"},
{file = "lxml-4.9.3-cp36-cp36m-win32.whl", hash = "sha256:50baa9c1c47efcaef189f31e3d00d697c6d4afda5c3cde0302d063492ff9b477"},
{file = "lxml-4.9.3-cp36-cp36m-win_amd64.whl", hash = "sha256:bef4e656f7d98aaa3486d2627e7d2df1157d7e88e7efd43a65aa5dd4714916cf"},
{file = "lxml-4.9.3-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:46f409a2d60f634fe550f7133ed30ad5321ae2e6630f13657fb9479506b00601"},
{file = "lxml-4.9.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:4c28a9144688aef80d6ea666c809b4b0e50010a2aca784c97f5e6bf143d9f129"},
{file = "lxml-4.9.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:141f1d1a9b663c679dc524af3ea1773e618907e96075262726c7612c02b149a4"},
{file = "lxml-4.9.3-cp37-cp37m-manylinux_2_28_x86_64.whl", hash = "sha256:53ace1c1fd5a74ef662f844a0413446c0629d151055340e9893da958a374f70d"},
{file = "lxml-4.9.3-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:17a753023436a18e27dd7769e798ce302963c236bc4114ceee5b25c18c52c693"},
{file = "lxml-4.9.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:7d298a1bd60c067ea75d9f684f5f3992c9d6766fadbc0bcedd39750bf344c2f4"},
{file = "lxml-4.9.3-cp37-cp37m-musllinux_1_1_aarch64.whl", hash = "sha256:081d32421db5df44c41b7f08a334a090a545c54ba977e47fd7cc2deece78809a"},
{file = "lxml-4.9.3-cp37-cp37m-musllinux_1_1_x86_64.whl", hash = "sha256:23eed6d7b1a3336ad92d8e39d4bfe09073c31bfe502f20ca5116b2a334f8ec02"},
{file = "lxml-4.9.3-cp37-cp37m-win32.whl", hash = "sha256:1509dd12b773c02acd154582088820893109f6ca27ef7291b003d0e81666109f"},
{file = "lxml-4.9.3-cp37-cp37m-win_amd64.whl", hash = "sha256:120fa9349a24c7043854c53cae8cec227e1f79195a7493e09e0c12e29f918e52"},
{file = "lxml-4.9.3-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:4d2d1edbca80b510443f51afd8496be95529db04a509bc8faee49c7b0fb6d2cc"},
{file = "lxml-4.9.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_24_aarch64.whl", hash = "sha256:8d7e43bd40f65f7d97ad8ef5c9b1778943d02f04febef12def25f7583d19baac"},
{file = "lxml-4.9.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:71d66ee82e7417828af6ecd7db817913cb0cf9d4e61aa0ac1fde0583d84358db"},
{file = "lxml-4.9.3-cp38-cp38-manylinux_2_28_x86_64.whl", hash = "sha256:6fc3c450eaa0b56f815c7b62f2b7fba7266c4779adcf1cece9e6deb1de7305ce"},
{file = "lxml-4.9.3-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:65299ea57d82fb91c7f019300d24050c4ddeb7c5a190e076b5f48a2b43d19c42"},
{file = "lxml-4.9.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:eadfbbbfb41b44034a4c757fd5d70baccd43296fb894dba0295606a7cf3124aa"},
{file = "lxml-4.9.3-cp38-cp38-musllinux_1_1_aarch64.whl", hash = "sha256:3e9bdd30efde2b9ccfa9cb5768ba04fe71b018a25ea093379c857c9dad262c40"},
{file = "lxml-4.9.3-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:fcdd00edfd0a3001e0181eab3e63bd5c74ad3e67152c84f93f13769a40e073a7"},
{file = "lxml-4.9.3-cp38-cp38-win32.whl", hash = "sha256:57aba1bbdf450b726d58b2aea5fe47c7875f5afb2c4a23784ed78f19a0462574"},
{file = "lxml-4.9.3-cp38-cp38-win_amd64.whl", hash = "sha256:92af161ecbdb2883c4593d5ed4815ea71b31fafd7fd05789b23100d081ecac96"},
{file = "lxml-4.9.3-cp39-cp39-macosx_11_0_x86_64.whl", hash = "sha256:9bb6ad405121241e99a86efff22d3ef469024ce22875a7ae045896ad23ba2340"},
{file = "lxml-4.9.3-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:8ed74706b26ad100433da4b9d807eae371efaa266ffc3e9191ea436087a9d6a7"},
{file = "lxml-4.9.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:fbf521479bcac1e25a663df882c46a641a9bff6b56dc8b0fafaebd2f66fb231b"},
{file = "lxml-4.9.3-cp39-cp39-manylinux_2_28_aarch64.whl", hash = "sha256:303bf1edce6ced16bf67a18a1cf8339d0db79577eec5d9a6d4a80f0fb10aa2da"},
{file = "lxml-4.9.3-cp39-cp39-manylinux_2_28_x86_64.whl", hash = "sha256:5515edd2a6d1a5a70bfcdee23b42ec33425e405c5b351478ab7dc9347228f96e"},
{file = "lxml-4.9.3-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:690dafd0b187ed38583a648076865d8c229661ed20e48f2335d68e2cf7dc829d"},
{file = "lxml-4.9.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:b6420a005548ad52154c8ceab4a1290ff78d757f9e5cbc68f8c77089acd3c432"},
{file = "lxml-4.9.3-cp39-cp39-musllinux_1_1_aarch64.whl", hash = "sha256:bb3bb49c7a6ad9d981d734ef7c7193bc349ac338776a0360cc671eaee89bcf69"},
{file = "lxml-4.9.3-cp39-cp39-musllinux_1_1_x86_64.whl", hash = "sha256:d27be7405547d1f958b60837dc4c1007da90b8b23f54ba1f8b728c78fdb19d50"},
{file = "lxml-4.9.3-cp39-cp39-win32.whl", hash = "sha256:8df133a2ea5e74eef5e8fc6f19b9e085f758768a16e9877a60aec455ed2609b2"},
{file = "lxml-4.9.3-cp39-cp39-win_amd64.whl", hash = "sha256:4dd9a263e845a72eacb60d12401e37c616438ea2e5442885f65082c276dfb2b2"},
{file = "lxml-4.9.3-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:6689a3d7fd13dc687e9102a27e98ef33730ac4fe37795d5036d18b4d527abd35"},
{file = "lxml-4.9.3-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:f6bdac493b949141b733c5345b6ba8f87a226029cbabc7e9e121a413e49441e0"},
{file = "lxml-4.9.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:05186a0f1346ae12553d66df1cfce6f251589fea3ad3da4f3ef4e34b2d58c6a3"},
{file = "lxml-4.9.3-pp37-pypy37_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:c2006f5c8d28dee289f7020f721354362fa304acbaaf9745751ac4006650254b"},
{file = "lxml-4.9.3-pp38-pypy38_pp73-macosx_11_0_x86_64.whl", hash = "sha256:5c245b783db29c4e4fbbbfc9c5a78be496c9fea25517f90606aa1f6b2b3d5f7b"},
{file = "lxml-4.9.3-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:4fb960a632a49f2f089d522f70496640fdf1218f1243889da3822e0a9f5f3ba7"},
{file = "lxml-4.9.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:50670615eaf97227d5dc60de2dc99fb134a7130d310d783314e7724bf163f75d"},
{file = "lxml-4.9.3-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:9719fe17307a9e814580af1f5c6e05ca593b12fb7e44fe62450a5384dbf61b4b"},
{file = "lxml-4.9.3-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:3331bece23c9ee066e0fb3f96c61322b9e0f54d775fccefff4c38ca488de283a"},
{file = "lxml-4.9.3-pp39-pypy39_pp73-macosx_11_0_x86_64.whl", hash = "sha256:ed667f49b11360951e201453fc3967344d0d0263aa415e1619e85ae7fd17b4e0"},
{file = "lxml-4.9.3-pp39-pypy39_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_24_i686.whl", hash = "sha256:8b77946fd508cbf0fccd8e400a7f71d4ac0e1595812e66025bac475a8e811694"},
{file = "lxml-4.9.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl", hash = "sha256:e4da8ca0c0c0aea88fd46be8e44bd49716772358d648cce45fe387f7b92374a7"},
{file = "lxml-4.9.3-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:fe4bda6bd4340caa6e5cf95e73f8fea5c4bfc55763dd42f1b50a94c1b4a2fbd4"},
{file = "lxml-4.9.3-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:f3df3db1d336b9356dd3112eae5f5c2b8b377f3bc826848567f10bfddfee77e9"},
{file = "lxml-4.9.3.tar.gz", hash = "sha256:48628bd53a426c9eb9bc066a923acaa0878d1e86129fd5359aee99285f4eed9c"},
]
[package.extras]
cssselect = ["cssselect (>=0.7)"]
html5 = ["html5lib"]
htmlsoup = ["BeautifulSoup4"]
source = ["Cython (>=0.29.7)"]
source = ["Cython (>=0.29.35)"]
[[package]]
name = "mccabe"
version = "0.6.1"
description = "McCabe checker, plugin for flake8"
category = "dev"
optional = false
python-versions = "*"
files = [
@ -412,6 +447,7 @@ files = [
name = "mwclient"
version = "0.10.1"
description = "MediaWiki API client"
category = "main"
optional = false
python-versions = "*"
files = [
@ -427,6 +463,7 @@ six = "*"
name = "nodeenv"
version = "1.8.0"
description = "Node.js virtual environment builder"
category = "dev"
optional = false
python-versions = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*"
files = [
@ -441,6 +478,7 @@ setuptools = "*"
name = "oauthlib"
version = "3.2.2"
description = "A generic, spec-compliant, thorough implementation of the OAuth request-signing logic"
category = "main"
optional = false
python-versions = ">=3.6"
files = [
@ -457,6 +495,7 @@ signedtoken = ["cryptography (>=3.0.0)", "pyjwt (>=2.0.0,<3)"]
name = "packaging"
version = "23.1"
description = "Core utilities for Python packages"
category = "dev"
optional = false
python-versions = ">=3.7"
files = [
@ -466,23 +505,25 @@ files = [
[[package]]
name = "platformdirs"
version = "3.8.0"
version = "3.10.0"
description = "A small Python package for determining appropriate platform-specific dirs, e.g. a \"user data dir\"."
category = "dev"
optional = false
python-versions = ">=3.7"
files = [
{file = "platformdirs-3.8.0-py3-none-any.whl", hash = "sha256:ca9ed98ce73076ba72e092b23d3c93ea6c4e186b3f1c3dad6edd98ff6ffcca2e"},
{file = "platformdirs-3.8.0.tar.gz", hash = "sha256:b0cabcb11063d21a0b261d557acb0a9d2126350e63b70cdf7db6347baea456dc"},
{file = "platformdirs-3.10.0-py3-none-any.whl", hash = "sha256:d7c24979f292f916dc9cbf8648319032f551ea8c49a4c9bf2fb556a02070ec1d"},
{file = "platformdirs-3.10.0.tar.gz", hash = "sha256:b45696dab2d7cc691a3226759c0d3b00c47c8b6e293d96f6436f733303f77f6d"},
]
[package.extras]
docs = ["furo (>=2023.5.20)", "proselint (>=0.13)", "sphinx (>=7.0.1)", "sphinx-autodoc-typehints (>=1.23,!=1.23.4)"]
test = ["appdirs (==1.4.4)", "covdefaults (>=2.3)", "pytest (>=7.3.1)", "pytest-cov (>=4.1)", "pytest-mock (>=3.10)"]
docs = ["furo (>=2023.7.26)", "proselint (>=0.13)", "sphinx (>=7.1.1)", "sphinx-autodoc-typehints (>=1.24)"]
test = ["appdirs (==1.4.4)", "covdefaults (>=2.3)", "pytest (>=7.4)", "pytest-cov (>=4.1)", "pytest-mock (>=3.11.1)"]
[[package]]
name = "pluggy"
version = "1.2.0"
description = "plugin and hook calling mechanisms for python"
category = "dev"
optional = false
python-versions = ">=3.7"
files = [
@ -498,6 +539,7 @@ testing = ["pytest", "pytest-benchmark"]
name = "poster3"
version = "0.8.1"
description = "Streaming HTTP uploads and multipart/form-data encoding"
category = "main"
optional = false
python-versions = "*"
files = [
@ -511,6 +553,7 @@ poster3 = ["buildutils", "sphinx"]
name = "pre-commit"
version = "2.21.0"
description = "A framework for managing and maintaining multi-language pre-commit hooks."
category = "dev"
optional = false
python-versions = ">=3.7"
files = [
@ -529,6 +572,7 @@ virtualenv = ">=20.10.0"
name = "pre-commit-poetry-export"
version = "0.1.2"
description = "pre-commit hook to keep requirements.txt updated"
category = "main"
optional = false
python-versions = ">=3.8,<4.0"
files = [
@ -540,6 +584,7 @@ files = [
name = "py"
version = "1.11.0"
description = "library with cross-python path, ini-parsing, io, code, log facilities"
category = "dev"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
files = [
@ -551,6 +596,7 @@ files = [
name = "pycodestyle"
version = "2.7.0"
description = "Python style guide checker"
category = "dev"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
files = [
@ -562,6 +608,7 @@ files = [
name = "pyflakes"
version = "2.3.1"
description = "passive checker of Python programs"
category = "dev"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
files = [
@ -569,10 +616,25 @@ files = [
{file = "pyflakes-2.3.1.tar.gz", hash = "sha256:f5bc8ecabc05bb9d291eb5203d6810b49040f6ff446a756326104746cc00c1db"},
]
[[package]]
name = "pymarkdown"
version = "0.1.4"
description = "Evaluate code in markdown"
category = "dev"
optional = false
python-versions = "*"
files = [
{file = "pymarkdown-0.1.4.tar.gz", hash = "sha256:680e36d2d81148a0f9ff4b932b8e30517a591f3f719215940a6672c163cf6de8"},
]
[package.dependencies]
toolz = "*"
[[package]]
name = "pymysql"
version = "1.1.0"
description = "Pure Python MySQL Driver"
category = "main"
optional = false
python-versions = ">=3.7"
files = [
@ -588,6 +650,7 @@ rsa = ["cryptography"]
name = "pytest"
version = "6.2.5"
description = "pytest: simple powerful testing with Python"
category = "dev"
optional = false
python-versions = ">=3.6"
files = [
@ -612,6 +675,7 @@ testing = ["argcomplete", "hypothesis (>=3.56)", "mock", "nose", "requests", "xm
name = "pywikibot"
version = "6.6.5"
description = "Python MediaWiki Bot Framework"
category = "main"
optional = false
python-versions = ">=3.5.0"
files = [
@ -646,57 +710,59 @@ wikitextparser = ["wikitextparser (>=0.47.0)", "wikitextparser (>=0.47.5)"]
[[package]]
name = "pyyaml"
version = "6.0"
version = "6.0.1"
description = "YAML parser and emitter for Python"
category = "dev"
optional = false
python-versions = ">=3.6"
files = [
{file = "PyYAML-6.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d4db7c7aef085872ef65a8fd7d6d09a14ae91f691dec3e87ee5ee0539d516f53"},
{file = "PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9df7ed3b3d2e0ecfe09e14741b857df43adb5a3ddadc919a2d94fbdf78fea53c"},
{file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77f396e6ef4c73fdc33a9157446466f1cff553d979bd00ecb64385760c6babdc"},
{file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a80a78046a72361de73f8f395f1f1e49f956c6be882eed58505a15f3e430962b"},
{file = "PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5"},
{file = "PyYAML-6.0-cp310-cp310-win32.whl", hash = "sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513"},
{file = "PyYAML-6.0-cp310-cp310-win_amd64.whl", hash = "sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a"},
{file = "PyYAML-6.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:d4b0ba9512519522b118090257be113b9468d804b19d63c71dbcf4a48fa32358"},
{file = "PyYAML-6.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:81957921f441d50af23654aa6c5e5eaf9b06aba7f0a19c18a538dc7ef291c5a1"},
{file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:afa17f5bc4d1b10afd4466fd3a44dc0e245382deca5b3c353d8b757f9e3ecb8d"},
{file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:dbad0e9d368bb989f4515da330b88a057617d16b6a8245084f1b05400f24609f"},
{file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:432557aa2c09802be39460360ddffd48156e30721f5e8d917f01d31694216782"},
{file = "PyYAML-6.0-cp311-cp311-win32.whl", hash = "sha256:bfaef573a63ba8923503d27530362590ff4f576c626d86a9fed95822a8255fd7"},
{file = "PyYAML-6.0-cp311-cp311-win_amd64.whl", hash = "sha256:01b45c0191e6d66c470b6cf1b9531a771a83c1c4208272ead47a3ae4f2f603bf"},
{file = "PyYAML-6.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86"},
{file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9758427f"},
{file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92"},
{file = "PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:98c4d36e99714e55cfbaaee6dd5badbc9a1ec339ebfc3b1f52e293aee6bb71a4"},
{file = "PyYAML-6.0-cp36-cp36m-win32.whl", hash = "sha256:0283c35a6a9fbf047493e3a0ce8d79ef5030852c51e9d911a27badfde0605293"},
{file = "PyYAML-6.0-cp36-cp36m-win_amd64.whl", hash = "sha256:07751360502caac1c067a8132d150cf3d61339af5691fe9e87803040dbc5db57"},
{file = "PyYAML-6.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:819b3830a1543db06c4d4b865e70ded25be52a2e0631ccd2f6a47a2822f2fd7c"},
{file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:473f9edb243cb1935ab5a084eb238d842fb8f404ed2193a915d1784b5a6b5fc0"},
{file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0ce82d761c532fe4ec3f87fc45688bdd3a4c1dc5e0b4a19814b9009a29baefd4"},
{file = "PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:231710d57adfd809ef5d34183b8ed1eeae3f76459c18fb4a0b373ad56bedcdd9"},
{file = "PyYAML-6.0-cp37-cp37m-win32.whl", hash = "sha256:c5687b8d43cf58545ade1fe3e055f70eac7a5a1a0bf42824308d868289a95737"},
{file = "PyYAML-6.0-cp37-cp37m-win_amd64.whl", hash = "sha256:d15a181d1ecd0d4270dc32edb46f7cb7733c7c508857278d3d378d14d606db2d"},
{file = "PyYAML-6.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0b4624f379dab24d3725ffde76559cff63d9ec94e1736b556dacdfebe5ab6d4b"},
{file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:213c60cd50106436cc818accf5baa1aba61c0189ff610f64f4a3e8c6726218ba"},
{file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9fa600030013c4de8165339db93d182b9431076eb98eb40ee068700c9c813e34"},
{file = "PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:277a0ef2981ca40581a47093e9e2d13b3f1fbbeffae064c1d21bfceba2030287"},
{file = "PyYAML-6.0-cp38-cp38-win32.whl", hash = "sha256:d4eccecf9adf6fbcc6861a38015c2a64f38b9d94838ac1810a9023a0609e1b78"},
{file = "PyYAML-6.0-cp38-cp38-win_amd64.whl", hash = "sha256:1e4747bc279b4f613a09eb64bba2ba602d8a6664c6ce6396a4d0cd413a50ce07"},
{file = "PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:055d937d65826939cb044fc8c9b08889e8c743fdc6a32b33e2390f66013e449b"},
{file = "PyYAML-6.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e61ceaab6f49fb8bdfaa0f92c4b57bcfbea54c09277b1b4f7ac376bfb7a7c174"},
{file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d67d839ede4ed1b28a4e8909735fc992a923cdb84e618544973d7dfc71540803"},
{file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cba8c411ef271aa037d7357a2bc8f9ee8b58b9965831d9e51baf703280dc73d3"},
{file = "PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:40527857252b61eacd1d9af500c3337ba8deb8fc298940291486c465c8b46ec0"},
{file = "PyYAML-6.0-cp39-cp39-win32.whl", hash = "sha256:b5b9eccad747aabaaffbc6064800670f0c297e52c12754eb1d976c57e4f74dcb"},
{file = "PyYAML-6.0-cp39-cp39-win_amd64.whl", hash = "sha256:b3d267842bf12586ba6c734f89d1f5b871df0273157918b0ccefa29deb05c21c"},
{file = "PyYAML-6.0.tar.gz", hash = "sha256:68fb519c14306fec9720a2a5b45bc9f0c8d1b9c72adf45c37baedfcd949c35a2"},
{file = "PyYAML-6.0.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d858aa552c999bc8a8d57426ed01e40bef403cd8ccdd0fc5f6f04a00414cac2a"},
{file = "PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:fd66fc5d0da6d9815ba2cebeb4205f95818ff4b79c3ebe268e75d961704af52f"},
{file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:69b023b2b4daa7548bcfbd4aa3da05b3a74b772db9e23b982788168117739938"},
{file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:81e0b275a9ecc9c0c0c07b4b90ba548307583c125f54d5b6946cfee6360c733d"},
{file = "PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ba336e390cd8e4d1739f42dfe9bb83a3cc2e80f567d8805e11b46f4a943f5515"},
{file = "PyYAML-6.0.1-cp310-cp310-win32.whl", hash = "sha256:bd4af7373a854424dabd882decdc5579653d7868b8fb26dc7d0e99f823aa5924"},
{file = "PyYAML-6.0.1-cp310-cp310-win_amd64.whl", hash = "sha256:fd1592b3fdf65fff2ad0004b5e363300ef59ced41c2e6b3a99d4089fa8c5435d"},
{file = "PyYAML-6.0.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6965a7bc3cf88e5a1c3bd2e0b5c22f8d677dc88a455344035f03399034eb3007"},
{file = "PyYAML-6.0.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:f003ed9ad21d6a4713f0a9b5a7a0a79e08dd0f221aff4525a2be4c346ee60aab"},
{file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:42f8152b8dbc4fe7d96729ec2b99c7097d656dc1213a3229ca5383f973a5ed6d"},
{file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:062582fca9fabdd2c8b54a3ef1c978d786e0f6b3a1510e0ac93ef59e0ddae2bc"},
{file = "PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d2b04aac4d386b172d5b9692e2d2da8de7bfb6c387fa4f801fbf6fb2e6ba4673"},
{file = "PyYAML-6.0.1-cp311-cp311-win32.whl", hash = "sha256:1635fd110e8d85d55237ab316b5b011de701ea0f29d07611174a1b42f1444741"},
{file = "PyYAML-6.0.1-cp311-cp311-win_amd64.whl", hash = "sha256:bf07ee2fef7014951eeb99f56f39c9bb4af143d8aa3c21b1677805985307da34"},
{file = "PyYAML-6.0.1-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:50550eb667afee136e9a77d6dc71ae76a44df8b3e51e41b77f6de2932bfe0f47"},
{file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1fe35611261b29bd1de0070f0b2f47cb6ff71fa6595c077e42bd0c419fa27b98"},
{file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:704219a11b772aea0d8ecd7058d0082713c3562b4e271b849ad7dc4a5c90c13c"},
{file = "PyYAML-6.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:afd7e57eddb1a54f0f1a974bc4391af8bcce0b444685d936840f125cf046d5bd"},
{file = "PyYAML-6.0.1-cp36-cp36m-win32.whl", hash = "sha256:fca0e3a251908a499833aa292323f32437106001d436eca0e6e7833256674585"},
{file = "PyYAML-6.0.1-cp36-cp36m-win_amd64.whl", hash = "sha256:f22ac1c3cac4dbc50079e965eba2c1058622631e526bd9afd45fedd49ba781fa"},
{file = "PyYAML-6.0.1-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:b1275ad35a5d18c62a7220633c913e1b42d44b46ee12554e5fd39c70a243d6a3"},
{file = "PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:18aeb1bf9a78867dc38b259769503436b7c72f7a1f1f4c93ff9a17de54319b27"},
{file = "PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:596106435fa6ad000c2991a98fa58eeb8656ef2325d7e158344fb33864ed87e3"},
{file = "PyYAML-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:baa90d3f661d43131ca170712d903e6295d1f7a0f595074f151c0aed377c9b9c"},
{file = "PyYAML-6.0.1-cp37-cp37m-win32.whl", hash = "sha256:9046c58c4395dff28dd494285c82ba00b546adfc7ef001486fbf0324bc174fba"},
{file = "PyYAML-6.0.1-cp37-cp37m-win_amd64.whl", hash = "sha256:4fb147e7a67ef577a588a0e2c17b6db51dda102c71de36f8549b6816a96e1867"},
{file = "PyYAML-6.0.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:1d4c7e777c441b20e32f52bd377e0c409713e8bb1386e1099c2415f26e479595"},
{file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a0cd17c15d3bb3fa06978b4e8958dcdc6e0174ccea823003a106c7d4d7899ac5"},
{file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:28c119d996beec18c05208a8bd78cbe4007878c6dd15091efb73a30e90539696"},
{file = "PyYAML-6.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7e07cbde391ba96ab58e532ff4803f79c4129397514e1413a7dc761ccd755735"},
{file = "PyYAML-6.0.1-cp38-cp38-win32.whl", hash = "sha256:184c5108a2aca3c5b3d3bf9395d50893a7ab82a38004c8f61c258d4428e80206"},
{file = "PyYAML-6.0.1-cp38-cp38-win_amd64.whl", hash = "sha256:1e2722cc9fbb45d9b87631ac70924c11d3a401b2d7f410cc0e3bbf249f2dca62"},
{file = "PyYAML-6.0.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9eb6caa9a297fc2c2fb8862bc5370d0303ddba53ba97e71f08023b6cd73d16a8"},
{file = "PyYAML-6.0.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:c8098ddcc2a85b61647b2590f825f3db38891662cfc2fc776415143f599bb859"},
{file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5773183b6446b2c99bb77e77595dd486303b4faab2b086e7b17bc6bef28865f6"},
{file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b786eecbdf8499b9ca1d697215862083bd6d2a99965554781d0d8d1ad31e13a0"},
{file = "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bc1bf2925a1ecd43da378f4db9e4f799775d6367bdb94671027b73b393a7c42c"},
{file = "PyYAML-6.0.1-cp39-cp39-win32.whl", hash = "sha256:faca3bdcf85b2fc05d06ff3fbc1f83e1391b3e724afa3feba7d13eeab355484c"},
{file = "PyYAML-6.0.1-cp39-cp39-win_amd64.whl", hash = "sha256:510c9deebc5c0225e8c96813043e62b680ba2f9c50a08d3724c7f28a747d1486"},
{file = "PyYAML-6.0.1.tar.gz", hash = "sha256:bfdf460b1736c775f2ba9f6a92bca30bc2095067b8a9d77876d1fad6cc3b4a43"},
]
[[package]]
name = "requests"
version = "2.31.0"
description = "Python HTTP for Humans."
category = "main"
optional = false
python-versions = ">=3.7"
files = [
@ -718,6 +784,7 @@ use-chardet-on-py3 = ["chardet (>=3.0.2,<6)"]
name = "requests-oauthlib"
version = "1.3.1"
description = "OAuthlib authentication support for Requests."
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
files = [
@ -736,6 +803,7 @@ rsa = ["oauthlib[signedtoken] (>=3.0.0)"]
name = "schema"
version = "0.7.5"
description = "Simple data validation library"
category = "main"
optional = false
python-versions = "*"
files = [
@ -748,24 +816,26 @@ contextlib2 = ">=0.5.5"
[[package]]
name = "setuptools"
version = "68.0.0"
version = "68.1.2"
description = "Easily download, build, install, upgrade, and uninstall Python packages"
category = "main"
optional = false
python-versions = ">=3.7"
python-versions = ">=3.8"
files = [
{file = "setuptools-68.0.0-py3-none-any.whl", hash = "sha256:11e52c67415a381d10d6b462ced9cfb97066179f0e871399e006c4ab101fc85f"},
{file = "setuptools-68.0.0.tar.gz", hash = "sha256:baf1fdb41c6da4cd2eae722e135500da913332ab3f2f5c7d33af9b492acb5235"},
{file = "setuptools-68.1.2-py3-none-any.whl", hash = "sha256:3d8083eed2d13afc9426f227b24fd1659489ec107c0e86cec2ffdde5c92e790b"},
{file = "setuptools-68.1.2.tar.gz", hash = "sha256:3d4dfa6d95f1b101d695a6160a7626e15583af71a5f52176efa5d39a054d475d"},
]
[package.extras]
docs = ["furo", "jaraco.packaging (>=9)", "jaraco.tidelift (>=1.4)", "pygments-github-lexers (==0.0.5)", "rst.linker (>=1.9)", "sphinx (>=3.5)", "sphinx-favicon", "sphinx-hoverxref (<2)", "sphinx-inline-tabs", "sphinx-lint", "sphinx-notfound-page (==0.8.3)", "sphinx-reredirects", "sphinxcontrib-towncrier"]
testing = ["build[virtualenv]", "filelock (>=3.4.0)", "flake8-2020", "ini2toml[lite] (>=0.9)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.2.0)", "pip (>=19.1)", "pip-run (>=8.8)", "pytest (>=6)", "pytest-black (>=0.3.7)", "pytest-checkdocs (>=2.4)", "pytest-cov", "pytest-enabler (>=1.3)", "pytest-mypy (>=0.9.1)", "pytest-perf", "pytest-ruff", "pytest-timeout", "pytest-xdist", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel"]
docs = ["furo", "jaraco.packaging (>=9.3)", "jaraco.tidelift (>=1.4)", "pygments-github-lexers (==0.0.5)", "rst.linker (>=1.9)", "sphinx (>=3.5,<=7.1.2)", "sphinx-favicon", "sphinx-hoverxref (<2)", "sphinx-inline-tabs", "sphinx-lint", "sphinx-notfound-page (==0.8.3)", "sphinx-reredirects", "sphinxcontrib-towncrier"]
testing = ["build[virtualenv]", "filelock (>=3.4.0)", "flake8-2020", "ini2toml[lite] (>=0.9)", "jaraco.develop (>=7.21)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.2.0)", "pip (>=19.1)", "pytest (>=6)", "pytest-black (>=0.3.7)", "pytest-checkdocs (>=2.4)", "pytest-cov", "pytest-enabler (>=2.2)", "pytest-mypy (>=0.9.1)", "pytest-perf", "pytest-ruff", "pytest-timeout", "pytest-xdist", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel"]
testing-integration = ["build[virtualenv]", "filelock (>=3.4.0)", "jaraco.envs (>=2.2)", "jaraco.path (>=3.2.0)", "pytest", "pytest-enabler", "pytest-xdist", "tomli", "virtualenv (>=13.0.0)", "wheel"]
[[package]]
name = "six"
version = "1.16.0"
description = "Python 2 and 3 compatibility utilities"
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*"
files = [
@ -777,6 +847,7 @@ files = [
name = "toml"
version = "0.10.2"
description = "Python Library for Tom's Obvious, Minimal Language"
category = "dev"
optional = false
python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"
files = [
@ -784,22 +855,35 @@ files = [
{file = "toml-0.10.2.tar.gz", hash = "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f"},
]
[[package]]
name = "toolz"
version = "0.12.0"
description = "List processing tools and functional utilities"
category = "dev"
optional = false
python-versions = ">=3.5"
files = [
{file = "toolz-0.12.0-py3-none-any.whl", hash = "sha256:2059bd4148deb1884bb0eb770a3cde70e7f954cfbbdc2285f1f2de01fd21eb6f"},
{file = "toolz-0.12.0.tar.gz", hash = "sha256:88c570861c440ee3f2f6037c4654613228ff40c93a6c25e0eba70d17282c6194"},
]
[[package]]
name = "tqdm"
version = "4.65.0"
version = "4.66.1"
description = "Fast, Extensible Progress Meter"
category = "main"
optional = false
python-versions = ">=3.7"
files = [
{file = "tqdm-4.65.0-py3-none-any.whl", hash = "sha256:c4f53a17fe37e132815abceec022631be8ffe1b9381c2e6e30aa70edc99e9671"},
{file = "tqdm-4.65.0.tar.gz", hash = "sha256:1871fb68a86b8fb3b59ca4cdd3dcccbc7e6d613eeed31f4c332531977b89beb5"},
{file = "tqdm-4.66.1-py3-none-any.whl", hash = "sha256:d302b3c5b53d47bce91fea46679d9c3c6508cf6332229aa1e7d8653723793386"},
{file = "tqdm-4.66.1.tar.gz", hash = "sha256:d88e651f9db8d8551a62556d3cff9e3034274ca5d66e93197cf2490e2dcb69c7"},
]
[package.dependencies]
colorama = {version = "*", markers = "platform_system == \"Windows\""}
[package.extras]
dev = ["py-make (>=0.1.0)", "twine", "wheel"]
dev = ["pytest (>=6)", "pytest-cov", "pytest-timeout", "pytest-xdist"]
notebook = ["ipywidgets (>=6)"]
slack = ["slack-sdk"]
telegram = ["requests"]
@ -808,6 +892,7 @@ telegram = ["requests"]
name = "urllib3"
version = "1.26.16"
description = "HTTP library with thread-safe connection pooling, file post, and more."
category = "main"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*"
files = [
@ -822,28 +907,30 @@ socks = ["PySocks (>=1.5.6,!=1.5.7,<2.0)"]
[[package]]
name = "virtualenv"
version = "20.23.1"
version = "20.24.3"
description = "Virtual Python Environment builder"
category = "dev"
optional = false
python-versions = ">=3.7"
files = [
{file = "virtualenv-20.23.1-py3-none-any.whl", hash = "sha256:34da10f14fea9be20e0fd7f04aba9732f84e593dac291b757ce42e3368a39419"},
{file = "virtualenv-20.23.1.tar.gz", hash = "sha256:8ff19a38c1021c742148edc4f81cb43d7f8c6816d2ede2ab72af5b84c749ade1"},
{file = "virtualenv-20.24.3-py3-none-any.whl", hash = "sha256:95a6e9398b4967fbcb5fef2acec5efaf9aa4972049d9ae41f95e0972a683fd02"},
{file = "virtualenv-20.24.3.tar.gz", hash = "sha256:e5c3b4ce817b0b328af041506a2a299418c98747c4b1e68cb7527e74ced23efc"},
]
[package.dependencies]
distlib = ">=0.3.6,<1"
filelock = ">=3.12,<4"
platformdirs = ">=3.5.1,<4"
distlib = ">=0.3.7,<1"
filelock = ">=3.12.2,<4"
platformdirs = ">=3.9.1,<4"
[package.extras]
docs = ["furo (>=2023.5.20)", "proselint (>=0.13)", "sphinx (>=7.0.1)", "sphinx-argparse (>=0.4)", "sphinxcontrib-towncrier (>=0.2.1a0)", "towncrier (>=23.6)"]
test = ["covdefaults (>=2.3)", "coverage (>=7.2.7)", "coverage-enable-subprocess (>=1)", "flaky (>=3.7)", "packaging (>=23.1)", "pytest (>=7.3.1)", "pytest-env (>=0.8.1)", "pytest-freezer (>=0.4.6)", "pytest-mock (>=3.10)", "pytest-randomly (>=3.12)", "pytest-timeout (>=2.1)", "setuptools (>=67.8)", "time-machine (>=2.9)"]
test = ["covdefaults (>=2.3)", "coverage (>=7.2.7)", "coverage-enable-subprocess (>=1)", "flaky (>=3.7)", "packaging (>=23.1)", "pytest (>=7.4)", "pytest-env (>=0.8.2)", "pytest-freezer (>=0.4.8)", "pytest-mock (>=3.11.1)", "pytest-randomly (>=3.12)", "pytest-timeout (>=2.1)", "setuptools (>=68)", "time-machine (>=2.10)"]
[[package]]
name = "wikitools3"
version = "3.0.1"
description = "Python package for interacting with a MediaWiki wiki. It is used by WikiTeam for archiving MediaWiki wikis."
category = "main"
optional = false
python-versions = ">=3.8,<4.0"
files = [
@ -857,4 +944,4 @@ poster3 = ">=0.8.1,<0.9.0"
[metadata]
lock-version = "2.0"
python-versions = "^3.8"
content-hash = "83bc813a2137937dc173270998983a3bddd9ae2f5c3d80468787b3b38ba2ac7c"
content-hash = "1eee6035c5660e8cba28942140937e2ceb36bf90482e76fa5ddd054efa3c659c"

4
pyproject.toml
Unescape Escape View File

@ -76,7 +76,11 @@ pytest = "^6.2.5"
requests = "^2.31.0"
flake8 = "^3.9.2"
pre-commit = "^2.17.0"
pymarkdown = "^0.1.4"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.pymarkdown]
disable-rules = "line-length,no-inline-html"

342
requirements.txt
Unescape Escape View File

@ -1,136 +1,234 @@
certifi==2021.10.8 ; python_version >= "2.7" and python_full_version < "3.0.0" or python_version >= "3.6" \
--hash=sha256:d62a0163eb4c2344ac042ab2bdf75399a71a2d8c7d47eac2e2ee91b9d6339569 \
--hash=sha256:78884e7c1d4b00ce3cea67b44566851c4343c120abd683433ce934a68ea58872
charset-normalizer==2.0.12 ; python_version >= "3.6" \
--hash=sha256:2857e29ff0d34db842cd7ca3230549d1a697f96ee6d3fb071cfa6c7393832597 \
--hash=sha256:6881edbebdb17b39b4eaaa821b438bf6eddffb4468cf344f09f89def34a8b1df
colorama==0.4.4 ; python_version >= "2.7" and python_full_version < "3.0.0" and platform_system == "Windows" or python_full_version >= "3.5.0" and platform_system == "Windows" \
--hash=sha256:9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2 \
--hash=sha256:5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b
contextlib2==21.6.0 ; python_version >= "3.6" \
certifi==2023.7.22 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:539cc1d13202e33ca466e88b2807e29f4c13049d6d87031a3c110744495cb082 \
--hash=sha256:92d6037539857d8206b8f6ae472e8b77db8058fec5937a1ef3f54304089edbb9
charset-normalizer==3.2.0 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:04e57ab9fbf9607b77f7d057974694b4f6b142da9ed4a199859d9d4d5c63fe96 \
--hash=sha256:09393e1b2a9461950b1c9a45d5fd251dc7c6f228acab64da1c9c0165d9c7765c \
--hash=sha256:0b87549028f680ca955556e3bd57013ab47474c3124dc069faa0b6545b6c9710 \
--hash=sha256:1000fba1057b92a65daec275aec30586c3de2401ccdcd41f8a5c1e2c87078706 \
--hash=sha256:1249cbbf3d3b04902ff081ffbb33ce3377fa6e4c7356f759f3cd076cc138d020 \
--hash=sha256:1920d4ff15ce893210c1f0c0e9d19bfbecb7983c76b33f046c13a8ffbd570252 \
--hash=sha256:193cbc708ea3aca45e7221ae58f0fd63f933753a9bfb498a3b474878f12caaad \
--hash=sha256:1a100c6d595a7f316f1b6f01d20815d916e75ff98c27a01ae817439ea7726329 \
--hash=sha256:1f30b48dd7fa1474554b0b0f3fdfdd4c13b5c737a3c6284d3cdc424ec0ffff3a \
--hash=sha256:203f0c8871d5a7987be20c72442488a0b8cfd0f43b7973771640fc593f56321f \
--hash=sha256:246de67b99b6851627d945db38147d1b209a899311b1305dd84916f2b88526c6 \
--hash=sha256:2dee8e57f052ef5353cf608e0b4c871aee320dd1b87d351c28764fc0ca55f9f4 \
--hash=sha256:2efb1bd13885392adfda4614c33d3b68dee4921fd0ac1d3988f8cbb7d589e72a \
--hash=sha256:2f4ac36d8e2b4cc1aa71df3dd84ff8efbe3bfb97ac41242fbcfc053c67434f46 \
--hash=sha256:3170c9399da12c9dc66366e9d14da8bf7147e1e9d9ea566067bbce7bb74bd9c2 \
--hash=sha256:3b1613dd5aee995ec6d4c69f00378bbd07614702a315a2cf6c1d21461fe17c23 \
--hash=sha256:3bb3d25a8e6c0aedd251753a79ae98a093c7e7b471faa3aa9a93a81431987ace \
--hash=sha256:3bb7fda7260735efe66d5107fb7e6af6a7c04c7fce9b2514e04b7a74b06bf5dd \
--hash=sha256:41b25eaa7d15909cf3ac4c96088c1f266a9a93ec44f87f1d13d4a0e86c81b982 \
--hash=sha256:45de3f87179c1823e6d9e32156fb14c1927fcc9aba21433f088fdfb555b77c10 \
--hash=sha256:46fb8c61d794b78ec7134a715a3e564aafc8f6b5e338417cb19fe9f57a5a9bf2 \
--hash=sha256:48021783bdf96e3d6de03a6e39a1171ed5bd7e8bb93fc84cc649d11490f87cea \
--hash=sha256:4957669ef390f0e6719db3613ab3a7631e68424604a7b448f079bee145da6e09 \
--hash=sha256:5e86d77b090dbddbe78867a0275cb4df08ea195e660f1f7f13435a4649e954e5 \
--hash=sha256:6339d047dab2780cc6220f46306628e04d9750f02f983ddb37439ca47ced7149 \
--hash=sha256:681eb3d7e02e3c3655d1b16059fbfb605ac464c834a0c629048a30fad2b27489 \
--hash=sha256:6c409c0deba34f147f77efaa67b8e4bb83d2f11c8806405f76397ae5b8c0d1c9 \
--hash=sha256:7095f6fbfaa55defb6b733cfeb14efaae7a29f0b59d8cf213be4e7ca0b857b80 \
--hash=sha256:70c610f6cbe4b9fce272c407dd9d07e33e6bf7b4aa1b7ffb6f6ded8e634e3592 \
--hash=sha256:72814c01533f51d68702802d74f77ea026b5ec52793c791e2da806a3844a46c3 \
--hash=sha256:7a4826ad2bd6b07ca615c74ab91f32f6c96d08f6fcc3902ceeedaec8cdc3bcd6 \
--hash=sha256:7c70087bfee18a42b4040bb9ec1ca15a08242cf5867c58726530bdf3945672ed \
--hash=sha256:855eafa5d5a2034b4621c74925d89c5efef61418570e5ef9b37717d9c796419c \
--hash=sha256:8700f06d0ce6f128de3ccdbc1acaea1ee264d2caa9ca05daaf492fde7c2a7200 \
--hash=sha256:89f1b185a01fe560bc8ae5f619e924407efca2191b56ce749ec84982fc59a32a \
--hash=sha256:8b2c760cfc7042b27ebdb4a43a4453bd829a5742503599144d54a032c5dc7e9e \
--hash=sha256:8c2f5e83493748286002f9369f3e6607c565a6a90425a3a1fef5ae32a36d749d \
--hash=sha256:8e098148dd37b4ce3baca71fb394c81dc5d9c7728c95df695d2dca218edf40e6 \
--hash=sha256:94aea8eff76ee6d1cdacb07dd2123a68283cb5569e0250feab1240058f53b623 \
--hash=sha256:95eb302ff792e12aba9a8b8f8474ab229a83c103d74a750ec0bd1c1eea32e669 \
--hash=sha256:9bd9b3b31adcb054116447ea22caa61a285d92e94d710aa5ec97992ff5eb7cf3 \
--hash=sha256:9e608aafdb55eb9f255034709e20d5a83b6d60c054df0802fa9c9883d0a937aa \
--hash=sha256:a103b3a7069b62f5d4890ae1b8f0597618f628b286b03d4bc9195230b154bfa9 \
--hash=sha256:a386ebe437176aab38c041de1260cd3ea459c6ce5263594399880bbc398225b2 \
--hash=sha256:a38856a971c602f98472050165cea2cdc97709240373041b69030be15047691f \
--hash=sha256:a401b4598e5d3f4a9a811f3daf42ee2291790c7f9d74b18d75d6e21dda98a1a1 \
--hash=sha256:a7647ebdfb9682b7bb97e2a5e7cb6ae735b1c25008a70b906aecca294ee96cf4 \
--hash=sha256:aaf63899c94de41fe3cf934601b0f7ccb6b428c6e4eeb80da72c58eab077b19a \
--hash=sha256:b0dac0ff919ba34d4df1b6131f59ce95b08b9065233446be7e459f95554c0dc8 \
--hash=sha256:baacc6aee0b2ef6f3d308e197b5d7a81c0e70b06beae1f1fcacffdbd124fe0e3 \
--hash=sha256:bf420121d4c8dce6b889f0e8e4ec0ca34b7f40186203f06a946fa0276ba54029 \
--hash=sha256:c04a46716adde8d927adb9457bbe39cf473e1e2c2f5d0a16ceb837e5d841ad4f \
--hash=sha256:c0b21078a4b56965e2b12f247467b234734491897e99c1d51cee628da9786959 \
--hash=sha256:c1c76a1743432b4b60ab3358c937a3fe1341c828ae6194108a94c69028247f22 \
--hash=sha256:c4983bf937209c57240cff65906b18bb35e64ae872da6a0db937d7b4af845dd7 \
--hash=sha256:c4fb39a81950ec280984b3a44f5bd12819953dc5fa3a7e6fa7a80db5ee853952 \
--hash=sha256:c57921cda3a80d0f2b8aec7e25c8aa14479ea92b5b51b6876d975d925a2ea346 \
--hash=sha256:c8063cf17b19661471ecbdb3df1c84f24ad2e389e326ccaf89e3fb2484d8dd7e \
--hash=sha256:ccd16eb18a849fd8dcb23e23380e2f0a354e8daa0c984b8a732d9cfaba3a776d \
--hash=sha256:cd6dbe0238f7743d0efe563ab46294f54f9bc8f4b9bcf57c3c666cc5bc9d1299 \
--hash=sha256:d62e51710986674142526ab9f78663ca2b0726066ae26b78b22e0f5e571238dd \
--hash=sha256:db901e2ac34c931d73054d9797383d0f8009991e723dab15109740a63e7f902a \
--hash=sha256:e03b8895a6990c9ab2cdcd0f2fe44088ca1c65ae592b8f795c3294af00a461c3 \
--hash=sha256:e1c8a2f4c69e08e89632defbfabec2feb8a8d99edc9f89ce33c4b9e36ab63037 \
--hash=sha256:e4b749b9cc6ee664a3300bb3a273c1ca8068c46be705b6c31cf5d276f8628a94 \
--hash=sha256:e6a5bf2cba5ae1bb80b154ed68a3cfa2fa00fde979a7f50d6598d3e17d9ac20c \
--hash=sha256:e857a2232ba53ae940d3456f7533ce6ca98b81917d47adc3c7fd55dad8fab858 \
--hash=sha256:ee4006268ed33370957f55bf2e6f4d263eaf4dc3cfc473d1d90baff6ed36ce4a \
--hash=sha256:eef9df1eefada2c09a5e7a40991b9fc6ac6ef20b1372abd48d2794a316dc0449 \
--hash=sha256:f058f6963fd82eb143c692cecdc89e075fa0828db2e5b291070485390b2f1c9c \
--hash=sha256:f25c229a6ba38a35ae6e25ca1264621cc25d4d38dca2942a7fce0b67a4efe918 \
--hash=sha256:f2a1d0fd4242bd8643ce6f98927cf9c04540af6efa92323e9d3124f57727bfc1 \
--hash=sha256:f7560358a6811e52e9c4d142d497f1a6e10103d3a6881f18d04dbce3729c0e2c \
--hash=sha256:f779d3ad205f108d14e99bb3859aa7dd8e9c68874617c72354d7ecaec2a054ac \
--hash=sha256:f87f746ee241d30d6ed93969de31e5ffd09a2961a051e60ae6bddde9ec3583aa
colorama==0.4.6 ; python_version >= "3.8" and python_version < "4.0" and platform_system == "Windows" \
--hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \
--hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6
contextlib2==21.6.0 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:3fbdb64466afd23abaf6c977627b75b6139a5a3e8ce38405c5b413aed7a0471f \
--hash=sha256:ab1e2bfe1d01d968e1b7e8d9023bc51ef3509bba217bb730cee3827e1ee82869
docopt==0.6.2 \
docopt==0.6.2 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:49b3a825280bd66b3aa83585ef59c4a8c82f2c8a522dbe754a8bc8d08c85c491
file-read-backwards==2.0.0 \
file-read-backwards==2.0.0 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:78684aafc0471e9e0f62a8662361bfaf70ade3d38333d7f166fd8024a028b1d1 \
--hash=sha256:fd50d9089b412147ea3c6027e2ad905f977002db2918cf315d64eed23d6d6eb8
idna==3.3 ; python_version >= "3.6" \
--hash=sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff \
--hash=sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d
internetarchive==2.3.0 \
--hash=sha256:fa89dc4be3e0a0aee24810a4a754e24adfd07edf710c645b4f642422c6078b8d
jsonpatch==1.32 ; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" \
--hash=sha256:26ac385719ac9f54df8a2f0827bb8253aa3ea8ab7b3368457bcdb8c14595a397 \
--hash=sha256:b6ddfe6c3db30d81a96aaeceb6baf916094ffa23d7dd5fa2c13e13f8b6e600c2
jsonpointer==2.2 ; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" \
--hash=sha256:26d9a47a72d4dc3e3ae72c4c6cd432afd73c680164cd2540772eab53cb3823b6 \
--hash=sha256:f09f8deecaaa5aea65b5eb4f67ca4e54e1a61f7a11c75085e360fe6feb6a48bf
lxml==4.9.1 \
--hash=sha256:98cafc618614d72b02185ac583c6f7796202062c41d2eeecdf07820bad3295ed \
--hash=sha256:c62e8dd9754b7debda0c5ba59d34509c4688f853588d75b53c3791983faa96fc \
--hash=sha256:21fb3d24ab430fc538a96e9fbb9b150029914805d551deeac7d7822f64631dfc \
--hash=sha256:86e92728ef3fc842c50a5cb1d5ba2bc66db7da08a7af53fb3da79e202d1b2cd3 \
--hash=sha256:4cfbe42c686f33944e12f45a27d25a492cc0e43e1dc1da5d6a87cbcaf2e95627 \
--hash=sha256:dad7b164905d3e534883281c050180afcf1e230c3d4a54e8038aa5cfcf312b84 \
--hash=sha256:a614e4afed58c14254e67862456d212c4dcceebab2eaa44d627c2ca04bf86837 \
--hash=sha256:f9ced82717c7ec65a67667bb05865ffe38af0e835cdd78728f1209c8fffe0cad \
--hash=sha256:d9fc0bf3ff86c17348dfc5d322f627d78273eba545db865c3cd14b3f19e57fa5 \
--hash=sha256:e5f66bdf0976ec667fc4594d2812a00b07ed14d1b44259d19a41ae3fff99f2b8 \
--hash=sha256:fe17d10b97fdf58155f858606bddb4e037b805a60ae023c009f760d8361a4eb8 \
--hash=sha256:8caf4d16b31961e964c62194ea3e26a0e9561cdf72eecb1781458b67ec83423d \
--hash=sha256:4780677767dd52b99f0af1f123bc2c22873d30b474aa0e2fc3fe5e02217687c7 \
--hash=sha256:b122a188cd292c4d2fcd78d04f863b789ef43aa129b233d7c9004de08693728b \
--hash=sha256:be9eb06489bc975c38706902cbc6888f39e946b81383abc2838d186f0e8b6a9d \
--hash=sha256:f1be258c4d3dc609e654a1dc59d37b17d7fef05df912c01fc2e15eb43a9735f3 \
--hash=sha256:927a9dd016d6033bc12e0bf5dee1dde140235fc8d0d51099353c76081c03dc29 \
--hash=sha256:9232b09f5efee6a495a99ae6824881940d6447debe272ea400c02e3b68aad85d \
--hash=sha256:04da965dfebb5dac2619cb90fcf93efdb35b3c6994fea58a157a834f2f94b318 \
--hash=sha256:4d5bae0a37af799207140652a700f21a85946f107a199bcb06720b13a4f1f0b7 \
--hash=sha256:4878e667ebabe9b65e785ac8da4d48886fe81193a84bbe49f12acff8f7a383a4 \
--hash=sha256:1355755b62c28950f9ce123c7a41460ed9743c699905cbe664a5bcc5c9c7c7fb \
--hash=sha256:bcaa1c495ce623966d9fc8a187da80082334236a2a1c7e141763ffaf7a405067 \
--hash=sha256:6eafc048ea3f1b3c136c71a86db393be36b5b3d9c87b1c25204e7d397cee9536 \
--hash=sha256:13c90064b224e10c14dcdf8086688d3f0e612db53766e7478d7754703295c7c8 \
--hash=sha256:206a51077773c6c5d2ce1991327cda719063a47adc02bd703c56a662cdb6c58b \
--hash=sha256:e8f0c9d65da595cfe91713bc1222af9ecabd37971762cb830dea2fc3b3bb2acf \
--hash=sha256:8f0a4d179c9a941eb80c3a63cdb495e539e064f8054230844dcf2fcb812b71d3 \
--hash=sha256:830c88747dce8a3e7525defa68afd742b4580df6aa2fdd6f0855481e3994d391 \
--hash=sha256:1e1cf47774373777936c5aabad489fef7b1c087dcd1f426b621fda9dcc12994e \
--hash=sha256:5974895115737a74a00b321e339b9c3f45c20275d226398ae79ac008d908bff7 \
--hash=sha256:1423631e3d51008871299525b541413c9b6c6423593e89f9c4cfbe8460afc0a2 \
--hash=sha256:2aaf6a0a6465d39b5ca69688fce82d20088c1838534982996ec46633dc7ad6cc \
--hash=sha256:9f36de4cd0c262dd9927886cc2305aa3f2210db437aa4fed3fb4940b8bf4592c \
--hash=sha256:ae06c1e4bc60ee076292e582a7512f304abdf6c70db59b56745cca1684f875a4 \
--hash=sha256:57e4d637258703d14171b54203fd6822fda218c6c2658a7d30816b10995f29f3 \
--hash=sha256:6d279033bf614953c3fc4a0aa9ac33a21e8044ca72d4fa8b9273fe75359d5cca \
--hash=sha256:a60f90bba4c37962cbf210f0188ecca87daafdf60271f4c6948606e4dabf8785 \
--hash=sha256:6ca2264f341dd81e41f3fffecec6e446aa2121e0b8d026fb5130e02de1402785 \
--hash=sha256:27e590352c76156f50f538dbcebd1925317a0f70540f7dc8c97d2931c595783a \
--hash=sha256:eea5d6443b093e1545ad0210e6cf27f920482bfcf5c77cdc8596aec73523bb7e \
--hash=sha256:f05251bbc2145349b8d0b77c0d4e5f3b228418807b1ee27cefb11f69ed3d233b \
--hash=sha256:487c8e61d7acc50b8be82bda8c8d21d20e133c3cbf41bd8ad7eb1aaeb3f07c97 \
--hash=sha256:8d1a92d8e90b286d491e5626af53afef2ba04da33e82e30744795c71880eaa21 \
--hash=sha256:b570da8cd0012f4af9fa76a5635cd31f707473e65a5a335b186069d5c7121ff2 \
--hash=sha256:5ef87fca280fb15342726bd5f980f6faf8b84a5287fcc2d4962ea8af88b35130 \
--hash=sha256:93e414e3206779ef41e5ff2448067213febf260ba747fc65389a3ddaa3fb8715 \
--hash=sha256:6653071f4f9bac46fbc30f3c7838b0e9063ee335908c5d61fb7a4a86c8fd2036 \
--hash=sha256:32a73c53783becdb7eaf75a2a1525ea8e49379fb7248c3eeefb9412123536387 \
--hash=sha256:1a7c59c6ffd6ef5db362b798f350e24ab2cfa5700d53ac6681918f314a4d3b94 \
--hash=sha256:1436cf0063bba7888e43f1ba8d58824f085410ea2025befe81150aceb123e345 \
--hash=sha256:4beea0f31491bc086991b97517b9683e5cfb369205dac0148ef685ac12a20a67 \
--hash=sha256:41fb58868b816c202e8881fd0f179a4644ce6e7cbbb248ef0283a34b73ec73bb \
--hash=sha256:bd34f6d1810d9354dc7e35158aa6cc33456be7706df4420819af6ed966e85448 \
--hash=sha256:edffbe3c510d8f4bf8640e02ca019e48a9b72357318383ca60e3330c23aaffc7 \
--hash=sha256:6d949f53ad4fc7cf02c44d6678e7ff05ec5f5552b235b9e136bd52e9bf730b91 \
--hash=sha256:079b68f197c796e42aa80b1f739f058dcee796dc725cc9a1be0cdb08fc45b000 \
--hash=sha256:9c3a88d20e4fe4a2a4a84bf439a5ac9c9aba400b85244c63a1ab7088f85d9d25 \
--hash=sha256:4e285b5f2bf321fc0857b491b5028c5f276ec0c873b985d58d7748ece1d770dd \
--hash=sha256:ef72013e20dd5ba86a8ae1aed7f56f31d3374189aa8b433e7b12ad182c0d2dfb \
--hash=sha256:10d2017f9150248563bb579cd0d07c61c58da85c922b780060dcc9a3aa9f432d \
--hash=sha256:0538747a9d7827ce3e16a8fdd201a99e661c7dee3c96c885d8ecba3c35d1032c \
--hash=sha256:0645e934e940107e2fdbe7c5b6fb8ec6232444260752598bc4d09511bd056c0b \
--hash=sha256:6daa662aba22ef3258934105be2dd9afa5bb45748f4f702a3b39a5bf53a1f4dc \
--hash=sha256:603a464c2e67d8a546ddaa206d98e3246e5db05594b97db844c2f0a1af37cf5b \
--hash=sha256:c4b2e0559b68455c085fb0f6178e9752c4be3bba104d6e881eb5573b399d1eb2 \
--hash=sha256:0f3f0059891d3254c7b5fb935330d6db38d6519ecd238ca4fce93c234b4a0f73 \
--hash=sha256:c852b1530083a620cb0de5f3cd6826f19862bafeaf77586f1aef326e49d95f0c \
--hash=sha256:287605bede6bd36e930577c5925fcea17cb30453d96a7b4c63c14a257118dbb9 \
--hash=sha256:fe749b052bb7233fe5d072fcb549221a8cb1a16725c47c37e42b0b9cb3ff2c3f
mwclient==0.10.1 \
--hash=sha256:7aad000c4c7f239c7c92c43e2d5fbdaf8573262accd6070524ff42e46950306e \
--hash=sha256:79363dd8d12f5e3b91b92b63152bf9dfef27da786c076a244e1f148c8dd67139
oauthlib==3.2.2 ; python_version >= "3.6" \
idna==3.4 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4 \
--hash=sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2
internetarchive==3.5.0 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:2a9625e1dcbe431e5b898402273a4345bf5cb46012599cbe92e664ffdc36e881
jsonpatch==1.33 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:0ae28c0cd062bbd8b8ecc26d7d164fbbea9652a1a3693f3b956c1eae5145dade \
--hash=sha256:9fcd4009c41e6d12348b4a0ff2563ba56a2923a7dfee731d004e212e1ee5030c
jsonpointer==2.4 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:15d51bba20eea3165644553647711d150376234112651b4f1811022aecad7d7a \
--hash=sha256:585cee82b70211fa9e6043b7bb89db6e1aa49524340dde8ad6b63206ea689d88
lxml==4.9.3 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:05186a0f1346ae12553d66df1cfce6f251589fea3ad3da4f3ef4e34b2d58c6a3 \
--hash=sha256:075b731ddd9e7f68ad24c635374211376aa05a281673ede86cbe1d1b3455279d \
--hash=sha256:081d32421db5df44c41b7f08a334a090a545c54ba977e47fd7cc2deece78809a \
--hash=sha256:0a3d3487f07c1d7f150894c238299934a2a074ef590b583103a45002035be120 \
--hash=sha256:0bfd0767c5c1de2551a120673b72e5d4b628737cb05414f03c3277bf9bed3305 \
--hash=sha256:0c0850c8b02c298d3c7006b23e98249515ac57430e16a166873fc47a5d549287 \
--hash=sha256:0e2cb47860da1f7e9a5256254b74ae331687b9672dfa780eed355c4c9c3dbd23 \
--hash=sha256:120fa9349a24c7043854c53cae8cec227e1f79195a7493e09e0c12e29f918e52 \
--hash=sha256:1247694b26342a7bf47c02e513d32225ededd18045264d40758abeb3c838a51f \
--hash=sha256:141f1d1a9b663c679dc524af3ea1773e618907e96075262726c7612c02b149a4 \
--hash=sha256:14e019fd83b831b2e61baed40cab76222139926b1fb5ed0e79225bc0cae14584 \
--hash=sha256:1509dd12b773c02acd154582088820893109f6ca27ef7291b003d0e81666109f \
--hash=sha256:17a753023436a18e27dd7769e798ce302963c236bc4114ceee5b25c18c52c693 \
--hash=sha256:1e224d5755dba2f4a9498e150c43792392ac9b5380aa1b845f98a1618c94eeef \
--hash=sha256:1f447ea5429b54f9582d4b955f5f1985f278ce5cf169f72eea8afd9502973dd5 \
--hash=sha256:23eed6d7b1a3336ad92d8e39d4bfe09073c31bfe502f20ca5116b2a334f8ec02 \
--hash=sha256:25f32acefac14ef7bd53e4218fe93b804ef6f6b92ffdb4322bb6d49d94cad2bc \
--hash=sha256:2c74524e179f2ad6d2a4f7caf70e2d96639c0954c943ad601a9e146c76408ed7 \
--hash=sha256:303bf1edce6ced16bf67a18a1cf8339d0db79577eec5d9a6d4a80f0fb10aa2da \
--hash=sha256:3331bece23c9ee066e0fb3f96c61322b9e0f54d775fccefff4c38ca488de283a \
--hash=sha256:3e9bdd30efde2b9ccfa9cb5768ba04fe71b018a25ea093379c857c9dad262c40 \
--hash=sha256:411007c0d88188d9f621b11d252cce90c4a2d1a49db6c068e3c16422f306eab8 \
--hash=sha256:42871176e7896d5d45138f6d28751053c711ed4d48d8e30b498da155af39aebd \
--hash=sha256:46f409a2d60f634fe550f7133ed30ad5321ae2e6630f13657fb9479506b00601 \
--hash=sha256:48628bd53a426c9eb9bc066a923acaa0878d1e86129fd5359aee99285f4eed9c \
--hash=sha256:48d6ed886b343d11493129e019da91d4039826794a3e3027321c56d9e71505be \
--hash=sha256:4930be26af26ac545c3dffb662521d4e6268352866956672231887d18f0eaab2 \
--hash=sha256:4aec80cde9197340bc353d2768e2a75f5f60bacda2bab72ab1dc499589b3878c \
--hash=sha256:4c28a9144688aef80d6ea666c809b4b0e50010a2aca784c97f5e6bf143d9f129 \
--hash=sha256:4d2d1edbca80b510443f51afd8496be95529db04a509bc8faee49c7b0fb6d2cc \
--hash=sha256:4dd9a263e845a72eacb60d12401e37c616438ea2e5442885f65082c276dfb2b2 \
--hash=sha256:4f1026bc732b6a7f96369f7bfe1a4f2290fb34dce00d8644bc3036fb351a4ca1 \
--hash=sha256:4fb960a632a49f2f089d522f70496640fdf1218f1243889da3822e0a9f5f3ba7 \
--hash=sha256:50670615eaf97227d5dc60de2dc99fb134a7130d310d783314e7724bf163f75d \
--hash=sha256:50baa9c1c47efcaef189f31e3d00d697c6d4afda5c3cde0302d063492ff9b477 \
--hash=sha256:53ace1c1fd5a74ef662f844a0413446c0629d151055340e9893da958a374f70d \
--hash=sha256:5515edd2a6d1a5a70bfcdee23b42ec33425e405c5b351478ab7dc9347228f96e \
--hash=sha256:56dc1f1ebccc656d1b3ed288f11e27172a01503fc016bcabdcbc0978b19352b7 \
--hash=sha256:578695735c5a3f51569810dfebd05dd6f888147a34f0f98d4bb27e92b76e05c2 \
--hash=sha256:57aba1bbdf450b726d58b2aea5fe47c7875f5afb2c4a23784ed78f19a0462574 \
--hash=sha256:57d6ba0ca2b0c462f339640d22882acc711de224d769edf29962b09f77129cbf \
--hash=sha256:5c245b783db29c4e4fbbbfc9c5a78be496c9fea25517f90606aa1f6b2b3d5f7b \
--hash=sha256:5c31c7462abdf8f2ac0577d9f05279727e698f97ecbb02f17939ea99ae8daa98 \
--hash=sha256:64f479d719dc9f4c813ad9bb6b28f8390360660b73b2e4beb4cb0ae7104f1c12 \
--hash=sha256:65299ea57d82fb91c7f019300d24050c4ddeb7c5a190e076b5f48a2b43d19c42 \
--hash=sha256:6689a3d7fd13dc687e9102a27e98ef33730ac4fe37795d5036d18b4d527abd35 \
--hash=sha256:690dafd0b187ed38583a648076865d8c229661ed20e48f2335d68e2cf7dc829d \
--hash=sha256:6fc3c450eaa0b56f815c7b62f2b7fba7266c4779adcf1cece9e6deb1de7305ce \
--hash=sha256:704f61ba8c1283c71b16135caf697557f5ecf3e74d9e453233e4771d68a1f42d \
--hash=sha256:71c52db65e4b56b8ddc5bb89fb2e66c558ed9d1a74a45ceb7dcb20c191c3df2f \
--hash=sha256:71d66ee82e7417828af6ecd7db817913cb0cf9d4e61aa0ac1fde0583d84358db \
--hash=sha256:7d298a1bd60c067ea75d9f684f5f3992c9d6766fadbc0bcedd39750bf344c2f4 \
--hash=sha256:8b77946fd508cbf0fccd8e400a7f71d4ac0e1595812e66025bac475a8e811694 \
--hash=sha256:8d7e43bd40f65f7d97ad8ef5c9b1778943d02f04febef12def25f7583d19baac \
--hash=sha256:8df133a2ea5e74eef5e8fc6f19b9e085f758768a16e9877a60aec455ed2609b2 \
--hash=sha256:8ed74706b26ad100433da4b9d807eae371efaa266ffc3e9191ea436087a9d6a7 \
--hash=sha256:92af161ecbdb2883c4593d5ed4815ea71b31fafd7fd05789b23100d081ecac96 \
--hash=sha256:97047f0d25cd4bcae81f9ec9dc290ca3e15927c192df17331b53bebe0e3ff96d \
--hash=sha256:9719fe17307a9e814580af1f5c6e05ca593b12fb7e44fe62450a5384dbf61b4b \
--hash=sha256:9767e79108424fb6c3edf8f81e6730666a50feb01a328f4a016464a5893f835a \
--hash=sha256:9a92d3faef50658dd2c5470af249985782bf754c4e18e15afb67d3ab06233f13 \
--hash=sha256:9bb6ad405121241e99a86efff22d3ef469024ce22875a7ae045896ad23ba2340 \
--hash=sha256:9e28c51fa0ce5674be9f560c6761c1b441631901993f76700b1b30ca6c8378d6 \
--hash=sha256:aca086dc5f9ef98c512bac8efea4483eb84abbf926eaeedf7b91479feb092458 \
--hash=sha256:ae8b9c6deb1e634ba4f1930eb67ef6e6bf6a44b6eb5ad605642b2d6d5ed9ce3c \
--hash=sha256:b0a545b46b526d418eb91754565ba5b63b1c0b12f9bd2f808c852d9b4b2f9b5c \
--hash=sha256:b4e4bc18382088514ebde9328da057775055940a1f2e18f6ad2d78aa0f3ec5b9 \
--hash=sha256:b6420a005548ad52154c8ceab4a1290ff78d757f9e5cbc68f8c77089acd3c432 \
--hash=sha256:b86164d2cff4d3aaa1f04a14685cbc072efd0b4f99ca5708b2ad1b9b5988a991 \
--hash=sha256:bb3bb49c7a6ad9d981d734ef7c7193bc349ac338776a0360cc671eaee89bcf69 \
--hash=sha256:bef4e656f7d98aaa3486d2627e7d2df1157d7e88e7efd43a65aa5dd4714916cf \
--hash=sha256:c0781a98ff5e6586926293e59480b64ddd46282953203c76ae15dbbbf302e8bb \
--hash=sha256:c2006f5c8d28dee289f7020f721354362fa304acbaaf9745751ac4006650254b \
--hash=sha256:c41bfca0bd3532d53d16fd34d20806d5c2b1ace22a2f2e4c0008570bf2c58833 \
--hash=sha256:cd47b4a0d41d2afa3e58e5bf1f62069255aa2fd6ff5ee41604418ca925911d76 \
--hash=sha256:cdb650fc86227eba20de1a29d4b2c1bfe139dc75a0669270033cb2ea3d391b85 \
--hash=sha256:cef2502e7e8a96fe5ad686d60b49e1ab03e438bd9123987994528febd569868e \
--hash=sha256:d27be7405547d1f958b60837dc4c1007da90b8b23f54ba1f8b728c78fdb19d50 \
--hash=sha256:d37017287a7adb6ab77e1c5bee9bcf9660f90ff445042b790402a654d2ad81d8 \
--hash=sha256:d3ff32724f98fbbbfa9f49d82852b159e9784d6094983d9a8b7f2ddaebb063d4 \
--hash=sha256:d73d8ecf8ecf10a3bd007f2192725a34bd62898e8da27eb9d32a58084f93962b \
--hash=sha256:dd708cf4ee4408cf46a48b108fb9427bfa00b9b85812a9262b5c668af2533ea5 \
--hash=sha256:e3cd95e10c2610c360154afdc2f1480aea394f4a4f1ea0a5eacce49640c9b190 \
--hash=sha256:e4da8ca0c0c0aea88fd46be8e44bd49716772358d648cce45fe387f7b92374a7 \
--hash=sha256:eadfbbbfb41b44034a4c757fd5d70baccd43296fb894dba0295606a7cf3124aa \
--hash=sha256:ed667f49b11360951e201453fc3967344d0d0263aa415e1619e85ae7fd17b4e0 \
--hash=sha256:f3df3db1d336b9356dd3112eae5f5c2b8b377f3bc826848567f10bfddfee77e9 \
--hash=sha256:f6bdac493b949141b733c5345b6ba8f87a226029cbabc7e9e121a413e49441e0 \
--hash=sha256:fbf521479bcac1e25a663df882c46a641a9bff6b56dc8b0fafaebd2f66fb231b \
--hash=sha256:fc9b106a1bf918db68619fdcd6d5ad4f972fdd19c01d19bdb6bf63f3589a9ec5 \
--hash=sha256:fcdd00edfd0a3001e0181eab3e63bd5c74ad3e67152c84f93f13769a40e073a7 \
--hash=sha256:fe4bda6bd4340caa6e5cf95e73f8fea5c4bfc55763dd42f1b50a94c1b4a2fbd4
mwclient==0.10.1 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:79363dd8d12f5e3b91b92b63152bf9dfef27da786c076a244e1f148c8dd67139 \
--hash=sha256:7aad000c4c7f239c7c92c43e2d5fbdaf8573262accd6070524ff42e46950306e
oauthlib==3.2.2 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:8139f29aac13e25d502680e9e19963e83f16838d48a0d71c287fe40e7067fbca \
--hash=sha256:9859c40929662bec5d64f34d01c99e093149682a3f38915dc0655d5a633dd918
poster3==0.8.1 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:1b27d7d63e3191e5d7238631fc828e4493590e94dcea034e386c079d853cce14
pre-commit-poetry-export==0.1.2 \
--hash=sha256:800d57df5ff96c6dcff1caf6f8f88bb08bd04f5017a125e94ce9e05ea27e60b8 \
--hash=sha256:4bdc5217f492083911fe6afe47a3532b2ab34108ed6cd7e159b603851f7b0357
pymysql==1.0.2 \
--hash=sha256:41fc3a0c5013d5f039639442321185532e3e2c8924687abe6537de157d403641 \
--hash=sha256:816927a350f38d56072aeca5dfb10221fe1dc653745853d30a216637f5d7ad36
pywikibot==6.6.5 \
pre-commit-poetry-export==0.1.2 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:4bdc5217f492083911fe6afe47a3532b2ab34108ed6cd7e159b603851f7b0357 \
--hash=sha256:800d57df5ff96c6dcff1caf6f8f88bb08bd04f5017a125e94ce9e05ea27e60b8
pymysql==1.1.0 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:4f13a7df8bf36a51e81dd9f3605fede45a4878fe02f9236349fd82a3f0612f96 \
--hash=sha256:8969ec6d763c856f7073c4c64662882675702efcb114b4bcbb955aea3a069fa7
pywikibot==6.6.5 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:72c2fc4615de0b7b0a158ac5407317ed3dd9d1250d371281cc18672e9d0f384d
requests-oauthlib==1.3.1 ; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" \
--hash=sha256:75beac4a47881eeb94d5ea5d6ad31ef88856affe2332b9aafb52c6452ccf0d7a \
--hash=sha256:2577c501a2fb8d05a304c09d090d6e47c306fef15809d102b327cf8364bddab5
requests==2.31.0 \
requests-oauthlib==1.3.1 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:2577c501a2fb8d05a304c09d090d6e47c306fef15809d102b327cf8364bddab5 \
--hash=sha256:75beac4a47881eeb94d5ea5d6ad31ef88856affe2332b9aafb52c6452ccf0d7a
requests==2.31.0 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f \
--hash=sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1
schema==0.7.5 \
--hash=sha256:f3ffdeeada09ec34bf40d7d79996d9f7175db93b7a5065de0faa7f41083c1e6c \
--hash=sha256:f06717112c61895cabc4707752b88716e8420a8819d71404501e114f91043197
six==1.16.0 ; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" \
--hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254 \
--hash=sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926
tqdm==4.63.1 ; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" \
--hash=sha256:6461b009d6792008d0000e1b0c7ca50195ec78c0e808a3a6b668a56a3236c3a5 \
--hash=sha256:4230a49119a416c88cc47d0d2d32d5d90f1a282d5e497d49801950704e49863d
urllib3==1.26.9 \
--hash=sha256:44ece4d53fb1706f667c9bd1c648f5469a2ec925fcf3a776667042d645472c14 \
--hash=sha256:aabaf16477806a5e1dd19aa41f8c2b7950dd3c746362d7e3223dbe6de6ac448e
wikitools3==3.0.1 \
schema==0.7.5 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:f06717112c61895cabc4707752b88716e8420a8819d71404501e114f91043197 \
--hash=sha256:f3ffdeeada09ec34bf40d7d79996d9f7175db93b7a5065de0faa7f41083c1e6c
setuptools==68.1.2 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:3d4dfa6d95f1b101d695a6160a7626e15583af71a5f52176efa5d39a054d475d \
--hash=sha256:3d8083eed2d13afc9426f227b24fd1659489ec107c0e86cec2ffdde5c92e790b
six==1.16.0 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926 \
--hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254
tqdm==4.66.1 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:d302b3c5b53d47bce91fea46679d9c3c6508cf6332229aa1e7d8653723793386 \
--hash=sha256:d88e651f9db8d8551a62556d3cff9e3034274ca5d66e93197cf2490e2dcb69c7
urllib3==1.26.16 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:8d36afa7616d8ab714608411b4a3b13e58f463aee519024578e062e141dce20f \
--hash=sha256:8f135f6502756bde6b2a9b28989df5fbe87c9970cecaa69041edcce7f0589b14
wikitools3==3.0.1 ; python_version >= "3.8" and python_version < "4.0" \
--hash=sha256:ab1a13d9be831eb1186cf0f2b8ac487dde0932dad82c6c6031f28fa415840fe4 \
--hash=sha256:b493e6806fb1985ad1af294c56364aeba6ea67a366553bc11b65b7f5279d0012

1
resources/listsofwikis/mediawiki/checkalive.py
Unescape Escape View File

@ -38,7 +38,6 @@ def checkcore(api):
try:
raw = urllib.request.urlopenurlopen(req, None, delay).read()
except URLError as reason: # https://docs.python.org/3/library/urllib.error.html
if reason.isinstance(HTTPError):
print(api + "is dead or has errors because:")
print(

33
resources/listsofwikis/mediawiki/fandom-spider.py
Unescape Escape View File

@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Copyright (C) 2022 Simon Liu
# This program is free software: you can redistribute it and/or modify
@ -17,34 +16,46 @@
import re
import time
import requests
from urllib import parse
import requests
from tqdm import tqdm
def main():
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0',
"User-Agent": "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0",
}
# grab lvl3 links
req = requests.get('https://community.fandom.com/wiki/Sitemap?level=2', headers=headers)
map_lvl3 = re.findall(r'<a class=\"title\" href=\"([^>]+?)\">', req.text)
req = requests.get(
"https://community.fandom.com/wiki/Sitemap?level=2", headers=headers
)
map_lvl3 = re.findall(r"<a class=\"title\" href=\"([^>]+?)\">", req.text)
# grab wiki links
wikis = []
for lvl3 in tqdm(map_lvl3):
time.sleep(0.3)
req = requests.get('https://community.fandom.com%s' % lvl3)
req = requests.get("https://community.fandom.com%s" % lvl3)
if req.status_code != 200:
time.sleep(5)
req = requests.get('https://community.fandom.com%s' % lvl3)
wikis.extend([wiki.replace('http://', 'https://') for wiki in re.findall(r'<a class=\"title\" href=\"([^>]+?)\">', req.text)])
req = requests.get("https://community.fandom.com%s" % lvl3)
wikis.extend(
[
wiki.replace("http://", "https://")
for wiki in re.findall(
r"<a class=\"title\" href=\"([^>]+?)\">", req.text
)
]
)
wikis = list(set(wikis))
wikis.sort()
with open('fandom.com', 'w') as f:
with open("fandom.com", "w") as f:
for wiki in wikis:
f.write(parse.urljoin(wiki, 'api.php') + '\n')
f.write(parse.urljoin(wiki, "api.php") + "\n")
if __name__ == '__main__':
if __name__ == "__main__":
main()

41
resources/listsofwikis/mediawiki/miraheze-spider.py
Unescape Escape View File

@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Copyright (C) 2022 Simon Liu
# This program is free software: you can redistribute it and/or modify
@ -17,37 +16,53 @@
import re
import time
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
def nextpage(soup):
try:
soup.find('span', text='Next page').parent['href']
soup.find("span", text="Next page").parent["href"]
return True
except:
return False
def main():
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0',
"User-Agent": "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0",
}
req = requests.get('https://meta.miraheze.org/wiki/Special:WikiDiscover')
soup = BeautifulSoup(req.content, features='lxml')
wikis = re.findall(r'<td class=\"TablePager_col_wiki_dbname\"><a href=\"([^>]+?)\">', req.text)
req = requests.get("https://meta.miraheze.org/wiki/Special:WikiDiscover")
soup = BeautifulSoup(req.content, features="lxml")
wikis = re.findall(
r"<td class=\"TablePager_col_wiki_dbname\"><a href=\"([^>]+?)\">", req.text
)
while nextpage(soup):
time.sleep(0.3)
req = requests.get(urljoin('https://meta.miraheze.org', soup.find('span', text='Next page').parent['href']))
soup = BeautifulSoup(req.content, features='lxml')
wikis.extend(re.findall(r'<td class=\"TablePager_col_wiki_dbname\"><a href=\"([^>]+?)\">', req.text))
req = requests.get(
urljoin(
"https://meta.miraheze.org",
soup.find("span", text="Next page").parent["href"],
)
)
soup = BeautifulSoup(req.content, features="lxml")
wikis.extend(
re.findall(
r"<td class=\"TablePager_col_wiki_dbname\"><a href=\"([^>]+?)\">",
req.text,
)
)
wikis = list(set(wikis))
wikis.sort()
with open('miraheze.org', 'w') as f:
with open("miraheze.org", "w") as f:
for wiki in wikis:
f.write(urljoin(wiki, 'w/api.php') + '\n')
f.write(urljoin(wiki, "w/api.php") + "\n")
if __name__ == '__main__':
if __name__ == "__main__":
main()

19
resources/listsofwikis/mediawiki/neoseeker-spider.py
Unescape Escape View File

@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Copyright (C) 2014-2022 WikiTeam developers
# This program is free software: you can redistribute it and/or modify
@ -16,21 +15,25 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
import re
import requests
def main():
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0',
"User-Agent": "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0",
}
url = 'https://neowiki.neoseeker.com/wiki/Special:WikiList'
url = "https://neowiki.neoseeker.com/wiki/Special:WikiList"
r = requests.get(url, headers=headers)
raw = r.text
m = re.findall(r'<li><a href=\'([^>]+?)/wiki/\'>', raw)
m = [w.replace('http://', 'https://') + '/w/api.php' for w in m]
m = re.findall(r"<li><a href=\'([^>]+?)/wiki/\'>", raw)
m = [w.replace("http://", "https://") + "/w/api.php" for w in m]
m = list(set(m))
m.sort()
with open('neoseeker.com', 'w') as f:
f.write('\n'.join(m))
with open("neoseeker.com", "w") as f:
f.write("\n".join(m))
if __name__ == '__main__':
if __name__ == "__main__":
main()

2
resources/listsofwikis/mediawiki/neoseeker.com vendored
Unescape Escape View File

@ -180,4 +180,4 @@ https://wikiguides.neoseeker.com/w/api.php
https://wow.neoseeker.com/w/api.php
https://xenoblade.neoseeker.com/w/api.php
https://yugioh.neoseeker.com/w/api.php
https://zelda.neoseeker.com/w/api.php
https://zelda.neoseeker.com/w/api.php

65
resources/listsofwikis/mediawiki/shoutwiki-spider.py
Unescape Escape View File

@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Copyright (C) 2022 Simon Liu
# This program is free software: you can redistribute it and/or modify
@ -16,59 +15,67 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
import time
import requests
from tqdm import tqdm
def main():
ids, wikis = [], []
gcont = 'tmp'
url = 'http://www.shoutwiki.com/w/api.php'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0'}
gcont = "tmp"
url = "http://www.shoutwiki.com/w/api.php"
headers = {
"User-Agent": "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0"
}
# grab wiki pages
params = {
'action': 'query',
'format': 'json',
'prop': 'info',
'generator': 'categorymembers',
'inprop': 'url',
'gcmtitle': 'Category:Flat_list_of_all_wikis',
'gcmlimit': 'max'
"action": "query",
"format": "json",
"prop": "info",
"generator": "categorymembers",
"inprop": "url",
"gcmtitle": "Category:Flat_list_of_all_wikis",
"gcmlimit": "max",
}
while gcont:
if gcont != 'tmp':
params['gcmcontinue'] = gcont
if gcont != "tmp":
params["gcmcontinue"] = gcont
json = requests.get(url, params=params, headers=headers).json()
gcont = json['continue']['gcmcontinue'] if 'continue' in json else ''
query = json['query']['pages']
gcont = json["continue"]["gcmcontinue"] if "continue" in json else ""
query = json["query"]["pages"]
for wiki in query:
ids.append(wiki)
# grab wiki API
params = {
'action': 'query',
'format': 'json',
'prop': 'revisions',
'formatversion': '2',
'rvprop': 'content',
'rvslots': '*'
"action": "query",
"format": "json",
"prop": "revisions",
"formatversion": "2",
"rvprop": "content",
"rvslots": "*",
}
for n in tqdm(range(0, len(ids), 50)):
params['pageids'] = '|'.join(ids[n:n+50])
params["pageids"] = "|".join(ids[n : n + 50])
json = requests.get(url, params=params, headers=headers).json()
for wiki in json['query']['pages']:
for val in wiki['revisions'][0]['slots']['main']['content'].split('\n|'):
if 'subdomain' in val:
wikis.append('http://%s.shoutwiki.com/w/api.php' % val.split('subdomain =')[-1].strip())
for wiki in json["query"]["pages"]:
for val in wiki["revisions"][0]["slots"]["main"]["content"].split("\n|"):
if "subdomain" in val:
wikis.append(
"http://%s.shoutwiki.com/w/api.php"
% val.split("subdomain =")[-1].strip()
)
break
time.sleep(0.3)
wikis = list(set(wikis))
wikis.sort()
with open('shoutwiki.com', 'w') as f:
f.write('\n'.join(wikis))
with open("shoutwiki.com", "w") as f:
f.write("\n".join(wikis))
if __name__ == '__main__':
if __name__ == "__main__":
main()

2
resources/listsofwikis/mediawiki/shoutwiki.com vendored
Unescape Escape View File

@ -2170,4 +2170,4 @@ http://zhishi.shoutwiki.com/w/api.php
http://zingarese.shoutwiki.com/w/api.php
http://zuowen.shoutwiki.com/w/api.php
http://zusnaties.shoutwiki.com/w/api.php
http://zuzendaritza.shoutwiki.com/w/api.php
http://zuzendaritza.shoutwiki.com/w/api.php

18
resources/listsofwikis/mediawiki/wiki-site-spider.py
Unescape Escape View File

@ -1,5 +1,4 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# Copyright (C) 2014-2022 WikiTeam developers
# This program is free software: you can redistribute it and/or modify
@ -16,17 +15,19 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
import re
import requests
from urllib import parse
import requests
def main():
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0',
"User-Agent": "Mozilla/5.0 (X11; Linux i686; rv:24.0) Gecko/20100101 Firefox/24.0",
}
urls = [
'http://www.wiki.co.il/active-wiki-all.html',
'http://www.wiki.co.il/active-wiki-en.html',
"http://www.wiki.co.il/active-wiki-all.html",
"http://www.wiki.co.il/active-wiki-en.html",
]
wikis = []
for url in urls:
@ -35,9 +36,10 @@ def main():
wikis = list(set(wikis))
wikis.sort()
with open('wiki-site.com', 'w') as f:
with open("wiki-site.com", "w") as f:
for wiki in wikis:
f.write(parse.urljoin(wiki, 'api.php') + '\n')
f.write(parse.urljoin(wiki, "api.php") + "\n")
if __name__ == '__main__':
if __name__ == "__main__":
main()

166
tools/xml2titles.py
Unescape Escape View File

@ -1,23 +1,9 @@
'''
"""
Extracts all titles from a XML dump file and writes them to `*-xml2titles.txt`.
requirements:
file_read_backwards
'''
import dataclasses
import os
import argparse
import tqdm
import sys
# import re
import xml.sax
from xml.sax.saxutils import unescape
from file_read_backwards import FileReadBackwards
'''
<page>
<title>abcde</title>
<ns>0</ns>
@ -35,11 +21,26 @@ from file_read_backwards import FileReadBackwards
<text xml:space="preserve" bytes="2326">text</text>
</revision>
</page>
'''
"""
import argparse
import dataclasses
import os
import sys
# import re
import xml.sax
from xml.sax.saxutils import unescape
import tqdm
from file_read_backwards import FileReadBackwards
class XMLBaseHandler(xml.sax.handler.ContentHandler):
'''only work on level <= 3 of the XML tree'''
"""only work on level <= 3 of the XML tree"""
fileSize = 0
class page__:
# TODO
pass
@ -47,7 +48,11 @@ class XMLBaseHandler(xml.sax.handler.ContentHandler):
def __init__(self, fileSize=0):
self.fileSize = fileSize
self.tqdm_progress = tqdm.tqdm(
total=self.fileSize, unit="B", unit_scale=True, unit_divisor=1024, desc="Parsing XML"
total=self.fileSize,
unit="B",
unit_scale=True,
unit_divisor=1024,
desc="Parsing XML",
)
self.globalParsedBytes = 0
self.debugCount = 0
@ -82,7 +87,7 @@ class XMLBaseHandler(xml.sax.handler.ContentHandler):
def close_tqdm(self):
self.tqdm_progress.close()
def __debugCount(self):
self.debugCount += 1
print(self.debugCount)
@ -93,11 +98,11 @@ class XMLBaseHandler(xml.sax.handler.ContentHandler):
# print("resetPageTag")
def startElement(self, name, attrs):
self.depth+=1
self.depth += 1
if self.depth > 3:
self.startElementOverDepth3(name, attrs)
return
if name == "page":
self.inPage = True
self.pageTagsCount += 1
@ -118,9 +123,9 @@ class XMLBaseHandler(xml.sax.handler.ContentHandler):
def endElement(self, name):
if self.depth > 3:
self.endElementOverDepth3(name)
self.depth-=1
self.depth -= 1
return
self.depth-=1
self.depth -= 1
if name == "page":
self.inPage = False
@ -147,20 +152,21 @@ class XMLBaseHandler(xml.sax.handler.ContentHandler):
bufferSize = len(content.encode("utf-8"))
self.globalParsedBytes += bufferSize
# print(bufferSize)
self.tqdm_progress.update(bufferSize) # NOTE: sum(bufferSize...) != fileSize
self.tqdm_progress.update(bufferSize) # NOTE: sum(bufferSize...) != fileSize
if self.inPage:
pass
if self.inTitle:
# self.__debugCount()
self.cjoin("title", content) if 'title' not in not_parse_tags else None
self.cjoin("title", content) if "title" not in not_parse_tags else None
if self.inNs:
self.cjoin("ns", content) if 'ns' not in not_parse_tags else None
self.cjoin("ns", content) if "ns" not in not_parse_tags else None
if self.inId:
self.cjoin("id", content) if 'id' not in not_parse_tags else None
self.cjoin("id", content) if "id" not in not_parse_tags else None
if self.inRevision:
self.cjoin("revision", content) if 'revision' not in not_parse_tags else None
self.cjoin(
"revision", content
) if "revision" not in not_parse_tags else None
def endDocument(self):
if self.depth != 0:
@ -171,18 +177,18 @@ class XMLBaseHandler(xml.sax.handler.ContentHandler):
def endElementOverDepth3(self, name):
pass
def cjoin(self, obj, content):
''' self.obj = self.obj + content if self.obj is not None else content
"""self.obj = self.obj + content if self.obj is not None else content
obj: str
'''
"""
if hasattr(self, obj):
if getattr(self, obj) is None:
setattr(self, obj, content)
else:
# assert ''.join((getattr(self, obj), content)) == content if getattr(self, obj) is None else getattr(self, obj) + content
setattr(self, obj, ''.join((getattr(self, obj), content)))
setattr(self, obj, "".join((getattr(self, obj), content)))
pass
else:
raise AttributeError("XMLBaseHandler has no attribute %s" % obj)
@ -194,21 +200,26 @@ class TitlesHandler(XMLBaseHandler):
super().__init__(*args, **kwargs)
self.set_titles = set()
self.list_titles = []
def endElement(self, name):
# print(self.revision) if name == "page" else None
super().endElement(name)
if name == "page":
if self.page['title'] is not None:
if self.page['title'] in self.set_titles:
print("Duplicate title found: %s" % self.page['title']) if not self.silent else None
if self.page["title"] is not None:
if self.page["title"] in self.set_titles:
print(
"Duplicate title found: %s" % self.page["title"]
) if not self.silent else None
else:
self.set_titles.add(self.page['title'])
self.list_titles.append(self.page['title']) # unique
self.set_titles.add(self.page["title"])
self.list_titles.append(self.page["title"]) # unique
if not self.silent:
print(self.page)
def characters(self, content):
return super().characters(content, not_parse_tags=["revision"])
class PagesHandler(XMLBaseHandler):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
@ -223,16 +234,16 @@ class PagesHandler(XMLBaseHandler):
# TODO
def startElementOverDepth3(self, name, attrs):
super().startElementOverDepth3(name, attrs)
if name == 'text' and attrs:
if name == "text" and attrs:
self.pageTextsAttrs.append(attrs.items())
self.page['textsAttrs'] = self.pageTextsAttrs
if name == 'text':
self.page["textsAttrs"] = self.pageTextsAttrs
if name == "text":
self.inText = True
self.textTagsCount += 1
def endElementOverDepth3(self, name):
super().endElementOverDepth3(name)
if name == 'text':
if name == "text":
self.inText = False
def resetPageTag(self):
@ -242,12 +253,14 @@ class PagesHandler(XMLBaseHandler):
self.pageTexts: str = None
def endElement(self, name):
self.pageTextsRealLength = len(self.pageTexts.encode('utf-8')) if self.pageTexts is not None else 0
self.page['textsRealLength'] = self.pageTextsRealLength
self.pageTextsRealLength = (
len(self.pageTexts.encode("utf-8")) if self.pageTexts is not None else 0
)
self.page["textsRealLength"] = self.pageTextsRealLength
super().endElement(name)
# if name == "page":
# print(self.page)
def characters(self, content, *args, **kwargs):
super().characters(content, *args, **kwargs)
if self.inText:
@ -260,33 +273,36 @@ class MediaNsHandler(XMLBaseHandler):
# self.mediaNsPages = []
self.mediaNsPagesName_set = set()
self.mediaNsPagesID_set = set()
def endElement(self, name):
super().endElement(name)
if name == "page":
if self.page['ns'] == '6':
if self.page['title'] in self.mediaNsPagesName_set:
if self.page["ns"] == "6":
if self.page["title"] in self.mediaNsPagesName_set:
if not self.silent:
print("Duplicate title found: %s" % self.page['title'])
print("Duplicate title found: %s" % self.page["title"])
else:
self.mediaNsPagesName_set.add(self.page['title'])
self.mediaNsPagesName_set.add(self.page["title"])
# self.mediaNsPages.append(self.page)
# print(self.page)
if self.page['id'] in self.mediaNsPagesID_set:
if self.page["id"] in self.mediaNsPagesID_set:
if not self.silent:
print("Duplicate id found: %s" % self.page['id'])
print("Duplicate id found: %s" % self.page["id"])
else:
self.mediaNsPagesID_set.add(self.page['id'])
self.mediaNsPagesID_set.add(self.page["id"])
# self.mediaNsPages.append(self.page)
print(self.page)
def characters(self, content):
return super().characters(content, not_parse_tags=["revision"])
def get_titles_from_xml(xmlfile, return_type="list", silent=False):
'''Return a list/set of titles from a XML dump file.\n
"""Return a list/set of titles from a XML dump file.\n
`xmlfile`: a system identifier or an InputSource.\n
`return_type`:`"list"` or `"set"` (default: `"list"`).
The `list` keeps the order of XML file, and is unique.
'''
"""
# xmlfile_size = os.path.getsize(xmlfile)
parser = xml.sax.make_parser()
handler = TitlesHandler(os.path.getsize(xmlfile))
@ -296,12 +312,19 @@ def get_titles_from_xml(xmlfile, return_type="list", silent=False):
parser.setContentHandler(handler)
parser.parse(xmlfile)
handler.close_tqdm()
print('',flush=True)
print('pageTagsCount:', handler.pageTagsCount,
'titleTagsCount:', handler.titleTagsCount,
'nsTagsCount:', handler.nsTagsCount,
'idTagsCount:', handler.idTagsCount,
'revisionTagsCount:', handler.revisionTagsCount)
print("", flush=True)
print(
"pageTagsCount:",
handler.pageTagsCount,
"titleTagsCount:",
handler.titleTagsCount,
"nsTagsCount:",
handler.nsTagsCount,
"idTagsCount:",
handler.idTagsCount,
"revisionTagsCount:",
handler.revisionTagsCount,
)
# print('MediaNsPages (Name):', len(handler.mediaNsPagesName_set))
# print('MediaNsPages (ID):', len(handler.mediaNsPagesID_set))
@ -309,7 +332,7 @@ def get_titles_from_xml(xmlfile, return_type="list", silent=False):
raise RuntimeError("len(set_titles) and (list_titles) are not equal!")
titles = handler.set_titles if return_type == "set" else handler.list_titles
return titles
@ -319,13 +342,14 @@ class Config:
dry: bool
verbose: bool
def getArguments():
parser = argparse.ArgumentParser()
parser.description = "Extracts all titles from a XML dump file and writes them to `*-xml2titles.txt`."
parser.add_argument("xmlfile", help="XML file of wiki dump")
parser.add_argument("--dry", help="Do not write to file",action="store_true")
parser.add_argument("--verbose", help="Verbose",action="store_true")
parser.add_argument("--dry", help="Do not write to file", action="store_true")
parser.add_argument("--verbose", help="Verbose", action="store_true")
args = parser.parse_args()
config = Config
@ -339,7 +363,7 @@ def getArguments():
if __name__ == "__main__":
args = getArguments()
print('Parsing...')
print("Parsing...")
xmlfile = args.xmlfile
if not os.path.exists(xmlfile):
@ -351,10 +375,12 @@ if __name__ == "__main__":
assert xml_basename.endswith(".xml")
"XML file name does not end with .xml!"
assert xml_basename.endswith("-current.xml") or xml_basename.endswith("-history.xml")
assert xml_basename.endswith("-current.xml") or xml_basename.endswith(
"-history.xml"
)
"XML file name does not end with -current.xml or -history.xml!"
with FileReadBackwards(xmlfile, encoding='utf-8') as frb:
with FileReadBackwards(xmlfile, encoding="utf-8") as frb:
seeked = 0
for line in frb:
seeked += 1
@ -362,7 +388,7 @@ if __name__ == "__main__":
# xml dump is complete
break
if seeked > 4:
raise Exception('xml dump is incomplete!')
raise Exception("xml dump is incomplete!")
_silent = not args.verbose
@ -372,7 +398,9 @@ if __name__ == "__main__":
print("Dry run. No file will be written.")
sys.exit(0)
titles_filename = xml_basename.replace("-current.xml", "-xml2titles.txt").replace("-history.xml", "-xml2titles.txt")
titles_filename = xml_basename.replace("-current.xml", "-xml2titles.txt").replace(
"-history.xml", "-xml2titles.txt"
)
titles_filepath = os.path.join(xml_dir, titles_filename)
with open(titles_filepath, "w") as f:
f.write("\n".join(titles))

2
wikiteam3/dumpgenerator/__main__.py
Unescape Escape View File

@ -1,4 +1,6 @@
if __name__ == "__main__":
import sys
from .__init__ import main
sys.exit(main())

23
wikiteam3/dumpgenerator/api/api.py
Unescape Escape View File

@ -1,16 +1,17 @@
from typing import *
import re
import time
from urllib.parse import urlparse, urlunparse, urljoin
from typing import *
from urllib.parse import urljoin, urlparse, urlunparse
import mwclient
import requests
from .get_json import getJSON
from wikiteam3.utils import getUserAgent
from .get_json import getJSON
def checkAPI(api="", session: requests.Session=None):
def checkAPI(api="", session: requests.Session = None):
"""Checking API availability"""
global cj
# handle redirects
@ -55,7 +56,7 @@ def checkAPI(api="", session: requests.Session=None):
return None
def mwGetAPIAndIndex(url="", session: requests.Session=None):
def mwGetAPIAndIndex(url="", session: requests.Session = None):
"""Returns the MediaWiki API and Index.php"""
api = ""
@ -114,7 +115,7 @@ def mwGetAPIAndIndex(url="", session: requests.Session=None):
return api, index
def checkRetryAPI(api="", apiclient=False, session: requests.Session=None):
def checkRetryAPI(api="", apiclient=False, session: requests.Session = None):
"""Call checkAPI and mwclient if necessary"""
check = None
try:
@ -126,7 +127,10 @@ def checkRetryAPI(api="", apiclient=False, session: requests.Session=None):
apiurl = urlparse(api)
try:
site = mwclient.Site(
apiurl.netloc, apiurl.path.replace("api.php", ""), scheme=apiurl.scheme, pool=session
apiurl.netloc,
apiurl.path.replace("api.php", ""),
scheme=apiurl.scheme,
pool=session,
)
except KeyError:
# Probably KeyError: 'query'
@ -144,7 +148,10 @@ def checkRetryAPI(api="", apiclient=False, session: requests.Session=None):
try:
site = mwclient.Site(
apiurl.netloc, apiurl.path.replace("api.php", ""), scheme=newscheme, pool=session
apiurl.netloc,
apiurl.path.replace("api.php", ""),
scheme=newscheme,
pool=session,
)
except KeyError:
check = False

2
wikiteam3/dumpgenerator/api/index_check.py
Unescape Escape View File

@ -3,7 +3,7 @@ import re
import requests
def checkIndex(index="", cookies="", session: requests.Session=None):
def checkIndex(index="", cookies="", session: requests.Session = None):
"""Checking index.php availability"""
r = session.post(url=index, data={"title": "Special:Version"}, timeout=30)
if r.status_code >= 400:

7
wikiteam3/dumpgenerator/api/namespaces.py
Unescape Escape View File

@ -1,10 +1,11 @@
import re
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.dumpgenerator.api import getJSON
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.dumpgenerator.config import Config
def getNamespacesScraper(config: Config=None, session=None):
def getNamespacesScraper(config: Config = None, session=None):
"""Hackishly gets the list of namespaces names and ids from the dropdown in the HTML of Special:AllPages"""
"""Function called if no API is available"""
namespaces = config.namespaces
@ -43,7 +44,7 @@ def getNamespacesScraper(config: Config=None, session=None):
return namespaces, namespacenames
def getNamespacesAPI(config: Config=None, session=None):
def getNamespacesAPI(config: Config = None, session=None):
"""Uses the API to get the list of namespaces names and ids"""
namespaces = config.namespaces
namespacenames = {0: ""} # main is 0, no prefix

59
wikiteam3/dumpgenerator/api/page_titles.py
Unescape Escape View File

@ -5,20 +5,25 @@ from urllib.parse import urlparse
import mwclient
from file_read_backwards import FileReadBackwards
from wikiteam3.dumpgenerator.api.namespaces import (
getNamespacesAPI,
getNamespacesScraper,
)
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.dumpgenerator.api.namespaces import getNamespacesAPI, getNamespacesScraper
from wikiteam3.utils import domain2prefix, cleanHTML, undoHTMLEntities
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.utils import cleanHTML, domain2prefix, undoHTMLEntities
from wikiteam3.utils.monkey_patch import DelaySession
def getPageTitlesAPI(config: Config=None, session=None):
def getPageTitlesAPI(config: Config = None, session=None):
"""Uses the API to get the list of page titles"""
titles = []
namespaces, namespacenames = getNamespacesAPI(config=config, session=session)
# apply delay to the session for mwclient.Site.allpages()
delay_session = DelaySession(session=session, msg="Session delay: "+__name__, config=config)
delay_session = DelaySession(
session=session, msg="Session delay: " + __name__, config=config
)
delay_session.hijack()
for namespace in namespaces:
if namespace in config.exnamespaces:
@ -29,7 +34,10 @@ def getPageTitlesAPI(config: Config=None, session=None):
print(" Retrieving titles in the namespace %d" % (namespace))
apiurl = urlparse(config.api)
site = mwclient.Site(
apiurl.netloc, apiurl.path.replace("api.php", ""), scheme=apiurl.scheme, pool=session
apiurl.netloc,
apiurl.path.replace("api.php", ""),
scheme=apiurl.scheme,
pool=session,
)
for page in site.allpages(namespace=namespace):
title = page.name
@ -44,15 +52,13 @@ def getPageTitlesAPI(config: Config=None, session=None):
delay_session.release()
def getPageTitlesScraper(config: Config=None, session=None):
def getPageTitlesScraper(config: Config = None, session=None):
"""Scrape the list of page titles from Special:Allpages"""
titles = []
namespaces, namespacenames = getNamespacesScraper(config=config, session=session)
for namespace in namespaces:
print(" Retrieving titles in the namespace", namespace)
url = "{}?title=Special:Allpages&namespace={}".format(
config.index, namespace
)
url = f"{config.index}?title=Special:Allpages&namespace={namespace}"
r = session.get(url=url, timeout=30)
raw = r.text
raw = cleanHTML(raw)
@ -141,8 +147,10 @@ def getPageTitlesScraper(config: Config=None, session=None):
)
Delay(config=config, session=session)
assert currfr is not None, "re.search found the pattern, but re.finditer fails, why?"
assert (
currfr is not None
), "re.search found the pattern, but re.finditer fails, why?"
oldfr = currfr
c += 1
@ -158,7 +166,7 @@ def getPageTitlesScraper(config: Config=None, session=None):
return titles
def getPageTitles(config: Config=None, session=None):
def getPageTitles(config: Config = None, session=None):
"""Get list of page titles"""
# http://en.wikipedia.org/wiki/Special:AllPages
# http://wiki.archiveteam.org/index.php?title=Special:AllPages
@ -193,9 +201,7 @@ def getPageTitles(config: Config=None, session=None):
titlesfilename = "{}-{}-titles.txt".format(
domain2prefix(config=config), config.date
)
titlesfile = open(
"{}/{}".format(config.path, titlesfilename), "wt", encoding="utf-8"
)
titlesfile = open(f"{config.path}/{titlesfilename}", "w", encoding="utf-8")
c = 0
for title in titles:
titlesfile.write(str(title) + "\n")
@ -210,16 +216,19 @@ def getPageTitles(config: Config=None, session=None):
print("%d page titles loaded" % (c))
return titlesfilename
def checkTitleOk(config: Config=None, ):
def checkTitleOk(
config: Config = None,
):
try:
with FileReadBackwards(
"%s/%s-%s-titles.txt"
% (
config.path,
domain2prefix(config=config),
config.date,
),
encoding="utf-8",
"%s/%s-%s-titles.txt"
% (
config.path,
domain2prefix(config=config),
config.date,
),
encoding="utf-8",
) as frb:
lasttitle = frb.readline().strip()
if lasttitle == "":
@ -232,7 +241,7 @@ def checkTitleOk(config: Config=None, ):
return True
def readTitles(config: Config=None, session=None, start=None, batch=False):
def readTitles(config: Config = None, session=None, start=None, batch=False):
"""Read title list from a file, from the title "start" """
if not checkTitleOk(config):
getPageTitles(config=config, session=session)
@ -240,7 +249,7 @@ def readTitles(config: Config=None, session=None, start=None, batch=False):
titlesfilename = "{}-{}-titles.txt".format(
domain2prefix(config=config), config.date
)
titlesfile = open("{}/{}".format(config.path, titlesfilename), encoding="utf-8")
titlesfile = open(f"{config.path}/{titlesfilename}", encoding="utf-8")
titlelist = []
seeking = False

2
wikiteam3/dumpgenerator/api/wiki_check.py
Unescape Escape View File

@ -5,7 +5,7 @@ import requests
from wikiteam3.utils import getUserAgent
def getWikiEngine(url="", session: requests.Session=None) -> str:
def getWikiEngine(url="", session: requests.Session = None) -> str:
"""Returns the wiki engine of a URL, if known"""
if not session:

2
wikiteam3/dumpgenerator/cli/__init__.py
Unescape Escape View File

@ -1,3 +1,3 @@
from .cli import getParameters
from .greeter import bye, welcome
from .delay import Delay
from .greeter import bye, welcome

168
wikiteam3/dumpgenerator/cli/cli.py
Unescape Escape View File

@ -1,4 +1,3 @@
import argparse
import datetime
import http
@ -7,24 +6,20 @@ import os
import queue
import re
import sys
from typing import *
import requests
import urllib3
from wikiteam3.dumpgenerator.api import checkRetryAPI, mwGetAPIAndIndex
from wikiteam3.utils.login import uniLogin
from .delay import Delay
from wikiteam3.utils import domain2prefix
from wikiteam3.dumpgenerator.api import checkRetryAPI, getWikiEngine, mwGetAPIAndIndex
from wikiteam3.dumpgenerator.api.index_check import checkIndex
from wikiteam3.utils import getUserAgent
from wikiteam3.dumpgenerator.version import getVersion
from wikiteam3.dumpgenerator.api import getWikiEngine
from wikiteam3.dumpgenerator.config import Config, newConfig
from wikiteam3.utils import mod_requests_text
from typing import *
from wikiteam3.dumpgenerator.version import getVersion
from wikiteam3.utils import domain2prefix, getUserAgent, mod_requests_text
from wikiteam3.utils.login import uniLogin
from ...utils.user_agent import setupUserAgent
from .delay import Delay
def getArgumentParser():
@ -36,7 +31,11 @@ def getArgumentParser():
"--cookies", metavar="cookies.txt", help="path to a cookies.txt file"
)
parser.add_argument(
"--delay", metavar="5", default=0.5, type=float, help="adds a delay (in seconds)"
"--delay",
metavar="5",
default=0.5,
type=float,
help="adds a delay (in seconds)",
)
parser.add_argument(
"--retries", metavar="5", default=5, help="Maximum number of retries for "
@ -48,34 +47,49 @@ def getArgumentParser():
help="resumes previous incomplete dump (requires --path)",
)
parser.add_argument("--force", action="store_true", help="")
parser.add_argument("--user", help="Username if MedaiWiki authentication is required.")
parser.add_argument(
"--pass", dest="password", help="Password if MediaWiki authentication is required."
"--user", help="Username if MedaiWiki authentication is required."
)
parser.add_argument(
"--http-user", dest="http_user", help="Username if HTTP authentication is required."
"--pass",
dest="password",
help="Password if MediaWiki authentication is required.",
)
parser.add_argument(
"--http-pass", dest="http_password", help="Password if HTTP authentication is required."
"--http-user",
dest="http_user",
help="Username if HTTP authentication is required.",
)
parser.add_argument(
'--insecure', action='store_true', help='Disable SSL certificate verification'
"--http-pass",
dest="http_password",
help="Password if HTTP authentication is required.",
)
parser.add_argument(
"--insecure", action="store_true", help="Disable SSL certificate verification"
)
parser.add_argument(
"--stdout-log-file", dest="stdout_log_path", default=None, help="Path to copy stdout to",
"--stdout-log-file",
dest="stdout_log_path",
default=None,
help="Path to copy stdout to",
)
# URL params
groupWikiOrAPIOrIndex = parser.add_argument_group()
groupWikiOrAPIOrIndex.add_argument(
"wiki", default="", nargs="?", help="URL to wiki (e.g. http://wiki.domain.org), auto detects API and index.php"
"wiki",
default="",
nargs="?",
help="URL to wiki (e.g. http://wiki.domain.org), auto detects API and index.php",
)
groupWikiOrAPIOrIndex.add_argument(
"--api", help="URL to API (e.g. http://wiki.domain.org/w/api.php)"
)
groupWikiOrAPIOrIndex.add_argument(
"--index", help="URL to index.php (e.g. http://wiki.domain.org/w/index.php), (not supported with --images on newer(?) MediaWiki without --api)"
"--index",
help="URL to index.php (e.g. http://wiki.domain.org/w/index.php), (not supported with --images on newer(?) MediaWiki without --api)",
)
# Download params
@ -88,7 +102,9 @@ def getArgumentParser():
help="Export XML dump using Special:Export (index.php). (supported with --curonly)",
)
groupDownload.add_argument(
"--curonly", action="store_true", help="store only the lastest revision of pages"
"--curonly",
action="store_true",
help="store only the lastest revision of pages",
)
groupDownload.add_argument(
"--xmlapiexport",
@ -124,7 +140,10 @@ def getArgumentParser():
help="comma-separated value of namespaces to exclude",
)
parser.add_argument(
"--api_chunksize", metavar="50", default=50, help="Chunk size for MediaWiki API (arvlimit, ailimit, etc.)"
"--api_chunksize",
metavar="50",
default=50,
help="Chunk size for MediaWiki API (arvlimit, ailimit, etc.)",
)
# Meta info params
@ -143,7 +162,6 @@ def getArgumentParser():
def checkParameters(args=argparse.Namespace()) -> bool:
passed = True
# Don't mix download params and meta info params
@ -162,29 +180,34 @@ def checkParameters(args=argparse.Namespace()) -> bool:
passed = False
# Check http-user and http-pass (one requires both)
if (args.http_user and not args.http_password) or (args.http_password and not args.http_user):
print("ERROR: Both --http-user and --http-pass are required for authentication.")
if (args.http_user and not args.http_password) or (
args.http_password and not args.http_user
):
print(
"ERROR: Both --http-user and --http-pass are required for authentication."
)
passed = False
# --curonly requires --xml
if args.curonly and not args.xml:
print("ERROR: --curonly requires --xml")
passed = False
# --xmlrevisions not supported with --curonly
if args.xmlrevisions and args.curonly:
print("ERROR: --xmlrevisions not supported with --curonly")
passed = False
# Check URLs
for url in [args.api, args.index, args.wiki]:
if url and (not url.startswith("http://") and not url.startswith("https://")):
print(url)
print("ERROR: URLs must start with http:// or https://")
passed = False
return passed
def getParameters(params=None) -> Tuple[Config, Dict]:
# if not params:
# params = sys.argv
@ -200,7 +223,7 @@ def getParameters(params=None) -> Tuple[Config, Dict]:
########################################
# Create session
mod_requests_text(requests) # monkey patch
mod_requests_text(requests) # monkey patch
session = requests.Session()
# Disable SSL verification
@ -217,12 +240,14 @@ def getParameters(params=None) -> Tuple[Config, Dict]:
# Courtesy datashaman https://stackoverflow.com/a/35504626
class CustomRetry(Retry):
def increment(self, method=None, url=None, *args, **kwargs):
if '_pool' in kwargs:
conn = kwargs['_pool'] # type: urllib3.connectionpool.HTTPSConnectionPool
if 'response' in kwargs:
if "_pool" in kwargs:
conn = kwargs[
"_pool"
] # type: urllib3.connectionpool.HTTPSConnectionPool
if "response" in kwargs:
try:
# drain conn in advance so that it won't be put back into conn.pool
kwargs['response'].drain_conn()
kwargs["response"].drain_conn()
except:
pass
# Useless, retry happens inside urllib3
@ -231,7 +256,7 @@ def getParameters(params=None) -> Tuple[Config, Dict]:
# adapters.poolmanager.clear()
# Close existing connection so that a new connection will be used
if hasattr(conn, 'pool'):
if hasattr(conn, "pool"):
pool = conn.pool # type: queue.Queue
try:
# Don't directly use this, This closes connection pool by making conn.pool = None
@ -239,22 +264,31 @@ def getParameters(params=None) -> Tuple[Config, Dict]:
except:
pass
conn.pool = pool
return super(CustomRetry, self).increment(method=method, url=url, *args, **kwargs)
return super().increment(method=method, url=url, *args, **kwargs)
def sleep(self, response=None):
backoff = self.get_backoff_time()
if backoff <= 0:
return
if response is not None:
msg = 'req retry (%s)' % response.status
msg = "req retry (%s)" % response.status
else:
msg = None
Delay(config=None, session=session, msg=msg, delay=backoff)
__retries__ = CustomRetry(
total=int(args.retries), backoff_factor=0.3,
total=int(args.retries),
backoff_factor=0.3,
status_forcelist=[500, 502, 503, 504, 429],
allowed_methods=['DELETE', 'PUT', 'GET', 'OPTIONS', 'TRACE', 'HEAD', 'POST']
allowed_methods=[
"DELETE",
"PUT",
"GET",
"OPTIONS",
"TRACE",
"HEAD",
"POST",
],
)
session.mount("https://", HTTPAdapter(max_retries=__retries__))
session.mount("http://", HTTPAdapter(max_retries=__retries__))
@ -271,7 +305,7 @@ def getParameters(params=None) -> Tuple[Config, Dict]:
# Setup user agent
session.headers.update({"User-Agent": getUserAgent()})
setupUserAgent(session) # monkey patch
setupUserAgent(session) # monkey patch
# Set HTTP Basic Auth
if args.http_user and args.http_password:
@ -319,7 +353,7 @@ def getParameters(params=None) -> Tuple[Config, Dict]:
# Replace the index URL we got from the API check
index2 = check[1]
api = checkedapi
print("API is OK: ", checkedapi)
print("API is OK: ", checkedapi)
else:
if index and not args.wiki:
print("API not available. Trying with index.php only.")
@ -331,7 +365,13 @@ def getParameters(params=None) -> Tuple[Config, Dict]:
# login if needed
# TODO: Re-login after session expires
if args.user and args.password:
_session = uniLogin(api=api, index=index, session=session, username=args.user, password=args.password)
_session = uniLogin(
api=api,
index=index,
session=session,
username=args.user,
password=args.password,
)
if _session:
session = _session
print("-- Login OK --")
@ -362,7 +402,6 @@ def getParameters(params=None) -> Tuple[Config, Dict]:
)
sys.exit(1)
namespaces = ["all"]
exnamespaces = []
# Process namespace inclusions
@ -398,28 +437,29 @@ def getParameters(params=None) -> Tuple[Config, Dict]:
else:
exnamespaces = [int(i) for i in ns.split(",")]
config = newConfig({
"curonly": args.curonly,
"date": datetime.datetime.now().strftime("%Y%m%d"),
"api": api,
"failfast": args.failfast,
"http_method": "POST",
"api_chunksize": int(args.api_chunksize),
"index": index,
"images": args.images,
"logs": False,
"xml": args.xml,
"xmlapiexport": args.xmlapiexport,
"xmlrevisions": args.xmlrevisions or args.xmlrevisions_page,
"xmlrevisions_page": args.xmlrevisions_page,
"namespaces": namespaces,
"exnamespaces": exnamespaces,
"path": args.path and os.path.normpath(args.path) or "",
"cookies": args.cookies or "",
"delay": args.delay,
"retries": int(args.retries),
})
config = newConfig(
{
"curonly": args.curonly,
"date": datetime.datetime.now().strftime("%Y%m%d"),
"api": api,
"failfast": args.failfast,
"http_method": "POST",
"api_chunksize": int(args.api_chunksize),
"index": index,
"images": args.images,
"logs": False,
"xml": args.xml,
"xmlapiexport": args.xmlapiexport,
"xmlrevisions": args.xmlrevisions or args.xmlrevisions_page,
"xmlrevisions_page": args.xmlrevisions_page,
"namespaces": namespaces,
"exnamespaces": exnamespaces,
"path": args.path and os.path.normpath(args.path) or "",
"cookies": args.cookies or "",
"delay": args.delay,
"retries": int(args.retries),
}
)
other = {
"resume": args.resume,

7
wikiteam3/dumpgenerator/cli/delay.py
Unescape Escape View File

@ -1,10 +1,11 @@
import itertools
import sys
import threading
import time
import sys
from wikiteam3.dumpgenerator.config import Config
class Delay:
done: bool = False
lock: threading.Lock = threading.Lock()
@ -20,7 +21,7 @@ class Delay:
time.sleep(0.3)
def __init__(self, config: Config=None, session=None, msg=None, delay=None):
def __init__(self, config: Config = None, session=None, msg=None, delay=None):
"""Add a delay if configured for that"""
self.ellipses: str = "."
@ -30,7 +31,7 @@ class Delay:
return
if msg:
self.ellipses = ("Delay %.1fs: %s " % (delay, msg)) + self.ellipses
self.ellipses = (f"Delay {delay:.1f}s: {msg} ") + self.ellipses
else:
self.ellipses = ("Delay %.1fs " % (delay)) + self.ellipses

62
wikiteam3/dumpgenerator/config.py
Unescape Escape View File

@ -1,19 +1,4 @@
import dataclasses
import json
import sys
from typing import *
def _dataclass_from_dict(klass_or_obj, d):
if isinstance(klass_or_obj, type): # klass
ret = klass_or_obj()
else:
ret = klass_or_obj
for k,v in d.items():
if hasattr(ret, k):
setattr(ret, k, v)
return ret
'''
"""
config = {
"curonly": args.curonly,
"date": datetime.datetime.now().strftime("%Y%m%d"),
@ -32,7 +17,25 @@ config = {
"delay": args.delay,
"retries": int(args.retries),
}
'''
"""
import dataclasses
import json
import sys
from typing import *
def _dataclass_from_dict(klass_or_obj, d):
if isinstance(klass_or_obj, type): # klass
ret = klass_or_obj()
else:
ret = klass_or_obj
for k, v in d.items():
if hasattr(ret, k):
setattr(ret, k, v)
return ret
@dataclasses.dataclass
class Config:
def asdict(self):
@ -41,13 +44,13 @@ class Config:
# General params
delay: float = 0.0
retries: int = 0
path: str = ''
path: str = ""
logs: bool = False
date: str = False
# URL params
index: str = ''
api: str = ''
index: str = ""
api: str = ""
# Download params
xml: bool = False
@ -60,27 +63,27 @@ class Config:
exnamespaces: List[int] = None
api_chunksize: int = 0 # arvlimit, ailimit, etc
export: str = '' # Special:Export page name
http_method: str = ''
export: str = "" # Special:Export page name
http_method: str = ""
# Meta info params
failfast: bool = False
templates: bool = False
def newConfig(configDict) -> Config:
return _dataclass_from_dict(Config, configDict)
def loadConfig(config: Config=None, configfilename=""):
def loadConfig(config: Config = None, configfilename=""):
"""Load config file"""
configDict = dataclasses.asdict(config)
if config.path:
try:
with open(
"{}/{}".format(config.path, configfilename), encoding="utf-8"
) as infile:
with open(f"{config.path}/{configfilename}", encoding="utf-8") as infile:
configDict.update(json.load(infile))
return newConfig(configDict)
except:
@ -89,10 +92,9 @@ def loadConfig(config: Config=None, configfilename=""):
print("There is no config file. we can't resume. Start a new dump.")
sys.exit()
def saveConfig(config: Config=None, configfilename=""):
def saveConfig(config: Config = None, configfilename=""):
"""Save config file"""
with open(
"{}/{}".format(config.path, configfilename), "w", encoding="utf-8"
) as outfile:
with open(f"{config.path}/{configfilename}", "w", encoding="utf-8") as outfile:
json.dump(dataclasses.asdict(config), outfile)

77
wikiteam3/dumpgenerator/dump/generator.py
Unescape Escape View File

@ -22,26 +22,23 @@ except ImportError:
from typing import *
from wikiteam3.dumpgenerator.config import loadConfig, saveConfig
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.dumpgenerator.cli import getParameters, bye, welcome
from wikiteam3.dumpgenerator.log import logerror
from wikiteam3.utils import domain2prefix
from wikiteam3.utils import undoHTMLEntities
from wikiteam3.utils import avoidWikimediaProjects
from wikiteam3.dumpgenerator.cli import bye, getParameters, welcome
from wikiteam3.dumpgenerator.config import Config, loadConfig, saveConfig
from wikiteam3.dumpgenerator.dump.image.image import Image
from wikiteam3.dumpgenerator.dump.misc.index_php import saveIndexPHP
from wikiteam3.dumpgenerator.dump.misc.site_info import saveSiteInfo
from wikiteam3.dumpgenerator.dump.misc.special_logs import saveLogs
from wikiteam3.dumpgenerator.dump.misc.special_version import saveSpecialVersion
from wikiteam3.dumpgenerator.dump.misc.site_info import saveSiteInfo
from wikiteam3.dumpgenerator.dump.xmldump.xml_dump import generateXMLDump
from wikiteam3.dumpgenerator.dump.xmldump.xml_integrity import checkXMLIntegrity
from wikiteam3.dumpgenerator.log import logerror
from wikiteam3.utils import avoidWikimediaProjects, domain2prefix, undoHTMLEntities
# From https://stackoverflow.com/a/57008707
class Tee(object):
class Tee:
def __init__(self, filename):
self.file = open(filename, 'w', encoding="utf-8")
self.file = open(filename, "w", encoding="utf-8")
self.stdout = sys.stdout
def __enter__(self):
@ -61,6 +58,7 @@ class Tee(object):
self.file.flush()
self.stdout.flush()
class DumpGenerator:
configfilename = "config.json"
@ -71,7 +69,11 @@ class DumpGenerator:
config, other = getParameters(params=params)
avoidWikimediaProjects(config=config, other=other)
with (Tee(other["stdout_log_path"]) if other["stdout_log_path"] is not None else contextlib.nullcontext()):
with (
Tee(other["stdout_log_path"])
if other["stdout_log_path"] is not None
else contextlib.nullcontext()
):
print(welcome())
print("Analysing %s" % (config.api if config.api else config.index))
@ -91,7 +93,7 @@ class DumpGenerator:
% (config.path, config.path, configfilename)
)
if reply.lower() in ["yes", "y"]:
if not os.path.isfile("{}/{}".format(config.path, configfilename)):
if not os.path.isfile(f"{config.path}/{configfilename}"):
print("No config file found. I can't resume. Aborting.")
sys.exit()
print("You have selected: YES")
@ -122,7 +124,7 @@ class DumpGenerator:
bye()
@staticmethod
def createNewDump(config: Config=None, other: Dict=None):
def createNewDump(config: Config = None, other: Dict = None):
# we do lazy title dumping here :)
images = []
print("Trying generating a new dump into a new directory...")
@ -139,11 +141,10 @@ class DumpGenerator:
saveLogs(config=config, session=other["session"])
@staticmethod
def resumePreviousDump(config: Config=None, other: Dict=None):
def resumePreviousDump(config: Config = None, other: Dict = None):
images = []
print("Resuming previous dump process...")
if config.xml:
# checking xml dump
xmliscomplete = False
lastxmltitle = None
@ -180,7 +181,10 @@ class DumpGenerator:
print("XML dump was completed in the previous session")
elif lastxmltitle:
# resuming...
print('Resuming XML dump from "%s" (revision id %s)' % (lastxmltitle, lastxmlrevid))
print(
'Resuming XML dump from "%s" (revision id %s)'
% (lastxmltitle, lastxmlrevid)
)
generateXMLDump(
config=config,
session=other["session"],
@ -194,14 +198,18 @@ class DumpGenerator:
if config.images:
# load images list
lastimage = ""
imagesFilePath = "%s/%s-%s-images.txt" % (config.path, domain2prefix(config=config), config.date)
imagesFilePath = "{}/{}-{}-images.txt".format(
config.path,
domain2prefix(config=config),
config.date,
)
if os.path.exists(imagesFilePath):
f = open(imagesFilePath)
lines = f.read().splitlines()
for l in lines:
if re.search(r"\t", l):
images.append(l.split("\t"))
if len(lines) == 0: # empty file
if len(lines) == 0: # empty file
lastimage = "--EMPTY--"
if lastimage == "":
lastimage = lines[-1].strip()
@ -209,10 +217,10 @@ class DumpGenerator:
lastimage = lines[-2].strip()
f.close()
if len(images)>0 and len(images[0]) < 5:
if len(images) > 0 and len(images[0]) < 5:
print(
"Warning: Detected old images list (images.txt) format.\n"+
"You can delete 'images.txt' manually and restart the script."
"Warning: Detected old images list (images.txt) format.\n"
+ "You can delete 'images.txt' manually and restart the script."
)
sys.exit(1)
if lastimage == "--END--":
@ -235,31 +243,36 @@ class DumpGenerator:
c_checked = 0
for filename, url, uploader, size, sha1 in images:
lastfilename = filename
if other["filenamelimit"] < len(filename.encode('utf-8')):
if other["filenamelimit"] < len(filename.encode("utf-8")):
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text=f"Filename too long(>240 bytes), skipping: {filename}",
)
continue
if filename in listdir:
c_images += 1
if filename+".desc" in listdir:
if filename + ".desc" in listdir:
c_desc += 1
c_checked += 1
if c_checked % 100000 == 0:
print(f"checked {c_checked}/{len(images)} records", end="\r")
print(f"{len(images)} records in images.txt, {c_images} images and {c_desc} .desc were saved in the previous session")
print(
f"{len(images)} records in images.txt, {c_images} images and {c_desc} .desc were saved in the previous session"
)
if c_desc < len(images):
complete = False
elif c_images < len(images):
complete = False
print("WARNING: Some images were not saved. You may want to delete their \n"
+".desc files and re-run the script to redownload the missing images.\n"
+"(If images URL are unavailable, you can ignore this warning.)\n"
+"(In most cases, if the number of .desc files equals the number of \n"
+ "images.txt records, you can ignore this warning, images dump was completed.)")
print(
"WARNING: Some images were not saved. You may want to delete their \n"
+ ".desc files and re-run the script to redownload the missing images.\n"
+ "(If images URL are unavailable, you can ignore this warning.)\n"
+ "(In most cases, if the number of .desc files equals the number of \n"
+ "images.txt records, you can ignore this warning, images dump was completed.)"
)
sys.exit()
else: # c_desc == c_images == len(images)
else: # c_desc == c_images == len(images)
complete = True
if complete:
# image dump is complete

18
wikiteam3/dumpgenerator/dump/image/html_regexs.py
Unescape Escape View File

@ -8,27 +8,27 @@ REGEX_CANDIDATES = [
# class="new" title="Usuario:Fernandocg (página no
# existe)">Fernandocg</a></td>
r'(?im)<td class="TablePager_col_img_name"><a href[^>]+title="[^:>]+:(?P<filename>[^>]+)">[^<]+</a>[^<]+<a href="(?P<url>[^>]+/[^>/]+)">[^<]+</a>[^<]+</td>\s*<td class="TablePager_col_img_user_text"><a[^>]+>(?P<uploader>[^<]+)</a></td>'
# [1]
# wikijuegos 1.9.5
# http://softwarelibre.uca.es/wikijuegos/Especial:Imagelist old
# mediawiki version
,r'(?im)<td class="TablePager_col_links"><a href[^>]+title="[^:>]+:(?P<filename>[^>]+)">[^<]+</a>[^<]+<a href="(?P<url>[^>]+/[^>/]+)">[^<]+</a></td>\s*<td class="TablePager_col_img_timestamp">[^<]+</td>\s*<td class="TablePager_col_img_name">[^<]+</td>\s*<td class="TablePager_col_img_user_text"><a[^>]+>(?P<uploader>[^<]+)</a></td>'
,
r'(?im)<td class="TablePager_col_links"><a href[^>]+title="[^:>]+:(?P<filename>[^>]+)">[^<]+</a>[^<]+<a href="(?P<url>[^>]+/[^>/]+)">[^<]+</a></td>\s*<td class="TablePager_col_img_timestamp">[^<]+</td>\s*<td class="TablePager_col_img_name">[^<]+</td>\s*<td class="TablePager_col_img_user_text"><a[^>]+>(?P<uploader>[^<]+)</a></td>'
# [2]
# gentoowiki 1.18
,r'(?im)<td class="TablePager_col_img_name"><a[^>]+title="[^:>]+:(?P<filename>[^>]+)">[^<]+</a>[^<]+<a href="(?P<url>[^>]+)">[^<]+</a>[^<]+</td><td class="TablePager_col_thumb"><a[^>]+><img[^>]+></a></td><td class="TablePager_col_img_size">[^<]+</td><td class="TablePager_col_img_user_text"><a[^>]+>(?P<uploader>[^<]+)</a></td>'
,
r'(?im)<td class="TablePager_col_img_name"><a[^>]+title="[^:>]+:(?P<filename>[^>]+)">[^<]+</a>[^<]+<a href="(?P<url>[^>]+)">[^<]+</a>[^<]+</td><td class="TablePager_col_thumb"><a[^>]+><img[^>]+></a></td><td class="TablePager_col_img_size">[^<]+</td><td class="TablePager_col_img_user_text"><a[^>]+>(?P<uploader>[^<]+)</a></td>'
# [3]
# http://www.memoryarchive.org/en/index.php?title=Special:Imagelist&sort=byname&limit=50&wpIlMatch=
# (<a href="/en/Image:109_0923.JPG" title="Image:109 0923.JPG">desc</a>) <a href="/en/upload/c/cd/109_0923.JPG">109 0923.JPG</a> . . 885,713 bytes . . <a href="/en/User:Bfalconer" title="User:Bfalconer">Bfalconer</a> . . 18:44, 17 November 2005<br />
,'(?ism)<a href=[^>]+ title="[^:>]+:(?P<filename>[^>]+)">[^<]+</a>[^<]+<a href="(?P<url>[^>]+)">[^<]+</a>[^<]+<a[^>]+>(?P<uploader>[^<]+)</a>'
,
'(?ism)<a href=[^>]+ title="[^:>]+:(?P<filename>[^>]+)">[^<]+</a>[^<]+<a href="(?P<url>[^>]+)">[^<]+</a>[^<]+<a[^>]+>(?P<uploader>[^<]+)</a>'
# [4]
,(
,
(
r'(?im)<td class="TablePager_col_img_name">\s*<a href[^>]*?>(?P<filename>[^>]+)</a>[^<]*?<a href="(?P<url>[^>]+)">[^<]*?</a>[^<]*?</td>\s*'
r'<td class="TablePager_col_thumb">[^\n\r]*?</td>\s*'
r'<td class="TablePager_col_img_size">[^<]*?</td>\s*'
r'<td class="(?:TablePager_col_img_user_text|TablePager_col_img_actor)">\s*(<a href="[^>]*?" title="[^>]*?">)?(?P<uploader>[^<]+?)(</a>)?\s*</td>'
)
),
]

27
wikiteam3/dumpgenerator/dump/image/html_regexs_test.py
Unescape Escape View File

@ -1,9 +1,10 @@
import os
import re
from pathlib import Path
from typing import Dict, List
import pytest
import requests
from pathlib import Path
from wikiteam3.dumpgenerator.dump.image.html_regexs import REGEX_CANDIDATES
@ -12,6 +13,7 @@ ONLINE = True
HTML_DIR = Path("test/data/html_regexs")
os.makedirs(HTML_DIR, exist_ok=True)
def prepare_raws_from_urls(urls: Dict[str, str]):
sess = requests.Session()
raws: Dict[str, str] = {}
@ -28,10 +30,14 @@ def prepare_raws_from_urls(urls: Dict[str, str]):
with open(HTML_DIR / f"{site}.html", "w", encoding="utf-8") as f:
f.write(resp.text)
else:
pytest.warns(UserWarning, match=f"Could not fetch {url}: status_code: {resp.status_code}")
pytest.warns(
UserWarning,
match=f"Could not fetch {url}: status_code: {resp.status_code}",
)
return raws
class TestRegexs:
class TestRegexsOnline:
listFiles_urls = {
@ -42,7 +48,6 @@ class TestRegexs:
"wiki.othing.xyz-20230701": "https://wiki.othing.xyz/index.php?title=Special:ListFiles&sort=byname",
"mediawiki.org-20230701": "https://www.mediawiki.org/w/index.php?title=Special:ListFiles&sort=byname&limit=7",
"asoiaf.fandom.com-20230701": "https://asoiaf.fandom.com/zh/wiki/Special:文件列表?sort=byname&limit=7",
# only for local testing:
# "commons.moegirl.org.cn-20230701": "https://commons.moegirl.org.cn/index.php?title=Special:ListFiles&sort=byname&limit=7",
# # login required:
@ -50,6 +55,7 @@ class TestRegexs:
# "group1.mediawiki.demo.save-web.org_mediawiki-1.27.7-20230701": "http://group1.mediawiki.demo.save-web.org/mediawiki-1.27.7/index.php?title=Special:ListFiles&limit=2",
}
raws: Dict[str, str] = {}
def test_online(self):
if not ONLINE:
pytest.skip("Online test skipped")
@ -65,18 +71,21 @@ class TestRegexs:
best_matched = _count
regexp_best = regexp
assert regexp_best is not None, f"Could not find a proper regexp to parse the HTML for {url} (online)"
assert (
regexp_best is not None
), f"Could not find a proper regexp to parse the HTML for {url} (online)"
if "limit=" in url:
limit = int(url.split("limit=")[-1])
assert len(re.findall(regexp_best, raw)) == limit, f"Could not find {limit} matches for {url} (online)"
assert (
len(re.findall(regexp_best, raw)) == limit
), f"Could not find {limit} matches for {url} (online)"
class TestRegexsOffline:
html_files = os.listdir(HTML_DIR)
raws: Dict[str, str] = {}
for html_file in html_files:
with open(HTML_DIR / html_file, "r", encoding="utf-8") as f:
with open(HTML_DIR / html_file, encoding="utf-8") as f:
raws[html_file] = f.read()
assert len(raws) != 0, f"Could not find any HTML files in {HTML_DIR}"
@ -92,4 +101,6 @@ class TestRegexs:
best_matched = _count
regexp_best = regexp
assert regexp_best is not None, f"Could not find a proper regexp to parse the HTML for {site} (local)"
assert (
regexp_best is not None
), f"Could not find a proper regexp to parse the HTML for {site} (local)"

203
wikiteam3/dumpgenerator/dump/image/image.py
Unescape Escape View File

@ -1,29 +1,26 @@
import os
import random
import re
import sys
import time
import random
import urllib.parse
from typing import Dict, List, Optional
import requests
from wikiteam3.dumpgenerator.api import getJSON, handleStatusCode
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.dumpgenerator.dump.image.html_regexs import R_NEXT, REGEX_CANDIDATES
from wikiteam3.utils import domain2prefix
from wikiteam3.dumpgenerator.exceptions import PageMissingError, FileSizeError
from wikiteam3.dumpgenerator.api import getJSON
from wikiteam3.dumpgenerator.api import handleStatusCode
from wikiteam3.dumpgenerator.log import logerror
from wikiteam3.dumpgenerator.dump.page.xmlexport.page_xml import getXMLPage
from wikiteam3.utils import sha1File
from wikiteam3.utils import cleanHTML, undoHTMLEntities
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.dumpgenerator.exceptions import FileSizeError, PageMissingError
from wikiteam3.dumpgenerator.log import logerror
from wikiteam3.utils import cleanHTML, domain2prefix, sha1File, undoHTMLEntities
class Image:
@staticmethod
def getXMLFileDesc(config: Config=None, title="", session=None):
def getXMLFileDesc(config: Config = None, title="", session=None):
"""Get XML for image description page"""
config.curonly = 1 # tricky to get only the most recent desc
return "".join(
@ -36,7 +33,12 @@ class Image:
)
@staticmethod
def generateImageDump(config: Config=None, other: Dict=None, images: List[List]=None, session: requests.Session=None):
def generateImageDump(
config: Config = None,
other: Dict = None,
images: List[List] = None,
session: requests.Session = None,
):
"""Save files and descriptions using a file list\n
Deprecated: `start` is not used anymore."""
@ -50,52 +52,59 @@ class Image:
c_savedImageFiles = 0
c_savedImageDescs = 0
bypass_cdn_image_compression: bool = other["bypass_cdn_image_compression"]
def modify_params(params: Optional[Dict] = None) -> Dict:
""" bypass Cloudflare Polish (image optimization) """
"""bypass Cloudflare Polish (image optimization)"""
if params is None:
params = {}
if bypass_cdn_image_compression is True:
# bypass Cloudflare Polish (image optimization)
# <https://developers.cloudflare.com/images/polish/>
params["_wiki_t"] = int(time.time()*1000)
params["_wiki_t"] = int(time.time() * 1000)
params[f"_wiki_{random.randint(10,99)}_"] = "random"
return params
def check_response(r: requests.Response) -> None:
assert not r.headers.get("cf-polished", ""), "Found cf-polished header in response, use --bypass-cdn-image-compression to bypass it"
assert not r.headers.get(
"cf-polished", ""
), "Found cf-polished header in response, use --bypass-cdn-image-compression to bypass it"
for filename, url, uploader, size, sha1 in images:
toContinue = 0
# saving file
filename2 = urllib.parse.unquote(filename)
if len(filename2.encode('utf-8')) > other["filenamelimit"]:
if len(filename2.encode("utf-8")) > other["filenamelimit"]:
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text=f"Filename is too long(>240 bytes), skipping: '{filename2}'",
)
continue
filename3 = f"{imagepath}/{filename2}"
# check if file already exists and has the same size and sha1
if ((size != 'False'
if (
size != "False"
and os.path.isfile(filename3)
and os.path.getsize(filename3) == int(size)
and sha1File(filename3) == sha1)
or (sha1 == 'False' and os.path.isfile(filename3))):
# sha1 is 'False' if file not in original wiki (probably deleted,
# you will get a 404 error if you try to download it)
and sha1File(filename3) == sha1
) or (sha1 == "False" and os.path.isfile(filename3)):
# sha1 is 'False' if file not in original wiki (probably deleted,
# you will get a 404 error if you try to download it)
c_savedImageFiles += 1
toContinue += 1
print_msg=f" {c_savedImageFiles}|sha1 matched: {filename2}"
print_msg = f" {c_savedImageFiles}|sha1 matched: {filename2}"
print(print_msg[0:70], end="\r")
if sha1 == 'False':
logerror(config=config, to_stdout=True,
text=f"sha1 is 'False' for {filename2}, file may not in wiki site (probably deleted). "
+"we will not try to download it...")
if sha1 == "False":
logerror(
config=config,
to_stdout=True,
text=f"sha1 is 'False' for {filename2}, file may not in wiki site (probably deleted). "
+ "we will not try to download it...",
)
else:
Delay(config=config, session=session)
original_url = url
@ -119,12 +128,14 @@ class Image:
):
url = "https://" + original_url.split("://")[1]
# print 'Maybe a broken http to https redirect, trying ', url
r = session.get(url=url, params=modify_params(), allow_redirects=False)
r = session.get(
url=url, params=modify_params(), allow_redirects=False
)
check_response(r)
if r.status_code == 200:
try:
if size == 'False' or len(r.content) == int(size):
if size == "False" or len(r.content) == int(size):
# size == 'False' means size is unknown
with open(filename3, "wb") as imagefile:
imagefile.write(r.content)
@ -133,22 +144,25 @@ class Image:
raise FileSizeError(file=filename3, size=size)
except OSError:
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text=f"File '{filename3}' could not be created by OS",
)
except FileSizeError as e:
# TODO: add a --force-download-image or --nocheck-image-size option to download anyway
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text=f"File '{e.file}' size is not match '{e.size}', skipping",
)
else:
logerror(
config=config, to_stdout=True,
text=f"Failled to donwload '{filename2}' with URL '{url}' due to HTTP '{r.status_code}', skipping"
config=config,
to_stdout=True,
text=f"Failled to donwload '{filename2}' with URL '{url}' due to HTTP '{r.status_code}', skipping",
)
if os.path.isfile(filename3+".desc"):
if os.path.isfile(filename3 + ".desc"):
toContinue += 1
else:
Delay(config=config, session=session)
@ -173,7 +187,8 @@ class Image:
except PageMissingError:
xmlfiledesc = ""
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text='The image description page "%s" was missing in the wiki (probably deleted)'
% (str(title)),
)
@ -185,34 +200,44 @@ class Image:
xmlfiledesc = ""
# Fixup the XML
if xmlfiledesc != "" and not re.search(r"</mediawiki>", xmlfiledesc):
if xmlfiledesc != "" and not re.search(
r"</mediawiki>", xmlfiledesc
):
xmlfiledesc += "</mediawiki>"
with open(f"{imagepath}/{filename2}.desc", "w", encoding="utf-8") as f:
with open(
f"{imagepath}/{filename2}.desc", "w", encoding="utf-8"
) as f:
f.write(xmlfiledesc)
c_savedImageDescs += 1
if xmlfiledesc == "":
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text=f"Created empty .desc file: '{imagepath}/{filename2}.desc'",
)
except OSError:
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text=f"File {imagepath}/{filename2}.desc could not be created by OS",
)
if toContinue == 2: # skip printing
if toContinue == 2: # skip printing
continue
print_msg = f" | {(len(images)-c_savedImageFiles)}=>{filename2[0:50]}"
print(print_msg, " "*(73 - len(print_msg)), end="\r")
print_msg = (
f" | {(len(images)-c_savedImageFiles)}=>{filename2[0:50]}"
)
print(print_msg, " " * (73 - len(print_msg)), end="\r")
print(f"Downloaded {c_savedImageFiles} images and {c_savedImageDescs} .desc files.")
print(
f"Downloaded {c_savedImageFiles} images and {c_savedImageDescs} .desc files."
)
@staticmethod
def getImageNames(config: Config=None, session: requests.Session=None):
def getImageNames(config: Config = None, session: requests.Session = None):
"""Get list of image names"""
print(")Retrieving image filenames")
@ -232,7 +257,7 @@ class Image:
return images
@staticmethod
def getImageNamesScraper(config: Config=None, session: requests.Session=None):
def getImageNamesScraper(config: Config = None, session: requests.Session = None):
"""Retrieve file list: filename, url, uploader"""
images = []
@ -280,7 +305,9 @@ class Image:
if _count > best_matched:
best_matched = _count
regexp_best = regexp
assert regexp_best is not None, "Could not find a proper regexp to parse the HTML"
assert (
regexp_best is not None
), "Could not find a proper regexp to parse the HTML"
m = re.compile(regexp_best).finditer(raw)
# Iter the image results
@ -293,10 +320,15 @@ class Image:
uploader = re.sub("_", " ", i.group("uploader"))
uploader = undoHTMLEntities(text=uploader)
uploader = urllib.parse.unquote(uploader)
images.append([
filename, url, uploader,
'False', 'False' # size, sha1 not available
])
images.append(
[
filename,
url,
uploader,
"False",
"False", # size, sha1 not available
]
)
# print (filename, url)
if re.search(R_NEXT, raw):
@ -319,12 +351,12 @@ class Image:
return images
@staticmethod
def getImageNamesAPI(config: Config=None, session: requests.Session=None):
def getImageNamesAPI(config: Config = None, session: requests.Session = None):
"""Retrieve file list: filename, url, uploader, size, sha1"""
oldAPI = False
# # Commented by @yzqzss:
# https://www.mediawiki.org/wiki/API:Allpages
# API:Allpages requires MW >= 1.8
# API:Allpages requires MW >= 1.8
# (Note: The documentation says that it requires MediaWiki >= 1.18, but that's not true.)
# (Read the revision history of [[API:Allpages]] and the source code of MediaWiki, you will
# know that it's existed since MW 1.8) (2023-05-09)
@ -335,7 +367,10 @@ class Image:
images = []
countImages = 0
while aifrom:
print(f'Using API:Allimages to get the list of images, {len(images)} images found so far...', end='\r')
print(
f"Using API:Allimages to get the list of images, {len(images)} images found so far...",
end="\r",
)
params = {
"action": "query",
"list": "allimages",
@ -352,10 +387,10 @@ class Image:
if "query" in jsonimages:
countImages += len(jsonimages["query"]["allimages"])
# oldAPI = True
# break
# # uncomment to force use API:Allpages generator
# # uncomment to force use API:Allpages generator
# # may also can as a fallback if API:Allimages response is wrong
aifrom = ""
@ -372,7 +407,9 @@ class Image:
aifrom = jsonimages["continue"]["aicontinue"]
elif "aifrom" in jsonimages["continue"]:
aifrom = jsonimages["continue"]["aifrom"]
print(countImages, aifrom[0:30]+" "*(60-len(aifrom[0:30])),end="\r")
print(
countImages, aifrom[0:30] + " " * (60 - len(aifrom[0:30])), end="\r"
)
for image in jsonimages["query"]["allimages"]:
url = image["url"]
@ -384,9 +421,7 @@ class Image:
# unquote() no longer supports bytes-like strings
# so unicode may require the following workaround:
# https://izziswift.com/how-to-unquote-a-urlencoded-unicode-string-in-python/
if (
".wikia." in config.api or ".fandom.com" in config.api
):
if ".wikia." in config.api or ".fandom.com" in config.api:
filename = urllib.parse.unquote(
re.sub("_", " ", url.split("/")[-3])
)
@ -402,7 +437,7 @@ class Image:
)
uploader = re.sub("_", " ", image.get("user", "Unknown"))
size = image.get("size", "False")
# size or sha1 is not always available (e.g. https://wiki.mozilla.org/index.php?curid=20675)
sha1 = image.get("sha1", "False")
images.append([filename, url, uploader, size, sha1])
@ -411,7 +446,9 @@ class Image:
break
if oldAPI:
print(" API:Allimages not available. Using API:Allpages generator instead.")
print(
" API:Allimages not available. Using API:Allpages generator instead."
)
gapfrom = "!"
images = []
while gapfrom:
@ -425,8 +462,8 @@ class Image:
"action": "query",
"generator": "allpages",
"gapnamespace": 6,
"gaplimit": config.api_chunksize, # The value must be between 1 and 500.
# TODO: Is it OK to set it higher, for speed?
"gaplimit": config.api_chunksize, # The value must be between 1 and 500.
# TODO: Is it OK to set it higher, for speed?
"gapfrom": gapfrom,
"prop": "imageinfo",
"iiprop": "url|user|size|sha1",
@ -440,7 +477,11 @@ class Image:
if "query" in jsonimages:
countImages += len(jsonimages["query"]["pages"])
print(countImages, gapfrom[0:30]+" "*(60-len(gapfrom[0:30])),end="\r")
print(
countImages,
gapfrom[0:30] + " " * (60 - len(gapfrom[0:30])),
end="\r",
)
gapfrom = ""
@ -450,7 +491,7 @@ class Image:
and "gapcontinue" in jsonimages["continue"]
):
gapfrom = jsonimages["continue"]["gapcontinue"]
# legacy code, not sure if it's still needed by some old wikis
elif (
"query-continue" in jsonimages
@ -461,7 +502,6 @@ class Image:
"gapfrom"
]
# print (gapfrom)
# print (jsonimages['query'])
@ -488,23 +528,30 @@ class Image:
return images
@staticmethod
def saveImageNames(config: Config=None, images: List[List]=None, session=None):
def saveImageNames(config: Config = None, images: List[List] = None, session=None):
"""Save image list in a file, including filename, url, uploader, size and sha1"""
imagesfilename = "{}-{}-images.txt".format(
domain2prefix(config=config), config.date
)
imagesfile = open(
"{}/{}".format(config.path, imagesfilename), "w", encoding="utf-8"
)
imagesfile = open(f"{config.path}/{imagesfilename}", "w", encoding="utf-8")
for line in images:
while 3 <= len(line) < 5:
line.append("False") # At this point, make sure all lines have 5 elements
line.append(
"False"
) # At this point, make sure all lines have 5 elements
filename, url, uploader, size, sha1 = line
print(line,end='\r')
print(line, end="\r")
imagesfile.write(
filename + "\t" + url + "\t" + uploader
+ "\t" + str(size) + "\t" + str(sha1)
filename
+ "\t"
+ url
+ "\t"
+ uploader
+ "\t"
+ str(size)
+ "\t"
+ str(sha1)
# sha1 or size may be `False` if file is missing, so convert bool to str
+ "\n"
)
@ -514,7 +561,7 @@ class Image:
print("Image filenames and URLs saved at...", imagesfilename)
@staticmethod
def curateImageURL(config: Config=None, url=""):
def curateImageURL(config: Config = None, url=""):
"""Returns an absolute URL for an image, adding the domain if missing"""
if config.index:
@ -524,7 +571,7 @@ class Image:
+ "://"
+ config.index.split("://")[1].split("/")[0]
)
elif config.api:
elif config.api:
domainalone = (
config.api.split("://")[0]
+ "://"

5
wikiteam3/dumpgenerator/dump/misc/index_php.py
Unescape Escape View File

@ -1,10 +1,11 @@
import os
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.utils import removeIP
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.utils import removeIP
def saveIndexPHP(config: Config=None, session=None):
def saveIndexPHP(config: Config = None, session=None):
"""Save index.php as .html, to preserve license details available at the botom of the page"""
if os.path.exists("%s/index.html" % (config.path)):

4
wikiteam3/dumpgenerator/dump/misc/site_info.py
Unescape Escape View File

@ -1,12 +1,12 @@
import json
import os
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.dumpgenerator.api import getJSON
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.dumpgenerator.config import Config
def saveSiteInfo(config: Config=None, session=None):
def saveSiteInfo(config: Config = None, session=None):
"""Save a file with site info"""
if config.api:

7
wikiteam3/dumpgenerator/dump/misc/site_info_test.py
Unescape Escape View File

@ -7,10 +7,11 @@ from wikiteam3.dumpgenerator.test.test_config import get_config
from .site_info import saveSiteInfo
def test_mediawiki_1_16():
with get_config('1.16.5') as config:
with get_config("1.16.5") as config:
sess = requests.Session()
saveSiteInfo(config, sess)
with open(config.path + '/siteinfo.json', 'r') as f:
with open(config.path + "/siteinfo.json") as f:
siteInfoJson = json.load(f)
assert siteInfoJson['query']['general']['generator'] == "MediaWiki 1.16.5"
assert siteInfoJson["query"]["general"]["generator"] == "MediaWiki 1.16.5"

5
wikiteam3/dumpgenerator/dump/misc/special_logs.py
Unescape Escape View File

@ -1,7 +1,8 @@
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.dumpgenerator.config import Config
def saveLogs(config: Config=None, session=None):
def saveLogs(config: Config = None, session=None):
"""Save Special:Log"""
# get all logs from Special:Log
"""parse

5
wikiteam3/dumpgenerator/dump/misc/special_version.py
Unescape Escape View File

@ -1,11 +1,11 @@
import os
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.utils import removeIP
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.utils import removeIP
def saveSpecialVersion(config: Config=None, session=None):
def saveSpecialVersion(config: Config = None, session=None):
"""Save Special:Version as .html, to preserve extensions details"""
if os.path.exists("%s/SpecialVersion.html" % (config.path)):
@ -22,4 +22,3 @@ def saveSpecialVersion(config: Config=None, session=None):
"%s/SpecialVersion.html" % (config.path), "w", encoding="utf-8"
) as outfile:
outfile.write(str(raw))

9
wikiteam3/dumpgenerator/dump/page/xmlexport/page_xml.py
Unescape Escape View File

@ -1,10 +1,15 @@
from wikiteam3.dumpgenerator.config import Config
from .page_xml_api import getXMLPageWithApi
from .page_xml_export import getXMLPageWithExport
def getXMLPage(config: Config = None, title="", verbose=True, session=None):
if config.xmlapiexport:
return getXMLPageWithApi(config=config, title=title, verbose=verbose, session=session)
return getXMLPageWithApi(
config=config, title=title, verbose=verbose, session=session
)
else:
return getXMLPageWithExport(config=config, title=title, verbose=verbose, session=session)
return getXMLPageWithExport(
config=config, title=title, verbose=verbose, session=session
)

255
wikiteam3/dumpgenerator/dump/page/xmlexport/page_xml_api.py
Unescape Escape View File

@ -7,11 +7,11 @@ import requests
from wikiteam3.dumpgenerator.api import handleStatusCode
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.dumpgenerator.exceptions import PageMissingError, ExportAbortedError
from wikiteam3.dumpgenerator.exceptions import ExportAbortedError, PageMissingError
from wikiteam3.dumpgenerator.log import logerror
try:
import xml.etree.cElementTree as ET
import xml.etree.ElementTree as ET
except ImportError:
import xml.etree.ElementTree as ET
@ -19,114 +19,133 @@ import xml.dom.minidom as MD
def reconstructRevisions(root=None):
#print ET.tostring(rev)
page = ET.Element('stub')
# print ET.tostring(rev)
page = ET.Element("stub")
edits = 0
for rev in root.find('query').find('pages').find('page').find('revisions').findall('rev'):
for rev in (
root.find("query").find("pages").find("page").find("revisions").findall("rev")
):
try:
rev_ = ET.SubElement(page,'revision')
rev_ = ET.SubElement(page, "revision")
# id
ET.SubElement(rev_,'id').text = rev.attrib['revid']
ET.SubElement(rev_, "id").text = rev.attrib["revid"]
# parentid (optional, export-0.7+)
if 'parentid' in rev.attrib:
ET.SubElement(rev_,'parentid').text = rev.attrib['parentid']
if "parentid" in rev.attrib:
ET.SubElement(rev_, "parentid").text = rev.attrib["parentid"]
# timestamp
ET.SubElement(rev_,'timestamp').text = rev.attrib['timestamp']
ET.SubElement(rev_, "timestamp").text = rev.attrib["timestamp"]
# contributor
contributor = ET.SubElement(rev_,'contributor')
if 'userhidden' not in rev.attrib:
ET.SubElement(contributor,'username').text = rev.attrib['user']
ET.SubElement(contributor,'id').text = rev.attrib['userid']
contributor = ET.SubElement(rev_, "contributor")
if "userhidden" not in rev.attrib:
ET.SubElement(contributor, "username").text = rev.attrib["user"]
ET.SubElement(contributor, "id").text = rev.attrib["userid"]
else:
contributor.set('deleted','deleted')
contributor.set("deleted", "deleted")
# comment (optional)
if 'commenthidden' in rev.attrib:
print('commenthidden')
comment = ET.SubElement(rev_,'comment')
comment.set('deleted','deleted')
elif 'comment' in rev.attrib and rev.attrib['comment']: # '' is empty
comment = ET.SubElement(rev_,'comment')
comment.text = rev.attrib['comment']
if "commenthidden" in rev.attrib:
print("commenthidden")
comment = ET.SubElement(rev_, "comment")
comment.set("deleted", "deleted")
elif "comment" in rev.attrib and rev.attrib["comment"]: # '' is empty
comment = ET.SubElement(rev_, "comment")
comment.text = rev.attrib["comment"]
else:
# no comment or empty comment, do not create comment element
pass
# minor edit (optional)
if 'minor' in rev.attrib:
ET.SubElement(rev_,'minor')
if "minor" in rev.attrib:
ET.SubElement(rev_, "minor")
# model and format (optional, export-0.8+)
if 'contentmodel' in rev.attrib:
ET.SubElement(rev_,'model').text = rev.attrib['contentmodel'] # default: 'wikitext'
if 'contentformat' in rev.attrib:
ET.SubElement(rev_,'format').text = rev.attrib['contentformat'] # default: 'text/x-wiki'
if "contentmodel" in rev.attrib:
ET.SubElement(rev_, "model").text = rev.attrib[
"contentmodel"
] # default: 'wikitext'
if "contentformat" in rev.attrib:
ET.SubElement(rev_, "format").text = rev.attrib[
"contentformat"
] # default: 'text/x-wiki'
# text
text = ET.SubElement(rev_,'text')
if 'texthidden' not in rev.attrib:
text.attrib['xml:space'] = "preserve"
text.attrib['bytes'] = rev.attrib['size']
text = ET.SubElement(rev_, "text")
if "texthidden" not in rev.attrib:
text.attrib["xml:space"] = "preserve"
text.attrib["bytes"] = rev.attrib["size"]
text.text = rev.text
else:
# NOTE: this is not the same as the text being empty
text.set('deleted','deleted')
text.set("deleted", "deleted")
# sha1
if not 'sha1' in rev.attrib:
if 'sha1hidden' in rev.attrib:
ET.SubElement(rev_,'sha1') # stub
if not "sha1" in rev.attrib:
if "sha1hidden" in rev.attrib:
ET.SubElement(rev_, "sha1") # stub
else:
# The sha1 may not have been backfilled on older wikis or lack for other reasons (Wikia).
pass
elif 'sha1' in rev.attrib:
sha1 = ET.SubElement(rev_,'sha1')
sha1.text = rev.attrib['sha1']
elif "sha1" in rev.attrib:
sha1 = ET.SubElement(rev_, "sha1")
sha1.text = rev.attrib["sha1"]
edits += 1
except Exception as e:
#logerror(config=config, text='Error reconstructing revision, xml:%s' % (ET.tostring(rev)))
# logerror(config=config, text='Error reconstructing revision, xml:%s' % (ET.tostring(rev)))
print(ET.tostring(rev))
traceback.print_exc()
page = None
edits = 0
raise e
return page,edits
return page, edits
def getXMLPageCoreWithApi(headers: Dict=None, params: Dict=None, config: Config=None, session=None):
""" """
def getXMLPageCoreWithApi(
headers: Dict = None, params: Dict = None, config: Config = None, session=None
):
""" """
# just send the API request
# if it fails, it will reduce params['rvlimit']
xml = ''
xml = ""
c = 0
maxseconds = 100 # max seconds to wait in a single sleeping
maxretries = config.retries # x retries and skip
increment = 20 # increment every retry
while not re.search(r'</api>' if not config.curonly else r'</mediawiki>', xml) or re.search(r'</error>', xml):
while not re.search(
r"</api>" if not config.curonly else r"</mediawiki>", xml
) or re.search(r"</error>", xml):
if c > 0 and c < maxretries:
wait = increment * c < maxseconds and increment * \
c or maxseconds # incremental until maxseconds
print(' In attempt %d, XML for "%s" is wrong. Waiting %d seconds and reloading...' % (
c, params['titles' if config.xmlapiexport else 'pages'], wait))
wait = (
increment * c < maxseconds and increment * c or maxseconds
) # incremental until maxseconds
print(
' In attempt %d, XML for "%s" is wrong. Waiting %d seconds and reloading...'
% (c, params["titles" if config.xmlapiexport else "pages"], wait)
)
time.sleep(wait)
# reducing server load requesting smallest chunks (if curonly then
# rvlimit = 1 from mother function)
if params['rvlimit'] > 1:
params['rvlimit'] = params['rvlimit'] / 2 # half
if params["rvlimit"] > 1:
params["rvlimit"] = params["rvlimit"] / 2 # half
if c >= maxretries:
print(' We have retried %d times' % (c))
print(' MediaWiki error for "%s", network error or whatever...' % (
params['titles' if config.xmlapiexport else 'pages']))
print(" We have retried %d times" % (c))
print(
' MediaWiki error for "%s", network error or whatever...'
% (params["titles" if config.xmlapiexport else "pages"])
)
# If it's not already what we tried: our last chance, preserve only the last revision...
# config.curonly means that the whole dump is configured to save only the last,
# params['curonly'] should mean that we've already tried this
# fallback, because it's set by the following if and passed to
# getXMLPageCore
# TODO: save only the last version when failed
print(' Saving in the errors log, and skipping...')
print(" Saving in the errors log, and skipping...")
logerror(
config=config,
text='Error while retrieving the last revision of "%s". Skipping.' %
(params['titles' if config.xmlapiexport else 'pages']).decode('utf-8'))
text='Error while retrieving the last revision of "%s". Skipping.'
% (params["titles" if config.xmlapiexport else "pages"]).decode(
"utf-8"
),
)
raise ExportAbortedError(config.index)
return '' # empty xml
return "" # empty xml
# FIXME HANDLE HTTP Errors HERE
try:
@ -135,8 +154,8 @@ def getXMLPageCoreWithApi(headers: Dict=None, params: Dict=None, config: Config=
xml = r.text
# print xml
except requests.exceptions.ConnectionError as e:
print(' Connection error: %s' % (str(e.args[0])))
xml = ''
print(" Connection error: %s" % (str(e.args[0])))
xml = ""
except requests.exceptions.ReadTimeout as e:
print(" Read timeout: %s" % (str(e.args[0])))
xml = ""
@ -144,39 +163,45 @@ def getXMLPageCoreWithApi(headers: Dict=None, params: Dict=None, config: Config=
return xml
def getXMLPageWithApi(config: Config=None, title="", verbose=True, session=None):
""" Get the full history (or current only) of a page using API:Query
if params['curonly'] is set, then using export&exportwrap to export
def getXMLPageWithApi(config: Config = None, title="", verbose=True, session=None):
"""Get the full history (or current only) of a page using API:Query
if params['curonly'] is set, then using export&exportwrap to export
"""
title_ = title
title_ = re.sub(' ', '_', title_)
title_ = re.sub(" ", "_", title_)
# do not convert & into %26, title_ = re.sub('&', '%26', title_)
# action=query&rvlimit=50&format=xml&prop=revisions&titles=TITLE_HERE
# &rvprop=timestamp%7Cuser%7Ccomment%7Ccontent%7Cids%7Cuserid%7Csha1%7Csize
# print 'current:%s' % (title_)
if not config.curonly:
params = {'titles': title_, 'action': 'query', 'format': 'xml',
'prop': 'revisions',
'rvprop': # rvprop: <https://www.mediawiki.org/wiki/API:Revisions#Parameter_history>
'timestamp|user|comment|content|' # MW v????
'ids|flags|size|' # MW v1.11
'userid|' # MW v1.17
'sha1|' # MW v1.19
'contentmodel|' # MW v1.21
,
'rvcontinue': None,
'rvlimit': config.api_chunksize
}
params = {
"titles": title_,
"action": "query",
"format": "xml",
"prop": "revisions",
"rvprop": "timestamp|user|comment|content|" # rvprop: <https://www.mediawiki.org/wiki/API:Revisions#Parameter_history> # MW v????
"ids|flags|size|" # MW v1.11
"userid|" # MW v1.17
"sha1|" # MW v1.19
"contentmodel|", # MW v1.21
"rvcontinue": None,
"rvlimit": config.api_chunksize,
}
else:
params = {'titles': title_, 'action': 'query', 'format': 'xml', 'export': 1, 'exportnowrap': 1}
params = {
"titles": title_,
"action": "query",
"format": "xml",
"export": 1,
"exportnowrap": 1,
}
# print 'params:%s' % (params)
if not config.curonly:
firstpartok = False
lastcontinue = None
numberofedits = 0
ret = ''
ret = ""
continueKey: Optional[str] = None
while True:
# in case the last request is not right, saving last time's progress
@ -191,23 +216,23 @@ def getXMLPageWithApi(config: Config=None, title="", verbose=True, session=None)
# just return so that we can continue, and getXMLPageCoreWithApi will log the error
return
try:
root = ET.fromstring(xml.encode('utf-8'))
root = ET.fromstring(xml.encode("utf-8"))
except:
continue
try:
retpage = root.find('query').find('pages').find('page')
retpage = root.find("query").find("pages").find("page")
except:
continue
if 'missing' in retpage.attrib or 'invalid' in retpage.attrib:
print('Page not found')
raise PageMissingError(params['titles'], xml)
if "missing" in retpage.attrib or "invalid" in retpage.attrib:
print("Page not found")
raise PageMissingError(params["titles"], xml)
if not firstpartok:
try:
# build the firstpart by ourselves to improve the memory usage
ret = ' <page>\n'
ret += ' <title>%s</title>\n' % (retpage.attrib['title'])
ret += ' <ns>%s</ns>\n' % (retpage.attrib['ns'])
ret += ' <id>%s</id>\n' % (retpage.attrib['pageid'])
ret = " <page>\n"
ret += " <title>%s</title>\n" % (retpage.attrib["title"])
ret += " <ns>%s</ns>\n" % (retpage.attrib["ns"])
ret += " <id>%s</id>\n" % (retpage.attrib["pageid"])
except:
firstpartok = False
continue
@ -216,67 +241,69 @@ def getXMLPageWithApi(config: Config=None, title="", verbose=True, session=None)
yield ret
continueVal = None
if root.find('continue') is not None:
if root.find("continue") is not None:
# uses continue.rvcontinue
# MW 1.26+
continueKey = 'rvcontinue'
continueVal = root.find('continue').attrib['rvcontinue']
elif root.find('query-continue') is not None:
revContinue = root.find('query-continue').find('revisions')
continueKey = "rvcontinue"
continueVal = root.find("continue").attrib["rvcontinue"]
elif root.find("query-continue") is not None:
revContinue = root.find("query-continue").find("revisions")
assert revContinue is not None, "Should only have revisions continue"
if 'rvcontinue' in revContinue.attrib:
if "rvcontinue" in revContinue.attrib:
# MW 1.21 ~ 1.25
continueKey = 'rvcontinue'
continueVal = revContinue.attrib['rvcontinue']
elif 'rvstartid' in revContinue.attrib:
continueKey = "rvcontinue"
continueVal = revContinue.attrib["rvcontinue"]
elif "rvstartid" in revContinue.attrib:
# TODO: MW ????
continueKey = 'rvstartid'
continueVal = revContinue.attrib['rvstartid']
continueKey = "rvstartid"
continueVal = revContinue.attrib["rvstartid"]
else:
# blindly assume the first attribute is the continue key
# may never happen
assert len(revContinue.attrib) > 0, "Should have at least one attribute"
assert (
len(revContinue.attrib) > 0
), "Should have at least one attribute"
for continueKey in revContinue.attrib.keys():
continueVal = revContinue.attrib[continueKey]
break
if continueVal is not None:
params[continueKey] = continueVal
try:
ret = ''
ret = ""
edits = 0
# transform the revision
rev_, edits = reconstructRevisions(root=root)
xmldom = MD.parseString(b'<stub1>' + ET.tostring(rev_) + b'</stub1>')
xmldom = MD.parseString(b"<stub1>" + ET.tostring(rev_) + b"</stub1>")
# convert it into text in case it throws MemoryError
# delete the first three line and last two line,which is for setting the indent
ret += ''.join(xmldom.toprettyxml(indent=' ').splitlines(True)[3:-2])
ret += "".join(xmldom.toprettyxml(indent=" ").splitlines(True)[3:-2])
yield ret
numberofedits += edits
if config.curonly or continueVal is None: # no continue
break
except:
traceback.print_exc()
params['rvcontinue'] = lastcontinue
ret = ''
yield ' </page>\n'
params["rvcontinue"] = lastcontinue
ret = ""
yield " </page>\n"
else:
xml = getXMLPageCoreWithApi(params=params, config=config, session=session)
if xml == "":
raise ExportAbortedError(config.index)
if not "</page>" in xml:
raise PageMissingError(params['titles'], xml)
raise PageMissingError(params["titles"], xml)
else:
# strip these sha1s sums which keep showing up in the export and
# which are invalid for the XML schema (they only apply to
# revisions)
xml = re.sub(r'\n\s*<sha1>\w+</sha1>\s*\n', r'\n', xml)
xml = re.sub(r'\n\s*<sha1/>\s*\n', r'\n', xml)
xml = re.sub(r"\n\s*<sha1>\w+</sha1>\s*\n", r"\n", xml)
xml = re.sub(r"\n\s*<sha1/>\s*\n", r"\n", xml)
yield xml.split("</page>")[0]
# just for looking good :)
r_timestamp = r'<timestamp>([^<]+)</timestamp>'
r_timestamp = r"<timestamp>([^<]+)</timestamp>"
numberofedits = 0
numberofedits += len(re.findall(r_timestamp, xml))
@ -284,7 +311,7 @@ def getXMLPageWithApi(config: Config=None, title="", verbose=True, session=None)
yield "</page>\n"
if verbose:
if (numberofedits == 1):
print(' %s, 1 edit' % (title.strip()))
if numberofedits == 1:
print(" %s, 1 edit" % (title.strip()))
else:
print(' %s, %d edits' % (title.strip(), numberofedits))
print(" %s, %d edits" % (title.strip(), numberofedits))

18
wikiteam3/dumpgenerator/dump/page/xmlexport/page_xml_export.py
Unescape Escape View File

@ -1,18 +1,20 @@
from typing import *
import re
import sys
import time
from typing import *
import requests
from wikiteam3.dumpgenerator.exceptions import ExportAbortedError, PageMissingError
from wikiteam3.dumpgenerator.api import handleStatusCode
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.dumpgenerator.exceptions import ExportAbortedError, PageMissingError
from wikiteam3.dumpgenerator.log import logerror
from wikiteam3.utils import uprint
from wikiteam3.dumpgenerator.config import Config
def getXMLPageCore(headers: Dict=None, params: Dict=None, config: Config=None, session=None) -> str:
def getXMLPageCore(
headers: Dict = None, params: Dict = None, config: Config = None, session=None
) -> str:
""""""
# returns a XML containing params['limit'] revisions (or current only), ending in </mediawiki>
# if retrieving params['limit'] revisions fails, returns a current only version
@ -55,7 +57,8 @@ def getXMLPageCore(headers: Dict=None, params: Dict=None, config: Config=None, s
print(" Trying to save only the last revision for this page...")
params["curonly"] = 1
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text='Error while retrieving the full history of "%s". Trying to save only the last revision for this page'
% (params["pages"]),
)
@ -65,7 +68,8 @@ def getXMLPageCore(headers: Dict=None, params: Dict=None, config: Config=None, s
else:
print(" Saving in the errors log, and skipping...")
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text='Error while retrieving the last revision of "%s". Skipping.'
% (params["pages"]),
)
@ -89,7 +93,7 @@ def getXMLPageCore(headers: Dict=None, params: Dict=None, config: Config=None, s
return xml
def getXMLPageWithExport(config: Config=None, title="", verbose=True, session=None):
def getXMLPageWithExport(config: Config = None, title="", verbose=True, session=None):
"""Get the full history (or current only) of a page"""
# if server errors occurs while retrieving the full page history, it may return [oldest OK versions] + last version, excluding middle revisions, so it would be partialy truncated

170
wikiteam3/dumpgenerator/dump/page/xmlrev/xml_revisions.py
Unescape Escape View File

@ -1,36 +1,47 @@
from datetime import datetime
from typing import *
import sys
import time
from datetime import datetime
from typing import *
from urllib.parse import urlparse
import lxml.etree
import lxml.etree
import mwclient
import requests
from wikiteam3.dumpgenerator.exceptions import PageMissingError
from wikiteam3.dumpgenerator.log import logerror
from wikiteam3.dumpgenerator.api.namespaces import getNamespacesAPI
from wikiteam3.dumpgenerator.api.page_titles import readTitles
from wikiteam3.dumpgenerator.dump.page.xmlrev.xml_revisions_page import makeXmlFromPage, makeXmlPageFromRaw
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.dumpgenerator.dump.page.xmlrev.xml_revisions_page import (
makeXmlFromPage,
makeXmlPageFromRaw,
)
from wikiteam3.dumpgenerator.exceptions import PageMissingError
from wikiteam3.dumpgenerator.log import logerror
ALL_NAMESPACE = -1
def getXMLRevisionsByAllRevisions(config: Config=None, session=None, site: mwclient.Site=None, nscontinue=None, arvcontinue=None):
def getXMLRevisionsByAllRevisions(
config: Config = None,
session=None,
site: mwclient.Site = None,
nscontinue=None,
arvcontinue=None,
):
if "all" not in config.namespaces:
namespaces = config.namespaces
else:
# namespaces, namespacenames = getNamespacesAPI(config=config, session=session)
namespaces = [ALL_NAMESPACE] # magic number refers to "all"
namespaces = [ALL_NAMESPACE] # magic number refers to "all"
_nscontinue = nscontinue
_arvcontinue = arvcontinue
for namespace in namespaces:
# Skip retrived namespace
if namespace == ALL_NAMESPACE:
assert len(namespaces) == 1, \
"Only one item shoule be there when 'all' namespace are specified"
assert (
len(namespaces) == 1
), "Only one item shoule be there when 'all' namespace are specified"
_nscontinue = None
else:
if _nscontinue is not None:
@ -48,9 +59,9 @@ def getXMLRevisionsByAllRevisions(config: Config=None, session=None, site: mwcli
"arvdir": "newer",
}
if namespace != ALL_NAMESPACE:
arvparams['arvnamespace'] = namespace
arvparams["arvnamespace"] = namespace
if _arvcontinue is not None:
arvparams['arvcontinue'] = _arvcontinue
arvparams["arvcontinue"] = _arvcontinue
if not config.curonly:
# We have to build the XML manually...
@ -64,14 +75,9 @@ def getXMLRevisionsByAllRevisions(config: Config=None, session=None, site: mwcli
)
while True:
try:
arvrequest = site.api(
http_method=config.http_method, **arvparams
)
arvrequest = site.api(http_method=config.http_method, **arvparams)
except requests.exceptions.HTTPError as e:
if (
e.response.status_code == 405
and config.http_method == "POST"
):
if e.response.status_code == 405 and config.http_method == "POST":
print("POST request to the API failed, retrying with GET")
config.http_method = "GET"
continue
@ -91,7 +97,9 @@ def getXMLRevisionsByAllRevisions(config: Config=None, session=None, site: mwcli
e.response_text.startswith("<!DOCTYPE html>")
and config.http_method == "POST"
):
print("POST request to the API failed (got HTML), retrying with GET")
print(
"POST request to the API failed (got HTML), retrying with GET"
)
config.http_method = "GET"
continue
else:
@ -112,14 +120,9 @@ def getXMLRevisionsByAllRevisions(config: Config=None, session=None, site: mwcli
# We only need the revision ID, all the rest will come from the raw export
arvparams["arvprop"] = "ids"
try:
arvrequest = site.api(
http_method=config.http_method, **arvparams
)
arvrequest = site.api(http_method=config.http_method, **arvparams)
except requests.exceptions.HTTPError as e:
if (
e.response.status_code == 405
and config.http_method == "POST"
):
if e.response.status_code == 405 and config.http_method == "POST":
print("POST request to the API failed, retrying with GET")
config.http_method = "GET"
continue
@ -156,12 +159,10 @@ def getXMLRevisionsByAllRevisions(config: Config=None, session=None, site: mwcli
)
except requests.exceptions.HTTPError as e:
if (
e.response.status_code == 405
and config.http_method == "POST"
e.response.status_code == 405
and config.http_method == "POST"
):
print(
"POST request to the API failed, retrying with GET"
)
print("POST request to the API failed, retrying with GET")
config.http_method = "GET"
exportrequest = site.api(
http_method=config.http_method, **exportparams
@ -185,12 +186,10 @@ def getXMLRevisionsByAllRevisions(config: Config=None, session=None, site: mwcli
)
except requests.exceptions.HTTPError as e:
if (
e.response.status_code == 405
and config.http_method == "POST"
e.response.status_code == 405
and config.http_method == "POST"
):
print(
"POST request to the API failed, retrying with GET"
)
print("POST request to the API failed, retrying with GET")
config.http_method = "GET"
arvrequest = site.api(
http_method=config.http_method, **arvparams
@ -208,7 +207,9 @@ def getXMLRevisionsByAllRevisions(config: Config=None, session=None, site: mwcli
break
def getXMLRevisionsByTitles(config: Config=None, session=None, site: mwclient.Site=None, start=None):
def getXMLRevisionsByTitles(
config: Config = None, session=None, site: mwclient.Site = None, start=None
):
if config.curonly:
# The raw XML export in the API gets a title and gives the latest revision.
# We could also use the allpages API as generator but let's be consistent.
@ -226,14 +227,9 @@ def getXMLRevisionsByTitles(config: Config=None, session=None, site: mwclient.Si
"export": "1",
}
try:
exportrequest = site.api(
http_method=config.http_method, **exportparams
)
exportrequest = site.api(http_method=config.http_method, **exportparams)
except requests.exceptions.HTTPError as e:
if (
e.response.status_code == 405
and config.http_method == "POST"
):
if e.response.status_code == 405 and config.http_method == "POST":
print("POST request to the API failed, retrying with GET")
config.http_method = "GET"
exportrequest = site.api(
@ -271,28 +267,24 @@ def getXMLRevisionsByTitles(config: Config=None, session=None, site: mwclient.Si
"action": "query",
"titles": "|".join(titlelist),
"prop": "revisions",
'rvlimit': config.api_chunksize,
"rvlimit": config.api_chunksize,
"rvprop": "ids|timestamp|user|userid|size|sha1|contentmodel|comment|content|flags",
}
try:
prequest = site.api(http_method=config.http_method, **pparams)
except requests.exceptions.HTTPError as e:
if (
e.response.status_code == 405
and config.http_method == "POST"
):
if e.response.status_code == 405 and config.http_method == "POST":
print("POST request to the API failed, retrying with GET")
config.http_method = "GET"
prequest = site.api(
http_method=config.http_method, **pparams
)
prequest = site.api(http_method=config.http_method, **pparams)
else:
raise
except mwclient.errors.InvalidResponse:
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text="Error: page inaccessible? Could not export page: %s"
% ("; ".join(titlelist)),
% ("; ".join(titlelist)),
)
continue
@ -305,9 +297,10 @@ def getXMLRevisionsByTitles(config: Config=None, session=None, site: mwclient.Si
pages = prequest["query"]["pages"]
except KeyError:
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text="Error: page inaccessible? Could not export page: %s"
% ("; ".join(titlelist)),
% ("; ".join(titlelist)),
)
break
# Go through the data we got to build the XML.
@ -317,9 +310,10 @@ def getXMLRevisionsByTitles(config: Config=None, session=None, site: mwclient.Si
yield xml
except PageMissingError:
logerror(
config=config, to_stdout=True,
config=config,
to_stdout=True,
text="Error: empty revision from API. Could not export page: %s"
% ("; ".join(titlelist)),
% ("; ".join(titlelist)),
)
continue
@ -335,19 +329,12 @@ def getXMLRevisionsByTitles(config: Config=None, session=None, site: mwclient.Si
break
try:
prequest = site.api(
http_method=config.http_method, **pparams
)
prequest = site.api(http_method=config.http_method, **pparams)
except requests.exceptions.HTTPError as e:
if (
e.response.status_code == 405
and config.http_method == "POST"
):
if e.response.status_code == 405 and config.http_method == "POST":
print("POST request to the API failed, retrying with GET")
config.http_method = "GET"
prequest = site.api(
http_method=config.http_method, **pparams
)
prequest = site.api(http_method=config.http_method, **pparams)
# We're done iterating for this title or titles.
c += len(titlelist)
@ -357,30 +344,40 @@ def getXMLRevisionsByTitles(config: Config=None, session=None, site: mwclient.Si
print(f"\n-> Downloaded {c} pages\n")
def getXMLRevisions(config: Config=None, session=None, useAllrevision=True, lastPage=None):
def getXMLRevisions(
config: Config = None, session=None, useAllrevision=True, lastPage=None
):
# FIXME: actually figure out the various strategies for each MediaWiki version
apiurl = urlparse(config.api)
# FIXME: force the protocol we asked for! Or don't verify SSL if we asked HTTP?
# https://github.com/WikiTeam/wikiteam/issues/358
site = mwclient.Site(
apiurl.netloc, apiurl.path.replace("api.php", ""), scheme=apiurl.scheme, pool=session
apiurl.netloc,
apiurl.path.replace("api.php", ""),
scheme=apiurl.scheme,
pool=session,
)
if useAllrevision:
# Find last title
if lastPage is not None:
try:
lastNs = int(lastPage.find('ns').text)
lastNs = int(lastPage.find("ns").text)
if False:
lastRevision = lastPage.find('revision')
lastTimestamp = lastRevision.find('timestamp').text
lastRevid = int(lastRevision.find('id').text)
lastDatetime = datetime.fromisoformat(lastTimestamp.rstrip('Z'))
lastArvcontinue = lastDatetime.strftime("%Y%m%d%H%M%S") + '|' + str(lastRevid)
lastRevision = lastPage.find("revision")
lastTimestamp = lastRevision.find("timestamp").text
lastRevid = int(lastRevision.find("id").text)
lastDatetime = datetime.fromisoformat(lastTimestamp.rstrip("Z"))
lastArvcontinue = (
lastDatetime.strftime("%Y%m%d%H%M%S") + "|" + str(lastRevid)
)
else:
lastArvcontinue = lastPage.attrib['arvcontinue']
lastArvcontinue = lastPage.attrib["arvcontinue"]
except Exception:
print("Failed to find title in last trunk XML: %s" % (lxml.etree.tostring(lastPage)))
print(
"Failed to find title in last trunk XML: %s"
% (lxml.etree.tostring(lastPage))
)
raise
nscontinue = lastNs
arvcontinue = lastArvcontinue
@ -391,19 +388,26 @@ def getXMLRevisions(config: Config=None, session=None, useAllrevision=True, last
arvcontinue = None
try:
return getXMLRevisionsByAllRevisions(config, session, site, nscontinue, arvcontinue)
return getXMLRevisionsByAllRevisions(
config, session, site, nscontinue, arvcontinue
)
except (KeyError, mwclient.errors.InvalidResponse) as e:
print(e)
# TODO: check whether the KeyError was really for a missing arv API
print("Warning. Could not use allrevisions. Wiki too old? Try to use --xmlrevisions_page")
print(
"Warning. Could not use allrevisions. Wiki too old? Try to use --xmlrevisions_page"
)
sys.exit()
else:
# Find last title
if lastPage is not None:
try:
start = lastPage.find('title')
start = lastPage.find("title")
except Exception:
print("Failed to find title in last trunk XML: %s" % (lxml.etree.tostring(lastPage)))
print(
"Failed to find title in last trunk XML: %s"
% (lxml.etree.tostring(lastPage))
)
raise
else:
start = None

68
wikiteam3/dumpgenerator/dump/page/xmlrev/xml_revisions_page.py
Unescape Escape View File

@ -3,13 +3,14 @@ from lxml.builder import E
from wikiteam3.dumpgenerator.exceptions import PageMissingError
def makeXmlPageFromRaw(xml, arvcontinue) -> str:
"""Discard the metadata around a <page> element in <mediawiki> string"""
root = etree.XML(xml)
find = etree.XPath("//*[local-name() = 'page']")
page = find(root)[0]
if arvcontinue is not None:
page.attrib['arvcontinue'] = arvcontinue
page.attrib["arvcontinue"] = arvcontinue
# The tag will inherit the namespace, like:
# <page xmlns="http://www.mediawiki.org/xml/export-0.10/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
# FIXME: pretty_print doesn't seem to work, only adds a newline
@ -25,7 +26,7 @@ def makeXmlFromPage(page: dict, arvcontinue) -> str:
E.id(str(page["pageid"])),
)
if arvcontinue is not None:
p.attrib['arvcontinue'] = arvcontinue
p.attrib["arvcontinue"] = arvcontinue
for rev in page["revisions"]:
# Older releases like MediaWiki 1.16 do not return all fields.
if "userid" in rev:
@ -38,26 +39,43 @@ def makeXmlFromPage(page: dict, arvcontinue) -> str:
size = 0
# Create rev object
revision = [E.id(str(rev["revid"])),
E.timestamp(rev["timestamp"]),]
revision = [
E.id(str(rev["revid"])),
E.timestamp(rev["timestamp"]),
]
# The text, user, comment, sha1 may be deleted/suppressed
if (('texthidden' in rev) or ('textmissing' in rev)):
print("Warning: text missing/hidden in pageid %d revid %d" % (page['pageid'], rev['revid']))
revision.append(E.text(**{
'bytes': str(size),
'deleted': 'deleted',
}))
if ("texthidden" in rev) or ("textmissing" in rev):
print(
"Warning: text missing/hidden in pageid %d revid %d"
% (page["pageid"], rev["revid"])
)
revision.append(
E.text(
**{
"bytes": str(size),
"deleted": "deleted",
}
)
)
else:
text = str(rev["*"])
revision.append(E.text(text, **{
'bytes': str(size),
'{http://www.w3.org/XML/1998/namespace}space': 'preserve',
}))
revision.append(
E.text(
text,
**{
"bytes": str(size),
"{http://www.w3.org/XML/1998/namespace}space": "preserve",
}
)
)
if not "user" in rev:
if not "userhidden" in rev:
print("Warning: user not hidden but missing user in pageid %d revid %d" % (page['pageid'], rev['revid']))
print(
"Warning: user not hidden but missing user in pageid %d revid %d"
% (page["pageid"], rev["revid"])
)
revision.append(E.contributor(deleted="deleted"))
else:
revision.append(
@ -69,15 +87,14 @@ def makeXmlFromPage(page: dict, arvcontinue) -> str:
if not "sha1" in rev:
if "sha1hidden" in rev:
revision.append(E.sha1()) # stub
revision.append(E.sha1()) # stub
else:
# The sha1 may not have been backfilled on older wikis or lack for other reasons (Wikia).
pass
elif "sha1" in rev:
revision.append(E.sha1(rev["sha1"]))
if 'commenthidden' in rev:
if "commenthidden" in rev:
revision.append(E.comment(deleted="deleted"))
elif "comment" in rev and rev["comment"]:
revision.append(E.comment(str(rev["comment"])))
@ -94,7 +111,19 @@ def makeXmlFromPage(page: dict, arvcontinue) -> str:
revision.append(E.minor())
# mwcli's dump.xml order
revisionTags = ['id', 'parentid', 'timestamp', 'contributor', 'minor', 'comment', 'origin', 'model', 'format', 'text', 'sha1']
revisionTags = [
"id",
"parentid",
"timestamp",
"contributor",
"minor",
"comment",
"origin",
"model",
"format",
"text",
"sha1",
]
revisionElementsDict = {elem.tag: elem for elem in revision}
_revision = E.revision()
for tag in revisionTags:
@ -107,4 +136,3 @@ def makeXmlFromPage(page: dict, arvcontinue) -> str:
print(e)
raise PageMissingError(page["title"], e)
return etree.tostring(p, pretty_print=True, encoding="unicode")

72
wikiteam3/dumpgenerator/dump/xmldump/xml_dump.py
Unescape Escape View File

@ -4,25 +4,39 @@ from typing import *
import lxml.etree
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.utils import domain2prefix
from wikiteam3.dumpgenerator.exceptions import PageMissingError
from wikiteam3.dumpgenerator.log import logerror
from wikiteam3.dumpgenerator.api.page_titles import readTitles
from wikiteam3.dumpgenerator.dump.page.xmlexport.page_xml import getXMLPage
from wikiteam3.dumpgenerator.cli import Delay
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.utils import cleanXML, undoHTMLEntities
from wikiteam3.dumpgenerator.dump.xmldump.xml_header import getXMLHeader
from wikiteam3.dumpgenerator.dump.page.xmlexport.page_xml import getXMLPage
from wikiteam3.dumpgenerator.dump.page.xmlrev.xml_revisions import getXMLRevisions
from wikiteam3.dumpgenerator.dump.xmldump.xml_truncate import truncateXMLDump, parseLastPageChunk
from wikiteam3.dumpgenerator.dump.xmldump.xml_header import getXMLHeader
from wikiteam3.dumpgenerator.dump.xmldump.xml_truncate import (
parseLastPageChunk,
truncateXMLDump,
)
from wikiteam3.dumpgenerator.exceptions import PageMissingError
from wikiteam3.dumpgenerator.log import logerror
from wikiteam3.utils import cleanXML, domain2prefix, undoHTMLEntities
def doXMLRevisionDump(config: Config=None, session=None, xmlfile=None, lastPage=None, useAllrevisions=False):
def doXMLRevisionDump(
config: Config = None,
session=None,
xmlfile=None,
lastPage=None,
useAllrevisions=False,
):
try:
r_timestamp = "<timestamp>([^<]+)</timestamp>"
r_arvcontinue = '<page arvcontinue="(.*?)">'
lastArvcontinue = None
for xml in getXMLRevisions(config=config, session=session, lastPage=lastPage, useAllrevision=useAllrevisions):
for xml in getXMLRevisions(
config=config,
session=session,
lastPage=lastPage,
useAllrevision=useAllrevisions,
):
numrevs = len(re.findall(r_timestamp, xml))
arvcontinueRe = re.findall(r_arvcontinue, xml)
if arvcontinueRe:
@ -36,7 +50,7 @@ def doXMLRevisionDump(config: Config=None, session=None, xmlfile=None, lastPage=
xmltitle = re.search(r"<title>([^<]+)</title>", xml)
title = undoHTMLEntities(text=xmltitle.group(1))
print(f'{title}, {numrevs} edits (--xmlrevisions)')
print(f"{title}, {numrevs} edits (--xmlrevisions)")
# Delay(config=config, session=session)
except AttributeError as e:
print(e)
@ -45,18 +59,20 @@ def doXMLRevisionDump(config: Config=None, session=None, xmlfile=None, lastPage=
except UnicodeEncodeError as e:
print(e)
def doXMLExportDump(config: Config=None, session=None, xmlfile=None, lastPage=None):
print(
'\nRetrieving the XML for every page\n'
)
def doXMLExportDump(config: Config = None, session=None, xmlfile=None, lastPage=None):
print("\nRetrieving the XML for every page\n")
lock = True
start = None
if lastPage is not None:
try:
start = lastPage.find('title').text
start = lastPage.find("title").text
except Exception:
print("Failed to find title in last trunk XML: %s" % (lxml.etree.tostring(lastPage)))
print(
"Failed to find title in last trunk XML: %s"
% (lxml.etree.tostring(lastPage))
)
raise
else:
# requested complete xml dump
@ -79,9 +95,9 @@ def doXMLExportDump(config: Config=None, session=None, xmlfile=None, lastPage=No
xmlfile.write(xml)
except PageMissingError:
logerror(
config=config, to_stdout=True,
text='The page "%s" was missing in the wiki (probably deleted)'
% title,
config=config,
to_stdout=True,
text='The page "%s" was missing in the wiki (probably deleted)' % title,
)
# here, XML is a correct <page> </page> chunk or
# an empty string due to a deleted page (logged in errors log) or
@ -90,7 +106,7 @@ def doXMLExportDump(config: Config=None, session=None, xmlfile=None, lastPage=No
c += 1
def generateXMLDump(config: Config=None, resume=False, session=None):
def generateXMLDump(config: Config = None, resume=False, session=None):
"""Generates a XML dump for a list of titles or from revision IDs"""
header, config = getXMLHeader(config=config, session=session)
@ -106,11 +122,9 @@ def generateXMLDump(config: Config=None, resume=False, session=None):
lastPageChunk = None
# start != None, means we are resuming a XML dump
if resume:
print(
"Removing the last chunk of past XML dump: it is probably incomplete."
)
print("Removing the last chunk of past XML dump: it is probably incomplete.")
# truncate XML dump if it already exists
lastPageChunk = truncateXMLDump("{}/{}".format(config.path, xmlfilename))
lastPageChunk = truncateXMLDump(f"{config.path}/{xmlfilename}")
if not lastPageChunk.strip():
print("Last page chunk is NULL, we'll directly start a new dump!")
resume = False
@ -123,14 +137,10 @@ def generateXMLDump(config: Config=None, resume=False, session=None):
sys.exit(1)
print(f"WARNING: will try to start the download...")
xmlfile = open(
"{}/{}".format(config.path, xmlfilename), "a", encoding="utf-8"
)
xmlfile = open(f"{config.path}/{xmlfilename}", "a", encoding="utf-8")
else:
print("\nRetrieving the XML for every page from the beginning\n")
xmlfile = open(
"{}/{}".format(config.path, xmlfilename), "w", encoding="utf-8"
)
xmlfile = open(f"{config.path}/{xmlfilename}", "w", encoding="utf-8")
xmlfile.write(header)
if config.xmlrevisions and not config.xmlrevisions_page:

11
wikiteam3/dumpgenerator/dump/xmldump/xml_header.py
Unescape Escape View File

@ -5,12 +5,13 @@ from typing import *
import requests
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.dumpgenerator.dump.page.xmlexport.page_xml import getXMLPage
from wikiteam3.dumpgenerator.exceptions import ExportAbortedError, PageMissingError
from wikiteam3.dumpgenerator.log import logerror
from wikiteam3.dumpgenerator.dump.page.xmlexport.page_xml import getXMLPage
from wikiteam3.dumpgenerator.config import Config
def getXMLHeader(config: Config=None, session=None) -> Tuple[str, Config]:
def getXMLHeader(config: Config = None, session=None) -> Tuple[str, Config]:
"""Retrieve a random page to extract XML headers (namespace info, etc)"""
# get the header of a random page, to attach it in the complete XML backup
# similar to: <mediawiki xmlns="http://www.mediawiki.org/xml/export-0.3/"
@ -125,6 +126,8 @@ def getXMLHeader(config: Config=None, session=None) -> Tuple[str, Config]:
else:
print(xml)
print("XML export on this wiki is broken, quitting.")
logerror(to_stdout=True, text="XML export on this wiki is broken, quitting.")
logerror(
to_stdout=True, text="XML export on this wiki is broken, quitting."
)
sys.exit()
return header, config

6
wikiteam3/dumpgenerator/dump/xmldump/xml_integrity.py
Unescape Escape View File

@ -1,7 +1,11 @@
from typing import *
from wikiteam3.dumpgenerator.config import Config
def checkXMLIntegrity(config: Config=None, titles: Iterable[str]=None, session=None):
def checkXMLIntegrity(
config: Config = None, titles: Iterable[str] = None, session=None
):
"""Check XML dump integrity, to detect broken XML chunks"""
# TODO: Fix XML Integrity Check
return

9
wikiteam3/dumpgenerator/dump/xmldump/xml_truncate.py
Unescape Escape View File

@ -1,6 +1,6 @@
import os
from io import StringIO
from typing import *
import os
import lxml.etree
from file_read_backwards import FileReadBackwards
@ -56,15 +56,14 @@ def truncateXMLDump(filename: str) -> str:
if endsWithNewlines(filename) == 0:
addNewline(filename)
elif endsWithNewlines(filename) > 1:
print(
f"WARNING: {filename} has {endsWithNewlines(filename)} newlines"
)
print(f"WARNING: {filename} has {endsWithNewlines(filename)} newlines")
return incomplete_segment
def parseLastPageChunk(chunk) -> Optional[lxml.etree._ElementTree]:
try:
parser = lxml.etree.XMLParser(recover=True)
tree = lxml.etree.parse(StringIO(chunk), parser)
return tree.getroot()
except lxml.etree.LxmlError:
return None
return None

4
wikiteam3/dumpgenerator/exceptions.py
Unescape Escape View File

@ -21,7 +21,7 @@ class FileSizeError(Exception):
self.size = size
def __str__(self):
return "File '%s' size is not match '%s'." % (self.file, self.size)
return f"File '{self.file}' size is not match '{self.size}'."
class FileSha1Error(Exception):
@ -30,4 +30,4 @@ class FileSha1Error(Exception):
self.sha1 = sha1
def __str__(self):
return "File '%s' sha1 is not match '%s'." % (self.file, self.sha1)
return f"File '{self.file}' sha1 is not match '{self.sha1}'."

3
wikiteam3/dumpgenerator/log/log_error.py
Unescape Escape View File

@ -2,7 +2,8 @@ import datetime
from wikiteam3.dumpgenerator.config import Config
def logerror(config: Config=None,to_stdout=False , text="") -> None:
def logerror(config: Config = None, to_stdout=False, text="") -> None:
"""Log error in errors.log"""
if text:
with open("%s/errors.log" % (config.path), "a", encoding="utf-8") as outfile:

50
wikiteam3/dumpgenerator/test/data/html_regexs/archiveteam.org-20230701.html
Unescape Escape View File

@ -30,10 +30,10 @@
</div>
<h1 id="firstHeading" class="firstHeading" >File list</h1>
<div id="bodyContent" class="vector-body">
<div id="contentSub"></div>
<div id="contentSub2"></div>
<div id="jump-to-nav"></div>
<a class="mw-jump-link" href="#mw-head">Jump to navigation</a>
<a class="mw-jump-link" href="#searchInput">Jump to search</a>
@ -126,30 +126,30 @@
<div id="mw-navigation">
<h2>Navigation menu</h2>
<div id="mw-head">
<nav id="p-personal" class="mw-portlet mw-portlet-personal vector-user-menu-legacy vector-menu" aria-labelledby="p-personal-label" role="navigation"
<nav id="p-personal" class="mw-portlet mw-portlet-personal vector-user-menu-legacy vector-menu" aria-labelledby="p-personal-label" role="navigation"
>
<h3 id="p-personal-label" class="vector-menu-heading"> <span>Personal tools</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="pt-createaccount" class="mw-list-item"><a href="/index.php?title=Special:CreateAccount&amp;returnto=Special%3AListFiles&amp;returntoquery=sort%3Dbyname%26limit%3D7" title="You are encouraged to create an account and log in; however, it is not mandatory">Create account</a></li><li id="pt-login" class="mw-list-item"><a href="/index.php?title=Special:UserLogin&amp;returnto=Special%3AListFiles&amp;returntoquery=sort%3Dbyname%26limit%3D7" title="You are encouraged to log in; however, it is not mandatory [o]" accesskey="o">Log in</a></li></ul>
</div>
</nav>
<div id="left-navigation">
<nav id="p-namespaces" class="mw-portlet mw-portlet-namespaces vector-menu vector-menu-tabs" aria-labelledby="p-namespaces-label" role="navigation"
<nav id="p-namespaces" class="mw-portlet mw-portlet-namespaces vector-menu vector-menu-tabs" aria-labelledby="p-namespaces-label" role="navigation"
>
<h3 id="p-namespaces-label" class="vector-menu-heading"> <span>Namespaces</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="ca-nstab-special" class="selected mw-list-item"><a href="/index.php?title=Special:ListFiles&amp;sort=byname&amp;limit=7" title="This is a special page, and it cannot be edited">Special page</a></li></ul>
</div>
</nav>
<nav id="p-variants" class="mw-portlet mw-portlet-variants emptyPortlet vector-menu-dropdown-noicon vector-menu vector-menu-dropdown" aria-labelledby="p-variants-label" role="navigation"
<nav id="p-variants" class="mw-portlet mw-portlet-variants emptyPortlet vector-menu-dropdown-noicon vector-menu vector-menu-dropdown" aria-labelledby="p-variants-label" role="navigation"
>
<input type="checkbox"
id="p-variants-checkbox"
@ -162,22 +162,22 @@
<span class="vector-menu-checkbox-collapsed">collapsed</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"></ul>
</div>
</nav>
</div>
<div id="right-navigation">
<nav id="p-views" class="mw-portlet mw-portlet-views emptyPortlet vector-menu vector-menu-tabs" aria-labelledby="p-views-label" role="navigation"
<nav id="p-views" class="mw-portlet mw-portlet-views emptyPortlet vector-menu vector-menu-tabs" aria-labelledby="p-views-label" role="navigation"
>
<h3 id="p-views-label" class="vector-menu-heading"> <span>Views</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"></ul>
</div>
</nav>
@ -194,9 +194,9 @@
<span class="vector-menu-checkbox-collapsed">collapsed</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"></ul>
</div>
</nav>
@ -218,35 +218,35 @@
</div>
</div>
<div id="mw-panel">
<div id="p-logo" role="banner">
<a class="mw-wiki-logo" href="/index.php/Main_Page"
title="Visit the main page"></a>
</div>
<nav id="p-navigation" class="mw-portlet mw-portlet-navigation vector-menu vector-menu-portal portal" aria-labelledby="p-navigation-label" role="navigation"
<nav id="p-navigation" class="mw-portlet mw-portlet-navigation vector-menu vector-menu-portal portal" aria-labelledby="p-navigation-label" role="navigation"
>
<h3 id="p-navigation-label" class="vector-menu-heading"> <span>Navigation</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="n-mainpage-description" class="mw-list-item"><a href="/index.php/Main_Page" title="Visit the main page [z]" accesskey="z">Main page</a></li><li id="n-recentchanges" class="mw-list-item"><a href="/index.php/Special:RecentChanges" title="A list of recent changes in the wiki [r]" accesskey="r">Recent changes</a></li><li id="n-randompage" class="mw-list-item"><a href="/index.php/Special:Random" title="Load a random page [x]" accesskey="x">Random page</a></li><li id="n-help-mediawiki" class="mw-list-item"><a href="https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents">Help about MediaWiki</a></li></ul>
</div>
</nav>
<nav id="p-tb" class="mw-portlet mw-portlet-tb vector-menu vector-menu-portal portal" aria-labelledby="p-tb-label" role="navigation"
<nav id="p-tb" class="mw-portlet mw-portlet-tb vector-menu vector-menu-portal portal" aria-labelledby="p-tb-label" role="navigation"
>
<h3 id="p-tb-label" class="vector-menu-heading"> <span>Tools</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="t-specialpages" class="mw-list-item"><a href="/index.php/Special:SpecialPages" title="A list of all special pages [q]" accesskey="q">Special pages</a></li><li id="t-print" class="mw-list-item"><a href="javascript:print();" rel="alternate" title="Printable version of this page [p]" accesskey="p">Printable version</a></li></ul>
</div>
</nav>
</div>
</div>
@ -268,4 +268,4 @@
<script>(RLQ=window.RLQ||[]).push(function(){mw.config.set({"wgBackendResponseTime":63});});</script>
</body></html>
</body></html>

2496
wikiteam3/dumpgenerator/test/data/html_regexs/asoiaf.fandom.com-20230701.html
View File

File diff suppressed because it is too large Load Diff

2
wikiteam3/dumpgenerator/test/data/html_regexs/commons.moegirl.org.cn-20230701.html
Unescape Escape View File

@ -28,7 +28,7 @@ function gtag() { dataLayer.push(arguments); }
gtag('js', new Date());
gtag('config', 'UA-173135727-1');
</script>
<script async src="https://fundingchoicesmessages.google.com/i/pub-1548272863545777?ers=1" nonce="pZjwXwYB8HZbw7pyXfFl9Q"></script><script nonce="pZjwXwYB8HZbw7pyXfFl9Q">(function() {function signalGooglefcPresent() {if (!window.frames['googlefcPresent']) {if (document.body) {const iframe = document.createElement('iframe'); iframe.style = 'width: 0; height: 0; border: none; z-index: -1000; left: -1000px; top: -1000px;'; iframe.style.display = 'none'; iframe.name = 'googlefcPresent'; document.body.appendChild(iframe);} else {setTimeout(signalGooglefcPresent, 0);}}}signalGooglefcPresent();})();</script>
<script>(function(){/*
Copyright The Closure Library Authors.

2
wikiteam3/dumpgenerator/test/data/html_regexs/group0.mediawiki.demo.save-web.org_mediawiki-1.16.5-20230701.html
Unescape Escape View File

@ -65,7 +65,7 @@ wgRestrictionMove=[];
<div id="globalWrapper">
<div id="column-content"><div id="content" >
<a id="top"></a>
<h1 id="firstHeading" class="firstHeading">文件列表</h1>
<div id="bodyContent">
<h3 id="siteSub">出自1165哈😂</h3>

2
wikiteam3/dumpgenerator/test/data/html_regexs/group2.mediawiki.demo.save-web.org_mediawiki-1.39.1-20230701.html
View File

File diff suppressed because one or more lines are too long

162
wikiteam3/dumpgenerator/test/data/html_regexs/mediawiki.org-20230701.html
Unescape Escape View File

@ -37,7 +37,7 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<header class="vector-header mw-header">
<div class="vector-header-start">
<nav class="vector-main-menu-landmark" aria-label="Site" role="navigation">
<div id="vector-main-menu-dropdown" class="vector-menu vector-dropdown vector-menu-dropdown vector-main-menu-dropdown vector-button-flush-left vector-button-flush-right" >
<input type="checkbox"
id="vector-main-menu-dropdown-checkbox"
@ -45,16 +45,16 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
aria-haspopup="true"
data-event-name="ui.dropdown-vector-main-menu-dropdown"
class="vector-dropdown-checkbox vector-menu-checkbox "
aria-label="Main menu"
/>
<label
id="vector-main-menu-dropdown-label"
for="vector-main-menu-dropdown-checkbox"
class="vector-dropdown-label vector-menu-heading cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only "
aria-hidden="true"
>
<span class="vector-icon mw-ui-icon-menu mw-ui-icon-wikimedia-menu"></span>
@ -65,7 +65,7 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<div id="vector-main-menu-unpinned-container" class="vector-unpinned-container">
<div id="vector-main-menu" class="vector-main-menu vector-pinnable-element">
<div
class="vector-pinnable-header vector-main-menu-pinnable-header vector-pinnable-header-unpinned"
@ -79,28 +79,28 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-unpin-button" data-event-name="pinnable-header.vector-main-menu.unpin">hide</button>
</div>
<div id="p-navigation" class="vector-menu mw-portlet mw-portlet-navigation" >
<div class="vector-menu-heading">
Navigation
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="n-mainpage-description" class="mw-list-item"><a href="/wiki/MediaWiki" title="Visit the main page [z]" accesskey="z"><span>Main page</span></a></li><li id="n-mw-download" class="mw-list-item"><a href="/wiki/Download"><span>Get MediaWiki</span></a></li><li id="n-mw-extensions" class="mw-list-item"><a href="/wiki/Special:MyLanguage/Category:Extensions"><span>Get extensions</span></a></li><li id="n-blog-text" class="mw-list-item"><a href="//techblog.wikimedia.org/"><span>Tech blog</span></a></li><li id="n-mw-contribute" class="mw-list-item"><a href="/wiki/Special:MyLanguage/How_to_contribute"><span>Contribute</span></a></li></ul>
</div>
</div>
<div id="p-support" class="vector-menu mw-portlet mw-portlet-support" >
<div class="vector-menu-heading">
Support
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="n-help" class="mw-list-item"><a href="/wiki/Special:MyLanguage/Help:Contents" title="The place to find out"><span>User help</span></a></li><li id="n-mw-faq" class="mw-list-item"><a href="/wiki/Special:MyLanguage/Manual:FAQ"><span>FAQ</span></a></li><li id="n-mw-manual" class="mw-list-item"><a href="/wiki/Special:MyLanguage/Manual:Contents"><span>Technical manual</span></a></li><li id="n-mw-supportdesk" class="mw-list-item"><a href="/wiki/Project:Support_desk"><span>Support desk</span></a></li><li id="n-mw-communication" class="mw-list-item"><a href="/wiki/Special:MyLanguage/Communication"><span>Communication</span></a></li></ul>
</div>
</div>
@ -109,9 +109,9 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
Development
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="n-mw-developerportal" class="mw-list-item"><a href="https://developer.wikimedia.org/"><span>Developer portal</span></a></li><li id="n-svn-statistics" class="mw-list-item"><a href="/wiki/Development_statistics"><span>Code statistics</span></a></li></ul>
</div>
</div>
@ -120,25 +120,25 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
MediaWiki.org
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="n-portal" class="mw-list-item"><a href="/wiki/Project:Help" title="About the project, what you can do, where to find things"><span>Community portal</span></a></li><li id="n-recentchanges" class="mw-list-item"><a href="/wiki/Special:RecentChanges" title="A list of recent changes in the wiki [r]" accesskey="r"><span>Recent changes</span></a></li><li id="n-mw-translate" class="mw-list-item"><a href="/wiki/Special:LanguageStats"><span>Translate content</span></a></li><li id="n-randompage" class="mw-list-item"><a href="/wiki/Special:Random" title="Load a random page [x]" accesskey="x"><span>Random page</span></a></li><li id="n-mw-discussion" class="mw-list-item"><a href="/wiki/Project:Village_Pump"><span>Village pump</span></a></li><li id="n-Sandboxlink-portlet-label" class="mw-list-item"><a href="/wiki/Project:Sandbox"><span>Sandbox</span></a></li></ul>
</div>
</div>
<div id="vector-main-menu" class="vector-menu " >
<div class="vector-menu-heading">
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"></ul>
</div>
</div>
</div>
</div>
@ -146,7 +146,7 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
</div>
</div>
</nav>
<a href="/wiki/MediaWiki" class="mw-logo">
<img class="mw-logo-icon" src="/static/images/icons/mediawikiwiki.svg" alt=""
aria-hidden="true" height="50" width="50">
@ -158,17 +158,17 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
</div>
<div class="vector-header-end">
<div id="p-search" role="search" class="vector-search-box-vue vector-search-box-collapses vector-search-box-show-thumbnail vector-search-box-auto-expand-width vector-search-box">
<a href="/wiki/Special:Search"
id=""
class="cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only search-toggle"
title="Search MediaWiki [f]"accesskey="f"><span class="vector-icon mw-ui-icon-search mw-ui-icon-wikimedia-search"></span>
<span>Search</span>
</a>
<div class="vector-typeahead-search-container">
<div class="cdx-typeahead-search cdx-typeahead-search--show-thumbnail cdx-typeahead-search--auto-expand-width">
<form action="/w/index.php" id="searchform" class="cdx-search-input cdx-search-input--has-end-button">
@ -177,7 +177,7 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<input
class="cdx-text-input__input"
type="search" name="search" placeholder="Search MediaWiki" aria-label="Search MediaWiki" autocapitalize="sentences" title="Search MediaWiki [f]" accesskey="f" id="searchInput"
>
<span class="cdx-text-input__icon cdx-text-input__start-icon"></span>
</div>
@ -190,16 +190,16 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
</div>
<nav class="vector-user-links" aria-label="Personal tools" role="navigation" >
<div id="p-vector-user-menu-overflow" class="vector-menu mw-portlet mw-portlet-vector-user-menu-overflow" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="ca-uls" class="user-links-collapsible-item mw-list-item active"><a href="#" class="uls-trigger cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet"><span class="vector-icon mw-ui-icon-wikimedia-language mw-ui-icon-wikimedia-wikimedia-language"></span> <span>English</span></a></li><li id="pt-createaccount-2" class="user-links-collapsible-item mw-list-item"><a href="/w/index.php?title=Special:CreateAccount&amp;returnto=Special%3AListFiles&amp;returntoquery=limit%3D7%26sort%3Dbyname" title="You are encouraged to create an account and log in; however, it is not mandatory"><span>Create account</span></a></li><li id="pt-login-2" class="user-links-collapsible-item mw-list-item"><a href="/w/index.php?title=Special:UserLogin&amp;returnto=Special%3AListFiles&amp;returntoquery=limit%3D7%26sort%3Dbyname" title="You are encouraged to log in; however, it is not mandatory [o]" accesskey="o"><span>Log in</span></a></li></ul>
</div>
</div>
<div id="vector-user-links-dropdown" class="vector-menu vector-dropdown vector-menu-dropdown vector-user-menu vector-button-flush-right vector-user-menu-logged-out" title="More options" >
<input type="checkbox"
id="vector-user-links-dropdown-checkbox"
@ -207,16 +207,16 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
aria-haspopup="true"
data-event-name="ui.dropdown-vector-user-links-dropdown"
class="vector-dropdown-checkbox vector-menu-checkbox "
aria-label="Personal tools"
/>
<label
id="vector-user-links-dropdown-label"
for="vector-user-links-dropdown-checkbox"
class="vector-dropdown-label vector-menu-heading cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only "
aria-hidden="true"
>
<span class="vector-icon mw-ui-icon-ellipsis mw-ui-icon-wikimedia-ellipsis"></span>
@ -226,12 +226,12 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<div class="vector-menu-content vector-dropdown-content">
<div id="p-personal" class="vector-menu mw-portlet mw-portlet-personal user-links-collapsible-item" title="User menu" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="pt-createaccount" class="user-links-collapsible-item mw-list-item"><a href="/w/index.php?title=Special:CreateAccount&amp;returnto=Special%3AListFiles&amp;returntoquery=limit%3D7%26sort%3Dbyname" title="You are encouraged to create an account and log in; however, it is not mandatory"><span class="vector-icon mw-ui-icon-userAdd mw-ui-icon-wikimedia-userAdd"></span> <span>Create account</span></a></li><li id="pt-login" class="user-links-collapsible-item mw-list-item"><a href="/w/index.php?title=Special:UserLogin&amp;returnto=Special%3AListFiles&amp;returntoquery=limit%3D7%26sort%3Dbyname" title="You are encouraged to log in; however, it is not mandatory [o]" accesskey="o"><span class="vector-icon mw-ui-icon-logIn mw-ui-icon-wikimedia-logIn"></span> <span>Log in</span></a></li></ul>
</div>
</div>
@ -240,13 +240,13 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
Pages for logged out editors <a href="/wiki/Help:Introduction" aria-label="Learn more about editing"><span>learn more</span></a>
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="pt-anoncontribs" class="mw-list-item"><a href="/wiki/Special:MyContributions" title="A list of edits made from this IP address [y]" accesskey="y"><span>Contributions</span></a></li><li id="pt-anontalk" class="mw-list-item"><a href="/wiki/Special:MyTalk" title="Discussion about edits from this IP address [n]" accesskey="n"><span>Talk</span></a></li></ul>
</div>
</div>
</div>
</div>
</nav>
@ -260,7 +260,7 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<div id="mw-navigation">
<nav id="mw-panel" class="vector-main-menu-landmark" aria-label="Site" role="navigation">
<div id="vector-main-menu-pinned-container" class="vector-pinned-container">
</div>
</nav>
</div>
@ -272,8 +272,8 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<main id="content" class="mw-body" role="main">
<header class="mw-body-header vector-page-titlebar">
<h1 id="firstHeading" class="firstHeading mw-first-heading">File list</h1>
<div class="mw-indicators">
<div id="mw-indicator-mw-helplink" class="mw-indicator"><a href="https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Managing_files" target="_blank" class="mw-helplink">Help</a></div>
</div>
@ -282,18 +282,18 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<div class="vector-page-toolbar-container">
<div id="left-navigation">
<nav aria-label="Namespaces">
<div id="p-associated-pages" class="vector-menu vector-menu-tabs mw-portlet mw-portlet-associated-pages emptyPortlet" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
</ul>
</div>
</div>
<div id="p-variants" class="vector-menu vector-dropdown vector-menu-dropdown mw-portlet mw-portlet-variants emptyPortlet" >
<input type="checkbox"
@ -303,25 +303,25 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
data-event-name="ui.dropdown-p-variants"
class="vector-dropdown-checkbox vector-menu-checkbox"
aria-label="Change language variant"
/>
<label
id="p-variants-label"
for="p-variants-checkbox"
class="vector-dropdown-label vector-menu-heading "
aria-hidden="true"
>
<span class="vector-dropdown-label-text vector-menu-heading-label">English</span>
</label>
<div class="vector-menu-content vector-dropdown-content">
<div class="vector-menu-content">
<ul class="vector-menu-content-list"></ul>
</div>
</div>
@ -330,21 +330,21 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
</div>
<div id="right-navigation" class="vector-collapsible">
<nav aria-label="Views">
<div id="p-views" class="vector-menu vector-menu-tabs mw-portlet mw-portlet-views emptyPortlet" >
<div class="vector-menu-content">
<ul class="vector-menu-content-list">
</ul>
</div>
</div>
</nav>
<nav class="vector-page-tools-landmark" aria-label="More options">
<div id="vector-page-tools-dropdown" class="vector-menu vector-dropdown vector-menu-dropdown vector-page-tools-dropdown" >
<input type="checkbox"
id="vector-page-tools-dropdown-checkbox"
@ -352,25 +352,25 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
aria-haspopup="true"
data-event-name="ui.dropdown-vector-page-tools-dropdown"
class="vector-dropdown-checkbox vector-menu-checkbox "
aria-label="Tools"
/>
<label
id="vector-page-tools-dropdown-label"
for="vector-page-tools-dropdown-checkbox"
class="vector-dropdown-label vector-menu-heading cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet"
aria-hidden="true"
>
<span class="vector-dropdown-label-text vector-menu-heading-label">Tools</span>
</label>
<div class="vector-menu-content vector-dropdown-content">
<div id="vector-page-tools-unpinned-container" class="vector-unpinned-container">
<div id="vector-page-tools" class="vector-page-tools vector-pinnable-element">
<div
class="vector-pinnable-header vector-page-tools-pinnable-header vector-pinnable-header-unpinned"
@ -384,15 +384,15 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<button class="vector-pinnable-header-toggle-button vector-pinnable-header-unpin-button" data-event-name="pinnable-header.vector-page-tools.unpin">hide</button>
</div>
<div id="p-cactions" class="vector-menu mw-portlet mw-portlet-cactions emptyPortlet" title="More options" >
<div class="vector-menu-heading">
Actions
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"></ul>
</div>
</div>
@ -401,16 +401,16 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
General
</div>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="t-upload" class="mw-list-item"><a href="//commons.wikimedia.org/wiki/Special:UploadWizard" title="Upload files [u]" accesskey="u"><span>Upload file</span></a></li><li id="t-specialpages" class="mw-list-item"><a href="/wiki/Special:SpecialPages" title="A list of all special pages [q]" accesskey="q"><span>Special pages</span></a></li><li id="t-print" class="mw-list-item"><a href="javascript:print();" rel="alternate" title="Printable version of this page [p]" accesskey="p"><span>Printable version</span></a></li></ul>
</div>
</div>
</div>
</div>
</div>
</div>
</nav>
@ -420,18 +420,18 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<div class="vector-column-end">
<nav class="vector-page-tools-landmark vector-sticky-pinned-container" aria-label="More options">
<div id="vector-page-tools-pinned-container" class="vector-pinned-container">
</div>
</nav>
</div>
<div id="bodyContent" class="vector-body" aria-labelledby="firstHeading" data-mw-ve-target-container>
<div class="vector-body-before-content">
</div>
<div id="contentSub"><div id="mw-content-subtitle"></div></div>
<div id="mw-content-text" class="mw-body-content"><div class="mw-specialpage-summary">
<p>This special page shows all uploaded files.
</p>
@ -509,10 +509,10 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
<div id="catlinks" class="catlinks catlinks-allhidden" data-mw="interface"></div>
</div>
</main>
</div>
<div class="mw-footer-container">
<footer id="footer" class="mw-footer" role="contentinfo" >
<ul id="footer-info">
</ul>
@ -536,10 +536,10 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
</footer>
</div>
</div>
</div>
</div>
</div>
<div class='vector-settings'>
<button
id=""
class="cdx-button cdx-button--icon-only vector-limited-width-toggle"
@ -552,4 +552,4 @@ false,"wgULSPosition":"personal","wgULSisCompactLinksEnabled":true,"wgULSisLangu
</div>
<script>(RLQ=window.RLQ||[]).push(function(){mw.config.set({"wgHostname":"mw-web.codfw.main-58544bbf85-6ztnc","wgBackendResponseTime":101,"wgPageParseReport":{"limitreport":{"cputime":"0.001","walltime":"0.001","ppvisitednodes":{"value":2,"limit":1000000},"postexpandincludesize":{"value":0,"limit":2097152},"templateargumentsize":{"value":0,"limit":2097152},"expansiondepth":{"value":1,"limit":100},"expensivefunctioncount":{"value":0,"limit":500},"unstrip-depth":{"value":0,"limit":20},"unstrip-size":{"value":0,"limit":5000000},"entityaccesscount":{"value":0,"limit":400},"timingprofile":["100.00% 0.000 1 -total"]},"cachereport":{"origin":"mw-web.codfw.main-58544bbf85-6ztnc","timestamp":"20230701154923","ttl":1814400,"transientcontent":false}}});});</script>
</body>
</html>
</html>

38
wikiteam3/dumpgenerator/test/data/html_regexs/wiki.othing.xyz-20230701.html
Unescape Escape View File

@ -39,10 +39,10 @@
</div>
<h1 id="firstHeading" class="firstHeading" >文件列表</h1>
<div id="bodyContent" class="mw-body-content">
<div id="contentSub"></div>
<div id="contentSub2"></div>
<div id="jump-to-nav"></div>
<a class="mw-jump-link" href="#mw-head">跳到导航</a>
<a class="mw-jump-link" href="#searchInput">跳到搜索</a>
@ -154,32 +154,32 @@
<h2>导航菜单</h2>
<div id="mw-head">
<!-- Please do not use role attribute as CSS selector, it is deprecated. -->
<nav id="p-personal" class="mw-portlet mw-portlet-personal vector-menu" aria-labelledby="p-personal-label" role="navigation"
<nav id="p-personal" class="mw-portlet mw-portlet-personal vector-menu" aria-labelledby="p-personal-label" role="navigation"
>
<h3 id="p-personal-label" class="vector-menu-heading">
<span>个人工具</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="pt-createaccount"><a href="/index.php?title=Special:%E5%88%9B%E5%BB%BA%E8%B4%A6%E6%88%B7&amp;returnto=Special%3A%E6%96%87%E4%BB%B6%E5%88%97%E8%A1%A8&amp;returntoquery=sort%3Dbyname" title="建议您创建一个账户并登录，不过这不是强制的">创建账户</a></li><li id="pt-login"><a href="/index.php?title=Special:%E7%94%A8%E6%88%B7%E7%99%BB%E5%BD%95&amp;returnto=Special%3A%E6%96%87%E4%BB%B6%E5%88%97%E8%A1%A8&amp;returntoquery=sort%3Dbyname" title="我们建议您登录，不过这不是强制的[o]" accesskey="o">登录</a></li></ul>
</div>
</nav>
<div id="left-navigation">
<!-- Please do not use role attribute as CSS selector, it is deprecated. -->
<nav id="p-namespaces" class="mw-portlet mw-portlet-namespaces vector-menu vector-menu-tabs" aria-labelledby="p-namespaces-label" role="navigation"
<nav id="p-namespaces" class="mw-portlet mw-portlet-namespaces vector-menu vector-menu-tabs" aria-labelledby="p-namespaces-label" role="navigation"
>
<h3 id="p-namespaces-label" class="vector-menu-heading">
<span>名字空间</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="ca-nstab-special" class="selected"><a href="/index.php?title=Special:%E6%96%87%E4%BB%B6%E5%88%97%E8%A1%A8&amp;sort=byname" title="这是特殊页面，并且它不能被编辑">特殊页面</a></li></ul>
</div>
</nav>
<!-- Please do not use role attribute as CSS selector, it is deprecated. -->
<nav id="p-variants" class="mw-portlet mw-portlet-variants emptyPortlet vector-menu vector-menu-dropdown" aria-labelledby="p-variants-label" role="navigation"
<nav id="p-variants" class="mw-portlet mw-portlet-variants emptyPortlet vector-menu vector-menu-dropdown" aria-labelledby="p-variants-label" role="navigation"
>
<input type="checkbox" class="vector-menu-checkbox" aria-labelledby="p-variants-label" />
<h3 id="p-variants-label" class="vector-menu-heading">
@ -187,26 +187,26 @@
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"></ul>
</div>
</nav>
</div>
<div id="right-navigation">
<!-- Please do not use role attribute as CSS selector, it is deprecated. -->
<nav id="p-views" class="mw-portlet mw-portlet-views emptyPortlet vector-menu vector-menu-tabs" aria-labelledby="p-views-label" role="navigation"
<nav id="p-views" class="mw-portlet mw-portlet-views emptyPortlet vector-menu vector-menu-tabs" aria-labelledby="p-views-label" role="navigation"
>
<h3 id="p-views-label" class="vector-menu-heading">
<span>视图</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"></ul>
</div>
</nav>
<!-- Please do not use role attribute as CSS selector, it is deprecated. -->
<nav id="p-cactions" class="mw-portlet mw-portlet-cactions emptyPortlet vector-menu vector-menu-dropdown" aria-labelledby="p-cactions-label" role="navigation"
<nav id="p-cactions" class="mw-portlet mw-portlet-cactions emptyPortlet vector-menu vector-menu-dropdown" aria-labelledby="p-cactions-label" role="navigation"
>
<input type="checkbox" class="vector-menu-checkbox" aria-labelledby="p-cactions-label" />
<h3 id="p-cactions-label" class="vector-menu-heading">
@ -214,7 +214,7 @@
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"></ul>
</div>
</nav>
@ -234,37 +234,37 @@
</div>
</div>
<div id="mw-panel">
<div id="p-logo" role="banner">
<a class="mw-wiki-logo" href="/wiki/%E9%A6%96%E9%A1%B5"
title="访问首页"></a>
</div>
<!-- Please do not use role attribute as CSS selector, it is deprecated. -->
<nav id="p-navigation" class="mw-portlet mw-portlet-navigation vector-menu vector-menu-portal portal" aria-labelledby="p-navigation-label" role="navigation"
<nav id="p-navigation" class="mw-portlet mw-portlet-navigation vector-menu vector-menu-portal portal" aria-labelledby="p-navigation-label" role="navigation"
>
<h3 id="p-navigation-label" class="vector-menu-heading">
<span>导航</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="n-mainpage-description"><a href="/wiki/%E9%A6%96%E9%A1%B5" title="访问首页[z]" accesskey="z">首页</a></li><li id="n-recentchanges"><a href="/wiki/Special:%E6%9C%80%E8%BF%91%E6%9B%B4%E6%94%B9" title="本wiki最近更改的列表[r]" accesskey="r">最近更改</a></li><li id="n-randompage"><a href="/wiki/Special:%E9%9A%8F%E6%9C%BA%E9%A1%B5%E9%9D%A2" title="载入一个随机页面[x]" accesskey="x">随机页面</a></li><li id="n-help-mediawiki"><a href="https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents">MediaWiki帮助</a></li></ul>
</div>
</nav>
<!-- Please do not use role attribute as CSS selector, it is deprecated. -->
<nav id="p-tb" class="mw-portlet mw-portlet-tb vector-menu vector-menu-portal portal" aria-labelledby="p-tb-label" role="navigation"
<nav id="p-tb" class="mw-portlet mw-portlet-tb vector-menu vector-menu-portal portal" aria-labelledby="p-tb-label" role="navigation"
>
<h3 id="p-tb-label" class="vector-menu-heading">
<span>工具</span>
</h3>
<div class="vector-menu-content">
<ul class="vector-menu-content-list"><li id="t-specialpages"><a href="/wiki/Special:%E7%89%B9%E6%AE%8A%E9%A1%B5%E9%9D%A2" title="所有特殊页面的列表[q]" accesskey="q">特殊页面</a></li><li id="t-print"><a href="javascript:print();" rel="alternate" title="本页面的可打印版本[p]" accesskey="p">打印版本</a></li></ul>
</div>
</nav>
</div>
</div>
@ -287,4 +287,4 @@
<script>(RLQ=window.RLQ||[]).push(function(){mw.config.set({"wgBackendResponseTime":698});});</script>
</body></html>
</body></html>

17
wikiteam3/dumpgenerator/test/test_config.py
Unescape Escape View File

@ -7,24 +7,29 @@ from wikiteam3.dumpgenerator.config import newConfig
CONFIG_CACHE = {}
@contextmanager
def _new_config_from_parameter(params):
_params = tuple(params)
if _params in CONFIG_CACHE:
return CONFIG_CACHE[_params]
config, _ = getParameters(['--path=.', '--xml'] + list(params))
config, _ = getParameters(["--path=.", "--xml"] + list(params))
CONFIG_CACHE[_params] = config
_config = newConfig(copy.deepcopy(config.asdict()))
try:
with tempfile.TemporaryDirectory(prefix='wikiteam3test_') as tmpdir:
with tempfile.TemporaryDirectory(prefix="wikiteam3test_") as tmpdir:
_config.path = tmpdir
yield _config
finally:
pass
def get_config(mediawiki_ver, api=True):
assert api == True
if mediawiki_ver == '1.16.5':
return _new_config_from_parameter([
"--api", "http://group0.mediawiki.demo.save-web.org/mediawiki-1.16.5/api.php",
])
if mediawiki_ver == "1.16.5":
return _new_config_from_parameter(
[
"--api",
"http://group0.mediawiki.demo.save-web.org/mediawiki-1.16.5/api.php",
]
)

21
wikiteam3/launcher.py
Unescape Escape View File

@ -34,7 +34,7 @@ def main():
parser.add_argument("wikispath")
parser.add_argument("--7z-path", dest="path7z", metavar="path-to-7z")
parser.add_argument("--generator-arg", "-g", dest="generator_args", action='append')
parser.add_argument("--generator-arg", "-g", dest="generator_args", action="append")
args = parser.parse_args()
@ -42,7 +42,7 @@ def main():
# None -> literal '7z', which will find the executable in PATH when running subprocesses
# otherwise -> resolve as path relative to current dir, then make absolute because we will change working dir later
path7z = str(Path(".", args.path7z).absolute()) if args.path7z is not None else '7z'
path7z = str(Path(".", args.path7z).absolute()) if args.path7z is not None else "7z"
generator_args = args.generator_args if args.generator_args is not None else []
@ -77,7 +77,12 @@ def main():
)
# Get the archive's file list.
if (sys.version_info[0] == 3) and (sys.version_info[1] > 0):
archivecontent = subprocess.check_output([path7z, "l", zipfilename, "-scsUTF-8"], text=True, encoding="UTF-8", errors="strict")
archivecontent = subprocess.check_output(
[path7z, "l", zipfilename, "-scsUTF-8"],
text=True,
encoding="UTF-8",
errors="strict",
)
if re.search(r"%s.+-history\.xml" % (prefix), archivecontent) is None:
# We should perhaps not create an archive in this case, but we continue anyway.
print("ERROR: The archive contains no history!")
@ -86,9 +91,7 @@ def main():
"WARNING: The archive doesn't contain SpecialVersion.html, this may indicate that download didn't finish."
)
else:
print(
"WARNING: Content of the archive not checked, we need 3.1+."
)
print("WARNING: Content of the archive not checked, we need 3.1+.")
# TODO: Find a way like grep -q below without doing a 7z l multiple times?
continue
@ -121,7 +124,8 @@ def main():
"--images",
"--resume",
f"--path={wikidir}",
] + generator_args,
]
+ generator_args,
shell=False,
env=subenv,
)
@ -134,7 +138,8 @@ def main():
f"--api={wiki}",
"--xml",
"--images",
] + generator_args,
]
+ generator_args,
shell=False,
env=subenv,
)

38
wikiteam3/uploader.py
Unescape Escape View File

@ -26,11 +26,12 @@ import urllib.parse
from io import BytesIO
from pathlib import Path
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.utils import getUserAgent, domain2prefix
import requests
from internetarchive import get_item
from wikiteam3.dumpgenerator.config import Config
from wikiteam3.utils import domain2prefix, getUserAgent
# Nothing to change below
convertlang = {
"ar": "Arabic",
@ -50,6 +51,7 @@ convertlang = {
def log(logfile, wiki, dump, msg):
logfile.write(f"\n{wiki};{dump.name};{msg}")
def read_ia_keys(config):
with open(config.keysfile) as f:
key_lines = f.readlines()
@ -57,10 +59,8 @@ def read_ia_keys(config):
accesskey = key_lines[0].strip()
secretkey = key_lines[1].strip()
return {
"access": accesskey,
"secret": secretkey
}
return {"access": accesskey, "secret": secretkey}
# We have to use md5 because the internet archive API doesn't provide
# sha1 for all files.
@ -80,6 +80,7 @@ def file_md5(path):
return digest.hexdigest()
def upload(wikis, logfile, config={}, uploadeddumps=[]):
ia_keys = read_ia_keys(config)
@ -99,7 +100,9 @@ def upload(wikis, logfile, config={}, uploadeddumps=[]):
wikiname = prefix.split("-")[0]
dumps = []
for f in dumpdir.iterdir():
if f.name.startswith("%s-" % (wikiname)) and (f.name.endswith("-wikidump.7z") or f.name.endswith("-history.xml.7z")):
if f.name.startswith("%s-" % (wikiname)) and (
f.name.endswith("-wikidump.7z") or f.name.endswith("-history.xml.7z")
):
print("%s found" % f)
dumps.append(f)
# Re-introduce the break here if you only need to upload one file
@ -113,7 +116,7 @@ def upload(wikis, logfile, config={}, uploadeddumps=[]):
for dump in dumps:
wikidate = dump.name.split("-")[1]
if first_item_exists and config.append_date and not config.admin:
identifier = "wiki-" + wikiname + "-" + wikidate
identifier = "wiki-" + wikiname + "-" + wikidate
item = get_item(identifier)
if dump.name in uploadeddumps:
if config.prune_directories:
@ -187,7 +190,11 @@ def upload(wikis, logfile, config={}, uploadeddumps=[]):
# Convert protocol-relative URLs
baseurl = re.sub("^//", "https://", baseurl)
if lang:
lang = (convertlang[lang.lower()] if (lang.lower() in convertlang) else lang.lower())
lang = (
convertlang[lang.lower()]
if (lang.lower() in convertlang)
else lang.lower()
)
# now copyright info from API
params = {
@ -326,21 +333,24 @@ def upload(wikis, logfile, config={}, uploadeddumps=[]):
retry = 20
while not item.exists and retry > 0:
retry -= 1
print('Waitting for item "%s" to be created... (%s)' % (identifier, retry))
print(
'Waitting for item "%s" to be created... (%s)'
% (identifier, retry)
)
time.sleep(10)
item = get_item(identifier)
# Update metadata
r = item.modify_metadata(md,
access_key=ia_keys["access"], secret_key=ia_keys["secret"])
r = item.modify_metadata(
md, access_key=ia_keys["access"], secret_key=ia_keys["secret"]
)
if r.status_code != 200:
print("Error when updating metadata")
print(r.status_code)
print(r.text)
print(
"You can find it in https://archive.org/details/%s"
% (identifier)
"You can find it in https://archive.org/details/%s" % (identifier)
)
uploadeddumps.append(dump.name)
except Exception as e:

9
wikiteam3/utils/__init__.py
Unescape Escape View File

@ -1,8 +1,7 @@
from .domain import domain2prefix
from .login import botLogin, clientLogin, fetchLoginToken, indexLogin, uniLogin
from .monkey_patch import mod_requests_text
from .uprint import uprint
from .util import removeIP, cleanXML, cleanHTML, undoHTMLEntities, sha1File
from .user_agent import getUserAgent
from .domain import domain2prefix
from .util import cleanHTML, cleanXML, removeIP, sha1File, undoHTMLEntities
from .wiki_avoid import avoidWikimediaProjects
from .monkey_patch import mod_requests_text
from .login import uniLogin, fetchLoginToken, botLogin, clientLogin, indexLogin

3
wikiteam3/utils/domain.py
Unescape Escape View File

@ -2,7 +2,8 @@ import re
from wikiteam3.dumpgenerator.config import Config
def domain2prefix(config: Config=None, session=None):
def domain2prefix(config: Config = None, session=None):
"""Convert domain name to a valid prefix filename."""
# At this point, both api and index are supposed to be defined

29
wikiteam3/utils/login/__init__.py
Unescape Escape View File

@ -1,37 +1,50 @@
""" Provide login functions """
import requests
import time
import requests
from wikiteam3.utils.login.api import botLogin, clientLogin, fetchLoginToken
from wikiteam3.utils.login.index import indexLogin
def uniLogin(api: str = '', index: str = '' ,session: requests.Session = requests.Session(), username: str = '', password: str = ''):
""" Try to login to a wiki using various methods.\n
def uniLogin(
api: str = "",
index: str = "",
session: requests.Session = requests.Session(),
username: str = "",
password: str = "",
):
"""Try to login to a wiki using various methods.\n
Return `session` if success, else return `None`.\n
Try: `cilent login (api) => bot login (api) => index login (index)` """
Try: `cilent login (api) => bot login (api) => index login (index)`"""
if (not api and not index) or (not username or not password):
print('uniLogin: api or index or username or password is empty')
print("uniLogin: api or index or username or password is empty")
return None
if api:
print("Trying to log in to the wiki using clientLogin... (MW 1.27+)")
_session = clientLogin(api=api, session=session, username=username, password=password)
_session = clientLogin(
api=api, session=session, username=username, password=password
)
if _session:
return _session
time.sleep(5)
print("Trying to log in to the wiki using botLogin... (MW 1.27+)")
_session = botLogin(api=api, session=session, username=username, password=password)
_session = botLogin(
api=api, session=session, username=username, password=password
)
if _session:
return _session
time.sleep(5)
if index:
print("Trying to log in to the wiki using indexLogin... (generic)")
_session = indexLogin(index=index, session=session, username=username, password=password)
_session = indexLogin(
index=index, session=session, username=username, password=password
)
if _session:
return _session

82
wikiteam3/utils/login/api.py
Unescape Escape View File

@ -6,76 +6,86 @@ import requests
def fetchLoginToken(session: requests.Session, api: str) -> Optional[str]:
""" fetch login token by API .(MediaWiki 1.27+)"""
"""fetch login token by API .(MediaWiki 1.27+)"""
response = session.get(
url=api,
params={
'action': "query",
'meta': "tokens",
'type': "login",
'format': "json"})
params={"action": "query", "meta": "tokens", "type": "login", "format": "json"},
)
data = response.json()
try:
token = data['query']['tokens']['logintoken']
token = data["query"]["tokens"]["logintoken"]
if type(token) is str:
return token
except KeyError:
print('fetch login token: Oops! Something went wrong -- ', data)
print("fetch login token: Oops! Something went wrong -- ", data)
return None
def clientLogin(api: str ,session: requests.Session, username: str, password: str) -> Optional[requests.Session]:
""" login to a wiki using username and password. (MediaWiki 1.27+)"""
def clientLogin(
api: str, session: requests.Session, username: str, password: str
) -> Optional[requests.Session]:
"""login to a wiki using username and password. (MediaWiki 1.27+)"""
login_token = fetchLoginToken(session=session, api=api)
if not login_token:
return None
response = session.post(url=api, data={
'action': "clientlogin",
'username': username,
'password': password,
'loginreturnurl': 'http://127.0.0.1:5000/',
'logintoken': login_token,
'format': "json"
})
response = session.post(
url=api,
data={
"action": "clientlogin",
"username": username,
"password": password,
"loginreturnurl": "http://127.0.0.1:5000/",
"logintoken": login_token,
"format": "json",
},
)
data = response.json()
try:
if data['clientlogin']['status'] == 'PASS':
print('client login: Success! Welcome, ' + data['clientlogin']['username'] + '!')
if data["clientlogin"]["status"] == "PASS":
print(
"client login: Success! Welcome, "
+ data["clientlogin"]["username"]
+ "!"
)
except KeyError:
print('client login: Oops! Something went wrong -- ', data)
print("client login: Oops! Something went wrong -- ", data)
return None
return session
def botLogin(api:str ,session: requests.Session, username: str, password: str) -> Optional[requests.Session]:
""" login to a wiki using BOT's name and password. (MediaWiki 1.27+) """
def botLogin(
api: str, session: requests.Session, username: str, password: str
) -> Optional[requests.Session]:
"""login to a wiki using BOT's name and password. (MediaWiki 1.27+)"""
login_token = fetchLoginToken(session=session, api=api)
if not login_token:
return None
response = session.post(url=api, data={
'action': "login",
'lgname': username,
'lgpassword': password,
'lgtoken': login_token,
'format': "json"
})
response = session.post(
url=api,
data={
"action": "login",
"lgname": username,
"lgpassword": password,
"lgtoken": login_token,
"format": "json",
},
)
data = response.json()
try:
if data['login']['result'] == 'Success':
print('bot login: Success! Welcome, ' + data['login']['lgusername'] + '!')
if data["login"]["result"] == "Success":
print("bot login: Success! Welcome, " + data["login"]["lgusername"] + "!")
except KeyError:
print('bot login: Oops! Something went wrong -- ' + data)
print("bot login: Oops! Something went wrong -- " + data)
return None
return session
return session

50
wikiteam3/utils/login/index.py
Unescape Escape View File

@ -1,19 +1,22 @@
""" Always available login methods.(mw 1.16-1.39)
Even oler versions of MW may work, but not tested. """
import lxml.html
from typing import *
import lxml.html
import requests
def indexLogin(index:str ,session: requests.Session, username: str, password: str) -> Optional[requests.Session]:
""" Try to login to a wiki using username and password through `Special:UserLogin`.
(tested on MW 1.16...1.39) """
def indexLogin(
index: str, session: requests.Session, username: str, password: str
) -> Optional[requests.Session]:
"""Try to login to a wiki using username and password through `Special:UserLogin`.
(tested on MW 1.16...1.39)"""
wpEditToken = None
wpLoginToken = None
params = {
'title': 'Special:UserLogin',
"title": "Special:UserLogin",
}
r = session.get(index, allow_redirects=True, params=params)
@ -21,31 +24,38 @@ def indexLogin(index:str ,session: requests.Session, username: str, password: st
# MW 1.16: <input type="hidden" name="wpLoginToken" value="adf5ed40243e9e5db368808b27dc289c" />
# MW 1.39: <input name="wpLoginToken" type="hidden" value="ad43f6cc89ef50ac3dbd6d03b56aedca63ec4c90+\"/>
html = lxml.html.fromstring(r.text)
if 'wpLoginToken' in r.text:
if "wpLoginToken" in r.text:
wpLoginToken = html.xpath('//input[@name="wpLoginToken"]/@value')[0]
# Sample r.text:
# MW 1.16: None
# MW 1.39: <input id="wpEditToken" type="hidden" value="+\" name="wpEditToken"/>
if 'wpEditToken' in r.text:
if "wpEditToken" in r.text:
wpEditToken = html.xpath('//input[@name="wpEditToken"]/@value')[0]
print('index login: wpEditToken found.')
print("index login: wpEditToken found.")
data = {
'wpName': username, # required
'wpPassword': password, # required
'wpLoginattempt': 'Log in', # required
'wpLoginToken': wpLoginToken, # required
'wpRemember': '1', # 0: not remember, 1: remember
'wpEditToken': wpEditToken, # introduced before MW 1.27, not sure whether it's required.
'authAction': 'login', # introduced before MW 1.39.
'title': 'Special:UserLogin', # introduced before MW 1.39.
'force': '', # introduced before MW 1.39, empty string is OK.
"wpName": username, # required
"wpPassword": password, # required
"wpLoginattempt": "Log in", # required
"wpLoginToken": wpLoginToken, # required
"wpRemember": "1", # 0: not remember, 1: remember
"wpEditToken": wpEditToken, # introduced before MW 1.27, not sure whether it's required.
"authAction": "login", # introduced before MW 1.39.
"title": "Special:UserLogin", # introduced before MW 1.39.
"force": "", # introduced before MW 1.39, empty string is OK.
}
r = session.post(index, allow_redirects=False, params=params, data=data)
if r.status_code == 302:
print('index login: Success! Welcome, ', username, '!')
print("index login: Success! Welcome, ", username, "!")
return session
else:
print('index login: Oops! Something went wrong -- ', r.status_code, 'wpLoginToken: ', wpLoginToken, 'wpEditToken: ', wpEditToken)
return None
print(
"index login: Oops! Something went wrong -- ",
r.status_code,
"wpLoginToken: ",
wpLoginToken,
"wpEditToken: ",
wpEditToken,
)
return None

16
wikiteam3/utils/monkey_patch.py
Unescape Escape View File

@ -2,15 +2,19 @@ import requests
from wikiteam3.dumpgenerator.cli.delay import Delay
def mod_requests_text(requests: requests):
""" Monkey patch `requests.Response.text` to remove BOM """
"""Monkey patch `requests.Response.text` to remove BOM"""
def new_text(self):
return self.content.lstrip(b'\xef\xbb\xbf').decode(self.encoding)
return self.content.lstrip(b"\xef\xbb\xbf").decode(self.encoding)
requests.Response.text = property(new_text)
class DelaySession:
""" Monkey patch `requests.Session.send` to add delay """
"""Monkey patch `requests.Session.send` to add delay"""
def __init__(self, session, msg=None, delay=None, config=None):
self.session = session
self.msg = msg
@ -19,14 +23,16 @@ class DelaySession:
self.config = config
def hijack(self):
''' Don't forget to call `release()` '''
"""Don't forget to call `release()`"""
def new_send(request, **kwargs):
Delay(msg=self.msg, delay=self.delay, config=self.config)
return self.old_send(request, **kwargs)
self.old_send = self.session.send
self.session.send = new_send
def release(self):
''' Undo monkey patch '''
"""Undo monkey patch"""
self.session.send = self.old_send
del self

6
wikiteam3/utils/user_agent.py
Unescape Escape View File

@ -314,12 +314,16 @@ def getUserAgents():
]
return useragents
def getUserAgent():
return random.choice(getUserAgents())
def setupUserAgent(session: requests.Session):
session._orirequest = session.request
def newrequest(*args, **kwargs):
session.headers.update({"User-Agent": getUserAgent()})
return session._orirequest(*args, **kwargs)
session.request = newrequest
session.request = newrequest

1
wikiteam3/utils/util.py
Unescape Escape View File

@ -76,6 +76,7 @@ def cleanXML(xml: str = "") -> str:
xml = xml.split("</mediawiki>")[0]
return xml
def sha1File(filename: str = "") -> str:
"""Return the SHA1 hash of a file"""

3
wikiteam3/utils/wiki_avoid.py
Unescape Escape View File

@ -4,7 +4,8 @@ from typing import *
from wikiteam3.dumpgenerator.config import Config
def avoidWikimediaProjects(config: Config=None, other: Dict=None):
def avoidWikimediaProjects(config: Config = None, other: Dict = None):
"""Skip Wikimedia projects and redirect to the dumps website"""
# notice about wikipedia dumps

Write Preview
Loading…
Cancel
Save