arkiver
|
0be16f775a
|
Version 20210114.04. Support cookies.
|
3 years ago |
|
arkiver
|
0d0e824421
|
Version 20210114.03. Do not accept 403.
|
3 years ago |
|
arkiver
|
94d8b551f8
|
Version 20210114.02. Actually add the user-agents file.
|
3 years ago |
|
arkiver
|
bc94cf036f
|
Version 20210114.01. Use a random user-agent.
|
3 years ago |
|
arkiver
|
df1f60079d
|
Version 20210109.01. Use browser user-agent.
|
3 years ago |
|
arkiver
|
2911934fd4
|
Version 20210108.09. Ignore over18 URLs on old.reddit.com (cookie fix coming up, not a problem on www.reddit.com).
|
3 years ago |
|
arkiver
|
f3d41ea2e1
|
Version 20210108.08. Do not archive URLs with utm_source for old.reddit.com.
|
3 years ago |
|
arkiver
|
992fb6b953
|
Version 20210108.07. Use tracker reddit.
|
3 years ago |
|
arkiver
|
6a8d5a62ac
|
Version 20210108.06.
|
3 years ago |
|
arkiver
|
1b220e014b
|
Fix for archiving videos.
|
3 years ago |
|
arkiver
|
4a371be167
|
Version 20210108.05.
|
3 years ago |
|
arkiver
|
3d20ca90af
|
Handle NULL byte seperated multi items. Support unicode chars in JSON permalink.
|
3 years ago |
|
arkiver
|
7c5ea717a8
|
Version 20210108.03.
|
3 years ago |
|
arkiver
|
5f3958c282
|
Merge branch 'master' of https://github.com/ArchiveTeam/reddit-grab
|
3 years ago |
|
arkiver
|
1924d5217e
|
Version 20210108.02.
|
3 years ago |
|
arkiver
|
16836ba201
|
Support single comment and post items. Queue outlinks to URLs project.
|
3 years ago |
|
arkiver
|
ae57a81baf
|
Use multi items.
|
3 years ago |
km09
|
eb945d2470
|
Use updated grab-base
|
4 years ago |
|
arkiver
|
4284d24b47
|
Version 20201031.01. Support Wget-AT version 1.20.3-at.20201030.01.
|
4 years ago |
|
arkiver
|
9ecf9a3a30
|
Version 20200902.01. Support Wget-AT version 1.20.3-at.20200902.01.
|
4 years ago |
|
arkiver
|
99875895b6
|
Version 20200821.02. Set tracker host to trackerproxy.archiveteam.org.
|
4 years ago |
|
arkiver
|
2087174a5c
|
Version 20200821.01. Ignore comment URL with utm_source param.
|
4 years ago |
|
arkiver
|
ace1a4f037
|
Version 20200805.01. Support Wget-AT version 1.20.3-at.20200804.01.
|
4 years ago |
|
arkiver
|
8b40429e95
|
Use new README template.
|
4 years ago |
|
arkiver
|
23bfe8b12c
|
Version 20200730.01. Support /user/ post better (like /r/).
|
4 years ago |
|
arkiver
|
450d4e0413
|
Version 20200728.01. Ignore non-reddit URLs. Fix extraction of tokens for morecomments.
|
4 years ago |
|
arkiver
|
9a6417ecbc
|
Version 20200727.03. Fix handling video URLs without extension.
|
4 years ago |
|
arkiver
|
869cdc4e6e
|
Remove unused cookies.txt file. Update README.
|
4 years ago |
|
arkiver
|
911c675e74
|
Version 20200727.02. Set TRACKER_ID to reddittest.
|
4 years ago |
|
arkiver
|
147c6416ed
|
Version 20200727.01. Use trackerproxy for dictionaries. Ignore irc: URLs.
|
4 years ago |
|
arkiver
|
910687b053
|
Version 20200726.06. Fix project name for ZSTD dictionary request.
|
4 years ago |
|
arkiver
|
496c018eef
|
Version 20200726.05. Add cookies to access some quarantines subreddits.
|
4 years ago |
|
arkiver
|
3d5e7e17f9
|
Version 20200726.04. Use reddittest tracker for size estimate.
|
4 years ago |
|
arkiver
|
23fec56409
|
Version 20200726.03. Support galleries and comments.
|
4 years ago |
|
arkiver
|
2f6a602313
|
Version 20200726.01. Fully support new and old design for posts.
|
4 years ago |
|
arkiver
|
56571306dd
|
Use default upload concurrent of 2.
|
4 years ago |
|
arkiver
|
40063adcaf
|
Use wget-at with ZSTD.
|
4 years ago |
|
Arkiver2
|
831f79f0d9
|
Do not import warcio. Update version to 20200102.03.
|
4 years ago |
|
Arkiver2
|
cf3f6c7af9
|
Skip URL on status code 204. Update version to 20200102.02.
|
4 years ago |
|
Arkiver2
|
ac65b0a818
|
Update version to 20200102.01.
|
4 years ago |
|
Arkiver2
|
0eb4b6205a
|
Fix string joining.
|
4 years ago |
|
Arkiver2
|
ad2cf89404
|
Split off checking if URL was processed. Do not add URL without trailing / already added with trailing /.
|
5 years ago |
|
Arkiver2
|
d4d5c9a93f
|
Skip amp.reddit.com post pages.
|
5 years ago |
|
Arkiver2
|
4cf7bd18f0
|
Version 20190729.01; do not get page requisites from outlinks; do not pip install warcio.
|
5 years ago |
|
Arkiver2
|
8902255c76
|
Version 20190405.01; support www.reddit.com; support videos; support outlinks
|
5 years ago |
|
Arkiver2
|
9d1ea0c688
|
rewrite
|
5 years ago |
Arkiver2
|
c08fd59a29
|
reddit.lua: ignore urls, fixes
|
9 years ago |
|
Arkiver2
|
11aef69a32
|
pipeline.py: cookies!
|
9 years ago |
|
Arkiver2
|
38074381c4
|
cookies
|
9 years ago |
|
Arkiver2
|
e87a2e4a51
|
README.md
|
9 years ago |