Archives
/
reddit-grab
mirror of https://github.com/ArchiveTeam/reddit-grab
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • daab40aa6e Version 20240216.01. Use fixed minimum Wget version 1.21.3-at.20231213.03. Use TLSv1.2. Fix check on svc comment content check. master arkiver 2024-02-16 12:19:43 +0100
  • 48dc016faf Version 20231201.01. Change protocol. arkiver 2023-12-01 01:04:50 +0100
  • 5f7cee8d3a Version 20231127.02. New --ciphers value. arkiver 2023-11-28 02:35:20 +0100
  • 2b41d8ef42 Version 20231127.01. Use --ciphers SECURE256. arkiver 2023-11-27 01:37:38 +0100
  • 7da27ab110 Version 20231118.01. Switch to gnutls. arkiver 2023-11-18 16:25:31 +0100
  • 0dc36e31e0 Version 20231115.01. Change cipher list. arkiver 2023-11-15 22:25:47 +0100
  • 8fc86a11ca Version 20231111.02. arkiver 2023-11-11 18:24:22 +0100
  • 6fdf778e19 Version 20231111.01. Switch ciphers again. arkiver 2023-11-11 17:39:35 +0100
  • e87de8969c Version 20231108.02. Move to another cipher. arkiver 2023-11-08 23:21:28 +0100
  • 9c9b59dafd Version 20231108.01. Do not install utf8 with luarocks, this is now in base parent image. arkiver 2023-11-08 23:18:20 +0100
  • 388e4325c5 Version 20231102.01. Do not keep partial files over rsync. arkiver 2023-11-02 22:47:00 +0100
  • 1c2723f9f2 Version 20231026.01. Use --ciphers HIGH:+SHA384. arkiver 2023-10-26 00:28:02 +0200
  • e350e69f89 Version 20231020.01. Use gnutls. Support new method of serving Reddit comments. arkiver 2023-10-20 00:30:12 +0200
  • 0e7392acd3 Version 20231019.01. Use --secure-protocol=TLSv1_2. arkiver 2023-10-19 03:23:51 +0200
  • 4bcc04734f Version 20231017.02. Use --secure-protocol=TLSv1_3. arkiver 2023-10-17 22:59:48 +0200
  • b1bf682030 Version 20231017.01. Use --secure-protocol=auto. Use new minimum Wget version checker. arkiver 2023-10-17 00:18:22 +0200
  • a0e35bb72d Version 20230910.05. Install Lua utf8 library through warrior-install.sh. arkiver 2023-09-10 22:54:30 +0200
  • 3add4f891c Version 20230910.04. Install lua utf8 library. Fix converting unicode codepoint to utf8 character support. arkiver 2023-09-10 22:49:35 +0200
  • 12abd58d4d Version 20230910.03. Increase hardcoded multi item size to 100, for soft limiting on tracker side. arkiver 2023-09-10 05:37:31 +0200
  • 8a46824231 Version 20230910.02. Remove old Lua files. arkiver 2023-09-10 05:36:35 +0200
  • a2ffd1f671 Version 20230910.01. Use cjson instead of JSON.lua. arkiver 2023-09-10 05:28:45 +0200
  • e6b1602e31 Version 20230827.01. Use --secure-protocol=TLSv1_3. arkiver 2023-08-27 22:25:47 +0200
  • d210e65967
         Merge pull request #18 from imerr/master-1 arkiver 2023-08-04 00:04:52 +0200
  • b7feddc147
         Extra docker container params Robin Rolf 2023-08-03 23:56:37 +0200
  • 29a6952edb Version 20230727.03. In the Warrior, do not use GnuTLS compiled Wget-AT. arkiver 2023-07-27 18:03:48 +0200
  • 6e73452ec5 Version 20230727.02. Only allow GNU Wget 1.21.3-at.20230623.01. Use Wget-AT option --reject-reserved-subnets. Remove old Wget files. Update README to latest. arkiver 2023-07-27 17:39:42 +0200
  • 288c9b731c Version 20230727.01. Use openssl instead of gnutls. arkiver 2023-07-27 16:23:21 +0200
  • bb6198cc1a Version 20230627.01. Queue outlinks directly to the urls project. arkiver 2023-06-27 12:59:40 +0200
  • f1ef7d1697 Version 20230619.02. Accept 404 on mediaembed URL. arkiver 2023-06-19 18:28:52 +0200
  • d2571cde06 Version 20230619.01. Primitive fix to user post verification problems. arkiver 2023-06-19 02:54:59 +0200
  • 2b19cdcd43 Version 20230617.01. Use --secure-protocol=auto for Wget-AT. arkiver 2023-06-17 15:16:03 +0200
  • 5a0dcd6dd9
         Merge pull request #17 from masterX244/master arkiver 2023-06-15 15:06:42 +0200
  • 488aaa2181
         Update pipeline.py masterX244 2023-06-15 13:09:42 +0200
  • 520e8b95d6
         Ignore for some garbge URLs that 404 masterX244 2023-06-15 13:08:24 +0200
  • bea971f375 Version 20230614.03. Better check for level error page on svc URL. arkiver 2023-06-15 01:45:13 +0200
  • be6e32cba5 Version 20230614.02. Extra validity checks. arkiver 2023-06-14 22:12:15 +0200
  • e84e804fc5 Version 20230614.01. Fix check for valid data. arkiver 2023-06-14 18:49:41 +0200
  • 4936505b0f Version 20230612.02. Add Reddit problem check for /comments/.../comment/ URL. arkiver 2023-06-14 03:07:27 +0200
  • 57adbb381c Version 20230612.01. Kill grab when reddit seems to have problems. arkiver 2023-06-12 19:50:28 +0200
  • 0ef6368945 Version 20230611.02. Multi item size 40. arkiver 2023-06-11 00:12:40 +0200
  • a974b81618 Version 20230611.01. Extra very simple check on validity of old.reddit.com returned body. arkiver 2023-06-11 00:12:10 +0200
  • 15a0a1a6f5 Version 20230607.06. Ignore discovered /r/FIFA URL if coming from a /r/EASportFC parent URL. arkiver 2023-06-07 23:13:42 +0200
  • fe17191306 Version 20230607.05. Better checking for video. Abort item if no post is found (during blackout for example). arkiver 2023-06-07 23:05:44 +0200
  • 7bb5c39419 Version 20230607.04. Abort on video for now. arkiver 2023-06-07 22:53:41 +0200
  • f63c8ab696 Version 20230607.03. Prevent getting URL ending with /". Ignore /message/compose URLs. arkiver 2023-06-07 22:39:57 +0200
  • 393407520b Version 20230607.02. Very simple content checks to check if response is complete. Properly prevent writing to WARC in cases and do not abort all items when finding a problematic URL. arkiver 2023-06-07 22:35:47 +0200
  • 37ba172c61 Version 20230607.01. Use GNU Wget 1.21.3-at.20230605.01 and arguments around DNS. arkiver 2023-06-07 15:46:23 +0200
  • da85457aae Version 20230531.01. Use --secure-protocol PFS. arkiver 2023-05-31 10:16:48 +0200
  • 48b24323c6 Version 20230530.01. Queue discovered outlinks to urls-stash-reddit. arkiver 2023-05-30 19:42:55 +0200
  • a3b5bcecc1 Version 20230529.01. Correctly extract more comment pages from comment pages in the new design. Print debug infrmation for comment pages on old design. arkiver 2023-05-29 17:56:36 +0200
  • 1a14af2095 Version 20230509.02. Support new Wget-AT. arkiver 2023-05-09 05:48:05 +0200
  • b2654e9317 Version 20230509.01. Support for new design. arkiver 2023-05-09 05:43:21 +0200
  • 7f4db17348 Version 20221021.01. Ignore /tailwind-build.css URL from comment in HTML. arkiver 2022-10-21 01:11:46 +0200
  • 8a27002fd3 Version 20221005.01. Max tries for backfeed to 10. arkiver 2022-10-05 16:20:17 +0200
  • 35e31af37f Queue redditstatic.com URLs as outlinks. arkiver 2022-10-05 16:19:53 +0200
  • bab4b4dcd2 Version 20220729.05. Fix aborting item on bad status code on url: item. Keep old retry code otherwise. arkiver 2022-07-29 04:52:08 +0200
  • 8c45a263aa Version 20220729.04. Queue extra found URLs on media URLs to backfeed. arkiver 2022-07-28 18:31:23 +0200
  • e8fe03fbd0 Version 20220729.03. Add url: prefix to url item. arkiver 2022-07-28 18:20:59 +0200
  • 2d8fa4034b Version 20220729.02. Support older Wget versions. arkiver 2022-07-28 18:15:54 +0200
  • f81b2ce97e Version 20220729.01. Queue media URLs back to reddit project and download individually. arkiver 2022-07-28 18:09:04 +0200
  • edacb2065a Fix README. arkiver 2022-05-07 04:49:30 +0200
  • cc83009a94 Version 20220605.01. Support GNU Wget 1.21.3-at.20220503.02. Fix killing crawl when items cannot be queued. arkiver 2022-05-06 18:31:38 +0200
  • 7c4cf4548e Version 20220415.02. arkiver 2022-04-15 21:39:33 +0200
  • 754fd256cb
         Merge pull request #13 from NGTmeaty/patch-1 arkiver 2022-04-15 21:38:46 +0200
  • 0ce1c59ca4 Version 20220415.01. Do not queue /r/undefined/ URLs. arkiver 2022-04-15 20:38:36 +0200
  • a858c33e29
         Add support for latest change in _options Jake L 2022-03-31 20:46:28 -0400
  • da28d3c902 Version 20220323.03. Fix items to maxtries variable name. Fix backfeed key name. arkiver 2022-03-23 21:59:52 +0100
  • 8944cf1fc6 Version 20220323.02. Fix items to maxtries variable name. arkiver 2022-03-23 16:36:23 +0100
  • 10eaa7c50c Version 20220323.01. Fix backfeed. Fix maxtries use. arkiver 2022-03-23 16:16:58 +0100
  • 28f132a052 Version 20220312.01. Fix backfeed. arkiver 2022-03-12 23:53:48 +0100
  • 4f50a0d699 Version 20220311.01. Use new backfeed endpoint for queuing. arkiver 2022-03-11 03:52:49 +0100
  • 383c101aef Version 20220109.02. Cut off URL at space when found between brackets without href= in front. arkiver 2022-01-09 17:19:29 +0100
  • df35317e0c Version 20220109.01. Add codepoint to utf8 support. Percent encode outlinks correctly. arkiver 2022-01-09 17:15:10 +0100
  • 0bcabda4d0 (chore) only update README.md and discard custom Dockerfiles T31M 2022-01-03 16:49:41 +0100
  • acadd8ee91 (feat) build zstd 1.4.4 from source to be compatible T31M 2022-01-02 22:06:45 +0100
  • 072c3b0261 (chore) Update build instructions for Alpine Linux README.md T31M 2022-01-02 19:42:15 +0100
  • ff1fceefe2 (feat) Add standalone reddit-grab Dockerfile T31M 2022-01-02 19:41:46 +0100
  • 71dcf25a0c
         Add new search path for Wget+At | Fixes #9 Julian Liebig 2021-10-20 21:36:00 +0200
  • 8a3f8cd1de Version 20211004.02. Fix incomplete facebook.com fix. arkiver 2021-10-04 21:09:21 +0200
  • d0070db67a Version 20211004.01. Do not check facebook.com while down at the moment. arkiver 2021-10-04 21:04:03 +0200
  • 0c5e8cd3bd Version 20211001.01. Use GNU Wget 1.20.3-at.20211001.01. arkiver 2021-10-01 02:44:01 +0200
  • ed80cb5a9d Version 20210707.01. Do not get media for cross posts. arkiver 2021-07-07 00:12:56 +0200
  • 4b976e2ea7 Version 20210521.01. Use TLS 1.2. arkiver 2021-05-21 22:37:19 +0200
  • f4619bb17f use onbuild-based image Katie Holly 2021-05-16 21:05:18 +0000
  • e6b876e9e6
         New day.. new wget-at 1.20.3-at.20210504.01 km09 2021-05-06 00:07:16 +0100
  • 1f9e995b4e
         20210410.01 - New day, new wget-at Thomas Glass 2021-04-10 15:20:27 +0100
  • 6e15841550 Version 20210407.01. Improve video archiving. Detect if video is still being processed by reddit. arkiver 2021-04-07 00:38:20 +0200
  • 1b3690d994 Version 20210330.04. Only decode unicode characters in URLs on v.redd.it URLs. arkiver 2021-03-30 22:20:43 +0200
  • ce7fff480d Version 20210330.03. Unescape unicode characters. Do not HLS for video. arkiver 2021-03-30 20:57:31 +0200
  • ad04f45d4f Fix typo. arkiver 2021-03-30 16:11:12 +0200
  • adc7f9c6fb Version 20210330.02. Skip images that are only in JSON and not on web page. arkiver 2021-03-30 02:21:55 +0200
  • 07ed16c44b Version 20210330.01. Handle 403 on v.redd.it on deleted post. arkiver 2021-03-30 01:49:48 +0200
  • 8849165130 Version 20210321.01. Do not get all video sizes. arkiver 2021-03-21 02:21:41 +0100
  • d3b6659419 Version 20210312.01. Get URLs with utm_* and context params. arkiver 2021-03-12 21:36:32 +0100
  • a5c798945c Version 20210306.01. Remove some AppleWebKir user-agents for getting 403s. arkiver 2021-03-06 00:27:31 +0100
  • eaad7cd7e7
         add 1.20.3-at.20210212.02 as supported wget-at version Katie Holly 2021-02-25 03:01:34 +0100
  • 3b4a2ef5a7
         20210225.01: update dict url Katie Holly 2021-02-25 02:58:24 +0100
  • e6c33f9433
         Updated warrior support Thomas Glass 2021-02-03 13:56:26 +0000
  • 261a7f76d2
         Update tracker host Thomas Glass 2021-02-03 13:31:32 +0000
  • 3d8f85a08a
         Support new wget-at location km09 2021-02-03 01:18:34 +0000