Commit Graph

3 Commits (main)

Author SHA1 Message Date
Aloïs Micard ec3357be5d
Big improvements
- Reduce debug noise
- Create scripts to blacklist 'famous' legit hostnames
- Make blacklister more resilient
- Merge archiver & indexer together
- Better prefix for cache key
- Rework scheduling process
- Update architecture.png
- Remove trandoshanctl
- Improve testing
3 years ago
Aloïs Micard 91a0dbb0ba
Move http folder into crawler 3 years ago
Aloïs Micard 5f657cfc74
Add mock http client & response 4 years ago