13 Commits (ec3357be5d402205d562500ee378dc36a330384c)

Author	SHA1	Message	Date
Aloïs Micard	ec3357be5d	Big improvements ... - Reduce debug noise - Create scripts to blacklist 'famous' legit hostnames - Make blacklister more resilient - Merge archiver & indexer together - Better prefix for cache key - Rework scheduling process - Update architecture.png - Remove trandoshanctl - Improve testing	3 years ago
Aloïs Micard	cc3c0d62d6	remove hacky check	3 years ago
Aloïs Micard	c8352d3299	Use url cache to determinate if crawling should be done	3 years ago
Aloïs Micard	e245e5d79a	last fixes	3 years ago
Aloïs Micard	60a23f7182	Fix ttl	3 years ago
Aloïs Micard	12362e0100	Fix tests case	3 years ago
Aloïs Micard	477092316b	Implement cache logic	3 years ago
Aloïs Micard	55ae36f3b9	s/database/index	3 years ago
Aloïs Micard	188df77541	improve logging	3 years ago
Aloïs Micard	ad808e6b31	indexer: do not publish duplicate URLs	3 years ago
Aloïs Micard	797c3df9a5	move api client into appropriate package	3 years ago
Aloïs Micard	4d250b6cb0	Finalize refactoring	3 years ago
Aloïs Micard	a996bf2d5b	Turn API into indexer	3 years ago