Aloïs Micard
|
6f79af997f
|
s/creekorful/darkspot(-org)/
|
3 years ago |
Aloïs Micard
|
02a0dca6ed
|
Add process description
|
3 years ago |
Aloïs Micard
|
96c75a85b1
|
Rename project
|
3 years ago |
Aloïs Micard
|
faee8b48c1
|
indexer: sort headers to have deterministic output
|
3 years ago |
Aloïs Micard
|
9b46dc205e
|
Indexer: support buffered indexing
|
3 years ago |
Aloïs Micard
|
71f82d4aad
|
process: Rework whole flags system
- Turn the flag into Feature system to allow easier configuration.
- Add prefetch flag to event feature
|
3 years ago |
Aloïs Micard
|
ec3357be5d
|
Big improvements
- Reduce debug noise
- Create scripts to blacklist 'famous' legit hostnames
- Make blacklister more resilient
- Merge archiver & indexer together
- Better prefix for cache key
- Rework scheduling process
- Update architecture.png
- Remove trandoshanctl
- Improve testing
|
3 years ago |
Aloïs Micard
|
cc3c0d62d6
|
remove hacky check
|
3 years ago |
Aloïs Micard
|
c8352d3299
|
Use url cache to determinate if crawling should be done
|
3 years ago |
Aloïs Micard
|
e245e5d79a
|
last fixes
|
3 years ago |
Aloïs Micard
|
60a23f7182
|
Fix ttl
|
3 years ago |
Aloïs Micard
|
12362e0100
|
Fix tests case
|
3 years ago |
Aloïs Micard
|
387a93b7b9
|
Create new flags for cache
|
3 years ago |
Aloïs Micard
|
477092316b
|
Implement cache logic
|
3 years ago |
Aloïs Micard
|
55ae36f3b9
|
s/database/index
|
3 years ago |
Aloïs Micard
|
2d6beb26ce
|
elastic: pre-declare index mapping
|
3 years ago |
Aloïs Micard
|
188df77541
|
improve logging
|
3 years ago |
Aloïs Micard
|
2eb416845e
|
improve logging
|
3 years ago |
Aloïs Micard
|
ad808e6b31
|
indexer: do not publish duplicate URLs
|
3 years ago |
Aloïs Micard
|
797c3df9a5
|
move api client into appropriate package
|
3 years ago |
Aloïs Micard
|
4d250b6cb0
|
Finalize refactoring
|
3 years ago |
Aloïs Micard
|
a996bf2d5b
|
Turn API into indexer
|
3 years ago |