Commit Graph

60 Commits (main)

Author SHA1 Message Date
Aloïs Micard 599e6ef4d3
Fix wrong endpoint being used by scheduler
Also b64 encode the URL.

Closes #6
4 years ago
Aloïs Micard 75fa6724c9
Allow to submit new URL trough the API
Closes #4
4 years ago
Aloïs Micard 6b28f074d1
Implement API
Now persister process will use API to save resource content.
Scheduler will also use the API to get resource by URL, and will later
determinate if scheduling should be done based on his own algorithm
4 years ago
Aloïs Micard 317a4eabbd
Add api process 4 years ago
Aloïs Micard 5f1dd4bec8
Implement persister
Add kibana & elasticsearch dependencies.
4 years ago
Aloïs Micard fd9d2e2b9e
Add persister process 4 years ago
Aloïs Micard 28f32042c6
Setup crawler to use tor proxy to reach hidden services 4 years ago
Aloïs Micard c6a857f45b
Add basic scheduler implementation
- Create separate proto package to store Trandoshan related protocol implementation
4 years ago
Aloïs Micard 06f31f8d9c
Implement crawler process
- Also change module URL.
- Create natsutil package
4 years ago
Aloïs Micard 2f17ee088a
Initial commit 4 years ago