Archives
/
bathyscaphe
mirror of https://github.com/creekorful/bathyscaphe
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
361 Commits
2 Branches
15 Tags
1021 KiB
main
devel
v1.0.0
v1.0.0-rc2
v1.0.0-rc1
v0.11.0
v0.10.0
v0.9.0
v0.8.0
v0.7.0
v0.6.0
v0.5.1
v0.5.0
v0.4.0
v0.3.0
v0.2.0
v0.1.0
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'main'
${ noResults }
Commit Graph

215 Commits (main)

Author SHA1 Message Date
Aloïs Micard f24f86fa6e
Fix build (disable goreportcard-action) 4 years ago
Aloïs Micard fd32c66774
Implement API authentication 
Also split source code into new architecture + start writing tests
 4 years ago
Aloïs Micard c2e501d0c2
Merge remote-tracking branch 'origin/master' into feature/api-authentication 4 years ago
Aloïs Micard cf92b5ff31
s/process/component 4 years ago
Aloïs Micard 87cff914b6
Start implement authentication endpoints 4 years ago
Aloïs Micard a1b17d7196
Merge remote-tracking branch 'origin/master' into feature/api-authentication 4 years ago
Aloïs Micard 362133bb23
Improve search rendering on trandoshanctl 4 years ago
Aloïs Micard 25d6452c65
Implement authorization trough JWT 4 years ago
Aloïs Micard 646da7dfdf
API contract: now pagine-able! 4 years ago
Aloïs Micard 7582af03f2
Release 0.4.0 4 years ago
Aloïs Micard b85e9944a2
API: fix startDate/endDate query param 4 years ago
Aloïs Micard 62b54bf385
[#12] Allow duplicate resource crawling 4 years ago
Aloïs Micard e61dc42d3c
Some cleanup 4 years ago
Aloïs Micard fa348dca5d
Last cleanups 
- API: implement pagination for search endpoints
- Crawler: do not save body when code > 302
- Scripts: add stop.sh
 4 years ago
Aloïs Micard e0dfc648b6
Implement search in trandoshanctl 4 years ago
Aloïs Micard 8d9d9524a7
Finalize search endpoint 4 years ago
Aloïs Micard cacf4f1236
Improve api search endpoint 4 years ago
Aloïs Micard 0e6477dd0a
Now follow redirect 4 years ago
Aloïs Micard 742ccbaa79
Finalize whole implementation 4 years ago
Aloïs Micard 6081a6a7c2
Move url extraction logic to extractor 4 years ago
Aloïs Micard f2b8984356
Little cleanup 4 years ago
Aloïs Micard ae5812c566
Make extractor publish found URLs 4 years ago
Aloïs Micard 560d7cb846
Implement extractor 4 years ago
Aloïs Micard 5b220de671
Move messaging into internal package 4 years ago
Aloïs Micard 20f67edd28
Create API client 4 years ago
Aloïs Micard 1c8368704c
Delete pkg/ package and split it 4 years ago
Aloïs Micard 11f04b1ca3
Add missing comments 4 years ago
Aloïs Micard a0be5160dc
Start implementing new architecture 4 years ago
Aloïs Micard 8eedbdd572
Release 0.3.0 4 years ago
Aloïs Micard 325c6ef175
Migrate to zerolog 
Closes: #20
 4 years ago
Aloïs Micard 42ee930160
Unit test scheduler 4 years ago
Aloïs Micard c043ad86f7
Release 0.2.0 4 years ago
Aloïs Micard a635722690
Fix docker image build 4 years ago
Aloïs Micard 82a4a9c527
Add tdsh- prefix to executables. 
API#searchResources:

- Serialize date
- Do not return body in get
 4 years ago
Aloïs Micard 5c739b5809 Release 0.1.0 4 years ago
Aloïs Micard 1413680121
[#9] Prevent from crawling binary, image, etc... 4 years ago
Aloïs Micard 45a9848395
Lint source code 4 years ago
Aloïs Micard b5b58a8d19
Cleanup code 4 years ago
Aloïs Micard 8ae38445cf
Move ResourceDto to proto package 4 years ago
Aloïs Micard 482dde3e17
[#8] Handle case no ES collection yet 4 years ago
Aloïs Micard 6519672b13
Api#addUrl: Fix sent message 4 years ago
Aloïs Micard ed7ea4596b
[#7] API should publish to URLFoundSubject 
this will allow scheduler to approve or not.
 4 years ago
Aloïs Micard 945651b93a
[#7] Fix publish URL 4 years ago
Aloïs Micard 05df5c56a4
Name apps, write test 4 years ago
Aloïs Micard 56cb94258f
Crawler: Allow to customize user agent 4 years ago
Aloïs Micard 599e6ef4d3
Fix wrong endpoint being used by scheduler 
Also b64 encode the URL.

Closes #6
 4 years ago
Aloïs Micard 75fa6724c9
Allow to submit new URL trough the API 
Closes #4
 4 years ago
Aloïs Micard 68ddf09aaa
Use logrus everywhere 4 years ago
Aloïs Micard 8a32bbe5fa
Finx lint issues 4 years ago
Aloïs Micard 6b28f074d1
Implement API 
Now persister process will use API to save resource content.
Scheduler will also use the API to get resource by URL, and will later
determinate if scheduling should be done based on his own algorithm
 4 years ago
Aloïs Micard 680eccef96
Fix wrong usage of logrus in trandoshan-api 4 years ago
Aloïs Micard 317a4eabbd
Add api process 4 years ago
Aloïs Micard 5f1dd4bec8
Implement persister 
Add kibana & elasticsearch dependencies.
 4 years ago
Aloïs Micard 7d2e666ba9
Refactor nats logic into natsutil 4 years ago
Aloïs Micard 82250b46ae
Centralize logging initialization 4 years ago
Aloïs Micard fd9d2e2b9e
Add persister process 4 years ago
Aloïs Micard 4e49e0aca9
Crawler: now publish message with resource body 4 years ago
Aloïs Micard 29da7859b4
Scheduler: normalized received URLs 4 years ago
Aloïs Micard cf8c2875cb
Run gofmt over the project 4 years ago
Aloïs Micard 28f32042c6
Setup crawler to use tor proxy to reach hidden services 4 years ago
Aloïs Micard 33269f7ffa
Centralize ReadJSON into natsutil 4 years ago
Aloïs Micard 6d349da3a6
Fix lint errors 4 years ago
Aloïs Micard c6a857f45b
Add basic scheduler implementation 
- Create separate proto package to store Trandoshan related protocol implementation
 4 years ago
Aloïs Micard 06f31f8d9c
Implement crawler process 
- Also change module URL.
- Create natsutil package
 4 years ago
Aloïs Micard 2f17ee088a
Initial commit 4 years ago