Aloïs Micard
|
ab9bf3d4c3
|
api: do not save forbidden hostname
|
3 years ago |
|
Aloïs Micard
|
592a1621d6
|
Bump app versions
|
3 years ago |
|
Aloïs Micard
|
a2db14cc7b
|
blacklister: do not publish duplicates
|
3 years ago |
|
Aloïs Micard
|
78b9e16ded
|
Implement blacklister process
|
3 years ago |
|
Aloïs Micard
|
95989b157b
|
make http client throw err timeout if one occurs
|
3 years ago |
|
Aloïs Micard
|
0cca8b0037
|
make crawler publish event in case crawling has failed
|
3 years ago |
|
Aloïs Micard
|
fcca98f99f
|
writeJson -> writeJSON
|
3 years ago |
|
Aloïs Micard
|
c114e091f9
|
api: return content-type
|
3 years ago |
|
Aloïs Micard
|
205235aa7b
|
fix api & configapi
|
3 years ago |
|
Aloïs Micard
|
5429bb70d8
|
Make api take refresh-delay from configapi
|
3 years ago |
|
Aloïs Micard
|
672c758897
|
Remove echo dependency
|
3 years ago |
|
Aloïs Micard
|
275bad8a6e
|
Finalize refactoring
|
3 years ago |
|
Aloïs Micard
|
c6a9038dad
|
Refactor base api
|
3 years ago |
|
Aloïs Micard
|
8555c5eb05
|
Refactor configapi
|
3 years ago |
|
Aloïs Micard
|
2b0c65ff28
|
refactor extractor
|
3 years ago |
|
Aloïs Micard
|
1feda6e3b9
|
Refactor archiver
|
3 years ago |
|
Aloïs Micard
|
63ad33f984
|
Refactor scheduler
|
3 years ago |
|
Aloïs Micard
|
d1633844c4
|
Refactor crawler using new process library
|
3 years ago |
|
Aloïs Micard
|
51f12b4cba
|
Simplify configapi
|
3 years ago |
|
Aloïs Micard
|
b7ad2097e4
|
Improve crawler test
|
3 years ago |
|
Aloïs Micard
|
0079f4b0ed
|
configapi: create default values if not exist
|
3 years ago |
|
Aloïs Micard
|
aacb28ea1c
|
Start refactoring crawler
|
3 years ago |
|
Aloïs Micard
|
62250bcf61
|
Unit test configapi client
|
3 years ago |
|
Aloïs Micard
|
18bb162ac7
|
event: Create new SubscribeAll function
|
3 years ago |
|
Aloïs Micard
|
2d86123c4a
|
SubscribeAsync -> Subscribe
|
3 years ago |
|
Aloïs Micard
|
e751e0e73e
|
Merge remote-tracking branch 'origin/develop' into 83-event-driven-configuration
|
3 years ago |
|
Aloïs Micard
|
485a109519
|
Improve extensions check
|
3 years ago |
|
Aloïs Micard
|
1b0efdf13e
|
Fix scheduler test
|
3 years ago |
|
Aloïs Micard
|
ec6bcd9d12
|
Finalize client implementation
|
3 years ago |
|
Aloïs Micard
|
a9e1d44e6c
|
Rework event system: add RawMessage
|
3 years ago |
|
Aloïs Micard
|
c50c4bb9c9
|
Working on config update trough event
|
3 years ago |
|
Aloïs Micard
|
a4326ab955
|
Add redis as config backend database
|
3 years ago |
|
Aloïs Micard
|
b9cea755de
|
Fix ConfigAPI server
|
3 years ago |
|
Aloïs Micard
|
2fd03d2112
|
Implement memory only database to test configapi
|
3 years ago |
|
Aloïs Micard
|
32cb7e63e0
|
Implement ConfigAPI client
|
3 years ago |
|
Aloïs Micard
|
901653dc02
|
Dockerize configapi
|
3 years ago |
|
Aloïs Micard
|
345f9c2669
|
Finalize ConfigAPI server
|
3 years ago |
|
Aloïs Micard
|
353880bc22
|
Implement ConfigAPI service layer
|
3 years ago |
|
Aloïs Micard
|
54ff87a130
|
event: add PublishJson method
|
3 years ago |
|
Aloïs Micard
|
278f8e58c2
|
Refactor scheduler to use new configapi
|
3 years ago |
|
Aloïs Micard
|
a1f8313246
|
Design configapi interfaces
|
3 years ago |
|
Aloïs Micard
|
08be8ddce5
|
Reduce API noise
|
3 years ago |
|
Aloïs Micard
|
bf884d16c2
|
Bump app versions
|
3 years ago |
|
Aloïs Micard
|
72458ea9c4
|
Improve forbidden hostname checks
|
3 years ago |
|
Aloïs Micard
|
1cb038727e
|
Allow to add forbidden hostnames
|
3 years ago |
|
Aloïs Micard
|
56c5fc8c9c
|
Cleanup test
|
3 years ago |
|
Aloïs Micard
|
b04c8a9e32
|
Refactor to use api.API everywhere
|
3 years ago |
|
Aloïs Micard
|
866b80f013
|
Little cleanup
|
3 years ago |
|
Aloïs Micard
|
579a39dddf
|
Implement archiver
|
3 years ago |
|
Aloïs Micard
|
480b1f8bc3
|
extractor: publish time as set by crawler
|
3 years ago |
|
Aloïs Micard
|
9e2186b97a
|
Add missing comments
|
3 years ago |
|
Aloïs Micard
|
5b37a4aeb0
|
Add time to new resource event
|
3 years ago |
|
Aloïs Micard
|
ac983b25ef
|
scheduler: fix errors perms
|
3 years ago |
|
Aloïs Micard
|
dea2cfe7b0
|
Refactor API
|
3 years ago |
|
Aloïs Micard
|
6b54772ac4
|
Improve scheduler error management
|
3 years ago |
|
Aloïs Micard
|
8516c8a00c
|
Refactor scheduler
|
3 years ago |
|
Aloïs Micard
|
ca47be907f
|
Fix state visibility
|
3 years ago |
|
Aloïs Micard
|
73cb76e1f7
|
Refactor extractor
|
3 years ago |
|
Aloïs Micard
|
4ff76ed552
|
Refactor crawler
|
3 years ago |
|
Aloïs Micard
|
5ad83d57a0
|
Rewrite event support
|
3 years ago |
|
Aloïs Micard
|
91a0dbb0ba
|
Move http folder into crawler
|
3 years ago |
|
Aloïs Micard
|
972c76a383
|
Remove unusued regex
|
4 years ago |
|
Aloïs Micard
|
4235145591
|
Create package for API
|
4 years ago |
|
Aloïs Micard
|
3a78e26ee2
|
Merge remote-tracking branch 'origin/master' into develop
|
4 years ago |
|
Aloïs Micard
|
39769e724f
|
Make message persistent by default
|
4 years ago |
|
Aloïs Micard
|
180182482c
|
Final (tested) fixes to the api search method
|
4 years ago |
|
Aloïs Micard
|
fa542b4bcb
|
Fix ES db query
|
4 years ago |
|
Aloïs Micard
|
de4779724f
|
Prevent duplicates (enough this time?)
|
4 years ago |
|
Aloïs Micard
|
8417539395
|
Fix RabbitMQ consumer
|
4 years ago |
Aloïs Micard
|
1eb45d82b8
|
Merge pull request #53 from creekorful/rabbitmq-refactoring
Refactor to use RabbitMQ
|
4 years ago |
|
Aloïs Micard
|
29ed1f2f5f
|
Put headers & meta in lowercase
|
4 years ago |
|
Aloïs Micard
|
ae31e70c42
|
Rename event-srv -> hub
|
4 years ago |
|
Aloïs Micard
|
db983c584b
|
Merge remote-tracking branch 'origin/develop' into rabbitmq-refactoring
|
4 years ago |
|
Aloïs Micard
|
3b320d49c7
|
Merge remote-tracking branch 'origin/develop' into 54-extract-http-headers
|
4 years ago |
|
Aloïs Micard
|
5f657cfc74
|
Add mock http client & response
|
4 years ago |
|
Aloïs Micard
|
59aa2cf86f
|
Crawl headers
|
4 years ago |
|
Aloïs Micard
|
82868521ab
|
extractor: prevent from publishing duplicates URLs
|
4 years ago |
|
Aloïs Micard
|
b365954e31
|
scheduler: filter protocol
|
4 years ago |
|
Aloïs Micard
|
0dc70f63f7
|
Refactor to use RabbitMQ
|
4 years ago |
|
Aloïs Micard
|
0c4013f0c1
|
Bump app versions
|
4 years ago |
|
Aloïs Micard
|
fec9d5c506
|
Make things more readable
|
4 years ago |
|
Aloïs Micard
|
c3d387b545
|
Finalize ACL implementation
|
4 years ago |
|
Aloïs Micard
|
5e3bc78ae1
|
Improve TestHandleMessage
|
4 years ago |
|
Aloïs Micard
|
4bcbcfaefd
|
Add todo about allow search by meta
|
4 years ago |
|
Aloïs Micard
|
1f106dca49
|
Add meta & description to resource
|
4 years ago |
|
Aloïs Micard
|
e4a01a1876
|
Harmonize error management
|
4 years ago |
|
Aloïs Micard
|
f297c9eab5
|
Harmonize logging messages
|
4 years ago |
|
Aloïs Micard
|
e1c0320a7b
|
Allow to skip scheduling for url with forbidden extensions
Closes: #42
|
4 years ago |
|
Aloïs Micard
|
0e05349f05
|
scheduler: reduce log noise
|
4 years ago |
|
Aloïs Micard
|
c752eb95e0
|
Release 0.5.1
|
4 years ago |
carter
|
a80360a8ce
|
Fixed doubling-up of URLs due to URL structure miss-match
|
4 years ago |
|
Aloïs Micard
|
8233880fb8
|
Fix api test
|
4 years ago |
|
Aloïs Micard
|
1e238a34d8
|
[#34] Improve search resources
|
4 years ago |
|
Aloïs Micard
|
2301a25dff
|
Move database in api folder
|
4 years ago |
|
Aloïs Micard
|
d55e0e4609
|
Release 0.5.0
|
4 years ago |
|
Aloïs Micard
|
8750830a62
|
Use resty client
|
4 years ago |
|
Aloïs Micard
|
a4f86fbee9
|
Finalize usage of authentication for components
|
4 years ago |
|
Aloïs Micard
|
4633cc7695
|
Unit test scheduler
|
4 years ago |
|
Aloïs Micard
|
73f52703f1
|
Unit test extractor
|
4 years ago |
|
Aloïs Micard
|
27e7c9d2fa
|
Implement basic user registration system
|
4 years ago |