s/process/component

pull/37/head
Aloïs Micard 4 years ago
parent 3ac0520aca
commit cf92b5ff31
No known key found for this signature in database
GPG Key ID: 1A0EB82F071F5EFE

@ -11,7 +11,7 @@ The first version of Trandoshan [(available here)](https://github.com/trandoshan
not really professional, the code start to be a mess, hard to manage since split in multiple repositories, etc.
I have therefore decided to create & maintain the project in this specific repository,
where all process code will be available (as a Go module).
where all components code will be available (as a Go module).
# How to start the crawler
@ -30,7 +30,7 @@ and wait for all containers to start.
# How to initiate crawling
Since the API is exposed on localhost:15005, one can use it to start the crawling process:
Since the API is exposed on localhost:15005, one can use it to start crawling:
using trandoshanctl executable:
@ -50,7 +50,7 @@ this will schedule given URL for crawling.
## How to speed up crawling
If one want to speed up the crawling process, he can scale the instance of crawling process in order
If one want to speed up the crawling, he can scale the instance of crawling component in order
to increase performances. This may be done by issuing the following command after the crawler is started:
```sh
@ -74,7 +74,7 @@ You will need to create an index pattern named 'resources', and when it asks for
# How to hack the crawler
If you've made a change to one of the crawler process and wish to use the updated version when
If you've made a change to one of the crawler component and wish to use the updated version when
running start.sh you just need to issue the following command:
```sh

@ -33,7 +33,7 @@ type ResourceDto struct {
Time time.Time `json:"time"`
}
// Client is the interface to interact with the API process
// Client is the interface to interact with the API component
type Client interface {
SearchResources(url, keyword string, startDate, endDate time.Time,
paginationPage, paginationSize int) ([]ResourceDto, int64, error)

@ -1,6 +1,6 @@
# Crawler
The crawler is the central process of Trandoshan.
The crawler is the central component of Trandoshan.
It consumes URL, crawl them and publish the page body while following redirects etc...
## Consumes
@ -13,7 +13,7 @@ It consumes URL, crawl them and publish the page body while following redirects
# Extractor
The extractor is the data extraction process of Trandoshan.
The extractor is the data extraction component of Trandoshan.
It consumes crawled resource, extract data (urls, metadata, etc...) from it,
store them into an ES instance (by calling the API), & publish found URLs.
@ -29,7 +29,7 @@ store them into an ES instance (by calling the API), & publish found URLs.
# Scheduler
The scheduler is the process responsible for crawling schedule part.
The scheduler is the component responsible for crawling schedule part.
It determinates which URL should be crawled and publish them.
## Consumes
@ -42,4 +42,4 @@ It determinates which URL should be crawled and publish them.
# API
The API process is mainly used to get data from ES.
The API component is mainly used to get data from ES.

@ -42,7 +42,7 @@ func GetApp() *cli.App {
return &cli.App{
Name: "tdsh-api",
Version: "0.4.0",
Usage: "Trandoshan API process",
Usage: "Trandoshan API component",
Flags: []cli.Flag{
logging.GetLogFlag(),
&cli.StringFlag{

@ -22,7 +22,7 @@ func GetApp() *cli.App {
return &cli.App{
Name: "tdsh-crawler",
Version: "0.4.0",
Usage: "Trandoshan crawler process",
Usage: "Trandoshan crawler component",
Flags: []cli.Flag{
logging.GetLogFlag(),
&cli.StringFlag{

@ -25,7 +25,7 @@ func GetApp() *cli.App {
return &cli.App{
Name: "tdsh-extractor",
Version: "0.4.0",
Usage: "Trandoshan extractor process",
Usage: "Trandoshan extractor component",
Flags: []cli.Flag{
logging.GetLogFlag(),
&cli.StringFlag{

@ -21,7 +21,7 @@ func GetApp() *cli.App {
return &cli.App{
Name: "tdsh-scheduler",
Version: "0.4.0",
Usage: "Trandoshan scheduler process",
Usage: "Trandoshan scheduler component",
Flags: []cli.Flag{
logging.GetLogFlag(),
&cli.StringFlag{

Loading…
Cancel
Save