Content fetch and aggregation bot for hugo data-driven websites
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Chakib Benziane 9c27502304 update readme 1 year ago
bitcoin initial 1 year ago
config initial 1 year ago
db initial 1 year ago
encoder initial 1 year ago
export initial 1 year ago
feeds initial 1 year ago
filters initial 1 year ago
github initial 1 year ago
handlers initial 1 year ago
logging initial 1 year ago
posts initial 1 year ago
static initial 1 year ago
types initial 1 year ago
utils initial 1 year ago
.gitignore initial 1 year ago
Dockerfile initial 1 year ago
Dockerfile-sqliteweb initial 1 year ago
LICENSE add license 1 year ago
Makefile initial 1 year ago
README.md update readme 1 year ago
api.go initial 1 year ago
commands.go initial 1 year ago
config.toml initial 1 year ago
docker-compose.yml initial 1 year ago
docker-entrypoint.sh initial 1 year ago
feed_commands.go initial 1 year ago
go.mod initial 1 year ago
go.sum initial 1 year ago
jobs.go initial 1 year ago
main.go initial 1 year ago
parse_test.go initial 1 year ago
posts_test.go initial 1 year ago
scheduler.go initial 1 year ago
server.go initial 1 year ago

README.md

MIRRORED FROM: https://git.sp4ke.com/sp4ke/hugobot

HUGOBOT

hugobot is a an automated content fetch and aggregation bot for Hugo data driven websites. It has the following features:

Data fetch

  • Use the feeds table to register feeds that will periodically get fetched, stored and exported into the hugo project.
  • Currently handles these types of feeds: RSS, Github Releases, Newsletters
  • Define your own feed types by implementing the JobHandler interface (see handlers/handlers.go).
  • Hugobot automatically fetches new posts from the registered.
  • Sqlite is used for storage. feeds and posts tables.
  • The scheduler can handle any number of tasks and uses leveldb for caching/resuming jobs.

Hugo export

  • Data is automatically exported to the configured Hugo website path.
  • It can export markdown files or json/toml data files.
  • All fields in the exported files can be customized.
  • You can define custom output formats by using the FormatHandler interface.
  • You can register custom filters and post processing on exported posts to avoid changing the raw data stored in the db.
  • You can force data export using the CLI.

API

  • Uses gin-gonic.

  • hugobot also includes a webserver API that can be used with Hugo Data Driven Mode.

  • Insert and query data from the db. This is still a WIP, you can easily add the missing code on the API side to automate adding/querying data from the DB.

  • An example usage is the automated generation of Bitcoin addresses for new articles on bitcointechweekly.com

Other

  • Some commands are available through the CLI (github.com/urfave/cli), you can add your own custom commands.

Sqliteweb interface

  • See Docker files

First time usage

  • The database is automatically generated the first time you run the program. You can add your feeds straight into the sqlite db using your favorite sqlite GUI or the provided web gui in the docker-compose file.

Contribution

  • PRs welcome, current priority is to add tests.
  • Check the TODO section.

TODO:

  • Add tests.
  • Handle more feed formats: tweets, mailing-list emails
  • TLS support in the API (not a priority, can be done with a reverse proxy).