parent
53a2bf355b
commit
241a83281f
@ -0,0 +1,55 @@
|
||||
# About
|
||||
## What is this?
|
||||
|
||||
This is a repository of all the tools I use to build and run torrent-paradise.ml. Some people asked for a source, so I'm just putting this out here. I did make *some* effort to clean it up. The 'code name' of the project is nextgen (next gen torrent search), so don't be surprised if it comes up somewhere.
|
||||
|
||||
## Can you help me?
|
||||
Maybe, open an issue. Be sure to demonstrate an effort that you tried to solve the problem yourself.
|
||||
|
||||
## This is a big mess. Fix it maybe?
|
||||
WIP ❤️
|
||||
|
||||
# Setup
|
||||
|
||||
Here's what the setup looks like rn:
|
||||
- VPS, Debian Stretch, 2 GB RAM
|
||||
- PostgreSQL 9.6. pg_hba.conf contains this:
|
||||
|
||||
```
|
||||
local all all peer
|
||||
# IPv4 local connections:
|
||||
host nextgen nextgen localhost md5
|
||||
```
|
||||
- IPFS v0.4.18
|
||||
- user with username nextgen on the server
|
||||
- my laptop w/ Linux
|
||||
- Go toolchain installed
|
||||
- node v10.9.0 & npm
|
||||
|
||||
Schema for the database is sth like this (taken from index-generator/README, runs on sqlite, probably also on pg.)
|
||||
```sql
|
||||
CREATE TABLE peercount ( infohash char(40), tracker varchar, seeders int, leechers int, completed int, scraped timestamp);
|
||||
|
||||
CREATE TABLE torrent( infohash char(40), name varchar, length bigint, added timestamp);
|
||||
```
|
||||
|
||||
|
||||
|
||||
What I did first after getting the server up and running was importing the TPB dump. Download https://thepiratebay.org/static/dump/csv/torrent_dump_full.csv.gz to the import-tpb-dump directory and run `go run`.
|
||||
|
||||
I probably forgot sth. Open an issue!
|
||||
|
||||
# Usage
|
||||
|
||||
## Generate the index
|
||||
|
||||
This is a half-broken process that is partially described in update-index.sh. Read the script to understand what it does.
|
||||
|
||||
## Spider the DHT
|
||||
|
||||
Run `go build` in spider/ to compile and scp the binary it to the server. You can use the systemd service file in `spider/spider.service` to start the spider on boot.
|
||||
|
||||
|
||||
# Contributing
|
||||
|
||||
Before working on something, open an issue to ask if it would be okay. I would love to [KISS](https://en.wikipedia.org/wiki/KISS_principle).
|
@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=nextgen DHT spider
|
||||
Requires=postgresql
|
||||
|
||||
[Service]
|
||||
User=nextgen
|
||||
WorkingDirectory=/home/nextgen
|
||||
ExecStart=/home/nextgen/spider
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
@ -0,0 +1,5 @@
|
||||
script to see how fast the spiders are adding new torrents
|
||||
|
||||
select added::date, count(infohash)
|
||||
from torrent
|
||||
group by added::date;
|
@ -0,0 +1,33 @@
|
||||
# This script updates the index and pushes it to IPFS. Should be run often.
|
||||
|
||||
echo "Scraping trackers for seed/leech data"
|
||||
mosh nextgen@dev.ipfsearch.xyz "~/tracker-scraper"
|
||||
echo "Generating SQL dump"
|
||||
ssh nextgen@dev.ipfsearch.xyz pg_dump --data-only --inserts nextgen > index-generator/dump.sql
|
||||
|
||||
sed -i -e 's/public.peercount/peercount/g' index-generator/dump.sql
|
||||
sed -i -e 's/public.torrent/torrent/g' index-generator/dump.sql
|
||||
tail -n +21 index-generator/dump.sql > index-generator/newdump.sql # remove headers
|
||||
mv index-generator/newdump.sql index-generator/dump.sql
|
||||
rm index-generator/db.sqlite3
|
||||
echo """Do the following:
|
||||
$ sqlite3 index-generator/db.sqlite3
|
||||
|
||||
sqlite> CREATE TABLE peercount ( infohash char(40), tracker varchar, seeders int, leechers int, completed int, scraped timestamp, ws boolean);
|
||||
sqlite> CREATE TABLE torrent( infohash char(40), name varchar, length bigint, added timestamp);
|
||||
sqlite> BEGIN;
|
||||
sqlite> .read index-generator/dump.sql
|
||||
sqlite> END;"""
|
||||
bash
|
||||
echo "Generating index now..."
|
||||
cd index-generator
|
||||
node --max-old-space-size=10000 main.js
|
||||
cd ..
|
||||
echo "Check meta.json, add resultPage='resultpage', fix invURLBase, inxURLBase"
|
||||
nano website/generated/inx.meta.json
|
||||
echo "Uploading website"
|
||||
cd website
|
||||
scp -r . user@server:/www/torrent-paradise.ml
|
||||
echo "Finished uploading website to server. Adding to IPFS"
|
||||
ssh user@server sudo -u ipfs ipfs add -r /www/torrent-paradise.ml/
|
||||
echo "Check if it works, maybe publish to IPNS."
|
Loading…
Reference in New Issue