From badf72838ed2691166ec4a192b84b987e41840e7 Mon Sep 17 00:00:00 2001 From: Jeff Date: Sat, 6 Aug 2022 09:34:05 -0400 Subject: [PATCH 1/5] move all code directory readmes into docs/project-structure.md --- contrib/readme.md | 75 -------------------------- daemon/readme.md | 10 ---- docs/project-structure.md | 110 ++++++++++++++++++++++++++++++++++++++ llarp/readme.md | 55 ------------------- 4 files changed, 110 insertions(+), 140 deletions(-) delete mode 100644 contrib/readme.md delete mode 100644 daemon/readme.md create mode 100644 docs/project-structure.md delete mode 100644 llarp/readme.md diff --git a/contrib/readme.md b/contrib/readme.md deleted file mode 100644 index 09cb78515..000000000 --- a/contrib/readme.md +++ /dev/null @@ -1,75 +0,0 @@ -grab bag directory for non core related platform specific non source code - -[Network Manager shims](NetworkManager) - -[apparmor profiles](apparmor) - -[default self signed network bootstrap files](bootstrap) - -[continuous integration scripts](ci) - -[cross compile cmake toolchains](cross) - -[embedded lokinet example code](liblokinet) - -[macos related hacks and associated hellscape items](macos) - -[static deps patch files](patches) - -[python scripts](py) - -[systemd-resolved profiles](systemd-resolved) - -android build shim: - -* android.sh -* android-configure.sh - -windows via mingw cross compiler build shim: - -* windows.sh -* windows-configure.sh - -macos build shim: - -* mac.sh - -iphone build shim: - -* ios.sh - -linux cross compile build shim: - -* cross.sh - -apply multiple patches script: - -* apply-patches.sh - -[bencode](https://www.bittorrent.org/beps/bep_0003.html#bencoding) pretty-print script: - -* bencode-dump.py - -deb.oxen.io apt repo pubkey: - -* deb.oxen.io.gpg - -clang-format / jsonnetfmt / swiftformat helper, will check or correct code style: - -* format.sh - -git hooks: - -* git-hook-pre-push.sh - -base16 to [z-base32](https://philzimmermann.com/docs/human-oriented-base-32-encoding.txt) converter - -* hex-to-base32z.py - -lokinet logo: - -* lokinet.svg - -cpack installer text: - -* readme-installer.txt diff --git a/daemon/readme.md b/daemon/readme.md deleted file mode 100644 index 32bbf198a..000000000 --- a/daemon/readme.md +++ /dev/null @@ -1,10 +0,0 @@ -this contains the main functions for all of our executables - -lokinet.cpp - lokinet full daemon - -lokinet.swift - macos sysex/appex - -lokinet-bootstrap.cpp - lokinet-bootstrap binary (deprecated / to be removed) - -lokinet-vpn.cpp - lokinet-vpn tool for rpc control of lokinet, needs a better name. - diff --git a/docs/project-structure.md b/docs/project-structure.md new file mode 100644 index 000000000..6415a69f9 --- /dev/null +++ b/docs/project-structure.md @@ -0,0 +1,110 @@ +# Lokinet project structure + +this codebase is a bit large. this is a high level map of the current code structure. + +## lokinet executable main functions `(/daemon)` + +* `lokinet.cpp`: lokinet daemon executable +* `lokinet.swift`: macos sysex/appex executable +* `lokinet-vpn.cpp`: lokinet rpc tool for controlling exit node usage +* `lokinet-bootstrap.cpp`: legacy util for windows, downloads a bootstrap file via https + + +## lokinet public headers `(/include)` + +`lokinet.h and lokinet/*.h`: C headers for embedded lokinet + +`llarp.hpp`: semi-internal C++ header for lokinet executables + + +## lokinet core library `(/llarp)` + +* `/llarp`: contains a few straggling compilation units +* `/llarp/android`: android platform compat shims +* `/llarp/apple`: all apple platform specific code +* `/llarp/config`: configuration structs, generation/parsing/validating of config files +* `/llarp/consensus`: network consenus and inter relay testing +* `/llarp/constants`: contains all compile time constants +* `/llarp/crypto`: cryptography interface and implementation, includes various secure helpers +* `/llarp/dht`: dht message structs, parsing, validation and handlers of dht related parts of the protocol +* `/llarp/dns`: dns subsytem, dns udp wire parsers, resolver, server, rewriter/intercepter, the works +* `/llarp/ev`: event loop interfaces and implementations +* `/llarp/exit`: `.snode` endpoint "backend" +* `/llarp/handlers`: packet endpoint "frontends" +* `/llarp/iwp`: "internet wire protocol", hacky homegrown durable udp wire protocol used in lokinet +* `/llarp/link`: linklayer (node to node) communcation subsystem +* `/llarp/messages`: linklayer message parsing and handling +* `/llarp/net`: wrappers and helpers for ip addresses / ip ranges / sockaddrs, hides platform specific implemenation details +* `/llarp/path`: onion routing path logic, both client and relay side, path selection algorithms. +* `/llarp/peerstats`: deprecated +* `/llarp/quic`: plainquic shims for quic protocol inside lokinet +* `/llarp/router`: the relm of the god objects +* `/llarp/routing`: routing messages (onion routed messages sent over paths), parsing, validation and handler interfaces. +* `/llarp/rpc`: lokinet zmq rpc server and zmq client for externalizing logic (like with blockchain state and custom `.loki` endpoint orchestration) +* `/llarp/service`: `.loki` endpoint "backend" +* `/llarp/simulation`: network simulation shims +* `/llarp/tooling`: network simulation tooling +* `/llarp/util`: utility function dumping ground +* `/llarp/vpn`: vpn tunnel implemenation for each supported platform +* `/llarp/win32`: windows specific code + + +## component relations + +### `/llarp/service` / `/llarp/handlers` / `/llarp/exit` + +for all codepaths for traffic over lokinet, there is 2 parts, the "frontend" and the "backend". + +the "backend" is responsible for sending and recieving data inside lokinet using our internal formats via paths, it handles flow management, lookups, timeouts, handover, and all state we have inside lokinet. + +the "fontend", is a translation layer that takes in IP Packets from the OS, and send it to the backend to go where ever it wants to go, and recieves data from the "backend" and sends it to the OS as an IP Packet. + +there are 2 'backends': `.snode` and `.loki` + +there are 2 'frontends': "tun" (generic OS vpn interface) and "null" (does nothing) + +* `//TODO: the backends need to be split up into multiple sub components as they are a kitchen sink.` +* `//TODO: the frontends blend into the backend too much and need to have their boundery clearer.` + + +### `/llarp/ev` / `/llarp/net` / `/llarp/vpn` + +these contain most of the os/platform specific bits + +* `//TODO: untangle these` + + +### `/llarp/link` / `/llarp/iwp` + +node to node traffic logic and wire protocol dialects + +* `//TODO: make better definitions of interfaces` +* `//TODO: separte implementation details from interfaces` + + +## platform contrib code `(/contrib)` + +grab bag directory for non core related platform specific non source code + +* `/contrib/format.sh`: clang-format / jsonnetfmt / swiftformat helper, will check or correct code style. + +system layer and packaging related: + +* `/contrib/NetworkManager` +* `/contrib/apparmor` +* `/contrib/systemd-resolved` +* `/contrib/lokinet-resolvconf` +* `/contrib/bootstrap` + +build shims / ci helpers + +* `/contrib/ci` +* `/contrib/patches` +* `/contrib/cross` +* `/contrib/android.sh` +* `/contrib/android-configure.sh` +* `/contrib/windows.sh` +* `/contrib/windows-configure.sh` +* `/contrib/mac.sh` +* `/contrib/ios.sh` +* `/contrib/cross.sh` diff --git a/llarp/readme.md b/llarp/readme.md deleted file mode 100644 index 6861afae4..000000000 --- a/llarp/readme.md +++ /dev/null @@ -1,55 +0,0 @@ -this directory contains the meat of the lokinet core implemenation - -components: - -[apple platform specific bits](apple) - -[configuration bits](config) - -[network consensus parts](consensus) - -[compile time constants](constants) - -[cryptography interface and implementations](crypto) - -[DHT related code](dht) - -[DNS client/server/parsing library](dns) - -[event loop interface and implementations](ev) - -[snode endpoint backend implementation](exit) - -[onion endpoint frontend adapters](handlers) - -[lokinet wire protocol](iwp) - -[lokinet link layer interface types](link) - -[link layer (node to node) messages](messages) - -[platform agnostic-ish network api](net) - -[onion path management](path) - -[peer stats collection (likely to be refactored)](peerstats) - -[quic over lokinet abstraction](quic) - -[central god objects](router) - -[routing (onion routed over paths) messages](routing) - -[rpc client/server](rpc) - -[onion endpoint backend implemenation](service) - -[net simulation code (likely to be binned)](simulation) - -[net simulation tooling (used by pybind)](tooling) - -[utility function dumping ground (to be fixed up)](util) - -[platform specific vpn bits with high level abstractions](vpn) - -[windows platform bits](win32) From f3533e9912926e9cdab96c3d9cdc9212473add6b Mon Sep 17 00:00:00 2001 From: Jeff Date: Sat, 6 Aug 2022 09:58:20 -0400 Subject: [PATCH 2/5] add initial high level usage docs --- docs/ideal-ux.md | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/docs/ideal-ux.md b/docs/ideal-ux.md index 2050892d5..e0706d2f9 100644 --- a/docs/ideal-ux.md +++ b/docs/ideal-ux.md @@ -1,3 +1,38 @@ -# How Do I use lokinet? +# What does Lokinet actually do? -`// TODO: this` +Lokinet is an onion routed authenticated unicast IP network. It exposes an IP tunnel to the user and provides a dns resolver that maps `.loki` and `.snode` gtld onto a user defined ip range. + +Lokinet allows users to tunnel arbitrary ip ranges to go to a `.loki` address to act as a tunnel broker via another network accessible via another lokinet client. This is commonly known as an "exit node" but the way lokinet does this is much more generic so that term is not very accurate given what it actually does. + +# How Do I use Lokinet? + +set system dns resolver to use the dns resolver provided by lokinet, make sure the upstream dns provider that lokinet uses for non lokinet gtlds is set as desired (see lokinet.ini `[dns]` section) + +configure exit traffic provider if you want to tunnel ip traffic via lokinet, by default this is off as we cannot provide a sane defualt that makes everyone happy. to enable an exit node, see lokinet.ini `[network]` section, add multiple `exit-node=exitaddrgoeshere.loki` lines for each endpoint you want to use for exit traffic. each `exit-node` entry will be used to randomly stripe across per IP you are sending to. + +note: per flow (ip+proto/port) isolation is trivial on a technical level but currently not implemented at this time. + +# Can I run lokinet on a soho router + +Yes and that is the best way to run it in practice. + +## The "easy" way + +We have a community maintained solution for ARM SBCs like rasperry pi: https://github.com/necro-nemesis/LabyrinthAP + +## The "fun" way (DIY) + +It is quite nice to DIY. if you choose to do so there is some assembly required: + +on the lokinet side, make sure that the... + +* ip ranges for `.loki` and `.snode` are statically set (see lokinet.ini `[network]` section `ifaddr=` option) +* network interace used by lokinet is statically set (see lokinet.ini `[network]` section `ifname=` option) +* dns socket is bound to an address the soho router's dns resolver can talk to, see `[dns]` section `bind=` option) + +on the soho router side: + +* route queries for `.loki` and `.snode` gtld to go to lokinet dns on soho router's dns resolver +* use dhcp options to set dns to use the soho router's dns resolver +* make sure that the ip ranges for lokinet are reachable via the LAN interface +* if you are tunneling over an exit ensure that LAN traffic will only forward to go over the lokinet vpn interface From f6613c9526477198b424d5b2d2671d20c682045e Mon Sep 17 00:00:00 2001 From: Jeff Date: Sat, 6 Aug 2022 10:09:42 -0400 Subject: [PATCH 3/5] wire up new docs pages reword a few things about the links rewired. --- docs/readme.md | 8 +++++--- readme.md | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/readme.md b/docs/readme.md index 1b132b26f..1bb99f2e9 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -10,6 +10,11 @@ This is where Lokinet documentation lives. [What Lokinet can't do](we-cannot-make-sandwiches.md) +## Lokinet Internals + +[High level layout of the git repo](project-structure.md) + +[Build Doxygen Docs for internals](doxygen.md) ## Lokinet (SN)Application Developer Portal @@ -19,7 +24,4 @@ This is where Lokinet documentation lives. [How do I embed lokinet into my application?](liblokinet-dev-guide.md) -## Lokinet Internals - -[Build Doxygen Docs for internals](doxygen.md) diff --git a/readme.md b/readme.md index fd5d51cd0..b1b128c2b 100644 --- a/readme.md +++ b/readme.md @@ -4,7 +4,7 @@ Lokinet is the reference implementation of LLARP (low latency anonymous routing protocol), a layer 3 onion routing protocol. -You can learn more about the high level design of LLARP [here](docs/) +You can learn more about the high level, how to use it and the internals of the protocol [here](docs/) [![Build Status](https://ci.oxen.rocks/api/badges/oxen-io/lokinet/status.svg?ref=refs/heads/dev)](https://ci.oxen.rocks/oxen-io/lokinet) From 256470229d3c7a21776927f2041c05852057a200 Mon Sep 17 00:00:00 2001 From: majestrate Date: Mon, 8 Aug 2022 21:28:07 -0400 Subject: [PATCH 4/5] add more info clarifiy what .loki and .snode gtld are --- docs/ideal-ux.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/ideal-ux.md b/docs/ideal-ux.md index e0706d2f9..7cda8e29a 100644 --- a/docs/ideal-ux.md +++ b/docs/ideal-ux.md @@ -4,6 +4,10 @@ Lokinet is an onion routed authenticated unicast IP network. It exposes an IP tu Lokinet allows users to tunnel arbitrary ip ranges to go to a `.loki` address to act as a tunnel broker via another network accessible via another lokinet client. This is commonly known as an "exit node" but the way lokinet does this is much more generic so that term is not very accurate given what it actually does. +The `.snode` gtld refers to a router on the network by its public ed25519 key. + +The `.loki` gtld refers to clients that publish the existence anonymously to the network by their ed25519 public key. (`.loki` also has the ability to use short names resolved via external consensus method, like a blockchain). + # How Do I use Lokinet? set system dns resolver to use the dns resolver provided by lokinet, make sure the upstream dns provider that lokinet uses for non lokinet gtlds is set as desired (see lokinet.ini `[dns]` section) From 6929a02842262c4c29705ddd99a07a4874d83a03 Mon Sep 17 00:00:00 2001 From: jeff Date: Fri, 12 Aug 2022 09:58:54 -0700 Subject: [PATCH 5/5] set up links to new docs pages --- docs/dns-overview.md | 36 ++++++++++++++++++++++++++++++++++++ docs/high-level-overview.md | 19 +++++++++++++++++++ docs/readme.md | 5 +++++ 3 files changed, 60 insertions(+) create mode 100644 docs/dns-overview.md create mode 100644 docs/high-level-overview.md diff --git a/docs/dns-overview.md b/docs/dns-overview.md new file mode 100644 index 000000000..d3fcab22a --- /dev/null +++ b/docs/dns-overview.md @@ -0,0 +1,36 @@ +# DNS in Lokinet + +Lokinet uses dns are its primary interface for resolving, mapping and querying resources inside of lokinet. +This was done not because DNS is *good* protocol, but because there is almost no relevent userland applications that are incapable of interacting with DNS, across every platform. +Using DNS in lokinet allows for the most zero config setup possible with the current set of standard protocols. + +Lokinet provides 2 internal gtld, `.loki` and `.snode` + +## .snode + +The `.snode` gtld is used to address a lokinet router in the form of `.snode`. +Traffic bound to a `.snode` tld will have its source authenticatable only if it originates from another valid lokinet router. +Clients can also send traffic to and from addresses mapped to `.snode` addresses, but the source address on the service node side is ephemeral. +In both cases, ip traffic to addresses mapped to `.snode` addresses will have the destination ip rewritten by the lokinet router to be its local interface ip, this ensures traffic stays on the lokinet router' interface for snode traffic and preventing usage as an exit node. + +## .loki + +The `.loki` gtld is used to address anonymously published routes to lokinet clients on the network. + + + +## What RR are provided? + +All `.loki` domains by default have the following dns rr synthesized by lokinet: + +* `A` record for initiating address mapping +* `MX` record pointing to the synthesizesd `A` record +* free wildcard entries for all of the above. + +Wildard entries are currently only pointing + +All `.snode` domains have by defult just an `A` record for initiating address mapping. + +Additionally both `.loki` and `.snode` can optionally provide multiple `SRV` records to advertise existence of services on or off of the name. + + diff --git a/docs/high-level-overview.md b/docs/high-level-overview.md new file mode 100644 index 000000000..1d2baea84 --- /dev/null +++ b/docs/high-level-overview.md @@ -0,0 +1,19 @@ +## onion routing overview + + + + + + + +## endpoint zmq api + + + +## DNS + + + + + + diff --git a/docs/readme.md b/docs/readme.md index 1bb99f2e9..3f113f5b7 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -6,14 +6,19 @@ This is where Lokinet documentation lives. [How is Lokinet different to \[insert network technology name here\] ?](net-comparisons.md) + + [How Do I use Lokinet?](ideal-ux.md) +[Lokinet and DNS](dns-overview.md) + [What Lokinet can't do](we-cannot-make-sandwiches.md) ## Lokinet Internals [High level layout of the git repo](project-structure.md) + [Build Doxygen Docs for internals](doxygen.md) ## Lokinet (SN)Application Developer Portal