From 552fdaeb286f8b6c341a1372efb3102a8695aa00 Mon Sep 17 00:00:00 2001 From: Oliver Gugger Date: Fri, 19 Mar 2021 18:25:40 +0100 Subject: [PATCH 1/3] regtest: add demo/quick start regtest environment To help with development of apps that use the Loop daemon (`loopd`) for liquidity management, it is often useful to test it in controlled environments such as on the Bitcoin `regtest` network. Lightning Labs provides a stripped down version of the Loop server that works only for `regtest` and is only published in compiled, binary form (currently as a Docker image). The `docker-compose.yml` shows an example setup and is accompanied by a quick start script that should make getting started very easy. --- regtest/README.md | 188 +++++++++++++++++++++++++++++++++++++ regtest/docker-compose.yml | 117 +++++++++++++++++++++++ regtest/regtest.sh | 96 +++++++++++++++++++ 3 files changed, 401 insertions(+) create mode 100644 regtest/README.md create mode 100644 regtest/docker-compose.yml create mode 100755 regtest/regtest.sh diff --git a/regtest/README.md b/regtest/README.md new file mode 100644 index 0000000..4d6a5fc --- /dev/null +++ b/regtest/README.md @@ -0,0 +1,188 @@ +# Local regtest setup + +To help with development of apps that use the Loop daemon (`loopd`) for +liquidity management, it is often useful to test it in controlled environments +such as on the Bitcoin `regtest` network. + +Lightning Labs provides a stripped down version of the Loop server that works +only for `regtest` and is only published in compiled, binary form (currently as +a Docker image). + +The `docker-compose.yml` shows an example setup and is accompanied by a quick +start script that should make getting started very easy. + +## Requirements + +To get this quick start demo environment operational, you need to have the +following tools installed on your system: + - Docker + - `docker-compose` + - `jq` + - `git` + +## Starting the environment + +Simply download the Loop repository and run the `start` command of the helper +script. This will boot up the `docker-compose` environment, mine some `regtest` +coins, connect and create channels between the two `lnd` nodes and hook up the +Loop client and servers to each other. + +```shell +$ git clone https://github.com/lightninglabs/loop +$ cd loop/regtest +$ ./regtest.sh start +``` + +## Looping out + +Looping out is the process of moving funds from your channel (off-chain) balance +into your on-chain wallet through what is called a submarine swap. We use the +`--fast` option here, otherwise the server would wait 30 minutes before +proceeding to allow for batched swaps. + +```shell +$ ./regtest.sh loop out --amt 500000 --fast +Send off-chain: 500000 sat +Receive on-chain: 492688 sat +Estimated total fee: 7312 sat + +Fast swap requested. + +CONTINUE SWAP? (y/n): y +Swap initiated +ID: f96b2e40fd75a63eba61a6ee1e0b17788d77804c0454beb5fbbe4a42a8a06245 +HTLC address: bcrt1qwgzsmag0579twr4d8hgfvarqxn27mdmdgfh7wc2hdlupsfqtxwyq5zltqn + +Run `loop monitor` to monitor progress. +``` + +The swap has been initiated now. Let's observe how it works: + +```shell +$ ./regtest.sh loop monitor +Note: offchain cost may report as 0 after loopd restart during swap +2021-03-19T17:01:39Z LOOP_OUT INITIATED 0.005 BTC - bcrt1qwgzsmag0579twr4d8hgfvarqxn27mdmdgfh7wc2hdlupsfqtxwyq5zltqn +``` + +Leave the command running, it will update with every new event. Open a second +terminal window in the same folder/location and mine one block: + +```shell +$ ./regtest.sh mine 1 +``` + +You should see your previous window with the `monitor` command update: + +```shell +2021-03-19T17:01:39Z LOOP_OUT INITIATED 0.005 BTC - bcrt1qwgzsmag0579twr4d8hgfvarqxn27mdmdgfh7wc2hdlupsfqtxwyq5zltqn +2021-03-19T17:04:37Z LOOP_OUT PREIMAGE_REVEALED 0.005 BTC - bcrt1qwgzsmag0579twr4d8hgfvarqxn27mdmdgfh7wc2hdlupsfqtxwyq5zltqn +``` + +As soon as the `PREIMAGE_REVEALED` message appears, you know the HTLC was +published to the chain. Finish the process by mining yet another block (in the +second terminal window): + +```shell +$ ./regtest.sh mine 1 +``` + +Your `monitor` window should now show the completion of the swap: + +```shell +2021-03-19T17:01:39Z LOOP_OUT INITIATED 0.005 BTC - bcrt1qwgzsmag0579twr4d8hgfvarqxn27mdmdgfh7wc2hdlupsfqtxwyq5zltqn +2021-03-19T17:04:37Z LOOP_OUT PREIMAGE_REVEALED 0.005 BTC - bcrt1qwgzsmag0579twr4d8hgfvarqxn27mdmdgfh7wc2hdlupsfqtxwyq5zltqn +2021-03-19T17:06:25Z LOOP_OUT SUCCESS 0.005 BTC - bcrt1qwgzsmag0579twr4d8hgfvarqxn27mdmdgfh7wc2hdlupsfqtxwyq5zltqn (cost: server 50, onchain 6687, offchain 0) +``` + +Congratulations, you just did a loop out! + +# Loop in + +Looping in is very similar but does the opposite: swap your on-chain balance for +off-chain (channel) balance: + +```shell +$ ./regtest.sh loop in --amt 500000 +Send on-chain: 500000 sat +Receive off-chain: 492279 sat +Estimated total fee: 7721 sat + +CONTINUE SWAP? (y/n): y +Swap initiated +ID: d95167e33e79df292e5ffeda29d34614a5a087c9378ffae003b9a25bb0f1c2e4 +HTLC address (P2WSH): bcrt1q4p0pead4tfepm683fff8fl8g3kpx8spjztuquu2sctfzed9rufls2z0g2f + +Run `loop monitor` to monitor progress. + + +$ ./regtest.sh loop monitor +Note: offchain cost may report as 0 after loopd restart during swap +2021-03-19T17:17:13Z LOOP_IN HTLC_PUBLISHED 0.005 BTC - P2WSH: bcrt1q4p0pead4tfepm683fff8fl8g3kpx8spjztuquu2sctfzed9rufls2z0g2f +``` + +In your second terminal window, go ahead and mine a block: + +```shell +$ ./regtest.sh mine 1 +``` + +You should see your previous window with the `monitor` command update: + +```shell +2021-03-19T17:17:13Z LOOP_IN HTLC_PUBLISHED 0.005 BTC - P2WSH: bcrt1q4p0pead4tfepm683fff8fl8g3kpx8spjztuquu2sctfzed9rufls2z0g2f +2021-03-19T17:18:38Z LOOP_IN INVOICE_SETTLED 0.005 BTC - P2WSH: bcrt1q4p0pead4tfepm683fff8fl8g3kpx8spjztuquu2sctfzed9rufls2z0g2f (cost: server -499929, onchain 7650, offchain 0) +``` + +After mining yet another block, the process should complete: + +```shell +$ ./regtest.sh mine 1 +``` + +```shell +2021-03-19T17:17:13Z LOOP_IN HTLC_PUBLISHED 0.005 BTC - P2WSH: bcrt1q4p0pead4tfepm683fff8fl8g3kpx8spjztuquu2sctfzed9rufls2z0g2f +2021-03-19T17:18:38Z LOOP_IN INVOICE_SETTLED 0.005 BTC - P2WSH: bcrt1q4p0pead4tfepm683fff8fl8g3kpx8spjztuquu2sctfzed9rufls2z0g2f (cost: server -499929, onchain 7650, offchain 0) +2021-03-19T17:19:21Z LOOP_IN SUCCESS 0.005 BTC - P2WSH: bcrt1q4p0pead4tfepm683fff8fl8g3kpx8spjztuquu2sctfzed9rufls2z0g2f (cost: server 71, onchain 7650, offchain 0) +``` + +# Using the Loop server in an existing setup + +This `docker-compose` is only meant as a demo and quick start help. You can of +course also integrate the Loop server Docker image into your existing `regtest` +environment. + +Simply pull the image and run it by pointing it to an existing `lnd` node: + +```shell +$ docker pull lightninglabs/loopserver:latest +$ docker run -d \ + -p 11009:11009 \ + -v /some/dir/to/lnd:/root/.lnd \ + lightninglabs/loopserver:latest \ + daemon \ + --maxamt=5000000 \ + --lnd.host=some-lnd-node:10009 \ + --lnd.macaroondir=/root/.lnd/data/chain/bitcoin/regtest \ + --lnd.tlspath=/root/.lnd/tls.cert \ +``` + +An existing Loop client (`loopd`) can then be pointed to this server with: + +```shell +$ loopd \ + --network=regtest \ + --debuglevel=debug \ + --server.host=localhost:11009 \ + --server.notls \ + --lnd.host=some-other-lnd-node:10009 \ + --lnd.macaroonpath=/root/.lnd/data/chain/bitcoin/regtest/admin.macaroon \ + --lnd.tlspath=/root/.lnd/tls.cert +``` + +The `--server.notls` is important here as the `regtest` version of the Loop +server only supports insecure, non-TLS connections. + +Also make sure you connect the Loop server and Loop client to a different `lnd` +node each since an `lnd` node cannot pay itself. Obviously there also need to be +some channels with enough liquidity between the server's and client's `lnd` +nodes (direct or indirect doesn't matter). diff --git a/regtest/docker-compose.yml b/regtest/docker-compose.yml new file mode 100644 index 0000000..bbc0c0b --- /dev/null +++ b/regtest/docker-compose.yml @@ -0,0 +1,117 @@ +version: '3' +services: + bitcoind: + image: ruimarinho/bitcoin-core:0.21-alpine + restart: unless-stopped + networks: + - regtest + command: + - "-txindex" + - "-regtest" + - "-rest" + - "-printtoconsole" + - "-zmqpubrawblock=tcp://0.0.0.0:28332" + - "-zmqpubrawtx=tcp://0.0.0.0:28333" + - "-rpcport=18443" + - "-rpcbind=0.0.0.0" + # This is just the hashed string "lightning" with a salt. + - "-rpcauth=lightning:8492220e715bbfdf5f165102bfd7ed4$$88090545821ed5e9db614588c0afbad575ccc14681fb77f3cae6899bc419af67" + - "-rpcallowip=172.0.0.0/8" + - "-rpcallowip=127.0.0.1" + - "-fallbackfee=0.0002" + - "-peerblockfilters=1" + - "-blockfilterindex=1" + - "-wallet=/home/bitcoin/.bitcoin/regtest/wallets/miner" + environment: + - HOME=/home/bitcoin + + lndserver: + image: lightninglabs/lnd:v0.12.1-beta + restart: unless-stopped + networks: + - regtest + volumes: + - "lndserver:/root/.lnd" + depends_on: + - bitcoind + command: + - "--alias=lndserver" + - "--rpclisten=0.0.0.0:10009" + - "--noseedbackup" + - "--bitcoin.active" + - "--bitcoin.regtest" + - "--bitcoin.node=bitcoind" + - "--bitcoind.rpchost=regtest_bitcoind_1" + - "--bitcoind.rpcuser=lightning" + - "--bitcoind.rpcpass=lightning" + - "--bitcoind.zmqpubrawblock=tcp://regtest_bitcoind_1:28332" + - "--bitcoind.zmqpubrawtx=tcp://regtest_bitcoind_1:28333" + - "--tlsextradomain=regtest_lndserver_1" + + loopserver: + image: lightninglabs/loopserver:latest + restart: unless-stopped + networks: + - regtest + volumes: + - "lndserver:/root/.lnd" + depends_on: + - lndserver + command: + - "daemon" + - "--maxamt=5000000" + - "--lnd.host=regtest_lndserver_1:10009" + - "--lnd.macaroondir=/root/.lnd/data/chain/bitcoin/regtest" + - "--lnd.tlspath=/root/.lnd/tls.cert" + + lndclient: + image: lightninglabs/lnd:v0.12.1-beta + restart: unless-stopped + networks: + - regtest + volumes: + - "lndclient:/root/.lnd" + depends_on: + - bitcoind + command: + - "--alias=lndclient" + - "--rpclisten=0.0.0.0:10009" + - "--noseedbackup" + - "--bitcoin.active" + - "--bitcoin.regtest" + - "--bitcoin.node=bitcoind" + - "--bitcoind.rpchost=regtest_bitcoind_1" + - "--bitcoind.rpcuser=lightning" + - "--bitcoind.rpcpass=lightning" + - "--bitcoind.zmqpubrawblock=tcp://regtest_bitcoind_1:28332" + - "--bitcoind.zmqpubrawtx=tcp://regtest_bitcoind_1:28333" + - "--tlsextradomain=regtest_lndclient_1" + + loopclient: + image: loopd + build: + context: ../ + dockerfile: Dockerfile + restart: unless-stopped + networks: + - regtest + volumes: + - "lndclient:/root/.lnd" + depends_on: + - lndclient + command: + - "loopd" + - "--network=regtest" + - "--debuglevel=debug" + - "--server.host=regtest_loopserver_1:11009" + - "--server.notls" + - "--lnd.host=regtest_lndclient_1:10009" + - "--lnd.macaroonpath=/root/.lnd/data/chain/bitcoin/regtest/admin.macaroon" + - "--lnd.tlspath=/root/.lnd/tls.cert" + +networks: + regtest: + +volumes: + lndserver: + lndclient: diff --git a/regtest/regtest.sh b/regtest/regtest.sh new file mode 100755 index 0000000..339e615 --- /dev/null +++ b/regtest/regtest.sh @@ -0,0 +1,96 @@ +#!/bin/bash + +# The absolute directory this file is located in. +COMPOSE="docker-compose -p regtest" + +function bitcoin() { + docker exec -ti regtest_bitcoind_1 bitcoin-cli -regtest "$@" +} + +function lndserver() { + docker exec -ti regtest_lndserver_1 lncli --network regtest "$@" +} + +function lndclient() { + docker exec -ti regtest_lndclient_1 lncli --network regtest "$@" +} + +function loop() { + docker exec -ti regtest_loopclient_1 loop --network regtest "$@" +} + +function start() { + $COMPOSE up --force-recreate -d + echo "Waiting for nodes to start" + waitnodestart + setup +} + +function waitnodestart() { + while ! lndserver getinfo | grep -q identity_pubkey; do + sleep 1 + done + while ! lndclient getinfo | grep -q identity_pubkey; do + sleep 1 + done +} + +function mine() { + NUMBLOCKS=6 + if [ ! -z "$1" ] + then + NUMBLOCKS=$1 + fi + bitcoin generatetoaddress $NUMBLOCKS $(bitcoin getnewaddress "" legacy) > /dev/null +} + +function setup() { + echo "Getting pubkeys" + LNDSERVER=$(lndserver getinfo | jq .identity_pubkey -r) + LNDCLIENT=$(lndclient getinfo | jq .identity_pubkey -r) + echo "Getting addresses" + + echo "Creating wallet" + bitcoin createwallet miner + + ADDR_BTC=$(bitcoin getnewaddress "" legacy) + echo "Generating blocks to $ADDR_BTC" + bitcoin generatetoaddress 106 "$ADDR_BTC" > /dev/null + + echo "Sending funds" + ADDR_SERVER=$(lndserver newaddress p2wkh | jq .address -r) + ADDR_CLIENT=$(lndclient newaddress p2wkh | jq .address -r) + bitcoin sendtoaddress "$ADDR_SERVER" 5 + bitcoin sendtoaddress "$ADDR_CLIENT" 5 + mine 6 + + lndserver openchannel --node_key $LNDCLIENT --connect regtest_lndclient_1:9735 --local_amt 16000000 + mine 6 + + lndclient openchannel --node_key $LNDSERVER --local_amt 16000000 + mine 6 +} + +function stop() { + $COMPOSE down --volumes +} + +function restart() { + stop + start +} + +function info() { + LNDSERVER=$(lndserver getinfo | jq -c '{pubkey: .identity_pubkey, channels: .num_active_channels, peers: .num_peers}') + LNDCLIENT=$(lndclient getinfo | jq -c '{pubkey: .identity_pubkey, channels: .num_active_channels, peers: .num_peers}') + echo "lnd server: $LNDSERVER" + echo "lnd client: $LNDCLIENT" +} + +if [[ $# -lt 1 ]]; then + echo "Usage: $0 start|stop|restart|info|loop" +fi + +CMD=$1 +shift +$CMD "$@" From fea9e3e7d8027fe81a5e383b610584dbdbf2903c Mon Sep 17 00:00:00 2001 From: Oliver Gugger Date: Fri, 19 Mar 2021 18:28:44 +0100 Subject: [PATCH 2/3] README: add section describing the regtest Loop server --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 4ae5197..17f8474 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,12 @@ problems. Community support is also available in the [LND Slack](https://lightning.engineering/slack.html) . +### Regtest Loop server + +To get started with local development against a stripped down dummy Loop server +running in a local `regtest` Bitcoin network, take a look at the +[`regtest` server environment example documentation](./regtest/README.md). + ## Setup and Install LND and the loop client are using Go modules. Make sure that the `GO111MODULE` From e5a191443370451f9b4ad5256f200ec71ff76b22 Mon Sep 17 00:00:00 2001 From: Oliver Gugger Date: Fri, 19 Mar 2021 18:33:16 +0100 Subject: [PATCH 3/3] release_notes: add regtest Loop server --- release_notes.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/release_notes.md b/release_notes.md index debc18c..f557b3f 100644 --- a/release_notes.md +++ b/release_notes.md @@ -20,6 +20,11 @@ This file tracks release notes for the loop client. verbose info, giving users a more intuitive view about money paid on/off-chain and fees incurred. Use `loop in -v`, `loop out -v`, `loop quote in -v` or `loop quote out -v` to view the details. +* A stripped down version of the Loop server is now provided as a + [Docker image](https://hub.docker.com/r/lightninglabs/loopserver). A quick + start script and example `docker-compose` environment as well as + [documentation on how to use the `regtest` Loop server](https://github.com/lightninglabs/loop/blob/master/regtest/README.md) + was added too. #### Breaking Changes