@ -10,7 +10,7 @@ Today, in 2021, the Lightning Network is still in its infancy. The Lightning Net
The concept of the Lightning Network was proposed in 2015, and the first implementation was launched in 2018. As of 2021, we're only beginning to see the opportunities the Lightning Network provides to Bitcoin including improved privacy, speed, and scale.
With core knowledge of the Lightning Network, you can help shape the future of the network while also building opportunities for yourself.
We assume you already have some basic knowledge about Bitcoin, but if not, don't worry – we will explain the most important Bitcoin concepts, those you must know to understand the Lightning Network, in <<bitcoin_fundamentals_review>>. If you want to learn more about Bitcoin, you can read _Mastering Bitcoin_, available for free online.
We assume you already have some basic knowledge about Bitcoin, but if not, don't worry – we will explain the most important Bitcoin concepts, those you must know to understand the Lightning Network, in XREF HERE. If you want to learn more about Bitcoin, you can read _Mastering Bitcoin_, available for free online.
While the bulk of this book is written for programmers, the first few chapters are written to be approachable by anyone regardless of technical experience. In this chapter, we'll start with some terminology, then move to look at trust and its application in these systems, and finally we'll discuss the history and future of the Lightning Network. Let's get started.
@ -47,7 +47,7 @@ Throughout this book, you will see "Bitcoin" with the first letter capitalized,
Now that you're familiar with these basic terms, let's move to a concept you are already comfortable with: trust.
=== Trust in decentralized networks
=== Trust in Decentralized Networks
You will often hear people calling Bitcoin and the Lightning Network "trustless". At first glance this is confusing. After all, isn't trust a good thing? Banks even use it in their names! Isn't a "trustless" system, a system devoid of trust, a bad thing?
@ -60,7 +60,7 @@ Trustless does not mean devoid of trust. It means that trust is not a necessary
Before we get into how the Lightning Network works, it's important to understand one basic concept that underlies Bitcoin, the Lightning Network and many other such systems: something we call a _fairness protocol_. A fairness protocol is a way to _achieve fair outcomes between participants, who do not need to trust each other, without the need for a central authority_ and it is the backbone of decentralized systems like Bitcoin.
=== Fairness without central authority
=== Fairness Without Central Authority
When people have competing interests, how can they establish enough trust to engage in some cooperative or transactional behavior? The answer to this question lies at the core of several scientific and humanistic disciplines, such as economics, sociology, behavioral psychology, and mathematics. Some of those disciplines give us "soft" answers that depend on concepts such as reputation, fairness, morality, and even religion. Other disciplines give us concrete answers that depend only on the assumption that the participants in these interactions will act rationally, with their self-interest as the main objective.
@ -74,7 +74,7 @@ In broad terms, there are a handful of ways to ensure fair outcomes in interacti
* Game theoretical fairness protocols - this last category emerges from the combination of the internet and cryptography and is the subject of this section. Let's see how it works and what its advantages and disadvantages are.
==== Trusted protocols without intermediaries
==== Trusted Protocols Without Intermediaries
Cryptographic systems like Bitcoin and the Lightning Network are systems that allow you to transact with people (and computers) that you don't trust. This is often referred to as "trustless" operation, even though it is not actually trustless. You have to trust in the software that you run, and you have to trust that the protocol implemented by that software will result in fair outcomes.
@ -88,7 +88,7 @@ While game theory and its use in cryptographic systems may appear confounding an
In this book, we call this pattern a _Fairness Protocol_ defined as a process that uses a system of incentives and/or disincentives to ensure fair outcomes for participants who don't trust each other. Enforcement of a fairness protocol is only necessary to ensure that the participants can't escape the incentives or disincentives.
==== A fairness protocol in action
==== A Fairness Protocol in Action
Let's look at an example of a fairness protocol, which you may already be familiar with.
@ -105,7 +105,7 @@ A much better solution exists: the siblings are taught to play a game called "sp
While the infamous chip battles of the 1980's neatly illustrate the point, any similarity between the scenario above and any of the authors' actual childhood experiences with their cousins is entirely coincidental... Or is it?
====
==== Security primitives as building blocks
==== Security Primitives as Building Blocks
In order for a fairness protocol like this to work, there need to be certain guarantees, or _security primitives_, that can be combined to ensure enforcement. The first security primitive is _strict time ordering/sequencing_: the "splitting" action must happen before the "choosing" action. It's not immediately obvious, but unless you can guarantee that action A happens before action B, then the protocol falls apart. The second security primitive is _commitment with non-repudiation_. Each sibling must commit to their choice of role: either splitter or chooser. Also, once the splitting has been completed, the splitter is committed to the split they created - they cannot repudiate that choice and go try again.
@ -121,7 +121,7 @@ In our real-life example, we saw one form of fairness protocol called "split and
Now that you understand this basic pattern, you will start seeing it everywhere in Bitcoin, the Lightning Network and many other systems. Let's look at some specific examples next.
==== Example of the fairness protocol
==== Example of the Fairness Protocol
The most prominent example of a "fairness protocol" is Bitcoin's consensus algorithm, _Proof of Work_ (PoW). In Bitcoin, miners compete to verify transactions and aggregate them in blocks. To ensure that the miners do not cheat, without entrusting them with authority, Bitcoin uses a system of incentives and disincentives. Miners have to use electricity and dedicate hardware doing "work" that is embedded as a "proof" inside every block. This is achieved because of a property of hash functions where the output value is randomly distributed across the entire range of possible outputs. If miners succeed in producing a valid block fast enough, they are rewarded by earning the block reward for that block. Forcing miners to use a lot of electricity before the network considers their block means that they have an incentive to correctly validate the transactions in the block. If they cheat or make any kind of mistake, their block is rejected and the electricity they used to "prove" it is wasted. No one needs to force miners to produce valid blocks; the reward and punishment incentivize them to do so. All the protocol needs to do is ensure that only valid blocks with proof of work are accepted.
@ -191,7 +191,7 @@ Aside from reducing the burden on nodes, payments on the Lightning Network are c
While the Lightning Network was initially conceived for Bitcoin, it can be implemented on any blockchain that meets some basic technical requirements. Other blockchains, such as Litecoin, already support the Lightning Network. Additionally, several other blockchains are developing similar "second layer" or "layer 2" solutions to help them scale.
=== The Lightning Network's defining features
=== The Lightning Network's Defining Features
The Lightning Network is a network that operates as a "second layer" protocol on top of Bitcoin and other blockchains. The Lightning Network enables fast, secure, private, trustless, and permissionless payments. Here are some of the features of the Lightning Network:
In this chapter, we will begin where most people start when encountering the Lightning Network for the first time - choosing software to participate in the Lightning Network economy. We will examine the choices of two users who represent a common use-case for the Lightning Network and learn by example. Alice, a coffee shop customer, will be using a Lightning wallet on her mobile device to buy coffee from Bob's Cafe. Bob, a merchant, will be using a Lightning node and wallet to run a point-of-sale system at his cafe, so he can accept payments over the Lightning Network.
=== Alice's first Lightning wallet
=== Alice's First Lightning Wallet
Alice is a long time Bitcoin user. We first met Alice in Chapter 1 of _"Mastering Bitcoin"_ footnote:["Mastering Bitcoin 2nd Edition, Chapter 1" Andreas M. Antonopoulos (https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch01.asciidoc).], when she bought a cup of coffee from Bob's cafe using a Bitcoin transaction. If you are not yet familiar with how Bitcoin transactions work or need a refresher, please read Mastering Bitcoin or the summary in <<bitcoin_fundamentals_review>>.
@ -11,14 +12,14 @@ Alice recently learned that Bob's Cafe just started accepting Lightning Network
Alice does not want to entrust custody of her bitcoin to third parties. She has learned enough about cryptocurrency to know how to use a wallet. She also wants a mobile wallet so that she can use it for small payments on-the-go, so she chooses the _Eclair_ wallet, a popular non-custodial mobile Lightning wallet. Let's learn more about how and why she's made these choices.
=== Lightning nodes
=== Lightning Nodes
The Lightning Network is accessed via software applications that can speak the Lightning Network protocol. A _Lightning Network node_ (or simply "node") is a software application that has three important characteristics. First, Lightning nodes are "wallets", so they send and receive payments over the Lightning Network as well as on the Bitcoin network. Second, nodes must communicate on a peer-to-peer basis with other Lightning nodes creating the network. Finally, Lightning nodes also need access to the Bitcoin blockchain (or other blockchains for other cryptocurrencies) to secure the funds used for payments.
Users have the highest degree of control by running their own Bitcoin node and Lightning node. However, Lightning nodes can also use a lightweight Bitcoin client, commonly referred to as Simplified Payment Verification (SPV), to interact with the Bitcoin blockchain.
[[ln_explorer]]
=== Lightning explorers
=== Lightning Explorers
Lightning network explorers are useful tools to show the statistics of nodes, channels and network capacity.
@ -41,7 +42,7 @@ Use Lightning explorers to visualize the network or gather information, but not
To have an authoritative view of the Lightning Network, run your own Lightning node that will build a channel graph and collect various statistics, which you can view with a web-based interface.
====
=== Lightning wallets
=== Lightning Wallets
The term "Lightning wallet" is somewhat ambiguous, as it can describe a broad variety of components combined with some user interface. The most common components of Lightning wallet software include:
All of the examples in this book can be replicated exactly on testnet with TBTC, so you can follow along if you want without risking real money.
=== Balancing complexity and control
=== Balancing Complexity and Control
Lightning wallets have to strike a careful balance between complexity and user control. Those that give the user the most control over their funds, the highest degree of privacy, and the greatest independence from third party services are necessarily more complex and difficult to operate. As the technology advances, some of these trade-offs will become less stark, and users may be able to get more control without more complexity. However, for now, different companies and projects are exploring different positions along this control-complexity spectrum and hoping to find the "sweet spot" for the users they are targeting.
@ -159,7 +160,7 @@ When selecting a wallet, keep in mind that even if you don't see these trade-off
In the next section, we will return to our first user and walk through her first Lightning wallet setup. She has chosen a wallet that is more sophisticated than the easier custodial wallets. This allows us to show some of the underlying complexity and introduce some of the inner workings of an advanced wallet. You may find that your first ideal wallet is further towards "ease of use", accepting some of the control and privacy trade-offs. Or perhaps you are more of a "power user" and want to run your own Lightning and Bitcoin nodes as part of your wallet solution.
=== Downloading and installing a Lightning wallet
=== Downloading and Installing a Lightning Wallet
When looking for a new cryptocurrency wallet, you must be very careful to select a secure source for the software.
@ -169,7 +170,7 @@ Alice uses an Android device and will use the Google Play Store to download and
[[eclair-playstore]]
.Eclair Mobile in the Google Play Store
image::images/eclair-playstore.png["Eclair wallet in the Google Play Store"]
image::images/mtln_0201.png["Eclair wallet in the Google Play Store"]
[TIP]
@ -184,11 +185,11 @@ Alice notices a few different elements on this page, that help her ascertain tha
Always exercise great care when installing software on any device. There are many fake cryptocurrency wallets that will not only steal your money but might also compromise all other applications on your device.
====
=== Creating a new wallet
=== Creating a New Wallet
When Alice opens the Eclair Mobile app for the first time, she is presented with a choice to "Create a New Wallet" or to "Import an Existing Wallet". Alice will create a new wallet, but let's first discuss why these options are presented here and what it means to "import an existing wallet".
==== Responsibility with key custody
==== Responsibility with Key Custody
As we mentioned at the beginning of this section, Eclair is a _non-custodial_ wallet, meaning that Alice has sole custody of the keys used to control her bitcoin. This also means that Alice is responsible for protecting and backing up those keys. If Alice loses the keys, no one can help her recover the bitcoin, and they will be lost forever.
@ -197,7 +198,7 @@ As we mentioned at the beginning of this section, Eclair is a _non-custodial_ wa
With the Eclair Mobile wallet, Alice has custody and control of the keys and, therefore, full responsibility to keep the keys safe and backed up. If she loses the keys, she loses the bitcoin, and no one can help her recover from that loss!
====
==== Mnemonic words
==== Mnemonic Words
Similar to most Bitcoin wallets, Eclair Mobile provides a _mnemonic phrase_ (also sometimes called a "seed" or "seed phrase") for Alice to back up. The mnemonic phrase consists of 24 English words, selected randomly by the software, and used as the basis for the keys that are generated by the wallet. The mnemonic phrase can be used by Alice to restore all the transactions and funds in the Eclair Mobile wallet in the case of an event such as a lost mobile device, a software bug, or memory corruption.
@ -210,12 +211,12 @@ When Alice chooses to "Create a New Wallet", she will be shown a screen with her
In <<eclair-mnemonic>>, we have purposely obscured part of the mnemonic phrase to prevent readers of this book from reusing the mnemonic.
[[mnemonic-storage]]
==== Storing the mnemonic safely
==== Storing the Mnemonic Safely
Alice needs to be careful to store the mnemonic phrase in a way that prevents theft but also avoids accidental loss. The recommended way to properly balance these risks is to write two copies of the mnemonic phrase on paper, with each of the words numbered - the order matters.
@ -230,7 +231,7 @@ Never attempt a "DIY" security scheme that deviates in any way from the best pra
After Alice initializes her Eclair Mobile wallet, she will see a brief tutorial that highlights the various elements of the user interface. We won't replicate the tutorial here, but we will explore all of those elements as we follow Alice's attempt to buy a cup of coffee!
=== Loading bitcoin onto the wallet
=== Loading Bitcoin Onto the Wallet
Alice now has a Lightning wallet. But, it's empty! She now faces one of the more challenging aspects of this experiment; she has to find a way to acquire some bitcoin and load it onto her Eclair wallet.
@ -240,7 +241,7 @@ If Alice already has bitcoin in another wallet, she could choose to send that bi
====
[[acquiring-bitcoin]]
==== Acquiring bitcoin
==== Acquiring Bitcoin
There are several ways Alice can acquire bitcoin:
@ -252,13 +253,13 @@ There are several ways Alice can acquire bitcoin:
All of these methods have varying degrees of difficulty, and many will involve paying a fee. Some will also require Alice to provide identification documents to comply with local banking regulations. However, with all these methods, Alice will be able to receive bitcoin.
==== Receiving bitcoin
==== Receiving Bitcoin
Let's assume Alice has found a local Bitcoin ATM and has decided to buy some bitcoin in exchange for cash. An example of a Bitcoin ATM, one built by the Lamassu company, is shown in <<bitcoin-atm>>. Such Bitcoin ATMs accept national currency (cash) through a cash slot and send bitcoin to a Bitcoin address scanned from a user's wallet using a built-in camera.
To receive the bitcoin in her Eclair Lightning wallet, Alice will need to present a Bitcoin address from the Eclair Lightning wallet to the ATM. The ATM can then send Alice's newly acquired bitcoin to this Bitcoin address.
@ -266,7 +267,7 @@ To see a Bitcoin address on the Eclair wallet, Alice must swipe to the left colu
[[eclair-receive]]
.Alice's bitcoin address, shown in Eclair
image::images/eclair-receive.png["Eclair bitcoin address QR code"]
image::images/mtln_0204.png["Eclair bitcoin address QR code"]
The QR code contains the same string of letters and numbers as shown below it, in an easy to scan format. This way, Alice doesn't have to type the Bitcoin address. In the screenshot <<eclair-receive>>, we have purposely blurred both, to prevent readers from inadvertently sending bitcoin to this address.
@ -279,7 +280,7 @@ Alice can take her mobile device to the ATM and show it to the built-in camera,
[[bitcoin-atm-receive]]
.Bitcoin ATM scans the QR code.
image::images/bitcoin-atm-receive.png["Bitcoin ATM scans the QR code"]
image::images/mtln_0205.png["Bitcoin ATM scans the QR code"]
Alice will see the transaction from the ATM in the "TRANSACTION HISTORY" tab of the Eclair wallet. While Eclair will detect the bitcoin transaction in just a few seconds, it will take approximately one hour for the bitcoin transaction to be "confirmed" on the Bitcoin blockchain. As you can see in <<eclair-tx1>>, Alice's Eclair wallet shows "6+ conf" below the transaction, indicating that the transaction has received the required minimum of six confirmations, and her funds are now ready to use.
@ -291,7 +292,7 @@ The number of "confirmations" on a transaction is the number of blocks mined sin
While in this example Alice used an ATM to acquire her first bitcoin, the same basic concepts would apply even if she used one of the other methods in <<acquiring-bitcoin>>. For example, if Alice wanted to sell a product or provide a professional service in exchange for bitcoin, her customers could scan the Bitcoin address with their wallets and pay her in bitcoin.
@ -304,7 +305,7 @@ Fortunately, the Lightning Network offers more private solutions to this, discus
Finally, if Alice bought bitcoin from a cryptocurrency exchange, she could (and should) "withdraw" the bitcoin by pasting her Bitcoin address into the exchange website. The exchange will then send the bitcoin to her address directly.
=== From Bitcoin to Lightning network
=== From Bitcoin to Lightning Network
Alice's bitcoin is now controlled by her Eclair wallet and has been recorded on the Bitcoin blockchain. At this point, Alice's bitcoin is "on-chain," meaning that the transaction has been broadcast to the entire Bitcoin network, verified by all Bitcoin nodes, and "mined" (recorded) onto the Bitcoin blockchain.
@ -312,7 +313,7 @@ So far, the Eclair Mobile wallet has behaved only as a Bitcoin wallet, and Alice
Now, Alice is ready to start using the Lightning Network by taking her bitcoin "off-chain" in order to take advantage of the fast, cheap, and private payments that the Lightning Network offers.
==== Lightning Network channels
==== Lightning Network Channels
Swiping right, Alice accesses the "LIGHTNING CHANNELS" section of Eclair. Here she can manage the channels that will connect her wallet to the Lightning Network.
@ -333,7 +334,7 @@ At first, there are no open channels, so as we see in <<eclair-channels>>, the "
Alice presses the plus symbol and is presented with four possible ways to open a channel:
@ -346,7 +347,7 @@ A "node URI" is a Universal Resource Identifier (URI) that identifies a specific
[[node-URI-QR]]
.node URI as a QR code
image::images/node-URI-QR.png["Lightning node URI QR code",width=120]
image::images/mtln_0208.png["Lightning node URI QR code",width=120]
[[node-URI-example]]
.node URI
@ -360,7 +361,7 @@ Choosing the ACINQ node will slightly reduce Alice's privacy, as it will give AC
Alice selects "ACINQ Node" and is ready to open her first channel on the Lightning network.
==== Opening a Lightning channel
==== Opening a Lightning Channel
When Alice selects a node to open a new channel, she is asked to select how much bitcoin she wants to allocate to this channel. In subsequent chapters, we will discuss the implications of these choices, but for now, Alice will allocate almost all her funds to the channel. Since she will have to pay transaction fees to open the channel, she will select an amount slightly less than her total balance footnote:[The Eclair wallet doesn't offer an option to automatically calculate the necessary fees and allocate the maximum amount of funds to a channel, so Alice has to calculate this herself.]
@ -368,7 +369,7 @@ Alice allocates 0.018BTC of her 0.020 total to her channel and accepts the defau
[[eclair-open-channel]]
.Opening a Lightning Channel
image::images/eclair-open-channel-detail.png["Opening a Lightning Channel"]
image::images/mtln_0209.png["Opening a Lightning Channel"]
Once she clicks "OPEN", her wallet constructs the special Bitcoin transaction that opens a Lightning channel, known as the _funding transaction_. The "on-chain" funding transaction is sent to the Bitcoin Network for confirmation.
@ -376,20 +377,20 @@ Alice now has to wait again (see <<eclair-channel-waiting>>) for the transaction
[[eclair-channel-waiting]]
.Waiting for the Funding Transaction to Open the Channel
image::images/eclair-channel-waiting.png["Waiting for the Funding Transaction to Open the Channel"]
image::images/mtln_0210.png["Waiting for the Funding Transaction to Open the Channel"]
Once the funding transaction is confirmed, Alice's channel to the ACINQ node is open, funded and ready, as shown in <<eclair-channel-open>>:
[[eclair-channel-open]]
.Channel is Open
image::images/eclair-channel-open.png["Channel is Open"]
image::images/mtln_0211.png["Channel is Open"]
[TIP]
====
Did you notice that the channel amount seems to have changed? It hasn't: the channel contains 0.018 BTC, but in the time between screenshots the BTC exchange rate changed, so the USD value is different. You can choose to show balances in BTC or USD, but keep in mind that USD values are calculated in real-time and will change!
====
=== Buying a cup of coffee using LN
=== Buying a Cup of Coffee Using Ln
Alice now has everything ready to start using the Lightning Network. As you can see, it took a bit of work and a bit of time waiting for confirmations. However, now subsequent actions are fast and easy. The Lightning Network enables payments without having to wait for confirmations, as funds get settled in seconds.
@ -409,21 +410,21 @@ On the counter at Bob's Cafe, there is a tablet device showing <<bob-cafe-posapp
Alice selects the "Cafe Latte" option from the screen and is presented with a _Lightning Invoice_ (also known as a "payment request") as shown in <<bob-cafe-invoice>>
[[bob-cafe-invoice]]
.Lightning Invoice for Alice's latte
image::images/bob-cafe-invoice.png["BTCPay Server Lightning invoice"]
image::images/mtln_0213.png["BTCPay Server Lightning invoice"]
To pay the invoice, Alice opens her Eclair wallet and selects the "Send" button (which looks like a right-facing arrow) under the "TRANSACTION HISTORY" tab, as shown in <<alice-send-start>>.
Alice presses "PAY," and a second later, Bob's tablet shows a successful payment. Alice has completed her first Lightning Network payment! It was fast, inexpensive, and easy. Now she can enjoy her latte which was purchased using bitcoin through a payment system that is fast, cheap and decentralized. From now on, Alice can simply select an item on Bob's tablet screen, scan the QR code with her cell phone, click pay, and will be served a coffee, all within seconds and all without an "on-chain" transaction.
Now that we've followed Alice as she set up a Lightning wallet and purchased a coffee from Bob, we'll look under the hood and unpack the different components of the Lightning Network involved in that process.
Now that we've followed Alice as she set up a Lightning Wallet and purchased a coffee from Bob, we'll look under the hood and unpack the different components of the Lightning Network involved in that process.
This chapter will give a high-level overview and will not delve into all the technical details.
The goal is rather to help you to become aware of the most important concepts and building blocks of the Lightning Network.
@ -27,7 +27,7 @@ We'll start with a one sentence definition of what the Lightning Network (LN) is
**The Lightning Network (LN) is a peer-to-peer network of _payment channels_ implemented as smart contracts on the _Bitcoin blockchain_ as well as a communication protocol that defines how participants set up and execute these smart contracts.**
[[what_is_payment_channel]]
=== What is a payment channel?
=== What Is a Payment Channel?
There are several ways to describe a payment channel, depending on the context. Let's start at a high level and then add some more detail.
@ -41,7 +41,7 @@ To summarize:
A payment channel is a financial relationship between nodes, allocating funds from a multi-signature address, through a strictly defined cryptographic protocol.
=== Payment channel basics
=== Payment Channel Basics
Underlying the payment channel is simply a 2-of-2 multisignature address on the Bitcoin blockchain, for which you hold one key and your channel partner holds the other key.
@ -61,7 +61,7 @@ If you have an unpublished transaction from a 2-of-2 multisignature address that
The ability to hold a partially-signed transaction, offline and unpublished, with the option to publish and own that balance at any time, is the basis of the Lightning Network.
====
=== Routing payments across channels
=== Routing Payments Across Channels
Once several participants have channels from one party to another, payments can also be "forwarded" from payment channel to payment channel, by setting up a _path_ across the network connecting several payment channels together.
@ -85,7 +85,7 @@ Alice, for example, needs the network topology information to be aware of the ch
Last but not least, it is important to understand that the Lightning Network is nothing more than an application on top of Bitcoin, using Bitcoin transactions and Bitcoin Script. There is no "Lightning coin" or "Lightning blockchain".
Beyond all the technical primitives, the Lightning Network protocol is a creative way to get more benefits out of Bitcoin by allowing an arbitrary amount of instant payments with instant settlements without the necessity of having to trust anyone else but the Bitcoin network.
=== Payment channels
=== Payment Channels
As we saw in the previous chapter, Alice used her wallet software to create a payment channel between her and another Lightning Network participant.
@ -114,7 +114,7 @@ The other two proposed methods are _Duplex Micropayment_ channels, introduced by
Eltoo channels have some interesting properties and simplify the implementation of payment channels. However, eltoo channels require a change in the Bitcoin scripting language and therefore cannot be implemented on the Bitcoin mainnet as of 2020.
==== Multisignature address
==== Multisignature Address
Payment channels are built on top of 2-of-2 multisignature addresses.
@ -122,7 +122,7 @@ In summary, a multi-signature address is where bitcoin is locked so that it requ
Multisignature scripts and addresses are explained in more detail in <<multisig>>.
==== Funding transaction
==== Funding Transaction
The fundamental building block of a payment channel, is a 2-of-2 multisignature address. One of the two channel partners will fund the payment channel by sending bitcoin to the multisignature address. This transaction is called the _funding transaction_, and is recorded on the Bitcoin blockchain. footnote:[While the original Lightning whitepaper described channels funded by both channel partners, the current specification, as of 2020, assumes that just one partner commits funds to the channel. As of May 2021 dual-funded lightning channels are experimental in the c-lightning LN implementation.]
@ -179,7 +179,7 @@ If Alice wants to send 30k satoshi to Bob, both would create a new version of th
Now that we understand commitment transactions, let's look at some of the more subtle details. You may notice that this protocol leaves a way for either Alice or Bob to cheat.
==== Cheating with prior state
==== Cheating with Prior State
How many commitment transactions does Alice hold after she pays 30k satoshi to Bob? She holds two: the original one paying her 100k satoshi and the more recent one, paying her 70k satoshi and Bob 30k satoshi.
@ -223,7 +223,7 @@ So, in our example, Bob holds a commitment transaction that pays Alice _immediat
The two channel partners hold half of the revocation secret, so that neither one knows the whole secret. If they share their half, then the other channel partner has the full secret and can use it to exercise the revocation condition. When signing a new commitment transaction, each channel partner revokes the previous commitment by giving the other party their half of the revocation secret.
We will examine the revocation mechanism in more detail in <<channel_operation>>, where we will learn the details of how revocation secrets are constructed and used.
We will examine the revocation mechanism in more detail in <<revocation>>, where we will learn the details of how revocation secrets are constructed and used.
In simple terms, Alice signs Bob's new commitment transaction only if Bob offers his half of the revocation secret for the previous commitment. Bob only signs Alice's new commitment transaction if she gives him her half of the revocation secret from the previous commitment.
@ -233,19 +233,19 @@ The timelock is set to a number of blocks up to 2016 (approximately two weeks).
The timelock is adjustable and can be negotiated between channel partners. Usually, it is longer for larger capacity channels, and shorter for smaller channels, to align the incentives with the value of the funds.
For every new update of the channel balance, new commitment transactions and new revocation secrets have to be created and saved. As long as a channel remains open, all revocation secrets _ever created_ for the channel need to be kept as they might be needed in the future. Fortunately, the secrets are rather small and it is only the channel partners who need to keep them, not the entire network. Furthermore, due to a smart derivation mechanism used to derive revocation secrets, we only need to store the most recent secret because previous secrets can be derived from it (See <<revocation_secret_derivation>>.)
For every new update of the channel balance, new commitment transactions and new revocation secrets have to be created and saved. As long as a channel remains open, all revocation secrets _ever created_ for the channel need to be kept as they might be needed in the future. Fortunately, the secrets are rather small and it is only the channel partners who need to keep them, not the entire network. Furthermore, due to a smart derivation mechanism used to derive revocation secrets we only need to store the most recent secret, because previous secrets can be derived from it (see <<revocation_secret_derivation>>).
Nevertheless, managing and storing the revocation secrets is one of the more elaborate parts of Lightning nodes that require node operators to maintain backups.
[NOTE]
===
====
Technologies such as watchtower services or changing the channel construction protocol to the "eltoo" protocol might be future strategies to mitigate these issues and reduce the need for revocation secrets, penalty transactions and channel backups.
====
Alice can close the channel at any time if Bob does not respond, claiming her fair share of the balance.
After publishing the *last* commitment transaction on-chain, Alice has to wait for the timelock to expire before she can spend her funds from the commitment transaction. As we will see later, there is an easier way to close a channel without waiting, as long as Alice and Bob are both online and cooperate to close the channel with the correct balance allocation. But the commitment transactions stored by each channel partner act as a failsafe, ensuring they do not lose funds if there is a problem with their channel partner.
==== Announcing the channel
==== Announcing the Channel
Channel partners can agree to announce their channel to the whole Lightning Network, making it a _public channel_. To announce the channel, they use the Lightning Network's gossip protocol to tell other nodes about the existence, capacity and fees of the channel.
@ -265,7 +265,7 @@ When a channel and its capacity is publicly announced using the gossip protocol,
When new nodes join the Lightning Network they collect the channel announcements propagated via the gossip protocol from their peers, building an internal "map" of the Lightning Network. This map can then be used to find paths for payments, connecting channels together end-to-end.
==== Closing the channel
==== Closing the Channel
The best way to close a channel is... to not close it!
Opening and closing channels require an on-chain transaction, which will incur transaction fees.
@ -336,7 +336,7 @@ When publishing a commitment transaction during a force close, the on-chain fees
[NOTE]
====
Hashed TimeLock Contracts (HTLCs) will be covered in detail in <<htlcs>>.
Hash Time-Locked Contracts (HTLCs) will be covered in detail in <<htlcs>>.
For now, assume that these are payments that are routed across the Lightning Network, rather than payments made directly between the two channel partners.
These HTLCs are carried as additional outputs in the commitment transactions, thereby increasing the transaction size and on-chain fees.
====
@ -388,7 +388,7 @@ This software is either:
Remember that the commitment transaction has a timeout period specified in a given number of blocks, up to a maximum of 2016 blocks.
As long as you run your Lightning node once before the timeout period is reached, it will catch all cheating attempts.
It is not advisable to take this kind of risk; it is important to keep a well maintained node running continuously (See <<node_operations>>).
It is not advisable to take this kind of risk; it is important to keep a well maintained node running continuously (See <<continuous_operation>>).
=== Invoices
@ -396,7 +396,7 @@ Most payments on the Lightning Network start with an invoice, generated by the r
[NOTE]
====
There is a way to send an "unsolicited" payment without an invoice, using a work-around in the protocol called _keysend_. We will examine this in <<keysend>>.
There is a way to send an "unsolicited" payment without an invoice, using a work-around in the protocol called _keysend_. We will examine this in <<keysend>>.
====
An invoice is a simple payment instruction containing information such as a unique payment identifier (called a payment hash), a recipient, an amount, and an optional text description.
@ -410,14 +410,14 @@ Invoices are usually encoded either as a long bech32-encoded string or as a QR c
Did you notice how this contrasts with Bitcoin and how different terms are used? In Bitcoin, the recipient passes an address to the sender. In Lightning, the recipient creates an invoice and sends an invoice to the sender. In Bitcoin, the sender sends funds to an address. In Lightning, the sender pays an invoice and the payment gets routed to the recipient. Bitcoin is based on the concept of an "address", and Lightning is a payment network based on the concept of an "invoice". In Bitcoin, we create a "transaction" whereas in Lightning we send a "payment".
==== Payment hash and preimage
==== Payment Hash and Preimage
The most important part of the invoice is the _payment hash_. When constructing the invoice, Bob will make a payment hash as follows:
1. Bob chooses a random number +r+. This random number is called the _preimage_ or _payment secret_.
2. Bob uses +SHA256+ to calculate the hash +H+ of +r+ called the _payment hash_
latexmath:[H = SHA256(r)].
latexmath:[$H = SHA256(r)$].
[NOTE]
====
@ -428,7 +428,7 @@ There is no known way to find the inverse of SHA256 (i.e. compute a preimage fro
The payment process of the Lightning Network is only secure if +r+ is chosen completely randomly and is not predictable. This security relies on the fact that hash functions cannot be inverted or feasibly brute-forced and therefore no one can find +r+ from +H+.
==== Additional metadata
==== Additional Metadata
Invoices can optionally include other useful metadata such as a short text description. If a user has several invoices to pay, the user can read the description and be reminded of what the invoice is about.
@ -444,7 +444,7 @@ While it is always possible to "fall back" to an on-chain Bitcoin transaction, i
Lightning invoices contain an expiry date. Since the recipient must keep the preimage +r+ for every invoice issued, it is useful to have invoices expire so that these preimages do not need to be kept forever. Once an invoice expires or is paid, the recipient can discard the preimage.
=== Delivering the payment
=== Delivering the Payment
We have seen how the recipient creates an invoice that contains a payment hash. This payment hash will be used to move the payment across a series of payment channels, from sender to recipient, even if they do not have a direct payment channel between them.
@ -452,7 +452,7 @@ In the next few sections, we will dive into the ideas and methods that are being
First, let's look at the Lightning Network's communication protocol.
==== The peer-to-peer gossip protocol
==== The Peer-To-Peer Gossip Protocol
As we mentioned previously, when a payment channel is constructed, the channel partners have the option of making it public, announcing its existence and details to the whole Lightning Network.
@ -491,7 +491,7 @@ For now, it is only important to know that the gossip protocol exists and that i
This topology information is crucial for delivering payments through the network of payment channels.
==== Pathfinding and routing
==== Pathfinding and Routing
Payments on the Lightning Network are forwarded along a _path_ made of channels linking one participant to another, from the payment source to the payment destination. The process of finding a path from source to destination is called _pathfinding_. The process of using that path to make the payment is called _routing_.
@ -502,7 +502,7 @@ A frequent criticism of the Lightning Network is that "routing" is not solved, o
As we will see next, the Lightning Network currently uses a _source-based_ protocol for pathfinding and an _onion routed_ protocol for routing payments. Source-based means that the sender of the payment has to find a path through the network to the intended destination. Onion-routed means that the elements of the path are layered (like an onion), with each layer encrypted so that it can only be seen by one node at a time. We will discuss onion routing in the next section.
=== Source-based pathfinding
=== Source-Based Pathfinding
If we knew the exact channel balances of every channel, we could easily compute a payment path using any of the standard pathfinding algorithms taught in any computer science class. This could even be solved in a way that optimizes the fees paid to nodes for forwarding the payment.
@ -529,7 +529,7 @@ On the Internet, we use the Internet protocol and an IP forwarding algorithm to
Of course, pathfinding is trivial if we want to pay our direct channel partner and we have enough balance on our side of the channel to do so. In all other cases, our node uses information from the gossip protocol to do pathfinding. This includes currently known public payment channels, known nodes, known topology (how known nodes are connected), known channel capacities, and known fee policies set by the node owners.
==== Onion routing
==== Onion Routing
The Lightning Network uses an _onion routing protocol_ similar to the famous Tor (The Onion Router) network.
The onion routing protocol used in Lightning is called the _SPHINX mixformat_ and will be explained in detail in a later chapter.
@ -570,7 +570,7 @@ The onion routing protocol used in Lightning has the following properties:
Onion routing will be examined in detail in <<onion_routing>>.
==== Payment forwarding algorithm
==== Payment Forwarding Algorithm
Once the sender of a payment finds a possible path across the network and constructs an onion, the payment is forwarded by each node in the path. Each node processes one layer of the onion and forwards it to the next node in the path.
@ -594,7 +594,7 @@ As the error propagates backwards on each channel along the path, the channel pa
While the likelihood for a payment failure is high if it does not settle quickly, a node should never initiate another payment attempt along a different path before the onion returns with an error. The sender would pay twice if both payment attempts eventually succeeded.
=== Peer-to-peer communication encryption
=== Peer-To-Peer Communication Encryption
The Lightning Network protocol is mainly a peer-to-peer protocol between its participants. As we saw in previous sections, there are two overlapping functions in the network, forming two logical networks that together are _The Lightning Network_:
@ -610,7 +610,7 @@ The use of the Noise Protocol Framework in the Lightning Network ensures that ev
https://github.com/nayutaco/lightning-dissector
=== Thoughts about trust
=== Thoughts About Trust
As long as a person follows the protocol and has their node secured, there is no major risk of losing funds when participating in the Lightning Network.
However, there is the cost of paying on-chain fees when opening a channel.
Any cost should come with a corresponding benefit.
@ -628,7 +628,7 @@ While the Lightning Network is built on top of Bitcoin and inherits many of its
Some of these differences are differences in terminology. There are also architectural differences and differences in the user experience. In the next few sections, we will examine the differences and similarities, explain the terminology and adjust our expectations.
==== Addresses vs. invoices, transactions vs. payments
==== Addresses vs. Invoices, Transactions vs. Payments
In a typical payment using Bitcoin, a user receives a Bitcoin address (e.g. scanning a QR code on a webpage, or receiving it in an instant message or email from a friend). They then use their Bitcoin wallet to create a transaction to send funds to this address.
@ -640,7 +640,7 @@ There are some differences in the user experience however. A Bitcoin address is
In Lightning however, each invoice can only be used once for a specific payment amount. You cannot pay more or less, you cannot use an invoice again and the invoice has an expiry time built in. In Lightning, a recipient has to generate a new invoice for each payment, specifying the payment amount in advance. There is an exception to this, a mechanism called _keysend_, which we will examine in <<keysend>>.
==== Selecting outputs vs. finding a path
==== Selecting Outputs vs. Finding a Path
In order to make a payment on the Bitcoin network, a sender needs to consume one or more Unspent Transaction Outputs (UTXOs).
If a user has multiple UTXOs, they (or rather their wallet) will need to select which UTXO(s) to send.
@ -652,14 +652,14 @@ As many possible channels and paths can be used to make a payment, the Lightning
With technologies such as Atomic Multi-Path (AMP) and Multi-Path Payments (MPP), which we will review in subsequent chapters, several Lightning paths can be aggregated into a single atomic payment, just like several Bitcoin UTXO can be aggregated into a single atomic Bitcoin transaction.
==== Change outputs on Bitcoin vs. no change on Lightning
==== Change Outputs On Bitcoin vs. No Change On Lightning
In order to make a payment on the Bitcoin network, the sender needs to consume one or more Unspent Transaction Outputs (UTXOs). UTXO can only be spent in full, they cannot be divided and partially spent. So if a user wishes to spend 0.8 BTC, but only has a 1 BTC UTXO, then they need to spend the entire 1 BTC UTXO by sending 0.8 BTC to the recipient and 0.2 BTC back to themselves as change. The 0.2 BTC change payment creates a new UTXO called a "change output".
On Lightning, the funding transaction spends some Bitcoin UTXO, creating a multi-signature UTXO to open the channel. Once the bitcoin is locked within that channel, portions of it can be sent back and forth within the channel, without the need to create any change.
This is because the channel partners simply update the channel balance and only create a new UTXO when the channel is eventually closed, with the channel closing transaction.
==== Mining fees vs. routing fees
==== Mining Fees vs. Routing Fees
On the Bitcoin network, users pay fees to miners to have their transactions included in a block.
These fees are paid to the miner who mines that particular block.
@ -669,7 +669,7 @@ As miners will typically mine the most profitable transactions first, a user who
On the Lightning Network, users pay fees to other (intermediary node) users to route payments through their channels.
In order to route a payment, an intermediary node will have to move funds in two or more channels they own, as well as transmit the data for the sender's payment. Typically, the routing user will charge the sender based on the _value_ of the payment, having established a minimum _base fee_ (a flat fee for each payment) and a _fee rate_ (a pro-rated fee proportional to the value of the payment). Higher value payments will thus cost more to route, and a market for liquidity is formed, where different users charge different fees for routing through their channels.
==== Varying fees depending on traffic vs. announced fees
==== Varying Fees Depending On Traffic vs. Announced Fees
On the Bitcoin network, miners are profit-seeking and will typically include as many transactions in a block as possible, while staying within the block capacity called the _block weight_.
@ -681,7 +681,7 @@ The scarce resource on the Bitcoin network is the space in the blocks. Bitcoin u
On the Lightning Network, users are paying fees to the users routing their payments. Routing a payment, in economic terms, is nothing more than providing and assigning capacity to the sender. Naturally, routers who charge lower fees for the same capacity will be more attractive to route through. Thus a fee market exists where routers are in competition with each other over the fees they charge to route payments through their channels.
==== Public Bitcoin transactions vs. private Lightning payments
==== Public Bitcoin Transactions vs. Private Lightning Payments
On the Bitcoin network, every transaction is publicly visible on the Bitcoin blockchain. While the addresses involved are pseudonymous and are not typically tied to an identity, they are still seen and validated by every other user on the network.
In addition, blockchain surveillance companies collect and analyze this data en-masse and sell it to interested parties such as private firms, governments and intelligence agencies.
@ -690,7 +690,7 @@ Lightning Network payments on the other hand, are almost completely private. Typ
In summary, Bitcoin transactions are broadcast publicly and stored forever. Lightning payments are executed between a few selected peers and information about them is privately stored and only until the channel is closed. Creating mass surveillance and analysis tools equivalent to those used on Bitcoin will be much harder on Lightning.
==== Waiting for confirmations vs. instant settlement
==== Waiting for Confirmations vs. Instant Settlement
On the Bitcoin network, transactions are only settled once they have been included in a block, in which case they are said to be "confirmed" in that block. As more blocks are mined, the transaction acquires more "confirmations" and is considered more secure.
@ -699,7 +699,7 @@ In practical terms, instant settlement means that payments take only a few secon
Finally, when the channel is closed, a transaction is made on the Bitcoin network; once that transaction is confirmed, the channel is considered closed.
==== Sending arbitrary amounts vs. capacity restrictions
==== Sending Arbitrary Amounts vs. Capacity Restrictions
On the Bitcoin network, a user can send any amount of bitcoin that they own to another user, without capacity restrictions. A single transaction can theoretically send up to 21 million bitcoin as a payment.
@ -711,7 +711,7 @@ If the payment is routed, every routing node along the routing path must have ch
Hence, capacity and connectivity are critical and scarce resources in the Lightning Network.
==== Incentives for large value payment vs. small value payments
==== Incentives for Large Value Payment vs. Small Value Payments
The fee structure in Bitcoin is independent of the transaction value.
A $1 million transaction has the same fee as a $1 transaction on Bitcoin, assuming a similar transaction size in bytes (more specifically "virtual" bytes after segwit).
@ -720,7 +720,7 @@ Therefore, in Lightning the payment fee increases with payment value.
These opposing fee structures create different incentives and lead to different usage in regards to transaction value.
A transaction of greater value will be cheaper on Bitcoin and hence users will prefer Bitcoin for large value transactions. Similarly, on the other end of the scale, users will prefer Lightning for small value transactions.
==== Using the Blockchain as a ledger vs. as a court system
==== Using the Blockchain As a Ledger vs. As a Court System
On the Bitcoin network, every transaction is eventually recorded in a block on the blockchain.
The blockchain thus forms a complete history of every transaction since Bitcoin's creation, and a way to fully audit every bitcoin in existence.
@ -734,7 +734,7 @@ If Alice tries to cheat by submitting the opening state of the channel to the Bi
For the Lightning Network, the Bitcoin blockchain acts as a court system.
Like a robotic judge, Bitcoin records the initial and final balances of each channel, and approves penalties if one of the parties tries to cheat.
==== Offline vs. online, asynchronous vs. synchronous
==== Offline vs. Online, Asynchronous vs. Synchronous
When a Bitcoin user sends funds to a destination address he does not need to know anything about the recipient. The recipient might be offline or online, and no interaction between sender and recipient is needed. The interaction is between sender and the Bitcoin blockchain. Receiving bitcoin on the Bitcoin blockchain is a _passive_ and _asynchronous_ activity that does not require any interaction by the recipient, or for the recipient to be online at any time. Bitcoin addresses can even be generated offline and are never "registered" with the Bitcoin network. Only spending bitcoin requires interaction.
@ -743,7 +743,7 @@ The recipient must run a node or have someone that runs a node on their behalf (
The synchronous and always-online nature of the Lightning network is probably the biggest difference in the user experience and often confounds users who are accustomed to Bitcoin.
==== Satoshis vs. milli-satoshis
==== Satoshis vs. Milli-Satoshis
On the Bitcoin network, the smallest amount is a _satoshi_ which cannot be divided any further. Lightning is a bit more flexible, and Lightning nodes work with _milli-satoshis_ (thousandths of a satoshi). This allows tiny payments to be sent via Lightning. A single milli-satoshi payment can be sent across a payment channel, an amount so small it should properly be characterized as a _nanopayment_.
@ -753,25 +753,25 @@ The milli-satoshi unit cannot, of course, be settled on the Bitcoin blockchain a
While the Lightning Network differs from Bitcoin in a number of ways, including in architecture and user experience, it is built from Bitcoin and retains many of Bitcoin’s core features.
==== Monetary unit
==== Monetary Unit
Both the Bitcoin network and the Lightning network use the same monetary units: bitcoin. Lightning payments use the very same bitcoin as Bitcoin transactions. As an implication, because the monetary unit is the same, the monetary limit is the same: less than 21 million bitcoin. Of Bitcoin's 21 million total bitcoin, some are already allocated to 2-of-2 multi-signature addresses as part of payment channels on the Lightning Network.
==== Irreversibility and finality of payments
==== Irreversibility and Finality of Payments
Both Bitcoin transactions and Lightning payments are irreversible and immutable. There is no "undo" operation or "chargeback" for either system. As a sender of either one, you have to act responsibly, but also, as a recipient you are guaranteed finality of your transactions.
==== Trust and counterparty risk
==== Trust and Counterparty Risk
As with Bitcoin, Lightning requires the user only to trust mathematics, encryption, and that the software does not have any critical bugs. Neither Bitcoin nor Lightning requires the user to trust a person, a company, an institution, or a government.
Since Lightning sits on top of Bitcoin and relies on Bitcoin as its underlying base layer, it is clear that the security model of Lightning reduces to the security of Bitcoin. This means that Lightning offers broadly the same security as Bitcoin under most circumstances, with only a slight reduction in security under some narrow circumstances.
==== Permissionless operation
==== Permissionless Operation
Both Bitcoin and Lightning can be used by anybody with access to the Internet and to the appropriate software, e.g. node and wallet.
Neither network requires users to get permission, vetting, or authorization from third-parties, companies, institutions or a government. Governments can outlaw Bitcoin or Lightning within their jurisdiction, but cannot prevent their global use.
==== Open source and open system
==== Open Source and Open System
Both, Bitcoin and Lightning are open-source software systems built by a decentralized global community of volunteers, available under open licenses. Both are based on open and interoperable protocols, which operate as open systems and open networks. Global, open and free.
As we have seen in previous chapters, a Lightning node is a computer system that participates in the Lightning Network. The Lightning Network is not a product or company, it is a set of open standards that define a baseline for interoperability. As such, Lightning node software has been built by a variety of companies and community groups. The vast majority of Lightning software is _open source_, meaning that the source code is open and licensed in such a way as to enable collaboration, sharing and community participation in the development process. Similarly, the Lightning node implementations we will present in this chapter are all open source and are collaboratively developed.
@ -13,11 +13,11 @@ Another major difference between Lightning node software and Bitcoin node softwa
In this chapter, you will learn how to set up each of the software packages for the most popular Lightning node implementations. We've presented them in alphabetical order to emphasize that we generally do not prefer or endorse one over the other. Each has its strengths and weaknesses and choosing one will depend on a variety of factors. Since they are developed in different programming languages (e.g. Go, C, etc.), your choice may also depend on your level of familiarity and expertise with a specific language and development toolset.
=== Lightning development environment
=== Lightning Development Environment
((("development environment", "setup")))If you're a developer, you will want to set up a development environment with all the tools, libraries, and support software for writing and running Lightning software. In this highly technical chapter, we'll walk through that process step-by-step. If the material becomes too dense or you're not actually setting up a development environment, then feel free to skip to the next chapter, which is less technical.
If you're a developer, you will want to set up a development environment with all the tools, libraries, and support software for writing and running Lightning software. In this highly technical chapter, we'll walk through that process step-by-step. If the material becomes too dense or you're not actually setting up a development environment, then feel free to skip to the next chapter, which is less technical.
==== Using the command-line
==== Using the Command-Line
The examples in this chapter, and more broadly in most of this book, use a command-line terminal. That means that you type commands into a terminal and receive text responses. Furthermore, the examples are demonstrated on an operating system based on the Linux kernel and GNU software system, specifically the latest long-term stable release of Ubuntu (Ubuntu 20.04 LTS). The majority of the examples can be replicated on other operating systems such as Windows or Mac OS, with small modifications to the commands. The biggest difference between operating systems is the _package manager_ which installs the various software libraries and their pre-requisites. In the given examples, we will use +apt+, which is the package manager for Ubuntu. On Mac OS, a common package manager used for open source development is Homebrew (command +brew+) found at https://brew.sh.
@ -25,12 +25,12 @@ In most of the examples here, we will be building the software directly from the
[TIP]
====
((("$ symbol")))((("shell commands")))((("terminal applications")))In many of the examples in this chapter we will be using the operating system's command-line interface (also known as a "shell"), accessed via a "terminal" application. The shell will first display a prompt as an indicator that it is ready for your command. Then you type a command and press "Enter" to which the shell responds with some text and a new prompt for your next command. The prompt may look different on your system, but in the following examples it is denoted by a +$+ symbol. In the examples, when you see text after a +$+ symbol, don't type the +$+ symbol but type the command immediately following it. Then press the Enter key to execute the command. In the examples, the lines below each command are the operating system's responses to that command. When you see the next +$+ prefix, you'll know it is a new command and you should repeat the process.
In many of the examples in this chapter we will be using the operating system's command-line interface (also known as a "shell"), accessed via a "terminal" application. The shell will first display a prompt as an indicator that it is ready for your command. Then you type a command and press "Enter" to which the shell responds with some text and a new prompt for your next command. The prompt may look different on your system, but in the following examples it is denoted by a +$+ symbol. In the examples, when you see text after a +$+ symbol, don't type the +$+ symbol but type the command immediately following it. Then press the Enter key to execute the command. In the examples, the lines below each command are the operating system's responses to that command. When you see the next +$+ prefix, you'll know it is a new command and you should repeat the process.
====
To keep things consistent, we use the +bash+ shell in all command-line examples. While other shells will behave in a similar way, and you will be able to run all the examples without it, some of the shell scripts are written specifically for the +bash+ shell and may require some changes or customizations to run in another shell. For consistency, you can install the +bash+ shell on Windows and Mac OS, and it comes installed by default on most Linux systems.
==== Downloading the book repository
==== Downloading the Book Repository
All the code examples are available in the book's online repository. Because the repository will be kept up-to-date as much as possible, you should always look for the latest version in the online repository instead of copying it from the printed book or the ebook.
@ -42,21 +42,21 @@ Alternatively, you can use the +git+ command to create a version-controlled clon
To make a local copy of the repository on your computer, run the git command as follows:
[git-clone-lnbook]
[[git-clone-lnbook]]
----
$ git clone https://github.com/lnbook/lnbook.git
----
You now have a complete copy of the book repository in a folder called +lnbook+. You will want to change to the newly downloaded directory by running:
[cd-lnbook]
[[cd-lnbook]]
----
$ cd lnbook
----
All subsequent examples will assume that you are running commands from inside this folder.
=== Docker containers
=== Docker Containers
Many developers use a _container_, which is a type of virtual machine, to install a pre-configured operating system and applications with all the necessary dependencies. Much of the Lightning software can also be installed using a container system such as _Docker_ found at https://docker.com. Container installations are a lot easier, especially for those who are not used to a command-line environment.
@ -68,12 +68,12 @@ The installation of use of Docker and its commands is detailed in <<appendix_doc
You can find the latest container definitions and build configurations in the book's repository under the +code/docker+ folder. Each container is in a separate folder as can be seen below:
[tree]
[[tree]]
----
$ tree -F --charset=asciii code/docker
----
[docker-dir-list]
[[docker-dir-list]]
----
code/docker
|-- bitcoind/
@ -126,7 +126,7 @@ code/docker
As we will see in the next few sections, you can build these containers locally, or you can pull them from the book's repository on _Docker Hub_. The following sections will assume that you have installed Docker and are familiar with the basic use of the +docker+ command.
=== Bitcoin Core and regtest
=== Bitcoin Core and Regtest
Most of the Lightning node implementations need access to a full Bitcoin node in order to work.
@ -136,7 +136,7 @@ A Bitcoin node can be operated in _regtest_ mode, where the node creates a local
The container for Bitcoin Core is +bitcoind+. It is configured to run Bitcoin Core in +regtest+ mode and to mine 6 new blocks every 10 seconds. Its RPC port is exposed on port 18443 and accessible for RPC calls with the username +regtest+ and the password +regtest+. You can also access it with an interactive shell and run +bitcoin-cli+ commands locally.
==== Building the Bitcoin Core container
==== Building the Bitcoin Core Container
Let's prepare the +bitcoind+ container. The easiest way is to pull the latest container from _Docker Hub_:
@ -208,7 +208,7 @@ As you can see, bitcoind starts up and mines 101 simulated blocks to get the cha
For now, there are no transactions. But we have some test bitcoin that has been mined in the wallet and is available to spend. When we connect some Lightning nodes to this chain, we will send some bitcoin to their wallets so that we can open some Lightning channels between the Lightning nodes.
===== Interacting with the Bitcoin Core container
===== Interacting with the bitcoin core container
In the mean time, we can also interact with the +bitcoind+ container by sending it shell commands. The container is sending a log file to the terminal, displaying the mining process of the bitcoind process. To interact with the shell we can issue commands in another terminal, using the +docker exec+ command. Since we previously named the running container with the +name+ argument, we can refer to it by that name when we run the +docker exec+ command. First, let's run an interactive +bash+ shell:
As you will see in the following sections, we can run several containers at the same time and then interact with them individually. We can issue commands to extract information such as the Lightning node public key or to take actions such as opening a Lightning channel to another node. The +docker run+ and +docker exec+ commands together with +jq+ for JSON decoding are all we need to build a working Lightning Network that mixes many different node implementations. This enables us to try out diverse experiments on our own computer.
=== The c-lightning Lightning node project
=== The c-lightning Lightning Node Project
C-lightning is a lightweight, highly customizable, and standard-compliant implementation of the Lightning Network protocol, developed by Blockstream as part of the Elements project. The project is open source and developed collaboratively on Github:
In the following sections, we will build a Docker container that runs a c-lightning node connecting to the bitcoind container we built previously. We will also show you how to configure and build the c-lightning software directly from the source code.
==== Building c-lightning as a Docker container
==== Building c-lightning As a Docker Container
The c-lightning software distribution has a Docker container, but it is designed for running c-lightning in production systems and along side a bitcoind node. We will be using a somewhat simpler container configured to run c-lightning for demonstration purposes.
@ -317,7 +317,7 @@ Our container is now built and ready to run. However, before we run the c-lightn
Docker containers can "talk" to each other over a virtual local area network managed by the Docker system. Each container can have a custom name and other containers can use that name to resolve its IP address and easily connect to it.
====
==== Setting up a Docker network
==== Setting Up a Docker Network
Once a Docker network is set up, Docker will activate the network on our local computer every time Docker starts, e.g. after rebooting. So we only need to set up a network once by using the +docker network create+ command. The network name itself is not important, but it has to be unique on our computer. By default, Docker has three networks named +host+, +bridge+, and +none+. We will name our new network +lnbook+ and create it like this:
@ -335,7 +335,7 @@ ee8824567c95 none null local
As you can see, running +docker network ls+ gives us a listing of the Docker networks. Our +lnbook+ network has been created. We can ignore the network ID, as it is automatically managed.
==== Running the bitcoind and c-lightning containers
==== Running the bitcoind and c-lightning Containers
The next step is to start the bitcoind and c-lightning containers and connect them to the +lnbook+ network. To run a container in a specific network, we must pass the +network+ argument to +docker run+. To make it easy for containers to find each other, we will also give each one a name with the +name+ argument. We start bitcoind like this:
@ -397,13 +397,13 @@ In the next section we will also look at how to download, configure and compile
If you are not planning on diving into the source code or programming of a Lightning node, you can skip the next section entirely. The Docker container we just built is sufficient for most of the examples in the book.
====
==== Installing c-lightning from source code
==== Installing c-lightning from Source Code
The c-lightning developers have provided detailed instructions for building c-lightning from source code. We will be following the instructions here:
==== Installing prerequisite libraries and packages
==== Installing Prerequisite Libraries and Packages
These installation instructions assume you are building c-lightning on a Linux or similar system with GNU build tools. If that is not the case, look for the instructions for your operating system in the ElementsProject repository above.
@ -446,7 +446,7 @@ $
After a few minutes and a lot of on-screen activity, you will have installed all the necessary packages and libraries. Many of these libraries are also used by other Lightning packages and needed for software development in general.
==== Copying the c-lightning source code
==== Copying the c-lightning Source Code
Next, we will copy the latest version of c-lightning from the source code repository. To do this, we will use the +git clone+ command which clones a version-controlled copy onto your local machine thereby allowing you to keep it synchronized with subsequent changes without having to download the whole repository again:
@ -467,7 +467,7 @@ $ cd lightning
We now have a copy of c-lightning cloned into the +lightning+ subfolder, and we have used the +cd+ (change directory) command to enter that subfolder.
==== Compiling the c-lightning source code
==== Compiling the c-lightning Source Code
Next, we use a set of _build scripts_ that are commonly available in many open source projects. These _build scripts_ use the +configure+ and +make+ commands which allow us to:
@ -561,7 +561,7 @@ v0.10.1-34-gfe86c11
The version consists of the latest release version (v0.10.1) followed by the number of changes since the release (34) and finally a hash identifying exactly which revision (fe86c11). You may see a different version from that shown above as the software continues to evolve long after this book is published. However, no matter what version you see, the fact that the commands execute and respond with version information means that you have succeeded in building the c-lightning software.
=== The Lightning Network Daemon (LND) node project
=== The Lightning Network Daemon (LND) Node Project
The Lightning Network Daemon (LND) is a complete implementation of a Lightning Network node by Lightning Labs. The LND project provides a number of executable applications, including +lnd+ (the daemon itself) and +lncli+ (the command-line utility). LND has several pluggable back-end chain services including btcd (a full-node), bitcoind (Bitcoin Core), and neutrino (a new experimental light client). LND is written in the Go programming language. The project is open source and developed collaboratively on Github:
In the next few sections we will build a Docker container to run LND, build LND from source code, and learn how to configure and run LND.
==== The LND docker container
==== The LND Docker Container
We can pull the LND example docker container from the book's Docker Hub repository:
@ -618,7 +618,7 @@ Our container is now ready to run. As with the c-lightning container we built pr
Normally, each node operator runs their own Lightning node and their own Bitcoin node on their own server. For us, a single bitcoind container can serve many Lightning nodes. On our simulated network we can run several Lightning nodes, all connecting to a single Bitcoin node in regtest mode.
====
==== Running the bitcoind and LND containers
==== Running the bitcoind and LND Containers
As before, we start the bitcoind container in one terminal and LND in another. If you already have the bitcoind container running, you do not need to restart it. Just leave it running and skip the next step. To start bitcoind in the +lnbook+ network we use +docker run+ like this:
@ -679,7 +679,7 @@ In the next section we will look at how to download and compile LND directly fro
If you are not planning on diving into the source code or programming of a Lightning node, you can skip the next section entirely. The Docker container we just built is sufficient for most of the examples in the book.
====
==== Installing LND from source code
==== Installing LND from Source Code
In this section we will build LND from scratch. LND is written in the Go programming language. IF you want to find out more about Go, search for +golang+ instead of +go+ to avoid irrelevant results. Because it is written in Go and not C or C++, it uses a different "build" framework than the GNU autotools/make framework we saw used in c-lightning previously. Don't fret though, it is quite easy to install and use the golang tools and we will show each step here. Go is a fantastic language for collaborative software development as it produces very consistent, precise, and easy to read code regardless of the number of authors. Go is focused and "minimalist" in a way that encourages consistency across versions of the language. As a compiled language, it is also quite efficient. Let's dive in.
To avoid having to set these environment variables every time you open a shell, you can add those two lines to the end of your bash shell configuration file +.bashrc+ in your home directory, using the editor of your choice.
==== Copying the LND source code
==== Copying the LND Source Code
As with many open source projects nowadays, the source code for LND is on Github (www.github.com). The +go get+ command can fetch it directly using the Git protocol:
@ -723,7 +723,7 @@ $ go get -d github.com/lightningnetwork/lnd
Once +go get+ finishes, you will have a sub-directory under +GOPATH+ that contains the LND source code.
==== Compiling the LND source code
==== Compiling the LND Source Code
LND uses the +make+ build system. To build the project, we change directory to LND's source code and then use +make+ like this:
@ -745,7 +745,7 @@ lncli version 0.10.99-beta commit=clock/v1.0.0-106-gc1ef5bb908606343d2636c8cd345
You will likely see a different version from that shown above, as the software continues to evolve long after this book is published. However, no matter what version you see, the fact that the commands execute and show you version information means that you have succeeded in building the LND software.
=== The Eclair Lightning node project
=== The Eclair Lightning Node Project
Eclair (French for Lightning) is a Scala implementation of the Lightning Network made by ACINQ. Eclair is also one of the most popular and pioneering mobile Lightning wallets which we used to demonstrate a Lightning payment in the second chapter. In this section we examine the Eclair server project which runs a Lightning node. Eclair is an open source project and can be found on GitHub:
In the next few sections we will build a Docker container to run Eclair, as we did previously with c-lightning and LND. We will also build Eclair directly from the source code.
==== The Eclair docker container
==== The Eclair Docker Container
Let's pull the book's eclair container from the Docker Hub repository:
@ -802,7 +802,7 @@ Our image is now ready to run. The Eclair container also depends on a running in
One notable difference between Eclair and LND or c-lightning is that Eclair doesn't contain a separate bitcoin wallet but instead relies directly on the bitcoin wallet in Bitcoin Core. Recall that using LND we "funded" its bitcoin wallet by executing a transaction to transfer bitcoin from Bitcoin Core's wallet to LND's bitcoin wallet. This step is not necessary using Eclair. When running Eclair, the Bitcoin Core wallet is used directly as the source of funds to open channels. As a result, unlike the LND or c-lightning containers, the Eclair container does not contain a script to transfer bitcoin into its wallet on startup.
==== Running the bitcoind and Eclair containers
==== Running the bitcoind and Eclair Containers
As before, we start the bitcoind container in one terminal and the Eclair container in another. If you already have the bitcoind container running, you do not need to restart it. Just leave it running and skip the next step. To start +bitcoind+ in the +lnbook+ network, we use +docker run+ like this:
@ -872,7 +872,7 @@ In the next section we will also look at how to download and compile Eclair dire
If you are not planning on diving into the source code or programming of a Lightning node, you can skip the next section entirely. The Docker container we just built is sufficient for most of the examples in the book.
====
==== Installing Eclair from source code
==== Installing Eclair from Source Code
In this section we will build Eclair from scratch. Eclair is written in the Scala programming language which is compiled using the Java compiler. To run Eclair, we first need to install Java and its build tools. We will be following the instructions found in the BUILD.md document of the Eclair project:
Once +git clone+ finishes you will have a sub-directory +eclair+ containing the source code for the Eclair server.
==== Compiling the Eclair source code
==== Compiling the Eclair Source Code
Eclair uses the +Maven+ build system. To build the project we change the working directory to Eclair's source code and then use +mvn package+ like this:
Congratulations! You have built Eclair from source and you are ready to code, test, fix bugs, and contribute to this project!
=== Building a complete network of diverse Lightning nodes
=== Building a Complete Network of Diverse Lightning Nodes
Our final example, presented in this section, will bring together all the various containers we've built to form a Lightning network made of diverse (LND, c-lightning, Eclair) node implementations. We'll compose the network by connecting the nodes together and opening channels from one node to another. As the final step, we'll route a payment across these channels!
In this example, we will replicate the Lightning network example from <<routing_on_a_network_of_payment_channels>>. Specifically, we will create four Lightning nodes named Alice, Bob, Chan, and Dina. We will connect Alice to Bob, Bob to Chan, and Chan to Dina. Finally, we will have Dina create an invoice and have Alice pay that invoice. Since Alice and Dina are not directly connected, the payment will be routed as an HTLC across all the payment channels.
In this example, we will build a demonstration Lightning network made of four Lightning nodes named Alice, Bob, Chan, and Dina. We will connect Alice to Bob, Bob to Chan, and Chan to Dina. This is shown in <<alice_bob_chan_dina_network_demo>>.
==== Using docker-compose to orchestrate Docker containers
[[alice_bob_chan_dina_network_demo]]
.A small demonstration network of four nodes
image::images/mtln_1002.png["A small demonstration network of four nodes"]
Finally, we will have Dina create an invoice and have Alice pay that invoice. Since Alice and Dina are not directly connected, the payment will be routed as an HTLC across all the payment channels.
==== Using Docker-Compose to Orchestrate Docker Containers
To make this example work, we will be using a _container orchestration_ tool that is available as a command called +docker-compose+. This command allows us to specify an application composed of several containers and run the application by launching all the cooperating containers together.
@ -977,7 +983,7 @@ docker-compose version 1.21.0, build unknown
The most common +docker-compose+ commands we will use are +up+ and +down+, e.g. +docker-compose up+.
==== Docker-compose configuration
==== Docker-Compose Configuration
The configuration file for +docker-compose+ is found in the +code/docker+ directory and is named +docker-compose.yml+. It contains a specification for a network and each of the four containers. The top looks like this:
@ -1007,7 +1013,7 @@ The fragment above defines a network called +lnnet+ and a container called +bitc
Since all these diverse implementations follow the Basis of Lightning Technologies (BOLT) specification and have been extensively tested for interoperability, they have no difficulty working together to build a Lightning network.
==== Starting the example Lightning network
==== Starting the Example Lightning Network
Before we get started, we should make sure we're not already running any of the containers. If a new container shares the same name as one that is already running, then it will fail to launch. Use +docker ps+, +docker stop+, and +docker rm+ as necessary to stop and remove any currently running containers!
@ -1058,7 +1064,7 @@ Following the start up, you will see a whole stream of log files as each of the
$ docker-compose logs -f Alice
----
==== Opening channels and routing a payment
==== Opening Channels and Routing a Payment
Our Lightning network should now be running. As we saw in the previous sections of this chapter, we can issue commands to a running Docker container with the +docker exec+ command. Regardless of whether we started the container with +docker run+ or started a bunch of them with +docker-compose up+, we can still access containers individually using the Docker commands.
@ -1136,7 +1142,7 @@ Of course, you can do a lot more with this test network than a 3-channel, 4-node
[TIP]
====
Lightning Polar (download from https://lightningpolar.com) allows you to visualize the network you have been experimenting with using Docker.
Lightning Polar (download from https://lightningpolar.com[]) allows you to visualize the network you have been experimenting with using Docker.
After having read this far, you have probably set up a Lightning wallet. In this chapter, we will take things one step further and set up a full Lightning node. In addition to setting one up, we will learn how to operate it and maintain it over time.
There are many reasons why you might want to set up your own Lightning node. They include:
* To be a full, active participant in the Lightning Network, not just an end-user.
* To run an e-commerce store or receive income via Lightning payments.
* To earn income from Lightning routing fees or by renting channel liquidity.
* To develop new services, applications, or plugins for the Lightning Network.
* To increase your financial privacy while using Lightning.
* To use some apps built on top of Lightning, like Lightning-powered instant messaging apps.
* For financial freedom, independence, and sovereignty.
* To be a full, active participant in the Lightning Network, not just an end user
* To run an ecommerce store or receive income via Lightning payments
* To earn income from Lightning routing fees or by renting channel liquidity
* To develop new services, applications, or plug-ins for the Lightning Network
* To increase your financial privacy while using Lightning
* To use some apps built on top of Lightning, like Lightning-powered instant messaging apps
* For financial freedom, independence, and sovereignty
There are costs associated with running a Lightning Network node. You need a computer, a permanent Internet connection, lots of disk space, and lots of time!
There are costs associated with running an LN node. You need a computer, a permanent internet connection, lots of disk space, and lots of time!
Operational costs will include electricity expenses.
But the skills you will learn from this experience are valuable and can be applied to a variety of other tasks too.
@ -26,62 +26,62 @@ Let's get started!
It is important that you set your own expectations correctly on accurate facts.
If you plan to operate a Lightning node _solely_ to gain income by earning routing fees,
please do your homework diligently first. Running a profitable business by operating a Lightning node is
definitely _not_ easy. Calculate all your initial and ongoing costs in a spreadsheet. Study Lightning Network statistics carefully.
definitely _not_ easy. Calculate all your initial and ongoing costs in a spreadsheet. Study LN statistics carefully.
What is the current payment volume? What is the volume per node? What are the current average routing fees? Consult forums and ask
for advice or feedback from other community members who have already gained real-world experience. Form your own educated opinion only
_after_ you have done this due diligence exercise. Most people will find their motivation for running a node not in financial gain,
but somewhere else.
====
=== Choosing your platform
=== Choosing Your Platform
There are many ways you can run a Lightning node ranging from a small mini-PC hosted in your home or a dedicated server to a hosted server in the cloud. The method you choose will depend on the resources you have and how much money you want to spend.
There are many ways you can run a Lightning node ranging from a small miniPC hosted in your home or a dedicated server to a hosted server in the cloud. The method you choose will depend on the resources you have and how much money you want to spend.
==== Why is reliability important for running a Lightning node?
[[continuous_operation]]
==== Why Is Reliability Important for Running a Lightning Node?
In Bitcoin, hardware is not particularly important unless one is specifically running a mining node.
The Bitcoin Core node software can be run on any machine that meets its minimum requirements and does not need to be online to receive payments; only to send them.
If a Bitcoin node goes down for an extended period of time, the user can simply reboot the node and once it connects to the rest of the network, it will re-sync the blockchain.
If a Bitcoin node goes down for an extended period of time, the user can simply reboot the node, and once it connects to the rest of the network, it will resync the blockchain.
In Lightning, however, the user needs to be online both to send _and_ to receive payments. If the Lightning node is offline, it cannot receive any payments from anyone and thus its open invoices cannot be fulfilled.
Furthermore, the open channels of an offline node cannot be used to route payments. Your channel partners will notice that you are offline and cannot contact you to route a payment. If you are offline too often, they may consider the bitcoin locked up in their channels with you to be underutilized capacity, and may close those channels. We already discussed the case of a protocol attack where your channel partner tries to cheat you by submitting an earlier commitment transaction. If you are offline and your channels aren't being monitored, then the attempted theft could succeed and you will have no recourse once the timelock expires.
In Lightning, however, the user needs to be online both to send _and_ to receive payments. If the Lightning node is offline, it cannot receive any payments from anyone, and thus its open invoices cannot be fulfilled.
Furthermore, the open channels of an offline node cannot be used to route payments. Your channel partners will notice that you are offline and cannot contact you to route a payment. If you are offline too often, they may consider the bitcoin locked up in their channels with you to be underutilized capacity, and may close those channels. We already discussed the case of a protocol attack in which your channel partner tries to cheat you by submitting an earlier commitment transaction. If you are offline and your channels aren't being monitored, then the attempted theft could succeed, and you will have no recourse once the timelock expires.
Hence, node reliability is extremely important for a Lightning node.
There are also the issues of hardware failure and loss of data. In Bitcoin, a hardware failure can be a trivial problem if the user has a backup of their mnemonic phrase or private keys. The Bitcoin wallet and the bitcoin inside the wallet can be easily restored from the private keys on a new computer. Most information can be re-downloaded from the blockchain.
There are also the issues of hardware failure and loss of data. In Bitcoin, a hardware failure can be a trivial problem if the user has a backup of their mnemonic phrase or private keys. The Bitcoin wallet and the bitcoin inside the wallet can be easily restored from the private keys on a new computer. Most information can be redownloaded from the blockchain.
In contrast, in Lightning the information about the user's channels, including the commitment transactions and revocation secrets, are not publicly known and are only stored on the individual user's hardware.
Thus, software and hardware failures in the Lightning Network can easily result in loss of funds.
==== Types of hardware Lightning nodes
==== Types of Hardware Lightning Nodes
There are three main types of hardware Lightning nodes:
* **General-purpose computers**: A Lightning Network node can be run on a home computer or laptop running Windows, Mac OS, or Linux. Typically this is run alongside a Bitcoin node.
* **Dedicated hardware**: A Lightning node can also be run on dedicated hardware like a Raspberry Pi, Rock64, or mini PC. This setup would usually run a software stack including a Bitcoin node and other applications. This setup is popular as the hardware is dedicated to running and maintaining the Lightning node only and is usually set up with an installation "helper".
* **Pre-configured hardware**: A Lightning Network node can also be run on purpose-built hardware specifically selected and configured for it. This would include "out-of-the-box" Lightning node solutions that can be purchased as a kit or a turn-key system.
* **General-purpose computers**: An LN node can be run on a home computer or laptop running Windows, macOS, or Linux. Typically this is run alongside a Bitcoin node.
* **Dedicated hardware**: A Lightning node can also be run on dedicated hardware like a Raspberry Pi, Rock64, or mini PC. This setup would usually run a software stack including a Bitcoin node and other applications. This setup is popular because the hardware is dedicated to running and maintaining the Lightning node only and is usually set up with an installation "helper."
* **Preconfigured hardware**: An LN node can also be run on purpose-built hardware specifically selected and configured for it. This would include "out-of-the-box" Lightning node solutions that can be purchased as a kit or a turnkey system.
==== Running in the "cloud"
==== Running in the "Cloud"
_Virtual Private Server_ (VPS) and "cloud computing" services such as Microsoft Azure, Google Cloud, Amazon Web Services (AWS), or DigitalOcean are quite affordable and can be set up very quickly. A Lightning node can be hosted for between $20 and $40 per month on such a service.
_Virtual private server_ (VPS) and cloud computing services such as Microsoft Azure, Google Cloud, Amazon Web Services (AWS), or DigitalOcean are quite affordable and can be set up very quickly. A Lightning node can be hosted for between $20 and $40 per month on such a service.
However, as the saying goes, "'Cloud' is just other people's computers". Using these services means running your node on other people's computers. This brings along the corresponding advantages and disadvantages. The key advantages are convenience, efficiency, uptime, and possibly even cost. The cloud operator manages and runs the node to a high degree automatically providing you with convenience and efficiency. They provide excellent uptime and availability, often much better than what an individual can achieve at home. If you consider that just the electricity cost of running a server in many western countries is around $10 per month, then add to that the cost of network bandwidth and the hardware itself, the VPS offering becomes financially competitive. Lastly, with a VPS you need no space for a PC at home, and don't have any issues with PC noise or heat.
On the other hand there are several notable disadvantages. A Lightning node running in the "cloud" will always be less secure and less private than one running on your own computer. Additionally, these cloud computing services are very centralized. The vast majority of Bitcoin and Lightning nodes running on such services are located in a handful of data centers in Virginia, Sunnyvale, Seattle, London, and Frankfurt. When the networks or data centers of these providers have service problems, it affects thousands of nodes on so-called "decentralized" networks.
However, as the saying goes, "‘Cloud’ is just other people's computers." Using these services means running your node on other people's computers. This brings along the corresponding advantages and disadvantages. The key advantages are convenience, efficiency, uptime, and possibly even cost. The cloud operator manages and runs the node to a high degree automatically providing you with convenience and efficiency. They provide excellent uptime and availability, often much better than what an individual can achieve at home. If you consider that just the electricity cost of running a server in many Western countries is around $10 per month, then add to that the cost of network bandwidth and the hardware itself, the VPS offering becomes financially competitive. Lastly, with a VPS you need no space for a PC at home and don't have any issues with PC noise or heat.
On the other hand, there are several notable disadvantages. A Lightning node running in the "cloud" will always be less secure and less private than one running on your own computer. Additionally, these cloud computing services are very centralized. The vast majority of Bitcoin and Lightning nodes running on such services are located in a handful of data centers in Virginia, Sunnyvale, Seattle, London, and Frankfurt. When the networks or data centers of these providers have service problems, it affects thousands of nodes on so-called "decentralized" networks.
If you have the possibility and capacity of running a node on your own computer at home or in your office, then this might be preferable to running it
in the cloud. Nonetheless, if running your own server is not an option, by all means consider running one on a VPS.
==== Running a node at home
==== Running a Node at Home
If you have a reasonable capacity Internet connection at home or in your office, you can certainly run a Lightning node there. Any "broadband" connection is sufficient for the purpose of running a lightweight node, and a fast connection will allow you to run a Bitcoin full node too.
If you have a reasonable-capacity internet connection at home or in your office, you can certainly run a Lightning node there. Any "broadband" connection is sufficient for the purpose of running a lightweight node, and a fast connection will allow you to run a Bitcoin full node too.
While you can run a Lightning node (and even a Bitcoin node) on your laptop, it will become annoying quite fast. These programs consume your computer's resources and need to run 24/7. Your user applications like your browser or your spreadsheet will find themselves competing against the Lightning background services for your computer's resources. In other words, your browser and other desktop workloads will be slowed down.
And when your word-processing app freezes up your laptop, your Lightning node will go down as well leaving you unable to receive transactions and potentially vulnerable to attacks. Furthermore, you should never turn off your laptop.
All this combined together results in a set-up that is not ideal. The same will apply to your daily-use personal desktop PC.
And when your word-processing app freezes up your laptop, your Lightning node will go down as well, leaving you unable to receive transactions and potentially vulnerable to attacks. Furthermore, you should never turn off your laptop.
All this combined together results in a setup that is not ideal. The same will apply to your daily-use personal desktop PC.
Instead, most users will choose to run a node on a dedicated computer.
Fortunately, you don't need a "server" class computer to do this.
You can run a Lightning node on a single-board computer, such as a Raspberry Pi or on a Mini PC (usually marketed as home theater PCs).
You can run a Lightning node on a single-board computer, such as a Raspberry Pi or on a mini PC (usually marketed as home theater PCs).
These are simple computers which are commonly used as a home automation hub or a media server.
They are relatively inexpensive, when compared to a PC or a laptop.
The advantage of a dedicated device as a platform for Lightning and Bitcoin nodes is that it can run continuously, silently, and unobtrusively on your home network, tucked behind your router or TV.
@ -89,113 +89,111 @@ No one will even know that this little box is actually part of a global banking
[WARNING]
====
Operating a node on a 32-bit operating system and/or 32-bit CPU is not recommended, as the node software may run into resource issues, causing a crash and possibly a loss of funds.
Operating a node on a 32-bit operating system and/or 32-bit CPU is not recommended, because the node software may run into resource issues, causing a crash and possibly a loss of funds.
====
==== What hardware is required to run a Lightning node?
==== What Hardware Is Required to Run a Lightning Node?
At a minimum, the following will be required to run a Lightning node:
* **CPU**: Sufficient processing power will be required to run a Bitcoin node, which will continuously download and validate new blocks. The user also needs to consider the Initial Block Download (IBD) when setting up a new Bitcoin node, which can take anywhere from several hours to several days. A 2-core or 4-core CPU is recommended.
* **CPU**: Sufficient processing power will be required to run a Bitcoin node, which will continuously download and validate new blocks. The user also needs to consider the initial block download (IBD) when setting up a new Bitcoin node, which can take anywhere from several hours to several days. A 2-core or 4-core CPU is recommended.
* **RAM**: A system with 2GB of RAM will _barely_ run both Bitcoin and Lightning nodes. It will perform much better with at least 4GB of RAM. The Initial Block Download will be especially challenging with less than 4GB of RAM. More than 8GB of RAM is unnecessary, because the CPU is the greater bottleneck for these types of services, due to cryptographic operations such as signature validation.
* **RAM**: A system with 2GB of RAM will _barely_ run both Bitcoin and Lightning nodes. It will perform much better with at least 4GB of RAM. The IBD will be especially challenging with less than 4GB of RAM. More than 8GB of RAM is unnecessary because the CPU is the greater bottleneck for these types of services, due to cryptographic operations such as signature validation.
* **Storage drive**: This can be a Hard Disk Drive (HDD) or a Solid State Drive (SSD).
* **Storage drive**: This can be a hard disk drive (HDD) or a solid state drive (SSD).
An SSD will be significantly quicker (but more expensive) for running a node.
Most of the storage is used for the Bitcoin blockchain, which is hundreds of gigabytes in size.
A fair tradeoff (cost for complexity) is to buy a small SSD to boot the OS from and a larger HDD to store large data objects (mostly databases).
A fair trade-off (cost for complexity) is to buy a small SSD to boot the OS from and a larger HDD to store large data objects (mostly databases).
[NOTE]
====
Raspberry Pis are a common choice for running node software, due to the cost and parts availability.
The OS that runs on the device usually boots from an SD card.
For most use-cases, this is a non-issue, but Bitcoin Core is notorious for being I/O heavy.
The OS that runs on the device usually boots from a secure digital (SD) card.
For most use cases, this is a nonissue, but Bitcoin Core is notorious for being I/O heavy.
You should make sure to place the Bitcoin blockchain and Lightning data directory on a different drive because long-term intensive I/O can cause an SD card to fail.
====
* **Internet connection**: A reliable Internet connection will be required to download new Bitcoin blocks, as well as to communicate with other Lightning peers. During operation the estimated data use ranges from 10GB to 100GB per month, depending on configuration. At startup, a Bitcoin full node downloads the full blockchain.
* **Internet connection**: A reliable internet connection will be required to download new Bitcoin blocks, as well as to communicate with other Lightning peers. During operation the estimated data use ranges from 10 to 100GB per month, depending on configuration. At startup, a Bitcoin full node downloads the full blockchain.
* **Power supply**: A reliable power supply is required as Lightning nodes need to be online at all times. A power failure will cause in-progress payments to fail. For heavy duty routing nodes, a backup or uninterruptible power supply (UPS) is useful in the event of power outages.
Ideally, you should connect your Internet router to this UPS as well.
* **Power supply**: A reliable power supply is required because Lightning nodes need to be online at all times. A power failure will cause in-progress payments to fail. For heavy duty routing nodes, a backup or uninterruptible power supply (UPS) is useful in the event of power outages.
Ideally, you should connect your internet router to this UPS as well.
* **Backup**: Backup is crucial as a failure can result in loss of data and hence in loss of funds.
* **Backup**: Backup is crucial because a failure can result in loss of data and hence in loss of funds.
The user will want to consider some kind of data backup solution. This could be a cloud-based automated backup to a server or web service the user controls. Alternatively, it could be an automated local hardware backup, such as a second hard drive. For the best results, both local and remote backup can be combined.
==== Switching server configuration in the cloud
==== Switching Server Configuration in the Cloud
When renting a cloud server, it is often cost effective to change the configuration between two phases of operation. A faster CPU and faster storage will be needed during the Initial Block Download (e.g. the first day). After the blockchain has synced, the CPU and storage speed requirements are much less, so the performance can be downgraded to a more cost-effective level.
When renting a cloud server, it is often cost effective to change the configuration between two phases of operation. A faster CPU and faster storage will be needed during the IBD (e.g., the first day). After the blockchain has synced, the CPU and storage speed requirements are much less, so the performance can be downgraded to a more cost-effective level.
For example, on Amazon's cloud, we would use a 8-16GB RAM, 8-core CPU (e.g. t3-large or m3.large) and faster 400GB SSD (1000+ provisioned IOPS) for the Initial Block Download, reducing its time to just 6-8 hours. Once that is complete, we would switch the server instance to a 2GB RAM, 2-core CPU (e.g. t3.small) and storage to a general purpose 1TB HDD. This will cost about the same as if you ran it on the slower server the entire time, but it will get you up and running in less than a day instead of having to wait almost a week for the IBD.
For example, on Amazon's cloud, we would use an 8–16 GB RAM, 8-core CPU (e.g. t3-large or m3.large) and faster 400GB SSD (1000+ provisioned input/output operations per second [IOPS]) for the IBD, reducing its time to just 6-8 hours. Once that is complete, we would switch the server instance to a 2GB RAM, 2-core CPU (e.g., t3.small) and storage to a general purpose 1TB HDD. This will cost about the same as if you ran it on the slower server the entire time, but it will get you up and running in less than a day instead of having to wait almost a week for the IBD.
===== Permanent data storage (drive)
If you use a mini PC or rent a server, the storage can be the most expensive part, costing as much as the computer and connectivity (data) added together.
Let's have a look at the different options available. First there are two main types of drives, Hard Disk Drives (HDDs) and Solid State Drives (SSDs). HDDs are cheaper and SSDs are faster, but both do the job.
Let's have a look at the different options available. First there are two main types of drives, HDDs and SSDs. HDDs are cheaper and SSDs are faster, but both do the job.
The fastest SSDs available today use the NVMe interface. The NVMe SSDs are faster in high-end machines, but also more costly.
The fastest SSDs available today use the Non-Volatile Memory Express (NVMe) interface. The NVMe SSDs are faster in high-end machines, but also more costly.
Traditional SATA-based SSDs are cheaper, but not as fast. SATA SSDs perform sufficiently well for your node setup.
Smaller computers might not be able to take advantage of NVMe SSDs.
For example, the Raspberry Pi 4 cannot benefit from them because of the limited bandwidth of its USB port.
To choose the size, let's look at the Bitcoin blockchain. As of August 2021, its size is 360GB including the transaction index and grows by roughly 60GB/year. If you want to have some margin available for future growth or to install other data on your node, purchase at least a 512GB drive or better yet a 1TB drive.
To choose the size, let's look at the Bitcoin blockchain. As of August 2021, its size is 360GB including the transaction index and grows by roughly 60 GB per year. If you want to have some margin available for future growth or to install other data on your node, purchase at least a 512GB drive or better yet a 1TB drive.
[[helpers]]
=== Using an installer or helper
=== Using an Installer or Helper
Installing a Lightning node or a Bitcoin node may be daunting if you are not familiar with a commandline environment. Luckily, there are a number of projects that make "helpers", i.e. software that installs and configures the various components for you. You will still need to learn some commandline incantations to interact with your node, but most of the initial work is done for you.
Installing a Lightning node or a Bitcoin node may be daunting if you are not familiar with a command-line environment. Luckily, there are a number of projects that make "helpers," i.e., software that installs and configures the various components for you. You will still need to learn some command-line incantations to interact with your node, but most of the initial work is done for you.
==== RaspiBlitz
One of the most popular and complete "helpers" is _RaspiBlitz_, a project built by Christian Rotzoll. It is intended to be installed on a Raspberry Pi 4. RaspiBlitz comes with a recommended hardware "kit" that you can build in a matter of hours or at most a weekend. If you attend a Lightning "hackathon" in your city, you are likely to see many people working on their RaspiBlitz setup, swapping tips, and helping each other. You can find the RaspiBlitz project here:
https://github.com/rootzoll/raspiblitz
One of the most popular and complete "helpers" is _RaspiBlitz_ (<<RaspiBlitz>>), a project built by Christian Rotzoll. It is intended to be installed on a Raspberry Pi 4. RaspiBlitz comes with a recommended hardware kit that you can build in a matter of hours or at most a weekend. If you attend a Lightning "hackathon" in your city, you are likely to see many people working on their RaspiBlitz setup, swapping tips, and helping each other. You can find the RaspiBlitz project on https://github.com/rootzoll/raspiblitz[GitHub].
image::images/raspiblitz.jpg[]
[[RaspiBlitz]]
.A RaspiBlitz node
image::images/mtln_0501.png[]
In addition to a Bitcoin and Lightning node, RaspiBlitz can install a number of additional services, such as:
* Tor (run as hidden service)
* ElectRS (Electrum server in Rust)
* BTCPayServer (cryptocurrency payment processor)
* BTCPayServer (cryptocurrency payment processor)
* BTC RPC Explorer (Bitcoin blockchain explorer)
* Ride The Lightning (Lightning node management GUI)
* LNDmanage (commandline interface for advanced channel management)
* LNDmanage (command-line interface for advanced channel management)
* Loop (submarine swaps service)
* JoinMarket (CoinJoin service)
==== myNode
_myNode_ is another popular open source "helper" project including a lot of Bitcoin related software. It is easy to install: you "flash" the installer onto an SD card and boot your mini-PC from the SD card. You do not need any monitor to use myNode as the administrative tools are accessible remotely from a browser. If your mini-PC has no monitor, mouse, or keyboard; you can manage it from another computer or even from your smartphone. Once installed, go to http://mynode.local/ and create a Lightning wallet and node in two clicks.
==== Mynode
You can find the myNode project here:
https://mynodebtc.com/
https://mynodebtc.com/[_myNode_] is another popular open source "helper" project including a lot of Bitcoin related software. It is easy to install: you "flash" the installer onto an SD card and boot your mini PC from the SD card. You do not need any monitor to use myNode because the administrative tools are accessible remotely from a browser. If your mini PC has no monitor, mouse, or keyboard, you can manage it from another computer or even from your smartphone. Once installed, go to http://mynode.local/ and create a Lightning wallet and node in two clicks.
In addition to a Bitcoin and Lightning node, myNode can optionally install a variety of additional services, such as:
* Ride The Lightning (Lightning node management GUI)
* OpenVPN (VPN Support for remote management or wallet)
* LNDmanage (commandline interface for advanced channel management)
* BTC RPC Explorer (A Bitcoin blockchain explorer)
* OpenVPN (virtual private network [VPN] support for remote management or wallet)
* LNDmanage (command-line interface for advanced channel management)
* BTC RPC Explorer (a Bitcoin blockchain explorer)
==== Umbrel
Famous for their UX/UI, _Umbrel_ provides a very easy and accessible way to get your Bitcoin and Lightning node up and running in no time, especially for beginners. A very distinctive feature is that _Umbrel_ utilizes Neutrino/SPV during the Initial Blockchain Download (IBD) so you can instantly start using your node. Once Bitcoin Core is fully synced in the background it automatically switches over and disables SPV mode. Umbrel OS supports the Raspberry Pi 4 and can also be installed on any Linux-based OS or on a virtual machine on macOS or Windows. You can also connect any wallet that supports Bitcoin Core P2P, Bitcoin Core RPC, the Electrum protocol or lndconnect.
Famous for their UX/UI (shown in <<umbrel>>), _Umbrel_ provides a very easy and accessible way to get your Bitcoin and Lightning node up and running in no time, especially for beginners. A very distinctive feature is that Umbrel utilizes Neutrino/SPV during the Initial Blockchain Download (IBD) so you can instantly start using your node. Once Bitcoin Core is fully synced in the background it automatically switches over and disables SPV mode. Umbrel OS supports the Raspberry Pi 4 and can also be installed on any Linux-based OS or on a virtual machine on macOS or Windows. You can also connect any wallet that supports Bitcoin Core P2P, Bitcoin Core RPC, the Electrum protocol, or lndconnect.
There's no need to wait for a rainy day - you can find the project here: https://getumbrel.com
There's no need to wait for a rainy day—you can go right to https://getumbrel.com[Umbrel] to learn more.
image::images/umbrel.png[]
[[umbrel]]
.The Umbrel web interface
image::images/mtln_0502.png["The Umbrel web interface"]
In addition to a Bitcoin and Lightning node, Umbrel introduced the Umbrel App Store, where you can easily install additional services, such as:
* Lightning Terminal (interface for managing channel liquidity, loop-in, and loop-out)
* Ride The Lightning (Lightning node management GUI)
* Specter Desktop (watch-only coordinator for multi-signature and single-key Bitcoin wallets)
* BTCPayServer (cryptocurrency payment processor)
* Specter Desktop (watch-only coordinator for multisignature and single-key Bitcoin wallets)
* BTCPayServer (cryptocurrency payment processor)
* BTC RPC Explorer (Bitcoin blockchain explorer)
* ThunderHub (monitor and manage your node)
* Sphinx Relay (handling connectivity and storage for Sphinx chat)
@ -206,14 +204,12 @@ Umbrel is currently still in beta and not considered secure.
==== BTCPay Server
While not initially designed as an installation "helper", the e-commerce and payment platform _BTCPay Server_ has an incredibly easy installation system that uses Docker containers and +docker-compose+ to install a Bitcoin node, Lightning node, and payment gateway, among many other services. It can be installed on a variety of hardware platforms, from a simple Raspberry Pi 4 (4GB recommended) to a mini PC, old laptop, desktop or server.
BTCPay Server is a fully-featured self-hosted self-custody e-commerce platform that can be integrated with many e-commerce platforms such as Wordpress Woocommerce and others. The installation of the full node is only a step of the e-commerce platform installation.
While originally developed as a feature-for-feature replacement of the _Bitpay_ commercial payment service and API, it has evolved past that to be a complete platform for BTC and Lightning services related to e-commerce. For many sellers or shops it is a one-shop turn-key solution to e-commerce.
While not initially designed as an installation "helper," the ecommerce and payment platform _BTCPay Server_ has an incredibly easy installation system that uses Docker containers and +docker-compose+ to install a Bitcoin node, Lightning node, and payment gateway, among many other services. It can be installed on a variety of hardware platforms, from a simple Raspberry Pi 4 (4 GB recommended) to a mini PC, old laptop, desktop, or server.
More information can be found at:
BTCPay Server is a fully featured self-hosted, self-custody ecommerce platform that can be integrated with many ecommerce platforms such as WordPress WooCommerce and others. The installation of the full node is only a step of the ecommerce platform installation.
While originally developed as a feature-for-feature replacement of the _Bitpay_ commercial payment service and API, it has evolved past that to be a complete platform for BTC and Lightning services related to ecommerce. For many sellers or shops it is a one-shop turnkey solution to ecommerce.
https://btcpayserver.org/
More information can be obtained from https://btcpayserver.org/[BTCPay].
In addition to a Bitcoin and Lightning node, BTCPay Server can also install a variety of services, including:
@ -221,34 +217,34 @@ In addition to a Bitcoin and Lightning node, BTCPay Server can also install a va
* Litecoin support
* Monero support
* Spark server (c-lightning web wallet)
* Charge server (c-lightning e-commerce API)
* Charge server (c-lightning ecommerce API)
* Ride The Lightning (Lightning node management web GUI)
* Many BTC forks
* BTCTransmuter (event-action automation service supporting currency exchange)
The number of additional services and features is growing rapidly, so the list above is only a small subset of what is available on the BTCPay Server platform.
The number of additional services and features is growing rapidly, so the preceding list is only a small subset of what is available on the BTCPay Server platform.
==== Bitcoin node or lightweight Lightning
==== Bitcoin Node or Lightweight Lightning
One critical choice for your setup will be the choice of the Bitcoin node and its configuration. _Bitcoin Core_, the reference implementation, is the most common choice but not the only choice available. One alternative choice is _btcd_, which is a Go-language implementation of a Bitcoin node. Btcd supports some features that are useful for running an LND Lightning node and are not available in Bitcoin Core.
A second consideration is whether you will run an _archival_ Bitcoin node with a full copy of the blockchain (some 350GB in mid-2021) or a _pruned_ blockchain that only keeps the most recent blocks. A pruned blockchain can save you some disk space, but you will still need to download the full blockchain at least once (during the Initial Block Download). Hence it won't save you any network traffic. Using a pruned node to run a Lightning node is still an experimental capability and might not support all the functionality. However, many people are running a node like that successfully.
A second consideration is whether you will run an _archival_ Bitcoin node with a full copy of the blockchain (some 350GB in mid-2021) or a _pruned_ blockchain that only keeps the most recent blocks. A pruned blockchain can save you some disk space, but you will still need to download the full blockchain at least once (during the IBD). Hence it won't save you any network traffic. Using a pruned node to run a Lightning node is still an experimental capability and might not support all the functionality. However, many people are running a node like that successfully.
Finally, you also have the option of not running a Bitcoin node at all. Instead you can operate the LND Lightning node in "lightweight" mode, using the _neutrino_ protocol to retrieve blockchain information from public Bitcoin nodes operated by others. Running like this means that you are taking resources from the Bitcoin network without offering any in return. Instead, you are offering your resources and contributing to the Lightning Network community. For smaller Lightning nodes this will generally reduce network traffic in comparison to running a full Bitcoin node.
Finally, you also have the option of not running a Bitcoin node at all. Instead you can operate the LND Lightning node in "lightweight" mode, using the Neutrino Protocol to retrieve blockchain information from public Bitcoin nodes operated by others. Running like this means that you are taking resources from the Bitcoin network without offering any in return. Instead, you are offering your resources and contributing to the LN community. For smaller Lightning nodes this will generally reduce network traffic in comparison to running a full Bitcoin node.
Keep in mind that operating a Bitcoin node allows you to support other services, besides and on top of a Lightning node. These other services may require an archival (not pruned) Bitcoin node and often can't run without a Bitcoin node. Consider upfront what other services you may want to run now or in the future to make an informed decision on the type of Bitcoin node you select.
The bottom line for this decision is: If you can afford a disk larger than 500GB, run a full archival Bitcoin node. You will be contributing resources to the Bitcoin system and helping others who cannot afford to do so. If you can't afford such a big disk, run a pruned node. If you can't afford the disk or the bandwidth for even a pruned node, run a lightweight LND node over neutrino.
The bottom line for this decision is: If you can afford a disk larger than 500GB, run a full archival Bitcoin node. You will be contributing resources to the Bitcoin system and helping others who cannot afford to do so. If you can't afford such a big disk, run a pruned node. If you can't afford the disk or the bandwidth for even a pruned node, run a lightweight LND node over Neutrino.
==== Operating system choice
==== Operating System Choice
The next step is to select an operating system for your node. The vast majority of Internet servers run on some variant of Linux. Linux is the platform of choice for the Internet because it is a powerful, open-source operating system. Linux, however, has a steep learning curve and requires familiarity with a commandline environment. It is often intimidating for new users.
The next step is to select an operating system for your node. The vast majority of internet servers run on some variant of Linux. Linux is the platform of choice for the internet because it is a powerful open source operating system. Linux, however, has a steep learning curve and requires familiarity with a command-line environment. It is often intimidating for new users.
Ultimately, most of the services can be run on any modern POSIX operating system, which includes Mac OS, Windows, and of course Linux. Your choice should be driven more by your familiarity and comfort with an operating system and your learning objectives. If you want to expand your knowledge and learn how to operate a Linux system, this is a great opportunity to do so with a specific project and a clear goal. If you just want to get a node up and running, go with what you know.
Ultimately, most of the services can be run on any modern POSIX operating system, which includes macOS, Windows, and of course Linux. Your choice should be driven more by your familiarity and comfort with an operating system and your learning objectives. If you want to expand your knowledge and learn how to operate a Linux system, this is a great opportunity to do so with a specific project and a clear goal. If you just want to get a node up and running, go with what you know.
Nowadays, many services are also delivered in the form of containers, usually based on the _Docker_ system. These containers can be deployed on a variety of operating systems, abstracting the underlying OS. You may need to learn some Linux CLI commands nonetheless, as most of the containers run some variant of Linux inside.
=== Choose your Lightning node implementation
=== Choose Your Lightning Node Implementation
As with the choice of operating system, your choice of Lightning node implementation should depend primarily on your familiarity with the programming language and development tools used by the projects. While there are some small differences in features between the various node implementations, those are relatively minor and most implementations converge on the common standards defined by the BOLTs.
@ -258,47 +254,47 @@ Familiarity with the programming language and build system, on the other hand, i
* go utilities for LND
* Java/Maven for Eclair
The programming language doesn't just influence the choice of build system, but also many other aspects of the program. Each programming language comes with a whole design philosophy and affects many other aspects, such as:
The programming language influences not only the choice of build system but also many other aspects of the program. Each programming language comes with a whole design philosophy and affects many other aspects, such as:
* format and syntax of configuration files
* file locations (in the filesystem)
* command line arguments and their syntax
* error message formatting
* prerequisite libraries
* Remote Procedure Call interfaces
* Format and syntax of configuration files
* File locations (in the filesystem)
* Command-line arguments and their syntax
* Error message formatting
* Prerequisite libraries
* Remote procedure call interfaces
When you choose your Lightning node, you are also choosing all of the above characteristics. So your familiarity with these tools and design philosophies will make it easier to run a node. Or harder, if you land in an unfamiliar domain.
When you choose your Lightning node, you are also choosing all the aforementioned characteristics. So your familiarity with these tools and design philosophies will make it easier to run a node. Or harder, if you land in an unfamiliar domain.
On the other hand, if this is your first foray into the commandline and server/service environment, you will find yourself unfamiliar with any implementation and have the opportunity to learn something completely new. In that case you might want to decide based on a number of other factors, such as:
On the other hand, if this is your first foray into the command-line and server/service environment, you will find yourself unfamiliar with any implementation and have the opportunity to learn something completely new. In that case you might want to decide based on a number of other factors, such as:
* quality of support forums and chat rooms
* quality of documentation
* degree of integration with other tools you want to run
* Quality of support forums and chat rooms
* Quality of documentation
* Degree of integration with other tools you want to run
As a final consideration, you may want to examine the performance and reliability of different node implementations. This is especially important if you will be using this node in a production environment and expect heavy traffic and high reliability requirements. This might be the case if you plan to run the payment system of a shop on it.
=== Installing a Bitcoin or Lightning node
=== Installing a Bitcoin or Lightning Node
You decided not to use an installation "helper" and instead to dive into the command line of a Linux operating system? That is a brave decision and we'll try to help you make it work. If you'd rather not try to do this manually, consider using an application that helps you install the node software or a container-based solution, as described in <<helpers>>.
You decided not to use an installation "helper" and instead to dive into the command line of a Linux operating system? That is a brave decision, and we'll try to help you make it work. If you'd rather not try to do this manually, consider using an application that helps you install the node software or a container-based solution, as described in <<helpers>>.
[WARNING]
====
This section will delve into the advanced topic of system administration from the command line. Linux administration is its own skill set that is outside the scope of this book. It is a complicated topic and there are many pitfalls. Proceed with caution!
====
In the next few sections we will briefly describe how to install and configure a Bitcoin and Lightning node on a Linux operating system. You will need to review the installation instructions for the specific Bitcoin and Lightning node applications you decided to use. You can usually find these in a file called +INSTALL+ or in the +docs+ sub-directory of each project. We will only describe some of the common steps that apply to all such services, and the instructions we offer will be necessarily incomplete.
In the next few sections we will briefly describe how to install and configure a Bitcoin and Lightning node on a Linux operating system. You will need to review the installation instructions for the specific Bitcoin and Lightning node applications you decided to use. You can usually find these in a file called _INSTALL_ or in the +docs+ subdirectory of each project. We will only describe some of the common steps that apply to all such services, and the instructions we offer will be necessarily incomplete.
==== Background services
==== Background Services
For those accustomed to running applications on their desktop or smartphone, an application always has a graphical user interface even if it may sometimes run in the background. The Bitcoin and Lightning node applications, however, are very different. These applications do not have a graphical user interface built in. Instead, they run as _headless_ background services, meaning they are always operating in the background and do not interact with the user directly.
This can create some confusion for users who are not used to running background services. How do you know if such a service is currently running? How do you start and stop it? How do you interact with it? The answers to these questions depend on the operating system you are using. For now we will assume you are using some Linux variant and answer them in that context.
==== Process isolation
==== Process Isolation
Background services usually run under a specific user account in order to isolate them from the operating system and each other. For example, Bitcoin Core is configured to run as user +bitcoin+. You will need to use the command line to create a user for each of the services you run.
Background services usually run under a specific user account to isolate them from the operating system and each other. For example, Bitcoin Core is configured to run as user +bitcoin+. You will need to use the command line to create a user for each of the services you run.
In addition, if you have connected an external drive, you will need to tell the operating system to relocate the user's home directory to that drive. That's because a service like Bitcoin Core will create files under the user's home directory. If you are setting it up to download the full Bitcoin blockchain, these files will take up several hundred gigabytes. Here, we assume you have connected the external drive and it is located on the +/external_drive/+ path of the operating system.
In addition, if you have connected an external drive, you will need to tell the operating system to relocate the user's home directory to that drive. That's because a service like Bitcoin Core will create files under the user's home directory. If you are setting it up to download the full Bitcoin blockchain, these files will take up several hundred gigabytes. Here, we assume you have connected the external drive and it is located on the _/external_drive/_ path of the operating system.
On most Linux systems you can create a new user with the +useradd+ command, like this:
@ -306,11 +302,11 @@ On most Linux systems you can create a new user with the +useradd+ command, like
The +m+ and +d+ flags create the user's home directory as specified by /external_drive/bitcoin in this case. The +s+ flag assigns the user's interactive shell. In this case, we set it to +/dev/null+ to disable interactive shell use. The last argument is the new user's username +bitcoin+.
The +m+ and +d+ flags create the user's home directory as specified by _/external_drive/bitcoin_ in this case. The +s+ flag assigns the user's interactive shell. In this case, we set it to _/dev/null_ to disable interactive shell use. The last argument is the new user's username +bitcoin+.
==== Node startup
==== Node Startup
For both Bitcoin and Lightning node services, "installation" also involves creating a socalled _startup script_ to make sure that the node starts when the computer boots. Startup and shutdown of background services is handled by an operating system process, which in Linux is called _init_ or _systemd_. You can usually find a system startup script in the +contrib+ sub-directory of each project. For example, if you are on a modern Linux OS that uses +systemd+, you would find a script called +bitcoind.service+, that can start and stop the Bitcoin Core node service.
For both Bitcoin and Lightning node services, "installation" also involves creating a so-called _startup script_ to make sure that the node starts when the computer boots. Startup and shutdown of background services is handled by an operating system process, which in Linux is called +init+ or +systemd+. You can usually find a system startup script in the +contrib+ subdirectory of each project. For example, if you are on a modern Linux OS that uses +systemd+, you would find a script called _bitcoind.service_ that can start and stop the Bitcoin Core node service.
Here's an example of what a Bitcoin node's startup script looks like, taken from the Bitcoin Core code repository:
To configure your node, you need to create and reference a configuration file. By convention, this file is usually created in +/etc+, under a directory with the name of the program. For example, Bitcoin Core and LND configurations would usually be stored in:
@ -422,15 +418,15 @@ bitcoin-rpcpassword=PASSWORD
In general, it is a good idea to minimize the amount of customization of these systems. The default configuration is carefully designed to support the most common deployments. If you modify a default value, it may cause problems later on or reduce the performance of your node. In short, modify only when necessary!
==== Network configuration
==== Network Configuration
Network configuration is normally not an issue when configuring a new application. However, peer-to-peer networks like Bitcoin and the Lightning network present some unique challenges for network configuration.
In a centralized service, your computer connects to the "big servers" of some corporation, and not vice-versa. Your home Internet connection is actually configured on the assumption that you are simply a consumer of services provided by others. But in a peer-to-peer system, every peer both consumes from and provides services to other nodes. If you're running a Bitcoin or Lightning node at your home, you're providing a service to other computers on the Internet. Your Internet service by default is not configured to allow you to run servers and may need some additional configuration to enable others to reach your node.
In a centralized service, your computer connects to the "big servers" of some corporation, and not vice-versa. Your home internet connection is actually configured on the assumption that you are simply a consumer of services provided by others. But in a peer-to-peer system, every peer both consumes from and provides services to other nodes. If you're running a Bitcoin or Lightning node at your home, you're providing a service to other computers on the internet. Your internet service by default is not configured to allow you to run servers and may need some additional configuration to enable others to reach your node.
If you want to run a Bitcoin or Lightning node, you need to make it possible for other nodes on the Internet to connect to you. That means enabling incoming TCP connections to the Bitcoin port (port 8333 by default) or Lightning port (port 9735 by default). While you can run a Bitcoin node without incoming connectivity, you can't do that with a Lightning node. A Lightning node must be accessible to others from outside your network.
If you want to run a Bitcoin or Lightning node, you need to make it possible for other nodes on the internet to connect to you. That means enabling incoming TCP connections to the Bitcoin port (port 8333 by default) or Lightning port (port 9735 by default). While you can run a Bitcoin node without incoming connectivity, you can't do that with a Lightning node. A Lightning node must be accessible to others from outside your network.
By default, your home Internet router does not expect incoming connections from the outside, and in fact incoming connections are blocked. Your Internet router IP address is the only externally accessible IP address, and all the computers you run inside your home network share that single IP address. This is achieved by a mechanism called _Network Address Translation (NAT)_ which allows your Internet router to act as an intermediary for all outbound connections. If you want to allow an inbound connection you have to set up _Port Forwarding_, which tells your Internet router that incoming connections on specific ports should be forwarded to specific computers inside the network. You can do this manually by changing your Internet router configuration or, if your router supports it, through an automatic port forwarding mechanism called _Universal Plug and Play (UPNP)_.
By default, your home internet router does not expect incoming connections from the outside, and in fact incoming connections are blocked. Your internet router IP address is the only externally accessible IP address, and all the computers you run inside your home network share that single IP address. This is achieved by a mechanism called _Network Address Translation (NAT)_ which allows your internet router to act as an intermediary for all outbound connections. If you want to allow an inbound connection you have to set up _Port Forwarding_, which tells your internet router that incoming connections on specific ports should be forwarded to specific computers inside the network. You can do this manually by changing your internet router configuration or, if your router supports it, through an automatic port forwarding mechanism called _Universal Plug and Play (UPNP)_.
An alternative mechanism to port forwarding is to enable The Onion Router (Tor), which provides a kind of virtual private network overlay that allows incoming connections to an _onion address_. If you run Tor, you don't need to do port forwarding nor enable incoming connections to Bitcoin or Lightning ports. If you run your nodes using Tor, all traffic goes through Tor and no other ports are used.
@ -438,7 +434,7 @@ Let's look at different ways you can make it possible for others to connect to y
===== It just works!
There is a possibility that your Internet service provider or router is configured to support UPNP by default and everything just works automatically. Let's try this approach first, just in case we are lucky.
There is a possibility that your internet service provider or router is configured to support UPNP by default and everything just works automatically. Let's try this approach first, just in case we are lucky.
Assuming you already have a Bitcoin or Lightning node running, we will try and see if they are accessible from the outside.
@ -457,15 +453,15 @@ By default, these services only allow you to check incoming connections to the I
[[ln_port_check]]
.Checking for incoming port 9735
image::images/ln_port_check.png[]
image::images/mtln_0503.png[]
In <<ln_port_check>> you can see the result of checking port 9735 on a server running Lightning, using the +whatismyip.org+ port scanner tool. It shows that the server is accepting incoming connections to the Lightning port. If you see a result like this, you are all set!
===== Automatic port forwarding using UPNP
===== Automatic port forwarding using upnp
Sometimes, even if your Internet router supports UPNP, it may be turned off by default. In that case you need to change your Internet router configuration from its web administration interface:
Sometimes, even if your internet router supports UPNP, it may be turned off by default. In that case you need to change your internet router configuration from its web administration interface:
. Connect to your Internet router's configuration website. Usually this can be done by connecting to the _gateway address_ of your home network using a web browser. You can find the gateway address by looking at the IP configuration of any computer on your home network. It is often the first address in one of the non-routable networks, like 192.168.0.1 or 10.0.0.1. Check all stickers on your router as well for the _gateway address_. Once found, open a browser and enter the IP address into the browser URL/Search box, e.g. "192.168.0.1" or "http://192.168.0.1".
. Connect to your internet router's configuration website. Usually this can be done by connecting to the _gateway address_ of your home network using a web browser. You can find the gateway address by looking at the IP configuration of any computer on your home network. It is often the first address in one of the non-routable networks, like 192.168.0.1 or 10.0.0.1. Check all stickers on your router as well for the _gateway address_. Once found, open a browser and enter the IP address into the browser URL/Search box, e.g. "192.168.0.1" or "http://192.168.0.1".
. Find the administrator username and password for the web configuration panel of the router. This is often written on a sticker on the router itself and may be as simple as "admin" and "password". A quick web search for your ISP and router model can also help you find this information.
@ -473,9 +469,9 @@ Sometimes, even if your Internet router supports UPNP, it may be turned off by d
Restart your Bitcoin and/or Lightning node and repeat the open port test with one of the websites we used in the previous section.
===== Using Tor for incoming connections
===== Using tor for incoming connections
_The Onion Router (Tor)_ is a virtual private network with the special property that it encrypts communications between hops, such that any intermediary node cannot determine the origin or destination of a packet. Both Bitcoin and Lightning nodes support operation over Tor, which enables you to operate a node without revealing your IP address or location. Hence, it provides a high level of privacy to your network traffic. An added benefit of running Tor is that because it operates as a VPN, it resolves the problem of port forwarding from your Internet router. Incoming connections are received over the Tor tunnel, and your node can be found through an ad-hoc generated _onion address_ instead of an IP address.
_The Onion Router (Tor)_ is a virtual private network with the special property that it encrypts communications between hops, such that any intermediary node cannot determine the origin or destination of a packet. Both Bitcoin and Lightning nodes support operation over Tor, which enables you to operate a node without revealing your IP address or location. Hence, it provides a high level of privacy to your network traffic. An added benefit of running Tor is that because it operates as a VPN, it resolves the problem of port forwarding from your internet router. Incoming connections are received over the Tor tunnel, and your node can be found through an ad-hoc generated _onion address_ instead of an IP address.
Enabling Tor requires two steps: First you must install the Tor router and proxy on your computer. Second, you must enable the use of the Tor proxy in your Bitcoin or Lightning configuration.
If everything is working properly, the response of this command should be +"Congratulations. This browser is configured to use Tor."+.
Due to the nature of Tor, you can't easily use an external service to check if your node is reachable via an onion address. Nonetheless, you should see your Tor onion address in the logs of your Lightning node. It is a long string of letters and numbers followed by the suffix +.onion+. Your node should now be reachable from the Internet, with the added bonus of privacy!
Due to the nature of Tor, you can't easily use an external service to check if your node is reachable via an onion address. Nonetheless, you should see your Tor onion address in the logs of your Lightning node. It is a long string of letters and numbers followed by the suffix +.onion+. Your node should now be reachable from the internet, with the added bonus of privacy!
===== Manual port forwarding
This is the most complex process and requires quite a bit of technical skill. The details depend on the type of Internet router you have, your service provider settings and policies, and a lot of other context. Try UPNP or Tor first, before you try this much more difficult mechanism.
This is the most complex process and requires quite a bit of technical skill. The details depend on the type of internet router you have, your service provider settings and policies, and a lot of other context. Try UPNP or Tor first, before you try this much more difficult mechanism.
The basic steps are as follows:
. Find the IP address of the computer your node is on. This is usually dynamically allocated by the Dynamic Host Configuration Protocol (DHCP) and is often somewhere in the 192.168.x.x or 10.x.x.x range.
. Find the Media Access Control (MAC) address of your node's network interface. This can be found in the Internet settings of that computer.
. Find the Media Access Control (MAC) address of your node's network interface. This can be found in the internet settings of that computer.
. Assign a static IP address for your node so that it is always the same one. You can use the IP address it currently has. On your Internet router, look for "Static Leases" under the DHCP configuraiton. Map the MAC address to the IP address you selected. Now your node will always have that IP address allocated to it. Alternatively, you can look at your router's DHCP configuration and find out what its DHCP address range is. Select an unused address _outside_ of the DHCP address range. Then, on the server, configure the network to stop using DHCP and hard-code the selected non-DHCP IP address into the operating system network configuration.
. Assign a static IP address for your node so that it is always the same one. You can use the IP address it currently has. On your internet router, look for "Static Leases" under the DHCP configuration. Map the MAC address to the IP address you selected. Now your node will always have that IP address allocated to it. Alternatively, you can look at your router's DHCP configuration and find out what its DHCP address range is. Select an unused address _outside_ of the DHCP address range. Then, on the server, configure the network to stop using DHCP and hard-code the selected non-DHCP IP address into the operating system network configuration.
. Finally, set up "Port Forwarding" on your Internet router to route incoming traffic on specific ports to the selected IP address of your server.
. Finally, set up "Port Forwarding" on your internet router to route incoming traffic on specific ports to the selected IP address of your server.
Once done re-configuring, repeat the port check using one of the websites from the previous sections.
Once done reconfiguring, repeat the port check using one of the websites from the previous sections.
=== Security of your node
=== Security of Your Node
A Lightning node is, by definition, a hot wallet. That means that the funds (both on-chain and off-chain) controlled by a Lightning node are directly controlled by keys that are loaded in the node's memory or stored on the node's hard disk. If a Lightning node is compromised, it is trivial to create on-chain or off-chain transactions to drain its funds. It is therefore critically important that you protect it from unauthorized access.
@ -533,7 +529,7 @@ Security is a holistic effort, meaning that you have to secure every layer of a
Despite all security measures you will take, remember that the Lightning Network is an early-stage experimental technology and there are likely to be exploitable bugs in the code of any project you use. *Do not put more money than you are willing to risk losing on the Lightning Network.*
==== Operating system security
==== Operating System Security
Securing an operating system is a vast topic that is beyond the scope of this book. However, we can establish some basic principles.
@ -548,11 +544,11 @@ To secure your operating system, here are some of the top items to consider:
. Two-factor authentication (2FA): Use two-factor authentication wherever possible, including Universal 2-Factor (U2F) with hardware security keys. This applies to all external services you might be using such as your cloud service provider. You can apply this also to your own set-up such as your own SSH configuration. Use 2FA also for indirect services. For example, say you are using a cloud service. You gave your cloud service provider an email address, so you should also protect your email address with 2FA.
. Backup: Make backups of your system, and make sure you protect the backups with encryption too. Perform these backups periodically. At least once test if you can restore your backup and that your backup is complete and accessible. If possible, keep one copy of your backups on a different disk to avoid that a single hard disk failure destroys _both_ your active node as well as your backup copies.
. Vulnerability and exposure management: Use remote scanning to ensure you have minimized the attack surface of your system. Close any unnecessary services or ports.
- Minimize: Install only software and packages that you really need and use. Uninstall packages that you no longer use. It is recommended that you do _not_ use your node computer for non-node activities that you can perform on another of your computers. Especially, if you can, do _not_ use your node computer for browsing, surfing the Internet, or reading your email.
- Minimize: Install only software and packages that you really need and use. Uninstall packages that you no longer use. It is recommended that you do _not_ use your node computer for non-node activities that you can perform on another of your computers. Especially, if you can, do _not_ use your node computer for browsing, surfing the internet, or reading your email.
This is a list of the most basic security measures. It is by no means exhaustive.
==== Node access
==== Node Access
Your Lightning node will expose a Remote Procedure Call (RPC) Application Programming Interface (API). This means that your node can be controlled remotely by commands sent to a specific TCP port. Access control to that RPC API is achieved by some form of user authentication. Depending on the type of Lightning node you set up, this will either be done by username/password authentication or by a mechanism called an authentication _macaroon_. As the name implies, a macaroon is a more sophisticated type of cookie. Unlike a cookie, it is cryptographically signed and can express a set of access capabilities.
@ -562,7 +558,7 @@ If you use a username/password authentication model, make sure you select a long
A plain alphanumeric sequence that is longer than 12 characters and randomly generated is usually sufficient. If you plan to store large amounts of money on your Lightning node and are concerned about remote brute-force attacks, select a password length of more than 20 characters to make such attacks practically infeasible.
=== Node and channel backups
=== Node and Channel Backups
A very important consideration when running a Lightning node is the issue of backups. Unlike a Bitcoin wallet, where a BIP39 mnemonic phrase can recover all the state of the wallet, in Lightning this is _not_ the case.
@ -583,21 +579,21 @@ Channel backup mechanisms are still a work-in-progress and a weakness in most Li
==== Static Channel Backups (SCB)
At the time of writing this book, only LND offers a built-in mechanism for static channel backups. Eclair has a similar mechanism deployed for server-side deployments, although Eclair mobile does offer optional backup to a Google Drive. C-lightning recently merged the necessary interfaces for a plugin to implement channel backups. Unfortunately, there is no consistent, agreed upon backup mechanism across different node implementations.
At the time of writing this book, only LND offers a built-in mechanism for static channel backups. Eclair has a similar mechanism deployed for server-side deployments, although Eclair mobile does offer optional backup to a Google Drive. C-lightning recently merged the necessary interfaces for a plug-in to implement channel backups. Unfortunately, there is no consistent, agreed upon backup mechanism across different node implementations.
File-based backups of the Lightning node databases are at best a partial solution because you run the risk of backing up an inconsistent database state. In addition, you may not reliably catch the latest state commitments. It is much better to have a backup mechanism that is triggered every time there is a state change in a channel, thereby ensuring data consistency.
To set up static channel backups in LND, set the +backupfilepath+ parameter either on the command line or in the configuration file. LND will then save an SCB file in that directory path. Of course, that's only the first step of the solution. Now, you have to setup a mechanism that monitors this file for changes. Each time the file changes the backup mechanism must copy this file to another, preferably "off-site" disk. Such backup mechanisms are beyond the scope of this book. Nonetheless, any sophisticated backup solution should be able to handle this scenario. Recall, the backup files should be encrypted too.
==== Hot wallet risk
==== Hot Wallet Risk
As we've discussed previously, the Lightning Network consists of a network of _hot wallets_. The funds you store in a Lightning wallet are *online all the time*. This makes them vulnerable. Hence, you should not store large amounts in a Lightning wallet. Large amounts should be kept in a _cold_ wallet that is _not_ online and which can transact only on-chain.
Even if you start small, as time passes you may still find you have a significant amount of money in a Lightning wallet. This is a typical scenario for store owners. If you use a Lightning node for an e-commerce operation, your wallet will likely receive funds often, but send funds rarely. You will therefore end up having two problems simultaneously: First, your channels will be imbalanced with large local balances outweighing small remote balances. Secondly, you will have too much money in the wallet. Fortunately, you can also solve both of these problems simultaneously.
Even if you start small, as time passes you may still find you have a significant amount of money in a Lightning wallet. This is a typical scenario for store owners. If you use a Lightning node for an ecommerce operation, your wallet will likely receive funds often, but send funds rarely. You will therefore end up having two problems simultaneously: First, your channels will be imbalanced with large local balances outweighing small remote balances. Secondly, you will have too much money in the wallet. Fortunately, you can also solve both of these problems simultaneously.
Let's look at some of the solutions you can use to reduce the funds exposed in a hot wallet.
==== Sweeping funds
==== Sweeping Funds
If your Lightning wallet balance becomes too large for your risk tolerance, you will need to "sweep" funds out of the wallet. You can do so in three ways: on-chain, off-chain, and loop-out. Let's look at each one of these options in the next few sections.
@ -605,7 +601,7 @@ If your Lightning wallet balance becomes too large for your risk tolerance, you
Sweeping funds on-chain is accomplished by moving the funds from the Lightning wallet to a Bitcoin wallet. You do that by closing channels. When you close a channel, all funds from your local balance are "swept" to a Bitcoin address. The Bitcoin address for on-chain funds is usually generated by your Lightning wallet so it is still a hot-wallet. You may need to do an additional on-chain transaction to move the funds to a more secure address, such as one generated on your hardware wallet.
Closing channels will incur an on-chain fee and will reduce your Lightning node's capacity and connectivity. However, if you run a popular e-commerce node, you will not lack incoming capacity and can strategically close channels with large local balances, essentially "bundling" your funds for movement on-chain. You may need to use some channel re-balancing techniques (see <<channel_rebalancing>>) before you close channels to maximize the benefits of this strategy.
Closing channels will incur an on-chain fee and will reduce your Lightning node's capacity and connectivity. However, if you run a popular ecommerce node, you will not lack incoming capacity and can strategically close channels with large local balances, essentially "bundling" your funds for movement on-chain. You may need to use some channel rebalancing techniques (see <<channel_rebalancing>>) before you close channels to maximize the benefits of this strategy.
===== Off-chain sweep
@ -627,11 +623,11 @@ A node operator can initiate a submarine swap and send all available channel bal
In the future, this could be a paid service offered by nodes on the Lightning Network who advertise exchange rates or charge a flat fee for the conversion.
The advantage of a submarine swap for sweeping funds is that no channel needs to be closed. That means that we preserve our channels, only re-balancing our channels through this operation. As we send a Lightning payment out, we shift some balance from local to remote on one or more of our channels. Not only does that reduce the balance exposed in our node's hot wallet, it also increases the balance available for future incoming payments.
The advantage of a submarine swap for sweeping funds is that no channel needs to be closed. That means that we preserve our channels, only rebalancing our channels through this operation. As we send a Lightning payment out, we shift some balance from local to remote on one or more of our channels. Not only does that reduce the balance exposed in our node's hot wallet, it also increases the balance available for future incoming payments.
You could do this by trusting an intermediary to act as a gateway, but this risks your coins being stolen. But in the case of a submarine swap, the operation does not require trust. Submarine swaps are non-custodial _atomic_ operations. That means that the counterparty in your submarine swap cannot steal your funds because the on-chain payment depends on the completion of the off-chain payment and vice-versa.
===== Submarine swaps with Loop
===== Submarine swaps with loop
One example of a submarine swap service is _Loop_ by Lightning Labs, the same company that builds LND. Loop comes in two variations: _Loop In_ and _Loop Out_. _Loop In_ accepts a Bitcoin on-chain payment and converts it into a Lightning off-chain payment. _Loop Out_ converts a Lightning payment into a Bitcoin payment.
@ -666,7 +662,7 @@ Run `loop monitor` to monitor progress.
Note that your maximum fee, which represents a worst-case scenario, will depend on the confirmation target that you select.
=== Lightning node uptime and availability
=== Lightning Node Uptime and Availability
Unlike Bitcoin, Lightning nodes need to be online almost continuously. Your node needs to be online to receive payments, open channels, close channels (cooperatively), and monitor protocol violations. Node availability is such an important requirement in the Lightning Network that it is a metric used by various automatic channel management tools (e.g. +autopilot+) to decide which nodes to open channels with. You can also see "availability" as a node metric on popular node explorers (see <<ln_explorer>>) such as +1ml.com+.
@ -674,7 +670,7 @@ Node availability is especially important to mitigate and resolve potential prot
Keeping a node online continuously is not easy, as various bugs and resource limitations can and will occasionally cause downtime. Especially if you run a busy and popular node, you will run into limitations of memory, swap space, number of open files, disk space, and so forth. A whole host of different problems will cause your node or your server to crash.
==== Tolerate faults and automate
==== Tolerate Faults and Automate
If you have the time and skills you should test some basic fault scenarios on the Lightning testnet. On the testnet you will learn valuable lessons without risking any funds. Any step you perform to automate your system will improve your availability.
@ -682,11 +678,11 @@ If you have the time and skills you should test some basic fault scenarios on th
- Automatic node restart: What happens when your node or one of your nodes crashes? Simulate this fault by killing the corresponding node processes. If a node crashes, it should automatically restart itself. Test it to make sure the node or nodes really restart automatically on failure without human intervention. If this is not the case, most likely your node is not set up correctly as an operating system service.
- Automatic network reconnection: What happens if your network goes down? What happens when your ISP goes temporarily down? What happens when your ISP assigns a new IP address to your router or your computer? When the network comes back, do the nodes you are running automatically reconnect to the network? Simulate this fault by unplugging and later re-plugging the Ethernet cable from the device hosting your nodes. The nodes should automatically reconnect and continue operation without human intervention.
- Automatic network reconnection: What happens if your network goes down? What happens when your ISP goes temporarily down? What happens when your ISP assigns a new IP address to your router or your computer? When the network comes back, do the nodes you are running automatically reconnect to the network? Simulate this fault by unplugging and later replugging the Ethernet cable from the device hosting your nodes. The nodes should automatically reconnect and continue operation without human intervention.
- Configure your log files: All of the above failures should leave textual entries behind in the corresponding log files. Turn up the verbosity of logging if needed. Find these error entries in the log files and use them for monitoring.
==== Monitoring node availability
==== Monitoring Node Availability
Monitoring your node is an important part of keeping it running. You need to monitor not only the availability of the computer itself, but also the availability and correct operation of the Lightning node software.
@ -747,21 +743,21 @@ OPTIONS:
--help, -h show help
----
C-lightning has the API hooks necessary for a watchtower client plugin, though no such plugin has been implemented yet.
C-lightning has the API hooks necessary for a watchtower client plug-in, though no such plug-in has been implemented yet.
Finally, a popular standalone watchtower server is _The Eye of Satoshi_ (TEOS). It can be found here:
https://github.com/talaia-labs/python-teos
=== Channel management
=== Channel Management
As a Lightning node operator, one of the recurring tasks you will need to perform is management of your channels. This means opening outbound channels from your node to other nodes, as well as getting other nodes to open inbound channels to your node. In the future, cooperative channel construction may be possible, so you can open symmetric channels that have funds committed on both ends on creation. For now, however, new channels only have funds on one end, on the originator's side. Hence, to make your node _balanced_ with both inbound and outbound capacity, you need to open channels to others and entice others to open channels to your node.
==== Opening outbound channels
==== Opening Outbound Channels
As soon as you get your Lightning node up and running, you can fund its Bitcoin wallet and then start opening channels with those funds.
You must choose channel partners carefully as your node's ability to send payments depends on who your channel partners are and how well connected they are to the rest of the Lightning Network. You also want to have more than one channel to avoid being susceptible to a single point of failure. Since Lightning now supports multi-path payments, you can split your initial funds into several channels and route bigger payments by combining their capacity. At the same time, avoid making your channels too small. Since you need to pay Bitcoin transaction fees to open and close a channel, the channel balance should not be so small that the on-chain fees consume a significant portion. It's all about balance!
You must choose channel partners carefully as your node's ability to send payments depends on who your channel partners are and how well connected they are to the rest of the Lightning Network. You also want to have more than one channel to avoid being susceptible to a single point of failure. Since Lightning now supports multipath payments, you can split your initial funds into several channels and route bigger payments by combining their capacity. At the same time, avoid making your channels too small. Since you need to pay Bitcoin transaction fees to open and close a channel, the channel balance should not be so small that the on-chain fees consume a significant portion. It's all about balance!
To summarize:
@ -782,7 +778,7 @@ Autopilots currently exist in 3 forms.
- +lnd+ incorporates an autopilot that is fully integrated with +lnd+ and runs constantly in the background while turned on.
- +lib_autopilot.py+ can offer autopilot computations for any node implementation based on the gossip and channel data.
- A +c-lightning+ plugin based on +lib_autopilot.py+ exists that provides an easy to use interface for +c-lightning+ users.
- A +c-lightning+ plug-in based on +lib_autopilot.py+ exists that provides an easy to use interface for +c-lightning+ users.
Be aware that the +lnd+ autopilot will start running in the background as soon as it is turned on via the config file. As a result it will start opening channels immediately if you have onchain outputs in your +lnd+ wallet.
If you want to have full control over the bitcoin transactions that you make and the channels that you open, make sure to turn the autopilot off _before_ you load your +lnd+ wallet with bitcoin funds.
@ -816,15 +812,15 @@ As is common, the amounts in the configuration file are enumerated in satoshi.
Currently channels below 1 mBTC are not very useful and we do not recommend you open channels that are too small and below this amount.
With the wider adoption of multipath payments, smaller channels are less of a burden. But for the time being, this is our recommendation.
The +c-lightning+ plugin which was orignally written by Rene Pickhardt works very differently in comparison to the +lnd+ autopilot.
The +c-lightning+ plug-in which was originally written by René Pickhardt works very differently in comparison to the +lnd+ autopilot.
First, it differs in the algorithms used to make the recommendations. We will not cover this here. Secondly, it differs in its user interface.
You will need to download the _autopilot plugin_ from the +c-lightning+ plugin repository at https://github.com/lightningd/plugins/tree/master/autopilot and activate it.
You will need to download the _autopilot plug-in_ from the +c-lightning+ plug-in repository at https://github.com/lightningd/plugins/tree/master/autopilot and activate it.
[NOTE]
====
In order to activate a plugin in +c-lightning+, place it into the +~/.lightning/plugins+ directory, ensure that it's executable (e.g. +chmod +x ~/.lightning/plugins/autopilot.py+), then restart +lightningd+.
In order to activate a plug-in in +c-lightning+, place it into the `~/.lightning/plugins` directory, ensure that it's executable (e.g. `chmod +x ~/.lightning/plugins/autopilot.py`), then restart +lightningd+.
Alternatively, if you don't want a plugin to automatically activate when you start +lightningd+ you can place it in a different directory and manually activate it with the +plugin+ argument to +lightningd+:
Alternatively, if you don't want a plug-in to automatically activate when you start +lightningd+ you can place it in a different directory and manually activate it with the +plugin+ argument to +lightningd+:
@ -857,11 +853,11 @@ These differences reflect personal preferences and could actually be the decidin
Keep in mind that current autopilots will primarily use public information from the gossip protocol about the current topology of the Lightning Network.
It is obvious that your personal requirements for channels can only be reflected to a certain degree.
More advanced autopilots would use historical and usage information that your node has gathered when running in the past, including information about routing successes, who you have paid in the past, and who paid you.
In the future, such improved autopilots might also use this collected data to make recommendations on closing channels and re-allocating funds.
In the future, such improved autopilots might also use this collected data to make recommendations on closing channels and reallocating funds.
Overall, at the time of writing of this book, be cautious not to depend or rely too heavily on autopilots.
==== Getting inbound liquidity
==== Getting Inbound Liquidity
In the current design of the Lightning Network, it is more typical for users to obtain outbound liquidity _before_ obtaining inbound liquidity.
They will do so by opening a channel with another node, and more often they'll be able to spend bitcoin before they can receive it.
@ -894,7 +890,7 @@ In the future these challenges can be partially mitigated by the implementation
Ultimately, Lightning users need to be strategic and proactive about channel management to ensure that sufficient inbound capacity is available to meet their needs.
==== Closing channels
==== Closing Channels
As discussed earlier in the book, a _mutual close_ is the preferred way of closing a channel. However, there are instances where a _force close_ is necessary.
@ -905,25 +901,25 @@ Some examples:
* Your channel partner is online and your nodes are negotiating a mutual close, but they become stuck and cannot reach a resolution.
[[channel_rebalancing]]
==== Re-balancing channels
==== Rebalancing Channels
In the course of transacting and routing payments on Lightning, the combination of inbound and outbound capacities can become unbalanced.
For example, if one of your channel partners is frequently routing payments through your node, you will exhaust the inbound capacity on that channel, while also exhausting the outbound capacity on the outgoing channels. Once that happens, you can no longer route payments through that route.
There are many ways to re-balance channels, each with different advantages and disadvantages. One way is to use a _submarine swap_ (e.g. +Loop-Out+) as described previously in this chapter. Another way to re-balance is to simply wait for routed payments that flow in the opposite direction. If your node is well connected, when a specific route becomes exhausted in one direction, the same route becomes available in the opposite direction. Other nodes may "discover" that route in the opposite direction and start using it as part of their payment path, thereby re-balancing the funds again.
There are many ways to rebalance channels, each with different advantages and disadvantages. One way is to use a _submarine swap_ (e.g. +Loop-Out+) as described previously in this chapter. Another way to rebalance is to simply wait for routed payments that flow in the opposite direction. If your node is well connected, when a specific route becomes exhausted in one direction, the same route becomes available in the opposite direction. Other nodes may "discover" that route in the opposite direction and start using it as part of their payment path, thereby rebalancing the funds again.
A third way to re-balance channels is to purposefully create a _circular route_ that sends a payment from your node back to your node, via the Lightning Network. By sending a payment out on a channel with large local capacity and arranging the path so that it returns to your node on a channel with large remote capacity, both of those channels will become more balanced. An example of a circular route re-balancing strategy can be seen in <<circular_rebalancing>>.
A third way to rebalance channels is to purposefully create a _circular route_ that sends a payment from your node back to your node, via the Lightning Network. By sending a payment out on a channel with large local capacity and arranging the path so that it returns to your node on a channel with large remote capacity, both of those channels will become more balanced. An example of a circular route rebalancing strategy can be seen in <<circular_rebalancing>>.
[[circular_rebalancing]]
.Circular route re-balancing
image::images/circular-rebalancing.png[]
.Circular route rebalancing
image::images/mtln_0504.png[]
Circular re-balancing is supported by most Lightning node implementations and can be done on the command line or via one of the web management interfaces such as _Ride the Lightning (RTL)_ (see <<rtl>>).
Circular rebalancing is supported by most Lightning node implementations and can be done on the command line or via one of the web management interfaces such as _Ride the Lightning (RTL)_ (see <<rtl>>).
Channel rebalancing is a complex issue that is the subject of active research and covered in more detail in <<channel_rebalancing>>.
=== Routing fees
=== Routing Fees
Running a Lightning node allows you to earn fees by routing payments across your channels. Routing fees are generally not a significant source of income and dwarfed by the cost of operating a node. For example, on a relatively busy node that routes a dozen payments a day, the fees amount to no more than 2000 satoshis.
@ -937,34 +933,33 @@ The unit of _millionths_ is often abbreviated with _ppm_ (parts per million).
For example, a _base fee_ of 1,000 (millisatoshi) and a _fee rate_ of 1,000 ppm (millionths) would result in the following charges for a 100,000 satoshi payment:
Broadly speaking, you can take one of two approaches to routing fees. You can route lots of payments with low fees, making up for low fees by high volume. Alternatively, you can choose to charge higher fees. If you choose to set higher fees, your node will be selected only when other cheaper routes don't exist. Therefore, you will route less frequently but earn more per successful routing.
For most nodes, it is usually best to use the default routing fee values. This way, your node is competing on a mostly level playing field with other nodes who use the default values.
You can also use the routing fee settings to re-balance channels. If most of your channels have the default fees but you want to rebalance a particular channel, just decrease the fees on that specific channel to zero or to very low rates. Then sit back and wait for someone to route a payment over your "cheap" route and re-balance your channels for you as a side-effect.
You can also use the routing fee settings to rebalance channels. If most of your channels have the default fees but you want to rebalance a particular channel, just decrease the fees on that specific channel to zero or to very low rates. Then sit back and wait for someone to route a payment over your "cheap" route and rebalance your channels for you as a side-effect.
=== Node management
=== Node Management
Managing your Lightning node on the command line is obviously not easy. It gives you the full flexibility of the node's API and the ability to write your own custom scripts to satisfy your personal requirements. But if you don't want to deal with the complexity of the command line and only need some basic node management capabilities, you should consider installing a web-based user interface that makes node management easier.
There are a number of competing projects that offer web-based Lightning node management. Some of the most popular ones are described below.
[[rtl]]
==== Ride The Lightning (RTL)
==== Ride the Lightning (RTL)
_RTL_ is a graphical web user interface to help users manage Lightning node operations for the three main Lightning node implementations (LND, c-lightning, and Eclair). RTL is an open source project developed by Suheb, Shahana Farooqui, and many other contributors. You can find the RTL software here:
The five layers of the Lightning Network, from the bottom up, are:
@ -36,7 +36,7 @@ Here's what we will cover:
<<payment_channels>>:: In this chapter we will look at how payment channels work, in significantly more depth than we saw in the earlier parts of the book. We will look at the structure and Bitcoin Script of the funding and commitment transactions and the process used by nodes to negotiate each step in the protocol.
<<routing_htlcs>>:: Next, we will put together several payment channels in a network and route a payment from one end to the other. In that process we will dive into the Hash Time-Locked Contract smart contract and the Bitcoin Script that we use to construct it.
<<routing>>:: Next, we will put together several payment channels in a network and route a payment from one end to the other. In that process we will dive into the Hash Time-Locked Contract smart contract and the Bitcoin Script that we use to construct it.
<<channel_operation>>:: Putting together the concepts of a simple payment channel and a routed payment using HTLCs, we will now look at how HTLCs are part of each channel's commitment transaction. We will also look at the protocol for adding, settling, failing and removing HTLCs from the commitments.
@ -46,10 +46,10 @@ Here's what we will cover:
<<path_finding>>:: Next, we will see how the information from the gossip protocol is used by each node to build a "map" of the entire network, which it can use to find paths from one point to another to route payments. We'll also look at the exiting innovations in path finding such as multi-part payments.
<<payment_requests>>:: A key part of the Lightning Network are payment requests, also known as Lighting Invoices. In this chapter we dissect the structure and encoding of an invoice.
<<invoices>>:: A key part of the Lightning Network are payment requests, also known as Lighting Invoices. In this chapter we dissect the structure and encoding of an invoice.
<<wire_protocol>>:: Underpinning the Lightning Network is the Peer-to-Peer protocol that nodes use to exchange messages about the network and about their channels. In this chapter we look at how those messages are constructed and the extension capabilities built into messages with feature bits and TLV encoding.
<<encrypted_transport>>:: Moving down to the lower-level part of the network, we will look at the underlying encrypted transport system that ensures the secrecy and integrity of all communications between nodes.
<<encrypted_message_transport>>:: Moving down to the lower-level part of the network, we will look at the underlying encrypted transport system that ensures the secrecy and integrity of all communications between nodes.
The Lightning Network is often described as a "Layer 2 Bitcoin Protocol", which makes it sound distinct from Bitcoin. Another way to describe Lightning is as a "smarter way to use Bitcoin" or just as an "Application on top of Bitcoin". Let's explore that.
@ -21,11 +21,11 @@ The Lightning Network is simply a different and creative way of using Bitcoin. I
[[on_off_chain]]
.Lightning payment channel made of on-chain and off-chain transactions
image::images/on_off_chain.png["Lightning payment channel made of on-chain and off-chain transactions"]
image::images/mtln_0702.png["Lightning payment channel made of on-chain and off-chain transactions"]
Lightning is Bitcoin. It's just a different way of using the Bitcoin system.
=== Bitcoin ownership and control
=== Bitcoin Ownership and Control
Before we understand payment channels, we have to take a small step back and understand how ownership and control work in Bitcoin.
@ -38,17 +38,17 @@ Keep in mind that the term "ownership" as used in Bitcoin is distinct from the t
Bitcoin ownership is only about control of keys and the ability to spend the Bitcoin with those keys. As the popular Bitcoin saying by goes: "Your keys, your coins - not your keys, not your coins"
====
==== Diversity of (independent) ownership and multisig
==== Diversity of (Independent) Ownership and Multisig
Ownership and control of private keys is not always in the hands of one person. That's where things get interesting and complicated. We know that more than one person can come to know the same private key, either through theft or because the original holder of the key makes a copy and gives it to someone else. Are all these people owners? In a practical sense they are, because any one of the people who know the private key can spend the bitcoin without the approval of any other.
Bitcoin also has multisignature addresses where multiple private keys are needed to sign before spending (see <<multisig>>). From a practical perspective, ownership in a multisignature address depends on the quorum (K) and total (N) defined in the K-of-N scheme. A 1-of-10 multisignature scheme would allow any one (K) of 10 (N) signers to spend a bitcoin amount locked in that address. This is similar to the scenario where ten people have a copy of the same private key and any of them can independently spend it.
==== Joint ownership without independent control
==== Joint Ownership Without Independent Control
There is also the scenario where *no one* has quorum. In a 2-of-2 scheme like that used in the Lightning Network, neither signer can spend the bitcoin without obtaining a signature from the other party. Who owns the bitcoin in that case? No one really has ownership because no one has control. They each own the equivalent of a voting share in the decision, but both votes are needed. A key problem (pun intended) with a 2-of-2 scheme, in both Bitcoin and the law, is what happens if one of the parties is unavailable, or if there is a vote deadlock and any one party refuses to cooperate.
==== Preventing "locked" and un-spendable Bitcoin
==== Preventing "Locked" and Un-Spendable Bitcoin
If one of the two signers of a 2-of-2 multisig cannot or will not sign, the funds become un-spendable. Not only can this scenario occur accidentally (loss of keys), but it can be used as a form of blackmail by either party: "I won't sign unless you pay me a part of the funds".
@ -57,13 +57,13 @@ Payment channels in Lightning are based on a 2-of-2 multisig address, with the t
In the next section, as we look at how to open (create) a Lightning channel, we will see how we can prevent loss of funds or any blackmail scenario between the two partners by implementing a fairness protocol for the channel construction with the help of pre-signed transactions that spend the multisig output in a way that gives the peers in the channel exclusive ability to spend one of the outputs which encodes the amount of Bitcoin they own in the channel.
=== Constructing a payment channel
=== Constructing a Payment Channel
In <<what_is_payment_channel>>, we described payment channels as a _financial relationship_ between two Lightning Nodes, which is established by funding a 2-of-2 multisignature address from the two channel partners.
Let's assume that Alice wants to construct a payment channel allowing her to connect to Bob's store directly. First, the two nodes (Alice's and Bob's) have to establish an internet connection to each other, so that they can negotiate a payment channel.
==== Node private and public keys
==== Node Private and Public Keys
Every node on the Lightning Network is identified by a _node public key_. The public key uniquely identifies the specific node and is usually presented as a hexadecimal encoding. For example, Rene Pickhardt currently runs a Lightning Node (+ln.rene-pickhardt.de+) that is identified by the following node public key:
@ -73,7 +73,7 @@ Every node on the Lightning Network is identified by a _node public key_. The pu
Each node generates a root private key when first initialized. The private key is kept private at all times (never shared) and securely stored in the node's wallet. From that private key, the node derives a public key which is the node identifier and shared with the network. Since the key space is enormous, as long as each node generates the private key randomly, it will have a unique public key, which can therefore uniquely identify it on the network.
==== Node network address
==== Node Network Address
Additionally, every node also advertises a network address where it can be reached, in one of several possible formats:
@ -94,7 +94,7 @@ For example, Rene's node with node public key +02a1ceb...45ea7b8+ currently adve
The default TCP port for the Lightning Network is 9735, but a node can choose to listen on any TCP port.
====
==== Node identifiers
==== Node Identifiers
Together the node public key and network address are written in the following format, separated by an +@+ sign, as +NodeID@Address:Port+
@ -113,7 +113,7 @@ The identifier above is often encoded in a QR code, making it easier for users t
Much like Bitcoin Nodes, Lightning Nodes advertise their presence on the Lightning Network by "gossiping" their node public key and network address. That way, other nodes can find them and keep an inventory (database) of all the known nodes that they can connect to and exchange the messages that are defined in the Lightning P2P message protocol.
==== Connecting nodes as direct peers
==== Connecting Nodes As Direct Peers
In order for Alice's node to connect to Bob's node, she will need Bob's node public key, or the full address containing the public key, IP or Tor address and port. Because Bob runs a store, Bob's node address can be retrieved from an invoice or a store payment page on the web. Alice can scan a QR code that contains the address and instruct her node to connect to Bob's node.
@ -124,7 +124,7 @@ Once Alice has connected to Bob's node, their nodes are now directly connected _
To open a payment channel, two nodes must first be _connected_ as direct peers by opening a connection over the internet (or Tor).
====
=== Constructing the channel
=== Constructing the Channel
Now that Alice's and Bob's Lightning nodes are connected, they can begin the process of constructing a payment channel. In this section we will review the communications between their nodes, known as the _Lightning Peer Protocol for Channel Management_, and the cryptographic protocol that they use to build Bitcoin transactions.
@ -138,13 +138,13 @@ We describe two different protocols in this scenario. First, there is a _message
The Lightning Peer Protocol for Channel Management is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md[BOLT #2 - Peer Protocol for Channel Management]. In this chapter we will be reviewing the "Channel Establishment" and "Channel Closing" sections of BOLT#2 in more detail.
==== Channel establishment message flow
==== Channel Establishment Message Flow
Channel establishment is achieved by the exchange of six messages between Alice and Bob's nodes (three from each peer): +open_channel+, +accept_channel+, +funding_created+, +funding_signed+, +funding_locked+ and +funding_locked+. The six messages are shown as a time-sequence diagram in <<funding_message_flow>>:
In "<<funding_message_flow>>" Alice and Bob's nodes are represented by the vertical lines "A" and "B" on either side of the diagram. A time-sequence diagram like this shows time flowing downwards, and messages flowing from one side to the other between the two communication peers. The lines are sloped down to represent the elapsed time needed to transmit each message and the direction of the message is shown by an arrow at the end of each line.
@ -163,10 +163,11 @@ Once the transaction has sufficient confirmations (as defined by the `minimum_de
Alice's node requests a payment channel with Bob's node, by sending an +open_channel+ message. The message contains information about Alice's _expectations_ for the channel setup, which Bob may accept or decline.
The structure of the +open_channel+ message (taken from BOLT#2) is shown in <<open_channel_message>> below:
The structure of the +open_channel+ message (taken from BOLT#2) is shown in <<open_channel_message>>.
[[open_channel_message]]
.The open_channel message
====
----
[chain_hash:chain_hash]
[32*byte:temporary_channel_id]
@ -188,6 +189,7 @@ The structure of the +open_channel+ message (taken from BOLT#2) is shown in <<op
[byte:channel_flags]
[open_channel_tlvs:tlvs]
----
====
The fields contained in this message specify the channel parameters that Alice wants as well as various configuration settings from Alice's nodes that reflect the security expectations for the operation of the channel.
@ -214,10 +216,11 @@ If you want to understand the other fields of this and Lightning peer protocol m
===== The accept_channel message
In response to Alice's +open_channel+ message, Bob sends back the +accept_channel+ message shown in <<accept_channel_message>> below:
In response to Alice's +open_channel+ message, Bob sends back the +accept_channel+ message shown in <<accept_channel_message>>.
[[accept_channel_message]]
.The accept_channel message
====
----
[32*byte:temporary_channel_id]
[u64:dust_limit_satoshis]
@ -235,6 +238,7 @@ In response to Alice's +open_channel+ message, Bob sends back the +accept_channe
[point:first_per_commitment_point]
[accept_channel_tlvs:tlvs]
----
====
As you can see, this is similar to the +open_channel+ message and contains Bob's node expectations and configuration values.
@ -244,15 +248,15 @@ funding_pubkey:: This is the public key Bob's node contributes for the 2-of-2 mu
minimum_depth:: This is the number of confirmations that Bob's node expects for the funding transaction before it considers the channel "open" and ready to use.
==== The funding transaction
==== The Funding Transaction
Once Alice's node receives Bob's +accept_channel+ message, it has the information necessary to construct the _funding transaction_ that anchors the channel to the Bitcoin blockchain. As we discussed in earlier chapters, a lightning payment channel is anchored by a 2-of-2 multisignature address. First, we need to generate that multisignature address in order to allow us to construct the funding transaction (and the refund transaction as described below).
==== Generating a multisignature address
==== Generating a Multisignature Address
The funding transaction sends some amount of bitcoin (+funding_satoshis+ from the +open_channel+ message) to a 2-of-2 multisignature output that is constructed from Alice and Bob's +funding_pubkey+ public keys.
Alice's node constructs a multisignature script as shown in <<A_B_multisig>> below:
Alice's node constructs a multisignature script as shown here:
[[A_B_multisig]]
@ -269,13 +273,13 @@ This script is encoded as a Pay-to-Witness-Script-Hash Bitcoin address, that loo
Alice's node can now construct a funding transaction, sending the amount agreed with Bob (funding_satoshis) to the 2-of-2 multisig address. Let's assume that funding_satoshis was 140,000 and Alice is spending a 200,000 satoshi output and creating 60,000 satoshi change. The transaction will look something like this:
[[A_B_funding_Tx]]
.Alice constructs the funding transaction
image::images/A_B_funding_Tx.png["Alice constructs the funding transaction"]
image::images/mtln_0704.png["Alice constructs the funding transaction"]
Alice *does not broadcast* this transaction, because doing so would put her 140,000 satoshi at risk. Once spent to the 2-of-2 multisig, there is no way for Alice to recover her money without Bob's signature.
@ -284,11 +288,11 @@ Alice *does not broadcast* this transaction, because doing so would put her 140,
In the current implementation of Lightning, channels are funded only by the node initiating the channel (Alice in our example). Dual-funded channels have been proposed, but not yet implemented. In a dual-funded channel, both Alice and Bob would contribute inputs to the funding transaction. Dual-funded channels require a slightly more complicated message flow and cryptographic protocol, so they have not been implemented yet but are planned for a future update to the Lightning BOLTs. The C-Lightning implementation includes an experimental version of a variant on dual funded channels.
****
==== Holding signed transactions without broadcasting
==== Holding Signed Transactions Without Broadcasting
An important Bitcoin feature that makes Lightning possible is the ability to construct and sign transactions, but not broadcast them. The transaction is *valid* in every way, but until it is broadcast and confirmed on the Bitcoin blockchain it is not recognized and its outputs are not spendable as they have not been created on the blockchain. We will use this capability many times in the Lightning Network and Alice's node uses the capability when constructing the funding transaction: holding it and not broadcasting it yet.
==== Refund before funding
==== Refund Before Funding
To prevent loss of funds, Alice cannot put her bitcoin into a 2-of-2 until she has a way to get a refund if things go wrong. Essentially, she must plan the "exit" from the channel before she enters into this arrangement.
@ -296,7 +300,7 @@ Consider the legal construct of a prenuptial agreement, also known as a "prenup"
We can create a similar agreement in Bitcoin. For example, we can create a refund transaction, which functions like a prenup, allowing the parties decide how the funds in their channel will be divided before their funds are actually locked into the multisignature funding address.
==== Constructing the pre-signed refund transaction
==== Constructing the Pre-Signed Refund Transaction
Alice will construct the "refund transaction" immediately after constructing (but not broadcasting) the funding transaction. The refund transaction spends the 2-of-2 multisig back to Alice's wallet. We call this refund transaction a _commitment transaction_ as it commits both channel partners to distributing the channel balance fairly. Since Alice funded the channel on her own, she gets the entire balance and both Alice and Bob commit to refunding Alice with this transaction.
@ -304,11 +308,11 @@ In practice, it is a bit more complicated as we will see in subsequent chapters,
[[A_B_fund_refund_Tx]]
.Alice also constructs the refund transaction
image::images/A_B_fund_refund_Tx.png["Alice also constructs the refund transaction"]
image::images/mtln_0705.png["Alice also constructs the refund transaction"]
Later in this chapter we will see how more commitment transactions can be made to distribute the balance of the channel in different amounts.
==== Chaining transactions without broadcasting
==== Chaining Transactions Without Broadcasting
So now, Alice has constructed the two transactions shown in <<A_B_fund_refund_Tx>>. But you might be wondering how is this possible? Alice hasn't broadcast the funding transaction to the Bitcoin blockchain. As far as everyone on the network is concerned, that transaction doesn't exist. The refund transaction is constructed so as to *spend* one of the outputs of the funding transaction, even though that output doesn't exist yet either. How can you spend an output that hasn't been confirmed on the Bitcoin blockchain?
@ -323,7 +327,7 @@ Notice how Alice has calculated +6da3c2...387710+ as the funding transaction has
This means that Alice can create a chained transaction by referencing an output that doesn't yet exist, knowing that the reference will be valid if the funding transaction is confirmed, making the refund transaction valid too. As we will see in the next section, this "trick" of chaining transactions before they are broadcast requires a very important feature of Bitcoin that was introduced in August of 2017: _Segregated Witness_.
==== Solving malleability (Segregated Witness)
==== Solving Malleability (Segregated Witness)
Alice has to depend on the transaction ID of the funding transaction being known before confirmation. But before the introduction of Segregated Witness (a.k.a SegWit) in August 2017, this was not sufficient to protect Alice. Because of the way transactions were constructed with the signatures (witnesses) included in the transaction ID, it was possible for a third party (e.g. Bob) to broadcast an alternative version of a transaction with a malleated (modified) transaction ID. This is known as _Transaction Malleability_ and made it difficult to implement indefinite lifetime payment channels securely.
@ -338,7 +342,7 @@ You might have wondered how Bob would be able to alter (malleate) a transaction
===== The funding_created message
Now that Alice has constructed the necessary transactions, the channel construction message flow continues. Alice transmits the +funding_created+ message to Bob. In <<funding_created_message>> below you can see the contents of this message:
Now that Alice has constructed the necessary transactions, the channel construction message flow continues. Alice transmits the +funding_created+ message to Bob. You can see the contents of this message here:
[[funding_created_message]]
.The funding_created message
@ -375,7 +379,7 @@ More precisely, a `channel_id`, which is the 32 byte representation of a funding
Bob will also need to send Alice his signature for the refund transaction, based on Bob's funding_pubkey which formed the 2-of-2 multisig. While Bob already has his local refund transaction this will allow Alice to complete the refund transaction with all necessary signatures and be sure her money is refundable in case something goes wrong.
Bob constructs a +funding_signed+ message and sends it to Alice. In <<funding_signed_message>> below, we see the contents of this message:
Bob constructs a +funding_signed+ message and sends it to Alice. Here we see the contents of this message:
[[funding_signed_message]]
.The funding_signed message
@ -386,7 +390,7 @@ Bob constructs a +funding_signed+ message and sends it to Alice. In <<funding_si
----
==== Broadcasting the funding transaction
==== Broadcasting the Funding Transaction
Upon receiving the +funding_signed+ message from Bob, Alice now has both signatures needed to sign the refund transaction. Her "exit plan" is now secure and therefore she can broadcast the funding transaction without fear of having her funds locked. If anything goes wrong, Alice can simply broadcast the refund transaction and get her money back, without any further help from Bob.
@ -401,7 +405,7 @@ Of course Alice will use the Bitcoin Protocol to verify that the signature that
As soon as the funding transaction has reached the required number of confirmations, both Alice and Bob send the +funding_locked+ message to each other and the channel is ready for use.
=== Sending payments across the channel
=== Sending Payments Across the Channel
The channel has been set up, but in its initial state, all the capacity (140,000 satoshis) is on Alice's side. This means that Alice can send payments to Bob across the channel, but Bob has no funds to send to Alice yet.
@ -409,7 +413,7 @@ In the next few sections we will show how payments are made across the payment c
Let's assume that Alice wants to send 70,000 satoshis to Bob to pay her bill at Bob's coffee shop.
==== Splitting the balance
==== Splitting the Balance
In principle, sending a payment from Alice to Bob is simply a matter of redistributing the balance of the channel. Before the payment is sent, Alice has 140,000 satoshis and Bob has none. After the 70,000 satoshi payment is sent, Alice has 70,000 satoshis and Bob has 70,000 satoshis.
@ -421,13 +425,13 @@ In <<competing_commitments_1>> we see several commitment transactions:
The first commitment transaction shown in <<competing_commitments_1>> is the "refund transaction" that Alice constructed before funding the channel. In the diagram, this is "Commitment #0". After Alice pays Bob 70,000 satoshis, the new commitment transaction ("Commitment #1") has two outputs paying Alice and Bob their respective balance. We have included two subsequent commitment transactions (Commitment #2 and Commitment #3) which represent Alice paying Bob an additional 10,000 satoshis and then 20,000 satoshis respectively.
Each signed and valid commitment transaction can be used by either channel partner at any time to close the channel by broadcasting it to the Bitcoin network. Since they both have the most recent commitment transaction and can use it at any time, they can also just hold it and not broadcast it. It's their guarantee of a fair exit from the channel.
==== Competing commitments
==== Competing Commitments
You may be wondering how it is possible for Alice and Bob to have multiple commitment transactions, all of them attempting to spend the same 2-of-2 output from the funding transaction. Aren't these commitment transactions conflicting? Isn't this a "double-spend" that the Bitcoin system is meant to prevent?
@ -437,13 +441,13 @@ As long as Alice and Bob hold these transactions and don't broadcast them, the f
If more than one commitment transaction are broadcast, there are many factors that will determine which one gets confirmed first: the amount of fees included, the speed of propagation of these competing transactions, network topology, etc. Essentially it becomes a race without a predictable outcome. That doesn't sound very secure. It sounds like someone could cheat.
==== Cheating with old commitment transactions
==== Cheating with Old Commitment Transactions
Let's look more carefully at the commitment transactions in <<competing_commitments_1>>. All four commitment transactions are signed and valid. But only the last one accurately reflects the most recent channel balances. In this particular scenario, Alice has an opportunity to cheat, by broadcasting an older commitment and getting it confirmed on the Bitcoin blockchain. Let's say Alice transmits Commitment #0 and gets it confirmed: she will effectively close the channel and take all 140,000 satoshis herself. In fact, in this particular example any commitment but #3 improves Alice's position and allows her to "cancel" at least part of the payments reflected in the channel.
In the next section we will see how the Lightning Network resolves this problem - preventing older commitment transactions from being used by the channel partners by a mechanism of revocation and penalties. There are other ways to prevent the transmission of older commitment transactions, such as _eltoo channels_ but they require an upgrade to Bitcoin called _input rebinding_.
==== Revoking old commitment transactions
==== Revoking Old Commitment Transactions
Bitcoin transactions do not expire and cannot be "canceled". Neither can they be stopped or censored once they have been broadcast. So how do we "revoke" a transaction that another person holds that has already been signed?
@ -465,19 +469,19 @@ There are three elements that make up the Lightning protocol's revocation and pe
Let's look at these three elements in turn.
==== Asymmetric commitment transactions
==== Asymmetric Commitment Transactions
Alice and Bob hold slightly different commitment transactions. Let's look specifically at Commitment #2 from <<competing_commitments_1>>, in more detail:
By convention, within the Lightning protocol, we refer to the two channel partners as _self_ (also known as _local_) and _remote_, depending on which side we're looking at. The outputs that pay each channel partner are called _to_local_ and _to_remote_, respectively.
@ -485,7 +489,7 @@ In <<asymmetric_1>> we see that Alice holds a transaction that pays 60,000 satos
Bob holds the mirror-image of that transaction, where the first output is 80,000 satoshis _to_self_ (can be spent by Bob's keys), and 60,000 satoshis _to_remote_ (can be spent by Alice's keys).
==== Delayed (timelocked) spending to_self
==== Delayed (Timelocked) Spending to_self
Using asymmetric transactions allows the protocol to easily ascribe _blame_ to the cheating party. An invariant that the _broadcasting_ party must always wait ensures that the "honest" party has time to refute the claim, and revoke their funds. This asymmetry is manifested in the form of differing outputs for each side: the _to_local_ output is always timelocked and can't be spent immediately, whereas the _to_remote_ output is not timelocked and can be spent immediately.
@ -493,7 +497,7 @@ In the commitment transaction held by Alice, for example, the _to_local_ output
[[asymmetric_delayed_1]]
.Asymmetric and delayed commitment transactions
image::images/asymmetric_delayed_1.png[Asymmetric and delayed commitment transactions]
image::images/mtln_0709.png[Asymmetric and delayed commitment transactions]
That means that if Alice closes the channel by broadcasting and confirming the commitment transaction she holds, she cannot spend her balance for 432 blocks, but Bob can claim his balance immediately. If Bob closes the channel using the commitment transaction he holds, he cannot spend his output for 432 blocks while Alice can immediately spend hers.
@ -501,7 +505,7 @@ The delay is there for one reason: to allow the *remote* party to exercise a pen
The delay is negotiated by Alice and Bob, during the initial channel construction message flow, as a field called +to_self_delay+. To ensure the security of the channel, the delay is scaled to the capacity of the channel - meaning a channel with more funds has longer delays in the +to_self+ outputs in commitments. Alice's node includes a desired +to_self_delay+ in the +open_channel+ message. If Bob find this acceptable, his node includes the same value for +to_self_delay+ in the +accept_channel+ message. If they do not agree, then the channel is rejected (see +shutdown+ message).
==== Revocation keys
==== Revocation Keys
As we discussed previously, the word "revocation" is a bit misleading because it implies that the "revoked" transaction cannot be used.
@ -513,10 +517,10 @@ So, in our example, each side holds a commitment transaction that includes a rev
[[asymmetric_delayed_revocable_1]]
.Asymmetric, delayed and revocable commitments
image::images/asymmetric_delayed_revocable_1.png[Asymmetric, delayed and revocable commitments]
image::images/mtln_0710.png["Asymmetric, delayed and revocable commitments"]
[[commitment_transaction]]
=== The commitment transaction
=== The Commitment Transaction
Now that we understand the structure of commitment transactions and why we need asymmetric, delayed, revocable commitments, let's look at the Bitcoin Script that implements this.
@ -565,7 +569,7 @@ The second output (to_remote) output of the commitment transaction, is defined i
Now that we've defined the commitment transactions in detail, let's see how Alice and Bob advance the state of the channel, create and sign new commitment transactions and revoke old commitment transactions.
=== Advancing the channel state
=== Advancing the Channel State
To advance the state of the channel, Alice and Bob exchange two messages: +commitment_signed+ and +revoke_and_ack+. The +commitment_signed+ message can be sent by either channel partner when they have an update to the channel state. The other channel partner then may respond with +revoke_and_ack+ to _revoke_ the old commitment and _acknowledge_ the new commitment.
@ -573,11 +577,11 @@ In <<commitment_message_flow>> we see the Alice and Bob exchanging two pairs of
[[commitment_message_flow]]
.Commitment and revocation message flow
image::images/commitment_message_flow.png[Commitment and revocation message flow]
image::images/mtln_0711.png[Commitment and revocation message flow]
==== The commitment_signed message
==== The Commitment_Signed Message
The structure of the +commitment_signed+ message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#committing-updates-so-far-commitment_signed[BOLT #2 - Peer Protocol - commitment_signed] and shown in <<commitment_signed_message>>, below:
The structure of the +commitment_signed+ message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#committing-updates-so-far-commitment_signed[BOLT #2 - Peer Protocol - commitment_signed] and shown here:
[[commitment_signed_message]]
.The commitment_signed message
@ -600,11 +604,11 @@ The use of HTLCs to commit updates will be explained in detail in <<htlcs>> and
Alice's +commitment_signed+ message gives Bob the signature needed (Alice's part of the 2-of-2) for a new commitment transaction.
==== The revoke_and_ack message
==== The revoke_and_ack Message
Now that Bob has a new commitment transaction, he can revoke the previous commitment by giving Alice a revocation key, and construct the new commitment with Alice's signature.
The +revoke_and_ack+ message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#completing-the-transition-to-the-updated-state-revoke_and_ack[BOLT #2 - Peer Protocol - revoke_and_ack] and show in <<revoke_and_ack_message>> below:
The +revoke_and_ack+ message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#completing-the-transition-to-the-updated-state-revoke_and_ack[BOLT #2 - Peer Protocol - revoke_and_ack] and shown here:
[[revoke_and_ack_message]]
.The revoke_and_ack message
@ -620,7 +624,8 @@ The +revoke_and_ack+ message is defined in https://github.com/lightningnetwork/l
+per_commitment_secret+:: Used to generate a revocation key for the previous (old) commitment, effectively revoking it.
+next_per_commitment_point+:: Used to build a revocation_pubkey for the new commitment, so that it can later be revoked.
==== Revoking and re-committing
[[revocation]]
==== Revoking and Re-Committing
Let's look at this interaction between Alice and Bob more closely.
@ -632,11 +637,11 @@ As soon as Bob has given this secret to Alice, he *must not* ever broadcast that
Once Alice has received the +revoke_and_ack+ from Bob she can be sure that Bob cannot broadcast the old commitment without being penalized. She now has the keys necessary to create a penalty transaction if Bob broadcasts an old commitment.
==== Cheating and penalty in practice
[[revocation_secret_derivation]]
==== Cheating and Penalty in Practice
In practice, both Alice and Bob have to monitor for "cheating". They are monitoring the Bitcoin blockchain for any commitment transactions related to any of the channels they are operating. If they see a commitment transaction confirmed on-chain they will check to see if it is the most recent commitment. If it is an "old" commitment, they must immediately construct and broadcast a penalty transaction. The penalty transaction spends *both* the +to_local+ and +to_remote+ outputs, closing the channel and sending both balances to the "cheated" channel partner.
[[revocation_secret_derivation]]
In order to more easily allow both sides to keep track of the commitment numbers of the passed revoke commitments, each commitment actually _encodes_ the number of the commitment within the lock time and sequence fields in a transition. Within the protocol, this special encoding is referred to as "state hints". Assuming a party knows the current commitment number, they're able to use the state hints to easily recognize if a broadcasted commitment was a revoked one, and if so, which commitment number was breached, as that number is used to easily look up which revocation secret should be used in the revocation secret tree (shachain).
Rather than encode the state hint in plain sight, an _obfuscated_ state hint is used in its place. This obfuscation is achieved by first XOR'ing the current commitment number with a set of random bytes generated deterministically using the funding public keys of both sides of the channel. A total of 6 bytes across the lock time and sequence (24 bits of the locktime and 24 bits of the sequence) are used to encode the state hint within the commitment transaction, so 6 random bytes are needed to use for XOR'ing. To obtain these 6 bytes, both sides obtain the SHA-256 hash of the initiator's funding key concatenated to the responder's funding key. Before encoding the current commitment height, the integer is XOR'd with this state hint obfuscater, and then encoded in the lower 24 bits of the locktime, and the upper 64 bits of the sequence.
@ -649,7 +654,7 @@ Let's review our channel between Alice and Bob and show a specific example of a
[[competing_commitments_2]]
.Revoked and current commitments
image::images/competing_commitments_2.png[Revoked and current commitments]
image::images/mtln_0712.png[Revoked and current commitments]
With each commitment, Alice has revoked the previous (older) commitment. The current state of the channel and the correct balance is represented by Commitment #3. All previous commitments have been revoked and Bob has the keys necessary to issue penalty transactions against them, in case Alice tries to broadcast one of them.
@ -659,7 +664,7 @@ Alice decides to take a huge risk and broadcast the revoked Commitment #1, to st
As you can see, Alice's old commitment has two outputs, one paying herself 70,000 satoshis (to_local output) and one paying Bob 70,000 satoshis. Alice can't yet spend her 70,000 to_local output because it has a 432 block (3 day) timelock. She is now hoping that Bob doesn't notice for three days.
@ -669,15 +674,15 @@ Bob's node will immediately broadcast a penalty transaction. Since this old comm
[[penalty_transaction]]
.Cheating and penalty
image::images/penalty_transaction.png[Cheating and penalty]
image::images/mtln_0714.png[Cheating and penalty]
Bob's penalty transaction pays 140,000 satoshis to his own wallet, taking the entire channel capacity. Alice has not only failed to cheat, she has lost everything in the attempt!
==== The channel reserve: ensuring skin in the game
==== The Channel Reserve: Ensuring Skin in the Game
You may have noticed there is a special situation that needs to be dealt with: if Alice could keep spending her balance until it is zero, she would be in a position to close the channel by broadcasting an old commitment transaction without risking a penalty: either the revoked commitment transaction succeeds after the delay, or the cheater gets caught but there's no consequence because the penalty is zero. From a game theory perspective, it is free money to attempt to cheat in this situation. This is why the channel reserve is in play, so a prospective cheater always faces the risk of a penalty.
=== Closing the channel (cooperative close)
=== Closing the Channel (Cooperative Close)
So far we've looked at the commitment transactions as one possible way to close a channel, unilaterally. This type of channel closure is not ideal, since it forces a timelock on the channel partner that uses it.
@ -687,11 +692,11 @@ The closing message flow is defined in https://github.com/lightningnetwork/light
[[closing_message_flow]]
.The channel close message flow
image::images/closing_message_flow.png[The channel close message flow]
image::images/mtln_0715.png[The channel close message flow]
==== The shutdown message
==== The Shutdown Message
Channel closing starts with one of the two channel partners sending the +shutdown+ message. The contents of this message are shown in <<shutdown_message>> below:
Channel closing starts with one of the two channel partners sending the +shutdown+ message. The contents of this message are shown here:
[[shutdown_message]]
.The shutdown message
@ -713,11 +718,11 @@ Bob will respond with his own +shutdown+ message indicating that he agrees to co
Now both Alice and Bob have each other's preferred wallet address, they can construct identical closing transactions to settle the channel balance.
==== The closing_signed message
==== The closing_signed Message
Assuming the channel has no outstanding commitments or updates and the channel partners have exchanged the +shutdown+ messages shown in the previous section, they can now finish this cooperative close.
The *funder* of the channel (Alice in our example) starts by sending a +closing_signed+ message to Bob. This message proposes a transaction fee for the on-chain transaction, and Alice's signature (the 2-of-2 multisig) for the closing transaction. The +closing_signed+ message is shown in <<closing_signed_message>> below:
The *funder* of the channel (Alice in our example) starts by sending a +closing_signed+ message to Bob. This message proposes a transaction fee for the on-chain transaction, and Alice's signature (the 2-of-2 multisig) for the closing transaction. The +closing_signed+ message is shown here:
[[closing_signed_message]]
.The closing_signed message
@ -737,7 +742,7 @@ This negotiation may continue with back-and-forth +closing_signed+ messages unti
Once Alice receives a +closing_signed+ message with the same fee as the one she proposed in her last message, the negotiation is complete. Alice signs and broadcasts the closing transaction and the channel is closed.
==== The cooperative close transaction
==== The Cooperative Close Transaction
The cooperative close transaction looks similar to the last commitment transaction that Alice and Bob had agreed on. However, unlike the last commitment transaction, it does not have timelocks or penalty revocation keys in the outputs. Since both parties cooperate to produce this transaction and they won't be making any further commitments, there is no need for the asymmetric, delayed and revocable elements in this transaction.
@ -747,7 +752,7 @@ Alice broadcasts a transaction shown in <<closing_transaction>> below to close t
[[closing_transaction]]
.The cooperative close transaction
image::images/closing_transaction.png[The cooperative close transaction]
image::images/mtln_0716.png[The cooperative close transaction]
As soon as this closing transaction is confirmed on the Bitcoin blockchain, the channel is closed. Now, Alice and Bob can spend their outputs as they please.
In this chapter we will finally unpack how payment channels can be connected to form a _network of payment channels_ via a process called _routing_. Specifically, we will look at the first part of the routing layer, the Atomic & Trustless Multihop Payment protocol. This is highlighted by a double outline in the protocol suite, shown in <<LN_protocol_routing_highlight>>:
In this section we will examine routing from the perspective of Dina, a gamer who receives donations from her fans while she streams her game sessions.
@ -20,7 +18,7 @@ In <<dina_routing_diagram>> we see a possible network layout created by various
[[dina_routing_diagram]]
.Fans connected indirectly to Dina on the Lightning Network
image::images/dina-routing-diagram.png["Fans connected indirectly to Dina on the Lightning Network"]
image::images/mtln_0802.png["Fans connected indirectly to Dina on the Lightning Network"]
The nodes along the path from the fan to Dina are intermediaries and called "routing nodes" in the context of routing a payment. There is no functional difference between the "routing nodes" and the nodes operated by Dina's fans. Any Lightning node is capable of routing payments across its payment channels.
@ -46,7 +44,7 @@ Routing refers to the series of interactions across the network that attempt to
An important rule of thumb is that it's possible for a _path_ to exist between Alice and Bob (perhaps even more than one), yet there may not be an active _route_ on which to send the payment. One example is the scenario where all the nodes connecting Alice and Bob are currently off-line. In this example, one can examine the _channel graph_ and connect a series of payment channels from Alice to Bob, hence a _path_ exists. However, as the intermediary nodes are offline, the payment cannot be sent and so no _route_ exists.
=== Creating a network of payment channels
=== Creating a Network of Payment Channels
Before we dive into the concept of an atomic trustless multihop payment, let's work through an example.
Let's return to Alice who, in previous chapters, purchased a coffee from Bob with whom she has an open channel.
@ -58,7 +56,7 @@ As you can see in <<routing_network>>, Alice has an open channel with Bob, the c
[[routing_network]]
.A network of payment channels between Alice and Dina
image::images/routing-network.png["A network of payment channels between Alice and Dina"]
image::images/mtln_0803.png["A network of payment channels between Alice and Dina"]
It's possible to trace a _path_ from Alice to Dina that uses Bob and Chan as intermediary routing nodes.
Alice can then craft a _route_ from this outlined path, and use it to send a tip of a few thousand satoshis to Dina, with the payment being _forwarded_ by Bob and Chan.
@ -66,7 +64,7 @@ Essentially, Alice will pay Bob, who will pay Chan, who will pay Dina. No direct
The main challenge is to do this in a way that prevents Bob and Chan from stealing the money that Alice wants delivered to Dina.
==== A physical example of "routing"
==== A Physical Example of "Routing"
To understand how the Lightning Network protects the payment while being routed, we can compare it to an example of routing physical payments with gold coins in the real world.
@ -74,7 +72,7 @@ Assume Alice wants to give 10 gold coins to Dina, but does not have direct acces
[[alice_dina_routing_1]]
.Alice wants to pay Dina 10 gold coins
image::images/gold-coins-network1.png[]
image::images/mtln_0804.png[]
Alice can pay Bob to pay Chan to pay Dina, but how does she make sure that Bob or Chan don't run off with the coins after receiving them?
In the physical world contracts could be used for safely carrying out a series of payments.
@ -118,7 +116,7 @@ The difference of 1 gold coin is the fee that Bob will earn for helping out with
[[alice_dina_routing_2]]
.Alice pays Bob, Bob pays Chan, Chan pays Dina
image::images/gold-coins-network2.png[]
image::images/mtln_0805.png[]
As there is still the issue of trust and the risk that either Alice or Bob won't honor the contract, all parties decide to use an escrow service.
At the start of the exchange, Alice could "lock up" these 12 gold coins in escrow that will only be paid to Bob once he proves that he's paid 11 gold coins to Chan.
@ -145,7 +143,7 @@ To facilitate Alice's payment, Dina will create the payment secret and the payme
[[alice_dina_routing_3]]
.Dina sends the hashed secret to Alice
image::images/gold-coins-network3.png["Dina sends the hashed secret to Alice"]
image::images/mtln_0806.png["Dina sends the hashed secret to Alice"]
Alice doesn't know the secret but she can rewrite her contract to use the hash of the secret as a proof of payment:
@ -241,7 +239,7 @@ If she does not provide the secret by this time, Chan's deposit will be refunded
With such a chain of contracts we can ensure that, after 24 hours, the payment will successfully go from Alice to Bob to Chan to Dina, or it will fail and everyone will be refunded.
Either the contract fails or succeeds, there's no middle ground.
In the context of the Lightning Network, we call this "all or nothing" property ((("atomicity")))_atomicity_.
In the context of the Lightning Network, we call this "all or nothing" property _atomicity_.
As long as the escrow is trustworthy and faithfully performs its duty, then no party will have their coins stolen in the process.
@ -279,7 +277,7 @@ The fairness protocol for routing used in the Lightning Network today is called
Another proposed mechanism for implementing routing is a _Point Time-Locked Contract (PTLC)_. PTLCs also achieve atomicity, trustless operation and multihop security, but do so with increased efficiency and better privacy. Efficient implementation of PTLCs depends on a new digital signature algorithm called _Schnorr signatures_, which is expected to be activated in Bitcoin in 2021.
=== Revisiting our example
=== Revisiting Our Example
Let's revisit our example from the first part of this chapter. Alice wants to "tip" Dina, with a Lightning payment. Let's say Alice wants to send Dina 50,000 satoshis as a tip.
@ -294,7 +292,7 @@ Alice visits Dina's site, enters the amount of 50,000 satoshis in a form and in
[[alice_dina_invoice_1]]
.Alice requests an invoice from Dina's website
image::images/alice-dina-invoice-1.png["Alice requests an invoice from Dina's website"]
image::images/mtln_0807.png["Alice requests an invoice from Dina's website"]
As we saw in previous examples, we assume that Alice does not have a direct payment channel to Dina. Instead, Alice has a channel to Bob, Bob has a channel to Chan and Chan has a channel to Dina. To pay Dina, Alice must find a path that connects her to Dina. We will discuss that step in more detail in <<path_finding>>. For now, let's assume that Alice is able to gather information about available channels and sees that there is a path from her to Dina, via Bob and Chan.
@ -306,7 +304,7 @@ Remember how Bob and Chan might expect a small compensation for routing the paym
Now, Alice's node can construct a Lightning payment. In the next few sections we will see how Alice's node constructs a Hash Time-Locked Contract (HTLCs) to pay Dina and how that HTLC is forwarded along the path from Alice to Dina.
==== On-chain vs Off-chain settlement of HTLCs
==== On-Chain vs. Off-Chain Settlement of HTLCs
The purpose of the Lightning Network is to enable _off-chain_ transactions that are trusted just the same as on-chain transactions, because no one can cheat. The reason no one can cheat is because at any time, any of the participants can take their off-chain transactions on-chain. Each off-chain transaction is ready to be submitted to the Bitcoin blockchain at any time. Thus, the Bitcoin blockchain acts as a dispute-resolution and final settlement mechanism if necessary.
@ -325,7 +323,7 @@ In <<alice_dina_invoice_2>> we see Alice getting a Lightning invoice from Dina.
[[alice_dina_invoice_2]]
.Alice gets a payment hash from Dina
image::images/alice-dina-invoice-2.png["Alice gets a payment hash from Dina"]
image::images/mtln_0808.png["Alice gets a payment hash from Dina"]
In the Lightning Network, Dina's payment pre-image won't be a phrase like "Dina's secret", but a random number generated by Dina's node. Let's call that random number +R+.
@ -337,9 +335,9 @@ Dina's node will calculate a cryptographic hash of +R+, such that:
H = SHA256(R)
----
In <<payment_hash_and_preimage>> +H+ is the hash, or _payment hash_ and +R+ is the secret or _payment pre-image_.
In this code, +H+ is the hash, or _payment hash_ and +R+ is the secret or _payment pre-image_.
The use of a cryptographic hash function is one element that guarantees ((("trustless")))_trustless operation_. The payment intermediaries do not need to trust each other because they know that no one can guess the secret or fake it.
The use of a cryptographic hash function is one element that guarantees _trustless operation_. The payment intermediaries do not need to trust each other because they know that no one can guess the secret or fake it.
==== HTLCs in Bitcoin Script
@ -354,7 +352,8 @@ Let's see how we would implement this as an HTLC in Bitcoin Script. In <<receive
Wow that looks complicated! Don't worry though, we will take it one step at a time and simplify it.
@ -382,7 +382,7 @@ In the following sections, we will focus on the main elements of the script and
The main part of the HTLC is in line 10 of <<received_htlc>>. Let's build it up from scratch!
==== Payment pre-image and hash verification
==== Payment Pre-Image and Hash Verification
The core of an HTLC is the "hash", where payment can be made if the recipient knows the payment pre-image. Alice locks the payment to a specific payment hash and Bob has to present a payment pre-image to claim the funds. The Bitcoin system can verify that Bob's payment pre-image is correct by hashing it and comparing the result to the payment hash that Alice used to lock the funds.
@ -422,7 +422,7 @@ Alice will now extend the HTLC across the network so that it reaches Dina.
[[alice_dina_htlc_1]]
.Propagating the HTLC across the network
image::images/alice-dina-htlc-1.png["Propagating the HTLC across the network"]
image::images/mtln_0809.png["Propagating the HTLC across the network"]
In <<alice_dina_htlc_1>> we see the HTLC propagated across the network from Alice to Dina. Alice has given Bob an HTLC for 50,200 satoshi. Bob can now create an HTLC for 50,100 satoshi and give it to Chan.
@ -434,13 +434,13 @@ Bob isn't taking a risk and isn't trusting Alice or Chan. Instead, Bob is trusti
Similarly, Chan can extend a 50,000 HTLC to Dina. He isn't risking anything or trusting Bob or Dina. To redeem the HTLC, Dina would have to broadcast the secret, which Chan could use to redeem Bob's HTLC. Chan would also earn 100 satoshis as a routing fee.
==== Back-propagating the secret
==== Back-Propagating the Secret
Once Dina receives a 50,000 HTLC from Chan, she can now get paid. Dina could simply commit this HTLC on-chain and spend it by revealing the secret in the spending transaction. Or, instead, Dina can update the channel balance with Chan by giving him the secret. There's no reason to incur a transaction fee and go on-chain. So, instead, Dina sends the secret to Chan and they agree to update their channel balances to reflect a 50,000 satoshi Lightning payment to Dina. In <<alice_dina_htlc_redeem_1>> we see Dina giving the secret to Chan, thereby fulfilling the HTLC.
Notice Dina's channel balance goes from 50,000 satoshi to 100,000 satoshi. Chan's channel balance is reduced from 200,000 satoshi to 150,000 satoshi. The channel capacity hasn't changed, but 50,000 has moved from Chan's side of the channel to Dina's side of the channel.
@ -448,7 +448,7 @@ Chan now has the secret and has paid Dina 50,000 satoshi. He can do this without
Chan has paid Dina 50,000 satoshi, and received 50,100 satoshi from Bob. So Chan has 100 satoshi more in his channel balances, which he earned as a routing fee.
@ -456,7 +456,7 @@ Bob now has the secret too. He can use it to spend Alice's HTLC on-chain. Or, he
So far we looked at the "Hash" part of HTLC, and how it would work if everyone cooperated and was online at the time of payment.
@ -603,7 +603,7 @@ The +OP_DROP+ operator simply drops the topmost item on the script stack. This i
Finally, once the stack has been cleaned up, there should be a public key and signature left behind, that +OP_CHECKSIG+ can verify. As we saw in <<preventing_theft>>, this is necessary to ensure that only the rightful owner of the funds can claim them, by binding this output to their public key and requiring a signature.
==== Decrementing time-locks
==== Decrementing Time-Locks
As the HTLCs are extended from Alice to Dina, the time-lock refund clause in each HTLC has a _different_ cltv_expiry value. We will see this in more detail as we talk about <<onion_routing>>. But suffice it to say that to ensure an orderly unwinding of a payment that fails, each hop needs to wait a bit less for their refund. The difference between time-locks for each hop is called the cltv_expiry_delta and is set by each node and advertized to the network as we will see in <<gossip>>.
=== Local (single channel) vs Routed (multiple channels)
=== Local (Single Channel) vs. Routed (Multiple Channels)
Even though it is possible to send payments across a payment channel simply by updating the channel balances and creating new commitment transactions, the Lightning protocol uses HTLCs even for "local" payments across a payment channel. The reason for this is to maintain the same protocol design regardless of whether a payment is only one hop (across a single payment channel), or several hops (routed across multiple payment channels).
By maintaining the same abstraction for both local and remote, we not only simplify the protocol design but we also improve privacy. For the recipient of a payment there is no discernible difference between a payment made directly by their channel partner and a payment forwarded by their channel partner on behalf of someone else.
=== Forwarding payments and updating commitments with HTLCs
=== Forwarding Payments and Updating Commitments with HTLCs
We will revisit our example from <<routing>>, to demonstrate how the HTLCs from Alice to Dina gets committed to each payment channel. As you recall in our example, Alice is paying Dina 50,000 satoshis, by routing an HTLC via Bob and Chan. The network is shown in <<alice_dina_htlc_2>> below:
[[alice_dina_htlc_2]]
.Alice pays Dina with an HTLC routed via Bob and Chan
image::images/alice-dina-htlc-1.png["Alice pays Dina with an HTLC routed via Bob and Chan"]
image::images/mtln_0809.png["Alice pays Dina with an HTLC routed via Bob and Chan"]
We will focus on the payment channel between Alice and Bob and review the messages and transactions that they use to process this HTLC.
==== HTLC and commitment message flow
==== HTLC and Commitment Message Flow
The message flow between Alice and Bob (and also between any pair of channel partners) is shown in <<HTLC_commitment_message_flow>>:
[[HTLC_commitment_message_flow]]
.The message flow for HTLC commitment between channel partners
image::images/HTLC_commitment_message_flow_1.png["The message flow for HTLC commitment between channel partners"]
image::images/mtln_0903.png["The message flow for HTLC commitment between channel partners"]
We've already seen the +commitment_signed+ and +revoke_and_ack+ in <<payment_channels>>. Now we will see how HTLCs fit into the commitment scheme. The two new messages are +update_add_htlc+ which Alice uses to ask Bob to add an HTLC and +update_fulfill_htlc+ which Bob uses to redeem the HTLC once he has received the payment secret (Dina's secret).
=== Forwarding payments with HTLCs
=== Forwarding Payments with HTLCs
Alice and Bob start with a payment channel that has 70,000 satoshi balance on each side.
@ -44,7 +44,7 @@ As we saw in <<payment_channels>> this means that Alice and Bob have negotiated
[[alice_bob_commitment_txs_1]]
.Alice and Bob's initial commitment transactions
image::images/alice_bob_commitment_txs_1.png["Alice and Bob's initial commitment transactions"]
image::images/mtln_0904.png["Alice and Bob's initial commitment transactions"]
==== Adding an HTLC
@ -53,7 +53,7 @@ Alice wants Bob to accept an HTLC worth 50,200 satoshis to forward to Dina. To d
To add the HTLC, Alice starts the flow we saw in <<HTLC_commitment_message_flow>> by sending the +update_add_htlc+ message to Bob.
[[update_add_htlc]]
==== The +update_add_htlc+ message
==== The update_add_HTLC Message
Alice sends the `update_add_HTLC` Lightning message to Bob. This message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#adding-an-htlc-update_add_htlc[BOLT #2 - Peer Protocol - update_add_HTLC], and is shown below:
@ -87,16 +87,17 @@ Alice sends the `update_add_HTLC` Lightning message to Bob. This message is defi
As a reminder, accounting within the Lightning Network is in units of milli-satoshis (thousandths of a satoshi), whereas Bitcoin accounting is in satoshis. Any amounts in HTLCs are milli-satoshis, which are then rounded to the nearest satoshi in the Bitcoin commitment transactions.
====
==== HTLC in commitment transaction
==== HTLC in Commitment Transaction
The received information is enough for Bob to create a new commitment transaction. The new commitment transaction has the same two outputs +to_self+ and +to_remote+ for Alice and Bob's balance, and a *new* output representing the HTLC offered by Alice.
We've already seen the basic structure of an HTLC in <<routing>>. The complete script of an offered HTLC is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/03-transactions.md#offered-htlc-outputs[BOLT #3 - Transactions - Offered HTLC output] and is shown in <<offered_htlc_output_script>> below:
There are three ways to claim this output. Try to read the script and see if you can figure it out (remember, it is a stack-based language so things appear "backwards").
==== New commitment with HTLC output
==== New Commitment with HTLC Output
Bob now has the necessary information to add this HTLC script as an additional output and create a new commitment transaction. Bob's new commitment will have 50,200 satoshis in the HTLC output. That amount will come from Alice's channel balance, so Alice's new balance will be 19,800 satoshis (70,000 - 50,200 = 19,800). Bob constructs this commitment as a tentative "Commitment #3", shown in <<add_commitment_3b>>, below:
[[add_commitment_3b]]
.Bob's new commitment with an HTLC output
image::images/add_commitment_3b.png["Bob's new commitment with an HTLC output"]
image::images/mtln_0905.png["Bob's new commitment with an HTLC output"]
==== Alice commits
==== Alice Commits
Shortly after sending the +update_add_htlc+ message, she will commit to the new state of the channel, so that the HTLC can be safely added by Bob. Bob has the HTLC information and has constructed a new commitment but does not yet have this new commitment signed by Alice.
Alice sends +commitment_signed+ to Bob, with the signature for the new commitment and for the HTLC within. We saw the +commitment_signed+ message in <<payment_channels>>, but now we can understand the rest of the fields. As a reminder, it is shown in <<commitment_signed_message>> below:
Alice sends +commitment_signed+ to Bob, with the signature for the new commitment and for the HTLC within. We saw the +commitment_signed+ message in <<payment_channels>>, but now we can understand the rest of the fields. As a reminder, it is shown in this code:
[[ops_commitment_signed_message]]
.The commitment_signed message
@ -159,9 +160,9 @@ Now, Bob has a new signed commitment transaction, as shown in <<signed_commitmen
[[signed_commitment_3b]]
.Bob has a new signed commitment
image::images/signed_commitment_3b.png[Bob has a new signed commitment]
image::images/mtln_0906.png[Bob has a new signed commitment]
==== Bob acknowledges new commitment and revokes old one
==== Bob Acknowledges New Commitment and Revokes Old One
Now that Bob has a new signed commitment, he needs to acknowledge it and revoke the old commitment. He does so by sending the +revoke_and_ack+ message, as we saw in <<payment_channels>> previously. As a reminder, that message is shown in <<revoke_and_ack_message_2>>, below:
@ -181,7 +182,7 @@ Bob has effectively moved the channel state forward, as shown in <<revoked_commi
[[revoked_commitment_2b]]
.Bob has revoked the old commitment
image::images/revoked_commitment_2b.png[Bob has revoked the old commitment]
image::images/mtln_0907.png[Bob has revoked the old commitment]
Despite the fact that Bob has a new (signed) commitment transaction and an HTLC output inside he cannot consider his HTLC as being set up successfully.
@ -193,7 +194,7 @@ Alice has constructed a mirror-image new commitment transaction containing the n
[[add_commitment_3a]]
.Alice's new commitment with an HTLC output
image::images/add_commitment_3a.png["Alice's new commitment with an HTLC output"]
image::images/mtln_0908.png["Alice's new commitment with an HTLC output"]
As we described in <<payment_channels>>, Alice's commitment is the mirror-image of Bob's, as it contains the asymmetric, delayed, revocable construct for revocation and penalty enforcement of old commitments. Alice's 19,800 satoshi balance (after deducting the HTLC value), is delayed and revocable. Bob's 70,000 satoshi balance is immediately redeemable.
@ -201,7 +202,7 @@ Next, the message flow for +commitment_signed+ and +revoke_and_ack+ is now repea
For completeness sake, let's quickly review the commitment transactions as this round of commitment/revocation happens.
==== Bob commits
==== Bob Commits
Bob now sends a +commitment_signed+ back to Alice, with his signatures for Alice's new commitment transaction, including the HTLC output she has added.
@ -209,7 +210,7 @@ Now Alice has the signature for the new commitment transaction. The state of the
[[signed_commitment_3a]]
.Alice has a new *signed* commitment
image::images/signed_commitment_3a.png[Alice has a new *signed* commitment]
image::images/mtln_0909.png[Alice has a new *signed* commitment]
Alice can now acknowledge the new commitment by revoking the old one. Alice sends the +revoke_and_ack+ message containing the necessary +per_commitment_point+ that will allow Bob to construct a revocation key and penalty transaction. Thus, Alice revokes her old commitment.
@ -217,7 +218,7 @@ The channel state is shown in <<revoked_commitment_2a>> below:
[[revoked_commitment_2a]]
.Alice has revoked the old commitment
image::images/revoked_commitment_2a.png[Alice has revoked the old commitment]
image::images/mtln_0910.png[Alice has revoked the old commitment]
=== Multiple HTLCs
@ -229,7 +230,7 @@ The current maximum number of HTLCs allowed on a channel is 483 HTLCs to account
As we will see in the next section, the maximum is only for _pending_ HTLCs, since once an HTLC is fulfilled (or fails due to timeout/error), it is removed from the commitment transaction.
=== HTLC fulfillment
=== HTLC Fulfillment
Now Bob and Alice both have a new commitment transaction with an additional HTLC output and we have achieved a major step towards updating a payment channel.
@ -237,15 +238,15 @@ The new balance of Alice and Bob does not reflect yet that Alice has successfull
However the HTLCs are now set up in a way that secure settlement in exchange for the proof of payment will be possible.
==== HTLC propagation
==== HTLC Propagation
Let's assume that Bob continues the chain and sets up an HTLC with Chan for 50,100 satoshis. The process will be exactly the same as we just saw between Alice and Bob. Bob will send +update_add_htlc+ to Chan, then they will exchange +commitment_signed+ and +revoke_and_ack+ messages in two rounds, progressing their channel to the next state.
Next, Chan will do the same with Dina: offer a 50,000 satoshi HTLC, commit and revoke etc. However, Dina is the final recipient of the HTLC. Dina is the only one that knows the payment secret (the pre-image of the payment hash). Therefore, Dina can fulfill the HTLC with Chan immediately!
==== Dina fulfills the HTLC with Chan
==== Dina Fulfills the HTLC with Chan
Dina can settle the HTLC by sending an +update_fulfill_htlc+ message to Chan. The +update_fulfill_htlc+ message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#removing-an-htlc-update_fulfill_htlc-update_fail_htlc-and-update_fail_malformed_htlc[BOLT #2 - Peer Protocol - update_fulfill_htlc] and is shown in <<update_fulfill_htlc_message>> below:
Dina can settle the HTLC by sending an +update_fulfill_htlc+ message to Chan. The +update_fulfill_htlc+ message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#removing-an-htlc-update_fulfill_htlc-update_fail_htlc-and-update_fail_malformed_htlc[BOLT #2 - Peer Protocol - update_fulfill_htlc] and is shown here:
[[update_fulfill_htlc_message]]
.The +update_fulfill_htlc+ message
@ -273,7 +274,7 @@ If the result +H+ matches the payment hash in the HTLC, Chan can do a little dan
Let's go back to Alice and Bob's channel and watch them unwind the HTLC. To get there, let's assume Dina sent the +update_fulfill_htlc+ to Chan, Chan sent +update_fulfill_htlc+ to Bob and Bob sent +update_fulfill_htlc+ to Alice. The payment pre-image has propagated all the way back to Alice.
==== Bob settles the HTLC with Alice
==== Bob Settles the HTLC with Alice
When Bob sends +update_fulfill_htlc+ to Alice, it will contain the same +payment_preimage+ that Dina selected for her invoice. That +payment_preimage+ has traveled backwards along the payment path. At each step, the +channel_id+ will be different and +id+ (HTLC ID) may be different. But the preimage is the same!
@ -283,7 +284,7 @@ The message flow between Alice and Bob is shown in <<htlc_fulfillment_message_fl
Both Alice and Bob can now remove the HTLC from the commitment transactions and update their channel balances.
@ -291,21 +292,21 @@ They create new commitments (Commitment #4), as shown in <<htlc_fulfillment_comm
[[htlc_fulfillment_commitments_added]]
.The HTLC is removed and balances updated in new commitments
image::images/htlc_fulfillment_commitments_added.png[The HTLC is removed and balances updated in new commitments]
image::images/mtln_0912.png[The HTLC is removed and balances updated in new commitments]
Next, they complete two rounds of commitment and revocation. First, Alice sends +commitment_signed+ to sign Bob's new commitment transaction. Bob responds with +revoke_and_ack+ to revoke his old commitment. Once Bob has moved the state of the channel forward, the commitments look like we see in <<htlc_fulfillment_commitments_bob_commit>> below:
[[htlc_fulfillment_commitments_bob_commit]]
.Alice signs Bob's new commitment and Bob revoked the old one
image::images/htlc_fulfillment_commitments_bob_commit.png[Alice signs Bob's new commitment and Bob revoked the old one]
image::images/mtln_0913.png[Alice signs Bob's new commitment and Bob revoked the old one]
Finally, Bob signs Alice's commitment by sending Alice a +commitment_signed+ message. Then Alice acknowledges and revokes her old commitment by sending +revoke_and_ack+ to Bob. The end result is that both Alice and Bob have moved their channel state to Commitment #4, have removed the HTLC and have updated their balances. Their current channel state is represented by the commitment transactions that are shown in <<alice_bob_htlc_fulfilled>> below:
[[alice_bob_htlc_fulfilled]]
.Alice and Bob settle the HTLC and update balances
image::images/alice_bob_htlc_fulfilled.png[Alice and Bob settle the HTLC and update balances]
image::images/mtln_0914.png[Alice and Bob settle the HTLC and update balances]
=== Removing an HTLC due to error or expiry
=== Removing an HTLC Due to Error or Expiry
If an HTLC cannot be fulfilled, it can be removed from the channel commitment using the same process of commitment and revocation.
@ -326,7 +327,7 @@ It's pretty self-explanatory. The multi-byte +reason+ field is defined in https:
If Alice received a +update_fail_htlc+ from Bob, the process would unfold in much the same way: The two channel partners would remove the HTLC, create updated commitment transactions and go through two rounds of commitment/revocation to move the channel state forward to the next commitment. The only difference: the end balances would revert back to what they were without the HTLC, essentially refunding Alice for the HTLC value.
=== Making a local payment
=== Making a Local Payment
At this point, you will easily understand why HTLCs are used for both "remote" and "local" payments. When Alice pays Bob for a coffee, she doesn't just update the channel balance and commit to a new state. Instead, the payment is made with an HTLC, in the same way Alice paid Dina. The fact that there's only one channel "hop" makes no difference. It would work like this:
Onion routing describes a method of encrypted communication where a message sender builds successive _nested layers of encryption_ that are "peeled" off by each intermediary node, until the innermost layer is delivered to the intended recipient. The name "onion routing" describes this use of layered encryption that is peeled off one layer at a time, like the skin of an onion.
@ -17,13 +17,13 @@ The Lightning Network uses an implementation of onion routing protocol based on
The implementation of onion routing in the Lightning Network is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md[BOLT #4 - Onion Routing Protocol]
=== A physical example illustrating onion routing
=== A Physical Example Illustrating Onion Routing
There are many ways to describe onion routing, but one of the easiest is to use the physical equivalent of sealed envelopes. An envelope represents a layer of encryption, allowing only the named recipient to open it and read the contents.
Let's say Alice wants to send a secret letter to Dina, indirectly via some intermediaries.
==== Selecting a path
==== Selecting a Path
The Lightning Network uses _source routing_, which means that the payment path is selected and specified by the sender, and only the sender. In this example, Alice's secret letter to Dina will be the equivalent of a payment. To make sure the letter reaches Dina, Alice will create a path from her to Dina, using Bob and Chan as intermediaries.
@ -36,7 +36,7 @@ As a reminder, the path selected by Alice is shown in <<alice_dina_path>>, below
[[alice_dina_path]]
.Path: Alice to Bob to Chan to Dina
image::images/alice_dina_path.png["Alice to Bob to Chan to Dina"]
image::images/mtln_1002.png["Alice to Bob to Chan to Dina"]
Let's see how Alice can use this path without revealing information to intermediaries Bob and Chan.
@ -47,39 +47,39 @@ Internet routing is based on _packet switching_ at each intermediary routing nod
However, the lack of privacy in such a routing mechanism, where every intermediary node sees the sender and recipient, make this a poor choice for use in a payment network.
****
==== Building the layers
==== Building the Layers
Alice starts by writing a secret letter to Dina. She then seals the letter inside an envelope and writes "To Dina" on the outside (see <<dina_envelope>>). The envelope represents encryption with Dina's public key, so that only Dina can open the envelope and read the letter.
[[dina_envelope]]
.Dina's secret letter, sealed in an envelope
image::images/dina_envelope.png[Dina's secret letter, sealed in an envelope]
image::images/mtln_1003.png["Dina's secret letter, sealed in an envelope"]
Dina's letter will be delivered to Dina by Chan, who is immediately before Dina in the "path". So, Alice puts Dina's envelope inside an envelope addressed to Chan (see <<chan_envelope>>). The only part that Chan can read is the destination (routing instructions): "To Dina". Sealing this inside an envelope addressed to Chan represents encrypting it with Chan's public key so that only Chan can read the envelope address. Chan still can't open Dina's envelope. All he sees is the instructions on the outside (the address).
Now, this letter will be delivered to Chan by Bob. So Alice puts it inside an envelope addressed to Bob (see <<bob_envelope>>). As before, the envelope represents a message encrypted to Bob that only Bob can read. Bob can only read the outside of Chan's envelope (the address), so he knows to send it to Chan.
Now, if we could look through the envelopes (with X-rays!) we would see the envelopes nested one inside the other, as shown in <<nested_envelopes>>, below:
Alice now has an envelope that says "To Bob" on the outside. It represents an encrypted message that only Bob can open (decrypt). Alice will now begin the process by sending this to Bob. The entire process is shown in <<sending_nested_envelopes>> below:
[[sending_nested_envelopes]]
.Sending the envelopes
image::images/sending_nested_envelopes.png[Sending the envelopes]
image::images/mtln_1007.png[Sending the envelopes]
As you can see, Bob receives the envelope from Alice. He knows it came from Alice, but doesn't know if Alice is the original sender or just someone forwarding envelopes. He opens it to find an envelope inside that says "To Chan". Since this is addressed to Chan, Bob can't open it. He doesn't know what's inside it and doesn't know if Chan is getting a letter or another envelope to forward. Bob doesn't know if Chan is the ultimate recipient or not. Bob forwards the envelope to Chan.
@ -91,7 +91,7 @@ This is the essence of onion routing. The sender wraps a message in layers, spec
Now, let's look at the details of the onion routing implementation in the Lightning Network.
=== Introduction to onion routing of HTLCs
=== Introduction to Onion Routing of HTLCs
Onion routing in the Lightning Network appears complex at first glance, but once you understand the basic concept is really quite simple.
@ -103,13 +103,13 @@ Each intermediary node, or Bob and Chan in our example, is called a _hop_. Every
Now that we know the terminology used in Lightning Onion Routing, let's restate Alice's task: Alice must construct an _onion_ with _hop data_, telling each _hop_ how to construct an _outgoing HTLC_ in order to send a payment to the _final node_ (Dina).
==== Alice selects the path
==== Alice Selects the Path
From <<routing>> we know that Alice will send a 50,000 satoshi payment to Dina via Bob and Chan. This payment is transmitted via a series of HTLCs, as shown in <<alice_dina_htlc_path>>, below:
[[alice_dina_htlc_path]]
.Payment path with HTLCs from Alice to Dina
image::images/alice-dina-htlc.png[Payment path with HTLCs from Alice to Dina]
image::images/mtln_1008.png[Payment path with HTLCs from Alice to Dina]
As we will see in <<gossip>>, Alice is able to construct this path to Dina because Lightning nodes announce their channels to the entire Lightning Network using the _Lightning Gossip Protocol_. After the initial channel announcement, Bob and Chan each sent out an additional "channel update" message with their routing fee and timelock expectations for payment routing.
@ -127,11 +127,11 @@ This information is used by Alice to identify the nodes, channels, fees, and tim
[[alice_dina_path_detail]]
.A detailed path constructed from gossiped channel and node information
image::images/alice_dina_path_detail.png[A path constructed from gossiped channel and node information]
image::images/mtln_1009.png[A path constructed from gossiped channel and node information]
Alice already knows her own channel to Bob, and therefore doesn't need this info to construct the path. Note also that Alice didn't need a channel update from Dina, because she has the update from Chan for that last channel in the path.
==== Alice constructs the payloads
==== Alice Constructs the Payloads
There are two possible formats that Alice can use for the information communicated to each hop: A legacy fixed-length format called the _hop data_ and a more flexible Type-Length-Value (TLV) based format called the _hop payload_. The TLV message format is explained in more detail in <<tlv>>. It offers flexibility by allowing fields to be added to the protocol at will.
@ -142,7 +142,7 @@ Both formats are specified in https://github.com/lightningnetwork/lightning-rfc/
Alice will start building the hop data from the end of the path backwards: Dina, Chan, Bob.
===== Final node payload for Dina
===== Final node payload for dina
Alice first builds the payload that will be delivered to Dina. Dina will not be constructing an "outgoing HTLC", because Dina is the final node and payment recipient. For this reason, the payload for Dina is different than all the others (uses all zeros for the `short_channel_id`), but only Dina will know this since it will be encrypted in the innermost layer of the onion. Essentially, this is the "secret letter to Dina" we saw in our physical envelope example.
@ -171,9 +171,9 @@ Alice serializes it in TLV format as shown (simplified) <<dina_onion_payload>> b
[[dina_onion_payload]]
.Dina's payload is constructed by Alice
image::images/dina_onion_payload.png[Dina's payload is constructed by Alice]
image::images/mtln_1010.png[Dina's payload is constructed by Alice]
===== Hop payload for Chan
===== Hop payload for chan
Next, Alice constructs the hop payload for Chan. This will tell Chan how to setup an outgoing HTLC to Dina.
@ -189,9 +189,9 @@ Alice serializes this payload in TLV format, as shown (simplified) in <<chan_oni
[[chan_onion_payload]]
.Chan's payload is constructed by Alice
image::images/chan_onion_payload.png[Chan's payload is constructed by Alice]
image::images/mtln_1011.png[Chan's payload is constructed by Alice]
===== Hop payload for Bob
===== Hop payload for bob
Finally, Alice constructs the hop payload for Bob, which also contains the same three fields as the hop payload for Chan, but with different values:
@ -214,7 +214,7 @@ Alice serializes this payload in TLV format, as shown (simplified) in <<bob_onio
[[bob_onion_payload]]
.Bob's payload is constructed by Alice
image::images/bob_onion_payload.png[Bob's payload is constructed by Alice]
image::images/mtln_1012.png[Bob's payload is constructed by Alice]
===== Finished hop payloads
@ -222,9 +222,9 @@ Alice has now built the three hop payloads that will be wrapped in an onion. A s
[[onion_hop_payloads]]
.Hop payloads for all the hops
image::images/onion_hop_payloads.png[Hop payloads for all the hops]
image::images/mtln_1013.png[Hop payloads for all the hops]
==== Key generation
==== Key Generation
Alice must now generate several keys that will be used to encrypt the various layers in the onion.
@ -258,7 +258,7 @@ The relationship between the various keys and how they are generated is shown in
@ -360,7 +360,7 @@ Note that the very first hop uses the original session key generated, as this ke
[[wrapping_the_onion]]
=== Wrapping the onion layers
=== Wrapping the Onion Layers
The process of wrapping the onion is detailed in https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md#packet-construction[BOLT #4 - Onion Routing - Packet Construction].
@ -368,7 +368,7 @@ In this section we will describe this process at a high-level and somewhat simpl
[[fixed_length_onions]]
==== Fixed-length Onions
==== Fixed-Length Onions
We've mentioned the fact that none of the "hop" nodes know how long the path is, or where they are in the path. How is this possible?
@ -389,7 +389,7 @@ The onion size is 1366 bytes structured as shown in <<onion_packet>> below:
[[onion_packet]]
.The onion packet
image::images/onion_packet.png[]
image::images/mtln_1015.png[]
* 1 byte: A version byte
* 33 bytes: A compressed public session key (<<session_key>>) from which the per-hop shared secret (<<shared_secret>>) can be generated without revealing Alice's identity
@ -400,7 +400,7 @@ A unique trait of Sphinx as a mix-net packet format is that rather than include
In practice, this little trick allows us to keep the onion packet as compact as possible while still retaining the desired security properties.
==== Wrapping the onion (outlined)
==== Wrapping the Onion (Outlined)
Here is the process of wrapping the onion, outlined below. Come back to this list, as we explore each step with our real-world example.
@ -440,7 +440,7 @@ Note that at each phase the _mu_ key is used to generate an HMAC over the _encry
This outer HMAC allows the node processing the packet to verify the integrity of the onion packet (no bytes modified).
The inner HMAC is then revealed during the inverse of the "shift and encrypt" routine described above, which serves as the _outer_ HMAC for the next hop.
==== Wrapping Dina's hop payload
==== Wrapping Dina's Hop Payload
As a reminder, the onion is wrapped by starting at the end of the path from Dina, the final node or recipient. Then the path is built in reverse all the way back to the sender, Alice.
@ -455,13 +455,13 @@ This is shown in <<onion_payload_filler>>:
[[onion_payload_filler]]
.Filling the onion payload with a random byte-stream
image::images/onion_payload_filler.png[]
image::images/mtln_1016.png[]
Alice will now insert Dina's hop payload into the left side of the 1300 byte array, shifting the filler to the right and discarding anything that overflows. This is visualized in <<onion_add_dina>>:
[[onion_add_dina]]
.Adding Dina's hop payload
image::images/onion_add_dina.png[]
image::images/mtln_1017.png[]
Another way to look at this is that Alice measures the length of Dina's hop payload, shifts the filler right to create an equal space in the left side of the onion payload and inserts Dina's payload in that space.
@ -473,7 +473,7 @@ To do this, Alice generates a byte-stream using the +rho+ key (which Dina also k
[[onion_obfuscate_dina]]
.Obfuscating the onion payload
image::images/onion_obfuscate_dina.png[]
image::images/mtln_1018.png[]
One of the properties of XOR is that if you do it twice you get back to the original data. As we will see in the "unwrapping" section, if Dina applies the same XOR operation with the byte-stream generated from +rho+, it will reveal the original onion payload.
@ -488,9 +488,9 @@ Finally, Alice calculates a Hash-based Message Authentication Code (HMAC) for Di
[[dina_hop_payload_hmac]]
.Adding an HMAC integrity checksum to Dina's hop payload
image::images/dina_hop_payload_hmac.png[]
image::images/mtln_1019.png[]
===== Onion Routing Replay Protection & Detection
===== Onion routing replay protection & detection
The HMAC acts as a secure checksum and helps Dina verify the integrity of the hop payload. The 32-byte HMAC is appended to Dina's hop payload.
Note that we compute the HMAC over the _encrypted_ data rather then over the plaintext data.
@ -519,13 +519,13 @@ Once again, due to the unique attributes of the HTLC construct, we can make a fu
Given that HTLCs are _time locked_ contracts based on the absolute block height, once an HTLC has expired, then the contract is effectively permanently closed.
As a result, nodes can use this CLTV expiration height as an indicator to know when it's safe to garbage collect an entry in the anti-replay log.
==== Wrapping Chan's hop payload
==== Wrapping Chan's Hop Payload
In <<chan_onion_wrapping>> we see the steps used to wrap Chan's hop payload in the onion. These are the same steps Alice used to wrap Dina's hop payload.
[[chan_onion_wrapping]]
.Wrapping the onion for Chan
image::images/chan_onion_wrapping.png[]
image::images/mtln_1020.png[]
Alice starts with the 1300 onion payload created for Dina. The first 65 (or less) bytes of this are Dina's payload obfuscated and the rest is filler. Alice must be careful not to overwrite Dina's payload.
@ -544,7 +544,7 @@ Now Chan's payload is in the front of the onion. When Chan sees this he has no i
Next, Alice obfuscates the entire payload by XOR with the byte-stream generated from the Alice-Chan +rho+ key. Only Alice and Chan have this +rho+ key and only they can produce the byte stream to obfuscate and de-obfuscate the onion.
Finally, as we did in the earlier step, we compute Chan's outer HMAC, which is what she'll use to verify the integrity of the encrypted onion packet.
==== Wrapping Bob's hop payload
==== Wrapping Bob's Hop Payload
In <<bob_onion_wrapping>> we see the steps used to wrap Bob's hop payload in the onion.
@ -552,7 +552,7 @@ All right, by now this is easy!
[[bob_onion_wrapping]]
.Wrapping the onion for Bob
image::images/bob_onion_wrapping.png[]
image::images/mtln_1021.png[]
Start with the onion payload (obfuscated) containing Chan's and Dina's hop payloads.
@ -565,7 +565,7 @@ Obfuscate the whole thing XOR with the _rho_ key from the Alice-Bob-shared-secre
Calculate the outer HMAC and stick it on the end of Bob's hop payload.
==== The final onion packet
==== The Final Onion Packet
The final onion payload is ready to be sent to Bob. Alice doesn't need to add any more hop payloads.
@ -581,13 +581,13 @@ The result can be seen in <<onion_packet_2>> below:
[[onion_packet_2]]
.The onion packet
image::images/onion_packet.png[]
image::images/mtln_1015.png[]
=== Sending the onion
=== Sending the Onion
In this section we will look at how the onion packet is forwarded and how HTLCs are deployed along the path.
==== The update_add_htlc message
==== The update_add_htlc Message
Onion packets are sent as part of the +update_add_htlc+ message. If you recall from <<update_add_htlc>>, in <<channel_operation>>, we saw the contents of the +update_add_htlc+ message were as follows:
@ -602,7 +602,7 @@ Onion packets are sent as part of the +update_add_htlc+ message. If you recall f
You will recall that this message is sent by one channel partner to ask the other channel partner to add an HTLC. This is how Alice will ask Bob to add an HTLC to pay Dina. Now you understand the purpose of the last field +onion_routing_packet+, which is 1366 bytes long. It's the fully-wrapped onion packet we just constructed!
==== Alice sends the onion to Bob
==== Alice Sends the Onion to Bob
Alice will send the +update_add_htlc+ message to Bob. Let's look at what this message will contain:
@ -618,7 +618,7 @@ cltv_expiry:: The expiry timelock for the HTLC will be 700,058. Alice adds 20 bl
onion_routing_packet:: The final onion packet Alice constructed with all the hop payloads!
==== Bob checks the onion
==== Bob Checks the Onion
As we saw in <<channel_operation>>, Bob will add the HTLC to the commitment transactions and update the state of the channel with Alice.
@ -632,7 +632,7 @@ Now that Bob has generated the shared key and verified the HMAC, he can start un
If Bob extracts and removes his hop payload, the remaining onion will not be 1300 bytes, it will be shorter! So the next hop will know that they are not the first hop and will be able to detect how long the path is. To prevent this, Bob needs to add more filler to refill the onion.
==== Bob generates filler
==== Bob Generates Filler
Bob generates filler in a slightly different way than Alice, but following the same general principle.
@ -640,11 +640,11 @@ First, Bob *extends* the onion payload by 1300 bytes and filles them with +0+ va
[[bob_extends]]
.Bob extends the onion payload by 1300 (zero-filled) bytes
image::images/bob_extends.png[Bob extends the onion payload by 1300 (zero-filled) bytes]
image::images/mtln_1023.png[Bob extends the onion payload by 1300 (zero-filled) bytes]
This empty space will become obfuscated and turn into "filler", by the same process that Bob uses to deobfuscate his own hop payload. Let's see how that works.
==== Bob de-obfuscates his hop payload
==== Bob De-Obfuscates His Hop Payload
Next, Bob will generate the +rho+ key from the Alice-Bob shared key. He will use this to generate a 2600 byte stream, using the +ChaCha20+ algorithm.
@ -661,20 +661,20 @@ At the same time, applying the +rho+ byte stream to the 1300 zeroes that were ad
[[bob_deobfuscates]]
.Bob de-obfuscates the onion, obfuscates the filler
image::images/bob_deobfuscates.png[Bob de-obfuscates the onion, obfuscates the filler]
image::images/mtln_1024.png["Bob de-obfuscates the onion, obfuscates the filler"]
==== Bob extracts the outer HMAC for the next hop
==== Bob Extracts the Outer HMAC for the Next Hop
Remember that an inner HMAC is included for each hop, then will then become the outer HMAC for the _next_ hop.
In this case, Bob extracts the inner HMAC (he's already verified the integrity of the encrypted packet w/ the outer HMAC), and puts it aside as he'll append it to the deobfuscated packet to allow Chan to verify the HMAC of her encrypted packet.
==== Bob removes his payload and left shifts the onion
==== Bob Removes His Payload and Left Shifts the Onion
Now Bob can remove his hop payload from the front of the onion and left-shift the remaining data. An amount of data equal to Bob's hop payload from the second-half 1300 bytes of filler will now shift into the onion payload space. This is shown in <<bob_removes_shifts>>:
[[bob_removes_shifts]]
.Bob removes the hop payload and left-shifts the rest, filling the gap with new filler
image::images/bob_removes_shifts.png[Bob removes the hop payload and left-shifts the rest, filling the gap with new filler]
image::images/mtln_1025.png["Bob removes the hop payload and left-shifts the rest, filling the gap with new filler"]
Now Bob can keep the first half 1300 bytes, and discard the extended (filler) 1300 bytes.
@ -682,7 +682,7 @@ Bob now has a 1300 byte onion packet to send to the next hop. It is almost ident
No one can tell the difference between filler put there by Alice and filler put there by Bob. Filler is filler! It's all random bytes anyway. Note that if Bob (or one of Bob's other aliases) is present in the route in two disctinct locations, then they can tell the difference as the base protocol always uses the same payment hash across the entire route. Atomic Multipath Payments (AMP), and Point Time Locked Contrats (PTLCs) eliminate the correlation vector by randomizing the payment identifier across each route/hop.
==== Bob constructs the new onion packet
==== Bob Constructs the New Onion Packet
Bob now copies the onion payload into the onion packet, appends the outer HMAC for chan, re-randomizes the session key (the same way Alice the sender does) with the Elliptic Curve multiplication operation, and appends a fresh version byte.
The `session_key_chan` public key will then be appended to the front of the onion packet for processing by Chan.
==== Bob verifies the HTLC details
==== Bob Verifies the HTLC Details
Bob's hop payload contains the instructions needed to create an HTLC for Chan.
@ -710,7 +710,7 @@ Next, Bob confirms that the outgoing amount (50,100 satoshis) is less than the i
Similarly, Bob checks that the outgoing cltv_expiry is less than the incoming cltv_expiry giving Bob enough time to claim the incoming HTLC if there is a breach.
==== Bob sends the update_add_htlc to Chan
==== Bob Sends the update_add_htlc to Chan
Bob now constructs and HTLC to send to Chan, as follows:
@ -726,7 +726,7 @@ cltv_expiry:: The expiry timelock for the HTLC will be 700,038.
onion_routing_packet:: The onion packet Bob reconstructed after removing his hop payload.
==== Chan forwards the onion
==== Chan Forwards the Onion
Chan repeats the exact same process as Bob:
@ -742,7 +742,7 @@ Chan repeats the exact same process as Bob:
6. Chan uses the generated byte stream to XOR and de-obfuscate the onion payload. Simultaneously the XOR operation obfuscates the extra 1300 zeroes, turning them into filler
7. Chan extracts the inner HMAC in the payload, which will become the outer HAMC for Dina.
7. Chan extracts the inner HMAC in the payload, which will become the outer HMAC for Dina.
8. Chan removes his hop payload and left-shifts the onion payload by the same amount. Some of the filler generated in the 1300 extended bytes move into the first-half 1300 bytes becoming part of the onion payload.
@ -754,13 +754,13 @@ Chan repeats the exact same process as Bob:
12. Chan re-randomizes the session key as Bob did in the prior hop for Dina.
==== Dina receives the final payload
==== Dina Receives the Final Payload
When Dina receives the +update_add_htlc+ message from Chan, she knows from the +payment_hash+ that this is a payment for her. She knows she is the last hop in the onion.
Dina follows the exact same process as Bob and Chan to verify and unwrap the onion, except she doesn't construct new filler and doesn't forward anything. Instead, Dina responds to Chan with +update_fulfill_htlc+ to redeem the HTLC. The +update_fulfill_htlc+ will flow backwards along the path until it reaches Alice. All the HTLCs are redeemed and channel balances are updated. The payment is complete!
=== Returning errors
=== Returning Errors
This far we've looked at the forward propagation of the onion establishing the HTLCs, and the backwards propagation of the payment secret unwinding the HTLCs once payment is successful.
@ -829,18 +829,18 @@ However, this means that the sender of the HTLC has to wait until expiry, and th
One proposed solution to this problem is called "stuckless payments", and it depends on Point Time-Locked Contracts (PTLCs), which are payment contract that use a different cryptographic primitive than HTLCs (i.e. point addition on the elliptic curve instead of a hash and secret pre-image). PTLCs are cumbersome using ECDSA but much easier with Bitcoin's Taproot and Schnorr Signature features which were recently locked in for activation in November 2021. It is expected that PTLCs will be implemented in the Lightning Network after these Bitcoin features become activated.
[[keysend]]
=== `keysend` Spontaenous Payments
=== Keysend Spontaneous Payments
In the payment flow described earlier in the chapter, we assumed that Dina
received an invoice from Alice "out of band", or obtained it via some mechanism
received an invoice from Alice "out of band," or obtained it via some mechanism
unrelated to the protocol (typically copy/paste or QR code scans). This trait
means that the payment process always takes two steps: first the sender
obtains an invoice and then uses the payment hash (encoded in the invoice) to
successfully route an HTLC. This extra round trip required to obtain an invoice
means that the payment process always takes two steps: first, the sender
obtains an invoice and second, uses the payment hash (encoded in the invoice) to
successfully route an HTLC. The extra round trip required to obtain an invoice
before making a payment may be a bottleneck in applications that involve
streaming micropayments over Lightning. What if we could just "push" a payment
streaming micropayments over Lightning. What if we could just "push" a payment
over spontaneously, without having to obtain an invoice from the recipient
first? The "keysend" protocol is an end-to-end extension (only the sender and
first? The `keysend` protocol is an end-to-end extension (only the sender and
receiver are aware) to the Lightning protocol that enables spontaneous push
payments.
@ -849,43 +849,42 @@ payments.
The modern Lightning protocol uses the TLV (Type-Length-Value) encoding in
the onion to encode information that tells each node _where_ and _how_ to
forward the payment. Leveraging the TLV format, each piece of routing information
(like the next node to pass the HTLC to) is assigned a specific type (or key)
(like the next node to which to pass the HTLC) is assigned a specific type (or key)
encoded as a `BigSize` variable length integer (max sized as as 64-bit
integer). These "essential" (reversed values below `65536`) types are defined
in BOLT-04 along with the rest of the onion routing details. Onion types with a
value greater than `65536` are intended to be used by wallets and applications
as "custom records".
as "custom records."
Custom records allow payment applications to attach additional metadata or
context to a payment as key/value pairs in the onion. Since the custom records
are included in the onion payload itself, like all other hop contents, the
records are end to end encrypted. As the custom records effectively consume a
portion of the fixed-sized `1300` bytes onion packet, encoding each key and
records are end-to-end encrypted. As the custom records effectively consume a
portion of the fixed-size `1300` bytes onion packet, encoding each key and
value of each custom record reduces the amount of available space for encoding
the rest of the route. In practice, this means that more onion space that is
used for custom records, the shorter the route can be. Given that each HTLC
packet is fixed sized, custom records don't _add_ any additional data to an
HTLC, rather they re-allocate bytes that would've been filled with random data
the rest of the route. In practice, this means that the more onion space used for custom records, the shorter the route can be. Given that each HTLC
packet is fixed-size, custom records don't _add_ any additional data to an
HTLC; rather, they reallocate bytes that would have been filled with random data
otherwise.
==== Sending & Receiving `keysend` payments
==== Sending and Receiving Keysend Payments
A `keysend` payment inverts the typical flow of an HTLC where the receiver
reveals a secret pre-image to the sender. Instead, the sender includes the
pre-image _within_ the onion to the receiver and routes the HTLC to the
receiver. The receiver then decrypts the onion payload, and uses the included
pre-image (which MUST match the payment hash of the HTLC) to settle the
pre-image (which _must_ match the payment hash of the HTLC) to settle the
payment. As a result, `keysend` payments can be carried out without first
obtaining an invoice from the receiver, as the pre-image is "pushed" over to
the receiver. A `keysend` payment uses a TLV custom record type of `5482373484`
to encode a 32-byte pre-image value.
==== `keysend` & Custom Records in Lightning Applications
==== Keysend and Custom Records in Lightning Applications
Many streaming Lightning applications use the `keysend` protocol to continually
stream satoshis to a destination identified by its public key in the network.
Typically an application will also include additional metadata such as a
tipping/donation note, or other application level information in addition to
Typically, an application will also include metadata such as a
tipping/donation note or other application-level information in addition to
In this chapter we will describe the Lightning Network's _Gossip Protocol_ and how it is used by nodes to construct and maintain a _Channel Graph_. We will also review the _DNS Bootstrap_ mechanism used to find peers to "gossip" with.
@ -7,7 +7,7 @@ The gossip protocol and routing information section is highlighted by a double o
As we've learned already, the Lightning Network uses a source-based onion routing protocol to deliver a payment from a sender to the recipient.
To do this, the sending node must be able to construct a path of payment channels that connects it with the recipient, as we will see in <<path_finding>>.
@ -252,24 +252,24 @@ Try some combinations of the various flags using the +dig+ DNS command-line tool
dig @8.8.8.8 r0.a6.nodes.lightning.directory SRV
----
=== The channel graph
=== The Channel Graph
Now that our new node is able to use the DNS bootstrapping protocol to connect to their very first peer, it can start to sync the channel graph! However, before we sync the channel graph, we'll need to learn exactly _what_ we mean by the channel graph. In this section we'll explore the precise _structure_ of the channel graph and examine the unique aspects of the channel graph compared to the typical abstract "graph" data structure which is well known/used in the field of computer science.
==== A directed graph
==== A Directed Graph
A graph in computer science is a special data structure composed of vertices (typically referred to as nodes) and edges (also known as links). Two nodes may be connected by one or more edges. The channel graph is also _directed_ given that a payment is able to flow in either direction over a given edge (a channel). An example of a _directed graph_ is shown below:
In the context of the Lightning Network, our vertices are the Lightning nodes themselves, with our edges being the payment channels connecting these nodes. As we're concerned with _routing payments_, in our model a node with no edges (no payment channels) isn't considered to be a part of the graph as it isn't useful.
As channels themselves are UTXOs (funded 2-of-2 multisig addresses), we can view the channel graph as a special subset of the Bitcoin UTXO set, on top of which we can add some additional information (the nodes, etc) to arrive at the final overlay structure which is the channel graph. This anchoring of fundamental components of the channel graph in the
base Bitcoin blockchain means that it's impossible to _fake_ a valid channel graph, which has useful properties when it comes to spam prevention as we'll see later.
=== Gossip protocol messages
=== Gossip Protocol Messages
The channel graph information is propagated across the Lightning P2P Network as three messages, which are described in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7]:
@ -285,7 +285,7 @@ themselves _authenticated_ allowing a 3rd party to ensure that the owner of a ch
With the high level structure of the channel graph laid out, we'll now dive down into the precise structure of each of the three messages used to gossip the channel graph. We'll also explain how one can also _verify_ each message and component of the channel graph.
[[node_announcement]]
==== The node_announcement message
==== The node_announcement Message
First, we have the +node_announcement+ message, which serves two primary
purposes:
@ -333,7 +333,7 @@ Validating an incoming +node_announcement+ is straight forward. The following as
* The included +alias+ bytes MUST be a valid UTF-8 string.
==== The channel_announcement message
==== The channel_announcement Message
Next, we have the +channel_announcement+ message, which is used to _announce_ a new _public_ channel to the wider network. Note that announcing a channel is _optional_. A channel only needs to be announced if it is intended to be used for routing by the Lightning network. Active routing nodes may wish to announce all their channels. However, certain nodes like mobile nodes likely don't have the
uptime or desire to be an active routing node. As a result, these
@ -345,7 +345,7 @@ An unannounced channel isn't part of the known public channel graph, but can sti
Due to the existence of unadvertised channels, the _true_ size of the channel graph (both the public and private components) is unknown.
===== Locating a channel on the Bitcoin blockchain
===== Locating a channel on the bitcoin blockchain
As mentioned earlier, the channel graph is authenticated due to its usage of public key cryptography, as well as the Bitcoin blockchain as a spam prevention system. In order to have a node accept a new +channel_announcement+, the advertisement must _prove_ that the channel actually exists in the Bitcoin blockchain. This proof system adds an upfront cost to adding a new entry to the channel graph (the on-chain fees one must pay to create the UTXO of the channel). As a result, we mitigate spam and ensure that a dishonest node on the network can't fill up the memory of an honest node at no cost with bogus channels.
@ -357,7 +357,7 @@ The +scid+ is used both in +channel_announcement+ (and +channel_update+) as well
[[short_channel_id]]
[[scid]]
===== The Short Channel ID
===== The short channel id
Based on the information above, we have three pieces of information we need to encode in order to uniquely reference a given channel. As we want a compact representation, we'll attempt to encode the information into a _single_ integer. Our integer format of choice is an unsigned 64-bit integer, comprised of 8 bytes.
@ -431,7 +431,7 @@ Finally, the verifier will reconstruct the multisig script from +bitcoin_key_1+
The verifier has now independently verified that the payment channel in the announcement is funded and confirmed on the Bitcoin blockchain!
==== The channel_update message
==== The channel_update Message
The third and final message used in the gossip protocol is the +channel_update+ message. Two of these are generated for each payment channel (one by each channel partner) announcing their routing fees, timelock expectations and capabilities.
@ -5,7 +5,7 @@ Payment delivery on the Lightning Network depends on finding a path from the sen
In this chapter we will examine the problem of path finding, understand how uncertainty about channel balances complicates this problem and look at how a typical path finding implementation attempts to solve it.
=== Path finding in the Lightning protocol suite
=== Path Finding in the Lightning Protocol Suite
Path finding, path selection, multi-path payments (MPP) and the payment attempt trial & error loop occupy the majority of the "Payment Layer" at the top of the protocol suite.
@ -13,15 +13,15 @@ These components are highlighted by a double outline in the protocol suite, show
So far we've looked at several technologies that are part of the Lightning Network and we have seen their exact specification as part of a BOLT standard. You may be surprised to find that path finding is not part of the BOLTs!
That's because path finding isn't an activity that requires any form of coordination or interoperability between different implementations. As we've seen, the path is selected by the sender. Even though the routing details are specified in detail in the BOLTs, the path discovery and selection are left entirely up to the sender. So each node implementation can choose a different strategy/algorithm to find paths. In fact, the different node/client and wallet implementations can even compete and use their path finding algorithm as a point of differentiation.
=== Path finding: what problem are we solving?
=== Path Finding: What Problem Are We Solving?
The term path finding may be somewhat misleading, because it implies a search for _a single path_ connecting two nodes. In the beginning, when the Lightning Network was small and not well interconnected, the problem was indeed about finding a way to join payment channels to reach the recipient.
@ -29,7 +29,7 @@ But, as the Lightning Network has grown explosively, the path finding problem's
[[lngraph]]
.A visualization of part of the Lightning Network as of July 2021
image::images/LNGraphJuly2021.png[]
image::images/mtln_1202.png[]
[NOTE]
====
@ -38,7 +38,7 @@ The network visualization above was produced with a simple python script you can
If the sender and recipient are connected to other well-connected nodes and have at least one channel with adequate capacity - there will be thousands of paths. The problem becomes: selecting the _best_ path that will succeed in payment delivery, out of a list of thousands of possible paths.
==== Selecting the best path
==== Selecting the Best Path
To select the "best" path, we have to first define what we mean by "best". There may be many different criteria such as:
@ -51,13 +51,13 @@ To select the "best" path, we have to first define what we mean by "best". There
All of these criteria may be desirable to some extent and selecting paths that are favorable across many dimensions is not an easy task. Optimization problems like this may be too complex to solve for the "best" solution, but often can be solved for some approximation of the optimal. Which is good news, because otherwise path finding would be an intractable problem.
==== Path finding in math and computer science
==== Path Finding in Math and Computer Science
Path finding in the Lightning Network falls under a general category of _graph theory_ in mathematics and the more specific category of _graph traversal_ in computer science.
A network such as the Lightning Network can be represented as a mathematical construct called a _graph_, where _nodes_ are connected to each other by _edges_ (equivalent to the payment channels). The Lightning Network forms a _directed graph_ because the nodes are linked _asymmetrically_, since the channel balance is split between the two channel partners and the payment liquidity is different in each direction. A directed graph with numerical capacity constrains on its edges is called a _flow network_, a mathematical construct used to optimize transportation and other similar networks. Flow networks can be used as a framework when solutions need to achieve a specific flow while minimizing cost, known as the Minimum Cost Flow Problem (MCFP).
==== Capacity, balance, liquidity
==== Capacity, Balance, Liquidity
In order to better understand the problem of transporting satoshis from point A to point B, we need to better define three important terms: capacity, balance, and liquidity. We use these terms to describe a payment channel's ability to route a payment.
If we knew the exact channel balances of every channel, we could compute one or more payment paths using any of the standard path finding algorithms taught in good computer science programs. But we don't know the channel balances, we only know the aggregate channel capacity, which is advertised by nodes in channel announcements. In order for a payment to succeed, there must be adequate balance on the sending side of the channel. If we don't know how the capacity is distributed between the channel partners, we don't know if there is enough balance in the direction we are trying to send the payment.
Our channel liquidity uncertainty range is the range between the minimum and maximum possible liquidity. This is unknown to the network, except the two channel partners. However, as we will see we can use failed HTLCs returned from our payment attempts to update our liquidity estimate and reduce uncertainty. If for example we get an HTLC failure code that tells us that a channel cannot fulfill an HTLC that is smaller than our estimate for maximum liquidity, that means the maximum liquidity can be updated to the amount of the failed HTLC. In simpler terms, if we think the liquidity can handle an HTLC of N satoshis and we find out it fails to deliver M satoshis (where M is smaller), then we can update our estimate to M-1 as the upper bound. We tried to find the ceiling and bumped against it, so it's lower than we thought!
==== Path finding complexity
==== Path Finding Complexity
Finding a path through a graph is a problem modern computers can solve rather efficiently.
Developers mainly choose breadth-first search if the edges are all of equal weight.
@ -113,7 +113,7 @@ But we can't be certain unless we try it out or ask the channel owners directly.
Even if we were able to ask the channel owners directly, their balance might change by the time we have asked others, computed a path, constructed an onion and send it along.
Not only do we have limited information but the information we have is highly dynamic and might change at any point in time without our knowledge.
==== Keeping it simple
==== Keeping It Simple
The path finding mechanism implemented in Lightning nodes is to first create a list of candidate paths, filtered and sorted by some function. Then, the node or wallet will probe paths (by attempting to deliver a payment) in a trial-and-error loop until a path is found that successfully delivers the payment.
@ -127,7 +127,7 @@ While "blind probing" is not optimal and leaves ample room for improvement, it s
Most Lightning node and wallet implementations improve on this approach, by ordering/weighting the list of candidate paths. Some implementations order the candidate paths by cost (fees), or some combination of cost/capacity.
=== Path finding and payment delivery process
=== Path Finding and Payment Delivery Process
Path finding and payment delivery involves several steps, which we list below. Different implementations may use different algorithms and strategies, but the basic steps are likely to be very similar:
@ -151,7 +151,7 @@ These three activities can be repeated in a _payment round_ if we use the failur
In the next sections we will look at each of these steps in more detail, as well as more advanced payment strategies.
=== Channel graph construction
=== Channel Graph Construction
In <<gossip>> we covered the three main messages that nodes "gossip": +node_announcement+, +channel_announcement+, and +channel_update+. These three messages allow any node to gradually construct a "map" of the Lightning Network in the form of a _channel graph_. Each of these messages provides a critical piece of information for the channel graph:
@ -194,13 +194,13 @@ Selena listens to node_announcement messages and discovers 4 other nodes (in add
[[channel_graph_nodes]]
.Node announcements
image::images/channel_graph_nodes.png[]
image::images/mtln_1203.png[]
Selena also receives seven channel_announcement messages with the corresponding channel capacities, allowing her to construct a basic "map" of the network, shown in <<channel_graph_1>>, below:
[[channel_graph_1]]
.The channel graph
image::images/channel_graph_1.png[]
image::images/mtln_1204.png[]
===== Uncertainty in the channel graph
@ -210,7 +210,7 @@ But wait: Selena does know *some* channel balances! She knows the balances of th
[[channel_graph_2]]
.Channel graph with known and unknown balances
image::images/channel_graph_2.png[]
image::images/mtln_1205.png[]
While the "?" symbol seems ominous, a lack of certainty is not the same as complete ignorance. We can _quantify_ the uncertainty and _reduce_ it by updating the graph with the successful/failed HTLCs we attempt.
@ -223,7 +223,7 @@ Once we attempt to send an HTLC we can learn more about channel balances: if we
You may be thinking "What's the point of learning from a successful HTLC?" After all, if it succeeded we're "done". But consider that we may be sending one part of a multi-part payment. We also may be sending other single-part payments within a short time. Anything we learn about liquidity is useful for the next attempt!
====
==== Liquidity uncertainty and probability
==== Liquidity Uncertainty and Probability
To quantify the uncertainty of a channel's liquidity, we can apply probability theory. A basic model of the probability of payment delivery will lead to some rather obvious, but important, conclusions:
@ -241,7 +241,7 @@ First, let's posit that a channel with capacity +c+ has liquidity on one side wi
More simply, if the possible values for the liquidity are 0,1,2,3,4,5 only one of those six possible values will be sufficient to send our payment. To continue this example, if our payment amount was 3, then we would succeed if the liquidity was 3, 4, or 5. So our chances of success are 3 in 6 (50%). Expressed in math, the success probability function for a single channel is:
latexmath:[P_c(a) = (c + 1 - a) / (c + 1)]
latexmath:[$P_c(a) = (c + 1 - a) / (c + 1)$]
where +a+ is the amount and +c+ is the capacity
@ -253,7 +253,7 @@ Now let's think about the probability of success across a path made of several c
We can express this as an equation that calculates the probability of a payment's success as the product of probabilities for each channel in the path(s):
latexmath:[P_{payment} = \prod_{i=1}^n P_i]
latexmath:[$P_{payment} = \prod_{i=1}^n P_i$]
Where P_i_ is the probability of success over one path or channel, and P_payment_ is the overall probability of a successful payment over all the paths/channels.
@ -266,7 +266,7 @@ In other words, "The more channels (hops) you use, the lower the chance of succe
There is a lot of mathmatical theory and modelling behind the uncertainty of the liquidity in the channels. Fundamental work about modeling the uncertainty intervals of the channel liquidity can be found in the paper https://arxiv.org/abs/2103.08576[Security and Privacy of Lightning Network Payments with Uncertain Channel Balances] by (co-author of this book) Pickhardt et. al.
====
==== Fees and other channel metrics
==== Fees and Other Channel Metrics
Next, our sender will add information to the graph from +channel_update+ messages received from the intermediary nodes. As a reminder, the +channel_update+ contains a wealth of information about a channel and the expectations of one of the channel partners.
@ -274,13 +274,13 @@ In <<channel_graph_3>> below we see how Selena can update the channel graph base
[[channel_graph_3]]
.Channel graph fees and other channel metrics
image::images/channel_graph_3.png[]
image::images/mtln_1206.png[]
The fee and timelock information are very important not just as path selection metrics. As we saw in <<onion_routing>>, the sender needs to add up fees and timelocks (cltv_expiry_delta) at each hop to make the onion. The process of calculating fees happens from the recipient to the sender *backwards* along the path, because each intermediary hop expects an incoming HTLC with higher amount and expiry timelock than the outgoing HTLC they will send to the next hop. So, for example, if Bob wants 1000 satoshis in fees and 30 blocks of expiry timelock delta, to send a payment to Rashid, then that amount and expiry delta must be added to the HTLC _from Alice_.
It is also important to note that a channel must have liquidity that is sufficient not only for the payment amount but also for the cumulative fees of all the subsequent hops. Even though Selena's channel to Xavier (S-->X) has enough liquidity for a 1m satoshi payment, it *does not* have enough liquidity once we consider fees. We need to know fees because only paths that have sufficient liquidity for *both payment and all fees* will be considered.
=== Finding candidate paths
=== Finding Candidate Paths
Finding a suitable path through a directed graph like this is a well-studied computer science problem (known broadly as the "Shortest Path problem"), which can be solved by a variety of algorithms depending on the desired optimization and resource constraints.
@ -303,7 +303,7 @@ But, as we saw previously, the channel +S->X+ does not have enough liquidity for
With two possible paths, Selena is ready to attempt delivery!
=== Payment delivery (Trial-and-error loop)
=== Payment Delivery (Trial-And-Error Loop)
Selena's node starts the trial-and-error loop, by constructing the HTLCs, building the onion and attempting delivery of the payment. For each attempt, there are three possible outcomes:
@ -315,19 +315,19 @@ If the payment fails, then it can be re-tried via a different path by updating t
We'll look at what happens if the payment is "stuck" in <<stuck_payments>>. The important detail is that a stuck payment is the worst outcome because we cannot retry with another HTLC as both (the stuck one and the retry one) might go through eventually and cause a double payment.
==== First attempt (path #1)
==== First Attempt (Path #1)
Selena attempts the first path (S->A->B->R). She constructs the onion and sends it, but receives a failure code from Bob's node. Bob reports back a +temporary channel failure+ with a +channel_update+ identifying the channel B->R as the one that can't deliver. This attempt is shown in <<path_1_fail>>:
[[path_1_fail]]
.Path 1 attempt fails
image::images/path_1_fail.png[]
image::images/mtln_1207.png[]
===== Learning from failure
From this failure code, Selena will deduce that Bob doesn't have enough liquidity to deliver the payment to Rashid on that channel. Importantly, this failure narrows the uncertainty of the liquidity of that channel! Previously, Selena's node assumed that the liquidity on Bob's side of the channel was somewhere in the range (0, 4m). Now, she can assume that the liquidity is in the range (0, 999999). Similarly, Selena can now assume that the liquidity of that channel on Rashid's side is in the range (1m, 4m), instead of (0, 4m). Selena has learned a lot from this failure.
==== Second attempt (path #4)
==== Second Attempt (Path #4)
Now Selena attempts the fourth candidate path (S->A->B->X->Y->R). This is a longer path and will incur more fees, but it's now the best option for delivery of the payment.
@ -335,7 +335,7 @@ Fortunately, Selena receives an +update_fulfill_htlc+ message from Alice, indica
[[path_4_success]]
.Path 4 attempt succeeds
image::images/path_4_success.png[]
image::images/mtln_1208.png[]
===== Learning from success
@ -389,18 +389,18 @@ Quantifying this optimal balance of size/number-of-splits for a given channel gr
[NOTE]
====
To read more about the optimization problem known as Minimum-Cost Flows involved when splitting payments into different sizes and allocating them to paths see the paper: https://arxiv.org/abs/2107.05322[Optimally Reliable & Cheap Payment Flows on the Lightning Network] by (co-author of this book) Rene Pickhardt & Stefan Richter.
To read more about the optimization problem known as Minimum-Cost Flows involved when splitting payments into different sizes and allocating them to paths see the paper: https://arxiv.org/abs/2107.05322[Optimally Reliable & Cheap Payment Flows on the Lightning Network] by (co-author of this book) René Pickhardt & Stefan Richter.
====
In our example, Selena's node will attempt to split the 1m satoshi payment into two parts with 600k and 400k satoshi respectively and send them on two different paths. This is shown in <<mpp_paths>>:
[[mpp_paths]]
.Sending two parts of a multi-path payment
image::images/mpp_paths.png[]
image::images/mtln_1209.png[]
Because the S->X channel can now be utilized, and (luckily for Selena), the B->R channel has sufficient liquidity for 600k satoshis, the two parts are successful along paths that were previously not possible.
==== Trial-and-error over multiple "rounds"
==== Trial-And-Error Over Multiple "Rounds"
Multi-Path Payments lead to a somewhat modified "trial-and-error" loop for payment delivery. Because we are attempting multiple paths in each attempt, we have four possible outcomes:
@ -428,7 +428,7 @@ This multi-round example of sending a payment using MPP is shown in <<mpp_rounds
[[mpp_rounds]]
.Sending a payment in multiple rounds with MPP
image::images/mpp_rounds.png[]
image::images/mtln_1210.png[]
In the end, Selena's node used three rounds of path finding to send the 1m satoshis in 30 parts.
@ -9,13 +9,13 @@ to being able to write a custom wire protocol parser, a reader of this chapter
will gain a deep understanding with respect of the various upgrade mechanisms
that have been built into the protocol.
=== Messaging Layer in the Lightning protocol suite
=== Messaging Layer in the Lightning Protocol Suite
The messaging layer, which is detailed in this chapter, consists of _Message Framing and Format_, _Type Length Value (TLV)_ encoding, and _Feature Bits_. These components are highlighted by a double outline in the protocol suite, shown in <<LN_protocol_wire_message_highlight>>:
@ -206,14 +206,14 @@ integer follows one of the four following options:
4. Otherwise, we just encode the value as a full-size 64-bit integer.
==== TLV encoding constraints
==== TLV Encoding Constraints
Within the context of a TLV message, record types below `2^16` are said to be _reserved_ for future use. Types beyond this
range are to be used for "custom" message extensions used by higher-level application protocols.
The `value` of a record depends on the `type`. In other words, it can take any form as parsers will attempt to interpret it depending on the context of the type itself.
==== TLV canonical encoding
==== TLV Canonical Encoding
One issue with the Protobuf format is that encodings of the same message may
output an entirely different set of bytes when encoded by two different
@ -258,7 +258,7 @@ designers of the original Lightning Network knew that there were many possible f
extensibility mechanisms within the system, which can be used to upgrade it partially or fully in a decoupled, desynchronized, and decentralized
manner.
==== Feature Bits as an Upgrade Discoverability Mechanism
==== Feature Bits As an Upgrade Discoverability Mechanism
An astute reader may have noticed the various locations that "feature bits" are
included within the Lightning Protocol. A "feature bit" is a bitfield that can
@ -315,7 +315,7 @@ not. The feature bits within the `init` message allow peers to understand if
they can maintain a connection, and also which features are negotiated for the
lifetime of a given connection.
==== TLV for forwards & backwards compatibility
==== TLV for Forward and Backward Compatibility
As we learned earlier in the chapter, Type-Length-Value, or TLV records can be
used to extend messages in a forwards and backwards compatible manner.
@ -339,14 +339,14 @@ consensus in order to adopt. Instead, at minimum, only two peers within the
network need to understand a new upgrade in order to start using it. Commonly these two peers may be the recipient and sender of a
payment, or may be the channel partners of a new payment channel.
==== A taxonomy of upgrade mechanisms
==== A Taxonomy of Upgrade Mechanisms
Rather than there being a single widely utilized upgrade mechanism within the
network (such as soft-forks for Bitcoin), there exist several possible upgrade mechanisms within the Lighting Network. In this
section, we enumerate these upgrade mechanisms, and
provide a real-world example of their use in the past.
===== Internal Network Upgrades
===== Internal network upgrades
We start with the upgrade type that requires the most protocol-level
coordination: internal network upgrades. An internal network upgrade is
@ -370,7 +370,7 @@ network upgrade, in that if a payment wasn't using any new feature beyond
that new routing information encoding, then a payment could be transmitted
using a mixed set of nodes.
===== End-to-End Upgrades
===== End-to-end upgrades
To contrast the internal network upgrade, in this section we describe the
_end to end_ network upgrade. This upgrade mechanism differs from the internal
@ -134,7 +132,7 @@ Lightning Network, the flavor of the Noise protocol used is sometimes referred t
as "Brontide". A brontide is a low billowing noise, similar to what one would
hear during a thunderstorm when very far away.
=== Lightning Encrypted Transport in detail
=== Lightning Encrypted Transport in Detail
In this section we will break down the Lightning Encrypted Transport protocol and delve into the details of the cryptographic algorithms and protocol used to establish encrypted, authenticated and integrity-assured communications between peers. Feel free to skip this section if you find this level of detail daunting.
@ -144,7 +142,7 @@ The Noise protocol is extremely flexible in that it advertises several
handshakes, each with different security and privacy properties for a would be
protocol implementer to select from. A deep exploration of each of the
handshakes, and their various trade-offs is out of the scope of this chapter.
With that said, the Lighting Network uses a specific handshake referred to as
With that said, the Lightning Network uses a specific handshake referred to as
`Noise_XK`. The unique property provided by this handshake is "identity
hiding": in order for a node to initiate a connection with another node, it
must first know it's public key. Mechanically, this means that the public key
@ -284,7 +282,7 @@ handshake as:
* `a || b` denotes the concatenation of two byte strings `a` and `b`
===== Handshake Session State Initialization
===== Handshake session state initialization
Before starting the handshake process, both sides need to initialize the
starting state that they'll use to advance the handshake process. To start,
@ -316,7 +314,7 @@ does indeed know the public key of the responder.
In this chapter, we look at some of the most important issues related to the security and privacy of the Lightning Network. First, we'll consider privacy, what it means, how to evaluate it, and some things you can do to protect your own privacy while using LN. Then we'll explore some common attacks and mitigation techniques.
=== Why is privacy important?
=== Why Is Privacy Important?
The key value proposition of cryptocurrency is censorship resistant money. Bitcoin offers participants the possibility of storing and transferring their wealth without interference by governments, banks, or corporations.The Lightning Network continues this mission.
Unlike trivial scaling solutions like custodial Bitcoin banks, LN aims to scale Bitcoin without compromising on self-custody, which should lead to greater censorship resistance in the Bitcoin ecosystem. However, the LN operates under a different security model, which introduces novel security and privacy challenges.
=== Definitions of privacy
=== Definitions of Privacy
The question: "Is Lightning private?" has no direct answer. Privacy is a complex topic; it is often difficult to precisely define what we mean by privacy, particularly if you are not a privacy researcher. Fortunately, privacy researchers use processes to analyze and evaluate the privacy characteristics of systems and we can use them too! Let's look at how a security researcher might seek to answer the question, "Is Lightning private?" in two general steps.
First, a privacy researcher would define a _security model_ that specifies what an adversary is capable of and aims to achieve.
Then, they would describe the relevant properties of the system and check whether it conforms to the requirements.
==== Process to evaluate privacy
==== Process to Evaluate Privacy
A security model is based on a set of underlying _security assumptions_.
In cryptographic systems, these assumptions are often centered around the mathematical properties of the cryptographic primitives such as ciphers, signatures, and hash functions.
@ -61,7 +61,7 @@ Finally, as a payment network, the LN depends on economic stimuli.
The size and structure of fees affect the routing algorithm and, therefore, can either aid the attacker by forwarding most payments through their nodes or prevent this from happening.
=== Anonymity set
=== Anonymity Set
What does it mean to deanonymize someone?
In simple terms, deanonymization implies linking some action with a person's "real world" identity, such as their name or physical address.
@ -102,7 +102,7 @@ However, the real network leaks information that allows an attacker to narrow do
The smaller the anonymity set, the higher the chance of successful deanonymization.
=== Differences between LN and Bitcoin in terms of privacy
=== Differences Between LN and Bitcoin in Terms of Privacy
While it's true that transactions on the Bitcoin network do not associate real world identities with Bitcoin addresses, all transactions are broadcast in cleartext and can be analyzed.
Multiple companies have been established to deanonymize users of Bitcoin and other cryptocurrencies.
@ -126,7 +126,7 @@ Therefore, the anonymity set of a node in Lightning roughly equals its neighbors
[[anonymity_set]]
.The anonymity set of Alice and Bob constitutes their neighbors.
image::images/anon-set.png["The anonymity set of Alice and Bob constitutes their neighbors"]
image::images/mtln_1601.png["The anonymity set of Alice and Bob constitutes their neighbors"]
Similar logic applies to payment receivers.
Many users open only a handful of payment channels, therefore, limiting their anonymity sets.
@ -147,11 +147,11 @@ The issues discussed in this part of the chapter summarize research available in
Now let's review some of the attacks on the Lightning Network privacy that have been described in academic literature.
=== Attacks on Lightning
=== Attacks On Lightning
Recent research describes various ways in which the security and privacy of the LN may be compromised.
==== Observing payment amounts
==== Observing Payment Amounts
One of the goals for a privacy-preserving payment system is to hide the payment amount from uninvolved parties.
The Lightning Network is an improvement over layer-one in this regard.
@ -162,7 +162,7 @@ The availability of payment amounts to intermediary nodes do not present an imme
However, an _honest-but-curious_ intermediary node may use it as a part of a larger attack.
==== Linking senders and receivers
==== Linking Senders and Receivers
An attacker might be interested in learning the sender and/or the receiver of a payment to reveal certain economic relationships.
This breach of privacy could harm censorship resistance, as an intermediary node could censor payments to or from certain receivers or senders.
@ -203,7 +203,7 @@ Even more, as it was recently shown, an attacker can successfully determine the
Finally, it's important to recognize that unknown or unstudied leakages probably exist that could aid deanonymizing attempts. For instance, because different Lightning wallets apply different routing algorithms, even knowing the applied routing algorithm could help exclude certain nodes from being a sender and/or receiver of a payment.
==== Revealing channel balances (probing)
==== Revealing Channel Balances (Probing)
The balances of Lightning channels are supposed to be hidden for privacy and efficiency reasons.
A Lightning node only knows the balances of its adjacent channels.
@ -241,7 +241,7 @@ Note that in any case, Mallory's estimation becomes twice as precise after just
She can continue probing, choosing the next probing amount such that it divides the current estimation interval in half.
This well-known search technique is called _binary search_.
With binary search, the number of probes is _logarithmic_ in the desired precision.
For example, to obtain Alice's balance in a channel of 1 million satoshis up to a single satoshi, Mallory would only have to perform latexmath:[\log_2(1000000) \approx 20] probings.
For example, to obtain Alice's balance in a channel of 1 million satoshis up to a single satoshi, Mallory would only have to perform latexmath:[$\log_2(1000000) \approx 20$] probings.
If one probing takes 3 seconds, one channel can be precisely probed in only about a minute!
Channel probing can be made even more efficient.
@ -274,7 +274,8 @@ First, it is hard for an attacker to target unannounced channels.
Second, nodes that implement just-in-time or JIT routing may be less prone to the attack.
Finally, as multi-part payments make the problem of insufficient capacity less severe, the protocol developers may consider hiding some of the error details without harming efficiency.
==== Denial of service
[[denial_of_service]]
==== Denial of Service
When resources are made publicly available, there is a risk that attackers may attempt to make that resource unavailable by executing a denial-of-service attack.
Generally, this is achieved by the attacker bombarding a resource with requests, which are indistinguishable from legitimate queries.
@ -283,7 +284,7 @@ The attacks seldom result in the target suffering financial loss aside from the
Typical mitigations for denial-of-service attacks require authentication for requests to separate legitimate users from malicious ones or to incur a trivial cost to regular users but will act as a sufficient deterrent to an attacker launching requests at scale.
Anti-denial-of-service measures can be seen everywhere on the internet - websites apply rate limits to ensure that no one user can consume all of their server's attention, film review sites require login authentication to keep angry r/prequelmemes (Reddit group) members at bay, and data services sell API keys to limit the number of queries.
===== DoS in Bitcoin
===== Dos in bitcoin
In Bitcoin, the bandwidth that nodes use to relay transactions and the space that they avail to the network in the form of their mempool are publicly available resources.
Any node on the network can consume bandwidth and mempool space by sending a valid transaction.
@ -296,7 +297,7 @@ This measure largely ensured that the transactions that consume network resource
The minimum relay fee is acceptable to regular users but would hurt attackers financially if they tried to spam the network.
While some transactions may not make it into valid blocks during high fee environments, these measures have largely been effective at disincentivizing this type of spam.
===== DoS in Lightning
===== Dos in lightning
Similarly to Bitcoin, the Lightning Network charges fees for the use of its public resources, but in this case, the resources are public channels, and the fees come in the form of routing fees.
The ability to route payments through nodes in exchange for fees provides the network with a large scalability benefit - nodes that are not directly connected can still transact - but it comes at the cost of exposing a public resource that must be protected against DoS attacks.
@ -309,14 +310,14 @@ This is great for legitimate users, who wouldn't like to pay for failed attempts
At the time of writing, a discussion is https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-June/002734.html[ongoing] on the lightning-dev mailing list as to how best address this issue.
===== Known DoS attacks
===== Known dos attacks
There are two known DoS attacks on public Lightning channels which render a target channel, or a set of target channels, unusable.
Both attacks involve routing payments through a public channel, then holding them until their timeout, thus maximizing the attack's duration.
The requirement to fail payments to not pay fees is fairly simple to meet because malicious nodes can simply reroute payments to themselves.
In the absence of fees for failed payments, the only cost to the attacker is the on-chain cost of opening a channel to dispatch these payments through, which can be trivial in low fee environments.
==== Commitment jamming
==== Commitment Jamming
Lightning nodes update their shared state using asymmetric commitment transactions, on which HTLCs are added and removed to facilitate payments.
Each party is limited to a total of https://github.com/lightningnetwork/lightning-rfc/blob/c053ce7afb4cbf88615877a0d5fc7b8dbe2b9ba0/02-peer-protocol.md#the-open_channel-message[483] HTLCs in the commitment transaction at a time.
@ -325,13 +326,13 @@ A channel jamming attack allows an attacker to render a channel unusable by rout
It should be noted that this limit was chosen in the specification to ensure that all the HTLCs can be swept in a https://github.com/lightningnetwork/lightning-rfc/blob/master/05-onchain.md#penalty-transaction-weight-calculation[single justice transaction].
While this limit _may_ be increased, transactions are still limited by the block size, so the number of slots available is likely to remain limited.
==== Channel liquidity lockup
==== Channel Liquidity Lockup
A channel liquidity lockup attack is comparable to a channel jamming attack in that it routes payments through a channel and holds them so that the channel is unusable.
Rather than locking up slots on the channel commitment, this attack routes large HTLCs through a target channel that consumes all the channel's available bandwidth.
This attack's capital commitment is higher than the commitment jamming attack because the attacking node needs more funds to route failed payments through the target.
=== Cross-layer deanonymization
=== Cross-Layer Deanonymization
Computer networks are often layered.
Layering allows for separation of concerns and makes the whole system manageable.
@ -413,7 +414,7 @@ The possible deanonymization of Bitcoin entities illustrates how important it is
//TODO from author: maybe here we should/could include the corresponding figures from the Romiti et al. paper. it would greatly improve and help the understanding of the section
=== Lightning graph
=== Lightning Graph
The Lightning Network, as the name suggests, is a peer-to-peer network of payment channels.
Therefore, many of its properties (privacy, robustness, connectivity, routing efficiency) are influenced and characterized by its network nature.
@ -421,7 +422,7 @@ Therefore, many of its properties (privacy, robustness, connectivity, routing ef
In this section, we discuss and analyze the LN from the point of view of network science.
We are particularly interested in understanding the LN channel graph, its robustness, connectivity, and other important characteristics.
==== How does the Lightning graph look in reality?
==== How Does the Lightning Graph Look in Reality?
One could have expected that the LN is a random graph, where edges are randomly formed between nodes.
If this was the case, then the LN's degree distribution would follow a Gaussian normal distribution.
In particular, most of the nodes would have approximately the same degree, and we would not expect nodes with extraordinarily large degrees.
@ -469,7 +470,7 @@ To conclude, even by design, it is possible to remove edges and nodes from the r
However, depending on the utilized attack vector, the adversary may have to provide more or fewer resources to carry out the attack.
===== Temporality of the LN
===== Temporality of the ln
The LN is a dynamically changing, permissionless network.
Nodes can freely join or leave the network, they can open and create payment channels anytime they want.
@ -477,8 +478,8 @@ Therefore, a single static snapshot of the LN graph is misleading. We need to co
Its effective diameter is also shrinking; that is, nodes become closer to each other, as we can see in <<temporal_ln>>:
[[temporal_ln]]
.The steady growth of the LN in nodes, channels and locked capacity (as of Sep'2021)
image::images/LN_over_time_Sep2021.png["The steady growth of the LN in terms of nodes, channels and locked capacity (as of Sep'2021)"]
.The steady growth of the LN in nodes, channels, and locked capacity (as of September 2021)
image::images/mtln_1602.png["The steady growth of the LN in terms of nodes, channels and locked capacity (as of September 2021)"]
In social networks, triangle closing behavior is common.
Specifically, in a graph where nodes represent people and friendships are represented as edges, it is somewhat expected that triangles will emerge in the graph.
@ -504,7 +505,7 @@ In general, our understanding of the dynamic nature of the LN channel graph is r
It is fruitful to analyze how protocol changes like multi-part payments can affect the dynamics of the LN.
It would be beneficial to explore the temporal nature of the LN graph in more depth.
=== Economic incentives and graph structure
=== Economic Incentives and Graph Structure
The Lightning graph forms spontaneously and nodes connect to each other based on mutual interest.
As a result, incentives drive graph development.
@ -523,16 +524,16 @@ This might change in the future if LN has significantly larger traffic or if a m
On the other hand, if a node wishes to optimize its routing fees, it would minimize the average shortest path lengths to every other node.
Put differently, a profit-seeker node will try to locate itself in the _center_ of the channel graph or close to it.
=== Practical advice for users to protect their privacy
=== Practical Advice for Users to Protect Their Privacy
We're still in the early stages of the Lightning Network.
Many of the concerns listed in this chapter are likely to be addressed as it matures and grows.
In the meantime, there are some measures that you can take to guard your node against malicious users; something as simple as updating the default parameters that your node runs with can go a long way in hardening your node.
== Unannounced channels
=== Unannounced Channels
If you intend to use the Lightning Network to send and receive funds between nodes and wallets you control, and have no interest in routing other users' payments, there is little need to announce your channels to the rest of the network.
You could open a channel between, say, your desktop PC running a full node and your mobile phone running a Lightning Wallet and simply forgo the channel announcement discussed in Chapter 3.
You could open a channel between, say, your desktop PC running a full node and your mobile phone running a Lightning Wallet and simply forgo the channel announcement discussed in <<ch03_How_Lightning_Works>>.
These are sometimes called "private" channels, however it is more correct to refer to them as "unannounced" channels as they are not strictly private.
Unannounced channels will not be known to the rest of the network and won't normally be used to route other users' payments.
@ -554,7 +555,8 @@ However, since the opening and commitment transactions are pseudonymous it will
In addition, the Taproot update of 2021 makes it difficult to distinguish between channel opening and closing transactions and other specific kinds of Bitcoin transactions.
Hence, while unannouned channels are not completely private, they do provide some privacy benefits when used carefully.
=== Routing considerations
[[routing_considerations]]
=== Routing Considerations
As covered in the <<denial_of_service>> section, nodes that open public channels expose themselves to the risk of a series of attacks on their channels.
While mitigations are being developed on the protocol level, there are many steps that a node can take to protect against denial of service attacks on their public channels.
@ -568,7 +570,7 @@ Some useful guidelines for a custom rate limiter are as follows:
* Shadow channels: nodes that wish to open large channels to a single target can instead open a single public channel to the target and support it with further private channels called [shadow channels](joost on TFTC).
These channels can still be used for routing but are not announced to potential attackers.
==== Accepting channels
==== Accepting Channels
At present, Lightning nodes struggle with bootstrapping inbound liquidity. While there are some paid
solutions to acquiring inbound liquidity, like swap services, channel markets, and paid channel opening services from known hubs, many nodes will gladly accept any legitimate looking channel opening request to increase their inbound liquidity.
@ -591,12 +593,12 @@ Some potential strategies are:
* No risk: do not accept any incoming channels.
* Low risk: accept channels from a known set of nodes that you have previously had successful channels open with
* Medium risk: only accept channels from nodes that have been present in the graph for a longer period, and have some long-lived channels.
* Higher risk: accept any incoming channels, and implement the mitigations described in the <<routing_considerations>> chapter.
* Higher risk: accept any incoming channels, and implement the mitigations described in <<routing_considerations>>.
=== Conclusion
In summary, privacy and security are nuanced, complex topics and while many researchers and developers are looking for network-wide improvements, it's important for everyone participating in the network to understand what they can do to protect their own privacy and increase security on an individual node level.
=== References and further reading
=== References and Further Reading
In this chapter, we used many references from ongoing research on Lightning security. You may find these useful articles and papers listed by topic below:
In just a few years, the Lightning Network has gone from a whitepaper to a rapidly growing global network. As Bitcoin's second layer it has delivered on the promise of fast, inexpensive, and private payments. Additionally, it has started a tsunami of innovation, as it unleashes developers from the constraints of lock-step consensus that exist in Bitcoin development.
Innovation in the Lightning Network is happening in several different levels:
* At Bitcoin's Core protocol, providing use and demand for new Bitcoin Script opcodes, signing algorithms, and optimizations
* At the Lightning protocol level with new features deployed rapidly network-wide.
* At the payment channel level, with new channel constructs and enhancements
* As distinct opt-in features deployed end-to-end by independent implementations that senders and recipients can use if they want.
* With new and exciting Lightning Applications (LApps) build on top of the clients and protocols.
Let's look at how these innovations are changing Lightning now and in the near future.
=== Decentralized and asynchronous innovation
Lightning isn't bound by lock-step consensus, as is the case with Bitcoin. That means that different Lightning clients can implement different features and negotiate their interactions (see <<feature_bits>>). As a result, innovation in the Lightning Network is occurring at a much faster rate than in Bitcoin.
Not only is Lightning advancing rapidly, but it is creating demand for new features in the Bitcoin system. Many recent and planned innovations in Bitcoin are both motivated and justified by their use in the Lightning Network. In fact, the Lightning Network is often mentioned as an example use-case for many of the new features.
==== Bitcoin protocol and Bitcoin Script innovation
The Bitcoin system is, by necessity, a conservative system that has to preserve compatibility with consensus rules to avoid unplanned forks of the blockchain or partitions of the P2P network. As a result, new features require a lot of coordination and testing before they are implemented in "mainnet", the live production system.
Here are some of the current or proposed innovations spurred by Lightning:
Neutrino:: A lightweight client protocol with improved privacy features over the legacy SPV protocol. Neutrino is mostly used by Lightning clients to access the Bitcoin blockchain.
Schnorr signatures:: Introduced as part of the _Taproot_ soft fork, Schnorr signatures will enable flexible Point Time-Locked Contracts (PTLC) for channel construction in Lightning. This might in particular make use of revocable signatures instead of revocable transactions.
Taproot:: Also part of the November 2021 soft-fork that introduces Schnorr signatures, Taproot allows complex scripts to appear as single-payer, single-payee payments, and indistinguishable from the most common type of payment on Bitcoin. This will allow Lightning channel cooperative (mutual) closure transactions to appear indistinguishable from simple payments and increase privacy for LN users.
Input re-binding:: Also known by the names SIGHAS_NOINPUT, or SIGHASH_ANYPREVOUT, this planned upgrade to the Bitcoin Script language is primarily motivated by advanced smart contracts such as the eltoo channel protocol.
Covenants:: Currently in the early stages of research, coventants allow transactions to create outputs that constrain future transactions which spend them. This mechanism could increase security for Lightning channels, by making it possible to enforce address whitelisting in commitment transactions.
==== Lightning protocol innovation
The Lightning P2P protocol is highly extensible and has undergone a lot of change since its inception. The "It's OK to be odd" rule used in feature bits (see <<feature_bits>>) ensure that nodes can negotiate the features they support, enabling multiple independent upgrades to the protocol.
===== TLV extensibility
The Type-Length-Value (see <<tlv>>) mechanism for extending the messaging protocol is extremely powerful and has already enabled the introduction of several new capabilities in Lightning while maintaining both forward and backward compatibility.
A prominent example that is currently being developed and makes use of this is path blinding and trampoline payments which allows a recipient to hide itself from the sender but also allows mobile clients to send payments without the necessity to store the full channel graph on their devices by using a third party to which they don't need to reveal the final recipient.
==== Payment channel construction
Payment channels are an abstraction that is operated by two channel partners. As long as those two are willing to run new code, they can implement a variety of channel mechanisms simultaneously. In fact, recent research suggests that channels could even be upgraded to a new mechanism dynamically, without closing the old channel and opening a new channel type.
Eltoo:: A proposed channel mechanism that uses input-rebinding to significantly simplify the operation of payment channels and remove the need for the penalty mechanism. It needs a new Bitcoin signature type before it can be implemented
==== Opt-in end-to-end features
Point Time-Locked Contracts (PTLCs):: A different approach to HTLCs, PTLCs can increase privacy, reduce information leaked to intermediary nodes and operate more efficiently than HTLC-based channels.
Large channels:: Large or "Wumbo" channels were introduced in a dynamic way to the network without requiring coordination. Channels that support large payments are advertized as part of the channel announcement messages and can be used in an opt-in manner.
Multi-Part Payments (MPP):: MPP was also introduced in an opt-in manner, but even better only requires the sender and recipient of a payment to be able to do MPP. The rest of the network simply routes HTLCs as if they are single-part payments.
JIT-Routing:: An optional method that can be used by routing nodes to increase their reliability and to protect themsleves from being spammed.
Keysend:: An upgrade introduced independently by Lightning client implementations, it allows the sender to send money in an "unsolicited" and asynchronous way without requiring an invoice first.
HODL invoices:: Payments where the final HTLC is not collected, committing the sender to the payment, but allowing the recipient to delay collection until some other condition is satisfied, or cancel the invoice without collection. This was also implemented independently by different Lightning clients and can be used opt-in.
Onion routed message services:: The onion routing mechanism, and the underlying public key databse of nodes can be used to send data that is unrelated to payments, such as text messages or forum posts. The use of Lightning to enable paid messaging as a solution to spam posts and sybil attacks (spam) is another innovation that was implemented independently of the core protocol.
Offers:: Currently Proposed as BOLT #12 but already implemented by some nodes this is a communication protocol to request (recurring) invoices from remote nodes via Onion messages.
[[lapps]]
=== Lightning Applications (LApps)
While still in their infancy, we are already seeing the emergence of interesting Lightning Applications. Broadly defined as an application that uses the Lightning Protocol or a Lightning client as a component, LApps are the application layer of Lightning.
A prominent example are LNURL that provides a similar functionality as BOLT #12 Offers but just over http and Lightning address that works on top of offers to provide users with an email style address to which others can send funds while the software in the background requests an invoice against the LNURL endpoint of the node.
Further LApps are being built for simple games, messaging applications, micro-services, payable-APIs, paid dispensers (eg. fuel pumps), derivative trading systems, and much more.
=== Ready, set, go!
The future is looking bright. The Lightning Network is taking Bitcoin to new unexplored markets and applications. Equipped with the knowledge in this book, you can explore this new frontier, or maybe even join as a pioneer and forge a new path.
[[conclusion_chapter]]
== Conclusion
In just a few years, the Lightning Network has gone from a whitepaper to a rapidly growing global network. As Bitcoin's second layer it has delivered on the promise of fast, inexpensive, and private payments. Additionally, it has started a tsunami of innovation, as it unleashes developers from the constraints of lock-step consensus that exist in Bitcoin development.
Innovation in the Lightning Network is happening in several different levels:
* At Bitcoin's Core protocol, providing use and demand for new Bitcoin Script opcodes, signing algorithms, and optimizations
* At the Lightning protocol level with new features deployed rapidly network-wide.
* At the payment channel level, with new channel constructs and enhancements
* As distinct opt-in features deployed end-to-end by independent implementations that senders and recipients can use if they want.
* With new and exciting Lightning Applications (LApps) build on top of the clients and protocols.
Let's look at how these innovations are changing Lightning now and in the near future.
=== Decentralized and Asynchronous Innovation
Lightning isn't bound by lock-step consensus, as is the case with Bitcoin. That means that different Lightning clients can implement different features and negotiate their interactions (see <<feature_bits>>). As a result, innovation in the Lightning Network is occurring at a much faster rate than in Bitcoin.
Not only is Lightning advancing rapidly, but it is creating demand for new features in the Bitcoin system. Many recent and planned innovations in Bitcoin are both motivated and justified by their use in the Lightning Network. In fact, the Lightning Network is often mentioned as an example use-case for many of the new features.
==== Bitcoin Protocol and Bitcoin Script Innovation
The Bitcoin system is, by necessity, a conservative system that has to preserve compatibility with consensus rules to avoid unplanned forks of the blockchain or partitions of the P2P network. As a result, new features require a lot of coordination and testing before they are implemented in "mainnet", the live production system.
Here are some of the current or proposed innovations spurred by Lightning:
Neutrino:: A lightweight client protocol with improved privacy features over the legacy SPV protocol. Neutrino is mostly used by Lightning clients to access the Bitcoin blockchain.
Schnorr signatures:: Introduced as part of the _Taproot_ soft fork, Schnorr signatures will enable flexible Point Time-Locked Contracts (PTLC) for channel construction in Lightning. This might in particular make use of revocable signatures instead of revocable transactions.
Taproot:: Also part of the November 2021 soft-fork that introduces Schnorr signatures, Taproot allows complex scripts to appear as single-payer, single-payee payments, and indistinguishable from the most common type of payment on Bitcoin. This will allow Lightning channel cooperative (mutual) closure transactions to appear indistinguishable from simple payments and increase privacy for LN users.
Input re-binding:: Also known by the names SIGHAS_NOINPUT, or SIGHASH_ANYPREVOUT, this planned upgrade to the Bitcoin Script language is primarily motivated by advanced smart contracts such as the eltoo channel protocol.
Covenants:: Currently in the early stages of research, coventants allow transactions to create outputs that constrain future transactions which spend them. This mechanism could increase security for Lightning channels, by making it possible to enforce address whitelisting in commitment transactions.
==== Lightning Protocol Innovation
The Lightning P2P protocol is highly extensible and has undergone a lot of change since its inception. The "It's OK to be odd" rule used in feature bits (see <<feature_bits>>) ensure that nodes can negotiate the features they support, enabling multiple independent upgrades to the protocol.
===== Tlv extensibility
The Type-Length-Value (see <<tlv>>) mechanism for extending the messaging protocol is extremely powerful and has already enabled the introduction of several new capabilities in Lightning while maintaining both forward and backward compatibility.
A prominent example that is currently being developed and makes use of this is path blinding and trampoline payments which allows a recipient to hide itself from the sender but also allows mobile clients to send payments without the necessity to store the full channel graph on their devices by using a third party to which they don't need to reveal the final recipient.
==== Payment Channel Construction
Payment channels are an abstraction that is operated by two channel partners. As long as those two are willing to run new code, they can implement a variety of channel mechanisms simultaneously. In fact, recent research suggests that channels could even be upgraded to a new mechanism dynamically, without closing the old channel and opening a new channel type.
Eltoo:: A proposed channel mechanism that uses input-rebinding to significantly simplify the operation of payment channels and remove the need for the penalty mechanism. It needs a new Bitcoin signature type before it can be implemented
==== Opt-In End-to-End Features
Point Time-Locked Contracts (PTLCs):: A different approach to HTLCs, PTLCs can increase privacy, reduce information leaked to intermediary nodes and operate more efficiently than HTLC-based channels.
Large channels:: Large or "Wumbo" channels were introduced in a dynamic way to the network without requiring coordination. Channels that support large payments are advertized as part of the channel announcement messages and can be used in an opt-in manner.
Multi-Part Payments (MPP):: MPP was also introduced in an opt-in manner, but even better only requires the sender and recipient of a payment to be able to do MPP. The rest of the network simply routes HTLCs as if they are single-part payments.
JIT-Routing:: An optional method that can be used by routing nodes to increase their reliability and to protect themsleves from being spammed.
Keysend:: An upgrade introduced independently by Lightning client implementations, it allows the sender to send money in an "unsolicited" and asynchronous way without requiring an invoice first.
HODL invoices:: Payments where the final HTLC is not collected, committing the sender to the payment, but allowing the recipient to delay collection until some other condition is satisfied, or cancel the invoice without collection. This was also implemented independently by different Lightning clients and can be used opt-in.
Onion routed message services:: The onion routing mechanism, and the underlying public key databse of nodes can be used to send data that is unrelated to payments, such as text messages or forum posts. The use of Lightning to enable paid messaging as a solution to spam posts and sybil attacks (spam) is another innovation that was implemented independently of the core protocol.
Offers:: Currently Proposed as BOLT #12 but already implemented by some nodes this is a communication protocol to request (recurring) invoices from remote nodes via Onion messages.
[[lapps]]
=== Lightning Applications (LApps)
While still in their infancy, we are already seeing the emergence of interesting Lightning Applications. Broadly defined as an application that uses the Lightning Protocol or a Lightning client as a component, LApps are the application layer of Lightning.
A prominent example are LNURL that provides a similar functionality as BOLT #12 Offers but just over http and Lightning address that works on top of offers to provide users with an email style address to which others can send funds while the software in the background requests an invoice against the LNURL endpoint of the node.
Further LApps are being built for simple games, messaging applications, micro-services, payable-APIs, paid dispensers (eg. fuel pumps), derivative trading systems, and much more.
=== Ready, Set, Go!
The future is looking bright. The Lightning Network is taking Bitcoin to new unexplored markets and applications. Equipped with the knowledge in this book, you can explore this new frontier, or maybe even join as a pioneer and forge a new path.
@ -21,17 +21,17 @@ This chapter covers several important concepts from Bitcoin, including:
* Timelocks
=== Keys and digital signatures
=== Keys and Digital Signatures
((("cryptography", "defined")))((("cryptography", see="also keys and addresses")))You may have heard that bitcoin is based on _cryptography_, which is a branch of mathematics used extensively in computer security. Cryptography can also be used to prove knowledge of a secret without revealing that secret (digital signature), or prove the authenticity of data (digital fingerprint). These types of cryptographic proofs are the mathematical tools critical to bitcoin and used extensively in bitcoin applications.
You may have heard that bitcoin is based on _cryptography_, which is a branch of mathematics used extensively in computer security. Cryptography can also be used to prove knowledge of a secret without revealing that secret (digital signature), or prove the authenticity of data (digital fingerprint). These types of cryptographic proofs are the mathematical tools critical to bitcoin and used extensively in bitcoin applications.
((("digital keys", see="keys and addresses")))((("keys and addresses", "overview of")))((("digital signatures", "purpose of")))Ownership of bitcoin is established through _digital keys_, _bitcoin addresses_, and _digital signatures_. The digital keys are not actually stored in the network, but are instead created and stored by users in a file, or simple database, called a _wallet_. The digital keys in a user's wallet are completely independent of the bitcoin protocol and can be generated and managed by the user's wallet software without reference to the blockchain or access to the internet.
Ownership of bitcoin is established through _digital keys_, _bitcoin addresses_, and _digital signatures_. The digital keys are not actually stored in the network, but are instead created and stored by users in a file, or simple database, called a _wallet_. The digital keys in a user's wallet are completely independent of the bitcoin protocol and can be generated and managed by the user's wallet software without reference to the blockchain or access to the internet.
Most bitcoin transactions require a valid digital signature to be included in the blockchain, which can only be generated with a secret key; therefore, anyone with a copy of that key has control of the bitcoin. ((("witnesses")))The digital signature used to spend funds is also referred to as a _witness_, a term used in cryptography. The witness data in a bitcoin transaction testifies to the true ownership of the funds being spent. ((("public and private keys", "key pairs")))((("public and private keys", see="also keys and addresses")))Keys come in pairs consisting of a private (secret) key and a public key. Think of the public key as similar to a bank account number and the private key as similar to the secret PIN.
Most bitcoin transactions require a valid digital signature to be included in the blockchain, which can only be generated with a secret key; therefore, anyone with a copy of that key has control of the bitcoin. The digital signature used to spend funds is also referred to as a _witness_, a term used in cryptography. The witness data in a bitcoin transaction testifies to the true ownership of the funds being spent. Keys come in pairs consisting of a private (secret) key and a public key. Think of the public key as similar to a bank account number and the private key as similar to the secret PIN.
==== Private and public keys
==== Private and Public Keys
((("keys and addresses", "overview of", "private key generation")))((("warnings and cautions", "private key protection")))A private key is simply a number, picked at random. In practice, and to make managing many keys easy, most bitcoin wallets generate a sequence of private keys from a single random _seed_, using a deterministic derivation algorithm. Simply put, a single random number is used to produce a repeatable sequence of seemingly random numbers that are used as private keys. This allows users to only backup the seed and be able to _derive_ all the keys they need from that seed.
A private key is simply a number, picked at random. In practice, and to make managing many keys easy, most bitcoin wallets generate a sequence of private keys from a single random _seed_, using a deterministic derivation algorithm. Simply put, a single random number is used to produce a repeatable sequence of seemingly random numbers that are used as private keys. This allows users to only backup the seed and be able to _derive_ all the keys they need from that seed.
Bitcoin, like many other cryptocurrencies and blockchains, uses _elliptic curves_ for security. In Bitcoin, elliptic curve multiplication on the _secp256k1_ elliptic curve is used as a _one-way function_. Simply put, the nature of elliptic curve math makes it trivial to calculate the scalar multiplication of a point but impossible to calculate the inverse ("division", or "discrete logarithm").
@ -51,7 +51,7 @@ A hash function, also known as a _digest function_, is a function that takes arb
For example, if we use a command-line terminal to feed the text "Mastering the Lightning Network" into the SHA256 function it will produce a fingerprint as follows:
@ -107,7 +107,7 @@ Proof-of-Work/Hash Grinding:: You can use a hash to prove you have done computat
Atomicity:: You can make a secret pre-image a pre-requisite of spending funds in several linked transactions. If any one of the parties reveals the pre-image in order to spend one of the transactions, all the other parties can now spend their transactions too. All or none become spendable, achieving atomicity across several transactions.
==== Digital signatures
==== Digital Signatures
The private key is used to create signatures that are required to spend bitcoin by proving ownership of funds used in a transaction.
@ -115,11 +115,11 @@ A digital signature is a number that is calculated from the application of the p
Given a message m and a private key k, a signature function F_sig_ can produce a signature S:
latexmath:[ S = F{sign}(m, k) ]
latexmath:[$ S = F{sign}(m, k) $]
This signature S can be independently verified by anyone who has the public key K (corresponding to private key k), and the message:
latexmath:[ F{verify}(m, K, S) ]
latexmath:[$ F{verify}(m, K, S) $]
If F_verify_ returns a true result, then the verifier can confirm that the message m was signed by someone who had access to the private key k. Importantly, the digital signature proves the possession of the private key k at the time of signing, without revealing k.
@ -133,18 +133,18 @@ Signatures are not always applied to the entire transaction. To provide signing
More information about signature hash types can be found in https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch06.asciidoc#sighash_types[_Mastering Bitcoin Second Edition - Chapter 6 - Signature Hash Types_]
=== Bitcoin transactions
=== Bitcoin Transactions
Transactions are data structures that encode the transfer of value between participants in the bitcoin system.
[[utxo]]
==== Inputs and outputs
==== Inputs and Outputs
The fundamental building block of a bitcoin transaction is a transaction output. Transaction outputs are indivisible chunks of bitcoin currency, recorded on the blockchain, and recognized as valid by the entire network. A transaction spends "inputs" and creates "outputs". Transaction inputs are simply references to outputs of previously recorded transactions. This way, each transaction spends the outputs of previous transactions and creates new outputs.
[[transaction_structure]]
.A transaction transfers value from inputs to outputs
image::images/tx1.png["transaction inputs and outputs"]
image::images/mtln_aa02.png["transaction inputs and outputs"]
Bitcoin full nodes track all available and spendable outputs, known as _unspent transaction outputs_, or UTXOs. The collection of all UTXOs is known as the UTXO set and currently numbers in the millions of UTXOs. The UTXO set grows as new UTXOs are created and shrinks when UTXOs are consumed. Every transaction represents a change (state transition) in the UTXO set, by consuming one or more UTXOs as _transaction inputs_ and creating one or more UTXOs as its _transaction outputs_.
@ -152,7 +152,7 @@ For example, let's assume that a user Alice has a 100,000 satoshi UTXO that she
[[alice_100ksat_to_bob]]
.Alice pays 100,000 satoshis to Bob
image::images/tx2.png["Alice pays 100,000 satoshis to Bob"]
image::images/mtln_aa03.png["Alice pays 100,000 satoshis to Bob"]
A transaction output can have an arbitrary (integer) value denominated in satoshis. Just as dollars can be divided down to two decimal places as cents, bitcoin can be divided down to eight decimal places as satoshis. Although an output can have any arbitrary value, once created it is indivisible. This is an important characteristic of outputs that needs to be emphasized: outputs are discrete and indivisible units of value, denominated in integer satoshis. An unspent output can only be consumed in its entirety by a transaction.
@ -160,7 +160,7 @@ So what if Alice wants to pay Bob 50,000 satoshi, but only has an indivisible 10
[[alice_50ksat_to_bob_change]]
.Alice pays 50k sat to Bob and 50k sat to herself as change
image::images/tx3.png["Alice pays 50,000 satoshis to Bob and 50,000 satoshis to herself as change"]
image::images/mtln_aa04.png["Alice pays 50,000 satoshis to Bob and 50,000 satoshis to herself as change"]
[TIP]
====
@ -171,7 +171,7 @@ Similarly, if Alice wants to pay Bob 85,000 satoshi but has two 50,000 satoshi U
[[tx_twoin_twoout]]
.Alice uses two 50k inputs to pay 85k sat to Bob and 50k sat to herself as change
image::images/tx4.png["Alice uses two 50k inputs to pay 85k sat to Bob and 50k sat to herself as change"]
image::images/mtln_aa05.png["Alice uses two 50k inputs to pay 85k sat to Bob and 50k sat to herself as change"]
The illustrations and examples above show how a Bitcoin transaction combines (spends) one or more inputs and creates one or more outputs. A transaction can have hundreds or even thousands of inputs and outputs.
@ -180,13 +180,13 @@ The illustrations and examples above show how a Bitcoin transaction combines (sp
While the transactions created by the Lightning Network have multiple outputs they do not have "change" per se, because the entire available balance of a channel is split between the two channel partners.
====
==== Transaction chains
==== Transaction Chains
Every output can be spent, as an input in a subsequent transaction. So for example, if Bob decided to spend 10,000 satoshi in a transaction paying Chan, and Chan spend 4,000 satoshi to pay Dina:
[[tx_chain]]
.Alice pays Bob who pays Chan who pays Dina
image::images/tx5.png["Alice pays Bob who pays Chan who pays Dina"]
image::images/mtln_aa06.png["Alice pays Bob who pays Chan who pays Dina"]
An output is considered "spent" if it is referenced as an input in another transaction that is recorded on the blockchain. An output is considered "unspent" (and available for spending) if no recorded transaction references it.
@ -194,7 +194,7 @@ The only type of transaction that doesn't have "inputs" is a special transaction
Since transactions are chained, if you pick a transaction at random, you can follow any one of its inputs backwards to the previous transaction that created it. If you keep doing that you will eventually reach a coinbase transaction where the bitcoin was first mined.
==== TxID: Transaction identifiers
==== TxID: Transaction Identifiers
Every transaction in the Bitcoin system is identified by a unique identifier (assuming the existence of BIP-0030), called the _transaction ID_ or _TxID_ for short. To produce a unique identifier, we use the SHA-256 cryptographic hash function to produce a hash of the transaction's data. This "fingerprint" serves as a universal identifier. A transaction can be referenced by its transaction ID and once a transaction is recorded on the Bitcoin blockchain, every node in the Bitcoin network knows that this transaction is valid.
@ -215,7 +215,7 @@ or use the short link (case sensitive):
http://bit.ly/AliceTx
==== Outpoints: output identifiers
==== Outpoints: Output Identifiers
As every transaction has a unique ID, we can also identify a transaction output within that transaction uniquely by reference to the TxID and the output index number. The first output in a transaction is output index 0, the second output is output index 1 and so on. An output identifier is commonly known as an _outpoint_.
@ -232,7 +232,7 @@ Here's the chain of transactions from Alice to Bob to Chan to Dina, this time wi
[[tx_chain_vout]]
.Transaction inputs refer to outpoints forming a chain
image::images/tx6.png["Transaction inputs refer to outpoints forming a chain"]
image::images/mtln_aa07.png["Transaction inputs refer to outpoints forming a chain"]
The input in Bob's transaction references Alice's transaction (by TxID) and the 0 indexed output.
@ -258,7 +258,9 @@ Let's look at a very simple example of Bitcoin Script, which adds the numbers 2
In the diagram below, we see how this script is executed (from left to right):
image::images/bitcoin-script-example-1.png["Example of Bitcoin Script execution"]
[[figa08]]
.Caption here
image::images/mtln_aa08.png["Example of Bitcoin Script execution"]
==== Locking and Unlocking Scripts
@ -272,7 +274,7 @@ Using a simplified model, for validation, the unlocking script and locking scrip
Obviously, this simplified example would make a very poor choice for locking an actual Bitcoin output because there is no secret, just basic arithmetic. Anyone could spend the output by providing the answer "2". Most locking scripts therefore require demonstrating knowledge of a secret.
==== Locking to a public key (signature)
==== Locking to a Public Key (Signature)
The simplest form of a locking script that requires a signature. Let's consider Alice's transaction that pays Bob 50,000 satoshis. The output Alice creates to pay Bob will have a locking script requiring Bob's signature and would look like this:
@ -296,7 +298,7 @@ In <<locking_unlocking_chain>> you can see the locking script in Alice's transac
[[locking_unlocking_chain]]
.A transaction chain showing the locking script (output) and unlocking script (input)
image::images/locking-unlocking-chain.png["A transaction chain showing the locking script (output) and unlocking script (input)"]
image::images/mtln_aa09.png["A transaction chain showing the locking script (output) and unlocking script (input)"]
To validate Bob's transaction, a Bitcoin node would do the following:
@ -307,7 +309,7 @@ To validate Bob's transaction, a Bitcoin node would do the following:
* Execute this script on the Bitcoin Script execution engine to see what result is produced
* If the result is +TRUE+, deduce that Bob's transaction is valid because it was able to fulfill the spending condition to spend that outpoint.
==== Locking to a hash (secret)
==== Locking to a Hash (Secret)
Another type of locking script, one that is used in the Lightning Network, is a _hash lock_. In order to unlock it you must know the secret _pre-image_ to the hash.
@ -352,7 +354,7 @@ Only Bob knows +R+, so only Bob can produce a transaction with an unlocking scri
Interestingly, Bob can give the +R+ value to anyone else, who can then spend that Bitcoin. This makes the secret value +R+ almost like a bitcoin "voucher", since anyone who has it can spend the output Alice created. We'll see how this is a useful property for the Lightning Network!
[[multisig]]
==== Multisignature scripts
==== Multisignature Scripts
The Bitcoin scripting language provides a multisignature building block (primitive), that can be used to build escrow services and complex ownership configurations between several stakeholders. An arrangement that requires multiple signatures to spend Bitcoin is called a _multisignature scheme_, further specified as an _K-of-N_ scheme, where:
@ -386,7 +388,7 @@ The two scripts together would form the combined validation script:
A multisignature locking script can be represented by a Bitcoin address, encoding the hash of the locking script. For example, the initial funding transaction of a Lightning payment channel is a transaction that pays to an address that encodes a locking script of a 2-of-2 multisig of the two channel partners.
==== Timelock scripts
==== Timelock Scripts
Another important building block that exists in Bitcoin and is used extensively in the Lightning Network is a _timelock_. A timelock is a restriction on spending that requires that a certain time or block height has elapsed before spending is allowed. It is a bit like a post-dated check drawn from a bank account that can't be "cashed" before the date on the check.
@ -401,7 +403,7 @@ Output-level absolute timelocks are implemented by the operator +CHECKLOCKTIMEVE
Output-level relative timelocks are implemented by the operator +CHECKSEQUENCEVERIFY+, often shortened in conversation as _CSV_. Relative timelocks implement a spending constraint that is relative to the confirmation of the transaction, expressing the equivalent of "can't be spent until 1024 blocks after confirmation".
[[conditional_scripts]]
==== Scripts with multiple conditions
==== Scripts with Multiple Conditions
One of the more powerful features of Bitcoin Script is flow control, also known as conditional clauses. You are probably familiar with flow control in various programming languages that use the construct +IF...THEN...ELSE+. Bitcoin conditional clauses look a bit different, but are essentially the same construct.
@ -411,7 +413,7 @@ Additionally, bitcoin conditional expressions can be "nested" indefinitely, mean
Bitcoin implements flow control using the +IF+, +ELSE+, +ENDIF+, and +NOTIF+ opcodes. Additionally, conditional expressions can contain boolean operators such as +BOOLAND+, +BOOLOR+, and +NOT+.
At first glance, you may find the bitcoin's flow control scripts confusing. That is because Bitcoin Script is a stack language. The same way that the arithmetic operation latexmath:[1 + 1] looks "backward" when expressed in Bitcoin Script as +1 1 ADD+, flow control clauses in
At first glance, you may find the bitcoin's flow control scripts confusing. That is because Bitcoin Script is a stack language. The same way that the arithmetic operation latexmath:[$1 + 1$] looks "backward" when expressed in Bitcoin Script as +1 1 ADD+, flow control clauses in
Bitcoin also look "backward."
In most traditional (procedural) programming languages, flow control looks like this:
@ -444,7 +446,7 @@ When reading Bitcoin Script, remember that the condition being evaluated comes _
A very common use for flow control in Bitcoin Script is to construct a locking script that offers multiple execution paths, each a different way of redeeming the UTXO.
((("use cases", "buying coffee")))Let's look at a simple example, where we have two signers, Alice and Bob, and either one is able to redeem. With multisig, this would be expressed as a 1-of-2 multisig script. For the sake of demonstration, we will do the same thing with an +IF+ clause:
Let's look at a simple example, where we have two signers, Alice and Bob, and either one is able to redeem. With multisig, this would be expressed as a 1-of-2 multisig script. For the sake of demonstration, we will do the same thing with an +IF+ clause:
----
IF
@ -481,6 +483,7 @@ In <<htlc_script_example>> you can see an example of the kind of complex script
Images of the _Gaia_ Bitcoin ATM (https://lamassu.is/product/gaia) seen in [[bitcoin-atm]] are used with permission of Lamassu Industries AG. The use of these images is not an endorsement of the product or company, but are provided as a visual example of a Bitcoin ATM.
Images of the _Gaia_ Bitcoin ATM (https://lamassu.is/product/gaia) seen in <<bitcoin-atm>> are used with permission of Lamassu Industries AG. The use of these images is not an endorsement of the product or company, but are provided as a visual example of a Bitcoin ATM.
The structure of the `init` message is defined as follows:
[[apdx_init_message]]
===== The `init` message
===== The init message
* type: *16*
* fields:
@ -151,7 +151,7 @@ broadcast the latest signed commitment)
The sole message in this category is the `error` message:
[[apdx_error_message]]
===== The `error` message
===== The error message
* type: *17*
* fields:
@ -178,7 +178,7 @@ transport being used to transmit the messages, a set of protocol level `ping`
and `pong` messages are defined.
[[apdx_ping_message]]
===== The `ping` message
===== The ping message
* type: *18*
* fields:
@ -189,7 +189,7 @@ and `pong` messages are defined.
Next it's companion, the `pong` message.
[[apdx_pong_message]]
===== The `pong` message
===== The pong message
* type: *19*
* fields:
@ -230,7 +230,7 @@ set of 5 messages: `open_channel`, `accept_channel`, `funding_created`,
The detailed protocol flow using these messages is described in <<payment_channels>>.
[[apdx_open_channel_message]]
===== The `open_channel` message
===== The open_channel message
* type: *32*
* fields:
@ -282,7 +282,7 @@ commonly referred to as a "private" channel.
The `accept_channel` message is the response to the `open_channel` message:
[[apdx_accept_channel_message]]
===== The `accept_channel` message
===== The accept_channel message
* type: *33*
* fields:
@ -313,7 +313,7 @@ channel.
In response, the initiator will send the `funding_created` message.
[[apdx_funding_created_message]]
===== The `funding_created` message
===== The funding_created message
* type: *34*
* fields:
@ -333,7 +333,7 @@ initiator, only needs to send the funding outpoint of the channel.
To conclude the responder sends the `funding_signed` message.
[[apdx_funding_signed_message]]
===== The `funding_signed` message
===== The funding_signed message
* type: *34*
* fields:
@ -356,7 +356,7 @@ Once the funding transaction has received enough confirmations, the
`funding_locked` is sent.
[[apdx_funding_locked_message]]
===== The `funding_locked` message
===== The funding_locked message
* type: *36*
* fields:
@ -372,7 +372,7 @@ message has been received, and sent can the channel being to be used.
Channel closing is a multi-step process. One node initiates by sending the `shutdown` message. The two channel partners then exchange a series of `channel_closing` messages to negotiate mutually acceptable fees for the closing transaction. The channel funder sends the first `closing_signed` message and the other side can accept by sending a `closing_signed` message with the same fee values.
[[apdx_shutdown_message]]
===== The `shutdown` message
===== The shutdown message
* type: *38*
* fields:
@ -381,7 +381,7 @@ Channel closing is a multi-step process. One node initiates by sending the `shut
** `len*byte` : `scriptpubkey`
[[apdx_closing_signed_message]]
===== The `closing_signed` message
===== The closing_signed message
* type: *39*
* fields:
@ -402,7 +402,7 @@ The `update_add_htlc` message allows either side to add a new HTLC to the
opposite commitment transaction.
[[apdx_update_add_htlc_message]]
===== The `update_add_htlc` message
===== The update_add_htlc message
* type: *128*
* fields:
@ -428,7 +428,7 @@ unique manner scoped to the channel.
The `update_fulfill_hltc` allow redemption (receipt) of an active HTLC.
[[apdx_update_fulfill_hltc_message]]
===== The `update_fulfill_hltc` message
===== The update_fulfill_hltc message
* type: *130*
* fields:
@ -443,7 +443,7 @@ provides the pre-image (which unlocks the HLTC) as well.
The `update_fail_htlc` is sent to remove an HTLC from a commitment transaction.
[[apdx_update_fail_htlc_message]]
===== The `update_fail_htlc` message
===== The update_fail_htlc message
* type: *131*
* fields:
@ -463,7 +463,7 @@ failure itself is a terminal one.
The `commitment_signed` message is used to stamp the creation of a new commitment transaction
[[apdx_commitment_signed_message]]
===== The `commitment_signed` message
===== The commitment_signed message
* type: *132*
* fields:
@ -480,7 +480,7 @@ present on the commitment transaction. This is due to the existence of the
The `revoke_and_ack` is sent to revoke a dated commitment:
[[apdx_revoke_and_ack_message]]
===== The `revoke_and_ack` message
===== The revoke_and_ack message
* type: *133*
* fields:
@ -499,7 +499,7 @@ The `update_fee` is sent to update the fee on the current commitment
transactions.
[[apdx_update_fee_message]]
===== The `update_fee` message
===== The update_fee message
* type: *134*
* fields:
@ -513,7 +513,7 @@ The `update_fail_malformed_htlc` is sent to remove a corrupted HTLC:
[[apdx_update_fail_malformed_htlc_message]]
===== The `update_fail_malformed_htlc` message
===== The update_fail_malformed_htlc message
* type: *135*
* fields:
@ -544,7 +544,7 @@ The `channel_announcement` is used to announce a new channel to the wider
network.
[[apdx_channel_announcement_message]]
===== The `channel_announcement` message
===== The channel_announcement message
* type: *256*
* fields:
@ -571,7 +571,7 @@ The `node_announcement` allows a node to announce/update it's vertex within the
greater Channel Graph.
[[apdx_node_announcement_message]]
===== The `node_announcement` message
===== The node_announcemen` message
* type: *257*
* fields:
@ -597,7 +597,7 @@ The `channel_update` message is sent to update the properties and policies of
an active channel edge within the Channel graph.
[[apdx_channel_update_message]]
===== The `channel_update` message
===== The channel_update message
* type: *258*
* fields:
@ -622,7 +622,7 @@ assemble the set of signatures required to produce a `channel_announcement`
message.
[[apdx_announce_signatures_message]]
===== The `announce_signatures` message
===== The announce_signatures message
* type: *259*
* fields:
@ -642,7 +642,7 @@ The `query_short_chan_ids` allows a peer to obtain the channel information
related to a series of short channel IDs.
[[apdx_query_short_chan_ids_message]]
===== The `query_short_chan_ids` message
===== The query_short_chan_ids message
* type: *261*
* fields:
@ -659,7 +659,7 @@ The `reply_short_chan_ids_end` message is sent after a peer finishes responding
to a prior `query_short_chan_ids` message.
[[apdx_reply_short_chan_ids_end_message]]
===== The `reply_short_chan_ids_end` message
===== The reply_short_chan_ids_end message
* type: *262*
* fields:
@ -673,7 +673,7 @@ The `query_channel_range` message allows a node to query for the set of channel
opened within a block range.
[[apdx_query_channel_range_message]]
===== The `query_channel_range` message
===== The query_channel_range message
* type: *263*
* fields:
@ -692,7 +692,7 @@ The `reply_channel_range` message is the response to `query_channel_range` and
includes the set of short channel IDs for known channels within that range.
[[apdx_reply_channel_range_message]]
===== The `reply_channel_range` message
===== The reply_channel_range message
* type: *264*
* fields:
@ -713,7 +713,7 @@ The `gossip_timestamp_range` message allows a peer to start receiving new
<p><strong>Andreas M. Antonopoulos</strong> is a best-selling author, speaker, educator, and highly sought after expert in Bitcoin and open blockchain technologies. He is known for making complex subjects easy to understand and highlighting both the positive and negative impacts these technologies can have on our global societies.</p>
<li><spanclass="printedition">September 2021:</span> First Edition</li>
<li><spanclass="printedition">December 2021:</span> First Edition</li>
</ul>
<!--Add additional revdate spans below as needed.-->
<div>
<h1class="revisions">Revision History for the Early Release</h1>
<h1class="revisions">Revision History for the First Edition</h1>
<ulclass="releases">
<li><spanclass="revdate">2021-02-16:</span> First Release</li>
<li><spanclass="revdate">2021-11-22:</span> First Release</li>
</ul>
</div>
@ -42,6 +43,8 @@
<p>The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. <em>Mastering the Lightning Network</em>, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.</p>
<p>The views expressed in this work are those of the authors, and do not represent the publisher's views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This book is not intended as legal or financial advice; use the appropriate advice. Please consult a qualified professional if you require legal/financial advice.</p>
<p><em>Mastering the Lightning Network</em> is offered under the Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 International License (CC BY-NC-ND 4.0).</p>
This quick glossary contains many of the terms used in relation to Bitcoin. These terms are used throughout the book, so bookmark this for a quick reference.
////
Proposed entries to remove:
AMP::
Atomic Multipath Payments is an extension to the protocol that allows triggering a spontaneous payment (no invoice required) that splits up a payment into multiple parts and uses additive secret sharing to ensure that the payment can only be pulled once all parts arrive. Additionally, each path of a AMP payment uses a distinct payment hash.
Breach Remedy Transaction::
A transaction claiming the outputs of a Revocable Sequence Maturity Contract with the help of the revocation key.
This can only happen if a channel partner was not following the protocol and tried to publish (willingly or due to a software bug) an old channel state.
computationally easy::
A problem is considered to be computationally easy if there exists an algorithm that is able to compute the solution to the problem relatively quickly (in polynomial time complexity).
computationally hard::
A problem is considered to be computationally hard if no algorithm exists or is known that is able to compute the solution to the problem relatively quickly (in polynomial time complexity).
downstream payment::
TBD.
HODL/Hold Invoices::
HODL/Hold invoices are effectively standard HTLC LN invoices with the exception that the recipient can “hold” the funds, deferring to settle the transaction until some condition has been met. The sender remains committed unless the recipient opts to cancel the transaction.
Neutrino:: (in the conclusion as a future thing also SPV not in glossary, too detailed)
Neutrino is a later alternative to SPV that also verifies whether certain transactions are contained in a block without downloading the entire block. However, it offers a number of improvements over SPV: Neutrino does not transmit any information that would allow a third party to determine users’ identities, it facilitates the use of non-custodial apps, and it reduces the computational load on full nodes. The trade-off for these improvements is that Neutrino requires more data from the full node than SPV.
penalty transaction::
See _Breach Remedy Transaction_.
Second stage HTLC::
TBD.
upstream payment::
TBD.
zombie channel::
An open channel where one of the channel partners has gone permanently offline.
Zombie channels cannot be used to route payments and have only downsides to the online partner.
It is good practice to close zombie channels although they can be tricky to identify as the online partner can't always be sure if the offline party will stay offline.
++++
TODO:
Some additional definitions, to be cleaned up and moved into alphabetic order are in the commented-out area below
++++
* blockchain: a single distributed ledger agreed upon by a network of participating nodes. The Lightning Network does not use a blockchain to transact, but requires transactions recorded in a blockchain in order for bitcoin to enter and leave the network.
@ -483,7 +444,7 @@ topology::
Also, the topology is important for features such as the autopilot.
satoshi::
A satoshi is the smallest denomination of bitcoin that can be recorded on the blockchain. It is the equivalent of 0.00000001 bitcoin and is named after the creator of Bitcoin, Satoshi Nakamoto. ((("satoshi")))
A satoshi is the smallest denomination of bitcoin that can be recorded on the blockchain. It is the equivalent of 0.00000001 bitcoin and is named after the creator of Bitcoin, Satoshi Nakamoto.
Satoshi Nakamoto::
Satoshi Nakamoto is the name used by the person or group of people who designed Bitcoin and created its original reference implementation, Bitcoin Core. As a part of the implementation, they also devised the first blockchain database. In the process, they were the first to solve the double-spending problem for digital currency. Their real identity remains unknown.
Andreas M. Antonopoulos
3 years ago
