Archives
/
lnbook
mirror of https://github.com/lnbook/lnbook
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

indexing complete

Browse Source
pull/910/head
Meghan Jones 3 years ago
parent 7f62859ee9
commit 24e7746e53
20 changed files with 10539 additions and 10539 deletions
Show Stats Download Patch File Download Diff File

466
01_introduction.asciidoc
Unescape Escape View File

@ -1,233 +1,233 @@
[role="pagenumrestart"]
[[intro_what_is_the_lightning_network]]
== Introduction
Welcome to _Mastering the Lightning Network_!
The Lightning Network (often abbreviated as LN), is changing the way people exchange value online, and it's one of the most exciting advancements to happen in Bitcoin's history.
Today, in 2021, the Lightning Network is still in its infancy. The Lightning Network is a protocol for using Bitcoin in a smart and nonobvious way. It is a second layer technology on top of Bitcoin.
The concept of the Lightning Network was proposed in 2015, and the first implementation was launched in 2018. As of 2021, we're only beginning to see the opportunities the Lightning Network provides to Bitcoin, including improved privacy, speed, and scale.
With core knowledge of the Lightning Network, you can help shape the future of the network while also building opportunities for yourself.
We assume you already have some basic knowledge about Bitcoin, but if not, don't worry—we will explain the most important Bitcoin concepts, those you must know to understand the Lightning Network, in <<bitcoin_fundamentals_review>>. If you want to learn more about Bitcoin, you can read _Mastering Bitcoin_, 2nd edition, by Andreas M. Antonopoulos (O'Reilly), available https://github.com/bitcoinbook/bitcoinbook[for free online].
While the bulk of this book is written for programmers, the first few chapters are written to be approachable by anyone regardless of technical experience. In this chapter, we'll start with some terminology, then move to look at trust and its application in these systems, and finally we'll discuss the history and future of the Lightning Network. Let's get started.
=== Lightning Network Basic Concepts
As we explore how the Lightning Network actually works, we will encounter some technical terminology that might, at first, be a bit confusing. While all of these concepts and terms will be explained in detail as we progress through the book and are defined in the glossary, some basic definitions now will make it easier to understand the concepts in the next two chapters. If you don't understand all of the words in these definitions yet, that's okay. You'll understand more as you move through the text.
Blockchain:: A distributed transaction ledger, produced by a network of computers. Bitcoin, for example, is a system that produces a blockchain. The Lightning Network is not itself a blockchain, nor does it produce a blockchain. It is a network that relies on an existing external blockchain for its security.
Digital Signature:: A digital signature is a mathematical scheme for verifying the authenticity of digital messages or documents. A valid digital signature gives a recipient reason to believe that the message was created by a known sender, that the sender cannot deny having sent the message, and that the message was not altered in transit.
Hash Function:: A cryptographic hash function is a mathematical algorithm that maps data of arbitrary size to a bit string of a fixed size (a hash) and is designed to be a one-way function, that is, a function which is infeasible to invert.
Node:: A computer that participates in a network. A Lightning node is a computer that participates in the Lightning Network. A Bitcoin node is a computer that participates in the Bitcoin network. Typically an LN user will run a Lightning node _and_ a Bitcoin node.
On-Chain versus Off-Chain:: A payment is _on-chain_ if it is recorded as a transaction on the Bitcoin (or other underlying) blockchain. Payments sent via payment channels between Lightning nodes, and which are not visible in the underlying blockchain, are called _off-chain_ payments. Usually in the Lightning Network, the only on-chain transactions are those used to open and close a Lightning payment channel. A third type of channel modifying transaction exists, called splicing, which can be used to add/remove the amount of funds committed in a channel.
Payment:: When value is exchanged on the Lightning Network, we call this a "payment" as compared to a "transaction" on the Bitcoin blockchain.
Payment Channel:: A _financial relationship_ between two nodes on the Lightning Network, typically implemented by multisignature Bitcoin transactions that share control over bitcoin between the two Lightning nodes.
Routing vs Sending:: Unlike Bitcoin where transactions are "sent" by broadcasting them to everyone, Lightning is a routed network where payments are "routed" across one or more payment channels following a _path_ from sender to recipient.
Transaction:: A data structure that records the transfer of control over some funds (e.g., some bitcoin). The Lightning Network relies on Bitcoin transactions (or those of another blockchain) to track control of funds.
More detailed definitions of these and many other terms can be found in the <<glossary>>. Throughout this book, we will explain what these concepts mean and how these technologies actually work.
[TIP]
====
Throughout this book, you will see "Bitcoin" with the first letter capitalized, which refers to the _Bitcoin System_ and is a proper noun. You will also see "bitcoin," with a lowercase _b_, which refers to the currency unit. Each bitcoin is further subdivided into 100 million units each called a "satoshi" (singular) or "satoshis" (plural)
====
Now that you're familiar with these basic terms, let's move to a concept you are already comfortable with: trust.
=== Trust in Decentralized Networks
You will often hear people calling Bitcoin and the Lightning Network "trustless." At first glance this is confusing. After all, isn't trust a good thing? Banks even use it in their names! Isn't a "trustless" system, a system devoid of trust, a bad thing?
The use of the word "trustless" is intended to convey the ability to operate without _needing_ trust in the other participants in the system. In a decentralized system like Bitcoin, you can always choose to transact with someone you trust. However, the system ensures you can't be cheated even if you can't trust the other party in a transaction. Trust is a nice-to-have instead of a must-have property of the system.
Contrast that to traditional systems like banking where you must place your trust in a third party, since it controls your money. If the bank violates your trust, you may be able to find some recourse from a regulator or court, but at an enormous cost of time, money, and effort.
Trustless does not mean devoid of trust. It means that trust is not a necessary prerequisite to all transactions and that you can transact even with people you don't trust because the system prevents cheating.
Before we get into how the Lightning Network works, it's important to understand one basic concept that underlies Bitcoin, the Lightning Network, and many other such systems: something we call a _fairness protocol_. A fairness protocol is a way to achieve fair outcomes between participants, who do not need to trust each other, without the need for a central authority, and it is the backbone of decentralized systems like Bitcoin.
=== Fairness Without Central Authority
When people have competing interests, how can they establish enough trust to engage in some cooperative or transactional behavior? The answer to this question lies at the core of several scientific and humanistic disciplines, such as economics, sociology, behavioral psychology, and mathematics. Some of those disciplines give us "soft" answers that depend on concepts such as reputation, fairness, morality, and even religion. Other disciplines give us concrete answers that depend only on the assumption that the participants in these interactions will act rationally, with their self-interest as the main objective.
In broad terms, there are a handful of ways to ensure fair outcomes in interactions between individuals who may have competing interests:
Require trust:: You only interact with people whom you already trust, due to prior interactions, reputation, or familial relationships. This works well enough at small scale, especially within families and small groups, that it is the most common basis for cooperative behavior. Unfortunately, it doesn't scale and it suffers from tribalist (in-group) bias.
Rule of law:: Establish rules for interactions that are enforced by an institution. This scales better, but it cannot scale globally due to differences in customs and traditions, as well as the inability to scale the institutions of enforcement. One nasty side-effect of this solution is that the institutions become more and more powerful as they get bigger and that may lead to corruption.
Trusted third parties:: Put an intermediary in every interaction to enforce fairness. Combined with the "rule of law" to provide oversight of intermediaries, this scales better, but suffers from the same imbalance of power: the intermediaries get very powerful and may attract corruption. Concentration of power leads to systemic risk and systemic failure ("Too big to fail").
Game theoretical fairness protocols:: This last category emerges from the combination of the internet and cryptography and is the subject of this section. Let's see how it works and what its advantages and disadvantages are.
==== Trusted Protocols Without Intermediaries
Cryptographic systems like Bitcoin and the Lightning Network are systems that allow you to transact with people (and computers) that you don't trust. This is often referred to as "trustless" operation, even though it is not actually trustless. You have to trust in the software that you run, and you have to trust that the protocol implemented by that software will result in fair outcomes.
The big distinction between a cryptographic system like this and a traditional financial system is that in traditional finance you have a _trusted third party_, for example a bank, to ensure that outcomes are fair. A significant problem with such systems is that they give too much power to the third party, and they are also vulnerable to a _single point of failure_. If the trusted third party itself violates trust or attempts to cheat, the basis of trust breaks.
As you study cryptographic systems, you will notice a certain pattern: instead of relying on a trusted third party, these systems attempt to prevent unfair outcomes by using a system of incentives and disincentives. In cryptographic systems you place trust in the _protocol_, which is effectively a system with a set of rules that, if properly designed, will correctly apply the desired incentives and disincentives. The advantage of this approach is twofold: not only do you avoid trusting a third party, you also reduce the need to enforce fair outcomes. So long as the participants follow the agreed protocol and stay within the system, the incentive mechanism in that protocol achieves fair outcomes without enforcement.
The use of incentives and disincentives to achieve fair outcomes is one aspect of a branch of mathematics called _game theory_, which studies "models of strategic interaction among rational decision makers."footnote:[The Wikipedia https://en.wikipedia.org/wiki/Game_theory[entry on game theory] provides more information.] Cryptographic systems that control financial interactions between participants, such as Bitcoin and the Lightning Network, rely heavily on game theory to prevent participants from cheating and allow participants who don't trust each other to achieve fair outcomes.
While game theory and its use in cryptographic systems may appear confounding and unfamiliar at first, chances are you're already familiar with these systems in your everyday life; you just don't recognize them yet. In the following we'll use a simple example from childhood to help us identify the basic pattern. Once you understand the basic pattern, you will see it everywhere in the blockchain space and you will come to recognize it quickly and intuitively.
In this book, we call this pattern a _fairness protocol_, defined as a process that uses a system of incentives and/or disincentives to ensure fair outcomes for participants who don't trust each other. Enforcement of a fairness protocol is only necessary to ensure that the participants can't escape the incentives or disincentives.
==== A Fairness Protocol in Action
Let's look at an example of a fairness protocol that you may already be familiar with.
Imagine a family lunch, with a parent and two children. The children are fussy eaters and the only thing they will agree to eat is fried potatoes. The parent has prepared a bowl of fried potatoes ("french fries" or "chips" depending on which English dialect you use). The two siblings must share the plate of chips. The parent must ensure a fair distribution of chips to each child; otherwise, the parent will have to hear constant complaining (maybe all day), and there's always a possibility of an unfair situation escalating to violence. What is a parent to do?
There are a few different ways that fairness can be achieved in this strategic interaction between two siblings that do not trust each other and have competing interests. The naive but commonly used method is for the parent to use their authority as a trusted third party: they split the bowl of chips into two servings. This is similar to traditional finance, where a bank, accountant, or lawyer acts as a trusted third party to prevent any cheating between two parties who want to transact.
The problem with this scenario is that it vests a lot of power and responsibility in the hands of the trusted third party. In this example, the parent is fully responsible for the equal allocation of chips, and the parties merely wait, watch, and complain. The children accuse the parent of playing favorites and not allocating the chips fairly. The siblings fight over the chips, yelling "that chip is bigger!", and dragging the parent into their fight. It sounds awful, doesn't it? Should the parent yell louder? Take all of the chips away? Threaten to never make chips again and let those ungrateful children go hungry?
A much better solution exists: the siblings are taught to play a game called "split and choose." At each lunch one sibling splits the bowl of chips into two servings and the _other_ sibling gets to choose which serving they want. Almost immediately, the siblings figure out the dynamic of this game. If the one splitting makes a mistake or tries to cheat, the other sibling can "punish" them by choosing the bigger bowl. It is in the best interest of both siblings, but especially the one splitting the bowl, to play fair. Only the cheater loses in this scenario. The parent doesn't even have to use their authority or enforce fairness. All the parent has to do is _enforce the protocol_; as long as the siblings cannot escape their assigned roles of "splitter" and "chooser," the protocol itself ensures a fair outcome without the need for any intervention. The parent can't play favorites or distort the outcome.
[WARNING]
====
While the infamous chip battles of the 1980s neatly illustrate the point, any similarity between the preceding scenario and any of the authors' actual childhood experiences with their cousins is entirely coincidental...Or is it?
====
==== Security Primitives as Building Blocks
In order for a fairness protocol like this to work, there need to be certain guarantees, or _security primitives_, that can be combined to ensure enforcement. The first security primitive is _strict time ordering/sequencing_: the "splitting" action must happen before the "choosing" action. It's not immediately obvious, but unless you can guarantee that action A happens before action B, then the protocol falls apart. The second security primitive is _commitment with nonrepudiation_. Each sibling must commit to their choice of role: either splitter or chooser. Also, once the splitting has been completed, the splitter is committed to the split they created—they cannot repudiate that choice and go try again.
Cryptographic systems offer a number of security primitives that can be combined in different ways to construct a fairness protocol. In addition to sequencing and commitment, we can also use many other tools:
- Hash functions to fingerprint data, as a form of commitment, or as the basis for a digital signature
- Digital signatures for authentication, nonrepudiation, and proof of ownership of a secret
- Encryption/decryption to restrict access to information to authorized participants only
This is only a small list of a whole "menagerie" of security and cryptographic primitives that are in use. More basic primitives and combinations are invented all the time.
In our real-life example, we saw one form of fairness protocol called "split and choose." This is just one of a myriad different fairness protocols that can be built by combining the building blocks of security primitives in different ways. But the basic pattern is always the same: two or more participants interact without trusting each other by engaging in a series of steps that are part of an agreed protocol. The protocol's steps arrange incentives and disincentives to ensure that if the participants are rational, cheating is counterproductive and fairness is the automatic outcome. Enforcement is not necessary to get fair outcomes—it is only necessary to keep the participants from breaking out of the agreed protocol.
Now that you understand this basic pattern, you will start seeing it everywhere in Bitcoin, the Lightning Network, and many other systems. Let's look at some specific examples next.
==== Example of the Fairness Protocol
The most prominent example of a fairness protocol is Bitcoin's consensus algorithm, Proof of Work (PoW). In Bitcoin, miners compete to verify transactions and aggregate them in blocks. To ensure that the miners do not cheat, without entrusting them with authority, Bitcoin uses a system of incentives and disincentives. Miners have to use electricity and dedicate hardware doing "work" that is embedded as a "proof" inside every block. This is achieved because of a property of hash functions where the output value is randomly distributed across the entire range of possible outputs. If miners succeed in producing a valid block fast enough, they are rewarded by earning the block reward for that block. Forcing miners to use a lot of electricity before the network considers their block means that they have an incentive to correctly validate the transactions in the block. If they cheat or make any kind of mistake, their block is rejected and the electricity they used to "prove" it is wasted. No one needs to force miners to produce valid blocks; the reward and punishment incentivize them to do so. All the protocol needs to do is ensure that only valid blocks with proof of work are accepted.
The fairness protocol pattern can also be found in many different aspects of the Lightning Network:
* Those who fund channels make sure that they have a refund transaction signed before they publish the funding transaction.
* Whenever a channel is moved to a new state, the old state is "revoked" by ensuring that if anyone tries to broadcast it, they lose the entire balance and get punished.
* Those who forward payments know that if they commit funds forward, they can either get a refund or get paid by the node preceding them.
Again and again, we see this pattern. Fair outcomes are not enforced by any authority. They emerge as the natural consequence of a protocol that rewards fairness and punishes cheating, a fairness protocol that harnesses self-interest by directing it toward fair outcomes.
Bitcoin and the Lightning Network are both implementations of fairness protocols. So why do we need the Lightning Network? Isn't Bitcoin enough?
=== Motivation for the Lightning Network
Bitcoin is a system that records transactions on a globally replicated public ledger. Every transaction is seen, validated, and stored by every participating computer. As you can imagine, this generates a lot of data and is difficult to scale.
As Bitcoin and the demand for transactions grew, the number of transactions in each block increased until it eventually reached the block size limit.
Once blocks are "full," excess transactions are left to wait in a queue. Many users will increase the fees they're willing to pay to buy space for their transactions in the next block.
If demand continues to outpace the capacity of the network, an increasing number of users' transactions are left waiting unconfirmed. Competition for fees also increases the cost of each transaction, making many smaller-value transactions (e.g., microtransactions) completely uneconomical during periods of particularly high demand.
To solve this problem, we could increase the block size limit to create space for more transactions. An increase in the "supply" of block space will lead to a lower price equilibrium for transaction fees.
However, increasing block size shifts the cost to node operators and requires them to expend more resources to validate and store the blockchain. Because blockchains are gossip protocols, each node is required to know and validate every single transaction that occurs on the network. Furthermore, once validated, each transaction and block must be propagated to the node's "neighbors," multiplying the bandwidth requirements. As such, the greater the block size, the greater the bandwidth, processing, and storage requirements for each individual node. Increasing transaction capacity in this way has the undesirable effect of centralizing the system by reducing the number of nodes and node operators. Since node operators are not compensated for running nodes, if nodes are very expensive to run, only a few well-funded node operators will continue to run nodes.
==== Scaling Blockchains
The side effects of increasing the block size or decreasing the block time with respect to centralization of the network are severe, as a few calculations with the numbers show.
Let us assume the usage of Bitcoin grows so that the network has to process 40,000 transactions per second, which is the approximate transaction processing level of the VISA network during peak usage.
Assuming 250 bytes on average per transaction, this would result in a data stream of 10 megabytes per second or 80 Mbit/s just to be able to receive all the transactions.
This does not include the traffic overhead of forwarding the transaction information to other peers.
While 10 MB/s does not seem extreme in the context of high-speed fiber optic and 5G mobile speeds, it would effectively exclude anyone who cannot meet this requirement from running a node, especially in countries where high-performance internet is not affordable or widely available.
Users also have many other demands on their bandwidth and cannot be expected to expend this much only to receive transactions.
Furthermore, storing this information locally would result in 864 gigabytes per day. This is roughly one terabyte of data, or the size of a hard drive.
Verifying 40,000 Elliptic Curve Digital Signature Algorithm (ECDSA) signatures per second is also barely feasible (see https://bitcoin.stackexchange.com/questions/95339/how-many-bitcoin-transactions-can-be-verified-per-second[this article on StackExchange]) making the _initial blockchain download (IBD)_ of the Bitcoin blockchain (synchronizing and verifying everything starting from the genesis block) almost impossible without very expensive hardware.
While 40,000 transactions per second seems like a lot, it only achieves parity with traditional financial payment networks at peak times. Innovations in machine-to-machine payments, microtransactions, and other applications are likely to push demand to many orders higher than that.
Simply put: You can't scale a blockchain to validate the entire world's transactions in a decentralized way.
_But what if each node wasn't required to know and validate every single transaction? What if there was a way to have scalable off-chain transactions, without losing the security of the Bitcoin network?_
In February 2015, Joseph Poon and Thaddeus Dryja proposed a possible solution to the Bitcoin Scalability Problem, with the publication of "The Bitcoin Lightning Network: Scalable Off-Chain Instant Payments."footnote:[Joseph Poon and Thaddeus Dryja. "The Bitcoin Lightning Network: Scalable Off-Chain Instant Payments." DRAFT Version 0.5.9.2. January 14, 2016. https://lightning.network/lightning-network-paper.pdf[].]
In the (now outdated) whitepaper, Poon and Dryja estimate that in order for Bitcoin to reach the 47,000 transactions per second processed at peak by Visa, it would require 8 GB blocks.
This would make running a node completely untenable for anyone but large-scale enterprises and industrial-grade operations.
The result would be a network in which only a few users could actually validate the state of the ledger.
Bitcoin relies on users validating the ledger for themselves, without explicitly trusting third parties, in order to stay decentralized.
Pricing users out of running nodes would force the average user to trust third parties to discover the state of the ledger, ultimately breaking the trust model of Bitcoin.
The Lightning Network proposes a new network, a second layer, where users can make payments to each other peer-to-peer, without the necessity of publishing a transaction to the Bitcoin blockchain for each payment.
Users may pay each other on the Lightning Network as many times as they want, without creating additional Bitcoin transactions or incurring on-chain fees.
They only make use of the Bitcoin blockchain to load bitcoin onto the Lightning Network initially and to _settle_, that is, to remove bitcoin from the Lightning Network.
The result is that many more Bitcoin payments can take place off-chain, with only the initial loading and final settlement transactions needing to be validated and stored by Bitcoin nodes.
Aside from reducing the burden on nodes, payments on the Lightning Network are cheaper for users because they do not need to pay blockchain fees and more private for users because they are not published to all participants of the network and furthermore not stored permanently.
While the Lightning Network was initially conceived for Bitcoin, it can be implemented on any blockchain that meets some basic technical requirements. Other blockchains, such as Litecoin, already support the Lightning Network. Additionally, several other blockchains are developing similar second layer or "layer 2" solutions to help them scale.
=== The Lightning Network's Defining Features
The Lightning Network is a network that operates as a second layer protocol on top of Bitcoin and other blockchains. The Lightning Network enables fast, secure, private, trustless, and permissionless payments. Here are some of the features of the Lightning Network:
* Users of the Lightning Network can route payments to each other for low cost and in real time.
* Users who exchange value over the Lightning Network do not need to wait for block confirmations for payments.
* Once a payment on the Lightning Network has completed, usually within a few seconds, it is final and cannot be reversed. Like a Bitcoin transaction, a payment on the Lightning Network can only be refunded by the recipient.
* Whereas on-chain Bitcoin transactions are broadcast and verified by all nodes in the network, payments routed on the Lightning Network are transmitted between pairs of nodes and are not visible to everyone, resulting in much greater privacy.
* Unlike transactions on the Bitcoin network, payments routed on the Lightning Network do not need to be stored permanently. Lightning thus uses fewer resources and hence is cheaper. This property also has benefits for privacy.
* The Lightning Network uses onion routing, similar to the protocol used by The Onion Router (Tor) privacy network, so that even the nodes involved in routing a payment are only directly aware of their predecessor and successor in the payment route.
* When used on top of Bitcoin, the Lightning Network uses real bitcoin, which is always in the possession (custody) and full control of the user. Lightning is not a separate token or coin, it _is_ Bitcoin.
[[user-stories]]
=== Lightning Network Use Cases, Users, and Their Stories
To better understand how the Lightning Network actually works, and why people use it, we'll be following a number of users and their stories.
In our examples, some of the people have already used Bitcoin and others are completely new to the Bitcoin network. Each person and their story, as listed here, illustrate one or more specific use cases. We'll be revisiting them throughout this book:
consumer::
Alice is a Bitcoin user who wants to make fast, secure, cheap, and private payments for small retail purchases. She buys coffee with bitcoin, using the Lightning Network.
merchant::
Bob owns a coffee shop, "Bob's Cafe." On-chain Bitcoin payments don't scale for small amounts like a cup of coffee, so he uses the Lightning Network to accept Bitcoin payments almost instantaneously and for low fees.
software service business::
Chan is a Chinese entrepreneur who sells information services related to the Lightning Network, as well as Bitcoin and other cryptocurrencies. Chan is selling these information services over the internet by implementing micropayments over the Lightning Network. Additionally, Chan has implemented a liquidity provider service that rents inbound channel capacity on the Lightning Network, charging a small bitcoin fee for each rental period.
gamer::
Dina is a teenage gamer from Russia. She plays many different computer games, but her favorite ones are those that have an "in-game economy" based on real money. As she plays games, she also earns money by acquiring and selling virtual in-game items. The Lightning Network allows her to transact in small amounts for in-game items as well as earn small amounts for completing quests.
=== Chapter Summary
In this chapter, we talked about the fundamental concept that underlies both Bitcoin and the Lightning Network: the fairness protocol.
We looked at the history of the Lightning Network and the motivations behind second layer scaling solutions for Bitcoin and other blockchain-based networks.
We learned basic terminology including node, payment channel, on-chain transactions, and off-chain payments.
Finally, we met Alice, Bob, Chan, and Dina, who we'll be following throughout the rest of the book. In the next chapter, we'll meet Alice and walk through her thought process as she selects a Lightning wallet and prepares to make her first Lightning payment to buy a cup of coffee from Bob's Cafe.
[role="pagenumrestart"]
[[intro_what_is_the_lightning_network]]
== Introduction
Welcome to _Mastering the Lightning Network_!
((("Lightning Network (generally)", seealso="innovations in Lightning", id="ix_01_introduction-asciidoc0", range="startofrange")))The Lightning Network (often abbreviated as LN), is changing the way people exchange value online, and it's one of the most exciting advancements to happen in Bitcoin's history.
Today, in 2021, the Lightning Network is still in its infancy. The Lightning Network is a protocol for using Bitcoin in a smart and nonobvious way. It is a second layer technology on top of Bitcoin.
The concept of the Lightning Network was proposed in 2015, and the first implementation was launched in 2018. As of 2021, we're only beginning to see the opportunities the Lightning Network provides to Bitcoin, including improved privacy, speed, and scale.
With core knowledge of the Lightning Network, you can help shape the future of the network while also building opportunities for yourself.
We assume you already have some basic knowledge about Bitcoin, but if not, don't worry—we will explain the most important Bitcoin concepts, those you must know to understand the Lightning Network, in <<bitcoin_fundamentals_review>>. If you want to learn more about Bitcoin, you can read _Mastering Bitcoin_, 2nd edition, by Andreas M. Antonopoulos (O'Reilly), available https://github.com/bitcoinbook/bitcoinbook[for free online].
While the bulk of this book is written for programmers, the first few chapters are written to be approachable by anyone regardless of technical experience. In this chapter, we'll start with some terminology, then move to look at trust and its application in these systems, and finally we'll discuss the history and future of the Lightning Network. Let's get started.
=== Lightning Network Basic Concepts
((("Lightning Network (generally)","basic concepts", id="ix_01_introduction-asciidoc1", range="startofrange")))As we explore how the Lightning Network actually works, we will encounter some technical terminology that might, at first, be a bit confusing. While all of these concepts and terms will be explained in detail as we progress through the book and are defined in the glossary, some basic definitions now will make it easier to understand the concepts in the next two chapters. If you don't understand all of the words in these definitions yet, that's okay. You'll understand more as you move through the text.
Blockchain:: ((("blockchain","defined")))A distributed transaction ledger, produced by a network of computers. Bitcoin, for example, is a system that produces a blockchain. The Lightning Network is not itself a blockchain, nor does it produce a blockchain. It is a network that relies on an existing external blockchain for its security.
Digital Signature:: ((("digital signatures")))A digital signature is a mathematical scheme for verifying the authenticity of digital messages or documents. A valid digital signature gives a recipient reason to believe that the message was created by a known sender, that the sender cannot deny having sent the message, and that the message was not altered in transit.
Hash Function:: ((("hash function, defined")))A cryptographic hash function is a mathematical algorithm that maps data of arbitrary size to a bit string of a fixed size (a hash) and is designed to be a one-way function, that is, a function which is infeasible to invert.
Node:: ((("node, defined")))A computer that participates in a network. A Lightning node is a computer that participates in the Lightning Network. A Bitcoin node is a computer that participates in the Bitcoin network. Typically an LN user will run a Lightning node _and_ a Bitcoin node.
On-Chain versus Off-Chain:: ((("on-chain payment","defined")))A payment is _on-chain_ if it is recorded as a transaction on the Bitcoin (or other underlying) blockchain. ((("off-chain payment")))Payments sent via payment channels between Lightning nodes, and which are not visible in the underlying blockchain, are called _off-chain_ payments. Usually in the Lightning Network, the only on-chain transactions are those used to open and close a Lightning payment channel. A third type of channel modifying transaction exists, called splicing, which can be used to add/remove the amount of funds committed in a channel.
Payment:: ((("payment","defined")))When value is exchanged on the Lightning Network, we call this a "payment" as compared to a "transaction" on the Bitcoin blockchain.
Payment Channel:: ((("payment channel", seealso="channel entries")))A _financial relationship_ between two nodes on the Lightning Network, typically implemented by multisignature Bitcoin transactions that share control over bitcoin between the two Lightning nodes.
Routing vs Sending:: ((("routing","sending versus")))((("sending, routing versus")))Unlike Bitcoin where transactions are "sent" by broadcasting them to everyone, Lightning is a routed network where payments are "routed" across one or more payment channels following a _path_ from sender to recipient.
Transaction:: ((("transaction, defined")))A data structure that records the transfer of control over some funds (e.g., some bitcoin). The Lightning Network relies on Bitcoin transactions (or those of another blockchain) to track control of funds.
More detailed definitions of these and many other terms can be found in the <<glossary>>. Throughout this book, we will explain what these concepts mean and how these technologies actually work.
[TIP]
====
Throughout this book, you will see "Bitcoin" with the first letter capitalized, which refers to the _Bitcoin System_ and is a proper noun. You will also see "bitcoin," with a lowercase _b_, which refers to the currency unit. Each bitcoin is further subdivided into 100 million units each called a "satoshi" (singular) or "satoshis" (plural)(((range="endofrange", startref="ix_01_introduction-asciidoc1")))
====
Now that you're familiar with these basic terms, let's move to a concept you are already comfortable with: trust.
=== Trust in Decentralized Networks
((("Lightning Network (generally)","trust in decentralized networks")))((("trustless systems","trust in decentralized networks")))You will often hear people calling Bitcoin and the Lightning Network "trustless." At first glance this is confusing. After all, isn't trust a good thing? Banks even use it in their names! Isn't a "trustless" system, a system devoid of trust, a bad thing?
The use of the word "trustless" is intended to convey the ability to operate without _needing_ trust in the other participants in the system. In a decentralized system like Bitcoin, you can always choose to transact with someone you trust. However, the system ensures you can't be cheated even if you can't trust the other party in a transaction. Trust is a nice-to-have instead of a must-have property of the system.
Contrast that to traditional systems like banking where you must place your trust in a third party, since it controls your money. If the bank violates your trust, you may be able to find some recourse from a regulator or court, but at an enormous cost of time, money, and effort.
Trustless does not mean devoid of trust. It means that trust is not a necessary prerequisite to all transactions and that you can transact even with people you don't trust because the system prevents cheating.
Before we get into how the Lightning Network works, it's important to understand one basic concept that underlies Bitcoin, the Lightning Network, and many other such systems: something we call a _fairness protocol_. A fairness protocol is a way to achieve fair outcomes between participants, who do not need to trust each other, without the need for a central authority, and it is the backbone of decentralized systems like Bitcoin.
=== Fairness Without Central Authority
((("fairness, ensuring")))((("Lightning Network (generally)","fairness without central authority")))When people have competing interests, how can they establish enough trust to engage in some cooperative or transactional behavior? The answer to this question lies at the core of several scientific and humanistic disciplines, such as economics, sociology, behavioral psychology, and mathematics. Some of those disciplines give us "soft" answers that depend on concepts such as reputation, fairness, morality, and even religion. Other disciplines give us concrete answers that depend only on the assumption that the participants in these interactions will act rationally, with their self-interest as the main objective.
In broad terms, there are a handful of ways to ensure fair outcomes in interactions between individuals who may have competing interests:
Require trust:: You only interact with people whom you already trust, due to prior interactions, reputation, or familial relationships. This works well enough at small scale, especially within families and small groups, that it is the most common basis for cooperative behavior. Unfortunately, it doesn't scale and it suffers from tribalist (in-group) bias.
Rule of law:: Establish rules for interactions that are enforced by an institution. This scales better, but it cannot scale globally due to differences in customs and traditions, as well as the inability to scale the institutions of enforcement. One nasty side-effect of this solution is that the institutions become more and more powerful as they get bigger and that may lead to corruption.
Trusted third parties:: Put an intermediary in every interaction to enforce fairness. Combined with the "rule of law" to provide oversight of intermediaries, this scales better, but suffers from the same imbalance of power: the intermediaries get very powerful and may attract corruption. Concentration of power leads to systemic risk and systemic failure ("Too big to fail").
Game theoretical fairness protocols:: This last category emerges from the combination of the internet and cryptography and is the subject of this section. Let's see how it works and what its advantages and disadvantages are.
==== Trusted Protocols Without Intermediaries
((("fairness protocol","trusted protocols without intermediaries")))Cryptographic systems like Bitcoin and the Lightning Network are systems that allow you to transact with people (and computers) that you don't trust. This is often referred to as "trustless" operation, even though it is not actually trustless. You have to trust in the software that you run, and you have to trust that the protocol implemented by that software will result in fair outcomes.
The big distinction between a cryptographic system like this and a traditional financial system is that in traditional finance you have a _trusted third party_, for example a bank, to ensure that outcomes are fair. A significant problem with such systems is that they give too much power to the third party, and they are also vulnerable to a _single point of failure_. If the trusted third party itself violates trust or attempts to cheat, the basis of trust breaks.
As you study cryptographic systems, you will notice a certain pattern: instead of relying on a trusted third party, these systems attempt to prevent unfair outcomes by using a system of incentives and disincentives. In cryptographic systems you place trust in the ((("protocol, defined")))_protocol_, which is effectively a system with a set of rules that, if properly designed, will correctly apply the desired incentives and disincentives. The advantage of this approach is twofold: not only do you avoid trusting a third party, you also reduce the need to enforce fair outcomes. So long as the participants follow the agreed protocol and stay within the system, the incentive mechanism in that protocol achieves fair outcomes without enforcement.
((("game theory")))The use of incentives and disincentives to achieve fair outcomes is one aspect of a branch of mathematics called _game theory_, which studies "models of strategic interaction among rational decision makers."footnote:[The Wikipedia https://en.wikipedia.org/wiki/Game_theory[entry on game theory] provides more information.] Cryptographic systems that control financial interactions between participants, such as Bitcoin and the Lightning Network, rely heavily on game theory to prevent participants from cheating and allow participants who don't trust each other to achieve fair outcomes.
While game theory and its use in cryptographic systems may appear confounding and unfamiliar at first, chances are you're already familiar with these systems in your everyday life; you just don't recognize them yet. In the following we'll use a simple example from childhood to help us identify the basic pattern. Once you understand the basic pattern, you will see it everywhere in the blockchain space and you will come to recognize it quickly and intuitively.
((("fairness protocol", id="ix_01_introduction-asciidoc2", range="startofrange")))In this book, we call this pattern a ((("fairness protocol","defined")))_fairness protocol_, defined as a process that uses a system of incentives and/or disincentives to ensure fair outcomes for participants who don't trust each other. Enforcement of a fairness protocol is only necessary to ensure that the participants can't escape the incentives or disincentives.
==== A Fairness Protocol in Action
((("fairness protocol","real-world example")))Let's look at an example of a fairness protocol that you may already be familiar with.
Imagine a family lunch, with a parent and two children. The children are fussy eaters and the only thing they will agree to eat is fried potatoes. The parent has prepared a bowl of fried potatoes ("french fries" or "chips" depending on which English dialect you use). The two siblings must share the plate of chips. The parent must ensure a fair distribution of chips to each child; otherwise, the parent will have to hear constant complaining (maybe all day), and there's always a possibility of an unfair situation escalating to violence. What is a parent to do?
There are a few different ways that fairness can be achieved in this strategic interaction between two siblings that do not trust each other and have competing interests. The naive but commonly used method is for the parent to use their authority as a trusted third party: they split the bowl of chips into two servings. This is similar to traditional finance, where a bank, accountant, or lawyer acts as a trusted third party to prevent any cheating between two parties who want to transact.
The problem with this scenario is that it vests a lot of power and responsibility in the hands of the trusted third party. In this example, the parent is fully responsible for the equal allocation of chips, and the parties merely wait, watch, and complain. The children accuse the parent of playing favorites and not allocating the chips fairly. The siblings fight over the chips, yelling "that chip is bigger!", and dragging the parent into their fight. It sounds awful, doesn't it? Should the parent yell louder? Take all of the chips away? Threaten to never make chips again and let those ungrateful children go hungry?
A much better solution exists: the siblings are taught to play a game called "split and choose." At each lunch one sibling splits the bowl of chips into two servings and the _other_ sibling gets to choose which serving they want. Almost immediately, the siblings figure out the dynamic of this game. If the one splitting makes a mistake or tries to cheat, the other sibling can "punish" them by choosing the bigger bowl. It is in the best interest of both siblings, but especially the one splitting the bowl, to play fair. Only the cheater loses in this scenario. The parent doesn't even have to use their authority or enforce fairness. All the parent has to do is _enforce the protocol_; as long as the siblings cannot escape their assigned roles of "splitter" and "chooser," the protocol itself ensures a fair outcome without the need for any intervention. The parent can't play favorites or distort the outcome.
[WARNING]
====
While the infamous chip battles of the 1980s neatly illustrate the point, any similarity between the preceding scenario and any of the authors' actual childhood experiences with their cousins is entirely coincidental...Or is it?
====
==== Security Primitives as Building Blocks
((("fairness protocol","security primitives as building blocks")))((("security primitives")))In order for a fairness protocol like this to work, there need to be certain guarantees, or _security primitives_, that can be combined to ensure enforcement. The first security primitive is _strict time ordering/sequencing_: the "splitting" action must happen before the "choosing" action. It's not immediately obvious, but unless you can guarantee that action A happens before action B, then the protocol falls apart. The second security primitive is _commitment with nonrepudiation_. Each sibling must commit to their choice of role: either splitter or chooser. Also, once the splitting has been completed, the splitter is committed to the split they created—they cannot repudiate that choice and go try again.
Cryptographic systems offer a number of security primitives that can be combined in different ways to construct a fairness protocol. In addition to sequencing and commitment, we can also use many other tools:
- Hash functions to fingerprint data, as a form of commitment, or as the basis for a digital signature
- Digital signatures for authentication, nonrepudiation, and proof of ownership of a secret
- Encryption/decryption to restrict access to information to authorized participants only
This is only a small list of a whole "menagerie" of security and cryptographic primitives that are in use. More basic primitives and combinations are invented all the time.
In our real-life example, we saw one form of fairness protocol called "split and choose." This is just one of a myriad different fairness protocols that can be built by combining the building blocks of security primitives in different ways. But the basic pattern is always the same: two or more participants interact without trusting each other by engaging in a series of steps that are part of an agreed protocol. The protocol's steps arrange incentives and disincentives to ensure that if the participants are rational, cheating is counterproductive and fairness is the automatic outcome. Enforcement is not necessary to get fair outcomes—it is only necessary to keep the participants from breaking out of the agreed protocol.
Now that you understand this basic pattern, you will start seeing it everywhere in Bitcoin, the Lightning Network, and many other systems. Let's look at some specific examples next.
==== Example of the Fairness Protocol
((("fairness protocol","Proof of Work example")))((("PoW (Proof of Work) algorithm")))((("Proof of Work (PoW) algorithm")))The most prominent example of a fairness protocol is Bitcoin's consensus algorithm, Proof of Work (PoW). In Bitcoin, miners compete to verify transactions and aggregate them in blocks. To ensure that the miners do not cheat, without entrusting them with authority, Bitcoin uses a system of incentives and disincentives. Miners have to use electricity and dedicate hardware doing "work" that is embedded as a "proof" inside every block. This is achieved because of a property of hash functions where the output value is randomly distributed across the entire range of possible outputs. If miners succeed in producing a valid block fast enough, they are rewarded by earning the block reward for that block. Forcing miners to use a lot of electricity before the network considers their block means that they have an incentive to correctly validate the transactions in the block. If they cheat or make any kind of mistake, their block is rejected and the electricity they used to "prove" it is wasted. No one needs to force miners to produce valid blocks; the reward and punishment incentivize them to do so. All the protocol needs to do is ensure that only valid blocks with proof of work are accepted.
The fairness protocol pattern can also be found in many different aspects of the Lightning Network:
* Those who fund channels make sure that they have a refund transaction signed before they publish the funding transaction.
* Whenever a channel is moved to a new state, the old state is "revoked" by ensuring that if anyone tries to broadcast it, they lose the entire balance and get punished.
* Those who forward payments know that if they commit funds forward, they can either get a refund or get paid by the node preceding them.
Again and again, we see this pattern. Fair outcomes are not enforced by any authority. They emerge as the natural consequence of a protocol that rewards fairness and punishes cheating, a fairness protocol that harnesses self-interest by directing it toward fair outcomes.
Bitcoin and the Lightning Network are both implementations of fairness protocols. So why do we need the Lightning Network? Isn't Bitcoin enough?(((range="endofrange", startref="ix_01_introduction-asciidoc2")))
=== Motivation for the Lightning Network
((("Lightning Network (generally)","motivation for", id="ix_01_introduction-asciidoc3", range="startofrange")))Bitcoin is a system that records transactions on a globally replicated public ledger. Every transaction is seen, validated, and stored by every participating computer. As you can imagine, this generates a lot of data and is difficult to scale.
As Bitcoin and the demand for transactions grew, the number of transactions in each block increased until it eventually reached the block size limit.
Once blocks are "full," excess transactions are left to wait in a queue. Many users will increase the fees they're willing to pay to buy space for their transactions in the next block.
If demand continues to outpace the capacity of the network, an increasing number of users' transactions are left waiting unconfirmed. Competition for fees also increases the cost of each transaction, making many smaller-value transactions (e.g., microtransactions) completely uneconomical during periods of particularly high demand.
To solve this problem, we could increase the block size limit to create space for more transactions. An increase in the "supply" of block space will lead to a lower price equilibrium for transaction fees.
However, increasing block size shifts the cost to node operators and requires them to expend more resources to validate and store the blockchain. Because blockchains are gossip protocols, each node is required to know and validate every single transaction that occurs on the network. Furthermore, once validated, each transaction and block must be propagated to the node's "neighbors," multiplying the bandwidth requirements. As such, the greater the block size, the greater the bandwidth, processing, and storage requirements for each individual node. Increasing transaction capacity in this way has the undesirable effect of centralizing the system by reducing the number of nodes and node operators. Since node operators are not compensated for running nodes, if nodes are very expensive to run, only a few well-funded node operators will continue to run nodes.
==== Scaling Blockchains
((("blockchain","scaling", id="ix_01_introduction-asciidoc4", range="startofrange")))((("Lightning Network (generally)","scaling blockchains", id="ix_01_introduction-asciidoc5", range="startofrange")))The side effects of increasing the block size or decreasing the block time with respect to centralization of the network are severe, as a few calculations with the numbers show.
Let us assume the usage of Bitcoin grows so that the network has to process 40,000 transactions per second, which is the approximate transaction processing level of the VISA network during peak usage.
Assuming 250 bytes on average per transaction, this would result in a data stream of 10 megabytes per second or 80 Mbit/s just to be able to receive all the transactions.
This does not include the traffic overhead of forwarding the transaction information to other peers.
While 10 MB/s does not seem extreme in the context of high-speed fiber optic and 5G mobile speeds, it would effectively exclude anyone who cannot meet this requirement from running a node, especially in countries where high-performance internet is not affordable or widely available.
Users also have many other demands on their bandwidth and cannot be expected to expend this much only to receive transactions.
Furthermore, storing this information locally would result in 864 gigabytes per day. This is roughly one terabyte of data, or the size of a hard drive.
Verifying 40,000 Elliptic Curve Digital Signature Algorithm (ECDSA) signatures per second is also barely feasible (see https://bitcoin.stackexchange.com/questions/95339/how-many-bitcoin-transactions-can-be-verified-per-second[this article on StackExchange]) making the _initial blockchain download (IBD)_ of the Bitcoin blockchain (synchronizing and verifying everything starting from the genesis block) almost impossible without very expensive hardware.
While 40,000 transactions per second seems like a lot, it only achieves parity with traditional financial payment networks at peak times. Innovations in machine-to-machine payments, microtransactions, and other applications are likely to push demand to many orders higher than that.
Simply put: You can't scale a blockchain to validate the entire world's transactions in a decentralized way.
_But what if each node wasn't required to know and validate every single transaction? What if there was a way to have scalable off-chain transactions, without losing the security of the Bitcoin network?_
In February 2015, Joseph Poon and Thaddeus Dryja proposed a possible solution to the Bitcoin Scalability Problem, with the publication of "The Bitcoin Lightning Network: Scalable Off-Chain Instant Payments."footnote:[Joseph Poon and Thaddeus Dryja. "The Bitcoin Lightning Network: Scalable Off-Chain Instant Payments." DRAFT Version 0.5.9.2. January 14, 2016. https://lightning.network/lightning-network-paper.pdf[].]
In the (now outdated) whitepaper, Poon and Dryja estimate that in order for Bitcoin to reach the 47,000 transactions per second processed at peak by Visa, it would require 8 GB blocks.
This would make running a node completely untenable for anyone but large-scale enterprises and industrial-grade operations.
The result would be a network in which only a few users could actually validate the state of the ledger.
Bitcoin relies on users validating the ledger for themselves, without explicitly trusting third parties, in order to stay decentralized.
Pricing users out of running nodes would force the average user to trust third parties to discover the state of the ledger, ultimately breaking the trust model of Bitcoin.
The Lightning Network proposes a new network, a second layer, where users can make payments to each other peer-to-peer, without the necessity of publishing a transaction to the Bitcoin blockchain for each payment.
Users may pay each other on the Lightning Network as many times as they want, without creating additional Bitcoin transactions or incurring on-chain fees.
They only make use of the Bitcoin blockchain to load bitcoin onto the Lightning Network initially and to _settle_, that is, to remove bitcoin from the Lightning Network.
The result is that many more Bitcoin payments can take place off-chain, with only the initial loading and final settlement transactions needing to be validated and stored by Bitcoin nodes.
Aside from reducing the burden on nodes, payments on the Lightning Network are cheaper for users because they do not need to pay blockchain fees and more private for users because they are not published to all participants of the network and furthermore not stored permanently.
While the Lightning Network was initially conceived for Bitcoin, it can be implemented on any blockchain that meets some basic technical requirements. Other blockchains, such as Litecoin, already support the Lightning Network. Additionally, several other blockchains are developing similar second layer or "layer 2" solutions to help them scale(((range="endofrange", startref="ix_01_introduction-asciidoc5")))(((range="endofrange", startref="ix_01_introduction-asciidoc4"))).(((range="endofrange", startref="ix_01_introduction-asciidoc3")))
=== The Lightning Network's Defining Features
((("Lightning Network (generally)","defining features")))The Lightning Network is a network that operates as a second layer protocol on top of Bitcoin and other blockchains. The Lightning Network enables fast, secure, private, trustless, and permissionless payments. Here are some of the features of the Lightning Network:
* Users of the Lightning Network can route payments to each other for low cost and in real time.
* Users who exchange value over the Lightning Network do not need to wait for block confirmations for payments.
* Once a payment on the Lightning Network has completed, usually within a few seconds, it is final and cannot be reversed. Like a Bitcoin transaction, a payment on the Lightning Network can only be refunded by the recipient.
* Whereas on-chain Bitcoin transactions are broadcast and verified by all nodes in the network, payments routed on the Lightning Network are transmitted between pairs of nodes and are not visible to everyone, resulting in much greater privacy.
* Unlike transactions on the Bitcoin network, payments routed on the Lightning Network do not need to be stored permanently. Lightning thus uses fewer resources and hence is cheaper. This property also has benefits for privacy.
* The Lightning Network uses onion routing, similar to the protocol used by The Onion Router (Tor) privacy network, so that even the nodes involved in routing a payment are only directly aware of their predecessor and successor in the payment route.
* When used on top of Bitcoin, the Lightning Network uses real bitcoin, which is always in the possession (custody) and full control of the user. Lightning is not a separate token or coin, it _is_ Bitcoin.
[[user-stories]]
=== Lightning Network Use Cases, Users, and Their Stories
((("Lightning Network (generally)","use cases and users")))To better understand how the Lightning Network actually works, and why people use it, we'll be following a number of users and their stories.
In our examples, some of the people have already used Bitcoin and others are completely new to the Bitcoin network. Each person and their story, as listed here, illustrate one or more specific use cases. We'll be revisiting them throughout this book:
consumer::
Alice is a Bitcoin user who wants to make fast, secure, cheap, and private payments for small retail purchases. She buys coffee with bitcoin, using the Lightning Network.
merchant::
Bob owns a coffee shop, "Bob's Cafe." On-chain Bitcoin payments don't scale for small amounts like a cup of coffee, so he uses the Lightning Network to accept Bitcoin payments almost instantaneously and for low fees.
software service business::
Chan is a Chinese entrepreneur who sells information services related to the Lightning Network, as well as Bitcoin and other cryptocurrencies. Chan is selling these information services over the internet by implementing micropayments over the Lightning Network. Additionally, Chan has implemented a liquidity provider service that rents inbound channel capacity on the Lightning Network, charging a small bitcoin fee for each rental period.
gamer::
Dina is a teenage gamer from Russia. She plays many different computer games, but her favorite ones are those that have an "in-game economy" based on real money. As she plays games, she also earns money by acquiring and selling virtual in-game items. The Lightning Network allows her to transact in small amounts for in-game items as well as earn small amounts for completing quests.
=== Chapter Summary
In this chapter, we talked about the fundamental concept that underlies both Bitcoin and the Lightning Network: the fairness protocol.
We looked at the history of the Lightning Network and the motivations behind second layer scaling solutions for Bitcoin and other blockchain-based networks.
We learned basic terminology including node, payment channel, on-chain transactions, and off-chain payments.
Finally, we met Alice, Bob, Chan, and Dina, who we'll be following throughout the rest of the book.(((range="endofrange", startref="ix_01_introduction-asciidoc0"))) In the next chapter, we'll meet Alice and walk through her thought process as she selects a Lightning wallet and prepares to make her first Lightning payment to buy a cup of coffee from Bob's Cafe.((("Bitcoin (system)","Lightning Network compared to", see="Bitcoin–Lightning Network comparisons")))((("channel", see="payment channel")))((("containers", see="Docker containers")))((("delivering payment", see="payment delivery")))((("DoS attacks", see="denial-of-service attacks")))((("encrypted message transport", see="Lightning encrypted transport protocol")))((("future issues", see="innovations in Lightning")))((("HTLCs", see="hash time-locked contracts")))((("invoices", see="Lightning invoices")))((("Lightning Network (generally)","Bitcoin compared to", see="Bitcoin–Lightning Network comparisons")))((("Lightning Network (generally)","invoices", see="Lightning invoices")))((("Lightning Network (generally)","network architecture", see="architecture, Lightning Network")))((("Lightning Network node", see="Lightning node entries")))((("Lightning payment requests", see="Lightning invoices")))((("LN node", see="Lightning node entries")))((("LND node project", see="Lightning Network Daemon node project")))((("message transport", see="Lightning encrypted transport protocol")))((("node", see="Lightning node entries")))((("payment channel","routing on network of", see="routing")))((("payment requests", see="Lightning invoices")))((("payment routing", see="routing")))((("privacy", see="breaches of privacy")))((("privacy", see="security and privacy")))((("private channels", see="unannounced channels")))((("TLV", see="Type-Length-Value")))((("wallet", see="Lightning wallet")))

904
02_getting_started.asciidoc
Unescape Escape View File

@ -1,452 +1,452 @@
[[getting-started]]
== Getting Started
In this chapter, we will begin where most people start when encountering the Lightning Network for the first time&#x2014;choosing software to participate in the LN economy. We will examine the choices of two users who represent a common use-case for the Lightning Network and learn by example. Alice, a coffee shop customer, will be using a Lightning wallet on her mobile device to buy coffee from Bob's Cafe. Bob, a merchant, will be using a Lightning node and wallet to run a point-of-sale system at his cafe, so he can accept payments over the Lightning Network.
=== Alice's First Lightning Wallet
Alice is a longtime Bitcoin user. We first met Alice in Chapter 1 of _"Mastering Bitcoin,"_ footnote:[_Mastering Bitcoin_, 2nd Edition, https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch01.asciidoc[Chapter 1] (O'Reilly) by Andreas M. Antonopoulos] when she bought a cup of coffee from Bob's cafe using a Bitcoin transaction. If you are not yet familiar with how Bitcoin transactions work or need a refresher, please read _Mastering Bitcoin_ or the summary in <<bitcoin_fundamentals_review>>.
Alice recently learned that Bob's Cafe just started accepting LN payments! Alice is eager to learn about and experiment with the Lightning Network; she wants to be one of Bob's first LN customers. To do this, first, Alice has to select a Lightning wallet that meets her needs.
Alice does not want to entrust custody of her bitcoin to third parties. She has learned enough about cryptocurrency to know how to use a wallet. She also wants a mobile wallet so that she can use it for small payments on-the-go, so she chooses the _Eclair_ wallet, a popular noncustodial mobile Lightning wallet. Let's learn more about how and why she's made these choices.
=== Lightning Nodes
The Lightning Network is accessed via software applications that can speak the LN protocol. A _Lightning Network node_ (LN node or simply node) is a software application that has three important characteristics. First, Lightning nodes are wallets so they send and receive payments over the Lightning Network as well as on the Bitcoin network. Second, nodes must communicate on a peer-to-peer basis with other Lightning nodes creating the network. Finally, Lightning nodes also need access to the Bitcoin blockchain (or other blockchains for other cryptocurrencies) to secure the funds used for payments.
Users have the highest degree of control by running their own Bitcoin node and Lightning node. However, Lightning nodes can also use a lightweight Bitcoin client, commonly referred to as simplified payment verification (SPV), to interact with the Bitcoin blockchain.
[[ln_explorer]]
=== Lightning Explorers
LN explorers are useful tools to show the statistics of nodes, channels, and network capacity.
Following is an inexhaustive list (in alphanumerical order):
* https://1ml.com/1ML[Lightning explorer]
* https://explorer.acinq.co/[ACINQ's Lightning explorer], with fancy visualization
* https://amboss.space/[Amboss Space Lightning explorer], with community metrics and intuitive visualizations
* https://ln.bigsun.xyz/[Fiatjaf's Lightning explorer] with many diagrams
* https://hashxp.org/lightning/node/[hashXP Lightning explorer]
[TIP]
====
Note that when using Lightning explorers, just like with other block explorers, privacy can be a concern.
If users are careless, the website may track their IP addresses and collect their behavior records (for example, the nodes users are interested in).
Also, it should be noted that because there is no global consensus of the current Lightning graph or the current state of any existing channel policy, users should never rely on Lightning explorers to retrieve the most current information.
Furthermore, as users open, close, and update channels, the graph will change and individual Lightning explorers may not be up to date.
Use Lightning explorers to visualize the network or gather information, but not as an authoritative source of what is happening on the Lightning Network.
To have an authoritative view of the Lightning Network, run your own Lightning node that will build a channel graph and collect various statistics, which you can view with a web-based interface.
====
=== Lightning Wallets
The term _Lightning wallet_ is somewhat ambiguous because it can describe a broad variety of components combined with some user interface. The most common components of Lightning wallet software include:
* A keystore that holds secrets, such as private keys
* A LN node (Lightning node) that communicates on the peer-to-peer network, as described previously
* A Bitcoin node that stores blockchain data and communicates with other Bitcoin nodes
* A database "map" of nodes and channels that are announced on the Lightning Network
* A channel manager that can open and close LN channels
* A close up system that can find a path of connected channels from payment source to payment destination
A Lightning wallet may contain all of these functions, acting as a "full" wallet, with no reliance on any third-party services. Or one or more of these components may rely (partially or entirely) on third-party services that mediate those functions.
A _key_ distinction (pun intended) is whether the keystore function is internal or outsourced. In blockchains, control of keys determines custody of funds, as memorialized by the phrase "your keys, your coins; not your keys, not your coins." Any wallet that outsources management of keys is called a _custodial_ wallet because a third party acting as custodian has control of the user's funds, not the user. A _noncustodial_ or _self-custody_ wallet, by comparison, is one where the keystore is part of the wallet, and keys are controlled directly by the user. The term noncustodial wallet just implies that the keystore is local and under the user's control. However, one or more of the other wallet components may or may not be outsourced and rely on trusted third parties.
Blockchains, especially open blockchains like Bitcoin, attempt to minimize or eliminate trust in third parties and empower users. This is often called a "trustless" model, though "trust-minimized" is a better term. In such systems, the user trusts the software rules, not third parties. Therefore, the issue of control over keys is a principal consideration when choosing a Lightning wallet.
Every other component of a Lightning wallet brings similar considerations of trust. If all the components are under the control of the user, then the amount of trust in third parties is minimized, bringing maximum power to the user. Of course, this brings a direct trade-off because with that power comes the corresponding responsibility to manage complex software.
Every user must consider their own technical skills before deciding what type of Lightning wallet to use. Those with strong technical skills should use a Lightning wallet that puts all of the components under the direct control of the user. Those with fewer technical skills, but with a desire to control their funds, should choose a noncustodial Lightning wallet.
Often the trust in these cases relates to privacy.
If users decide to outsource some functionality to a third party, they usually give up some privacy as the third party will learn some information about them.
Finally, those seeking simplicity and convenience, even at the expense of control and security, may choose a custodial Lightning wallet. This is the least technically challenging option, but it _undermines the trust model of cryptocurrency_ and should therefore be considered only as a stepping stone toward more control and self-reliance.
There are many ways wallets can be characterized or categorized.
The most important questions to ask about a specific wallet are:
. Does this Lightning wallet have a full Lightning node or does it use a third-party Lightning node?
. Does this Lightning wallet have a full Bitcoin node or does it use a third-party Bitcoin node?
. Does this Lightning wallet store its own keys under user control (self-custody) or are the keys held by a third-party custodian?
[TIP]
====
If a Lightning wallet uses a third-party Lightning node, it is this third-party Lightning node that decides how to communicate with Bitcoin. Hence, using a third-party Lightning node implies that you are also using a third-party Bitcoin node. Only when the Lightning wallet uses its own Lightning node does the choice between full Bitcoin node and third-party Bitcoin node exist.
====
At the highest level of abstraction, Questions 1 and 3 are the most elementary ones.
From these two questions, we can derive four possible categories.
We can place these four categories into a quadrant as seen in <<lnwallet-categories>>.
But remember that this is just one way of categorizing Lightning wallets.
[[lnwallet-categories]]
.Lightning wallets quadrant
[options="header"]
|===
| | *Full Lightning node* | *3rd-party Lightning node*
| *Self-Custody* | Q1: High technical skill, least trust in 3rd parties, most permissionless | Q2: Below medium technical skills, below medium trust in 3rd parties, requires some permissions
| *Custodial* | Q3: Above medium technical skills, above medium trust in 3rd parties, requires some permissions | Q4: Low technical skills, high trust in 3rd parties, least permissionless
|===
Quadrant 3 (Q3), where a full Lightning node is used, but the keys are held by a custodian, is currently not common.
Future wallets from that quadrant may let a user worry about the operational aspects of their node, but then delegate access to the keys to a third party which primarily uses cold storage.
Lightning wallets can be installed on a variety of devices, including laptops, servers, and mobile devices. To run a full Lightning node you will need to use a server or desktop computer because mobile devices and laptops are usually not powerful enough in terms of capacity, processing, battery life, and connectivity.
The category third-party Lightning nodes can again be subdivided:
Lightweight::
This means that the wallet does not operate a Lightning node and thus needs to obtain information about the Lightning Network over the internet from someone else's Lightning node.
None::
This means that not only is the Lightning node operated by a third party, but most of the wallet is operated by a third party in the cloud. This is a custodial wallet where someone else controls custody of the funds.
These subcategories are used in <<lnwallet-examples>>.
Other terms that need explanation in <<lnwallet-examples>> in the column "Bitcoin node" are:
Neutrino::
This wallet does not operate a Bitcoin node. Instead, a Bitcoin node operated by someone else (a third party) is accessed via the Neutrino Protocol.
Electrum::
This wallet does not operate a Bitcoin node. Instead, a Bitcoin node operated by someone else (a third party) is accessed via the Electrum protocol.
Bitcoin Core::
This is an implementation of a Bitcoin node.
btcd::
This is another implementation of a Bitcoin node.
In <<lnwallet-examples>>, we see some examples of currently popular Lightning node and wallet applications for different types of device. The list is sorted first by device type and then alphabetically.
[[lnwallet-examples]]
.Examples of popular Lightning wallets
[options="header"]
|===
| Application | Device | Lightning node | Bitcoin node | Keystore
| Blue Wallet | Mobile | None | None | Custodial
| Breez Wallet | Mobile | Full node | Neutrino | Self-Custody
| Eclair Mobile | Mobile | Lightweight | Electrum | Self-Custody
| lntxbot | Mobile | None | None | Custodial
| Muun | Mobile | Lightweight | Neutrino | Self-Custody
| Phoenix Wallet | Mobile | Lightweight | Electrum | Self-Custody
| Zeus | Mobile | Full node | Bitcoin Core/btcd | Self-Custody
| Electrum | Desktop | Full node | Bitcoin Core/Electrum | Self-Custody
| Zap Desktop | Desktop | Full node | Neutrino | Self-Custody
| c-lightning | Server | Full node | Bitcoin Core | Self-Custody
| Eclair Server | Server | Full node | Bitcoin Core/Electrum | Self-Custody
| lnd | Server | Full node | Bitcoin Core/btcd | Self-Custody
|===
[[testnet-bitcoin]]
==== Testnet Bitcoin
The Bitcoin system offers an alternative chain for testing purposes called _testnet_, in contrast with the "normal" Bitcoin chain which is referred to as _mainnet_. On testnet, the currency is _testnet bitcoin_ (_TBTC_), which is a worthless copy of bitcoin used exclusively for testing. Every function of Bitcoin is replicated exactly, but the money is worth nothing, so you literally have nothing to lose!
Some Lightning wallets can also operate on testnet, allowing you to make Lightning payments with testnet bitcoin, without risking real funds. This is a great way to experiment with Lightning safely. Eclair Mobile, which Alice uses in this chapter, is one example of a Lightning wallet that supports testnet operation.
You can get some TBTC to play with from a _testnet bitcoin faucet_, which gives out free TBTC on demand. Here are a few testnet faucets:
++++
<ul class="simplelist">
<li><a href="https://coinfaucet.eu/en/btc-testnet/"><em>https://coinfaucet.eu/en/btc-testnet</em></a></li>
<li><a href="https://testnet-faucet.mempool.co/"><em>https://testnet-faucet.mempool.co/</em></a></li>
<li><a href="https://bitcoinfaucet.uo1.net/"><em>https://bitcoinfaucet.uo1.net/</em></a></li>
<li><a href="https://testnet.help/en/btcfaucet/testnet"><em>https://testnet.help/en/btcfaucet/testnet</em></a></li>
</ul>
++++
All of the examples in this book can be replicated exactly on testnet with TBTC, so you can follow along if you want without risking real money.
=== Balancing Complexity and Control
Lightning wallets have to strike a careful balance between complexity and user control. Those that give the user the most control over their funds, the highest degree of privacy, and the greatest independence from third-party services are necessarily more complex and difficult to operate. As the technology advances, some of these trade-offs will become less stark, and users may be able to get more control without more complexity. However, for now, different companies and projects are exploring different positions along this control-complexity spectrum and hoping to find the "sweet spot" for the users they are targeting.
When selecting a wallet, keep in mind that even if you don't see these trade-offs, they still exist. For example, many wallets will attempt to remove the burden of channel management from their users. To do so, they introduce central _hub nodes_ that all their wallets connect to automatically. While this trade-off simplifies the user interface and user experience, it introduces a single point of failure (SPoF) as these hub nodes become indispensable for the wallet's operation. Furthermore, relying on a "hub" like this can reduce user privacy since the hub knows the sender and potentially (if constructing the payment route on behalf of the user) also the recipient of each payment made by the user's wallet.
In the next section, we will return to our first user and walk through her first Lightning wallet setup. She has chosen a wallet that is more sophisticated than the easier custodial wallets. This allows us to show some of the underlying complexity and introduce some of the inner workings of an advanced wallet. You may find that your first ideal wallet is oriented toward ease of use, accepting some of the control and privacy trade-offs. Or perhaps you are more of a power user and want to run your own Lightning and Bitcoin nodes as part of your wallet solution.
=== Downloading and Installing a Lightning Wallet
When looking for a new cryptocurrency wallet, you must be very careful to select a secure source for the software.
Unfortunately, many fake wallet applications will steal your money, and some of these even find their way onto reputable and supposedly vetted software sites like the Apple and Google application stores. Whether you are installing your first or your tenth wallet, always exercise extreme caution. A rogue app may not just steal any money you entrust it with, but it might also be able to steal keys and passwords from other applications by compromising your mobile device operating system.
Alice uses an Android device and will use the Google Play Store to download and install the Eclair wallet. Searching on Google Play, she finds an entry for "Eclair Mobile," as shown in <<eclair-playstore>>.
[[eclair-playstore]]
.Eclair Mobile in the Google Play Store
image::images/mtln_0201.png["Eclair wallet in the Google Play Store"]
[TIP]
====
It is possible to experiment and test all Bitcoin-type software with zero risk (except for your own time) by using testnet bitcoins. You can also download Eclair testnet wallet to try Lightning (on testnet) by going to the Google Play Store.
====
Alice notices a few different elements on this page that help her ascertain that this is, most likely, the correct "Eclair Mobile" wallet she is looking for. Firstly, the organization ACINQfootnote:[ACINQ: Developers of the Eclair Mobile Lightning wallet (https://acinq.co/).] is listed as the developer of this mobile wallet, which Alice knows from her research is the correct developer. Secondly, the wallet has been installed "10,000+" times and has more than 320 positive reviews. It is unlikely this is a rogue app that has snuck into the Google Play Store. As a third step, she goes to the https://acinq.co[ACINQ website]. She verifies that the web page is secure by checking that the address begins with https, or prefixed by a padlock in some browsers. On the website she goes to the Download section or looks for the link to the Google App Store. She finds the link and clicks it. She compares that this link brings her to the very same app in the Google App Store. Satisfied by these findings, Alice installs the Eclair app on her mobile device.
[WARNING]
====
Always exercise great care when installing software on any device. There are many fake cryptocurrency wallets that will not only steal your money but might also compromise all other applications on your device.
====
=== Creating a New Wallet
When Alice opens the Eclair Mobile app for the first time, she is presented with a choice to "Create a New Wallet" or to "Import an Existing Wallet." Alice will create a new wallet, but let's first discuss why these options are presented here and what it means to import an existing wallet.
==== Responsibility with Key Custody
As we mentioned at the beginning of this section, Eclair is a _noncustodial_ wallet, meaning that Alice has sole custody of the keys used to control her bitcoin. This also means that Alice is responsible for protecting and backing up those keys. If Alice loses the keys, no one can help her recover the bitcoin, and they will be lost forever.
[WARNING]
====
With the Eclair Mobile wallet, Alice has custody and control of the keys and, therefore, full responsibility to keep the keys safe and backed up. If she loses the keys, she loses the bitcoin, and no one can help her recover from that loss!
====
==== Mnemonic Words
Similar to most Bitcoin wallets, Eclair Mobile provides a _mnemonic phrase_ (also sometimes called a "seed" or "seed phrase") for Alice to back up. The mnemonic phrase consists of 24 English words, selected randomly by the software, and used as the basis for the keys that are generated by the wallet. The mnemonic phrase can be used by Alice to restore all the transactions and funds in the Eclair Mobile wallet in the case of an event such as a lost mobile device, a software bug, or memory corruption.
[TIP]
====
The correct term for these backup words is "mnemonic phrase." We avoid the use of the term "seed" to refer to a mnemonic phrase because even though its use is common, it is incorrect.
====
When Alice chooses to Create a New Wallet, she will be shown a screen with her mnemonic phrase, which looks like the screenshot in <<eclair-mnemonic>>.
[[eclair-mnemonic]]
.New wallet mnemonic phrase
image::images/mtln_0202.png["New Wallet Mnemonic Phrase"]
In <<eclair-mnemonic>>, we have purposely obscured part of the mnemonic phrase to prevent readers of this book from reusing the mnemonic.
[[mnemonic-storage]]
==== Storing the Mnemonic Safely
Alice needs to be careful to store the mnemonic phrase in a way that prevents theft but also avoids accidental loss. The recommended way to properly balance these risks is to write two copies of the mnemonic phrase on paper, with each of the words numbered&#x2014;the order matters.
Once Alice has recorded the mnemonic phrase, after touching "OK GOT IT" on her screen, she will be presented with a quiz to make sure that she correctly recorded the mnemonic. The quiz will ask for three or four of the words at random. Alice wasn't expecting a quiz, but since she recorded the mnemonic correctly, she passes without any difficulty.
Once Alice has recorded the mnemonic phrase and passed the quiz, she should store each copy in a separate secure location such as a locked desk drawer or a fireproof safe.
[WARNING]
====
Never attempt a "DIY" security scheme that deviates in any way from the best practice recommendation in <<mnemonic-storage>>. Do not cut your mnemonic in half, make screenshots, store on USB drives or cloud drives, encrypt it, or try any other nonstandard method. You will tip the balance in such a way as to risk permanent loss. Many people have lost funds, not from theft, but because they tried a nonstandard solution without having the expertise to balance the risks involved. The best practice recommendation is carefully considered by experts and suitable for the vast majority of users.
====
After Alice initializes her Eclair Mobile wallet, she will see a brief tutorial that highlights the various elements of the user interface. We won't replicate the tutorial here, but we will explore all of those elements as we follow Alice's attempt to buy a cup of coffee!
=== Loading Bitcoin Onto the Wallet
Alice now has a Lightning wallet. But it's empty! She now faces one of the more challenging aspects of this experiment; she has to find a way to acquire some bitcoin and load it onto her Eclair wallet.
[TIP]
====
If Alice already has bitcoin in another wallet, she could choose to send that bitcoin to her Eclair wallet instead of acquiring new bitcoin to load onto her new wallet.
====
[[acquiring-bitcoin]]
==== Acquiring Bitcoin
There are several ways Alice can acquire bitcoin:
* She can exchange some of her national currency (e.g., USD) on a cryptocurrency exchange.
* She can buy some from a friend, or an acquaintance from a Bitcoin meetup, in exchange for cash.
* She can find a _Bitcoin ATM_ in her area, which acts as a vending machine, selling bitcoin for cash.
* She can offer her skills or a product she sells and accept payment in bitcoin.
* She can ask her employer or clients to pay her in bitcoin.
All of these methods have varying degrees of difficulty, and many will involve paying a fee. Some will also require Alice to provide identification documents to comply with local banking regulations. However, with all these methods, Alice will be able to receive bitcoin.
==== Receiving Bitcoin
Let's assume Alice has found a local Bitcoin ATM and has decided to buy some bitcoin in exchange for cash. An example of a Bitcoin ATM, one built by the Lamassu Company, is shown in <<bitcoin-atm>>. Such Bitcoin ATMs accept national currency (cash) through a cash slot and send bitcoin to a Bitcoin address scanned from a user's wallet using a built-in camera.
[[bitcoin-atm]]
.A Lamassu Bitcoin ATM
image::images/mtln_0203.png["Lamassu Bitcoin ATM"]
To receive the bitcoin in her Eclair Lightning wallet, Alice will need to present a Bitcoin address from the Eclair Lightning wallet to the ATM. The ATM can then send Alice's newly acquired bitcoin to this Bitcoin address.
To see a Bitcoin address on the Eclair wallet, Alice must swipe to the left column titled YOUR BITCOIN ADDRESS (see <<eclair-receive>>), where she will see a square barcode (called a _QR code_) and a string of letters and numbers below that.
[[eclair-receive]]
.Alice's bitcoin address, shown in Eclair
image::images/mtln_0204.png["Eclair bitcoin address QR code"]
The QR code contains the same string of letters and numbers as shown below it, in an easy to scan format. This way, Alice doesn't have to type the Bitcoin address. In the screenshot (<<eclair-receive>>), we have purposely blurred both, to prevent readers from inadvertently sending bitcoin to this address.
[NOTE]
====
Both Bitcoin addresses and QR codes contain error detection information that prevents any typing or scanning errors from producing a "wrong" Bitcoin address. If there is a mistake in the address, any Bitcoin wallet will notice the error and refuse to accept the Bitcoin address as valid.
====
Alice can take her mobile device to the ATM and show it to the built-in camera, as shown in <<bitcoin-atm-receive>>. After inserting some cash into the slot, she will receive bitcoin in Eclair!
[[bitcoin-atm-receive]]
.Bitcoin ATM scans the QR code.
image::images/mtln_0205.png["Bitcoin ATM scans the QR code"]
Alice will see the transaction from the ATM in the TRANSACTION HISTORY tab of the Eclair wallet. Although Eclair will detect the bitcoin transaction in just a few seconds, it will take approximately one hour for the bitcoin transaction to be "confirmed" on the Bitcoin blockchain. As you can see in <<eclair-tx1>>, Alice's Eclair wallet shows "6+ conf" below the transaction, indicating that the transaction has received the required minimum of six confirmations, and her funds are now ready to use.
[TIP]
====
The number of confirmations on a transaction is the number of blocks mined since (and inclusive of) the block that contained that transaction. Six confirmations is best practice, but different Lightning wallets can consider a channel open after any number of confirmations. Some wallets even scale up the number of expected confirmations by the monetary value of the channel.
====
[[eclair-tx1]]
.Alice receives bitcoin
image::images/mtln_0206.png["Bitcoin transaction received"]
Although in this example Alice used an ATM to acquire her first bitcoin, the same basic concepts would apply even if she used one of the other methods in <<acquiring-bitcoin>>. For example, if Alice wanted to sell a product or provide a professional service in exchange for bitcoin, her customers could scan the Bitcoin address with their wallets and pay her in bitcoin.
Similarly, if she billed a client for a service offered over the internet, Alice could send an email or instant message with the Bitcoin address or the QR code to her client, and they could paste or scan the information into a Bitcoin wallet to pay her.
Alice could even print the QR code and affix it to a sign and display it publicly to receive tips. For example, she could have a QR code affixed to her guitar and receive tips while performing on the street!footnote:[It is generally not advisable to reuse the same Bitcoin address for multiple payments because all Bitcoin transactions are public.
A nosy person passing by could scan Alice's QR code and see how many tips Alice has already received to this address on the Bitcoin blockchain.
Fortunately, the Lightning Network offers more private solutions to this, discussed later in the book!]
Finally, if Alice bought bitcoin from a cryptocurrency exchange, she could (and should) "withdraw" the bitcoin by pasting her Bitcoin address into the exchange website. The exchange will then send the bitcoin to her address directly.
=== From Bitcoin to Lightning Network
Alice's bitcoin is now controlled by her Eclair wallet and has been recorded on the Bitcoin blockchain. At this point, Alice's bitcoin is _on-chain_, meaning that the transaction has been broadcast to the entire Bitcoin network, verified by all Bitcoin nodes, and _mined_ (recorded) onto the Bitcoin blockchain.
So far, the Eclair Mobile wallet has behaved only as a Bitcoin wallet, and Alice hasn't used the Lightning Network features of Eclair. As is the case with many Lightning wallets, Eclair bridges Bitcoin and the Lightning Network by acting as both a Bitcoin wallet and a Lightning wallet.
Now, Alice is ready to start using the Lightning Network by taking her bitcoin off-chain to take advantage of the fast, cheap, and private payments that the Lightning Network offers.
==== Lightning Network Channels
Swiping right, Alice accesses the LIGHTNING CHANNELS section of Eclair. Here she can manage the channels that will connect her wallet to the Lightning Network.
Let's review the definition of an LN channel at this point, to make things a bit clearer. Firstly, the word "channel" is a metaphor for a _financial relationship_ between Alice's Lightning wallet and another Lightning wallet. We call it a channel because it is a means for Alice's wallet and this other wallet to exchange many payments with each other on the Lightning Network (off-chain) without committing transactions to the Bitcoin blockchain (on-chain).
The wallet or _node_ that Alice opens a channel to is called her _channel peer_. Once "opened," a channel can be used to send many payments back and forth between Alice's wallet and her channel peer.
Furthermore, Alice's channel peer can _forward_ payments via other channels further into the Lightning Network. This way, Alice can _route_ a payment to any wallet (e.g., Bob's Lightning wallet) as long as Alice's wallet can find a viable _path_ made by hopping from channel to channel, all the way to Bob's wallet.
[TIP]
====
Not all channel peers are _good_ peers for routing payments. Well-connected peers will be able to route payments over shorter paths to the destination, increasing the chance of success. Channel peers with ample funds will be able to route larger payments.
====
In other words, Alice needs one or more channels that connects her to one or more other nodes on the Lightning Network. She doesn't need a channel to connect her wallet directly to Bob's Cafe in order to send Bob a payment, though she can choose to open a direct channel, too. Any node in the Lightning Network can be used for Alice's first channel. The more well-connected a node is, the more people Alice can reach. In this example, since we want to also demonstrate payment routing, we won't have Alice open a channel directly to Bob's wallet. Instead, we will have Alice open a channel to a well-connected node and then later use that node to forward her payment, routing it through any other nodes as necessary to reach Bob.
At first, there are no open channels, so as we see in <<eclair-channels>>, the LIGHTNING CHANNELS tab displays an empty list. If you notice, on the bottom right corner, there is a plus symbol (+), which is a button to open a new channel.
[[eclair-channels]]
.Lightning Channels Tab
image::images/mtln_0207.png["Lightning Channels Tab"]
Alice presses the plus symbol and is presented with four possible ways to open a channel:
* Paste a node URI
* Scan a node URI
* Random node
* ACINQ node
A "node URI" is a Universal Resource Identifier (URI) that identifies a specific Lightning node. Alice can either paste such a URI from her clipboard or scan a QR code containing that same information. An example of a node URI is shown as a QR code in <<node-URI-QR>> and then as a text string.
[[node-URI-QR]]
.Node URI as a QR code
image::images/mtln_0208.png["Lightning node URI QR code",width=120]
[[node-URI-example]]
.node URI
----
0237fefbe8626bf888de0cad8c73630e32746a22a2c4faa91c1d9877a3826e1174@1.ln.aantonop.com:9735
----
While Alice could select a specific Lightning node, or use the "Random node" option to have the Eclair wallet select a node at random, she will select the ACINQ Node option to connect to one of ACINQ's well-connected Lightning nodes.
Choosing the ACINQ node will slightly reduce Alice's privacy, because it will give ACINQ the ability to see all of Alice's transactions. It will also create a single point of failure, since Alice will only have one channel, and if the ACINQ node is not available, Alice will not be able to make payments. To keep things simple at first, we will accept these trade-offs. In subsequent chapters, we will gradually learn how to gain more independence and make fewer trade-offs!
Alice selects ACINQ Node and is ready to open her first channel on the Lightning Network.
==== Opening a Lightning Channel
When Alice selects a node to open a new channel, she is asked to select how much bitcoin she wants to allocate to this channel. In subsequent chapters, we will discuss the implications of these choices, but for now, Alice will allocate almost all her funds to the channel. Since she will have to pay transaction fees to open the channel, she will select an amount slightly less than her total balance footnote:[The Eclair wallet doesn't offer an option to automatically calculate the necessary fees and allocate the maximum amount of funds to a channel, so Alice has to calculate this herself.]
Alice allocates 0.018BTC of her 0.020 total to her channel and accepts the default fee rate, as shown in <<eclair-open-channel>>.
[[eclair-open-channel]]
.Opening a Lightning channel
image::images/mtln_0209.png["Opening a Lightning Channel"]
Once she clicks OPEN, her wallet constructs the special Bitcoin transaction that opens a Lightning channel, known as the _funding transaction_. The on-chain funding transaction is sent to the Bitcoin network for confirmation.
Alice now has to wait again (see <<eclair-channel-waiting>>) for the transaction to be recorded on the Bitcoin blockchain. As with the initial Bitcoin transaction that she used to acquire her bitcoin, she has to wait for six or more confirmations (approximately one hour).
[[eclair-channel-waiting]]
.Waiting for the funding transaction to open the channel
image::images/mtln_0210.png["Waiting for the Funding Transaction to Open the Channel"]
Once the funding transaction is confirmed, Alice's channel to the ACINQ node is open, funded, and ready, as shown in <<eclair-channel-open>>.
[[eclair-channel-open]]
.Channel is open
image::images/mtln_0211.png["Channel is Open"]
[TIP]
====
Did you notice that the channel amount seems to have changed? It hasn't: the channel contains 0.018 BTC, but in the time between screenshots the BTC exchange rate changed, so the USD value is different. You can choose to show balances in BTC or USD, but keep in mind that USD values are calculated in real time and will change!
====
=== Buying a Cup of Coffee Using Lightning Network
Alice now has everything ready to start using the Lightning Network. As you can see, it took a bit of work and a bit of time waiting for confirmations. However, now subsequent actions are fast and easy. The Lightning Network enables payments without having to wait for confirmations, as funds get settled in seconds.
Alice grabs her mobile device and runs to Bob's Cafe in her neighborhood. She is excited to try her new Lightning wallet and use it to buy something!
==== Bob's Cafe
Bob has a simple point-of-sale (PoS) application for the use of any customer who wants to pay with bitcoin over the Lightning Network. As we will see in the next chapter, Bob uses the popular open source platform _BTCPay Server_ which contains all the necessary components for an ecommerce or retail solution, such as:
* A Bitcoin Node using the Bitcoin Core software
* A Lightning Node using the c-lightning software
* A simple PoS application for a tablet
BTCPay Server makes it simple to install all the necessary software, upload pictures and product prices, and launch a store quickly.
On the counter at Bob's Cafe, there is a tablet device showing <<bob-cafe-posapp>>.
[[bob-cafe-posapp]]
.Bob's point-of-sale application
image::images/mtln_0212.png["Bob's Point-of-Sale Application"]
==== A Lightning Invoice
Alice selects the Cafe Latte option from the screen and is presented with a _Lightning invoice_ (also known as a "payment request") as shown in <<bob-cafe-invoice>>.
[[bob-cafe-invoice]]
.Lightning invoice for Alice's latte
image::images/mtln_0213.png["BTCPay Server Lightning invoice"]
To pay the invoice, Alice opens her Eclair wallet and selects the Send button (which looks like a right-facing arrow) under the TRANSACTION HISTORY tab, as shown in <<alice-send-start>>.
[[alice-send-start]]
.Alice Send
image::images/mtln_0214.png["Lightning transaction send",width=300]
[TIP]
====
The term "payment request" can refer to a Bitcoin payment request or a Lightning invoice, and the terms "invoice" and "payment request" are often used interchangeably. The correct technical term is "Lightning invoice," regardless of how it is named in the wallet.
====
Alice selects the option to "scan a payment request" and scans the QR code displayed on the screen of the tablet (see <<bob-cafe-invoice>>), and is prompted to confirm her payment, as shown in <<alice-send-detail>>.
[[alice-send-detail]]
.Alice's send confirmation
image::images/mtln_0215.png["Lightning transaction send confirmation",width=300]
Alice presses PAY, and a second later, Bob's tablet shows a successful payment. Alice has completed her first LN payment! It was fast, inexpensive, and easy. Now she can enjoy her latte which was purchased using bitcoin through a payment system that is fast, cheap, and decentralized. From now on, Alice can simply select an item on Bob's tablet screen, scan the QR code with her cell phone, click pay, and will be served a coffee, all within seconds and all without an on-chain transaction.
Lightning payments are better for Bob too. He's confident that he will be paid for Alice's latte without waiting for an on-chain confirmation. In the future, whenever Alice feels like drinking a coffee at Bob's Cafe she can choose to pay with bitcoin on the Bitcoin network or the Lightning Network. Which one do you think she will choose?
=== Conclusion
In this chapter, we followed Alice as she downloaded and installed her first Lightning wallet, acquired and transferred some bitcoin, opened her first Lightning channel, and bought a cup of coffee by making her first payment on the Lightning Network. In the following chapters, we will look "under the covers" at how each component in the Lightning Network works and how Alice's payment reached Bob's Cafe.
[[getting-started]]
== Getting Started
((("Lightning Network (generally)","example", id="ix_02_getting_started-asciidoc0", range="startofrange")))In this chapter, we will begin where most people start when encountering the Lightning Network for the first time&#x2014;choosing software to participate in the LN economy. We will examine the choices of two users who represent a common use-case for the Lightning Network and learn by example. Alice, a coffee shop customer, will be using a Lightning wallet on her mobile device to buy coffee from Bob's Cafe. Bob, a merchant, will be using a Lightning node and wallet to run a point-of-sale system at his cafe, so he can accept payments over the Lightning Network.
=== Alice's First Lightning Wallet
((("Lightning Network (generally)","Lightning wallet")))((("Lightning wallet")))Alice is a longtime Bitcoin user. We first met Alice in Chapter 1 of _"Mastering Bitcoin,"_ footnote:[_Mastering Bitcoin_, 2nd Edition, https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch01.asciidoc[Chapter 1] (O'Reilly) by Andreas M. Antonopoulos] when she bought a cup of coffee from Bob's cafe using a Bitcoin transaction. If you are not yet familiar with how Bitcoin transactions work or need a refresher, please read _Mastering Bitcoin_ or the summary in <<bitcoin_fundamentals_review>>.
Alice recently learned that Bob's Cafe just started accepting LN payments! Alice is eager to learn about and experiment with the Lightning Network; she wants to be one of Bob's first LN customers. To do this, first, Alice has to select a Lightning wallet that meets her needs.
Alice does not want to entrust custody of her bitcoin to third parties. She has learned enough about cryptocurrency to know how to use a wallet. She also wants a mobile wallet so that she can use it for small payments on-the-go, so she chooses the _Eclair_ wallet, a popular noncustodial mobile Lightning wallet. Let's learn more about how and why she's made these choices.
=== Lightning Nodes
((("Lightning node operation")))The Lightning Network is accessed via software applications that can speak the LN protocol. A _Lightning Network node_ (LN node or simply node) is a software application that has three important characteristics. First, Lightning nodes are wallets so they send and receive payments over the Lightning Network as well as on the Bitcoin network. Second, nodes must communicate on a peer-to-peer basis with other Lightning nodes creating the network. Finally, Lightning nodes also need access to the Bitcoin blockchain (or other blockchains for other cryptocurrencies) to secure the funds used for payments.
Users have the highest degree of control by running their own Bitcoin node and Lightning node. However, ((("simplified payment verification (SPV)")))((("SPV (simplified payment verification)")))Lightning nodes can also use a lightweight Bitcoin client, commonly referred to as simplified payment verification (SPV), to interact with the Bitcoin blockchain.
[[ln_explorer]]
=== Lightning Explorers
((("Lightning explorers")))LN explorers are useful tools to show the statistics of nodes, channels, and network capacity.
Following is an inexhaustive list (in alphanumerical order):
* https://1ml.com/1ML[Lightning explorer]
* https://explorer.acinq.co/[ACINQ's Lightning explorer], with fancy visualization
* https://amboss.space/[Amboss Space Lightning explorer], with community metrics and intuitive visualizations
* https://ln.bigsun.xyz/[Fiatjaf's Lightning explorer] with many diagrams
* https://hashxp.org/lightning/node/[hashXP Lightning explorer]
[TIP]
====
Note that when using Lightning explorers, just like with other block explorers, privacy can be a concern.
If users are careless, the website may track their IP addresses and collect their behavior records (for example, the nodes users are interested in).
Also, it should be noted that because there is no global consensus of the current Lightning graph or the current state of any existing channel policy, users should never rely on Lightning explorers to retrieve the most current information.
Furthermore, as users open, close, and update channels, the graph will change and individual Lightning explorers may not be up to date.
Use Lightning explorers to visualize the network or gather information, but not as an authoritative source of what is happening on the Lightning Network.
To have an authoritative view of the Lightning Network, run your own Lightning node that will build a channel graph and collect various statistics, which you can view with a web-based interface.
====
=== Lightning Wallets
((("Lightning wallet","basics", id="ix_02_getting_started-asciidoc1", range="startofrange")))The term _Lightning wallet_ is somewhat ambiguous because it can describe a broad variety of components combined with some user interface. The most common components of Lightning wallet software include:
* A keystore that holds secrets, such as private keys
* A LN node (Lightning node) that communicates on the peer-to-peer network, as described previously
* A Bitcoin node that stores blockchain data and communicates with other Bitcoin nodes
* A database "map" of nodes and channels that are announced on the Lightning Network
* A channel manager that can open and close LN channels
* A close up system that can find a path of connected channels from payment source to payment destination
A Lightning wallet may contain all of these functions, acting as a "full" wallet, with no reliance on any third-party services. Or one or more of these components may rely (partially or entirely) on third-party services that mediate those functions.
A _key_ distinction (pun intended) is whether the keystore function is internal or outsourced. In blockchains, control of keys determines custody of funds, as memorialized by the phrase "your keys, your coins; not your keys, not your coins." ((("custodial wallet")))Any wallet that outsources management of keys is called a _custodial_ wallet because a third party acting as custodian has control of the user's funds, not the user. ((("noncustodial wallet")))A _noncustodial_ or ((("self-custody wallet")))_self-custody_ wallet, by comparison, is one where the keystore is part of the wallet, and keys are controlled directly by the user. The term noncustodial wallet just implies that the keystore is local and under the user's control. However, one or more of the other wallet components may or may not be outsourced and rely on trusted third parties.
Blockchains, especially open blockchains like Bitcoin, attempt to minimize or eliminate trust in third parties and empower users. ((("trustless systems","blockchains as")))This is often called a "trustless" model, though "trust-minimized" is a better term. In such systems, the user trusts the software rules, not third parties. Therefore, the issue of control over keys is a principal consideration when choosing a Lightning wallet.
Every other component of a Lightning wallet brings similar considerations of trust. If all the components are under the control of the user, then the amount of trust in third parties is minimized, bringing maximum power to the user. Of course, this brings a direct trade-off because with that power comes the corresponding responsibility to manage complex software.
Every user must consider their own technical skills before deciding what type of Lightning wallet to use. Those with strong technical skills should use a Lightning wallet that puts all of the components under the direct control of the user. Those with fewer technical skills, but with a desire to control their funds, should choose a noncustodial Lightning wallet.
Often the trust in these cases relates to privacy.
If users decide to outsource some functionality to a third party, they usually give up some privacy as the third party will learn some information about them.
Finally, those seeking simplicity and convenience, even at the expense of control and security, may choose a custodial Lightning wallet. This is the least technically challenging option, but it _undermines the trust model of cryptocurrency_ and should therefore be considered only as a stepping stone toward more control and self-reliance.
There are many ways wallets can be characterized or categorized.
The most important questions to ask about a specific wallet are:
. Does this Lightning wallet have a full Lightning node or does it use a third-party Lightning node?
. Does this Lightning wallet have a full Bitcoin node or does it use a third-party Bitcoin node?
. Does this Lightning wallet store its own keys under user control (self-custody) or are the keys held by a third-party custodian?
[TIP]
====
If a Lightning wallet uses a third-party Lightning node, it is this third-party Lightning node that decides how to communicate with Bitcoin. Hence, using a third-party Lightning node implies that you are also using a third-party Bitcoin node. Only when the Lightning wallet uses its own Lightning node does the choice between full Bitcoin node and third-party Bitcoin node exist.
====
At the highest level of abstraction, Questions 1 and 3 are the most elementary ones.
From these two questions, we can derive four possible categories.
We can place these four categories into a quadrant as seen in <<lnwallet-categories>>.
But remember that this is just one way of categorizing Lightning wallets.
[[lnwallet-categories]]
.Lightning wallets quadrant
[options="header"]
|===
| | *Full Lightning node* | *3rd-party Lightning node*
| *Self-Custody* | Q1: High technical skill, least trust in 3rd parties, most permissionless | Q2: Below medium technical skills, below medium trust in 3rd parties, requires some permissions
| *Custodial* | Q3: Above medium technical skills, above medium trust in 3rd parties, requires some permissions | Q4: Low technical skills, high trust in 3rd parties, least permissionless
|===
Quadrant 3 (Q3), where a full Lightning node is used, but the keys are held by a custodian, is currently not common.
Future wallets from that quadrant may let a user worry about the operational aspects of their node, but then delegate access to the keys to a third party which primarily uses cold storage.
Lightning wallets can be installed on a variety of devices, including laptops, servers, and mobile devices. To run a full Lightning node you will need to use a server or desktop computer because mobile devices and laptops are usually not powerful enough in terms of capacity, processing, battery life, and connectivity.
The category third-party Lightning nodes can again be subdivided:
Lightweight::
This means that the wallet does not operate a Lightning node and thus needs to obtain information about the Lightning Network over the internet from someone else's Lightning node.
None::
This means that not only is the Lightning node operated by a third party, but most of the wallet is operated by a third party in the cloud. This is a custodial wallet where someone else controls custody of the funds.
These subcategories are used in <<lnwallet-examples>>.
Other terms that need explanation in <<lnwallet-examples>> in the column "Bitcoin node" are:
Neutrino::
This wallet does not operate a Bitcoin node. Instead, a Bitcoin node operated by someone else (a third party) is accessed via the Neutrino Protocol.
Electrum::
This wallet does not operate a Bitcoin node. Instead, a Bitcoin node operated by someone else (a third party) is accessed via the Electrum protocol.
Bitcoin Core::
This is an implementation of a Bitcoin node.
btcd::
This is another implementation of a Bitcoin node.
In <<lnwallet-examples>>, we see some examples of currently popular Lightning node and wallet applications for different types of device. The list is sorted first by device type and then alphabetically.
[[lnwallet-examples]]
.Examples of popular Lightning wallets
[options="header"]
|===
| Application | Device | Lightning node | Bitcoin node | Keystore
| Blue Wallet | Mobile | None | None | Custodial
| Breez Wallet | Mobile | Full node | Neutrino | Self-Custody
| Eclair Mobile | Mobile | Lightweight | Electrum | Self-Custody
| lntxbot | Mobile | None | None | Custodial
| Muun | Mobile | Lightweight | Neutrino | Self-Custody
| Phoenix Wallet | Mobile | Lightweight | Electrum | Self-Custody
| Zeus | Mobile | Full node | Bitcoin Core/btcd | Self-Custody
| Electrum | Desktop | Full node | Bitcoin Core/Electrum | Self-Custody
| Zap Desktop | Desktop | Full node | Neutrino | Self-Custody
| c-lightning | Server | Full node | Bitcoin Core | Self-Custody
| Eclair Server | Server | Full node | Bitcoin Core/Electrum | Self-Custody
| lnd | Server | Full node | Bitcoin Core/btcd | Self-Custody
|===
[[testnet-bitcoin]]
==== Testnet Bitcoin
((("Lightning wallet","testnet bitcoin and")))((("testnet bitcoin (TBTC)")))The Bitcoin system offers an alternative chain for testing purposes called _testnet_, in contrast with the "normal" Bitcoin chain which is referred to as _mainnet_. On testnet, the currency is _testnet bitcoin_ (_TBTC_), which is a worthless copy of bitcoin used exclusively for testing. Every function of Bitcoin is replicated exactly, but the money is worth nothing, so you literally have nothing to lose!
Some Lightning wallets can also operate on testnet, allowing you to make Lightning payments with testnet bitcoin, without risking real funds. This is a great way to experiment with Lightning safely. Eclair Mobile, which Alice uses in this chapter, is one example of a Lightning wallet that supports testnet operation.
You can get some TBTC to play with from a _testnet bitcoin faucet_, which gives out free TBTC on demand. Here are a few testnet faucets:
++++
<ul class="simplelist">
<li><a href="https://coinfaucet.eu/en/btc-testnet/"><em>https://coinfaucet.eu/en/btc-testnet</em></a></li>
<li><a href="https://testnet-faucet.mempool.co/"><em>https://testnet-faucet.mempool.co/</em></a></li>
<li><a href="https://bitcoinfaucet.uo1.net/"><em>https://bitcoinfaucet.uo1.net/</em></a></li>
<li><a href="https://testnet.help/en/btcfaucet/testnet"><em>https://testnet.help/en/btcfaucet/testnet</em></a></li>
</ul>
++++
All of the examples in this book can be replicated exactly on testnet with TBTC, so you can follow along if you want without risking real money.(((range="endofrange", startref="ix_02_getting_started-asciidoc1")))
=== Balancing Complexity and Control
((("Lightning wallet","balancing complexity and control")))Lightning wallets have to strike a careful balance between complexity and user control. Those that give the user the most control over their funds, the highest degree of privacy, and the greatest independence from third-party services are necessarily more complex and difficult to operate. As the technology advances, some of these trade-offs will become less stark, and users may be able to get more control without more complexity. However, for now, different companies and projects are exploring different positions along this control-complexity spectrum and hoping to find the "sweet spot" for the users they are targeting.
When selecting a wallet, keep in mind that even if you don't see these trade-offs, they still exist. For example, many wallets will attempt to remove the burden of channel management from their users. To do so, they introduce central _hub nodes_ that all their wallets connect to automatically. While this trade-off simplifies the user interface and user experience, it introduces a single point of failure (SPoF) as these hub nodes become indispensable for the wallet's operation. Furthermore, relying on a "hub" like this can reduce user privacy since the hub knows the sender and potentially (if constructing the payment route on behalf of the user) also the recipient of each payment made by the user's wallet.
In the next section, we will return to our first user and walk through her first Lightning wallet setup. She has chosen a wallet that is more sophisticated than the easier custodial wallets. This allows us to show some of the underlying complexity and introduce some of the inner workings of an advanced wallet. You may find that your first ideal wallet is oriented toward ease of use, accepting some of the control and privacy trade-offs. Or perhaps you are more of a power user and want to run your own Lightning and Bitcoin nodes as part of your wallet solution.
=== Downloading and Installing a Lightning Wallet
((("Lightning wallet","downloading/installing")))When looking for a new cryptocurrency wallet, you must be very careful to select a secure source for the software.
Unfortunately, many fake wallet applications will steal your money, and some of these even find their way onto reputable and supposedly vetted software sites like the Apple and Google application stores. Whether you are installing your first or your tenth wallet, always exercise extreme caution. A rogue app may not just steal any money you entrust it with, but it might also be able to steal keys and passwords from other applications by compromising your mobile device operating system.
((("Eclair wallet, downloading/installing")))Alice uses an Android device and will use the Google Play Store to download and install the Eclair wallet. Searching on Google Play, she finds an entry for "Eclair Mobile," as shown in <<eclair-playstore>>.
[[eclair-playstore]]
.Eclair Mobile in the Google Play Store
image::images/mtln_0201.png["Eclair wallet in the Google Play Store"]
[TIP]
====
It is possible to experiment and test all Bitcoin-type software with zero risk (except for your own time) by using testnet bitcoins. You can also download Eclair testnet wallet to try Lightning (on testnet) by going to the Google Play Store.
====
Alice notices a few different elements on this page that help her ascertain that this is, most likely, the correct "Eclair Mobile" wallet she is looking for. Firstly, the organization ACINQfootnote:[ACINQ: Developers of the Eclair Mobile Lightning wallet (https://acinq.co/).] is listed as the developer of this mobile wallet, which Alice knows from her research is the correct developer. Secondly, the wallet has been installed "10,000+" times and has more than 320 positive reviews. It is unlikely this is a rogue app that has snuck into the Google Play Store. As a third step, she goes to the https://acinq.co[ACINQ website]. She verifies that the web page is secure by checking that the address begins with https, or prefixed by a padlock in some browsers. On the website she goes to the Download section or looks for the link to the Google App Store. She finds the link and clicks it. She compares that this link brings her to the very same app in the Google App Store. Satisfied by these findings, Alice installs the Eclair app on her mobile device.
[WARNING]
====
Always exercise great care when installing software on any device. There are many fake cryptocurrency wallets that will not only steal your money but might also compromise all other applications on your device.
====
=== Creating a New Wallet
((("Lightning wallet","creating a new wallet", id="ix_02_getting_started-asciidoc2", range="startofrange")))When Alice opens the Eclair Mobile app for the first time, she is presented with a choice to "Create a New Wallet" or to "Import an Existing Wallet." Alice will create a new wallet, but let's first discuss why these options are presented here and what it means to import an existing wallet.
==== Responsibility with Key Custody
((("keys","Lightning wallet and")))((("Lightning wallet","responsibility with key custody")))As we mentioned at the beginning of this section, Eclair is a _noncustodial_ wallet, meaning that Alice has sole custody of the keys used to control her bitcoin. This also means that Alice is responsible for protecting and backing up those keys. If Alice loses the keys, no one can help her recover the bitcoin, and they will be lost forever.
[WARNING]
====
With the Eclair Mobile wallet, Alice has custody and control of the keys and, therefore, full responsibility to keep the keys safe and backed up. If she loses the keys, she loses the bitcoin, and no one can help her recover from that loss!
====
==== Mnemonic Words
((("Lightning wallet","mnemonic phrase")))((("mnemonic phrase")))((("seed (mnemonic) phrase")))Similar to most Bitcoin wallets, Eclair Mobile provides a _mnemonic phrase_ (also sometimes called a "seed" or "seed phrase") for Alice to back up. The mnemonic phrase consists of 24 English words, selected randomly by the software, and used as the basis for the keys that are generated by the wallet. The mnemonic phrase can be used by Alice to restore all the transactions and funds in the Eclair Mobile wallet in the case of an event such as a lost mobile device, a software bug, or memory corruption.
[TIP]
====
The correct term for these backup words is "mnemonic phrase." We avoid the use of the term "seed" to refer to a mnemonic phrase because even though its use is common, it is incorrect.
====
When Alice chooses to Create a New Wallet, she will be shown a screen with her mnemonic phrase, which looks like the screenshot in <<eclair-mnemonic>>.
[[eclair-mnemonic]]
.New wallet mnemonic phrase
image::images/mtln_0202.png["New Wallet Mnemonic Phrase"]
In <<eclair-mnemonic>>, we have purposely obscured part of the mnemonic phrase to prevent readers of this book from reusing the mnemonic.
[[mnemonic-storage]]
==== Storing the Mnemonic Safely
((("Lightning wallet","mnemonic phrase storage")))Alice needs to be careful to store the mnemonic phrase in a way that prevents theft but also avoids accidental loss. The recommended way to properly balance these risks is to write two copies of the mnemonic phrase on paper, with each of the words numbered&#x2014;the order matters.
Once Alice has recorded the mnemonic phrase, after touching "OK GOT IT" on her screen, she will be presented with a quiz to make sure that she correctly recorded the mnemonic. The quiz will ask for three or four of the words at random. Alice wasn't expecting a quiz, but since she recorded the mnemonic correctly, she passes without any difficulty.
Once Alice has recorded the mnemonic phrase and passed the quiz, she should store each copy in a separate secure location such as a locked desk drawer or a fireproof safe.
[WARNING]
====
Never attempt a "DIY" security scheme that deviates in any way from the best practice recommendation in <<mnemonic-storage>>. Do not cut your mnemonic in half, make screenshots, store on USB drives or cloud drives, encrypt it, or try any other nonstandard method. You will tip the balance in such a way as to risk permanent loss. Many people have lost funds, not from theft, but because they tried a nonstandard solution without having the expertise to balance the risks involved. The best practice recommendation is carefully considered by experts and suitable for the vast majority of users.
====
After Alice initializes her Eclair Mobile wallet, she will see a brief tutorial that highlights the various elements of the user interface. We won't replicate the tutorial here, but we will explore all of those elements as we follow Alice's attempt to buy a cup of coffee!(((range="endofrange", startref="ix_02_getting_started-asciidoc2")))
=== Loading Bitcoin Onto the Wallet
((("bitcoin (currency)","loading onto Lightning wallet", id="ix_02_getting_started-asciidoc3", range="startofrange")))((("Lightning wallet","loading bitcoin onto", id="ix_02_getting_started-asciidoc4", range="startofrange")))Alice now has a Lightning wallet. But it's empty! She now faces one of the more challenging aspects of this experiment; she has to find a way to acquire some bitcoin and load it onto her Eclair wallet.
[TIP]
====
If Alice already has bitcoin in another wallet, she could choose to send that bitcoin to her Eclair wallet instead of acquiring new bitcoin to load onto her new wallet.
====
[[acquiring-bitcoin]]
==== Acquiring Bitcoin
((("bitcoin (currency)","acquiring for Lightning wallet")))((("Lightning wallet","acquiring bitcoin for")))There are several ways Alice can acquire bitcoin:
* She can exchange some of her national currency (e.g., USD) on a cryptocurrency exchange.
* She can buy some from a friend, or an acquaintance from a Bitcoin meetup, in exchange for cash.
* She can find a _Bitcoin ATM_ in her area, which acts as a vending machine, selling bitcoin for cash.
* She can offer her skills or a product she sells and accept payment in bitcoin.
* She can ask her employer or clients to pay her in bitcoin.
All of these methods have varying degrees of difficulty, and many will involve paying a fee. Some will also require Alice to provide identification documents to comply with local banking regulations. However, with all these methods, Alice will be able to receive bitcoin.
==== Receiving Bitcoin
((("bitcoin (currency)","receiving for Lightning wallet", id="ix_02_getting_started-asciidoc5", range="startofrange")))((("Bitcoin ATM", id="ix_02_getting_started-asciidoc6", range="startofrange")))((("Lightning wallet","receiving bitcoin", id="ix_02_getting_started-asciidoc7", range="startofrange")))Let's assume Alice has found a local Bitcoin ATM and has decided to buy some bitcoin in exchange for cash. An example of a Bitcoin ATM, one built by the Lamassu Company, is shown in <<bitcoin-atm>>. Such Bitcoin ATMs accept national currency (cash) through a cash slot and send bitcoin to a Bitcoin address scanned from a user's wallet using a built-in camera.
[[bitcoin-atm]]
.A Lamassu Bitcoin ATM
image::images/mtln_0203.png["Lamassu Bitcoin ATM"]
To receive the bitcoin in her Eclair Lightning wallet, Alice will need to present a Bitcoin address from the Eclair Lightning wallet to the ATM. The ATM can then send Alice's newly acquired bitcoin to this Bitcoin address.
To see a Bitcoin address on the Eclair wallet, Alice must swipe to the left column titled YOUR BITCOIN ADDRESS (see <<eclair-receive>>), where she will see a square barcode (called a _QR code_) and a string of letters and numbers below that.
[[eclair-receive]]
.Alice's bitcoin address, shown in Eclair
image::images/mtln_0204.png["Eclair bitcoin address QR code"]
The QR code contains the same string of letters and numbers as shown below it, in an easy to scan format. This way, Alice doesn't have to type the Bitcoin address. In the screenshot (<<eclair-receive>>), we have purposely blurred both, to prevent readers from inadvertently sending bitcoin to this address.
[NOTE]
====
Both Bitcoin addresses and QR codes contain error detection information that prevents any typing or scanning errors from producing a "wrong" Bitcoin address. If there is a mistake in the address, any Bitcoin wallet will notice the error and refuse to accept the Bitcoin address as valid.
====
Alice can take her mobile device to the ATM and show it to the built-in camera, as shown in <<bitcoin-atm-receive>>. After inserting some cash into the slot, she will receive bitcoin in Eclair!
[[bitcoin-atm-receive]]
.Bitcoin ATM scans the QR code.
image::images/mtln_0205.png["Bitcoin ATM scans the QR code"]
Alice will see the transaction from the ATM in the TRANSACTION HISTORY tab of the Eclair wallet. Although Eclair will detect the bitcoin transaction in just a few seconds, it will take approximately one hour for the bitcoin transaction to be "confirmed" on the Bitcoin blockchain. As you can see in <<eclair-tx1>>, Alice's Eclair wallet shows "6+ conf" below the transaction, indicating that the transaction has received the required minimum of six confirmations, and her funds are now ready to use.
[TIP]
====
The number of confirmations on a transaction is the number of blocks mined since (and inclusive of) the block that contained that transaction. Six confirmations is best practice, but different Lightning wallets can consider a channel open after any number of confirmations. Some wallets even scale up the number of expected confirmations by the monetary value of the channel.
====
[[eclair-tx1]]
.Alice receives bitcoin
image::images/mtln_0206.png["Bitcoin transaction received"]
Although in this example Alice used an ATM to acquire her first bitcoin, the same basic concepts would apply even if she used one of the other methods in <<acquiring-bitcoin>>. For example, if Alice wanted to sell a product or provide a professional service in exchange for bitcoin, her customers could scan the Bitcoin address with their wallets and pay her in bitcoin.
Similarly, if she billed a client for a service offered over the internet, Alice could send an email or instant message with the Bitcoin address or the QR code to her client, and they could paste or scan the information into a Bitcoin wallet to pay her.
Alice could even print the QR code and affix it to a sign and display it publicly to receive tips. For example, she could have a QR code affixed to her guitar and receive tips while performing on the street!footnote:[It is generally not advisable to reuse the same Bitcoin address for multiple payments because all Bitcoin transactions are public.
A nosy person passing by could scan Alice's QR code and see how many tips Alice has already received to this address on the Bitcoin blockchain.
Fortunately, the Lightning Network offers more private solutions to this, discussed later in the book!]
Finally, if Alice bought bitcoin from a cryptocurrency exchange, she could (and should) "withdraw" the bitcoin by pasting her Bitcoin address into the exchange website. The exchange will then send the bitcoin to her address directly(((range="endofrange", startref="ix_02_getting_started-asciidoc7")))(((range="endofrange", startref="ix_02_getting_started-asciidoc6")))(((range="endofrange", startref="ix_02_getting_started-asciidoc5"))).(((range="endofrange", startref="ix_02_getting_started-asciidoc4")))(((range="endofrange", startref="ix_02_getting_started-asciidoc3")))
=== From Bitcoin to Lightning Network
((("Lightning wallet","bridging of Bitcoin and Lightning networks", id="ix_02_getting_started-asciidoc8", range="startofrange")))Alice's bitcoin is now controlled by her Eclair wallet and has been recorded on the Bitcoin blockchain. At this point, Alice's bitcoin is _on-chain_, meaning that the transaction has been broadcast to the entire Bitcoin network, verified by all Bitcoin nodes, and _mined_ (recorded) onto the Bitcoin blockchain.
So far, the Eclair Mobile wallet has behaved only as a Bitcoin wallet, and Alice hasn't used the Lightning Network features of Eclair. As is the case with many Lightning wallets, Eclair bridges Bitcoin and the Lightning Network by acting as both a Bitcoin wallet and a Lightning wallet.
Now, Alice is ready to start using the Lightning Network by taking her bitcoin off-chain to take advantage of the fast, cheap, and private payments that the Lightning Network offers.
==== Lightning Network Channels
((("Lightning Network channels","basics", id="ix_02_getting_started-asciidoc9", range="startofrange")))((("Lightning Network channels","opening a channel", id="ix_02_getting_started-asciidoc10", range="startofrange")))((("Lightning wallet","LN channels and", id="ix_02_getting_started-asciidoc11", range="startofrange")))Swiping right, Alice accesses the LIGHTNING CHANNELS section of Eclair. Here she can manage the channels that will connect her wallet to the Lightning Network.
Let's review the definition of an LN channel at this point, to make things a bit clearer. Firstly, the word "channel" is a metaphor for a _financial relationship_ between Alice's Lightning wallet and another Lightning wallet. We call it a channel because it is a means for Alice's wallet and this other wallet to exchange many payments with each other on the Lightning Network (off-chain) without committing transactions to the Bitcoin blockchain (on-chain).
((("channel peer")))The wallet or _node_ that Alice opens a channel to is called her _channel peer_. Once "opened," a channel can be used to send many payments back and forth between Alice's wallet and her channel peer.
Furthermore, Alice's channel peer can _forward_ payments via other channels further into the Lightning Network. This way, Alice can _route_ a payment to any wallet (e.g., Bob's Lightning wallet) as long as Alice's wallet can find a viable _path_ made by hopping from channel to channel, all the way to Bob's wallet.
[TIP]
====
Not all channel peers are _good_ peers for routing payments. Well-connected peers will be able to route payments over shorter paths to the destination, increasing the chance of success. Channel peers with ample funds will be able to route larger payments.
====
In other words, Alice needs one or more channels that connects her to one or more other nodes on the Lightning Network. She doesn't need a channel to connect her wallet directly to Bob's Cafe in order to send Bob a payment, though she can choose to open a direct channel, too. Any node in the Lightning Network can be used for Alice's first channel. The more well-connected a node is, the more people Alice can reach. In this example, since we want to also demonstrate payment routing, we won't have Alice open a channel directly to Bob's wallet. Instead, we will have Alice open a channel to a well-connected node and then later use that node to forward her payment, routing it through any other nodes as necessary to reach Bob.
At first, there are no open channels, so as we see in <<eclair-channels>>, the LIGHTNING CHANNELS tab displays an empty list. If you notice, on the bottom right corner, there is a plus symbol (+), which is a button to open a new channel.
[[eclair-channels]]
.Lightning Channels Tab
image::images/mtln_0207.png["Lightning Channels Tab"]
Alice presses the plus symbol and is presented with four possible ways to open a channel:
* Paste a node URI
* Scan a node URI
* Random node
* ACINQ node
A "node URI" is a Universal Resource Identifier (URI) that identifies a specific Lightning node. Alice can either paste such a URI from her clipboard or scan a QR code containing that same information. An example of a node URI is shown as a QR code in <<node-URI-QR>> and then as a text string.
[[node-URI-QR]]
.Node URI as a QR code
image::images/mtln_0208.png["Lightning node URI QR code",width=120]
[[node-URI-example]]
.node URI
----
0237fefbe8626bf888de0cad8c73630e32746a22a2c4faa91c1d9877a3826e1174@1.ln.aantonop.com:9735
----
While Alice could select a specific Lightning node, or use the "Random node" option to have the Eclair wallet select a node at random, she will select the ACINQ Node option to connect to one of ACINQ's well-connected Lightning nodes.
Choosing the ACINQ node will slightly reduce Alice's privacy, because it will give ACINQ the ability to see all of Alice's transactions. It will also create a single point of failure, since Alice will only have one channel, and if the ACINQ node is not available, Alice will not be able to make payments. To keep things simple at first, we will accept these trade-offs. In subsequent chapters, we will gradually learn how to gain more independence and make fewer trade-offs!
Alice selects ACINQ Node and is ready to open her first channel on the Lightning Network.(((range="endofrange", startref="ix_02_getting_started-asciidoc11")))(((range="endofrange", startref="ix_02_getting_started-asciidoc10")))(((range="endofrange", startref="ix_02_getting_started-asciidoc9")))
==== Opening a Lightning Channel
((("Lightning wallet","opening a Lightning channel", id="ix_02_getting_started-asciidoc12", range="startofrange")))When Alice selects a node to open a new channel, she is asked to select how much bitcoin she wants to allocate to this channel. In subsequent chapters, we will discuss the implications of these choices, but for now, Alice will allocate almost all her funds to the channel. Since she will have to pay transaction fees to open the channel, she will select an amount slightly less than her total balance footnote:[The Eclair wallet doesn't offer an option to automatically calculate the necessary fees and allocate the maximum amount of funds to a channel, so Alice has to calculate this herself.]
Alice allocates 0.018BTC of her 0.020 total to her channel and accepts the default fee rate, as shown in <<eclair-open-channel>>.
[[eclair-open-channel]]
.Opening a Lightning channel
image::images/mtln_0209.png["Opening a Lightning Channel"]
Once she clicks OPEN, her wallet constructs the special Bitcoin transaction that ((("funding transaction")))opens a Lightning channel, known as the _funding transaction_. The on-chain funding transaction is sent to the Bitcoin network for confirmation.
Alice now has to wait again (see <<eclair-channel-waiting>>) for the transaction to be recorded on the Bitcoin blockchain. As with the initial Bitcoin transaction that she used to acquire her bitcoin, she has to wait for six or more confirmations (approximately one hour).
[[eclair-channel-waiting]]
.Waiting for the funding transaction to open the channel
image::images/mtln_0210.png["Waiting for the Funding Transaction to Open the Channel"]
Once the funding transaction is confirmed, Alice's channel to the ACINQ node is open, funded, and ready, as shown in <<eclair-channel-open>>.
[[eclair-channel-open]]
.Channel is open
image::images/mtln_0211.png["Channel is Open"]
[TIP]
====
Did you notice that the channel amount seems to have changed? It hasn't: the channel contains 0.018 BTC, but in the time between screenshots the BTC exchange rate changed, so the USD value is different. You can choose to show balances in BTC or USD, but keep in mind that USD values are calculated in real time and will change(((range="endofrange", startref="ix_02_getting_started-asciidoc12")))!(((range="endofrange", startref="ix_02_getting_started-asciidoc8")))
====
=== Buying a Cup of Coffee Using Lightning Network
((("Lightning Network (generally)","example: buying a cup of coffee", id="ix_02_getting_started-asciidoc13", range="startofrange")))((("Lightning wallet","example: buying a cup of coffee", id="ix_02_getting_started-asciidoc14", range="startofrange")))Alice now has everything ready to start using the Lightning Network. As you can see, it took a bit of work and a bit of time waiting for confirmations. However, now subsequent actions are fast and easy. The Lightning Network enables payments without having to wait for confirmations, as funds get settled in seconds.
Alice grabs her mobile device and runs to Bob's Cafe in her neighborhood. She is excited to try her new Lightning wallet and use it to buy something!
==== Bob's Cafe
Bob has a simple point-of-sale (PoS) application for the use of any customer who wants to pay with bitcoin over the Lightning Network. As we will see in the next chapter, Bob uses the popular open source platform _BTCPay Server_ which contains all the necessary components for an ecommerce or retail solution, such as:
* A Bitcoin Node using the Bitcoin Core software
* A Lightning Node using the c-lightning software
* A simple PoS application for a tablet
BTCPay Server makes it simple to install all the necessary software, upload pictures and product prices, and launch a store quickly.
On the counter at Bob's Cafe, there is a tablet device showing <<bob-cafe-posapp>>.
[[bob-cafe-posapp]]
.Bob's point-of-sale application
image::images/mtln_0212.png["Bob's Point-of-Sale Application"]
==== A Lightning Invoice
((("Lightning invoices", id="ix_02_getting_started-asciidoc15", range="startofrange")))((("Lightning wallet","invoices", id="ix_02_getting_started-asciidoc16", range="startofrange")))Alice selects the Cafe Latte option from the screen and is presented with a _Lightning invoice_ (also known as a "payment request") as shown in <<bob-cafe-invoice>>.
[[bob-cafe-invoice]]
.Lightning invoice for Alice's latte
image::images/mtln_0213.png["BTCPay Server Lightning invoice"]
To pay the invoice, Alice opens her Eclair wallet and selects the Send button (which looks like a right-facing arrow) under the TRANSACTION HISTORY tab, as shown in <<alice-send-start>>.
[[alice-send-start]]
.Alice Send
image::images/mtln_0214.png["Lightning transaction send",width=300]
[TIP]
====
The term "payment request" can refer to a Bitcoin payment request or a Lightning invoice, and the terms "invoice" and "payment request" are often used interchangeably. The correct technical term is "Lightning invoice," regardless of how it is named in the wallet.
====
Alice selects the option to "scan a payment request" and scans the QR code displayed on the screen of the tablet (see <<bob-cafe-invoice>>), and is prompted to confirm her payment, as shown in <<alice-send-detail>>.
[[alice-send-detail]]
.Alice's send confirmation
image::images/mtln_0215.png["Lightning transaction send confirmation",width=300]
Alice presses PAY, and a second later, Bob's tablet shows a successful payment. Alice has completed her first LN payment! It was fast, inexpensive, and easy. Now she can enjoy her latte which was purchased using bitcoin through a payment system that is fast, cheap, and decentralized. From now on, Alice can simply select an item on Bob's tablet screen, scan the QR code with her cell phone, click pay, and will be served a coffee, all within seconds and all without an on-chain transaction.
Lightning payments are better for Bob too. He's confident that he will be paid for Alice's latte without waiting for an on-chain confirmation. In the future, whenever Alice feels like drinking a coffee at Bob's Cafe she can choose to pay with bitcoin on the Bitcoin network or the Lightning Network. Which one do you think she will choose(((range="endofrange", startref="ix_02_getting_started-asciidoc16")))(((range="endofrange", startref="ix_02_getting_started-asciidoc15")))?(((range="endofrange", startref="ix_02_getting_started-asciidoc14")))(((range="endofrange", startref="ix_02_getting_started-asciidoc13")))
=== Conclusion
In this chapter, we followed Alice as she downloaded and installed her first Lightning wallet, acquired and transferred some bitcoin, opened her first Lightning channel, and bought a cup of coffee by making her first payment on the Lightning Network.(((range="endofrange", startref="ix_02_getting_started-asciidoc0"))) In the following chapters, we will look "under the covers" at how each component in the Lightning Network works and how Alice's payment reached Bob's Cafe.

1552
03_how_ln_works.asciidoc
View File

File diff suppressed because it is too large Load Diff

2262
04_node_client.asciidoc
View File

File diff suppressed because it is too large Load Diff

1938
05_node_operations.asciidoc
View File

File diff suppressed because it is too large Load Diff

110
06_lightning_architecture.asciidoc
Unescape Escape View File

@ -1,55 +1,55 @@
== Lightning Network Architecture
In the first part of this book we introduced the main concepts of the Lightning Network and worked through a comprehensive example of routing a payment and setting up the tools we can use to explore further. In the second part of the book we will explore the Lightning Network in a lot more technical detail, dissecting each of the building blocks.
In this section we will outline the components of the Lightning Network in more detail and provide a "big picture" perspective to guide you through the following chapters.
=== The Lightning Network Protocol Suite
The Lightning Network is composed of a complex collection of protocols that run on top of the internet. We can broadly classify these protocols into five distinct layers that make up a _protocol stack_, where each layer builds upon and uses the protocols in the layer below. Also, each protocol layer abstracts the underlying layers and "hides" some of the complexity.
The architecture diagram shown in <<lightning_network_protocol_suite>> provides an overview of these layers and their component protocols.
[[lightning_network_protocol_suite]]
.The Lightning Network protocol suite
image::images/mtln_0601.png[]
The five layers of the Lightning Network, from the bottom up, are:
Network connection layer:: This contains the protocols that interact directly with the internet core protocols (TCP/IP), overlay protocols (Tor v2/v3), and internet services (DNS). This layer also contains the cryptographic transport protocols that protect Lightning messages.
Messaging layer:: This layer contains the protocols that nodes use to negotiate features, format messages, and encode message fields.
Peer-to-Peer (P2P) layer:: This layer is the primary protocol layer for communication between Lightning nodes and contains all the different messages exchanged between nodes.
Routing layer:: This layer contains the protocols used to route payments between nodes, end-to-end and atomically. This layer contains the core functionality of the Lightning Network: routed payments.
Payment layer:: The highest layer of the network, which presents a reliable payment interface to applications.
=== Lightning in Detail
Over the next 10 chapters, we will diseect the protocol suite and examine each component of the Lightning Network in detail.
We spent quite some time trying to decide the best order of presenting this detail. It's not an easy choice because there is so much interdependence between different components: as you start explaining one, you find that it pulls in quite a few of the other componenents. Instead of a top-down or bottom-up approach, we ended up choosing a more meandering path that starts with the most fundamental building blocks that are unique to the Lightning Network-Payment Channels and moves outward from there. But since that path is not obvious, we will use the Lightning Protocol Suite shown in <<lightning_network_protocol_suite>> as a map. In each chapter will focus on one or more related components, and you will see them highlighted in the protocol suite. Kind of like a map marker saying "You are here!".
Here's what we will cover:
<<payment_channels>>:: In this chapter we will look at how payment channels work, in significantly more depth than we saw in the earlier parts of the book. We will look at the structure and Bitcoin Script of the funding and commitment transactions and the process used by nodes to negotiate each step in the protocol.
<<routing>>:: Next, we will put together several payment channels in a network and route a payment from one end to the other. In that process we will dive into the hash time-locked contract smart contract and the Bitcoin Script that we use to construct it.
<<channel_operation>>:: Putting together the concepts of a simple payment channel and a routed payment using HTLCs, we will now look at how HTLCs are part of each channel's commitment transaction. We will also look at the protocol for adding, settling, failing, and removing HTLCs from the commitments.
<<onion_routing>>:: Next, we will look at how the HTLC information is propagated across the network inside the onion routing protocol. We will look at the mechanism for layered encryption and decryption that gives the Lightning Network some of its privacy characteristics.
<<gossip>>:: In this chapter we will look at how Lightning nodes find each other and learn about published channels to construct a channel graph that they can use to find paths across the network.
<<path_finding>>:: Next, we will see how the information from the gossip protocol is used by each node to build a "map" of the entire network, which it can use to find paths from one point to another to route payments. We'll also look at the exiting innovations in pathfinding such as multipart payments.
<<invoices>>:: A key part of the Lightning Network is payment requests, also known as Lightning Invoices. In this chapter we dissect the structure and encoding of an invoice.
<<wire_protocol>>:: Underpinning the Lightning Network is the peer-to-peer protocol that nodes use to exchange messages about the network and about their channels. In this chapter we look at how those messages are constructed and the extension capabilities built into messages with feature bits and Type-Length-Value (TLV) encoding.
<<encrypted_message_transport>>:: Moving down to the lower-level part of the network, we will look at the underlying encrypted transport system that ensures the secrecy and integrity of all communications between nodes.
Let's dive in!
== Lightning Network Architecture
((("architecture, Lightning Network", id="ix_06_lightning_architecture-asciidoc0", range="startofrange")))In the first part of this book we introduced the main concepts of the Lightning Network and worked through a comprehensive example of routing a payment and setting up the tools we can use to explore further. In the second part of the book we will explore the Lightning Network in a lot more technical detail, dissecting each of the building blocks.
In this section we will outline the components of the Lightning Network in more detail and provide a "big picture" perspective to guide you through the following chapters.
=== The Lightning Network Protocol Suite
((("architecture, Lightning Network","protocol suite")))((("protocol stack")))The Lightning Network is composed of a complex collection of protocols that run on top of the internet. We can broadly classify these protocols into five distinct layers that make up a _protocol stack_, where each layer builds upon and uses the protocols in the layer below. Also, each protocol layer abstracts the underlying layers and "hides" some of the complexity.
The architecture diagram shown in <<lightning_network_protocol_suite>> provides an overview of these layers and their component protocols.
[[lightning_network_protocol_suite]]
.The Lightning Network protocol suite
image::images/mtln_0601.png[]
((("architecture, Lightning Network","layers")))The five layers of the Lightning Network, from the bottom up, are:
Network connection layer:: This contains the protocols that interact directly with the internet core protocols (TCP/IP), overlay protocols (Tor v2/v3), and internet services (DNS). This layer also contains the cryptographic transport protocols that protect Lightning messages.
Messaging layer:: This layer contains the protocols that nodes use to negotiate features, format messages, and encode message fields.
Peer-to-Peer (P2P) layer:: This layer is the primary protocol layer for communication between Lightning nodes and contains all the different messages exchanged between nodes.
Routing layer:: This layer contains the protocols used to route payments between nodes, end-to-end and atomically. This layer contains the core functionality of the Lightning Network: routed payments.
Payment layer:: The highest layer of the network, which presents a reliable payment interface to applications.
=== Lightning in Detail
((("architecture, Lightning Network","outline of details")))Over the next 10 chapters, we will diseect the protocol suite and examine each component of the Lightning Network in detail.
We spent quite some time trying to decide the best order of presenting this detail. It's not an easy choice because there is so much interdependence between different components: as you start explaining one, you find that it pulls in quite a few of the other componenents. Instead of a top-down or bottom-up approach, we ended up choosing a more meandering path that starts with the most fundamental building blocks that are unique to the Lightning Network-Payment Channels and moves outward from there. But since that path is not obvious, we will use the Lightning Protocol Suite shown in <<lightning_network_protocol_suite>> as a map. In each chapter will focus on one or more related components, and you will see them highlighted in the protocol suite. Kind of like a map marker saying "You are here!".
Here's what we will cover:
<<payment_channels>>:: In this chapter we will look at how payment channels work, in significantly more depth than we saw in the earlier parts of the book. We will look at the structure and Bitcoin Script of the funding and commitment transactions and the process used by nodes to negotiate each step in the protocol.
<<routing>>:: Next, we will put together several payment channels in a network and route a payment from one end to the other. In that process we will dive into the hash time-locked contract smart contract and the Bitcoin Script that we use to construct it.
<<channel_operation>>:: Putting together the concepts of a simple payment channel and a routed payment using HTLCs, we will now look at how HTLCs are part of each channel's commitment transaction. We will also look at the protocol for adding, settling, failing, and removing HTLCs from the commitments.
<<onion_routing>>:: Next, we will look at how the HTLC information is propagated across the network inside the onion routing protocol. We will look at the mechanism for layered encryption and decryption that gives the Lightning Network some of its privacy characteristics.
<<gossip>>:: In this chapter we will look at how Lightning nodes find each other and learn about published channels to construct a channel graph that they can use to find paths across the network.
<<path_finding>>:: Next, we will see how the information from the gossip protocol is used by each node to build a "map" of the entire network, which it can use to find paths from one point to another to route payments. We'll also look at the exiting innovations in pathfinding such as multipart payments.
<<invoices>>:: A key part of the Lightning Network is payment requests, also known as Lightning Invoices. In this chapter we dissect the structure and encoding of an invoice.
<<wire_protocol>>:: Underpinning the Lightning Network is the peer-to-peer protocol that nodes use to exchange messages about the network and about their channels. In this chapter we look at how those messages are constructed and the extension capabilities built into messages with feature bits and Type-Length-Value (TLV) encoding.
<<encrypted_message_transport>>:: Moving down to the lower-level part of the network, we will look at the underlying encrypted transport system that ensures the secrecy and integrity of all communications between nodes.(((range="endofrange", startref="ix_06_lightning_architecture-asciidoc0")))
Let's dive in!

1556
07_payment_channels.asciidoc
View File

File diff suppressed because it is too large Load Diff

1248
08_routing_htlcs.asciidoc
View File

File diff suppressed because it is too large Load Diff

704
09_channel_operation.asciidoc
Unescape Escape View File

@ -1,352 +1,352 @@
[[channel_operation]]
== Channel Operation and Payment Forwarding
In this chapter we will bring together payment channels and hash time-locked contracts (HTLCs). In <<payment_channels>> we explained the way Alice and Bob construct a payment channel between their two nodes. We also looked at the commitment and penalty mechanisms that secure the payment channel. In <<routing>> we looked at hash time-locked contracts (HTLCs) and how these can be used to route a payment across a path made of multiple payment channels. In this chapter we bring the two concepts together by looking at how HTLCs are managed on each payment channel, how the HTLCs are committed to the channel state, and how they are settled to update the channel balances.
Specifically, we will be discussing "Adding, settling, failing HTLCs" and the "Channel state machine" that form the overlap between the peer 2 peer layer and the routing layer, as highlighted by a double outline in <<LN_protocol_channelops_highlight>>.
[[LN_protocol_channelops_highlight]]
.The Lightning Network protocol suite
image::images/mtln_0901.png["The Lightning Network Protocol Suite"]
=== Local (Single Channel) versus Routed (Multiple Channels)
Even though it is possible to send payments across a payment channel simply by updating the channel balances and creating new commitment transactions, the Lightning protocol uses HTLCs even for "local" payments across a payment channel. The reason for this is to maintain the same protocol design regardless of whether a payment is only one hop (across a single payment channel), or several hops (routed across multiple payment channels).
By maintaining the same abstraction for both local and remote, we not only simplify the protocol design but also improve privacy. For the recipient of a payment there is no discernible difference between a payment made directly by their channel partner and a payment forwarded by their channel partner on behalf of someone else.
=== Forwarding Payments and Updating Commitments with HTLCs
We will revisit our example from <<routing>> to demonstrate how the HTLCs from Alice to Dina get committed to each payment channel. As you recall in our example, Alice is paying Dina 50,000 satoshis by routing an HTLC via Bob and Chan. The network is shown in <<alice_dina_htlc_2>>.
[[alice_dina_htlc_2]]
.Alice pays Dina with an HTLC routed via Bob and Chan
image::images/mtln_0809.png["Alice pays Dina with an HTLC routed via Bob and Chan"]
We will focus on the payment channel between Alice and Bob and review the messages and transactions that they use to process this HTLC.
==== HTLC and Commitment Message Flow
The message flow between Alice and Bob (and also between any pair of channel partners) is shown in <<HTLC_commitment_message_flow>>.
[[HTLC_commitment_message_flow]]
.The message flow for HTLC commitment between channel partners
image::images/mtln_0903.png["The message flow for HTLC commitment between channel partners"]
We've already seen the +commitment_signed+ and +revoke_and_ack+ in <<payment_channels>>. Now we will see how HTLCs fit into the commitment scheme. The two new messages are +update_add_htlc+, which Alice uses to ask Bob to add an HTLC, and +update_fulfill_htlc+, which Bob uses to redeem the HTLC once he has received the payment secret (Dina's secret).
=== Forwarding Payments with HTLCs
Alice and Bob start with a payment channel that has a 70,000 satoshi balance on each side.
As we saw in <<payment_channels>>, this means that Alice and Bob have negotiated and each hold commitment transactions. These commitment transactions are asymmetric, delayed, and revocable, and look like the example in <<alice_bob_commitment_txs_1>>.
[[alice_bob_commitment_txs_1]]
.Alice and Bob's initial commitment transactions
image::images/mtln_0904.png["Alice and Bob's initial commitment transactions"]
==== Adding an HTLC
Alice wants Bob to accept an HTLC worth 50,200 satoshis to forward to Dina. To do so, Alice must send the details of this HTLC, including the payment hash and amount to Bob. Bob will also need to know where to forward it, which is something we discuss in detail in <<onion_routing>>.
To add the HTLC, Alice starts the flow we saw in <<HTLC_commitment_message_flow>> by sending the +update_add_htlc+ message to Bob.
[[update_add_htlc]]
==== The update_add_HTLC Message
Alice sends the `update_add_HTLC` Lightning message to Bob. This message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#adding-an-htlc-update_add_htlc[BOLT #2-Peer Protocol -update_add_HTLC], and is shown in Example 9-1.
[[update_add_HTLC_message_fields]]
.The `update_add_HTLC` message
====
----
[channel_id:channel_id]
[u64:id]
[u64:amount_msat]
[sha256:payment_hash]
[u32:cltv_expiry]
[1366*byte:onion_routing_packet]
----
====
+channel_id+:: This is the channel that Alice has with Bob that she wants to add the HTLC. Remember that Alice and Bob may have multiple channels with each other.
+id+:: This is an HTLC counter and starts at +0+ for the first HTLC offered to Bob by Alice and is incremented for each subsequent offered HTLC.
+amount_msat+:: This is the amount (value) of the HTLC in millisatoshis. In our example this is 50,200,000 millisatoshis (i.e., 50,200 satoshis).
+payment_hash+:: This is the payment hash calculated from Dina's invoice. It is _H_ = RIPEMD160(SHA-256(_R_)), where _R_ is Dina's secret that is known only by Dina and will be revealed if Dina is paid.
+cltv_expiry+:: This is the expiry time for this HTLC, which will be encoded as a timelocked refund in case the HTLC fails to reach Dina in this time.
+onion_routing_packet+:: This is an onion-encrypted route that tells Bob where to forward this HTLC next (to Chan). Onion routing is covered in detail in <<onion_routing>>.
[TIP]
====
As a reminder, accounting within the Lightning Network is in units of millisatoshis (thousandths of a satoshi), whereas Bitcoin accounting is in satoshis. Any amounts in HTLCs are millisatoshis, which are then rounded to the nearest satoshi in the Bitcoin commitment transactions.
====
==== HTLC in Commitment Transactions
The received information is enough for Bob to create a new commitment transaction. The new commitment transaction has the same two outputs +to_self+ and +to_remote+ for Alice and Bob's balance, and a _new_ output representing the HTLC offered by Alice.
We've already seen the basic structure of an HTLC in <<routing>>. The complete script of an offered HTLC is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/03-transactions.md#offered-htlc-outputs[BOLT #3-Transactions-Offered HTLC output] and is shown in <<offered_htlc_output_script>>.
[[offered_htlc_output_script]]
.Offered HTLC output script:
====
[source,text,linenums]
----
# Revocation <1>
OP_DUP OP_HASH160 <RIPEMD160(SHA256(revocationpubkey))> OP_EQUAL
OP_IF
OP_CHECKSIG
OP_ELSE
<remote_HTLCpubkey> OP_SWAP OP_SIZE 32 OP_EQUAL
OP_IF
# Redemption <2>
OP_HASH160 <RIPEMD160(payment_hash)> OP_EQUALVERIFY
2 OP_SWAP <local_HTLCpubkey> 2 OP_CHECKMULTISIG
OP_ELSE
# Refund <3>
OP_DROP <cltv_expiry> OP_CHECKLOCKTIMEVERIFY OP_DROP
OP_CHECKSIG
OP_ENDIF
OP_ENDIF
----
<1> The first clause of the `OP_IF` conditional is redeemable by Alice with a revocation key. If this commitment is later revoked, Alice will have a revocation key to claim this output in a penalty transaction, taking the whole channel balance.
<2> The second clause is redeemable by the preimage (payment secret, or in our example Dina's secret) if it is revealed. This allows Bob to claim this output if he has the secret from Dina, meaning he has successfully delivered the payment to Dina.
<3> The third and final clause is a refund of the HTLC to Alice, if the HTLC expires without reaching Dina. It is timelocked with the expiration +cltv_expiry+. This ensures that Alice's balance is not "stuck" in an HTLC that can't be routed to Dina.
====
There are three ways to claim this output. Try to read the script and see if you can figure it out (remember, it is a stack-based language so things appear "backward").
==== New Commitment with HTLC Output
Bob now has the necessary information to add this HTLC script as an additional output and create a new commitment transaction. Bob's new commitment will have 50,200 satoshis in the HTLC output. That amount will come from Alice's channel balance, so Alice's new balance will be 19,800 satoshis (70,000–50,200 = 19,800). Bob constructs this commitment as a tentative "Commitment #3," shown in <<add_commitment_3b>>.
[[add_commitment_3b]]
.Bob's new commitment with an HTLC output
image::images/mtln_0905.png["Bob's new commitment with an HTLC output"]
==== Alice Commits
Shortly after sending the +update_add_htlc+ message, she will commit to the new state of the channel, so that the HTLC can be safely added by Bob. Bob has the HTLC information and has constructed a new commitment but does not yet have this new commitment signed by Alice.
Alice sends +commitment_signed+ to Bob, with the signature for the new commitment and for the HTLC within. We saw the +commitment_signed+ message in <<payment_channels>>, but now we can understand the rest of the fields. As a reminder, it is shown in Example 9-3.
[[ops_commitment_signed_message]]
.The commitment_signed message
====
----
[channel_id:channel_id]
[signature:signature]
[u16:num_htlcs]
[num_htlcs*signature:htlc_signature]
----
====
The fields +num_htlcs+ and +htlc_signature+ now make more sense:
+num_htlcs+:: This is the number of HTLCs that are outstanding in the commitment transaction. In our example, just one HTLC, the one Alice offered.
+htlc_signature+:: This is an array of signatures (+num_htlcs+ in length), containing signatures for the HTLC outputs.
Alice can send these signatures without hesitation: she can always get a refund if the HTLC expires without being routed to Dina.
Now, Bob has a new signed commitment transaction, as shown in <<signed_commitment_3b>>.
[[signed_commitment_3b]]
.Bob has a new signed commitment
image::images/mtln_0906.png[Bob has a new signed commitment]
==== Bob Acknowledges New Commitment and Revokes Old One
Now that Bob has a new signed commitment, he needs to acknowledge it and revoke the old commitment. He does so by sending the +revoke_and_ack+ message, as we saw in <<payment_channels>> previously. As a reminder, that message is shown in <<revoke_and_ack_message_2>>.
[[revoke_and_ack_message_2]]
.The +revoke_and_ack+ message
====
----
[channel_id:channel_id]
[32*byte:per_commitment_secret]
[point:next_per_commitment_point]
----
====
Bob sends the +per_commitment_secret+ that allows Alice to construct a revocation key to build a penalty transaction spending Bob's old commitment. Once Bob has sent this, he cannot ever publish "Commitment #2" without risking a penalty transaction and losing all his money. So, the old commitment is effectively revoked.
Bob has effectively moved the channel state forward, as shown in <<revoked_commitment_2b>>.
[[revoked_commitment_2b]]
.Bob has revoked the old commitment
image::images/mtln_0907.png[Bob has revoked the old commitment]
Despite the fact that Bob has a new (signed) commitment transaction and an HTLC output inside he cannot consider his HTLC as being set up successfully.
He first needs to have Alice revoke her old commitment because, otherwise, Alice can roll back her balance to 70,000 satoshis. Bob needs to make sure that Alice also has a commitment transaction containing the HTLC and has revoked the old commitment.
That is why, if Bob is not the final recipient of the HTLC funds, he should not forward the HTLC yet, by offering an HTLC on the next channel with Chan.
Alice has constructed a mirror-image new commitment transaction containing the new HTLC, but it is yet to be signed by Bob. We can see it in <<add_commitment_3a>>.
[[add_commitment_3a]]
.Alice's new commitment with an HTLC output
image::images/mtln_0908.png["Alice's new commitment with an HTLC output"]
As we described in <<payment_channels>>, Alice's commitment is the mirror-image of Bob's, as it contains the asymmetric, delayed, revocable construct for revocation and penalty enforcement of old commitments. Alice's 19,800 satoshi balance (after deducting the HTLC value), is delayed and revocable. Bob's 70,000 satoshi balance is immediately redeemable.
Next, the message flow for +commitment_signed+ and +revoke_and_ack+ is now repeated, but in the opposite direction. Bob sends +commitment_signed+ to sign Alice's new commitment and Alice responds by revoking her old commitment.
For completeness sake, let's quickly review the commitment transactions as this round of commitment/revocation happens.
==== Bob Commits
Bob now sends a +commitment_signed+ back to Alice, with his signatures for Alice's new commitment transaction, including the HTLC output she has added.
Now Alice has the signature for the new commitment transaction. The state of the channel is shown in <<signed_commitment_3a>>.
[[signed_commitment_3a]]
.Alice has a new *signed* commitment
image::images/mtln_0909.png[Alice has a new signed commitment]
Alice can now acknowledge the new commitment by revoking the old one. Alice sends the +revoke_and_ack+ message containing the necessary +per_commitment_point+ that will allow Bob to construct a revocation key and penalty transaction. Thus, Alice revokes her old commitment.
The channel state is shown in <<revoked_commitment_2a>>.
[[revoked_commitment_2a]]
.Alice has revoked the old commitment
image::images/mtln_0910.png[Alice has revoked the old commitment]
=== Multiple HTLCs
At any point in time, Alice and Bob may have dozens or even hundreds of HTLCs across a single channel. Each HTLC is offered and added to the commitment transaction as an additional output. A commitment transaction therefore always has two outputs for the channel partner balances and any number of HTLC outputs, one per HTLC.
As we saw in the +commitment_signed+ message, there is an array for HTLC signatures so that multiple HTLC commitments can be transmitted at the same time.
The current maximum number of HTLCs allowed on a channel is 483 HTLCs to account for the maximum Bitcoin transaction size and ensure that the commitment transactions continue to be valid Bitcoin transactions.
As we will see in the next section, the maximum is only for _pending_ HTLCs because, once an HTLC is fulfilled (or fails due to timeout/error), it is removed from the commitment transaction.
=== HTLC Fulfillment
Now Bob and Alice both have a new commitment transaction with an additional HTLC output and we have achieved a major step toward updating a payment channel.
The new balance of Alice and Bob does not reflect yet that Alice has successfully sent 50,200 satoshis to Bob.
However, the HTLCs are now set up in a way that secure settlement in exchange for the proof of payment will be possible.
==== HTLC Propagation
Let's assume that Bob continues the chain and sets up an HTLC with Chan for 50,100 satoshis. The process will be exactly the same as we just saw between Alice and Bob. Bob will send +update_add_htlc+ to Chan, then they will exchange commitment_signed and +revoke_and_ack+ messages in two rounds, progressing their channel to the next state.
Next, Chan will do the same with Dina: offer a 50,000 satoshi HTLC, commit, and revoke, etc. However, Dina is the final recipient of the HTLC. Dina is the only one that knows the payment secret (the preimage of the payment hash). Therefore, Dina can fulfill the HTLC with Chan immediately!
==== Dina Fulfills the HTLC with Chan
Dina can settle the HTLC by sending an +update_fulfill_htlc+ message to Chan. The +update_fulfill_htlc+ message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#removing-an-htlc-update_fulfill_htlc-update_fail_htlc-and-update_fail_malformed_htlc[BOLT #2-Peer Protocol-update_fulfill_htlc] and is shown here:
[[update_fulfill_htlc_message]]
.The +update_fulfill_htlc+ message
----
[channel_id:channel_id]
[u64:id]
[32*byte:payment_preimage]
----
It's a really simple message:
+channel_id+:: The channel ID on which the HTLC is committed
+id+:: The ID of the HTLC (we started with 0 and incremented for each HTLC on the channel)
+payment_preimage+:: The secret that proves payment was made and redeems the HTLC. This is the +R+ value that was hashed by Dina to produce the payment hash in the invoice to Alice.
When Chan receives this message, he will immediately check if the payment_preimage (let's call it _R_) produces the payment hash (let's call it _H_) in the HTLC that he offered to Dina. He hashes it like this:
++++
<ul class="simplelist">
<li><em>H</em> = RIPEMD160( SHA-256 (<em>R</em>) )</li>
</ul>
++++
If the result _H_ matches the payment hash in the HTLC, Chan can do a little dance of celebration. This long-awaited secret can be used to redeem the HTLC, and will be passed back along the chain of payment channels all the way to Alice, resolving every HTLC that was part of this payment to Dina.
Let's go back to Alice and Bob's channel and watch them unwind the HTLC. To get there, let's assume Dina sent the +update_fulfill_htlc+ to Chan, Chan sent +update_fulfill_htlc+ to Bob, and Bob sent +update_fulfill_htlc+ to Alice. The payment preimage has propagated all the way back to Alice.
==== Bob Settles the HTLC with Alice
When Bob sends +update_fulfill_htlc+ to Alice, it will contain the same payment_preimage that Dina selected for her invoice. That +payment_preimage+ has traveled backward along the payment path. At each step, the +channel_id+ will be different and +id+ (HTLC ID) may be different. But the preimage is the same!
Alice will also validate the +payment_preimage+ received from Bob. She will compare its hash to the HTLC payment hash in the HTLC she offered Bob. She will also find this preimage matches the hash in Dina's invoice. This is proof that Dina was paid.
The message flow between Alice and Bob is shown in <<htlc_fulfillment_message_flow>>.
[[htlc_fulfillment_message_flow]]
.The HTLC fulfillment message flow
image::images/mtln_0911.png[The HTLC fulfillment message flow]
Both Alice and Bob can now remove the HTLC from the commitment transactions and update their channel balances.
They create new commitments (Commitment #4), as shown in <<htlc_fulfillment_commitments_added>>.
[[htlc_fulfillment_commitments_added]]
.The HTLC is removed and balances updated in new commitments
image::images/mtln_0912.png[The HTLC is removed and balances updated in new commitments]
Next, they complete two rounds of commitment and revocation. First, Alice sends +commitment_signed+ to sign Bob's new commitment transaction. Bob responds with +revoke_and_ack+ to revoke his old commitment. Once Bob has moved the state of the channel forward, the commitments look like we see in <<htlc_fulfillment_commitments_bob_commit>>.
[[htlc_fulfillment_commitments_bob_commit]]
.Alice signs Bob's new commitment and Bob revoked the old one
image::images/mtln_0913.png[Alice signs Bob's new commitment and Bob revoked the old one]
Finally, Bob signs Alice's commitment by sending Alice a +commitment_signed+ message. Then Alice acknowledges and revokes her old commitment by sending +revoke_and_ack+ to Bob. The end result is that both Alice and Bob have moved their channel state to Commitment #4, have removed the HTLC, and have updated their balances. Their current channel state is represented by the commitment transactions that are shown in <<alice_bob_htlc_fulfilled>>.
[[alice_bob_htlc_fulfilled]]
.Alice and Bob settle the HTLC and update balances
image::images/mtln_0914.png[Alice and Bob settle the HTLC and update balances]
=== Removing an HTLC Due to Error or Expiry
If an HTLC cannot be fulfilled, it can be removed from the channel commitment using the same process of commitment and revocation.
Instead of +update_fulfill_htlc+, Bob would send a +update_fail_htlc+ or +update_fail_malformed_htlc+. These two messages are defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#removing-an-htlc-update_fulfill_htlc-update_fail_htlc-and-update_fail_malformed_htlc[BOLT #2-Peer Protocol-Removing an HTLC].
The +update_fail_htlc+ message is shown in the following:
[[update_fail_htlc_message]]
.The +update_fail_htlc+ message
----
[channel_id:channel_id]
[u64:id]
[u16:len]
[len*byte:reason]
----
It's pretty self-explanatory. The multi-byte +reason+ field is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md#failure-messages[BOLT #4-Onion Routing], which we will describe in more detail in <<onion_routing>>.
If Alice received an +update_fail_htlc+ from Bob, the process would unfold in much the same way: the two channel partners would remove the HTLC, create updated commitment transactions, and go through two rounds of commitment/revocation to move the channel state forward to the next commitment. The only difference: the end balances would revert back to what they were without the HTLC, essentially refunding Alice for the HTLC value.
=== Making a Local Payment
At this point, you will easily understand why HTLCs are used for both remote and local payments. When Alice pays Bob for a coffee, she doesn't just update the channel balance and commit to a new state. Instead, the payment is made with an HTLC, in the same way Alice paid Dina. The fact that there's only one channel hop makes no difference. It would work like this:
[start=1]
. Alice orders a coffee from Bob's shop page.
. Bob's shop sends an invoice with a payment hash.
. Alice constructs an HTLC from that payment hash.
. Alice offers the HTLC to Bob with +update_add_htlc+.
. Alice and Bob exchange commitments and revocations adding the HTLC to their commitment transactions.
. Bob sends +update_fulfill_htlc+ to Alice with the payment preimage.
. Alice and Bob exchange commitments and revocations removing the HTLC and updating the channel balances.
Whether an HTLC is forwarded across many channels or just fulfilled in a single channel "hop," the process is exactly the same
=== Conclusion
In this chapter we saw how commitment transactions (from <<payment_channels>>) and HTLCs (from <<routing>>) work together. We saw how an HTLC is added to a commitment transaction, and how it is fulfilled. We saw how the asymmetric, delayed, revocable system for enforcing channel state is extended to HTLCs.
We also saw how a local payment and a multi-hop routed payment are handled identically: using HTLCs.
In the next chapter we will look at the encrypted message routing system called _onion routing_
[[channel_operation]]
== Channel Operation and Payment Forwarding
((("payment channel","operation", id="ix_09_channel_operation-asciidoc0", range="startofrange")))In this chapter we will bring together payment channels and hash time-locked contracts (HTLCs). In <<payment_channels>> we explained the way Alice and Bob construct a payment channel between their two nodes. We also looked at the commitment and penalty mechanisms that secure the payment channel. In <<routing>> we looked at hash time-locked contracts (HTLCs) and how these can be used to route a payment across a path made of multiple payment channels. In this chapter we bring the two concepts together by looking at how HTLCs are managed on each payment channel, how the HTLCs are committed to the channel state, and how they are settled to update the channel balances.
Specifically, we will be discussing "Adding, settling, failing HTLCs" and the "Channel state machine" that form the overlap between the peer 2 peer layer and the routing layer, as highlighted by a double outline in <<LN_protocol_channelops_highlight>>.
[[LN_protocol_channelops_highlight]]
.The Lightning Network protocol suite
image::images/mtln_0901.png["The Lightning Network Protocol Suite"]
=== Local (Single Channel) versus Routed (Multiple Channels)
((("payment channel","local channel versus routed channels")))Even though it is possible to send payments across a payment channel simply by updating the channel balances and creating new commitment transactions, the Lightning protocol uses HTLCs even for "local" payments across a payment channel. The reason for this is to maintain the same protocol design regardless of whether a payment is only one hop (across a single payment channel), or several hops (routed across multiple payment channels).
By maintaining the same abstraction for both local and remote, we not only simplify the protocol design but also improve privacy. For the recipient of a payment there is no discernible difference between a payment made directly by their channel partner and a payment forwarded by their channel partner on behalf of someone else.
=== Forwarding Payments and Updating Commitments with HTLCs
((("commitment transactions","updating commitments with HTLCs", id="ix_09_channel_operation-asciidoc1", range="startofrange")))((("hash time-locked contracts (HTLCs)","updating commitments with", id="ix_09_channel_operation-asciidoc2", range="startofrange")))((("payment channel","updating commitments with HTLCs", id="ix_09_channel_operation-asciidoc3", range="startofrange")))We will revisit our example from <<routing>> to demonstrate how the HTLCs from Alice to Dina get committed to each payment channel. As you recall in our example, Alice is paying Dina 50,000 satoshis by routing an HTLC via Bob and Chan. The network is shown in <<alice_dina_htlc_2>>.
[[alice_dina_htlc_2]]
.Alice pays Dina with an HTLC routed via Bob and Chan
image::images/mtln_0809.png["Alice pays Dina with an HTLC routed via Bob and Chan"]
We will focus on the payment channel between Alice and Bob and review the messages and transactions that they use to process this HTLC.
==== HTLC and Commitment Message Flow
((("hash time-locked contracts (HTLCs)","commitment message flow")))The message flow between Alice and Bob (and also between any pair of channel partners) is shown in <<HTLC_commitment_message_flow>>.
[[HTLC_commitment_message_flow]]
.The message flow for HTLC commitment between channel partners
image::images/mtln_0903.png["The message flow for HTLC commitment between channel partners"]
We've already seen the +commitment_signed+ and +revoke_and_ack+ in <<payment_channels>>. Now we will see how HTLCs fit into the commitment scheme. The two new messages are +update_add_htlc+, which Alice uses to ask Bob to add an HTLC, and +update_fulfill_htlc+, which Bob uses to redeem the HTLC once he has received the payment secret (Dina's secret).(((range="endofrange", startref="ix_09_channel_operation-asciidoc3")))(((range="endofrange", startref="ix_09_channel_operation-asciidoc2")))(((range="endofrange", startref="ix_09_channel_operation-asciidoc1")))
=== Forwarding Payments with HTLCs
((("hash time-locked contracts (HTLCs)","forwarding payments with", id="ix_09_channel_operation-asciidoc4", range="startofrange")))((("payment forwarding","with HTLCs", id="ix_09_channel_operation-asciidoc5", range="startofrange")))Alice and Bob start with a payment channel that has a 70,000 satoshi balance on each side.
As we saw in <<payment_channels>>, this means that Alice and Bob have negotiated and each hold commitment transactions. These commitment transactions are asymmetric, delayed, and revocable, and look like the example in <<alice_bob_commitment_txs_1>>.
[[alice_bob_commitment_txs_1]]
.Alice and Bob's initial commitment transactions
image::images/mtln_0904.png["Alice and Bob's initial commitment transactions"]
==== Adding an HTLC
((("hash time-locked contracts (HTLCs)","adding an HTLC")))Alice wants Bob to accept an HTLC worth 50,200 satoshis to forward to Dina. To do so, Alice must send the details of this HTLC, including the payment hash and amount to Bob. Bob will also need to know where to forward it, which is something we discuss in detail in <<onion_routing>>.
To add the HTLC, Alice starts the flow we saw in <<HTLC_commitment_message_flow>> by sending the +update_add_htlc+ message to Bob.
[[update_add_htlc]]
==== The update_add_HTLC Message
((("hash time-locked contracts (HTLCs)","update_add_HTLC message")))((("update_add_HTLC message")))Alice sends the `update_add_HTLC` Lightning message to Bob. This message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#adding-an-htlc-update_add_htlc[BOLT #2-Peer Protocol -update_add_HTLC], and is shown in Example 9-1.
[[update_add_HTLC_message_fields]]
.The `update_add_HTLC` message
====
----
[channel_id:channel_id]
[u64:id]
[u64:amount_msat]
[sha256:payment_hash]
[u32:cltv_expiry]
[1366*byte:onion_routing_packet]
----
====
+channel_id+:: This is the channel that Alice has with Bob that she wants to add the HTLC. Remember that Alice and Bob may have multiple channels with each other.
+id+:: This is an HTLC counter and starts at +0+ for the first HTLC offered to Bob by Alice and is incremented for each subsequent offered HTLC.
+amount_msat+:: This is the amount (value) of the HTLC in millisatoshis. In our example this is 50,200,000 millisatoshis (i.e., 50,200 satoshis).
+payment_hash+:: This is the payment hash calculated from Dina's invoice. It is _H_ = RIPEMD160(SHA-256(_R_)), where _R_ is Dina's secret that is known only by Dina and will be revealed if Dina is paid.
+cltv_expiry+:: This is the expiry time for this HTLC, which will be encoded as a timelocked refund in case the HTLC fails to reach Dina in this time.
+onion_routing_packet+:: This is an onion-encrypted route that tells Bob where to forward this HTLC next (to Chan). Onion routing is covered in detail in <<onion_routing>>.
[TIP]
====
As a reminder, accounting within the Lightning Network is in units of millisatoshis (thousandths of a satoshi), whereas Bitcoin accounting is in satoshis. Any amounts in HTLCs are millisatoshis, which are then rounded to the nearest satoshi in the Bitcoin commitment transactions.
====
==== HTLC in Commitment Transactions
((("commitment transactions","HTLC in")))((("hash time-locked contracts (HTLCs)","commitment transactions and")))The received information is enough for Bob to create a new commitment transaction. The new commitment transaction has the same two outputs +to_self+ and +to_remote+ for Alice and Bob's balance, and a _new_ output representing the HTLC offered by Alice.
We've already seen the basic structure of an HTLC in <<routing>>. The complete script of an offered HTLC is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/03-transactions.md#offered-htlc-outputs[BOLT #3-Transactions-Offered HTLC output] and is shown in <<offered_htlc_output_script>>.
[[offered_htlc_output_script]]
.Offered HTLC output script:
====
[source,text,linenums]
----
# Revocation <1>
OP_DUP OP_HASH160 <RIPEMD160(SHA256(revocationpubkey))> OP_EQUAL
OP_IF
OP_CHECKSIG
OP_ELSE
<remote_HTLCpubkey> OP_SWAP OP_SIZE 32 OP_EQUAL
OP_IF
# Redemption <2>
OP_HASH160 <RIPEMD160(payment_hash)> OP_EQUALVERIFY
2 OP_SWAP <local_HTLCpubkey> 2 OP_CHECKMULTISIG
OP_ELSE
# Refund <3>
OP_DROP <cltv_expiry> OP_CHECKLOCKTIMEVERIFY OP_DROP
OP_CHECKSIG
OP_ENDIF
OP_ENDIF
----
<1> The first clause of the `OP_IF` conditional is redeemable by Alice with a revocation key. If this commitment is later revoked, Alice will have a revocation key to claim this output in a penalty transaction, taking the whole channel balance.
<2> The second clause is redeemable by the preimage (payment secret, or in our example Dina's secret) if it is revealed. This allows Bob to claim this output if he has the secret from Dina, meaning he has successfully delivered the payment to Dina.
<3> The third and final clause is a refund of the HTLC to Alice, if the HTLC expires without reaching Dina. It is timelocked with the expiration +cltv_expiry+. This ensures that Alice's balance is not "stuck" in an HTLC that can't be routed to Dina.
====
There are three ways to claim this output. Try to read the script and see if you can figure it out (remember, it is a stack-based language so things appear "backward").
==== New Commitment with HTLC Output
((("commitment transactions","new commitment with HTLC output", id="ix_09_channel_operation-asciidoc6", range="startofrange")))((("hash time-locked contracts (HTLCs)","new commitment with HTLC output", id="ix_09_channel_operation-asciidoc7", range="startofrange")))Bob now has the necessary information to add this HTLC script as an additional output and create a new commitment transaction. Bob's new commitment will have 50,200 satoshis in the HTLC output. That amount will come from Alice's channel balance, so Alice's new balance will be 19,800 satoshis (70,000–50,200 = 19,800). Bob constructs this commitment as a tentative "Commitment #3," shown in <<add_commitment_3b>>.
[[add_commitment_3b]]
.Bob's new commitment with an HTLC output
image::images/mtln_0905.png["Bob's new commitment with an HTLC output"]
==== Alice Commits
Shortly after sending the +update_add_htlc+ message, she will commit to the new state of the channel, so that the HTLC can be safely added by Bob. Bob has the HTLC information and has constructed a new commitment but does not yet have this new commitment signed by Alice.
Alice sends +commitment_signed+ to Bob, with the signature for the new commitment and for the HTLC within. We saw the +commitment_signed+ message in <<payment_channels>>, but now we can understand the rest of the fields. As a reminder, it is shown in Example 9-3.
[[ops_commitment_signed_message]]
.The commitment_signed message
====
----
[channel_id:channel_id]
[signature:signature]
[u16:num_htlcs]
[num_htlcs*signature:htlc_signature]
----
====
The fields +num_htlcs+ and +htlc_signature+ now make more sense:
+num_htlcs+:: This is the number of HTLCs that are outstanding in the commitment transaction. In our example, just one HTLC, the one Alice offered.
+htlc_signature+:: This is an array of signatures (+num_htlcs+ in length), containing signatures for the HTLC outputs.
Alice can send these signatures without hesitation: she can always get a refund if the HTLC expires without being routed to Dina.
Now, Bob has a new signed commitment transaction, as shown in <<signed_commitment_3b>>.
[[signed_commitment_3b]]
.Bob has a new signed commitment
image::images/mtln_0906.png[Bob has a new signed commitment]
==== Bob Acknowledges New Commitment and Revokes Old One
((("hash time-locked contracts (HTLCs)","acknowledging new commitment/revoking old commitment")))Now that Bob has a new signed commitment, he needs to acknowledge it and revoke the old commitment. ((("revoke_and_ack message", id="ix_09_channel_operation-asciidoc8", range="startofrange")))He does so by sending the +revoke_and_ack+ message, as we saw in <<payment_channels>> previously. As a reminder, that message is shown in <<revoke_and_ack_message_2>>.
[[revoke_and_ack_message_2]]
.The +revoke_and_ack+ message
====
----
[channel_id:channel_id]
[32*byte:per_commitment_secret]
[point:next_per_commitment_point]
----
====
Bob sends the +per_commitment_secret+ that allows Alice to construct a revocation key to build a penalty transaction spending Bob's old commitment. Once Bob has sent this, he cannot ever publish "Commitment #2" without risking a penalty transaction and losing all his money. So, the old commitment is effectively revoked.
Bob has effectively moved the channel state forward, as shown in <<revoked_commitment_2b>>.
[[revoked_commitment_2b]]
.Bob has revoked the old commitment
image::images/mtln_0907.png[Bob has revoked the old commitment]
Despite the fact that Bob has a new (signed) commitment transaction and an HTLC output inside he cannot consider his HTLC as being set up successfully.
He first needs to have Alice revoke her old commitment because, otherwise, Alice can roll back her balance to 70,000 satoshis. Bob needs to make sure that Alice also has a commitment transaction containing the HTLC and has revoked the old commitment.
That is why, if Bob is not the final recipient of the HTLC funds, he should not forward the HTLC yet, by offering an HTLC on the next channel with Chan.
Alice has constructed a mirror-image new commitment transaction containing the new HTLC, but it is yet to be signed by Bob. We can see it in <<add_commitment_3a>>.
[[add_commitment_3a]]
.Alice's new commitment with an HTLC output
image::images/mtln_0908.png["Alice's new commitment with an HTLC output"]
As we described in <<payment_channels>>, Alice's commitment is the mirror-image of Bob's, as it contains the asymmetric, delayed, revocable construct for revocation and penalty enforcement of old commitments. Alice's 19,800 satoshi balance (after deducting the HTLC value), is delayed and revocable. Bob's 70,000 satoshi balance is immediately redeemable.
Next, the message flow for +commitment_signed+ and +revoke_and_ack+ is now repeated, but in the opposite direction. Bob sends +commitment_signed+ to sign Alice's new commitment and Alice responds by revoking her old commitment.(((range="endofrange", startref="ix_09_channel_operation-asciidoc8")))
For completeness sake, let's quickly review the commitment transactions as this round of commitment/revocation happens.
==== Bob Commits
Bob now sends a +commitment_signed+ back to Alice, with his signatures for Alice's new commitment transaction, including the HTLC output she has added.
Now Alice has the signature for the new commitment transaction. The state of the channel is shown in <<signed_commitment_3a>>.
[[signed_commitment_3a]]
.Alice has a new *signed* commitment
image::images/mtln_0909.png[Alice has a new signed commitment]
Alice can now acknowledge the new commitment by revoking the old one. Alice sends the +revoke_and_ack+ message containing the necessary +per_commitment_point+ that will allow Bob to construct a revocation key and penalty transaction. Thus, Alice revokes her old commitment.
The channel state is shown in <<revoked_commitment_2a>>.(((range="endofrange", startref="ix_09_channel_operation-asciidoc7")))(((range="endofrange", startref="ix_09_channel_operation-asciidoc6"))) (((range="endofrange", startref="ix_09_channel_operation-asciidoc5")))(((range="endofrange", startref="ix_09_channel_operation-asciidoc4")))
[[revoked_commitment_2a]]
.Alice has revoked the old commitment
image::images/mtln_0910.png[Alice has revoked the old commitment]
=== Multiple HTLCs
((("hash time-locked contracts (HTLCs)","multiple contracts")))At any point in time, Alice and Bob may have dozens or even hundreds of HTLCs across a single channel. Each HTLC is offered and added to the commitment transaction as an additional output. A commitment transaction therefore always has two outputs for the channel partner balances and any number of HTLC outputs, one per HTLC.
As we saw in the +commitment_signed+ message, there is an array for HTLC signatures so that multiple HTLC commitments can be transmitted at the same time.
The current maximum number of HTLCs allowed on a channel is 483 HTLCs to account for the maximum Bitcoin transaction size and ensure that the commitment transactions continue to be valid Bitcoin transactions.
As we will see in the next section, the maximum is only for _pending_ HTLCs because, once an HTLC is fulfilled (or fails due to timeout/error), it is removed from the commitment transaction.
=== HTLC Fulfillment
((("hash time-locked contracts (HTLCs)","fulfillment", id="ix_09_channel_operation-asciidoc9", range="startofrange")))((("payment forwarding","HTLC fulfillment", id="ix_09_channel_operation-asciidoc10", range="startofrange")))Now Bob and Alice both have a new commitment transaction with an additional HTLC output and we have achieved a major step toward updating a payment channel.
The new balance of Alice and Bob does not reflect yet that Alice has successfully sent 50,200 satoshis to Bob.
However, the HTLCs are now set up in a way that secure settlement in exchange for the proof of payment will be possible.
==== HTLC Propagation
((("hash time-locked contracts (HTLCs)","propagation", id="ix_09_channel_operation-asciidoc11", range="startofrange")))((("payment forwarding","HTLC propagation", id="ix_09_channel_operation-asciidoc12", range="startofrange")))Let's assume that Bob continues the chain and sets up an HTLC with Chan for 50,100 satoshis. The process will be exactly the same as we just saw between Alice and Bob. Bob will send +update_add_htlc+ to Chan, then they will exchange commitment_signed and +revoke_and_ack+ messages in two rounds, progressing their channel to the next state.
Next, Chan will do the same with Dina: offer a 50,000 satoshi HTLC, commit, and revoke, etc. However, Dina is the final recipient of the HTLC. Dina is the only one that knows the payment secret (the preimage of the payment hash). Therefore, Dina can fulfill the HTLC with Chan immediately!
==== Dina Fulfills the HTLC with Chan
Dina can settle the HTLC by sending an +update_fulfill_htlc+ message to Chan. The +update_fulfill_htlc+ message is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#removing-an-htlc-update_fulfill_htlc-update_fail_htlc-and-update_fail_malformed_htlc[BOLT #2-Peer Protocol-update_fulfill_htlc] and is shown here:
[[update_fulfill_htlc_message]]
.The +update_fulfill_htlc+ message
----
[channel_id:channel_id]
[u64:id]
[32*byte:payment_preimage]
----
It's a really simple message:
+channel_id+:: The channel ID on which the HTLC is committed
+id+:: The ID of the HTLC (we started with 0 and incremented for each HTLC on the channel)
+payment_preimage+:: The secret that proves payment was made and redeems the HTLC. This is the +R+ value that was hashed by Dina to produce the payment hash in the invoice to Alice.
When Chan receives this message, he will immediately check if the payment_preimage (let's call it _R_) produces the payment hash (let's call it _H_) in the HTLC that he offered to Dina. He hashes it like this:
++++
<ul class="simplelist">
<li><em>H</em> = RIPEMD160( SHA-256 (<em>R</em>) )</li>
</ul>
++++
If the result _H_ matches the payment hash in the HTLC, Chan can do a little dance of celebration. This long-awaited secret can be used to redeem the HTLC, and will be passed back along the chain of payment channels all the way to Alice, resolving every HTLC that was part of this payment to Dina.
Let's go back to Alice and Bob's channel and watch them unwind the HTLC. To get there, let's assume Dina sent the +update_fulfill_htlc+ to Chan, Chan sent +update_fulfill_htlc+ to Bob, and Bob sent +update_fulfill_htlc+ to Alice. The payment preimage has propagated all the way back to Alice.
==== Bob Settles the HTLC with Alice
When Bob sends +update_fulfill_htlc+ to Alice, it will contain the same payment_preimage that Dina selected for her invoice. That +payment_preimage+ has traveled backward along the payment path. At each step, the +channel_id+ will be different and +id+ (HTLC ID) may be different. But the preimage is the same!
Alice will also validate the +payment_preimage+ received from Bob. She will compare its hash to the HTLC payment hash in the HTLC she offered Bob. She will also find this preimage matches the hash in Dina's invoice. This is proof that Dina was paid.
The message flow between Alice and Bob is shown in <<htlc_fulfillment_message_flow>>.
[[htlc_fulfillment_message_flow]]
.The HTLC fulfillment message flow
image::images/mtln_0911.png[The HTLC fulfillment message flow]
Both Alice and Bob can now remove the HTLC from the commitment transactions and update their channel balances.
They create new commitments (Commitment #4), as shown in <<htlc_fulfillment_commitments_added>>.
[[htlc_fulfillment_commitments_added]]
.The HTLC is removed and balances updated in new commitments
image::images/mtln_0912.png[The HTLC is removed and balances updated in new commitments]
Next, they complete two rounds of commitment and revocation. First, Alice sends +commitment_signed+ to sign Bob's new commitment transaction. Bob responds with +revoke_and_ack+ to revoke his old commitment. Once Bob has moved the state of the channel forward, the commitments look like we see in <<htlc_fulfillment_commitments_bob_commit>>.
[[htlc_fulfillment_commitments_bob_commit]]
.Alice signs Bob's new commitment and Bob revoked the old one
image::images/mtln_0913.png[Alice signs Bob's new commitment and Bob revoked the old one]
Finally, Bob signs Alice's commitment by sending Alice a +commitment_signed+ message. Then Alice acknowledges and revokes her old commitment by sending +revoke_and_ack+ to Bob. The end result is that both Alice and Bob have moved their channel state to Commitment #4, have removed the HTLC, and have updated their balances. Their current channel state is represented by the commitment transactions that are shown in <<alice_bob_htlc_fulfilled>>(((range="endofrange", startref="ix_09_channel_operation-asciidoc12")))(((range="endofrange", startref="ix_09_channel_operation-asciidoc11"))). (((range="endofrange", startref="ix_09_channel_operation-asciidoc10")))(((range="endofrange", startref="ix_09_channel_operation-asciidoc9")))
[[alice_bob_htlc_fulfilled]]
.Alice and Bob settle the HTLC and update balances
image::images/mtln_0914.png[Alice and Bob settle the HTLC and update balances]
=== Removing an HTLC Due to Error or Expiry
((("hash time-locked contracts (HTLCs)","removing due to error/expiry")))((("payment forwarding","removing an HTLC due to error/expiry")))If an HTLC cannot be fulfilled, it can be removed from the channel commitment using the same process of commitment and revocation.
Instead of +update_fulfill_htlc+, Bob would send a +update_fail_htlc+ or +update_fail_malformed_htlc+. These two messages are defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/02-peer-protocol.md#removing-an-htlc-update_fulfill_htlc-update_fail_htlc-and-update_fail_malformed_htlc[BOLT #2-Peer Protocol-Removing an HTLC].
The +update_fail_htlc+ message is shown in the following:
[[update_fail_htlc_message]]
.The +update_fail_htlc+ message
----
[channel_id:channel_id]
[u64:id]
[u16:len]
[len*byte:reason]
----
It's pretty self-explanatory. The multi-byte +reason+ field is defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/04-onion-routing.md#failure-messages[BOLT #4-Onion Routing], which we will describe in more detail in <<onion_routing>>.
If Alice received an +update_fail_htlc+ from Bob, the process would unfold in much the same way: the two channel partners would remove the HTLC, create updated commitment transactions, and go through two rounds of commitment/revocation to move the channel state forward to the next commitment. The only difference: the end balances would revert back to what they were without the HTLC, essentially refunding Alice for the HTLC value.
=== Making a Local Payment
((("hash time-locked contracts (HTLCs)","local payment with")))((("local payments")))((("payment forwarding","local payments")))At this point, you will easily understand why HTLCs are used for both remote and local payments. When Alice pays Bob for a coffee, she doesn't just update the channel balance and commit to a new state. Instead, the payment is made with an HTLC, in the same way Alice paid Dina. The fact that there's only one channel hop makes no difference. It would work like this:
[start=1]
. Alice orders a coffee from Bob's shop page.
. Bob's shop sends an invoice with a payment hash.
. Alice constructs an HTLC from that payment hash.
. Alice offers the HTLC to Bob with +update_add_htlc+.
. Alice and Bob exchange commitments and revocations adding the HTLC to their commitment transactions.
. Bob sends +update_fulfill_htlc+ to Alice with the payment preimage.
. Alice and Bob exchange commitments and revocations removing the HTLC and updating the channel balances.
Whether an HTLC is forwarded across many channels or just fulfilled in a single channel "hop," the process is exactly the same
=== Conclusion
In this chapter we saw how commitment transactions (from <<payment_channels>>) and HTLCs (from <<routing>>) work together. We saw how an HTLC is added to a commitment transaction, and how it is fulfilled. We saw how the asymmetric, delayed, revocable system for enforcing channel state is extended to HTLCs.
We also saw how a local payment and a multi-hop routed payment are handled identically: using HTLCs.(((range="endofrange", startref="ix_09_channel_operation-asciidoc0")))
In the next chapter we will look at the encrypted message routing system called _onion routing_

1860
10_onion_routing.asciidoc
View File

File diff suppressed because it is too large Load Diff

976
11_gossip_channel_graph.asciidoc
Unescape Escape View File

@ -1,488 +1,488 @@
[[gossip]]
== Gossip and the Channel Graph
In this chapter we will describe the Lightning Network's gossip protocol and how it is used by nodes to construct and maintain a channel graph. We will also review the DNS bootstrap mechanism used to find peers to "gossip" with.
The "Routing fees and Gossip relaying" section is highlighted by a double outline spanning the routing layer and peer 2 peer layer of <<LN_protocol_gossip_highlight>>.
[[LN_protocol_gossip_highlight]]
.The Lightning Network protocol suite
image::images/mtln_1101.png["The Lightning Network Protocol Suite"]
As we've learned already, the Lightning Network uses a source-based onion routing protocol to deliver a payment from a sender to the recipient.
To do this, the sending node must be able to construct a path of payment channels that connects it with the recipient, as we will see in <<path_finding>>.
Thus, the sender has to be able to map the Lightning Network by constructing a channel graph.
The _channel graph_ is the interconnected set of publicly advertised channels and the nodes that these channels interlink.
As channels are backed by a funding transaction that is happening on-chain, one might falsely believe that Lightning nodes could just extract the existing channels from the Bitcoin blockchain.
However this is only possible to a certain extent.
The funding transactions are Pay-to-Witness-Script-Hash (P2WSH) addresses, and the nature of the script (a 2-of-2 multisig) will only be revealed once the funding transaction output is spent.
Even if the nature of the script were known, it's important to remember that not all 2-of-2 multisig scripts correspond to payment channels.
There are even more reasons why looking at the Bitcoin blockchain might not be helpful.
For example, on the Lightning Network the Bitcoin keys that are used for signing are rotated by the nodes for every channel and update.
Thus, even if we could reliably detect funding transactions on the Bitcoin blockchain, we would not know which two nodes on the Lightning Network own that particular channel.
The Lightning Network solves this problem by implementing a _gossip protocol_.
Gossip protocols are typical for peer-to-peer (P2P) networks and allow nodes to share information with the whole network with just a few direct connections to peers.
Lightning nodes open encrypted peer-to-peer connections to each other and share (gossip) information that they have received from other peers.
As soon as a node wants to share some information, for example about a newly created channel, it sends a message to all its peers.
Upon receiving a message a node decides if the received message was novel and, if so, forwards the information to its peers.
In this way, if the peer-to-peer network is well connected, all new information that is necessary for the operation of the network will eventually be propagated to all other peers.
Obviously, if a new peer joins the network for the first time, it needs to know some other peers on the network, so it can connect to others and participate in the network.
In this chapter, we'll explore exactly _how_ Lightning nodes discover each other, discover and update their node status, and communicate with one another.
When most refer to the _network_ part of the Lightning Network, they're referring to the _channel graph_ which itself is a unique authenticated data structure _anchored_ in the base Bitcoin
blockchain.
However, the Lightning Network is also a peer-to-peer network of nodes that gossip information about payment channels and nodes. Usually for two peers to maintain a payment channel they need to talk to each other directly, which means that there will be a peer connection between them.
This suggests that the channel graph is a subnetwork of the peer-to-peer network.
However, this is not true because payment channels can remain open even if one or both peers go temporarily offline.
Let's revisit some of the terminology that we have used throughout the book, specifically looking at what they mean in terms of the channel graph and the peer-to-peer network (see <<network_terminology>>).
[[network_terminology]]
.Terminology of the different networks
[options="header"]
|===
| Channel graph |Peer-to-peer network
| channel | connection
| open | connect
| close | disconnect
| funding transaction | encrypted TCP/IP connection
| send | transmit
| payment | message
|===
Because the Lightning Network is a peer-to-peer network, some initial bootstrapping is required in order for peers to discover each other. Within this chapter we'll follow the story of a new peer connecting to the network for the first time and examine each step in the bootstrapping process from initial peer discovery to channel graph syncing and validation.
As an initial step, our new node needs to somehow _discover_ at least _one_ peer that is already connected to the network and has a full channel graph (as we'll see later, there's no canonical version of the channel graph). Using one of many initial bootstrapping protocols to find that first peer, after a connection is established, our new
peer now needs to _download_ and _validate_ the channel graph. Once the channel graph has been fully validated, our new peer is ready to start opening channels and sending payments on the network.
After initial bootstrap, a node on the network needs to continue to maintain its view of the channel graph by processing new channel routing policy updates, discovering and validating new channels, removing channels that have been closed on-chain, and finally pruning channels that fail to send out a proper "heartbeat" every two weeks or so.
Upon completion of this chapter, you will understand a key component of
the peer-to-peer Lightning Network: namely how peers discover each other and maintain a local copy (perspective) of the channel graph. We'll begin by exploring the story of a new node that has just booted up and needs to find other peers to connect to on the network.
=== Peer Discovery
In this section, we'll begin to follow a new Lightning node that wishes to join the network through three steps:
. Discover a set of bootstrap peers
. Download and validate the channel graph
. Begin the process of ongoing maintenance of the channel graph itself
==== P2P Bootstrapping
Before doing any thing else, our new node first needs to discover a set of peers who are already part of the network. We call this process initial peer bootstrapping, and it's something that every peer-to-peer network needs to implement properly to ensure a robust, healthy network.
Bootstrapping new peers to existing peer-to-peer networks is a very well studied problem with several known solutions, each with their own distinct trade-offs. The simplest solution to this problem is simply to package a set of _hardcoded_ bootstrap peers into the packaged P2P node software. This is simple in that each new node has a list of bootstrap peers in the software they're running, but rather fragile given that if the set of bootstrap peers goes offline, then no new nodes will be able to join the network. Due to this fragility, this
option is usually used as a fallback in case none of the other P2P bootstrapping mechanisms work properly.
Rather than hardcoding the set of bootstrap peers within the software/binary itself, we can instead allow peers to dynamically obtain a fresh/new set of bootstrap peers they can use to join the network. We'll call this process _initial peer discovery_. Typically we'll leverage
existing internet protocols to maintain and distribute a set of bootstrapping peers. A nonexhaustive list of protocols that have been used in the past to accomplish initial peer discovery includes:
* Domain Name Service (DNS)
* Internet Relay Chat (IRC)
* Hyper-Text Transfer Protocol (HTTP)
Similar to the Bitcoin protocol, the primary initial peer discovery mechanism used in the Lightning Network happens via DNS. Because initial peer discovery is a critical and universal task for the network, the process has been _standardized_ in https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10: DNS Bootstrap].
==== DNS Bootstrapping
The https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10] document describes a standardized way of implementing peer
discovery using the DNS. Lightning's flavor of DNS based bootstrapping uses up to three distinct record types:
* +SRV+ records for discovering a set of _node public keys_.
* +A+ records for mapping a node's public key to its current +IPv4+ address.
* +AAA+ records for mapping a node's public key to its current +IPv6+ address.
Those somewhat familiar with the DNS protocol may already be familiar with the +A+ (name to IPv4 address) and +AAA+ (name to IPv6 address) record types, but not the +SRV+ type. The +SRV+ record type is used by protocols built on top of DNS to determine the _location_ for a specified service. In our context, the service in question is a given Lightning node, and the location its IP address. We need to use this additional record type because, unlike nodes within the Bitcoin protocol, we need both a public key _and_ an IP address to connect to a node. As we see in <<wire_protocol>>, the transport encryption protocol used in LN requires knowledge of the public key of a node before connecting, so as to implement identity hiding for nodes in the network.
===== A new peer's bootstrapping workflow
Before diving into the specifics of https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10], we'll first outline the high-level flow of a new node that wishes to use BOLT #10 to join the network.
First, a node needs to identify a single DNS server or set of DNS servers that understand BOLT #10 so they can be used for P2P bootstrapping.
While BOLT #10 uses lseed.bitcoinstats.com as the seed server, there exists no "official" set of DNS seeds for this purpose, but each of the major implementations maintains their own DNS seed, and they cross query each other's seeds for redundancy purposes. In <<dns_seeds>> you'll see a nonexhaustive list of some popular DNS seed servers.
[[dns_seeds]]
.Table of known Lightning DNS seed servers
[options="header"]
|===
| DNS server | Maintainer
| lseed.bitcoinstats.com | Christian Decker
| nodes.lightning.directory | Lightning Labs (Olaoluwa Osuntokun)
| soa.nodes.lightning.directory | Lightning Labs (Olaoluwa Osuntokun)
| lseed.darosior.ninja | Antoine Poinsot
|===
DNS seeds exist for both Bitcoin's mainnet and testnet. For the sake
of our example, we'll assume the existence of a valid BOLT #10 DNS seed at +nodes.lightning.directory+.
Next, our new node will issue an +SRV+ query to obtain a set of _candidate bootstrap peers_. The response to our query will be a series of bech32 encoded public keys. Because DNS is a text-based protocol, we can't send raw binary data, so an encoding scheme is required. BOLT #10 specifies a bech32 encoding due to its use in the wider Bitcoin ecosystem. The number of encoded public keys returned depends on the server returning the query, as well as all the resolvers that stand between the client and the authoritative server.
Using the widely available +dig+ command-line tool, we can query the _testnet_ version of the DNS seed mentioned previously with the following command:
----
$ dig @8.8.8.8 test.nodes.lightning.directory SRV
----
We use the +@+ argument to force resolution via Google's nameserver (with IP address 8.8.8.8) because they do not filter large SRV query responses. At the end of the command, we specify that we only want +SRV+ records to be returned. A sample response looks something like <<ex1101>>.
[[ex1101]]
.Title here
====
----
$ dig @8.8.8.8 test.nodes.lightning.directory SRV
; <<>> DiG 9.10.6 <<>> @8.8.8.8 test.nodes.lightning.directory SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43610
;; flags: qr rd ra; QUERY: 1, ANSWER: 25, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
;test.nodes.lightning.directory. IN SRV
;; ANSWER SECTION:
test.nodes.lightning.directory. 59 IN SRV 10 10 9735 <1>
ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory. <2>
test.nodes.lightning.directory. 59 IN SRV 10 10 15735 ln1qtgsl3efj8verd4z27k44xu0a59kncvsarxatahm334exgnuvwhnz8dkhx8.test.nodes.lightning.directory.
[...]
;; Query time: 89 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Dec 31 16:41:07 PST 2020
----
<1> TCP port number where LN node can be reached.
<2> Node public key (ID) encoded as a virtual domain name.
====
We've truncated the response for brevity and show only two of the returned responses. The responses contain a "virtual" domain name for a target node, then to the left we have the _TCP port_ where this node can be reached. The first response uses the standard TCP port for LN: +9735+. The second response uses a custom port, which is permitted by the protocol.
Next, we'll attempt to obtain the other piece of information we need to connect to a node: its IP address. Before we can query for this, however, we'll first _decode_ the bech32 encoding of the public key from the virtual domain name:
----
ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7
----
Decoding this bech32 string we obtain the following valid
+secp256k1+ public key:
----
026c64f5a7f24c6f7f0e1d6ec877f23b2f672fb48967c2545f227d70636395eaf3
----
Now that we have the raw public key, we'll ask the DNS server to _resolve_ the virtual host given so we can obtain the IP information (+A+ record) for the node, as shown in <<ex1102>>.
++++
<div id="ex1102" data-type="example">
<h5>Title here</h5>
<pre data-type="programlisting">$ dig ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory A
; &lt;&lt;&gt;&gt; DiG 9.10.6 &lt;&lt;&gt;&gt; ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory A
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 41934
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory. IN A
;; ANSWER SECTION:
ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory. 60 IN A <em>X.X.X.X</em> <a class="co" id="comarker1" href="#c01"><img src="callouts/1.png" alt="1"/></a>
;; Query time: 83 msec
;; SERVER: 2600:1700:6971:6dd0::1#53(2600:1700:6971:6dd0::1)
;; WHEN: Thu Dec 31 16:59:22 PST 2020
;; MSG SIZE rcvd: 138</pre>
<dl class="calloutlist">
<dt><a class="co" id="c01" href="#comarker1"><img src="callouts/1.png" alt="1"/></a></dt>
<dd><p>The DNS server returns an IP Address <em>X.X.X.X</em>. We’ve replaced it with X’s in the text here so as to avoid presenting a real IP address.</p></dd>
</dl></div>
++++
In the preceding command, we've queried the server so we can obtain an +IPv4+ (+A+ record) address for our target node (replaced by _X.X.X.X_ in the preceding example). Now that we have the raw public key, IP address, and TCP port, we can connect to the node transport protocol at:
----
026c64f5a7f24c6f7f0e1d6ec877f23b2f672fb48967c2545f227d70636395eaf3@X.X.X.X:9735
----
Querying the current DNS +A+ record for a given node can also be used to look up the _latest_ set of addresses. Such queries can be used to more quickly sync the latest addressing information for a node, compared to waiting for address updates on the gossip network (see <<node_announcement>>).
At this point in our journey, our new Lightning node has found its first
peer and established its first connection! Now we can begin the second phase of new peer bootstrapping: channel graph synchronization and validation.
First, we'll explore more of the intricacies of BOLT #10 itself to take a deeper look into how things work under the hood.
==== SRV Query Options
The https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10] standard is highly extensible due to its usage of nested
subdomains as a communication layer for additional query options. The
bootstrapping protocol allows clients to further specify the _type_ of nodes they're attempting to query for versus the default of receiving a random subset of nodes in the query responses.
The query option subdomain scheme uses a series of key-value pairs where the key itself is a _single letter_ and the remaining set of text is the value itself. The following query types exist in the current version of the https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10] standards document:
+r+:: The _realm_ byte which is used to determine which chain or realm queries should be returned for. As is, the only value for this key is +0+ which denotes "Bitcoin."
+a+:: Allows clients to filter out returned nodes based on the _types_ of addresses they advertise. As an example, this can be used to only obtain nodes that advertise a valid IPv6 address. The value that follows this type is based on a bitfled that _indexes_ into the set of specified address _types_ that are defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7]. The default value for this field is +6+, which represents both IPv4 and IPv6 (bits 1 and 2 are set).
+l+:: A valid node public key serialized in compressed format. This allows a client to query for a specified node rather than receiving a set of random nodes.
+n+:: The number of records to return. The default value for this field is +25+.
An example query with additional query options looks something like the following:
----
r0.a2.n10.nodes.lightning.directory
----
Breaking down the query one key-value pair at a time, we gain the following
insights:
* +r0+: The query targets the Bitcoin realm
* +a2+: The query only wants IPv4 addresses to be returned
* +n10+: The query requests
Try some combinations of the various flags using the +dig+ DNS command-line tool yourself:
----
dig @8.8.8.8 r0.a6.nodes.lightning.directory SRV
----
=== The Channel Graph
Now that our new node is able to use the DNS bootstrapping protocol to connect to their very first peer, it can start to sync the channel graph! However, before we sync the channel graph, we'll need to learn exactly _what_ we mean by the channel graph. In this section we'll explore the precise _structure_ of the channel graph and examine the unique aspects of the channel graph compared to the typical abstract "graph" data structure which is well known/used in the field of computer science.
==== A Directed Graph
A _graph_ in computer science is a special data structure composed of vertices (typically referred to as nodes) and edges (also known as links). Two nodes may be connected by one or more edges. The channel graph is also _directed_ given that a payment is able to flow in either direction over a given edge (a channel). An example of a _directed graph_ is shown in <<directed_graph>>.
[[directed_graph]]
.A directed graph (Source: Wikimedia Commons)
image::images/mtln_1102.png["A directed graph"]
In the context of the Lightning Network, our vertices are the Lightning nodes themselves, with our edges being the payment channels connecting these nodes. Because we're concerned with _routing payments_, in our model a node with no edges (no payment channels) isn't considered to be a part of the graph since it isn't useful.
Because channels themselves are UTXOs (funded 2-of-2 multisig addresses), we can view the channel graph as a special subset of the Bitcoin UTXO set, on top of which we can add some additional information (the nodes, etc.) to arrive at the final overlay structure, which is the channel graph. This anchoring of fundamental components of the channel graph in the
base Bitcoin blockchain means that it's impossible to _fake_ a valid channel graph, which has useful properties when it comes to spam prevention as we'll see later.
=== Gossip Protocol Messages
The channel graph information is propagated across the Lightning P2P Network as three messages, which are described in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7]:
+node_announcement+:: The vertex in our graph which communicates the public key of a node, as well as how to reach the node over the internet and some additional metadata describing the set of _features_ the node supports.
+channel_announcement+:: A blockchain anchored proof of the existence of a channel between two individual nodes. Any third party can verify this proof to ensure that a _real_ channel is actually being advertised. Similar to the +node_announcement+, this message also contains information describing the _capabilities_ of the channel, which is useful when attempting to route a payment.
+channel_update+:: A _pair_ of structures that describes the set of routing policies for a given channel. +channel_update+ messages come in a _pair_ because a channel is a directed edge, so each side of the channel is able to specify its own custom routing policy.
It's important to note that each of components of the channel graph are
themselves _authenticated_ allowing a third party to ensure that the owner of a channel/update/node is actually the one sending out an update. This effectively makes the channel graph a unique type of _authenticated data structure_ that cannot be counterfeited. For authentication, we use an +secp256k1+ ECDSA digital signature (or a series of them) over the serialized digest of the message itself. We won't get into the specific of the messaging framing/serialization used in the Lightning Network in this chapter, as we'll cover that information in <<wire_protocol>>.
With the high-level structure of the channel graph laid out, we'll now dive down into the precise structure of each of the three messages used to gossip the channel graph. We'll also explain how one can also verify each message and component of the channel graph.
[[node_announcement]]
==== The node_announcement Message
First, we have the +node_announcement+ message, which serves two primary
purposes:
1. To advertise connection information so other nodes can connect to a node either to bootstrap to the network or to attempt to establish a new payment channel with that node.
2. To communicate the set of protocol-level features (capabilities) a node understands/supports. Feature negotiation between nodes allows developers to add new features independently and support them with any other node on an opt-in basis.
Unlike channel announcements, node announcements are not anchored in
the base blockchain. Therefore, node announcements are
only considered valid if they have propagated with a corresponding channel announcement. In other words, we always reject nodes without payment channels to ensure a malicious peer can't flood the network with bogus nodes that are not part of the channel graph.
===== The node_announcement message structure
The +node_announcement+ is comprised of
the following fields:
+signature+:: A valid ECDSA signature that covers the serialized digest of all fields listed below. This signature must correspond to the public key of the advertised node.
+features+:: A bit vector that describes the set of protocol features that this node understands. We'll cover this field in more detail in <<feature_bits>> on the extensibility of the Lightning protocol. At a high level, this field carries a set of bits that represent the features a node understands. As an example, a node may signal that it understands the latest channel type.
+timestamp+:: A UNIX epoch encoded timestamp. This allows clients to enforce a partial ordering over the updates to a node's announcement.
+node_id+:: The +secp256k1+ public key that this node announcement belongs to. There can only be a single +node_announcement+ for a given node in the channel graph at any given time. As a result, a +node_announcement+ can supersede a prior +node_announcement+ for the same node if it carries a higher (later) timestamp.
+rgb_color+:: A field that allows a node to specify an RGB color to be associated with it, often used in channel graph visualizations and node directories.
+alias+:: A UTF-8 string to serve as the nickname for a given node. Note that these aliases aren't required to be globally unique, nor are they verified in any way. As a result, they should not be relied on as a form of identity—they can be easily spoofed.
+addresses+:: A set of public internet reachable addresses that are to be associated with a given node. In the current version of the protocol four address types are supported: IPv4 (type: 1), IPv6 (type: 2), Tor v2 (type: 3), Tor v3 (type: 4). In the +node_announcement+ message, each of these address types are denoted by an integer type which is included in parenthesis after the address type.
===== Validating node announcements
Validating an incoming +node_announcement+ is straightforward. The following assertions should be upheld when examining a node announcement:
* If an existing +node_announcement+ for that node is already known, then the +timestamp+ field of a new incoming +node_announcement+ must be greater than the prior one.
* With this constraint, we enforce a forced level of "freshness."
* If no +node_announcement+ exists for the given node, then an existing +channel_announcement+ that references the given node (more on that later) must already exist in one's local channel graph.
* The included +signature+ must be a valid ECDSA signature verified using the included +node_id+ public key and the double–SHA-256 digest of the raw message encoding (minus the signature and frame header) as the message.
* All included +addresses+ must be sorted in ascending order based on their address identifier.
* The included +alias+ bytes must be a valid UTF-8 string.
==== The channel_announcement Message
Next, we have the +channel_announcement+ message, which is used to _announce_ a new _public_ channel to the wider network. Note that announcing a channel is _optional_. A channel only needs to be announced if it is intended to be used for routing by the Lightning Network. Active routing nodes may wish to announce all their channels. However, certain nodes like mobile nodes likely don't have the
uptime or desire to be an active routing node. As a result, these
mobile nodes (which typically use light clients to connect to the Bitcoin P2P network) instead may have purely _unannounced_ (private) channels.
===== Unannounced (private) channels
An unannounced channel isn't part of the known public channel graph, but can still be used to send/receive payments. An astute reader may now be wondering how a channel which isn't part of the public channel graph is able to receive payments. The solution to this problem is a set of "pathfinding helpers" that we call routing hints. As we'll see in <<invoices>>, invoices created by nodes with unadvertised channels will include information to help the sender route to them assuming the node has at least a single channel with an existing public routing node.
Due to the existence of unadvertised channels, the _true_ size of the channel graph (both the public and private components) is unknown.
===== Locating a channel on the bitcoin blockchain
As mentioned earlier, the channel graph is authenticated due to its usage of public key cryptography, as well as the Bitcoin blockchain as a spam prevention system. To have a node accept a new +channel_announcement+, the advertisement must _prove_ that the channel actually exists in the Bitcoin blockchain. This proof system adds an upfront cost to adding a new entry to the channel graph (the on-chain fees one must pay to create the UTXO of the channel). As a result, we mitigate spam and ensure that a dishonest node on the network can't fill up the memory of an honest node at no cost with bogus channels.
Given that we need to construct a proof of the existence of a channel, a
natural question that arises is: how do we "point to" or reference a given channel for the verifier? Given that a payment channel is anchored in an unspent transaction output (see <<utxo>>), an initial thought might be to first attempt to advertise the full outpoint (+txid:index+) of the channel. Given the outpoint is globally unique and confirmed in the chain, this sounds like a good idea; however, it has a drawback: the verifier must maintain a full copy of the UTXO set to verify channels. This works fine for Bitcoin full nodes, but clients that rely on lightweight verification don't typically maintain a full UTXO set. Because we want to ensure we can support mobile nodes in the Lightning Network, we're forced to find another solution.
What if rather than referencing a channel by its UTXO, we reference it based on its "location" in the chain? To do this, we'll need a scheme that allows us to reference a given block, then a transaction within that block, and finally a specific output created by that transaction. Such an identifier is described in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7] and is referred to as a _short channel ID_, or +scid+.
The +scid+ is used both in +channel_announcement+ (and +channel_update+) as well as within the onion encrypted routing packet included within HTLCs as we learned <<onion_routing>>.
[[short_channel_id]]
[[scid]]
===== The short channel id
Based on the preceding information, we have three pieces of information we need to encode to uniquely reference a given channel. Because we want a compact representation, we'll attempt to encode the information into a _single_ integer. Our integer format of choice is an unsigned 64-bit integer, comprised of 8 bytes.
First, the block height: Using 3 bytes (24 bits) we can encode 16,777,216 blocks. That leaves 5 bytes for us to encode the transaction index and the output index, respectively. We'll use the next 3
bytes to encode the transaction index _within_ a block. This is more than enough given that it's only possible to fix tens of thousands of transactions in a block at current block sizes. This leaves 2 bytes left for us to encode the output index of the channel within the transaction.
Our final +scid+ format resembles:
----
block_height (3 bytes) || transaction_index (3 bytes) || output_index (2 bytes)
----
Using bit packing techniques, we first encode the most significant 3 bytes as the block height, the next 3 bytes as the transaction index, and the least significant 2 bytes as the output index of that creates the channel UTXO.
A short channel ID can be represented as a single integer
(+695313561322258433+) or as a more human friendly string: +632384x1568x1+. Here we see the channel was mined in block +632384+, was the ++1568++th transaction in the block, with the channel output as the second (UTXOs are zero-indexed) output produced by the transaction.
Now that we're able to succinctly point to a given channel funding output in the chain, we can examine the full structure of the +channel_announcement+ message, as well as how to verify the proof-of-existence included within the message.
===== The channel_announcement message structure
A +channel_announcement+ primarily communicates two things:
1. A proof that a channel exists between node A and node B with both nodes controlling the mulitsig keys in that channel output.
2. The set of capabilities of the channel (what types of HTLCs can it route, etc.).
When describing the proof, we'll typically refer to node +1+ and node +2+. Out of the two nodes that a channel connects, the "first" node is the node that has a "lower" public key encoding when we compare the public key of the two nodes in compressed format hex-encoded in lexicographical order. Correspondingly, in addition to a node public key on the network, each node should also control a public key within the Bitcoin blockchain.
Similar to the +node_announcement+ message, all included signatures of the +channel_announcement+ message should be signed/verified against the raw encoding of the message (minus the header) that follows _after_ the final signature (because it isn't possible for a digital signature to sign itself).
With that said, a +channel_announcement+ message has the following fields:
+node_signature_1+:: The signature of the first node over the message digest.
+node_signature_2+:: The signature of the second node over the message digest.
+bitcoin_signature_1+:: The signature of the multisig key (in the funding output) of the first node over the message digest.
+bitcoin_signature_2+:: The signature of the multisig key (in the funding output) of the second node over the message digest.
+features+:: A feature bit vector that describes the set of protocol level features supported by this channel.
+chain_hash+:: A 32-byte hash which is typically the genesis block hash of the blockchain (e.g., Bitcoin mainnet) the channel was opened within.
+short_channel_id+:: The +scid+ that uniquely locates the given channel funding output within the blockchain.
+node_id_1+:: The public key of the first node in the network.
+node_id_2+:: The public key of the second node in the network.
+bitcoin_key_1+:: The raw multisig key for the channel funding output for the first node in the network.
+bitcoin_key_2+:: The raw multisig key for the channel funding output for the second node in the network.
===== Channel announcement validation
Now that we know what a +channel_announcement+ contains, we can look at how to verify the channel's existence on-chain.
Armed with the information in the +channel_announcement+, any Lightning node (even one without a full copy of the Bitcoin blockchain) can verify the existence and authenticity of the payment channel.
First, the verifier will use the short channel ID to find which Bitcoin block contains the channel funding output. With the block height information, the verifier can request only that specific block from a Bitcoin node. The block can then be linked back to the genesis block by following the block header chain backward (verifying the proof-of-work), confirming that this is in fact a block belonging to the Bitcoin blockchain.
Next, the verifier uses the transaction index number to identify the transaction ID of the transaction containing the payment channel. Most modern Bitcoin libraries will allow indexing into the transaction of a block based on the index of the transaction within the greater block.
Next, the verifier uses a Bitcoin library (in the verifier's language) to extract the relevant transaction according to its index within the block. The verifier will validate the transaction (checking that it is properly signed and produces the same transaction ID when hashed).
Next, the verifier will extract the Pay-to-Witness-Script-Hash output referenced by the output index number of the short channel ID. This is the address of the channel funding output. Additionally, the verifier will ensure that the size of the alleged channel matches the value of the output produced at the specified output index.
Finally, the verifier will reconstruct the multisig script from +bitcoin_key_1+ and +bitcoin_key_2+ and confirm that it produces the same address as in the output.
The verifier has now independently verified that the payment channel in the announcement is funded and confirmed on the Bitcoin blockchain!
==== The channel_update Message
The third and final message used in the gossip protocol is the +channel_update+ message. Two of these are generated for each payment channel (one by each channel partner) announcing their routing fees, timelock expectations, and capabilities.
The +channel_update+ message also contains a timestamp, allowing a node to update its routing fees and other expectations and capabilities by sending a new +channel_update+ message with a higher (later) timestamp that supersedes any older updates.
The +channel_update+ message contains the following fields:
+signature+:: A digital signature matching the node's public key, to authenticate the source and integrity of the channel update
+chain_hash+:: The hash of the genesis block of the chain containing the channel
+short_channel_id+:: The short channel ID to identify the channel
+timestamp+:: The timestamp of this update, to allow recipients to sequence updates and replace older updates
+message_flags+:: A bit field indicating the presence of additional fields in the +channel_update+ message
+channel_flags+:: A bit field showing the direction of the channel and other channel options
+cltv_expiry_delta+:: The timelock delta expectations of this node for routing (see <<onion_routing>>)
+htlc_minimum_msat+:: The minimum HTLC amount that will be routed
+fee_base_msat+:: The base fee that will be charged for routing
+fee_proportional_millionths+:: The proportional fee rate that will be charged for routing
+htlc_maximum_msat+ (+option_channel_htlc_max+):: The maximum amount that will be routed
A node that receives the +channel_update+ message can attach this metadata to the channel graph edge to enable pathfinding, as we will see in <<path_finding>>.
=== Ongoing Channel Graph Maintenance
The construction of a channel graph is not a one-time event, but rather an ongoing activity. As a node bootstraps into the network it will start receiving "gossip," in the form of the three update messages. It will use these messages to immediately start building a validated channel graph.
The more information a node receives, the better its "map" of the Lightning Network becomes and the more effective it can be at pathfinding and payment delivery.
A node won't only add information to the channel graph. It will also keep track of the last time a channel was updated and will delete "stale" channels that have not been updated in more than two weeks. Finally, if it sees that some node no longer has any channels, it will also remove that node.
The information collected from the gossip protocol is not the only information that can be stored in the channel graph. Different Lightning node implementations may attach other metadata to nodes and channels. For example, some node implementations calculate a "score" that evaluates a node's "quality" as a routing peer. This score is used as part of pathfinding to prioritize or deprioritize paths.
=== Conclusion
In this chapter, we've learned how Lightning nodes discover each
other, discover and update their node status, and communicate with one another. We've learned how channel graphs are created and maintained, and we've explored a few ways that the Lightning Network discourages bad actors or dishonest nodes from spamming the network.
[[gossip]]
== Gossip and the Channel Graph
((("channel graph", id="ix_11_gossip_channel_graph-asciidoc0", range="startofrange")))((("gossip protocol", id="ix_11_gossip_channel_graph-asciidoc1", range="startofrange")))In this chapter we will describe the Lightning Network's gossip protocol and how it is used by nodes to construct and maintain a channel graph. We will also review the DNS bootstrap mechanism used to find peers to "gossip" with.
The "Routing fees and Gossip relaying" section is highlighted by a double outline spanning the routing layer and peer 2 peer layer of <<LN_protocol_gossip_highlight>>.
[[LN_protocol_gossip_highlight]]
.The Lightning Network protocol suite
image::images/mtln_1101.png["The Lightning Network Protocol Suite"]
As we've learned already, the Lightning Network uses a source-based onion routing protocol to deliver a payment from a sender to the recipient.
To do this, the sending node must be able to construct a path of payment channels that connects it with the recipient, as we will see in <<path_finding>>.
Thus, the sender has to be able to map the Lightning Network by constructing a channel graph.
The _channel graph_ is the interconnected set of publicly advertised channels and the nodes that these channels interlink.
As channels are backed by a funding transaction that is happening on-chain, one might falsely believe that Lightning nodes could just extract the existing channels from the Bitcoin blockchain.
However this is only possible to a certain extent.
((("P2WSH (Pay-to-Witness-Script-Hash)")))((("Pay-to-Witness-Script-Hash (P2WSH)")))The funding transactions are Pay-to-Witness-Script-Hash (P2WSH) addresses, and the nature of the script (a 2-of-2 multisig) will only be revealed once the funding transaction output is spent.
Even if the nature of the script were known, it's important to remember that not all 2-of-2 multisig scripts correspond to payment channels.
There are even more reasons why looking at the Bitcoin blockchain might not be helpful.
For example, on the Lightning Network the Bitcoin keys that are used for signing are rotated by the nodes for every channel and update.
Thus, even if we could reliably detect funding transactions on the Bitcoin blockchain, we would not know which two nodes on the Lightning Network own that particular channel.
The Lightning Network solves this problem by implementing a _gossip protocol_.
Gossip protocols are typical for peer-to-peer (P2P) networks and allow nodes to share information with the whole network with just a few direct connections to peers.
Lightning nodes open encrypted peer-to-peer connections to each other and share (gossip) information that they have received from other peers.
As soon as a node wants to share some information, for example about a newly created channel, it sends a message to all its peers.
Upon receiving a message a node decides if the received message was novel and, if so, forwards the information to its peers.
In this way, if the peer-to-peer network is well connected, all new information that is necessary for the operation of the network will eventually be propagated to all other peers.
Obviously, if a new peer joins the network for the first time, it needs to know some other peers on the network, so it can connect to others and participate in the network.
In this chapter, we'll explore exactly _how_ Lightning nodes discover each other, discover and update their node status, and communicate with one another.
When most refer to the _network_ part of the Lightning Network, they're referring to the _channel graph_ which itself is a unique authenticated data structure _anchored_ in the base Bitcoin
blockchain.
However, the Lightning Network is also a peer-to-peer network of nodes that gossip information about payment channels and nodes. Usually for two peers to maintain a payment channel they need to talk to each other directly, which means that there will be a peer connection between them.
This suggests that the channel graph is a subnetwork of the peer-to-peer network.
However, this is not true because payment channels can remain open even if one or both peers go temporarily offline.
Let's revisit some of the terminology that we have used throughout the book, specifically looking at what they mean in terms of the channel graph and the peer-to-peer network (see <<network_terminology>>).
[[network_terminology]]
.Terminology of the different networks
[options="header"]
|===
| Channel graph |Peer-to-peer network
| channel | connection
| open | connect
| close | disconnect
| funding transaction | encrypted TCP/IP connection
| send | transmit
| payment | message
|===
Because the Lightning Network is a peer-to-peer network, some initial bootstrapping is required in order for peers to discover each other. Within this chapter we'll follow the story of a new peer connecting to the network for the first time and examine each step in the bootstrapping process from initial peer discovery to channel graph syncing and validation.
As an initial step, our new node needs to somehow _discover_ at least _one_ peer that is already connected to the network and has a full channel graph (as we'll see later, there's no canonical version of the channel graph). Using one of many initial bootstrapping protocols to find that first peer, after a connection is established, our new
peer now needs to _download_ and _validate_ the channel graph. Once the channel graph has been fully validated, our new peer is ready to start opening channels and sending payments on the network.
After initial bootstrap, a node on the network needs to continue to maintain its view of the channel graph by processing new channel routing policy updates, discovering and validating new channels, removing channels that have been closed on-chain, and finally pruning channels that fail to send out a proper "heartbeat" every two weeks or so.
Upon completion of this chapter, you will understand a key component of
the peer-to-peer Lightning Network: namely how peers discover each other and maintain a local copy (perspective) of the channel graph. We'll begin by exploring the story of a new node that has just booted up and needs to find other peers to connect to on the network.(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc1")))(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc0")))
=== Peer Discovery
((("gossip protocol","peer discovery", id="ix_11_gossip_channel_graph-asciidoc2", range="startofrange")))((("peer discovery", id="ix_11_gossip_channel_graph-asciidoc3", range="startofrange")))In this section, we'll begin to follow a new Lightning node that wishes to join the network through three steps:
. Discover a set of bootstrap peers
. Download and validate the channel graph
. Begin the process of ongoing maintenance of the channel graph itself
==== P2P Bootstrapping
((("bootstrapping","P2P")))((("initial peer bootstrapping")))((("P2P bootstrapping")))((("peer discovery","P2P bootstrapping")))Before doing any thing else, our new node first needs to discover a set of peers who are already part of the network. We call this process initial peer bootstrapping, and it's something that every peer-to-peer network needs to implement properly to ensure a robust, healthy network.
Bootstrapping new peers to existing peer-to-peer networks is a very well studied problem with several known solutions, each with their own distinct trade-offs. The simplest solution to this problem is simply to package a set of _hardcoded_ bootstrap peers into the packaged P2P node software. This is simple in that each new node has a list of bootstrap peers in the software they're running, but rather fragile given that if the set of bootstrap peers goes offline, then no new nodes will be able to join the network. Due to this fragility, this
option is usually used as a fallback in case none of the other P2P bootstrapping mechanisms work properly.
((("initial peer discovery")))Rather than hardcoding the set of bootstrap peers within the software/binary itself, we can instead allow peers to dynamically obtain a fresh/new set of bootstrap peers they can use to join the network. We'll call this process _initial peer discovery_. Typically we'll leverage
existing internet protocols to maintain and distribute a set of bootstrapping peers. A nonexhaustive list of protocols that have been used in the past to accomplish initial peer discovery includes:
* Domain Name Service (DNS)
* Internet Relay Chat (IRC)
* Hyper-Text Transfer Protocol (HTTP)
Similar to the Bitcoin protocol, the primary initial peer discovery mechanism used in the Lightning Network happens via DNS. Because initial peer discovery is a critical and universal task for the network, the process has been _standardized_ in https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10: DNS Bootstrap].
==== DNS Bootstrapping
((("bootstrapping","DNS", id="ix_11_gossip_channel_graph-asciidoc4", range="startofrange")))((("DNS bootstrapping", id="ix_11_gossip_channel_graph-asciidoc5", range="startofrange")))((("peer discovery","DNS bootstrapping", id="ix_11_gossip_channel_graph-asciidoc6", range="startofrange")))The https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10] document describes a standardized way of implementing peer
discovery using the DNS. Lightning's flavor of DNS based bootstrapping uses up to three distinct record types:
* +SRV+ records for discovering a set of _node public keys_.
* +A+ records for mapping a node's public key to its current +IPv4+ address.
* +AAA+ records for mapping a node's public key to its current +IPv6+ address.
Those somewhat familiar with the DNS protocol may already be familiar with the +A+ (name to IPv4 address) and +AAA+ (name to IPv6 address) record types, but not the +SRV+ type. The +SRV+ record type is used by protocols built on top of DNS to determine the _location_ for a specified service. In our context, the service in question is a given Lightning node, and the location its IP address. We need to use this additional record type because, unlike nodes within the Bitcoin protocol, we need both a public key _and_ an IP address to connect to a node. As we see in <<wire_protocol>>, the transport encryption protocol used in LN requires knowledge of the public key of a node before connecting, so as to implement identity hiding for nodes in the network.
===== A new peer's bootstrapping workflow
Before diving into the specifics of https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10], we'll first outline the high-level flow of a new node that wishes to use BOLT #10 to join the network.
First, a node needs to identify a single DNS server or set of DNS servers that understand BOLT #10 so they can be used for P2P bootstrapping.
While BOLT #10 uses lseed.bitcoinstats.com as the seed server, there exists no "official" set of DNS seeds for this purpose, but each of the major implementations maintains their own DNS seed, and they cross query each other's seeds for redundancy purposes. In <<dns_seeds>> you'll see a nonexhaustive list of some popular DNS seed servers.
[[dns_seeds]]
.Table of known Lightning DNS seed servers
[options="header"]
|===
| DNS server | Maintainer
| lseed.bitcoinstats.com | Christian Decker
| nodes.lightning.directory | Lightning Labs (Olaoluwa Osuntokun)
| soa.nodes.lightning.directory | Lightning Labs (Olaoluwa Osuntokun)
| lseed.darosior.ninja | Antoine Poinsot
|===
DNS seeds exist for both Bitcoin's mainnet and testnet. For the sake
of our example, we'll assume the existence of a valid BOLT #10 DNS seed at +nodes.lightning.directory+.
Next, our new node will issue an +SRV+ query to obtain a set of _candidate bootstrap peers_. The response to our query will be a series of bech32 encoded public keys. Because DNS is a text-based protocol, we can't send raw binary data, so an encoding scheme is required. BOLT #10 specifies a bech32 encoding due to its use in the wider Bitcoin ecosystem. The number of encoded public keys returned depends on the server returning the query, as well as all the resolvers that stand between the client and the authoritative server.
Using the widely available +dig+ command-line tool, we can query the _testnet_ version of the DNS seed mentioned previously with the following command:
----
$ dig @8.8.8.8 test.nodes.lightning.directory SRV
----
We use the +@+ argument to force resolution via Google's nameserver (with IP address 8.8.8.8) because they do not filter large SRV query responses. At the end of the command, we specify that we only want +SRV+ records to be returned. A sample response looks something like <<ex1101>>.
[[ex1101]]
.Title here
====
----
$ dig @8.8.8.8 test.nodes.lightning.directory SRV
; <<>> DiG 9.10.6 <<>> @8.8.8.8 test.nodes.lightning.directory SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43610
;; flags: qr rd ra; QUERY: 1, ANSWER: 25, AUTHORITY: 0, ADDITIONAL: 1
;; QUESTION SECTION:
;test.nodes.lightning.directory. IN SRV
;; ANSWER SECTION:
test.nodes.lightning.directory. 59 IN SRV 10 10 9735 <1>
ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory. <2>
test.nodes.lightning.directory. 59 IN SRV 10 10 15735 ln1qtgsl3efj8verd4z27k44xu0a59kncvsarxatahm334exgnuvwhnz8dkhx8.test.nodes.lightning.directory.
[...]
;; Query time: 89 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Dec 31 16:41:07 PST 2020
----
<1> TCP port number where LN node can be reached.
<2> Node public key (ID) encoded as a virtual domain name.
====
We've truncated the response for brevity and show only two of the returned responses. The responses contain a "virtual" domain name for a target node, then to the left we have the _TCP port_ where this node can be reached. The first response uses the standard TCP port for LN: +9735+. The second response uses a custom port, which is permitted by the protocol.
Next, we'll attempt to obtain the other piece of information we need to connect to a node: its IP address. Before we can query for this, however, we'll first _decode_ the bech32 encoding of the public key from the virtual domain name:
----
ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7
----
Decoding this bech32 string we obtain the following valid
+secp256k1+ public key:
----
026c64f5a7f24c6f7f0e1d6ec877f23b2f672fb48967c2545f227d70636395eaf3
----
Now that we have the raw public key, we'll ask the DNS server to _resolve_ the virtual host given so we can obtain the IP information (+A+ record) for the node, as shown in <<ex1102>>.
++++
<div id="ex1102" data-type="example">
<h5>Title here</h5>
<pre data-type="programlisting">$ dig ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory A
; &lt;&lt;&gt;&gt; DiG 9.10.6 &lt;&lt;&gt;&gt; ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory A
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 41934
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory. IN A
;; ANSWER SECTION:
ln1qfkxfad87fxx7lcwr4hvsalj8vhkwta539nuy4zlyf7hqcmrjh40xx5frs7.test.nodes.lightning.directory. 60 IN A <em>X.X.X.X</em> <a class="co" id="comarker1" href="#c01"><img src="callouts/1.png" alt="1"/></a>
;; Query time: 83 msec
;; SERVER: 2600:1700:6971:6dd0::1#53(2600:1700:6971:6dd0::1)
;; WHEN: Thu Dec 31 16:59:22 PST 2020
;; MSG SIZE rcvd: 138</pre>
<dl class="calloutlist">
<dt><a class="co" id="c01" href="#comarker1"><img src="callouts/1.png" alt="1"/></a></dt>
<dd><p>The DNS server returns an IP Address <em>X.X.X.X</em>. We’ve replaced it with X’s in the text here so as to avoid presenting a real IP address.</p></dd>
</dl></div>
++++
In the preceding command, we've queried the server so we can obtain an +IPv4+ (+A+ record) address for our target node (replaced by _X.X.X.X_ in the preceding example). Now that we have the raw public key, IP address, and TCP port, we can connect to the node transport protocol at:
----
026c64f5a7f24c6f7f0e1d6ec877f23b2f672fb48967c2545f227d70636395eaf3@X.X.X.X:9735
----
Querying the current DNS +A+ record for a given node can also be used to look up the _latest_ set of addresses. Such queries can be used to more quickly sync the latest addressing information for a node, compared to waiting for address updates on the gossip network (see <<node_announcement>>).
At this point in our journey, our new Lightning node has found its first
peer and established its first connection! Now we can begin the second phase of new peer bootstrapping: channel graph synchronization and validation.
First, we'll explore more of the intricacies of BOLT #10 itself to take a deeper look into how things work under the hood.(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc6")))(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc5")))(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc4")))
==== SRV Query Options
The https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10] standard is highly extensible due to its usage of nested
subdomains as a communication layer for additional query options. The
bootstrapping protocol allows clients to further specify the _type_ of nodes they're attempting to query for versus the default of receiving a random subset of nodes in the query responses.
The query option subdomain scheme uses a series of key-value pairs where the key itself is a _single letter_ and the remaining set of text is the value itself. The following query types exist in the current version of the https://github.com/lightningnetwork/lightning-rfc/blob/master/10-dns-bootstrap.md[BOLT #10] standards document:
+r+:: The _realm_ byte which is used to determine which chain or realm queries should be returned for. As is, the only value for this key is +0+ which denotes "Bitcoin."
+a+:: Allows clients to filter out returned nodes based on the _types_ of addresses they advertise. As an example, this can be used to only obtain nodes that advertise a valid IPv6 address. The value that follows this type is based on a bitfled that _indexes_ into the set of specified address _types_ that are defined in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7]. The default value for this field is +6+, which represents both IPv4 and IPv6 (bits 1 and 2 are set).
+l+:: A valid node public key serialized in compressed format. This allows a client to query for a specified node rather than receiving a set of random nodes.
+n+:: The number of records to return. The default value for this field is +25+.
An example query with additional query options looks something like the following:
----
r0.a2.n10.nodes.lightning.directory
----
Breaking down the query one key-value pair at a time, we gain the following
insights:
* +r0+: The query targets the Bitcoin realm
* +a2+: The query only wants IPv4 addresses to be returned
* +n10+: The query requests
Try some combinations of the various flags using the +dig+ DNS command-line tool yourself(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc3")))(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc2"))):
----
dig @8.8.8.8 r0.a6.nodes.lightning.directory SRV
----
=== The Channel Graph
((("channel graph","structure of")))Now that our new node is able to use the DNS bootstrapping protocol to connect to their very first peer, it can start to sync the channel graph! However, before we sync the channel graph, we'll need to learn exactly _what_ we mean by the channel graph. In this section we'll explore the precise _structure_ of the channel graph and examine the unique aspects of the channel graph compared to the typical abstract "graph" data structure which is well known/used in the field of computer science.
==== A Directed Graph
((("channel graph","directed graph")))((("directed graph")))A _graph_ in computer science is a special data structure composed of vertices (typically referred to as nodes) and edges (also known as links). Two nodes may be connected by one or more edges. The channel graph is also _directed_ given that a payment is able to flow in either direction over a given edge (a channel). An example of a _directed graph_ is shown in <<directed_graph>>.
[[directed_graph]]
.A directed graph (Source: Wikimedia Commons)
image::images/mtln_1102.png["A directed graph"]
In the context of the Lightning Network, our vertices are the Lightning nodes themselves, with our edges being the payment channels connecting these nodes. Because we're concerned with _routing payments_, in our model a node with no edges (no payment channels) isn't considered to be a part of the graph since it isn't useful.
Because channels themselves are UTXOs (funded 2-of-2 multisig addresses), we can view the channel graph as a special subset of the Bitcoin UTXO set, on top of which we can add some additional information (the nodes, etc.) to arrive at the final overlay structure, which is the channel graph. This anchoring of fundamental components of the channel graph in the
base Bitcoin blockchain means that it's impossible to _fake_ a valid channel graph, which has useful properties when it comes to spam prevention as we'll see later.
=== Gossip Protocol Messages
((("channel graph","gossip protocol messages", id="ix_11_gossip_channel_graph-asciidoc7", range="startofrange")))((("gossip protocol","messages", id="ix_11_gossip_channel_graph-asciidoc8", range="startofrange")))The channel graph information is propagated across the Lightning P2P Network as three messages, which are described in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7]:
+node_announcement+:: The vertex in our graph which communicates the public key of a node, as well as how to reach the node over the internet and some additional metadata describing the set of _features_ the node supports.
+channel_announcement+:: A blockchain anchored proof of the existence of a channel between two individual nodes. Any third party can verify this proof to ensure that a _real_ channel is actually being advertised. Similar to the +node_announcement+, this message also contains information describing the _capabilities_ of the channel, which is useful when attempting to route a payment.
+channel_update+:: A _pair_ of structures that describes the set of routing policies for a given channel. +channel_update+ messages come in a _pair_ because a channel is a directed edge, so each side of the channel is able to specify its own custom routing policy.
It's important to note that each of components of the channel graph are
themselves _authenticated_ allowing a third party to ensure that the owner of a channel/update/node is actually the one sending out an update. This effectively makes the channel graph a unique type of _authenticated data structure_ that cannot be counterfeited. For authentication, we use an +secp256k1+ ECDSA digital signature (or a series of them) over the serialized digest of the message itself. We won't get into the specific of the messaging framing/serialization used in the Lightning Network in this chapter, as we'll cover that information in <<wire_protocol>>.
With the high-level structure of the channel graph laid out, we'll now dive down into the precise structure of each of the three messages used to gossip the channel graph. We'll also explain how one can also verify each message and component of the channel graph.
[[node_announcement]]
==== The node_announcement Message
((("gossip protocol","node_announcement message", id="ix_11_gossip_channel_graph-asciidoc9", range="startofrange")))((("node_announcement message", id="ix_11_gossip_channel_graph-asciidoc10", range="startofrange")))First, we have the +node_announcement+ message, which serves two primary
purposes:
1. To advertise connection information so other nodes can connect to a node either to bootstrap to the network or to attempt to establish a new payment channel with that node.
2. To communicate the set of protocol-level features (capabilities) a node understands/supports. Feature negotiation between nodes allows developers to add new features independently and support them with any other node on an opt-in basis.
Unlike channel announcements, node announcements are not anchored in
the base blockchain. Therefore, node announcements are
only considered valid if they have propagated with a corresponding channel announcement. In other words, we always reject nodes without payment channels to ensure a malicious peer can't flood the network with bogus nodes that are not part of the channel graph.
===== The node_announcement message structure
((("node_announcement message","structure")))The +node_announcement+ is comprised of
the following fields:
+signature+:: A valid ECDSA signature that covers the serialized digest of all fields listed below. This signature must correspond to the public key of the advertised node.
+features+:: A bit vector that describes the set of protocol features that this node understands. We'll cover this field in more detail in <<feature_bits>> on the extensibility of the Lightning protocol. At a high level, this field carries a set of bits that represent the features a node understands. As an example, a node may signal that it understands the latest channel type.
+timestamp+:: A UNIX epoch encoded timestamp. This allows clients to enforce a partial ordering over the updates to a node's announcement.
+node_id+:: The +secp256k1+ public key that this node announcement belongs to. There can only be a single +node_announcement+ for a given node in the channel graph at any given time. As a result, a +node_announcement+ can supersede a prior +node_announcement+ for the same node if it carries a higher (later) timestamp.
+rgb_color+:: A field that allows a node to specify an RGB color to be associated with it, often used in channel graph visualizations and node directories.
+alias+:: A UTF-8 string to serve as the nickname for a given node. Note that these aliases aren't required to be globally unique, nor are they verified in any way. As a result, they should not be relied on as a form of identity—they can be easily spoofed.
+addresses+:: A set of public internet reachable addresses that are to be associated with a given node. In the current version of the protocol four address types are supported: IPv4 (type: 1), IPv6 (type: 2), Tor v2 (type: 3), Tor v3 (type: 4). In the +node_announcement+ message, each of these address types are denoted by an integer type which is included in parenthesis after the address type.
===== Validating node announcements
((("node_announcement message","validating")))Validating an incoming +node_announcement+ is straightforward. The following assertions should be upheld when examining a node announcement:
* If an existing +node_announcement+ for that node is already known, then the +timestamp+ field of a new incoming +node_announcement+ must be greater than the prior one.
* With this constraint, we enforce a forced level of "freshness."
* If no +node_announcement+ exists for the given node, then an existing +channel_announcement+ that references the given node (more on that later) must already exist in one's local channel graph.
* The included +signature+ must be a valid ECDSA signature verified using the included +node_id+ public key and the double–SHA-256 digest of the raw message encoding (minus the signature and frame header) as the message.
* All included +addresses+ must be sorted in ascending order based on their address identifier.
* The included +alias+ bytes must be a valid UTF-8 string.(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc10")))(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc9")))
==== The channel_announcement Message
((("channel_announcement message", id="ix_11_gossip_channel_graph-asciidoc11", range="startofrange")))((("gossip protocol","channel_announcement message", id="ix_11_gossip_channel_graph-asciidoc12", range="startofrange")))Next, we have the +channel_announcement+ message, which is used to _announce_ a new _public_ channel to the wider network. Note that announcing a channel is _optional_. A channel only needs to be announced if it is intended to be used for routing by the Lightning Network. Active routing nodes may wish to announce all their channels. However, certain nodes like mobile nodes likely don't have the
uptime or desire to be an active routing node. As a result, these
mobile nodes (which typically use light clients to connect to the Bitcoin P2P network) instead may have purely _unannounced_ (private) channels.
===== Unannounced (private) channels
((("channel_announcement message","unannounced (private) channels")))((("unannounced channels")))An unannounced channel isn't part of the known public channel graph, but can still be used to send/receive payments. An astute reader may now be wondering how a channel which isn't part of the public channel graph is able to receive payments. The solution to this problem is a set of "pathfinding helpers" that we call routing hints. As we'll see in <<invoices>>, invoices created by nodes with unadvertised channels will include information to help the sender route to them assuming the node has at least a single channel with an existing public routing node.
Due to the existence of unadvertised channels, the _true_ size of the channel graph (both the public and private components) is unknown.
===== Locating a channel on the bitcoin blockchain
((("blockchain","locating a channel on the Bitcoin blockchain")))((("channel_announcement message","locating a channel on the Bitcoin blockchain")))As mentioned earlier, the channel graph is authenticated due to its usage of public key cryptography, as well as the Bitcoin blockchain as a spam prevention system. To have a node accept a new +channel_announcement+, the advertisement must _prove_ that the channel actually exists in the Bitcoin blockchain. This proof system adds an upfront cost to adding a new entry to the channel graph (the on-chain fees one must pay to create the UTXO of the channel). As a result, we mitigate spam and ensure that a dishonest node on the network can't fill up the memory of an honest node at no cost with bogus channels.
Given that we need to construct a proof of the existence of a channel, a
natural question that arises is: how do we "point to" or reference a given channel for the verifier? Given that a payment channel is anchored in an unspent transaction output (see <<utxo>>), an initial thought might be to first attempt to advertise the full outpoint (+txid:index+) of the channel. Given the outpoint is globally unique and confirmed in the chain, this sounds like a good idea; however, it has a drawback: the verifier must maintain a full copy of the UTXO set to verify channels. This works fine for Bitcoin full nodes, but clients that rely on lightweight verification don't typically maintain a full UTXO set. Because we want to ensure we can support mobile nodes in the Lightning Network, we're forced to find another solution.
What if rather than referencing a channel by its UTXO, we reference it based on its "location" in the chain? To do this, we'll need a scheme that allows us to reference a given block, then a transaction within that block, and finally a specific output created by that transaction. Such an identifier is described in https://github.com/lightningnetwork/lightning-rfc/blob/master/07-routing-gossip.md[BOLT #7] and is referred to as a _short channel ID_, or +scid+.
The +scid+ is used both in +channel_announcement+ (and +channel_update+) as well as within the onion encrypted routing packet included within HTLCs as we learned <<onion_routing>>.
[[short_channel_id]]
[[scid]]
===== The short channel id
((("blockchain","short channel ID")))Based on the preceding information, we have three pieces of information we need to encode to uniquely reference a given channel. Because we want a compact representation, we'll attempt to encode the information into a _single_ integer. Our integer format of choice is an unsigned 64-bit integer, comprised of 8 bytes.
First, the block height: Using 3 bytes (24 bits) we can encode 16,777,216 blocks. That leaves 5 bytes for us to encode the transaction index and the output index, respectively. We'll use the next 3
bytes to encode the transaction index _within_ a block. This is more than enough given that it's only possible to fix tens of thousands of transactions in a block at current block sizes. This leaves 2 bytes left for us to encode the output index of the channel within the transaction.
Our final +scid+ format resembles:
----
block_height (3 bytes) || transaction_index (3 bytes) || output_index (2 bytes)
----
Using bit packing techniques, we first encode the most significant 3 bytes as the block height, the next 3 bytes as the transaction index, and the least significant 2 bytes as the output index of that creates the channel UTXO.
A short channel ID can be represented as a single integer
(+695313561322258433+) or as a more human friendly string: +632384x1568x1+. Here we see the channel was mined in block +632384+, was the ++1568++th transaction in the block, with the channel output as the second (UTXOs are zero-indexed) output produced by the transaction.
Now that we're able to succinctly point to a given channel funding output in the chain, we can examine the full structure of the +channel_announcement+ message, as well as how to verify the proof-of-existence included within the message.
===== The channel_announcement message structure
((("channel_announcement message","message structure")))A +channel_announcement+ primarily communicates two things:
1. A proof that a channel exists between node A and node B with both nodes controlling the mulitsig keys in that channel output.
2. The set of capabilities of the channel (what types of HTLCs can it route, etc.).
When describing the proof, we'll typically refer to node +1+ and node +2+. Out of the two nodes that a channel connects, the "first" node is the node that has a "lower" public key encoding when we compare the public key of the two nodes in compressed format hex-encoded in lexicographical order. Correspondingly, in addition to a node public key on the network, each node should also control a public key within the Bitcoin blockchain.
Similar to the +node_announcement+ message, all included signatures of the +channel_announcement+ message should be signed/verified against the raw encoding of the message (minus the header) that follows _after_ the final signature (because it isn't possible for a digital signature to sign itself).
With that said, a +channel_announcement+ message has the following fields:
+node_signature_1+:: The signature of the first node over the message digest.
+node_signature_2+:: The signature of the second node over the message digest.
+bitcoin_signature_1+:: The signature of the multisig key (in the funding output) of the first node over the message digest.
+bitcoin_signature_2+:: The signature of the multisig key (in the funding output) of the second node over the message digest.
+features+:: A feature bit vector that describes the set of protocol level features supported by this channel.
+chain_hash+:: A 32-byte hash which is typically the genesis block hash of the blockchain (e.g., Bitcoin mainnet) the channel was opened within.
+short_channel_id+:: The +scid+ that uniquely locates the given channel funding output within the blockchain.
+node_id_1+:: The public key of the first node in the network.
+node_id_2+:: The public key of the second node in the network.
+bitcoin_key_1+:: The raw multisig key for the channel funding output for the first node in the network.
+bitcoin_key_2+:: The raw multisig key for the channel funding output for the second node in the network.
===== Channel announcement validation
((("channel_announcement message","validation")))Now that we know what a +channel_announcement+ contains, we can look at how to verify the channel's existence on-chain.
Armed with the information in the +channel_announcement+, any Lightning node (even one without a full copy of the Bitcoin blockchain) can verify the existence and authenticity of the payment channel.
First, the verifier will use the short channel ID to find which Bitcoin block contains the channel funding output. With the block height information, the verifier can request only that specific block from a Bitcoin node. The block can then be linked back to the genesis block by following the block header chain backward (verifying the proof-of-work), confirming that this is in fact a block belonging to the Bitcoin blockchain.
Next, the verifier uses the transaction index number to identify the transaction ID of the transaction containing the payment channel. Most modern Bitcoin libraries will allow indexing into the transaction of a block based on the index of the transaction within the greater block.
Next, the verifier uses a Bitcoin library (in the verifier's language) to extract the relevant transaction according to its index within the block. The verifier will validate the transaction (checking that it is properly signed and produces the same transaction ID when hashed).
Next, the verifier will extract the Pay-to-Witness-Script-Hash output referenced by the output index number of the short channel ID. This is the address of the channel funding output. Additionally, the verifier will ensure that the size of the alleged channel matches the value of the output produced at the specified output index.
Finally, the verifier will reconstruct the multisig script from +bitcoin_key_1+ and +bitcoin_key_2+ and confirm that it produces the same address as in the output.
The verifier has now independently verified that the payment channel in the announcement is funded and confirmed on the Bitcoin blockchain!(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc12")))(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc11")))
==== The channel_update Message
((("channel_update message")))((("gossip protocol","channel_update message")))The third and final message used in the gossip protocol is the +channel_update+ message. Two of these are generated for each payment channel (one by each channel partner) announcing their routing fees, timelock expectations, and capabilities.
The +channel_update+ message also contains a timestamp, allowing a node to update its routing fees and other expectations and capabilities by sending a new +channel_update+ message with a higher (later) timestamp that supersedes any older updates.
The +channel_update+ message contains the following fields:
+signature+:: A digital signature matching the node's public key, to authenticate the source and integrity of the channel update
+chain_hash+:: The hash of the genesis block of the chain containing the channel
+short_channel_id+:: The short channel ID to identify the channel
+timestamp+:: The timestamp of this update, to allow recipients to sequence updates and replace older updates
+message_flags+:: A bit field indicating the presence of additional fields in the +channel_update+ message
+channel_flags+:: A bit field showing the direction of the channel and other channel options
+cltv_expiry_delta+:: The timelock delta expectations of this node for routing (see <<onion_routing>>)
+htlc_minimum_msat+:: The minimum HTLC amount that will be routed
+fee_base_msat+:: The base fee that will be charged for routing
+fee_proportional_millionths+:: The proportional fee rate that will be charged for routing
+htlc_maximum_msat+ (+option_channel_htlc_max+):: The maximum amount that will be routed
A node that receives the +channel_update+ message can attach this metadata to the channel graph edge to enable pathfinding, as we will see in <<path_finding>>.(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc8")))(((range="endofrange", startref="ix_11_gossip_channel_graph-asciidoc7")))
=== Ongoing Channel Graph Maintenance
((("channel graph","ongoing maintenance")))The construction of a channel graph is not a one-time event, but rather an ongoing activity. As a node bootstraps into the network it will start receiving "gossip," in the form of the three update messages. It will use these messages to immediately start building a validated channel graph.
The more information a node receives, the better its "map" of the Lightning Network becomes and the more effective it can be at pathfinding and payment delivery.
A node won't only add information to the channel graph. It will also keep track of the last time a channel was updated and will delete "stale" channels that have not been updated in more than two weeks. Finally, if it sees that some node no longer has any channels, it will also remove that node.
The information collected from the gossip protocol is not the only information that can be stored in the channel graph. Different Lightning node implementations may attach other metadata to nodes and channels. For example, some node implementations calculate a "score" that evaluates a node's "quality" as a routing peer. This score is used as part of pathfinding to prioritize or deprioritize paths.
=== Conclusion
In this chapter, we've learned how Lightning nodes discover each
other, discover and update their node status, and communicate with one another. We've learned how channel graphs are created and maintained, and we've explored a few ways that the Lightning Network discourages bad actors or dishonest nodes from spamming the network.

906
12_path_finding.asciidoc
Unescape Escape View File

@ -1,453 +1,453 @@
[[path_finding]]
== Pathfinding and Payment Delivery
Payment delivery on the Lightning Network depends on finding a path from the sender to the recipient, a process called _pathfinding_. Since the routing is done by the sender, the sender must find a suitable path to reach the destination. This path is then encoded in an onion, as we saw in <<onion_routing>>.
In this chapter we will examine the problem of pathfinding, understand how uncertainty about channel balances complicates this problem, and look at how a typical pathfinding implementation attempts to solve it.
=== Pathfinding in the Lightning Protocol Suite
Pathfinding, path selection, multipart payments (MPP), and the payment attempt trial and error loop occupy the majority of the payment layer at the top of the protocol suite.
These components are highlighted by a double outline in the protocol suite, shown in <<LN_protocol_pathfinding_highlight>>.
[[LN_protocol_pathfinding_highlight]]
.The Lightning Network protocol suite
image::images/mtln_1201.png["The Lightning Network Protocol Suite"]
==== Where Is the BOLT?
So far we've looked at several technologies that are part of the Lightning Network and we have seen their exact specification as part of a BOLT standard. You may be surprised to find that pathfinding is not part of the BOLTs!
That's because pathfinding isn't an activity that requires any form of coordination or interoperability between different implementations. As we've seen, the path is selected by the sender. Even though the routing details are specified in detail in the BOLTs, the path discovery and selection are left entirely up to the sender. So each node implementation can choose a different strategy/algorithm to find paths. In fact, the different node/client and wallet implementations can even compete and use their pathfinding algorithm as a point of differentiation.
=== Pathfinding: What Problem Are We Solving?
The term pathfinding may be somewhat misleading because it implies a search for _a single path_ connecting two nodes. In the beginning, when the Lightning Network was small and not well interconnected, the problem was indeed about finding a way to join payment channels to reach the recipient.
But, as the Lightning Network has grown explosively, the pathfinding problem's nature has shifted. In mid-2021, as we finish this book, the Lightning Network consists of 20,000 nodes connected by at least 55,000 public channels with an aggregate capacity of almost 2,000 BTC. A node has on average 8.8 channels, while the top 10 most connected nodes have between 400 and 2,000 channels _each_. A visualization of just a small subset of the LN channel graph is shown in <<lngraph>>.
[[lngraph]]
.A visualization of part of the Lightning Network as of July 2021
image::images/mtln_1202.png[]
[NOTE]
====
The network visualization in <<lngraph>> was produced with a simple python script you can find in code/lngraph in the book's repository.
====
If the sender and recipient are connected to other well-connected nodes and have at least one channel with adequate capacity, there will be thousands of paths. The problem becomes selecting the _best_ path that will succeed in payment delivery, out of a list of thousands of possible paths.
==== Selecting the Best Path
To select the best path, we have to first define what we mean by "best." There may be many different criteria such as:
* Paths with enough liquidity. Obviously if a path doesn't have enough liquidity to route our payment, then it is not a suitable path.
* Paths with low fees. If we have several candidates, we may want to select ones with lower fees.
* Paths with short timelocks. We may want to avoid locking our funds for too long and therefore select paths with shorter timelocks.
All of these criteria may be desirable to some extent, and selecting paths that are favorable across many dimensions is not an easy task. Optimization problems like this may be too complex to solve for the "best" solution, but often can be solved for some approximation of the optimal, which is good news because otherwise pathfinding would be an intractable problem.
==== Pathfinding in Math and Computer Science
Pathfinding in the Lightning Network falls under a general category of _graph theory_ in mathematics and the more specific category of _graph traversal_ in computer science.
A network such as the Lightning Network can be represented as a mathematical construct called a _graph_, where _nodes_ are connected to each other by _edges_ (equivalent to the payment channels). The Lightning Network forms a _directed graph_ because the nodes are linked _asymmetrically_, since the channel balance is split between the two channel partners and the payment liquidity is different in each direction. A directed graph with numerical capacity constraints on its edges is called a _flow network_, a mathematical construct used to optimize transportation and other similar networks. Flow networks can be used as a framework when solutions need to achieve a specific flow while minimizing cost, known as the minimum cost flow problem (MCFP).
==== Capacity, Balance, Liquidity
To better understand the problem of transporting satoshis from point A to point B, we need to better define three important terms: capacity, balance, and liquidity. We use these terms to describe a payment channel's ability to route a payment.
In a payment channel connecting A<-->B:
Capacity:: This is the aggregate amount of satoshis that were funded into the 2-of-2 multisig with the funding transaction. It represents the maximum amount of value held in the channel. The channel capacity is announced by the gossip protocol and is known to nodes.
Balance:: This is the amount of satoshis held by each channel partner that can be sent to the other channel partner. A subset of the balance of A can be sent in the direction (A->B) toward node B. A subset of the balance of B can be sent in the opposite direction (A<-B).
Liquidity:: The available (subset) balance that can actually be sent across the channel in one direction. Liquidity of A is equal to the balance of A minus the channel reserve and any pending HTLCs committed by A.
The only value known to the network (via gossip announcements) is the aggregate capacity of the channel. Some unknown portion of that capacity is distributed as each partner's balance. Some subset of that balance is available to send across the channel in one direction:
++++
<ul class="simplelist">
<li>capacity = balance(A) + balance(B)</li>
<li>liquidity(A) = balance(A) - channel_reserve(A) - pending_HTLCs(A)</li>
</ul>
++++
==== Uncertainty of Balances
If we knew the exact channel balances of every channel, we could compute one or more payment paths using any of the standard pathfinding algorithms taught in good computer science programs. But we don't know the channel balances; we only know the aggregate channel capacity, which is advertised by nodes in channel announcements. In order for a payment to succeed, there must be adequate balance on the sending side of the channel. If we don't know how the capacity is distributed between the channel partners, we don't know if there is enough balance in the direction we are trying to send the payment.
Balances are not announced in channel updates for two reasons: privacy and scalability. First, announcing balances would reduce the privacy of the Lightning Network because it would allow surveillance of payment by statistical analysis of the changes in balances. Second, if nodes announced balances (globally) with every payment, the Lightning Network's scaling would be as bad as that of on-chain Bitcoin transactions which are broadcast to all participants. Therefore, balances are not announced. To solve the pathfinding problem in the face of uncertainty of balances, we need innovative pathfinding strategies. These strategies must relate closely to the routing algorithm that is used, which is source-based onion routing in which it is the responsibility of the sender to find a path through the network.
The uncertainty problem can be described mathematically as a _range of liquidity_, indicating the lower and upper bounds of liquidity based on the information that is known. Since we know the capacity of the channel and we know the channel reserve balance (the minimum allowed balance on each end), the liquidity can be defined as:
++++
<ul class="simplelist">
<li>min(liquidity) = channel_reserve</li>
<li>max(liquidity) = capacity – channel_reserve</li>
</ul>
++++
or as a range:
++++
<ul class="simplelist">
<li>channel_reserve &lt;= liquidity &lt;= (capacity – channel_reserve)</li>
</ul>
++++
Our channel liquidity uncertainty range is the range between the minimum and maximum possible liquidity. This is unknown to the network, except the two channel partners. However, as we will see, we can use failed HTLCs returned from our payment attempts to update our liquidity estimate and reduce uncertainty. If for example we get an HTLC failure code that tells us that a channel cannot fulfill an HTLC that is smaller than our estimate for maximum liquidity, that means the maximum liquidity can be updated to the amount of the failed HTLC. In simpler terms, if we think the liquidity can handle an HTLC of _N_ satoshis and we find out it fails to deliver _M_ satoshis (where _M_ is smaller), then we can update our estimate to __M__–1 as the upper bound. We tried to find the ceiling and bumped against it, so it's lower than we thought!
==== Pathfinding Complexity
Finding a path through a graph is a problem modern computers can solve rather efficiently.
Developers mainly choose breadth-first search if the edges are all of equal weight.
In cases where the edges are not of equal weight, an algorithm based on Dijkstra's algorithm is used, such as https://en.wikipedia.org/wiki/A*_search_algorithm[A* (pronounced "A-star")].
In our case the weights of the edges can represent the routing fees.
Only edges with a capacity larger than the amount to be sent will be included in the search.
In this basic form, pathfinding in the Lightning Network is very simple and straightforward.
However, channel liquidity is unknown to the sender. This turns our easy theoretical computer science problem into a rather complex real-world problem.
We now have to solve a pathfinding problem with only partial knowledge.
For example, we suspect which edges might be able to forward a payment because their capacity seems big enough.
But we can't be certain unless we try it out or ask the channel owners directly.
Even if we were able to ask the channel owners directly, their balance might change by the time we have asked others, computed a path, constructed an onion, and sent it along.
Not only do we have limited information but the information we have is highly dynamic and might change at any point in time without our knowledge.
==== Keeping It Simple
The pathfinding mechanism implemented in Lightning nodes is to first create a list of candidate paths, filtered and sorted by some function. Then, the node or wallet will probe paths (by attempting to deliver a payment) in a trial-and-error loop until a path is found that successfully delivers the payment.
[NOTE]
====
This probing is done by the Lightning node or wallet and is not directly observed by the user of the software.
However, the user might suspect that probing is taking place if the payment is not completed instantly.
====
While blind probing is not optimal and leaves ample room for improvement, it should be noted that even this simplistic strategy works surprisingly well for smaller payments and well-connected nodes.
Most Lightning node and wallet implementations improve on this approach by ordering/weighting the list of candidate paths. Some implementations order the candidate paths by cost (fees) or some combination of cost and capacity.
=== Pathfinding and Payment Delivery Process
Pathfinding and payment delivery involves several steps, which we list here. Different implementations may use different algorithms and strategies, but the basic steps are likely to be very similar:
. Create a _channel graph_ from announcements and updates containing the capacity of each channel and filter the graph ignoring any channels with insufficient capacity for the amount we want to send.
. Find paths connecting the sender to the recipient.
. Order the paths by some weight (this may be part of the previous step's algorithm).
. Try each path in order until payment succeeds (the trial-and-error loop).
. Optionally use the HTLC failure returns to update our graph, reducing uncertainty.
We can group these steps into three primary activities:
* Channel graph construction
* Pathfinding (filtered and ordered by some heuristics)
* Payment attempt(s)
These three activities can be repeated in a _payment round_ if we use the failure returns to update the graph, or if we are doing multipart payments (see <<mpp>>).
In the next sections we will look at each of these steps in more detail, as well as more advanced payment strategies.
=== Channel Graph Construction
In <<gossip>> we covered the three main messages that nodes use in their gossip: +node_announcement+, +channel_announcement+, and +channel_update+. These three messages allow any node to gradually construct a "map" of the Lightning Network in the form of a _channel graph_. Each of these messages provides a critical piece of information for the channel graph:
+node_announcement+:: This contains the information about a node on the Lightning Network, such as its node ID (public key), network address (e.g., IPv4/6 or Tor), capabilities/features, etc.
+channel_announcement+:: This contains the capacity and channel ID of a public (announced) channel between two nodes and proof of the channel's existence and ownership.
+channel_update+:: This contains a node's fee and timelock (CLTV) expectations for routing an outgoing (from that node's perspective) payment over a specified channel.
In terms of a mathematical graph, the +node_announcement+ is the information needed to create the nodes or _vertices_ of the graph. The +channel_announcement+ allows us to create the _edges_ of the graph representing the payment channels. Since each direction of the payment channel has its own balance, we create a directed graph. The +channel_update+ allows us to incorporate fees and timelocks to set the _cost_ or _weight_ of the graph edges.
Depending on the algorithm we will use for pathfinding, we may establish a number of different cost functions for the edges of the graph.
For now, let's ignore the cost function and simply establish a channel graph showing nodes and channels, using the +node_announcement+ and +channel_announcement+ messages.
In this chapter we will see how Selena attempts to find a path to pay Rashid 1,000,000 (1M) satoshis. To start, Selena is constructing a channel graph using the information from Lightning Network gossip to discover nodes and channels. Selena will then explore her channel graph to find a path to send a payment to Rashid.
This is _Selena's_ channel graph. There is no such thing as _the_ channel graph, there is only ever _a channel graph_, and it is always from the perspective of the node that has constructed it (see <<map_territory_relation>>).
[TIP]
====
Selena does not contruct a channel graph only when sending a payment. Rather, Selena's node is _continuously_ building and updating a channel graph. From the moment Selena's node starts and connects to any peer on the network it will participate in the gossip and use every message to learn as much as possible about the network.
====
[[map_territory_relation]]
.The Map-Territory Relation
****
From Wikipedia's https://en.wikipedia.org/wiki/Map%E2%80%93territory_relation[page on the Map–Territory Relation], "The map–territory relation describes the relationship between an object and a representation of that object, as in the relation between a geographical territory and a map of it."
The map–territory relation is best illustrated in "Sylvie and Bruno Concluded," a short story by Lewis Carroll which describes a fictional map that is 1:1 scale of the territory it maps, therefore having perfect accuracy but becoming completely useless as it would cover the entire territory if unfolded.
What does this mean for the Lightning Network? The Lightning Network is the territory, and a channel graph is a map of that territory.
While we could imagine a theoretical (Platonic ideal) channel graph that represents the complete, up-to-date map of the Lightning Network, such a map is simply the Lightning Network itself. Each node has its own channel graph which is constructed from announcements and is necessarily incomplete, incorrect, and out-of-date!
The map can never completely and accurately describe the territory.
****
Selena listens to +node_announcement+ messages and discovers four other nodes (in addition to Rashid, the intended recipient). The resulting graph represents a network of six nodes: Selena and Rashid are the Sender and Recipient respectively; Alice, Bob, Xavier and Yan are intermediary nodes. Selena's initial graph is just a list of nodes, shown in <<channel_graph_nodes>>.
[[channel_graph_nodes]]
.Node announcements
image::images/mtln_1203.png[]
Selena also receives seven +channel_announcement+ messages with the corresponding channel capacities, allowing her to construct a basic "map" of the network, shown in <<channel_graph_1>>. (The names Alice, Bob, Selena, Xavier, Yan, and Rashid have been replaced by their initials: A, B, S, X, and R, respectively.)
[[channel_graph_1]]
.The channel graph
image::images/mtln_1204.png[]
===== Uncertainty in the channel graph
As you can see from <<channel_graph_1>>, Selena does not know any of the balances of the channels. Her initial channel graph contains the highest level of uncertainty.
But wait: Selena does know _some_ channel balances! She knows the balances of the channels that her own node has connected with other nodes. While this does not seem like much, it is in fact very important information for constructing a path—Selena knows the actual liquidity of her own channels. Let's update the channel graph to show this information. We will use a "?" symbol to represent the unknown balances, as shown in <<channel_graph_2>>.
[[channel_graph_2]]
.Channel graph with known and unknown balances
image::images/mtln_1205.png[]
While the "?" symbol seems ominous, a lack of certainty is not the same as complete ignorance. We can _quantify_ the uncertainty and _reduce_ it by updating the graph with the successful/failed HTLCs we attempt.
Uncertainty can be quantified, because we know the maximum and minimum possible liquidity and can calculate probabilities for smaller (more precise) ranges.
Once we attempt to send an HTLC we can learn more about channel balances: if we succeed, then the balance was _at least_ sufficient to transport the specific amount. Meanwhile if we get a "temporary channel failure" error, the most likely reason is a lack of liquidity for the specific amount.
[TIP]
====
You may be thinking "What's the point of learning from a successful HTLC?" After all, if it succeeded we're "done." But consider that we may be sending one part of a multipart payment. We also may be sending other single-part payments within a short time. Anything we learn about liquidity is useful for the next attempt!
====
==== Liquidity Uncertainty and Probability
To quantify the uncertainty of a channel's liquidity, we can apply probability theory. A basic model of the probability of payment delivery will lead to some rather obvious, but important, conclusions:
* Smaller payments have a better chance of successful delivery across a path.
* Larger capacity channels will give us a better chance of payment delivery for a specific amount.
* The more channels (hops), the lower the chance of success.
While these may be obvious, they have important implications, especially for the use of multipart payments (see <<mpp>>). The math is not difficult to follow.
Let's use probability theory to see how we arrived at these conclusions.
First, let's posit that a channel with capacity +c+ has liquidity on one side with an unknown value in the range of +(0, c)+ or "range between 0 and c." For example, if the capacity is +5+, then the liquidity will be in the range +(0, 5)+. Now, from this we see that if we want to send 5 satoshis, our chance of success is only 1 in 6 (16.66%), because we will only succeed if the liquidity is exactly 5.
More simply, if the possible values for the liquidity are 0, 1, 2, 3, 4, and 5, only one of those six possible values will be sufficient to send our payment. To continue this example, if our payment amount was 3, then we would succeed if the liquidity was 3, 4, or 5. So our chances of success are 3 in 6 (50%). Expressed in math, the success probability function for a single channel is:
[latexmath]
++++
$P_c(a) = (c + 1 - a) / (c + 1)$
++++
where _a_ is the amount and _c_ is the capacity.
From the equation we see that if the amount is close to 0, the probability is close to 1, whereas if the amount exceeds the capacity, the probability is zero.
In other words: "Smaller payments have a better chance of successful delivery" or "Larger capacity channels give us better chances of delivery for a specific amount" and "You can't send a payment on a channel with insufficient capacity."
Now let's think about the probability of success across a path made of several channels. Let's say our first channel has 50% chance of success (_P_ = 0.5). Then if our second channel has 50% chance of success (_P_ = 0.5), it is intuitive that our overall chance is 25% (_P_ = 0.25).
We can express this as an equation that calculates the probability of a payment's success as the product of probabilities for each channel in the path(s):
[latexmath]
++++
$P_{payment} = \prod_{i=1}^n P_i$
++++
Where __P__~__i__~ is the probability of success over one path or channel, and __P__~__payment__~ is the overall probability of a successful payment over all the paths/channels.
From the equation we see that since the probability of success over a single channel is always less than or equal to 1, the probability across many channels will _drop exponentially_.
In other words, "The more channels (hops) you use, the lower the chance of success."
[NOTE]
====
There is a lot of mathematical theory and modeling behind the uncertainty of the liquidity in the channels. Fundamental work about modeling the uncertainty intervals of the channel liquidity can be found in the paper https://arxiv.org/abs/2103.08576["Security and Privacy of Lightning Network Payments with Uncertain Channel Balances"] by (coauthor of this book) Pickhardt et al.
====
==== Fees and Other Channel Metrics
Next, our sender will add information to the graph from +channel_update+ messages received from the intermediary nodes. As a reminder, the +channel_update+ contains a wealth of information about a channel and the expectations of one of the channel partners.
In <<channel_graph_3>> we see how Selena can update the channel graph based on +channel_update+ messages from A, B, X, and Y. Note that the channel ID and channel direction (included in +channel_flags+) tell Selena which channel and which direction this update refers to. Each channel partner gossips one or more +channel_update+ messages to announce their fee expectations and other information about the channel. For example, in the top left we see the +channel_update+ sent by Alice for the channel A--B and the direction A-to-B. With this update, Alice tells the network how much she will charge in fees to route an HTLC to Bob over that specific channel. Bob may announce a channel update (not shown in this diagram) for the opposite direction with completely different fee expectations. Any node may send a new +channel_update+ to change the fees or timelock expectations at any time.
[[channel_graph_3]]
.Channel graph fees and other channel metrics
image::images/mtln_1206.png[]
The fee and timelock information are very important not just as path selection metrics. As we saw in <<onion_routing>>, the sender needs to add up fees and timelocks (+cltv_expiry_delta+) at each hop to make the onion. The process of calculating fees happens from the recipient to the sender _backward_ along the path because each intermediary hop expects an incoming HTLC with higher amount and expiry timelock than the outgoing HTLC they will send to the next hop. So, for example, if Bob wants 1,000 satoshis in fees and 30 blocks of expiry timelock delta to send a payment to Rashid, then that amount and expiry delta must be added to the HTLC _from Alice_.
It is also important to note that a channel must have liquidity that is sufficient not only for the payment amount but also for the cumulative fees of all the subsequent hops. Even though Selena's channel to Xavier (S->X) has enough liquidity for a 1M satoshi payment, it _does not_ have enough liquidity once we consider fees. We need to know fees because only paths that have sufficient liquidity for _both payment and all fees_ will be considered.
=== Finding Candidate Paths
Finding a suitable path through a directed graph like this is a well-studied computer science problem (known broadly as the _shortest path problem_), which can be solved by a variety of algorithms depending on the desired optimization and resource constraints.
The most famous algorithm solving this problem was invented by Dutch mathematician E. W. Dijkstra in 1956, known simply as https://en.wikipedia.org/wiki/Dijkstra's_algorithm[_Dijkstra's algorithm_]. In addition to the original Dijkstra's algorithm, there are many variations and optimizations, such as https://en.wikipedia.org/wiki/A*_search_algorithm[A* ("A-star")], which is a heuristic-based algorithm.
As mentioned previously, the "search" must be applied _backward_ to account for fees that are accumulated from recipient to sender. Thus, Dijkstra, A*, or some other algorithm would search for a path from the recipient to the sender, using fees, estimated liquidity, and timelock delta (or some combination) as a cost function for each hop.
Using one such algorithm, Selena calculates several possible paths to Rashid, sorted by shortest path:
1. S->A->B->R
2. S->X->Y->R
3. S->X->B->R
4. S->A->B->X->Y->R
But, as we saw previously, the channel +S->X+ does not have enough liquidity for a 1M satoshi payment once fees are considered. So Paths 2 and 3 are not viable. That leaves Paths 1 and 4 as possible paths for the payment.
With two possible paths, Selena is ready to attempt delivery!
=== Payment Delivery (Trial-And-Error Loop)
Selena's node starts the trial-and-error loop by constructing the HTLCs, building the onion, and attempting delivery of the payment. For each attempt, there are three possible outcomes:
- A successful result (+update_fulfill_htlc+)
- An error (+update_fail_htlc+)
- A "stuck" payment with no response (neither success nor failure)
If the payment fails, then it can be retried via a different path by updating the graph (changing a channel's metrics) and recalculating an alternative path.
We looked at what happens if the payment is "stuck" in <<stuck_payments>>. The important detail is that a stuck payment is the worst outcome because we cannot retry with another HTLC because both (the stuck one and the retry one) might go through eventually and cause a double payment.
==== First Attempt (Path #1)
Selena attempts the first path (S->A->B->R). She constructs the onion and sends it, but receives a failure code from Bob's node. Bob reports back a +temporary channel failure+ with a +channel_update+ identifying the channel B->R as the one that can't deliver. This attempt is shown in <<path_1_fail>>.
[[path_1_fail]]
.Path 1 attempt fails
image::images/mtln_1207.png[]
===== Learning from failure
From this failure code, Selena will deduce that Bob doesn't have enough liquidity to deliver the payment to Rashid on that channel. Importantly, this failure narrows the uncertainty of the liquidity of that channel! Previously, Selena's node assumed that the liquidity on Bob's side of the channel was somewhere in the range (0, 4M). Now, she can assume that the liquidity is in the range (0, 999999). Similarly, Selena can now assume that the liquidity of that channel on Rashid's side is in the range (1M, 4M), instead of (0, 4M). Selena has learned a lot from this failure.
==== Second Attempt (Path #4)
Now Selena attempts the fourth candidate path (S->A->B->X->Y->R). This is a longer path and will incur more fees, but it's now the best option for delivery of the payment.
Fortunately, Selena receives an +update_fulfill_htlc+ message from Alice, indicating that the payment was successful, as shown in <<path_4_success>>.
[[path_4_success]]
.Path 4 attempt succeeds
image::images/mtln_1208.png[]
===== Learning from success
Selena has also learnt a lot from this successful payment. She now knows that all the channels on the path S->A->B->X->Y->R had enough liquidity to deliver the payment. Furthermore, she now knows that each of these channels has moved the HTLC amount (1M &#x2b; fees) to the other end of the channel. This allows Selena to recalculate the range of liquidity on the receiving side of all the channels in that path, replacing the minimum liquidity with 1M &#x2b; fees.
===== Stale knowledge?
Selena now has a much better "map" of the Lightning Network (at least as far as these seven channels go). This knowledge will be useful for any subsequent payments that Selena attempts to make.
However, this knowledge becomes somewhat "stale" as the other nodes send or route payments. Selena will never see any of these payments (unless she is the sender). Even if she is involved in routing payments, the onion routing mechanism means she can only see the changes for one hop (her own channels).
Therefore, Selena's node must consider how long to keep this knowledge before assuming that it is stale and no longer useful.
[[mpp]]
=== Multipart Payments (MPP)
_Multipart payments (MPP)_ are a feature that was introduced in the Lightning Network in 2020 and is already very widely available. Multipart payments allow a payment to be split into multiple _parts_ which are sent as HTLCs over several different paths to the intended recipient, preserving the _atomicity_ of the overall payment. In this context, atomicity means that either all the HTLC parts of a payment are eventually fulfilled or the entire payment fails and all the HTLC parts fail. There is no possibility of a partially successful payment.
Multipart payments are a significant improvement in the Lightning Network because they make it possible to send amounts that won't "fit" in any single channel by splitting them into smaller amounts for which there is sufficient liquidity. Furthermore, multipart payments have been shown to increase the probability of a successful payment, as compared to a single-path payment.
[TIP]
====
Now that MPP is available it is best to think of a single-path payment as a subcategory of an MPP. Essentially, a single-path is just a multipart of size one. All payments can be considered as multipart payments unless the size of the payment and liquidity available make it possible to deliver with a single part.
====
==== Using MPP
MPP is not something that a user will select, but rather it is a node pathfinding and payment delivery strategy. The same basic steps are implemented: create a graph, select paths, and the trial-and-error loop. The difference is that during path selection we must also consider how to split the payment to optimize delivery.
In our example we can see some immediate improvements to our pathfinding problem that become possible with MPP. First, we can utilize the S->X channel that has known insufficient liquidity to transport 1M satoshis plus fees. By sending a smaller part along that channel, we can use paths that were previously unavailable. Second, we have the unknown liquidity of the B->R channel, which is insufficient to transport the 1M amount, but might be sufficient to transport a smaller amount.
===== Splitting payments
The fundamental question is how to split the payments. More specifically, what are the optimal number of splits and the optimal amounts for each split?
This is an area of ongoing research where novel strategies are emerging. Multipart payments lead to a different algorithmic approach than single-path payments, even though single-path solutions can emerge from a multipart optimization (i.e., a single path may be the optimal solution suggested by a multipart pathfinding algorithm).
If you recall, we found that the uncertainty of liquidity/balances leads to some (somewhat obvious) conclusions that we can apply in MPP pathfinding, namely:
* Smaller payments have a higher chance of succeeding.
* The more channels you use, the chance of success becomes (exponentially) lower.
From the first of these insights, we might conclude that splitting a large payment (e.g., 1 million satoshis) into tiny payments increases the chance that each of those smaller payments will succeed. The number of possible paths with sufficient liquidity will be greater if we send smaller amounts.
To take this idea to an extreme, why not split the 1M satoshi payment into one million separate 1-satoshi parts? Well, the answer lies in our second insight: since we would be using more channels/paths to send our million single-satoshi HTLCs, our chance of success would drop exponentially.
If it's not obvious, the two preceding insights create a "sweet spot" where we can maximize our chances of success: splitting into smaller payments but not too many splits!
Quantifying this optimal balance of size/number-of-splits for a given channel graph is out of the scope of this book, but it is an active area of research. Some current implementations use a very simple strategy of splitting the payment in two halves, four quarters, etc.
[NOTE]
====
To read more about the optimization problem known as minimum-cost flows involved when splitting payments into different sizes and allocating them to paths see the paper https://arxiv.org/abs/2107.05322["Optimally Reliable & Cheap Payment Flows on the Lightning Network"] by (coauthor of this book) René Pickhardt and Stefan Richter.
====
In our example, Selena's node will attempt to split the 1M satoshi payment into two parts with 600k and 400k satoshi, respectively, and send them on two different paths. This is shown in <<mpp_paths>>.
[[mpp_paths]]
.Sending two parts of a multipart payment
image::images/mtln_1209.png[]
Because the S->X channel can now be utilized, and (luckily for Selena), the B->R channel has sufficient liquidity for 600k satoshis, the two parts are successful along paths that were previously not possible.
==== Trial-And-Error Over Multiple "Rounds"
Multipart payments lead to a somewhat modified trial-and-error loop for payment delivery. Because we are attempting multiple paths in each attempt, we have four possible outcomes:
* All parts succeed, the payment is successful
* Some parts succeed, some fail with errors returned
* All parts fail with errors returned
* Some parts are "stuck," no errors are returned
In the second case, where some parts fail with errors returned and some parts succeed, we can now _repeat_ the trial-and-error loop, but _only for the residual amount_.
Let's assume for example that Selena had a much larger channel graph with hundreds of possible paths to reach Rashid. Her pathfinding algorithm might find an optimal payment split consisting of 26 parts of varying sizes. After attempting to send all 26 parts in the first round, three of those parts failed with errors.
If those three parts consisted of, say 155k satoshis, then Selena would restart the pathfinding effort, only for 155k satoshis. The next round could find completely different paths (optimized for the residual amount of 155k), and split the 155k amount into completely different splits!
[TIP]
====
While it seems like 26 split parts are a lot, tests on the Lightning Network have successfully delivered a payment of 0.3679 BTC by splitting it into 345 parts.
====
Furthermore, Selena's node would update the channel graph using the information gleaned from the successes and errors of the first round to find the most optimal paths and splits for the second round.
Let's say that Selena's node calculates that the best way to send the 155k residual is six parts split as 80k, 42k, 15k, 11k, 6.5k, and 500 satoshis. In the next round, Selena gets only one error, indicating that the 11k satoshi part failed. Again, Selena updates the channel graph based on the information gleaned and runs the pathfinding again to send the 11k residual. This time, she succeeds with 2 parts of 6k and 5k satoshis, respectively.
This multiround example of sending a payment using MPP is shown in <<mpp_rounds>>.
[[mpp_rounds]]
.Sending a payment in multiple rounds with MPP
image::images/mtln_1210.png[]
In the end, Selena's node used three rounds of pathfinding to send the 1M satoshis in 30 parts.
=== Conclusion
In this chapter we looked at pathfinding and payment delivery. We saw how to use the channel graph to find paths from a sender to a recipient. We also saw how the sender will attempt to deliver payments on a candidate path and repeat in a trial-and-error loop.
We also examined the uncertainty of channel liquidity (from the perspective of the sender) and the implications that has for pathfinding. We saw how we can quantify the uncertainty and use probability theory to draw some useful conclusions. We also saw how we can reduce uncertainty by learning from both successful and failed payments.
Finally, we saw how the newly deployed multipart payments feature allows us to split payments into parts, increasing the probability of success even for larger payments.
[[path_finding]]
== Pathfinding and Payment Delivery
((("pathfinding", id="ix_12_path_finding-asciidoc0", range="startofrange")))Payment ((("payment delivery", id="ix_12_path_finding-asciidoc1", range="startofrange")))delivery on the Lightning Network depends on finding a path from the sender to the recipient, a process called _pathfinding_. Since the routing is done by the sender, the sender must find a suitable path to reach the destination. This path is then encoded in an onion, as we saw in <<onion_routing>>.
In this chapter we will examine the problem of pathfinding, understand how uncertainty about channel balances complicates this problem, and look at how a typical pathfinding implementation attempts to solve it.
=== Pathfinding in the Lightning Protocol Suite
((("Lightning Network Protocol","pathfinding in")))((("pathfinding","Lightning Protocol Suite and")))Pathfinding, path selection, multipart payments (MPP), and the payment attempt trial and error loop occupy the majority of the payment layer at the top of the protocol suite.
These components are highlighted by a double outline in the protocol suite, shown in <<LN_protocol_pathfinding_highlight>>.
[[LN_protocol_pathfinding_highlight]]
.The Lightning Network protocol suite
image::images/mtln_1201.png["The Lightning Network Protocol Suite"]
==== Where Is the BOLT?
((("BOLT (Basis of Lightning Technology) standards documents","pathfinding and")))((("pathfinding","BOLT standard and")))So far we've looked at several technologies that are part of the Lightning Network and we have seen their exact specification as part of a BOLT standard. You may be surprised to find that pathfinding is not part of the BOLTs!
That's because pathfinding isn't an activity that requires any form of coordination or interoperability between different implementations. As we've seen, the path is selected by the sender. Even though the routing details are specified in detail in the BOLTs, the path discovery and selection are left entirely up to the sender. So each node implementation can choose a different strategy/algorithm to find paths. In fact, the different node/client and wallet implementations can even compete and use their pathfinding algorithm as a point of differentiation.
=== Pathfinding: What Problem Are We Solving?
((("pathfinding","nature of problem solved by", id="ix_12_path_finding-asciidoc2", range="startofrange")))The term pathfinding may be somewhat misleading because it implies a search for _a single path_ connecting two nodes. In the beginning, when the Lightning Network was small and not well interconnected, the problem was indeed about finding a way to join payment channels to reach the recipient.
But, as the Lightning Network has grown explosively, the pathfinding problem's nature has shifted. In mid-2021, as we finish this book, the Lightning Network consists of 20,000 nodes connected by at least 55,000 public channels with an aggregate capacity of almost 2,000 BTC. A node has on average 8.8 channels, while the top 10 most connected nodes have between 400 and 2,000 channels _each_. A visualization of just a small subset of the LN channel graph is shown in <<lngraph>>.
[[lngraph]]
.A visualization of part of the Lightning Network as of July 2021
image::images/mtln_1202.png[]
[NOTE]
====
The network visualization in <<lngraph>> was produced with a simple python script you can find in code/lngraph in the book's repository.
====
If the sender and recipient are connected to other well-connected nodes and have at least one channel with adequate capacity, there will be thousands of paths. The problem becomes selecting the _best_ path that will succeed in payment delivery, out of a list of thousands of possible paths.
==== Selecting the Best Path
((("pathfinding","selecting the best path")))To select the best path, we have to first define what we mean by "best." There may be many different criteria such as:
* Paths with enough liquidity. Obviously if a path doesn't have enough liquidity to route our payment, then it is not a suitable path.
* Paths with low fees. If we have several candidates, we may want to select ones with lower fees.
* Paths with short timelocks. We may want to avoid locking our funds for too long and therefore select paths with shorter timelocks.
All of these criteria may be desirable to some extent, and selecting paths that are favorable across many dimensions is not an easy task. Optimization problems like this may be too complex to solve for the "best" solution, but often can be solved for some approximation of the optimal, which is good news because otherwise pathfinding would be an intractable problem.
==== Pathfinding in Math and Computer Science
((("pathfinding","math and computer science")))Pathfinding in the Lightning Network falls under a general category of _graph theory_ in mathematics and the more specific category of _graph traversal_ in computer science.
A network such as the Lightning Network can be represented as a mathematical construct called a _graph_, where _nodes_ are connected to each other by _edges_ (equivalent to the payment channels). ((("directed graph")))The Lightning Network forms a _directed graph_ because the nodes are linked _asymmetrically_, since the channel balance is split between the two channel partners and the payment liquidity is different in each direction. ((("flow network")))A directed graph with numerical capacity constraints on its edges is called a _flow network_, a mathematical construct used to optimize transportation and other similar networks. Flow networks can be used as a framework when solutions need to achieve a specific flow while minimizing cost, known as the minimum cost flow problem (MCFP).
==== Capacity, Balance, Liquidity
((("pathfinding","capacity, balance, and liquidity")))To better understand the problem of transporting satoshis from point A to point B, we need to better define three important terms: capacity, balance, and liquidity. We use these terms to describe a payment channel's ability to route a payment.
In a payment channel connecting A<-->B:
Capacity:: ((("capacity, payment channel")))This is the aggregate amount of satoshis that were funded into the 2-of-2 multisig with the funding transaction. It represents the maximum amount of value held in the channel. The channel capacity is announced by the gossip protocol and is known to nodes.
Balance:: ((("balance, in payment channel")))This is the amount of satoshis held by each channel partner that can be sent to the other channel partner. A subset of the balance of A can be sent in the direction (A->B) toward node B. A subset of the balance of B can be sent in the opposite direction (A<-B).
Liquidity:: ((("liquidity","in payment channel")))The available (subset) balance that can actually be sent across the channel in one direction. Liquidity of A is equal to the balance of A minus the channel reserve and any pending HTLCs committed by A.
The only value known to the network (via gossip announcements) is the aggregate capacity of the channel. Some unknown portion of that capacity is distributed as each partner's balance. Some subset of that balance is available to send across the channel in one direction:
++++
<ul class="simplelist">
<li>capacity = balance(A) + balance(B)</li>
<li>liquidity(A) = balance(A) - channel_reserve(A) - pending_HTLCs(A)</li>
</ul>
++++
==== Uncertainty of Balances
((("pathfinding","uncertainty of balances")))If we knew the exact channel balances of every channel, we could compute one or more payment paths using any of the standard pathfinding algorithms taught in good computer science programs. But we don't know the channel balances; we only know the aggregate channel capacity, which is advertised by nodes in channel announcements. In order for a payment to succeed, there must be adequate balance on the sending side of the channel. If we don't know how the capacity is distributed between the channel partners, we don't know if there is enough balance in the direction we are trying to send the payment.
Balances are not announced in channel updates for two reasons: privacy and scalability. First, announcing balances would reduce the privacy of the Lightning Network because it would allow surveillance of payment by statistical analysis of the changes in balances. Second, if nodes announced balances (globally) with every payment, the Lightning Network's scaling would be as bad as that of on-chain Bitcoin transactions which are broadcast to all participants. Therefore, balances are not announced. To solve the pathfinding problem in the face of uncertainty of balances, we need innovative pathfinding strategies. These strategies must relate closely to the routing algorithm that is used, which is source-based onion routing in which it is the responsibility of the sender to find a path through the network.
((("range of liquidity")))The uncertainty problem can be described mathematically as a _range of liquidity_, indicating the lower and upper bounds of liquidity based on the information that is known. Since we know the capacity of the channel and we know the channel reserve balance (the minimum allowed balance on each end), the liquidity can be defined as:
++++
<ul class="simplelist">
<li>min(liquidity) = channel_reserve</li>
<li>max(liquidity) = capacity – channel_reserve</li>
</ul>
++++
or as a range:
++++
<ul class="simplelist">
<li>channel_reserve &lt;= liquidity &lt;= (capacity – channel_reserve)</li>
</ul>
++++
Our channel liquidity uncertainty range is the range between the minimum and maximum possible liquidity. This is unknown to the network, except the two channel partners. However, as we will see, we can use failed HTLCs returned from our payment attempts to update our liquidity estimate and reduce uncertainty. If for example we get an HTLC failure code that tells us that a channel cannot fulfill an HTLC that is smaller than our estimate for maximum liquidity, that means the maximum liquidity can be updated to the amount of the failed HTLC. In simpler terms, if we think the liquidity can handle an HTLC of _N_ satoshis and we find out it fails to deliver _M_ satoshis (where _M_ is smaller), then we can update our estimate to __M__–1 as the upper bound. We tried to find the ceiling and bumped against it, so it's lower than we thought!
==== Pathfinding Complexity
((("pathfinding","complexity")))Finding a path through a graph is a problem modern computers can solve rather efficiently.
Developers mainly choose breadth-first search if the edges are all of equal weight.
In cases where the edges are not of equal weight, an algorithm based on ((("Dijkstra&apos;s algorithm")))Dijkstra's algorithm is used, such as https://en.wikipedia.org/wiki/A*_search_algorithm[A* (pronounced "A-star")].
In our case the weights of the edges can represent the routing fees.
Only edges with a capacity larger than the amount to be sent will be included in the search.
In this basic form, pathfinding in the Lightning Network is very simple and straightforward.
However, channel liquidity is unknown to the sender. This turns our easy theoretical computer science problem into a rather complex real-world problem.
We now have to solve a pathfinding problem with only partial knowledge.
For example, we suspect which edges might be able to forward a payment because their capacity seems big enough.
But we can't be certain unless we try it out or ask the channel owners directly.
Even if we were able to ask the channel owners directly, their balance might change by the time we have asked others, computed a path, constructed an onion, and sent it along.
Not only do we have limited information but the information we have is highly dynamic and might change at any point in time without our knowledge.
==== Keeping It Simple
((("pathfinding","simplicity")))The pathfinding mechanism implemented in Lightning nodes is to first create a list of candidate paths, filtered and sorted by some function. Then, the node or wallet will probe paths (by attempting to deliver a payment) in a trial-and-error loop until a path is found that successfully delivers the payment.
[NOTE]
====
This probing is done by the Lightning node or wallet and is not directly observed by the user of the software.
However, the user might suspect that probing is taking place if the payment is not completed instantly.
====
While blind probing is not optimal and leaves ample room for improvement, it should be noted that even this simplistic strategy works surprisingly well for smaller payments and well-connected nodes.
Most Lightning node and wallet implementations improve on this approach by ordering/weighting the list of candidate paths. Some implementations order the candidate paths by cost (fees) or some combination of cost and capacity.(((range="endofrange", startref="ix_12_path_finding-asciidoc2")))
=== Pathfinding and Payment Delivery Process
((("pathfinding","payment delivery process")))((("payment delivery","pathfinding and delivery process")))Pathfinding and payment delivery involves several steps, which we list here. Different implementations may use different algorithms and strategies, but the basic steps are likely to be very similar:
. Create a _channel graph_ from announcements and updates containing the capacity of each channel and filter the graph ignoring any channels with insufficient capacity for the amount we want to send.
. Find paths connecting the sender to the recipient.
. Order the paths by some weight (this may be part of the previous step's algorithm).
. Try each path in order until payment succeeds (the trial-and-error loop).
. Optionally use the HTLC failure returns to update our graph, reducing uncertainty.
We can group these steps into three primary activities:
* Channel graph construction
* Pathfinding (filtered and ordered by some heuristics)
* Payment attempt(s)
These three activities can be repeated in a _payment round_ if we use the failure returns to update the graph, or if we are doing multipart payments (see <<mpp>>).
In the next sections we will look at each of these steps in more detail, as well as more advanced payment strategies.
=== Channel Graph Construction
((("channel graph","construction of", id="ix_12_path_finding-asciidoc3", range="startofrange")))((("pathfinding","channel graph construction", id="ix_12_path_finding-asciidoc4", range="startofrange")))In <<gossip>> we covered the three main messages that nodes use in their gossip: +node_announcement+, +channel_announcement+, and +channel_update+. These three messages allow any node to gradually construct a "map" of the Lightning Network in the form of a _channel graph_. Each of these messages provides a critical piece of information for the channel graph:
+node_announcement+:: ((("node_announcement message")))This contains the information about a node on the Lightning Network, such as its node ID (public key), network address (e.g., IPv4/6 or Tor), capabilities/features, etc.
+channel_announcement+:: ((("channel_announcement message","channel graph and")))((("channel_update message")))This contains the capacity and channel ID of a public (announced) channel between two nodes and proof of the channel's existence and ownership.
+channel_update+:: This contains a node's fee and timelock (CLTV) expectations for routing an outgoing (from that node's perspective) payment over a specified channel.
In terms of a mathematical graph, the +node_announcement+ is the information needed to create the nodes or _vertices_ of the graph. The +channel_announcement+ allows us to create the _edges_ of the graph representing the payment channels. Since each direction of the payment channel has its own balance, we create a directed graph. The +channel_update+ allows us to incorporate fees and timelocks to set the _cost_ or _weight_ of the graph edges.
Depending on the algorithm we will use for pathfinding, we may establish a number of different cost functions for the edges of the graph.
For now, let's ignore the cost function and simply establish a channel graph showing nodes and channels, using the +node_announcement+ and +channel_announcement+ messages.
In this chapter we will see how Selena attempts to find a path to pay Rashid 1,000,000 (1M) satoshis. To start, Selena is constructing a channel graph using the information from Lightning Network gossip to discover nodes and channels. Selena will then explore her channel graph to find a path to send a payment to Rashid.
This is _Selena's_ channel graph. There is no such thing as _the_ channel graph, there is only ever _a channel graph_, and it is always from the perspective of the node that has constructed it (see <<map_territory_relation>>).
[TIP]
====
Selena does not contruct a channel graph only when sending a payment. Rather, Selena's node is _continuously_ building and updating a channel graph. From the moment Selena's node starts and connects to any peer on the network it will participate in the gossip and use every message to learn as much as possible about the network.
====
[[map_territory_relation]]
.The Map-Territory Relation
****
((("channel graph","map–territory relation")))From Wikipedia's https://en.wikipedia.org/wiki/Map%E2%80%93territory_relation[page on the Map–Territory Relation], "The map–territory relation describes the relationship between an object and a representation of that object, as in the relation between a geographical territory and a map of it."
The map–territory relation is best illustrated in "Sylvie and Bruno Concluded," a short story by Lewis Carroll which describes a fictional map that is 1:1 scale of the territory it maps, therefore having perfect accuracy but becoming completely useless as it would cover the entire territory if unfolded.
What does this mean for the Lightning Network? The Lightning Network is the territory, and a channel graph is a map of that territory.
While we could imagine a theoretical (Platonic ideal) channel graph that represents the complete, up-to-date map of the Lightning Network, such a map is simply the Lightning Network itself. Each node has its own channel graph which is constructed from announcements and is necessarily incomplete, incorrect, and out-of-date!
The map can never completely and accurately describe the territory.
****
Selena listens to +node_announcement+ messages and discovers four other nodes (in addition to Rashid, the intended recipient). The resulting graph represents a network of six nodes: Selena and Rashid are the Sender and Recipient respectively; Alice, Bob, Xavier and Yan are intermediary nodes. Selena's initial graph is just a list of nodes, shown in <<channel_graph_nodes>>.
[[channel_graph_nodes]]
.Node announcements
image::images/mtln_1203.png[]
Selena also receives seven +channel_announcement+ messages with the corresponding channel capacities, allowing her to construct a basic "map" of the network, shown in <<channel_graph_1>>. (The names Alice, Bob, Selena, Xavier, Yan, and Rashid have been replaced by their initials: A, B, S, X, and R, respectively.)
[[channel_graph_1]]
.The channel graph
image::images/mtln_1204.png[]
===== Uncertainty in the channel graph
((("channel graph","uncertainty in")))As you can see from <<channel_graph_1>>, Selena does not know any of the balances of the channels. Her initial channel graph contains the highest level of uncertainty.
But wait: Selena does know _some_ channel balances! She knows the balances of the channels that her own node has connected with other nodes. While this does not seem like much, it is in fact very important information for constructing a path—Selena knows the actual liquidity of her own channels. Let's update the channel graph to show this information. We will use a "?" symbol to represent the unknown balances, as shown in <<channel_graph_2>>.
[[channel_graph_2]]
.Channel graph with known and unknown balances
image::images/mtln_1205.png[]
While the "?" symbol seems ominous, a lack of certainty is not the same as complete ignorance. We can _quantify_ the uncertainty and _reduce_ it by updating the graph with the successful/failed HTLCs we attempt.
Uncertainty can be quantified, because we know the maximum and minimum possible liquidity and can calculate probabilities for smaller (more precise) ranges.
Once we attempt to send an HTLC we can learn more about channel balances: if we succeed, then the balance was _at least_ sufficient to transport the specific amount. Meanwhile if we get a "temporary channel failure" error, the most likely reason is a lack of liquidity for the specific amount.
[TIP]
====
You may be thinking "What's the point of learning from a successful HTLC?" After all, if it succeeded we're "done." But consider that we may be sending one part of a multipart payment. We also may be sending other single-part payments within a short time. Anything we learn about liquidity is useful for the next attempt!
====
==== Liquidity Uncertainty and Probability
((("channel graph","liquidity uncertainty and probability")))((("liquidity","uncertainty and probability")))To quantify the uncertainty of a channel's liquidity, we can apply probability theory. A basic model of the probability of payment delivery will lead to some rather obvious, but important, conclusions:
* Smaller payments have a better chance of successful delivery across a path.
* Larger capacity channels will give us a better chance of payment delivery for a specific amount.
* The more channels (hops), the lower the chance of success.
While these may be obvious, they have important implications, especially for the use of multipart payments (see <<mpp>>). The math is not difficult to follow.
Let's use probability theory to see how we arrived at these conclusions.
First, let's posit that a channel with capacity +c+ has liquidity on one side with an unknown value in the range of +(0, c)+ or "range between 0 and c." For example, if the capacity is +5+, then the liquidity will be in the range +(0, 5)+. Now, from this we see that if we want to send 5 satoshis, our chance of success is only 1 in 6 (16.66%), because we will only succeed if the liquidity is exactly 5.
More simply, if the possible values for the liquidity are 0, 1, 2, 3, 4, and 5, only one of those six possible values will be sufficient to send our payment. To continue this example, if our payment amount was 3, then we would succeed if the liquidity was 3, 4, or 5. So our chances of success are 3 in 6 (50%). Expressed in math, the success probability function for a single channel is:
[latexmath]
++++
$P_c(a) = (c + 1 - a) / (c + 1)$
++++
where _a_ is the amount and _c_ is the capacity.
From the equation we see that if the amount is close to 0, the probability is close to 1, whereas if the amount exceeds the capacity, the probability is zero.
In other words: "Smaller payments have a better chance of successful delivery" or "Larger capacity channels give us better chances of delivery for a specific amount" and "You can't send a payment on a channel with insufficient capacity."
Now let's think about the probability of success across a path made of several channels. Let's say our first channel has 50% chance of success (_P_ = 0.5). Then if our second channel has 50% chance of success (_P_ = 0.5), it is intuitive that our overall chance is 25% (_P_ = 0.25).
We can express this as an equation that calculates the probability of a payment's success as the product of probabilities for each channel in the path(s):
[latexmath]
++++
$P_{payment} = \prod_{i=1}^n P_i$
++++
Where __P__~__i__~ is the probability of success over one path or channel, and __P__~__payment__~ is the overall probability of a successful payment over all the paths/channels.
From the equation we see that since the probability of success over a single channel is always less than or equal to 1, the probability across many channels will _drop exponentially_.
In other words, "The more channels (hops) you use, the lower the chance of success."
[NOTE]
====
There is a lot of mathematical theory and modeling behind the uncertainty of the liquidity in the channels. Fundamental work about modeling the uncertainty intervals of the channel liquidity can be found in the paper https://arxiv.org/abs/2103.08576["Security and Privacy of Lightning Network Payments with Uncertain Channel Balances"] by (coauthor of this book) Pickhardt et al.
====
==== Fees and Other Channel Metrics
((("channel graph","fees and other channel metrics", id="ix_12_path_finding-asciidoc5", range="startofrange")))((("fees","channel graph and", id="ix_12_path_finding-asciidoc6", range="startofrange")))Next, our sender will add information to the graph from +channel_update+ messages received from the intermediary nodes. As a reminder, the +channel_update+ contains a wealth of information about a channel and the expectations of one of the channel partners.
In <<channel_graph_3>> we see how Selena can update the channel graph based on +channel_update+ messages from A, B, X, and Y. Note that the channel ID and channel direction (included in +channel_flags+) tell Selena which channel and which direction this update refers to. Each channel partner gossips one or more +channel_update+ messages to announce their fee expectations and other information about the channel. For example, in the top left we see the +channel_update+ sent by Alice for the channel A--B and the direction A-to-B. With this update, Alice tells the network how much she will charge in fees to route an HTLC to Bob over that specific channel. Bob may announce a channel update (not shown in this diagram) for the opposite direction with completely different fee expectations. Any node may send a new +channel_update+ to change the fees or timelock expectations at any time.
[[channel_graph_3]]
.Channel graph fees and other channel metrics
image::images/mtln_1206.png[]
The fee and timelock information are very important not just as path selection metrics. As we saw in <<onion_routing>>, the sender needs to add up fees and timelocks (+cltv_expiry_delta+) at each hop to make the onion. The process of calculating fees happens from the recipient to the sender _backward_ along the path because each intermediary hop expects an incoming HTLC with higher amount and expiry timelock than the outgoing HTLC they will send to the next hop. So, for example, if Bob wants 1,000 satoshis in fees and 30 blocks of expiry timelock delta to send a payment to Rashid, then that amount and expiry delta must be added to the HTLC _from Alice_.
It is also important to note that a channel must have liquidity that is sufficient not only for the payment amount but also for the cumulative fees of all the subsequent hops. Even though Selena's channel to Xavier (S->X) has enough liquidity for a 1M satoshi payment, it _does not_ have enough liquidity once we consider fees. We need to know fees because only paths that have sufficient liquidity for _both payment and all fees_ will be considered(((range="endofrange", startref="ix_12_path_finding-asciidoc6")))(((range="endofrange", startref="ix_12_path_finding-asciidoc5"))).(((range="endofrange", startref="ix_12_path_finding-asciidoc4")))(((range="endofrange", startref="ix_12_path_finding-asciidoc3")))
=== Finding Candidate Paths
((("pathfinding","finding candidate paths")))Finding a suitable path through a directed graph like this is a well-studied computer science problem (known broadly as the _shortest path problem_), which can be solved by a variety of algorithms depending on the desired optimization and resource constraints.
((("Dijkstra&apos;s algorithm")))The most famous algorithm solving this problem was invented by Dutch mathematician E. W. Dijkstra in 1956, known simply as https://en.wikipedia.org/wiki/Dijkstra's_algorithm[_Dijkstra's algorithm_]. In addition to the original Dijkstra's algorithm, there are many variations and optimizations, such as https://en.wikipedia.org/wiki/A*_search_algorithm[A* ("A-star")], which is a heuristic-based algorithm.
As mentioned previously, the "search" must be applied _backward_ to account for fees that are accumulated from recipient to sender. Thus, Dijkstra, A*, or some other algorithm would search for a path from the recipient to the sender, using fees, estimated liquidity, and timelock delta (or some combination) as a cost function for each hop.
Using one such algorithm, Selena calculates several possible paths to Rashid, sorted by shortest path:
1. S->A->B->R
2. S->X->Y->R
3. S->X->B->R
4. S->A->B->X->Y->R
But, as we saw previously, the channel +S->X+ does not have enough liquidity for a 1M satoshi payment once fees are considered. So Paths 2 and 3 are not viable. That leaves Paths 1 and 4 as possible paths for the payment.
With two possible paths, Selena is ready to attempt delivery!
=== Payment Delivery (Trial-And-Error Loop)
((("payment delivery", id="ix_12_path_finding-asciidoc7", range="startofrange")))Selena's ((("payment delivery","trial-and error loop", id="ix_12_path_finding-asciidoc8", range="startofrange")))((("trial-and error loop", id="ix_12_path_finding-asciidoc9", range="startofrange")))node starts the trial-and-error loop by constructing the HTLCs, building the onion, and attempting delivery of the payment. For each attempt, there are three possible outcomes:
- A successful result (+update_fulfill_htlc+)
- An error (+update_fail_htlc+)
- A "stuck" payment with no response (neither success nor failure)
If the payment fails, then it can be retried via a different path by updating the graph (changing a channel's metrics) and recalculating an alternative path.
We looked at what happens if the payment is "stuck" in <<stuck_payments>>. The important detail is that a stuck payment is the worst outcome because we cannot retry with another HTLC because both (the stuck one and the retry one) might go through eventually and cause a double payment.
==== First Attempt (Path #1)
Selena attempts the first path (S->A->B->R). She constructs the onion and sends it, but receives a failure code from Bob's node. Bob reports back a +temporary channel failure+ with a +channel_update+ identifying the channel B->R as the one that can't deliver. This attempt is shown in <<path_1_fail>>.
[[path_1_fail]]
.Path 1 attempt fails
image::images/mtln_1207.png[]
===== Learning from failure
From this failure code, Selena will deduce that Bob doesn't have enough liquidity to deliver the payment to Rashid on that channel. Importantly, this failure narrows the uncertainty of the liquidity of that channel! Previously, Selena's node assumed that the liquidity on Bob's side of the channel was somewhere in the range (0, 4M). Now, she can assume that the liquidity is in the range (0, 999999). Similarly, Selena can now assume that the liquidity of that channel on Rashid's side is in the range (1M, 4M), instead of (0, 4M). Selena has learned a lot from this failure.
==== Second Attempt (Path #4)
Now Selena attempts the fourth candidate path (S->A->B->X->Y->R). This is a longer path and will incur more fees, but it's now the best option for delivery of the payment.
Fortunately, Selena receives an +update_fulfill_htlc+ message from Alice, indicating that the payment was successful, as shown in <<path_4_success>>.
[[path_4_success]]
.Path 4 attempt succeeds
image::images/mtln_1208.png[]
===== Learning from success
Selena has also learnt a lot from this successful payment. She now knows that all the channels on the path S->A->B->X->Y->R had enough liquidity to deliver the payment. Furthermore, she now knows that each of these channels has moved the HTLC amount (1M &#x2b; fees) to the other end of the channel. This allows Selena to recalculate the range of liquidity on the receiving side of all the channels in that path, replacing the minimum liquidity with 1M &#x2b; fees.
===== Stale knowledge?
Selena now has a much better "map" of the Lightning Network (at least as far as these seven channels go). This knowledge will be useful for any subsequent payments that Selena attempts to make.
However, this knowledge becomes somewhat "stale" as the other nodes send or route payments. Selena will never see any of these payments (unless she is the sender). Even if she is involved in routing payments, the onion routing mechanism means she can only see the changes for one hop (her own channels).
Therefore, Selena's node must consider how long to keep this knowledge before assuming that it is stale and no longer useful(((range="endofrange", startref="ix_12_path_finding-asciidoc9")))(((range="endofrange", startref="ix_12_path_finding-asciidoc8"))).(((range="endofrange", startref="ix_12_path_finding-asciidoc7")))
[[mpp]]
=== Multipart Payments (MPP)
((("multipart payments (MPP)", id="ix_12_path_finding-asciidoc10", range="startofrange")))((("payment delivery","multipart payments", id="ix_12_path_finding-asciidoc11", range="startofrange")))_Multipart payments (MPP)_ are a feature that was introduced in the Lightning Network in 2020 and is already very widely available. Multipart payments allow a payment to be split into multiple _parts_ which are sent as HTLCs over several different paths to the intended recipient, preserving the _atomicity_ of the overall payment. In this context, atomicity means that either all the HTLC parts of a payment are eventually fulfilled or the entire payment fails and all the HTLC parts fail. There is no possibility of a partially successful payment.
Multipart payments are a significant improvement in the Lightning Network because they make it possible to send amounts that won't "fit" in any single channel by splitting them into smaller amounts for which there is sufficient liquidity. Furthermore, multipart payments have been shown to increase the probability of a successful payment, as compared to a single-path payment.
[TIP]
====
Now that MPP is available it is best to think of a single-path payment as a subcategory of an MPP. Essentially, a single-path is just a multipart of size one. All payments can be considered as multipart payments unless the size of the payment and liquidity available make it possible to deliver with a single part.
====
==== Using MPP
MPP is not something that a user will select, but rather it is a node pathfinding and payment delivery strategy. The same basic steps are implemented: create a graph, select paths, and the trial-and-error loop. The difference is that during path selection we must also consider how to split the payment to optimize delivery.
In our example we can see some immediate improvements to our pathfinding problem that become possible with MPP. First, we can utilize the S->X channel that has known insufficient liquidity to transport 1M satoshis plus fees. By sending a smaller part along that channel, we can use paths that were previously unavailable. Second, we have the unknown liquidity of the B->R channel, which is insufficient to transport the 1M amount, but might be sufficient to transport a smaller amount.
===== Splitting payments
((("multipart payments (MPP)","splitting payments", id="ix_12_path_finding-asciidoc12", range="startofrange")))((("payment","splitting", id="ix_12_path_finding-asciidoc13", range="startofrange")))The fundamental question is how to split the payments. More specifically, what are the optimal number of splits and the optimal amounts for each split?
This is an area of ongoing research where novel strategies are emerging. Multipart payments lead to a different algorithmic approach than single-path payments, even though single-path solutions can emerge from a multipart optimization (i.e., a single path may be the optimal solution suggested by a multipart pathfinding algorithm).
If you recall, we found that the uncertainty of liquidity/balances leads to some (somewhat obvious) conclusions that we can apply in MPP pathfinding, namely:
* Smaller payments have a higher chance of succeeding.
* The more channels you use, the chance of success becomes (exponentially) lower.
From the first of these insights, we might conclude that splitting a large payment (e.g., 1 million satoshis) into tiny payments increases the chance that each of those smaller payments will succeed. The number of possible paths with sufficient liquidity will be greater if we send smaller amounts.
To take this idea to an extreme, why not split the 1M satoshi payment into one million separate 1-satoshi parts? Well, the answer lies in our second insight: since we would be using more channels/paths to send our million single-satoshi HTLCs, our chance of success would drop exponentially.
If it's not obvious, the two preceding insights create a "sweet spot" where we can maximize our chances of success: splitting into smaller payments but not too many splits!
Quantifying this optimal balance of size/number-of-splits for a given channel graph is out of the scope of this book, but it is an active area of research. Some current implementations use a very simple strategy of splitting the payment in two halves, four quarters, etc.
[NOTE]
====
To read more about the optimization problem known as minimum-cost flows involved when splitting payments into different sizes and allocating them to paths see the paper https://arxiv.org/abs/2107.05322["Optimally Reliable & Cheap Payment Flows on the Lightning Network"] by (coauthor of this book) René Pickhardt and Stefan Richter.
====
In our example, Selena's node will attempt to split the 1M satoshi payment into two parts with 600k and 400k satoshi, respectively, and send them on two different paths. This is shown in <<mpp_paths>>.
[[mpp_paths]]
.Sending two parts of a multipart payment
image::images/mtln_1209.png[]
Because the S->X channel can now be utilized, and (luckily for Selena), the B->R channel has sufficient liquidity for 600k satoshis, the two parts are successful along paths that were previously not possible.(((range="endofrange", startref="ix_12_path_finding-asciidoc13")))(((range="endofrange", startref="ix_12_path_finding-asciidoc12")))
==== Trial-And-Error Over Multiple "Rounds"
((("multipart payments (MPP)","trial-and error over multiple &ldquo;rounds&rdquo;")))((("payment delivery","trial-and error loop")))((("trial-and error loop")))Multipart payments lead to a somewhat modified trial-and-error loop for payment delivery. Because we are attempting multiple paths in each attempt, we have four possible outcomes:
* All parts succeed, the payment is successful
* Some parts succeed, some fail with errors returned
* All parts fail with errors returned
* Some parts are "stuck," no errors are returned
In the second case, where some parts fail with errors returned and some parts succeed, we can now _repeat_ the trial-and-error loop, but _only for the residual amount_.
Let's assume for example that Selena had a much larger channel graph with hundreds of possible paths to reach Rashid. Her pathfinding algorithm might find an optimal payment split consisting of 26 parts of varying sizes. After attempting to send all 26 parts in the first round, three of those parts failed with errors.
If those three parts consisted of, say 155k satoshis, then Selena would restart the pathfinding effort, only for 155k satoshis. The next round could find completely different paths (optimized for the residual amount of 155k), and split the 155k amount into completely different splits!
[TIP]
====
While it seems like 26 split parts are a lot, tests on the Lightning Network have successfully delivered a payment of 0.3679 BTC by splitting it into 345 parts.
====
Furthermore, Selena's node would update the channel graph using the information gleaned from the successes and errors of the first round to find the most optimal paths and splits for the second round.
Let's say that Selena's node calculates that the best way to send the 155k residual is six parts split as 80k, 42k, 15k, 11k, 6.5k, and 500 satoshis. In the next round, Selena gets only one error, indicating that the 11k satoshi part failed. Again, Selena updates the channel graph based on the information gleaned and runs the pathfinding again to send the 11k residual. This time, she succeeds with 2 parts of 6k and 5k satoshis, respectively.
This multiround example of sending a payment using MPP is shown in <<mpp_rounds>>.
[[mpp_rounds]]
.Sending a payment in multiple rounds with MPP
image::images/mtln_1210.png[]
In the end, Selena's node used three rounds of pathfinding to send the 1M satoshis in 30 parts.(((range="endofrange", startref="ix_12_path_finding-asciidoc11")))(((range="endofrange", startref="ix_12_path_finding-asciidoc10")))
=== Conclusion
In this chapter we looked at pathfinding and payment delivery. We saw how to use the channel graph to find paths from a sender to a recipient. We also saw how the sender will attempt to deliver payments on a candidate path and repeat in a trial-and-error loop.
We also examined the uncertainty of channel liquidity (from the perspective of the sender) and the implications that has for pathfinding. We saw how we can quantify the uncertainty and use probability theory to draw some useful conclusions. We also saw how we can reduce uncertainty by learning from both successful and failed payments.
Finally, we saw how the newly deployed multipart payments feature allows us to split payments into parts, increasing the probability of success even for larger payments(((range="endofrange", startref="ix_12_path_finding-asciidoc1"))).(((range="endofrange", startref="ix_12_path_finding-asciidoc0")))

826
13_wire_protocol.asciidoc
Unescape Escape View File

@ -1,413 +1,413 @@
[[wire_protocol]]
== Wire Protocol: Framing and Extensibility
In this chapter, we dive into the wire protocol of the Lightning Network
and also cover all the various extensibility levers that have been built into
the protocol. By the end of this chapter, an ambitious reader should be able to
write their very own wire protocol parser for the Lightning Network. In addition
to being able to write a custom wire protocol parser, a reader of this chapter
will gain a deep understanding of the various upgrade mechanisms
that have been built into the protocol.
=== Messaging Layer in the Lightning Protocol Suite
The messaging layer, which is detailed in this chapter, consists of "Framing and message format," "Type Length Value (TLV)" encoding, and "Feature bits." These components are highlighted by a double outline in the protocol suite, shown in <<LN_protocol_wire_message_highlight>>.
[[LN_protocol_wire_message_highlight]]
.The Lightning Network protocol suite
image::images/mtln_1301.png["The Lightning Network Protocol Suite"]
=== Wire Framing
We begin by describing the high-level structure of the wire _framing_
within the protocol. When we say framing, we mean the way that the bytes are
packed on the wire to _encode_ a particular protocol message. Without knowledge
of the framing system used in the protocol, a string of bytes on the wire would
resemble a series of random bytes because no structure has been imposed. By applying
proper framing to decode these bytes on the wire, we'll be able to extract
structure and finally parse this structure into protocol messages within our
higher-level language.
It's important to note that the Lightning Network is an _end-to-end
encrypted_ protocol, and the wire framing is itself encapsulated within an
_encrypted_ message transport layer. As we see in <<encrypted_message_transport>>, the Lightning
Network uses a custom variant of the Noise Protocol to handle
transport encryption. Within this chapter, whenever we give an example of wire
framing, we assume the encryption layer has already been stripped away (when
decoding), or that we haven't yet encrypted the set of bytes before we send
them on the wire (encoding).
==== High-Level Wire Framing
With that said, we're ready to describe the high-level schema used to
encode messages on the wire:
* Messages on the wire begin with a _2-byte_ type field, followed by a
message payload.
* The message payload itself can be up to 65 KB in size.
* All integers are encoded in big-endian (network order).
* Any bytes that follow after a defined message can be safely ignored.
Yep, that's it. As the protocol relies on an _encapsulating_ transport protocol
encryption layer, we don't need an explicit length for each message type. This
is due to the fact that transport encryption works at the _message_ level, so
by the time we're ready to decode the next message, we already know the total
number of bytes of the message itself. Using 2 bytes for the message type
(encoded in big-endian) means that the protocol can have up to `2^16 – 1` or
`65,535` distinct messages. Continuing, because we know all messages must be below
65 KB, this simplifies our parsing as we can use a _fixed_ size buffer and
maintain strong bounds on the total amount of memory required to parse an
incoming wire message.
The final bullet point allows for a degree of _backward_ compatibility because new nodes are able to provide information in the wire messages that older nodes
(which may not understand them) can safely ignore. As we see subsequently, this
feature combined with a very flexible wire message extensibility format also
allows the protocol to achieve _forward_ compatibility as well.
==== Type Encoding
With this high-level background provided, we now start at the most primitive
layer: parsing primitive types. In addition to encoding integers, the Lightning
Protocol also allows for encoding of a vast array of types including variable
length byte slices, elliptic curve public keys, Bitcoin addresses, and
signatures. When we describe the _structure_ of wire messages later in this
chapter, we refer to the high-level type (the abstract type) rather than the
lower-level representation of said type. In this section, we peel back this
abstraction layer to ensure our future wire parser is able to properly
encoding/decode any of the higher-level types.
In <<message_types>>, we map the name of a given message type to the
high-level routine used to encode/decode the type.
[[message_types]]
.High-level message types
[options="header"]
|===
| High-level type | Framing | Comment
| `node_alias` | A 32-byte fixed-length byte slice | When decoding, reject if contents are not a valid UTF-8 string
| `channel_id` | A 32-byte fixed-length byte slice that maps an outpoint to a 32-byte value | Given an outpoint, one can convert it to a `channel_id` by taking the TxID of the outpoint and XORing it with the index (interpreted as the lower 2 bytes)
| `short_chan_id` | An unsigned 64-bit integer (`uint64`) | Composed of the block height (24 bits), transaction index (24 bits), and output index (16 bits) packed into 8 bytes
| `milli_satoshi` | An unsigned 64-bit integer (`uint64`) | Represents 1000th of a satoshi
| `satoshi` | An unsigned 64-bit integer (`uint64`) | The base unit of bitcoin
| `pubkey` | An secp256k1 public key encoded in _compressed_ format, occupying 33 bytes | Occupies a fixed 33-byte length on the wire
| `sig` | An ECDSA signature of the secp256k1 elliptic curve. | Encoded as a _fixed_ 64-byte byte slice, packed as `R \|\| S`
| `uint8` | An 8-bit integer |
| `uint16` | A 16-bit integer |
| `uint64` | A 64-bit integer |
| `[]byte` | A variable length byte slice | Prefixed with a 16-bit integer denoting the length of the bytes
| `color_rgb` | RGB color encoding | Encoded as a series if 8-bit integers
| `net_addr` | The encoding of a network address | Encoded with a 1-byte prefix that denotes the type of address, followed by the address body
|===
In the next section, we describe the structure of each of the wire messages
including the prefix type of the message along with the contents of its message
body.
[[tlv_message_extensions]]
=== Type Length Value (TLV) Message Extensions
Earlier in this chapter we mentioned that messages can be up to 65 KB in size,
and if while parsing a message, extra bytes are left over, then those bytes
are to be ignored. At an initial glance, this requirement may appear to be
somewhat arbitrary; however, this requirement allows for decoupled desynchronized evolution of the Lightning
Protocol itself. We discuss this more toward the end of the chapter. But first, we turn our attention to exactly what those "extra bytes" at
the end of a message can be used for.
==== The Protocol Buffers Message Format
The Protocol Buffers (Protobuf) message serialization format started out as an
internal format used at Google and has blossomed into one of the most popular
message serialization formats used by developers globally. The Protobuf format
describes how a message (usually some sort of data structure related to an API)
is encoded on the wire and decoded on the other end. Several "Protobuf
compilers" exists in dozens of languages which act as a bridge that allows any
language to encode a Protobuf that will be able to decode by a compliant decode
in another language. Such cross-language data structure compatibility allows
for a wide range of innovation because it's possible to transmit structure and even
typed data structures across language and abstraction boundaries.
Protobufs are also known for their flexibility with respect to how they
handle changes in the underlying messages structure. As long as the field
numbering schema is adhered to, then it's possible for a newer write of
Protobufs to include information within a Protobuf that may be unknown to any
older readers. When the old reader encounters the new serialized format, if
there are types/fields that it doesn't understand, then it simply _ignores_
them. This allows old clients and new clients to coexist because all clients can
parse some portion of the newer message format.
==== Forward and Backward Compatibility
Protobufs are extremely popular amongst developers because they have built in
support for both forward and backward compatibility. Most developers are
likely familiar with the concept of backward compatibility. In simple terms,
the principle states that any changes to a message format or API should be
done in a manner that doesn't break support for older clients. Within our preceding Protobuf extensibility examples, backward compatibility is achieved by
ensuring that new additions to the Protobuf format don't break the known portions
of older readers. Forward compatibility, on the other hand, is just as important
for desynchronized updates; however, it's less commonly known. For a change to
be forward compatible, clients are to simply ignore any information
they don't understand. The soft fork mechanism of upgrading the Bitcoin
consensus system can be said to be both forward and backward compatible: any
clients that don't update can still use Bitcoin, and if they encounter any
transactions they don't understand, then they simply ignore them as their funds
aren't using those new features.
[[tlv]]
=== Type-Length-Value (TLV) Format
To be able to upgrade messages in a manner that is both forward and backward
compatible, in addition to feature bits (more on that later), the Lightning Network utilizes a custom message serialization format plainly called Type-Length-Value, or TLV for short. The format was inspired by the widely used Protobuf
format and borrows many concepts by significantly simplifying the
implementation as well as the software that interacts with message parsing. A
curious reader might ask "why not just use Protobufs?" In response, the
Lightning developers would respond that we're able to have the best of the
extensibility of Protobufs while also having the benefit of a smaller
implementation and thus smaller attack. As of version 3.15.6, the Protobuf
compiler weighs in at over 656,671 lines of code. In comparison LND's
implementation of the TLV message format weighs in at only 2.3k lines of code
(including tests).
With the necessary background presented, we're now ready to describe the TLV
format in detail. A TLV message extension is said to be a stream of
individual TLV records. A single TLV record has three components: the type of
the record, the length of the record, and finally the opaque value of the
record:
`type`:: An integer representing the name of the record being encoded.
`length`:: The length of the record.
`value`:: The opaque value of the record.
Both the `type` and `length` are encoded using a variable sized integer that's inspired by the variable sized integer (varint) used in Bitcoin's P2P protocol, called `BigSize` for short.
==== BigSize Integer Encoding
In its fullest form, a `BigSize`
integer can represent value up to 64 bits. In contrast to Bitcoin's varint
format, the `BigSize` format instead encodes integers using a big-endian byte
ordering.
The `BigSize` varint has two components: the discriminant and the body. In the
context of the `BigSize` integer, the discriminant communicates to the decoder
the size of the variable sized integer that follows. Remember that the unique thing about
variable sized integers is that they allow a parser to use fewer bytes to encode
smaller integers than larger ones, saving space. Encoding of a `BigSize`
integer follows one of the four following options:
1. If the value is less than `0xfd` (`253`): Then the discriminant isn't really used, and the encoding is simply the integer itself. This allows us to encode very small integers with no additional overhead.
2. If the value is less than or equal to `0xffff` (`65535`):The discriminant is encoded as `0xfd`, which indicates that the value that follows is larger than `0xfd`, but smaller than `0xffff`). The number is then encoded as a 16-bit integer. Including the discriminant, then we can encode a value that is greater than 253, but less than 65,535 using 3 bytes.
3. If the value is less than `0xffffffff` (`4294967295`): The discriminant is encoded as `0xfe`. The body is encoded using 32-bit integer, including the discriminant, then we can encode a value that's less than `4,294,967,295` using 5 bytes.
4. Otherwise, we just encode the value as a full-size 64-bit integer.
==== TLV Encoding Constraints
Within the context of a TLV message, record types below `2^16` are said to be _reserved_ for future use. Types beyond this
range are to be used for "custom" message extensions used by higher-level application protocols.
The `value` of a record depends on the `type`. In other words, it can take any form because parsers will attempt to interpret it depending on the context of the type itself.
==== TLV Canonical Encoding
One issue with the Protobuf format is that encodings of the same message may
output an entirely different set of bytes when encoded by two different
versions of the compiler. Such instances of a noncanonical encoding are not
acceptable within the context of Lightning, as many messages contain a
signature of the message digest. If it's possible for a message to be encoded
in two different ways, then it would be possible to break the authentication of
a signature inadvertently by re-encoding a message using a slightly different
set of bytes on the wire.
To ensure that all encoded messages are canonical, the following
constraints are defined when encoding:
* All records within a TLV stream must be encoded in order of strictly
increasing type.
* All records must minimally encode the `type` and `length` fields. In other words, the smallest `BigSize` representation for an integer must be used at all times.
* Each `type` may only appear once within a given TLV stream.
In addition to these encoding constraints, a series of higher-level
interpretation requirements is also defined based on the _arity_ of a given `type` integer. We dive further into these details toward the end of the
chapter once we describe how the Lightning Protocol is upgraded in practice and
in theory.
[[feature_bits]]
=== Feature Bits and Protocol Extensibility
Because the Lightning Network is a decentralized system, no single entity can enforce a
protocol change or modification upon all the users of the system. This
characteristic is also seen in other decentralized networks such as Bitcoin.
However, unlike Bitcoin, overwhelming consensus _is not_ required to change a
subset of the Lightning Network. Lightning is able to evolve at will without a
strong requirement of coordination because, unlike Bitcoin, there is no global consensus required in the Lightning Network. Due to this fact and the several
upgrade mechanisms embedded in the Lightning Network, only the
participants that wish to use these new Lightning Network features need to
upgrade, and then they are able to interact with each other.
In this section, we explore the various ways that developers and users are
able to design and deploy new features to the Lightning Network. The
designers of the original Lightning Network knew that there were many possible future directions for the network and the underlying protocol. As a result, they made sure to implement several
extensibility mechanisms within the system, which can be used to upgrade it partially or fully in a decoupled, desynchronized, and decentralized
manner.
==== Feature Bits As an Upgrade Discoverability Mechanism
An astute reader may have noticed the various locations that feature bits are
included within the Lightning Protocol. A _feature bit_ is a bitfield that can
be used to advertise understanding or adherence to a possible network protocol
update. Feature bits are commonly assigned in pairs, meaning that each
potential new feature/upgrade always defines two bits within the bitfield.
One bit signals that the advertised feature is _optional_, meaning that the
node knows about the feature and can use it, but doesn't
consider it required for normal operation. The other bit signals that the
feature is instead _required_, meaning that the node will not continue
operation if a prospective peer doesn't understand that feature.
Using these two bits (optional and required), we can construct a simple
compatibility matrix that nodes/users can consult to determine if a peer is compatible with a desired feature, as shown in <<table1302>>.
[[table1302]]
.Feature bit compatibility matrix
[options="header"]
|===
|Bit type|Remote optional|Remote required|Remote unknown
|Local optional|✅|✅|✅
|Local required|✅|✅|❌
|Local unknown|✅|❌|❌
|===
From this simplified compatibility matrix, we can see that as long as the other
party knows about our feature bit, then we can interact with them using the
protocol. If the party doesn't even know about what bit we're referring to
_and_ they require the feature, then we are incompatible with them. Within the
network, optional features are signaled using an _odd bit number_ while
required features are signaled using an _even bit number_. As an example, if a peer signals that they known of a feature that uses bit +15+, then we know that
this is an optional feature, and we can interact with them or respond to
their messages even if we don't know about the feature. If
they instead signaled the feature using bit +16+, then we know this is a
required feature, and we can't interact with them unless our node also
understands that feature.
The Lightning developers have come up with an easy to remember phrase that
encodes this matrix: "it's OK to be odd." This simple rule allows for a
rich set of interactions within the protocol, as a simple bitmask operation
between two feature bit vectors allows peers to determine if certain
interactions are compatible with each other or not. In other words, feature
bits are used as an upgrade discoverability mechanism: they easily allow to
peers to understand if they are compatible or not based on the concepts of
optional, required, and unknown feature bits.
Feature bits are found in the `node_announcement`, `channel_announcement`, and
`init` messages within the protocol. As a result, these three messages can be
used to signal the knowledge and/or understanding of in-flight protocol
updates within the network. The feature bits found in the `node_announcement`
message can allow a peer to determine if their _connections_ are compatible or
not. The feature bits within the `channel_announcement` messages allow a peer
to determine if a given payment type or HTLC can transit through a given peer or
not. The feature bits within the `init` message allow peers to understand if
they can maintain a connection, and also which features are negotiated for the
lifetime of a given connection.
==== TLV for Forward and Backward Compatibility
As we learned earlier in the chapter, TLV records can be
used to extend messages in a forward and backward compatible manner.
Over time, these records have been used to extend existing messages without
breaking the protocol by utilizing the "undefined" area within a message beyond
that set of known bytes.
As an example, the original Lightning Protocol didn't have a concept of the
"largest amount HTLC" that could traverse through a channel as dictated by a routing
policy. Later on, the `max_htlc` field was added to the `channel_update`
message to phase in this concept over time. Peers that receive a
`channel_update` that sets such a field but don't even know the upgrade existed
are unaffected by the change, but have their HTLCs rejected if they are
beyond the limit. Newer peers, on the other hand, are able to parse, verify,
and utilize the new field.
Those familiar with the concept of soft forks in Bitcoin may now see some
similarities between the two mechanisms. Unlike Bitcoin consensus-level
soft forks, upgrades to the Lightning Network don't require overwhelming
consensus to be adopted. Instead, at minimum, only two peers within the
network need to understand a new upgrade to start using it. Commonly these two peers may be the recipient and sender of a
payment, or may be the channel partners of a new payment channel.
==== A Taxonomy of Upgrade Mechanisms
Rather than there being a single widely utilized upgrade mechanism within the
network (such as soft forks for Bitcoin), there exist several possible upgrade mechanisms within the Lightning Network. In this
section, we enumerate these upgrade mechanisms and
provide a real-world example of their use in the past.
===== Internal network upgrades
We start with the upgrade type that requires the most protocol-level
coordination: internal network upgrades. An internal network upgrade is
characterized by one that requires _every single node_ within a prospective payment path to understand the new feature. Such an upgrade is similar to any
upgrade within the internet that requires hardware-level upgrades within
the core-relay portion of the upgrade. In the context of the Lightning Network, however, we deal
with pure software, so such upgrades are easier to deploy, yet they still
require much more coordination than any other upgrade mechanism in the
network.
One example of such an upgrade within the network was the introduction of a TLV
encoding for the routing information encoded within the onion
packets. The prior format used a hardcoded fixed-length message
format to communicate information such as the next hop.
Because this format was fixed, it meant that new protocol-level upgrades weren't possible. The move to the more flexible TLV
format meant that after this upgrade, any sort of feature that
modified the type of information communicated at each hop could be rolled out at will.
It's worth mentioning that the TLV onion upgrade was a sort of "soft" internal
network upgrade, in that if a payment wasn't using any new feature beyond
that new routing information encoding, then a payment could be transmitted
using a mixed set of nodes.
===== End-to-end upgrades
To contrast the internal network upgrade, in this section we describe the
_end-to-end_ network upgrade. This upgrade mechanism differs from the internal
network upgrade in that it only requires the "ends" of the payment, the sender
and recipient, to upgrade.
This type of upgrade allows
for a wide array of unrestricted innovation within the network. Because of the
onion encrypted nature of payments within the network, those forwarding HTLCs
within the center of the network may not even know that new features are being
utilized.
One example of an end-to-end upgrade within the network was the rollout of multipart payments (MPP). MPP is a protocol-level feature that enables a
single payment to be split into multiple parts or paths, to be assembled at the
recipient for settlement. The rollout of MPP was coupled with a new
`node_announcement` level feature bit that indicates that the recipient knows
how to handle partial payments. Assuming a sender and recipient know about each
other (possibly via a BOLT #11 invoice), then they're able to use the new
feature without any further negotiation.
Another example of an end-to-end upgrade are the various types of
_spontaneous_ payments deployed within the network. One early type of
spontaneous payments called _keysend_ worked by simply placing the preimage of a payment within the encrypted onion. Upon receipt, the destination would decrypt the
preimage, then use that to settle the payment. Because the entire packet is end-to-end encrypted, this payment type was safe, since none of the intermediate nodes
are able to fully unwrap the onion to uncover the payment preimage.
==== Channel Construction Level Updates
The final broad category of updates are those that happen at
the channel construction level, but which don't modify the structure of the HTLC used widely within the network. When we say channel construction, we mean
how the channel is funded or created. As an example, the eltoo channel type
can be rolled out within the network using a new `node_announcement` level
feature bit as well as a `channel_announcement` level feature bit. Only the two
peers on the sides of the channels need to understand and advertise these new
features. This channel pair can then be used to forward any payment type
granted the channel supports it.
Another is the _anchor outputs_ channel format which allows the commitment fee to be
bumped via Bitcoin's Child-Pays-For-Parent (CPFP) fee management mechanism.
=== Conclusion
Lightning's wire protocol is incredibly flexible and allows for rapid innovation and interoperability without strict consensus. It is one of the reasons that the Lightning Network is experiencing much faster development and is attractive to many developers, who might otherwise find Bitcoin's development style too conservative and slow.
[[wire_protocol]]
== Wire Protocol: Framing and Extensibility
((("wire protocol", id="ix_13_wire_protocol-asciidoc0", range="startofrange")))In this chapter, we dive into the wire protocol of the Lightning Network
and also cover all the various extensibility levers that have been built into
the protocol. By the end of this chapter, an ambitious reader should be able to
write their very own wire protocol parser for the Lightning Network. In addition
to being able to write a custom wire protocol parser, a reader of this chapter
will gain a deep understanding of the various upgrade mechanisms
that have been built into the protocol.
=== Messaging Layer in the Lightning Protocol Suite
((("Lightning Network Protocol","messaging layer")))((("wire protocol","messaging layer in the Lightning Protocol Suite")))The messaging layer, which is detailed in this chapter, consists of "Framing and message format," "Type Length Value (TLV)" encoding, and "Feature bits." These components are highlighted by a double outline in the protocol suite, shown in <<LN_protocol_wire_message_highlight>>.
[[LN_protocol_wire_message_highlight]]
.The Lightning Network protocol suite
image::images/mtln_1301.png["The Lightning Network Protocol Suite"]
=== Wire Framing
((("wire framing", id="ix_13_wire_protocol-asciidoc1", range="startofrange")))((("wire protocol","wire framing", id="ix_13_wire_protocol-asciidoc2", range="startofrange")))We begin by describing the high-level structure of the wire _framing_
within the protocol. When we say framing, we mean the way that the bytes are
packed on the wire to _encode_ a particular protocol message. Without knowledge
of the framing system used in the protocol, a string of bytes on the wire would
resemble a series of random bytes because no structure has been imposed. By applying
proper framing to decode these bytes on the wire, we'll be able to extract
structure and finally parse this structure into protocol messages within our
higher-level language.
It's important to note that the Lightning Network is an _end-to-end
encrypted_ protocol, and the wire framing is itself encapsulated within an
_encrypted_ message transport layer. As we see in <<encrypted_message_transport>>, the Lightning
Network uses a custom variant of the Noise Protocol to handle
transport encryption. Within this chapter, whenever we give an example of wire
framing, we assume the encryption layer has already been stripped away (when
decoding), or that we haven't yet encrypted the set of bytes before we send
them on the wire (encoding).
==== High-Level Wire Framing
((("wire framing","high-level schema")))With that said, we're ready to describe the high-level schema used to
encode messages on the wire:
* Messages on the wire begin with a _2-byte_ type field, followed by a
message payload.
* The message payload itself can be up to 65 KB in size.
* All integers are encoded in big-endian (network order).
* Any bytes that follow after a defined message can be safely ignored.
Yep, that's it. As the protocol relies on an _encapsulating_ transport protocol
encryption layer, we don't need an explicit length for each message type. This
is due to the fact that transport encryption works at the _message_ level, so
by the time we're ready to decode the next message, we already know the total
number of bytes of the message itself. Using 2 bytes for the message type
(encoded in big-endian) means that the protocol can have up to `2^16 – 1` or
`65,535` distinct messages. Continuing, because we know all messages must be below
65 KB, this simplifies our parsing as we can use a _fixed_ size buffer and
maintain strong bounds on the total amount of memory required to parse an
incoming wire message.
The final bullet point allows for a degree of _backward_ compatibility because new nodes are able to provide information in the wire messages that older nodes
(which may not understand them) can safely ignore. As we see subsequently, this
feature combined with a very flexible wire message extensibility format also
allows the protocol to achieve _forward_ compatibility as well.
==== Type Encoding
((("wire framing","type encoding")))With this high-level background provided, we now start at the most primitive
layer: parsing primitive types. In addition to encoding integers, the Lightning
Protocol also allows for encoding of a vast array of types including variable
length byte slices, elliptic curve public keys, Bitcoin addresses, and
signatures. When we describe the _structure_ of wire messages later in this
chapter, we refer to the high-level type (the abstract type) rather than the
lower-level representation of said type. In this section, we peel back this
abstraction layer to ensure our future wire parser is able to properly
encoding/decode any of the higher-level types.
In <<message_types>>, we map the name of a given message type to the
high-level routine used to encode/decode the type.
[[message_types]]
.High-level message types
[options="header"]
|===
| High-level type | Framing | Comment
| `node_alias` | A 32-byte fixed-length byte slice | When decoding, reject if contents are not a valid UTF-8 string
| `channel_id` | A 32-byte fixed-length byte slice that maps an outpoint to a 32-byte value | Given an outpoint, one can convert it to a `channel_id` by taking the TxID of the outpoint and XORing it with the index (interpreted as the lower 2 bytes)
| `short_chan_id` | An unsigned 64-bit integer (`uint64`) | Composed of the block height (24 bits), transaction index (24 bits), and output index (16 bits) packed into 8 bytes
| `milli_satoshi` | An unsigned 64-bit integer (`uint64`) | Represents 1000th of a satoshi
| `satoshi` | An unsigned 64-bit integer (`uint64`) | The base unit of bitcoin
| `pubkey` | An secp256k1 public key encoded in _compressed_ format, occupying 33 bytes | Occupies a fixed 33-byte length on the wire
| `sig` | An ECDSA signature of the secp256k1 elliptic curve. | Encoded as a _fixed_ 64-byte byte slice, packed as `R \|\| S`
| `uint8` | An 8-bit integer |
| `uint16` | A 16-bit integer |
| `uint64` | A 64-bit integer |
| `[]byte` | A variable length byte slice | Prefixed with a 16-bit integer denoting the length of the bytes
| `color_rgb` | RGB color encoding | Encoded as a series if 8-bit integers
| `net_addr` | The encoding of a network address | Encoded with a 1-byte prefix that denotes the type of address, followed by the address body
|===
In the next section, we describe the structure of each of the wire messages
including the prefix type of the message along with the contents of its message
body.(((range="endofrange", startref="ix_13_wire_protocol-asciidoc2")))(((range="endofrange", startref="ix_13_wire_protocol-asciidoc1")))
[[tlv_message_extensions]]
=== Type Length Value (TLV) Message Extensions
((("Type-Length-Value (TLV) message extensions","message extensions in wire protocol")))((("wire protocol","TLV message extensions")))Earlier in this chapter we mentioned that messages can be up to 65 KB in size,
and if while parsing a message, extra bytes are left over, then those bytes
are to be ignored. At an initial glance, this requirement may appear to be
somewhat arbitrary; however, this requirement allows for decoupled desynchronized evolution of the Lightning
Protocol itself. We discuss this more toward the end of the chapter. But first, we turn our attention to exactly what those "extra bytes" at
the end of a message can be used for.
==== The Protocol Buffers Message Format
((("Protocol Buffers (Protobuf) message serialization format")))((("Type-Length-Value (TLV) message extensions","Protocol Buffers message format")))The Protocol Buffers (Protobuf) message serialization format started out as an
internal format used at Google and has blossomed into one of the most popular
message serialization formats used by developers globally. The Protobuf format
describes how a message (usually some sort of data structure related to an API)
is encoded on the wire and decoded on the other end. Several "Protobuf
compilers" exists in dozens of languages which act as a bridge that allows any
language to encode a Protobuf that will be able to decode by a compliant decode
in another language. Such cross-language data structure compatibility allows
for a wide range of innovation because it's possible to transmit structure and even
typed data structures across language and abstraction boundaries.
Protobufs are also known for their flexibility with respect to how they
handle changes in the underlying messages structure. As long as the field
numbering schema is adhered to, then it's possible for a newer write of
Protobufs to include information within a Protobuf that may be unknown to any
older readers. When the old reader encounters the new serialized format, if
there are types/fields that it doesn't understand, then it simply _ignores_
them. This allows old clients and new clients to coexist because all clients can
parse some portion of the newer message format.
==== Forward and Backward Compatibility
((("Protocol Buffers (Protobuf) message serialization format")))((("Type-Length-Value (TLV) message extensions","forward/backward compatibility")))Protobufs are extremely popular amongst developers because they have built in
support for both forward and backward compatibility. Most developers are
likely familiar with the concept of backward compatibility. In simple terms,
the principle states that any changes to a message format or API should be
done in a manner that doesn't break support for older clients. Within our preceding Protobuf extensibility examples, backward compatibility is achieved by
ensuring that new additions to the Protobuf format don't break the known portions
of older readers. Forward compatibility, on the other hand, is just as important
for desynchronized updates; however, it's less commonly known. For a change to
be forward compatible, clients are to simply ignore any information
they don't understand. The soft fork mechanism of upgrading the Bitcoin
consensus system can be said to be both forward and backward compatible: any
clients that don't update can still use Bitcoin, and if they encounter any
transactions they don't understand, then they simply ignore them as their funds
aren't using those new features.
[[tlv]]
=== Type-Length-Value (TLV) Format
((("Type-Length-Value (TLV) format", id="ix_13_wire_protocol-asciidoc3", range="startofrange")))((("Type-Length-Value (TLV) format","wire protocol and", id="ix_13_wire_protocol-asciidoc4", range="startofrange")))((("wire protocol","TLV format", id="ix_13_wire_protocol-asciidoc5", range="startofrange")))To be able to upgrade messages in a manner that is both forward and backward
compatible, in addition to feature bits (more on that later), the Lightning Network utilizes a custom message serialization format plainly called Type-Length-Value, or TLV for short. The format was inspired by the widely used Protobuf
format and borrows many concepts by significantly simplifying the
implementation as well as the software that interacts with message parsing. A
curious reader might ask "why not just use Protobufs?" In response, the
Lightning developers would respond that we're able to have the best of the
extensibility of Protobufs while also having the benefit of a smaller
implementation and thus smaller attack. As of version 3.15.6, the Protobuf
compiler weighs in at over 656,671 lines of code. In comparison LND's
implementation of the TLV message format weighs in at only 2.3k lines of code
(including tests).
With the necessary background presented, we're now ready to describe the TLV
format in detail. A TLV message extension is said to be a stream of
individual TLV records. A single TLV record has three components: the type of
the record, the length of the record, and finally the opaque value of the
record:
`type`:: An integer representing the name of the record being encoded.
`length`:: The length of the record.
`value`:: The opaque value of the record.
Both the `type` and `length` are encoded using a variable sized integer that's inspired by the variable sized integer (varint) used in Bitcoin's P2P protocol, called `BigSize` for short.
==== BigSize Integer Encoding
((("BigSize integer encoding")))((("Type-Length-Value (TLV) format","BigSize integer encoding")))In its fullest form, a `BigSize`
integer can represent value up to 64 bits. In contrast to Bitcoin's varint
format, the `BigSize` format instead encodes integers using a big-endian byte
ordering.
The `BigSize` varint has two components: the discriminant and the body. In the
context of the `BigSize` integer, the discriminant communicates to the decoder
the size of the variable sized integer that follows. Remember that the unique thing about
variable sized integers is that they allow a parser to use fewer bytes to encode
smaller integers than larger ones, saving space. Encoding of a `BigSize`
integer follows one of the four following options:
1. If the value is less than `0xfd` (`253`): Then the discriminant isn't really used, and the encoding is simply the integer itself. This allows us to encode very small integers with no additional overhead.
2. If the value is less than or equal to `0xffff` (`65535`):The discriminant is encoded as `0xfd`, which indicates that the value that follows is larger than `0xfd`, but smaller than `0xffff`). The number is then encoded as a 16-bit integer. Including the discriminant, then we can encode a value that is greater than 253, but less than 65,535 using 3 bytes.
3. If the value is less than `0xffffffff` (`4294967295`): The discriminant is encoded as `0xfe`. The body is encoded using 32-bit integer, including the discriminant, then we can encode a value that's less than `4,294,967,295` using 5 bytes.
4. Otherwise, we just encode the value as a full-size 64-bit integer.
==== TLV Encoding Constraints
((("Type-Length-Value (TLV) format","encoding constraints")))Within the context of a TLV message, record types below `2^16` are said to be _reserved_ for future use. Types beyond this
range are to be used for "custom" message extensions used by higher-level application protocols.
The `value` of a record depends on the `type`. In other words, it can take any form because parsers will attempt to interpret it depending on the context of the type itself.
==== TLV Canonical Encoding
One issue with the Protobuf format is that encodings of the same message may
output an entirely different set of bytes when encoded by two different
versions of the compiler. Such instances of a noncanonical encoding are not
acceptable within the context of Lightning, as many messages contain a
signature of the message digest. If it's possible for a message to be encoded
in two different ways, then it would be possible to break the authentication of
a signature inadvertently by re-encoding a message using a slightly different
set of bytes on the wire.
To ensure that all encoded messages are canonical, the following
constraints are defined when encoding:
* All records within a TLV stream must be encoded in order of strictly
increasing type.
* All records must minimally encode the `type` and `length` fields. In other words, the smallest `BigSize` representation for an integer must be used at all times.
* Each `type` may only appear once within a given TLV stream.
In addition to these encoding constraints, a series of higher-level
interpretation requirements is also defined based on the _arity_ of a given `type` integer. We dive further into these details toward the end of the
chapter once we describe how the Lightning Protocol is upgraded in practice and
in theory.(((range="endofrange", startref="ix_13_wire_protocol-asciidoc5")))(((range="endofrange", startref="ix_13_wire_protocol-asciidoc4")))(((range="endofrange", startref="ix_13_wire_protocol-asciidoc3")))
[[feature_bits]]
=== Feature Bits and Protocol Extensibility
((("feature bits", id="ix_13_wire_protocol-asciidoc6", range="startofrange")))((("wire protocol","feature bits/protocol extensibility", id="ix_13_wire_protocol-asciidoc7", range="startofrange")))Because the Lightning Network is a decentralized system, no single entity can enforce a
protocol change or modification upon all the users of the system. This
characteristic is also seen in other decentralized networks such as Bitcoin.
However, unlike Bitcoin, overwhelming consensus _is not_ required to change a
subset of the Lightning Network. Lightning is able to evolve at will without a
strong requirement of coordination because, unlike Bitcoin, there is no global consensus required in the Lightning Network. Due to this fact and the several
upgrade mechanisms embedded in the Lightning Network, only the
participants that wish to use these new Lightning Network features need to
upgrade, and then they are able to interact with each other.
In this section, we explore the various ways that developers and users are
able to design and deploy new features to the Lightning Network. The
designers of the original Lightning Network knew that there were many possible future directions for the network and the underlying protocol. As a result, they made sure to implement several
extensibility mechanisms within the system, which can be used to upgrade it partially or fully in a decoupled, desynchronized, and decentralized
manner.
==== Feature Bits As an Upgrade Discoverability Mechanism
((("feature bits","upgrade discoverability mechanism")))An astute reader may have noticed the various locations that feature bits are
included within the Lightning Protocol. A _feature bit_ is a bitfield that can
be used to advertise understanding or adherence to a possible network protocol
update. Feature bits are commonly assigned in pairs, meaning that each
potential new feature/upgrade always defines two bits within the bitfield.
One bit signals that the advertised feature is _optional_, meaning that the
node knows about the feature and can use it, but doesn't
consider it required for normal operation. The other bit signals that the
feature is instead _required_, meaning that the node will not continue
operation if a prospective peer doesn't understand that feature.
Using these two bits (optional and required), we can construct a simple
compatibility matrix that nodes/users can consult to determine if a peer is compatible with a desired feature, as shown in <<table1302>>.
[[table1302]]
.Feature bit compatibility matrix
[options="header"]
|===
|Bit type|Remote optional|Remote required|Remote unknown
|Local optional|✅|✅|✅
|Local required|✅|✅|❌
|Local unknown|✅|❌|❌
|===
From this simplified compatibility matrix, we can see that as long as the other
party knows about our feature bit, then we can interact with them using the
protocol. If the party doesn't even know about what bit we're referring to
_and_ they require the feature, then we are incompatible with them. Within the
network, optional features are signaled using an _odd bit number_ while
required features are signaled using an _even bit number_. As an example, if a peer signals that they known of a feature that uses bit +15+, then we know that
this is an optional feature, and we can interact with them or respond to
their messages even if we don't know about the feature. If
they instead signaled the feature using bit +16+, then we know this is a
required feature, and we can't interact with them unless our node also
understands that feature.
The Lightning developers have come up with an easy to remember phrase that
encodes this matrix: "it's OK to be odd." This simple rule allows for a
rich set of interactions within the protocol, as a simple bitmask operation
between two feature bit vectors allows peers to determine if certain
interactions are compatible with each other or not. In other words, feature
bits are used as an upgrade discoverability mechanism: they easily allow to
peers to understand if they are compatible or not based on the concepts of
optional, required, and unknown feature bits.
Feature bits are found in the `node_announcement`, `channel_announcement`, and
`init` messages within the protocol. As a result, these three messages can be
used to signal the knowledge and/or understanding of in-flight protocol
updates within the network. The feature bits found in the `node_announcement`
message can allow a peer to determine if their _connections_ are compatible or
not. The feature bits within the `channel_announcement` messages allow a peer
to determine if a given payment type or HTLC can transit through a given peer or
not. The feature bits within the `init` message allow peers to understand if
they can maintain a connection, and also which features are negotiated for the
lifetime of a given connection.
==== TLV for Forward and Backward Compatibility
((("feature bits","TLV for forward/backward compatibility")))((("Type-Length-Value (TLV) format","forward/backward compatibility and")))((("wire protocol","TLV for forward/backward compatibility")))As we learned earlier in the chapter, TLV records can be
used to extend messages in a forward and backward compatible manner.
Over time, these records have been used to extend existing messages without
breaking the protocol by utilizing the "undefined" area within a message beyond
that set of known bytes.
As an example, the original Lightning Protocol didn't have a concept of the
"largest amount HTLC" that could traverse through a channel as dictated by a routing
policy. Later on, the `max_htlc` field was added to the `channel_update`
message to phase in this concept over time. Peers that receive a
`channel_update` that sets such a field but don't even know the upgrade existed
are unaffected by the change, but have their HTLCs rejected if they are
beyond the limit. Newer peers, on the other hand, are able to parse, verify,
and utilize the new field.
Those familiar with the concept of soft forks in Bitcoin may now see some
similarities between the two mechanisms. Unlike Bitcoin consensus-level
soft forks, upgrades to the Lightning Network don't require overwhelming
consensus to be adopted. Instead, at minimum, only two peers within the
network need to understand a new upgrade to start using it. Commonly these two peers may be the recipient and sender of a
payment, or may be the channel partners of a new payment channel.
==== A Taxonomy of Upgrade Mechanisms
((("Lightning Network (generally)","taxonomy of upgrade mechanisms", id="ix_13_wire_protocol-asciidoc8", range="startofrange")))((("upgrades","taxonomy of upgrade mechanisms", id="ix_13_wire_protocol-asciidoc9", range="startofrange")))((("wire protocol","taxonomy of upgrade mechanisms", id="ix_13_wire_protocol-asciidoc10", range="startofrange")))Rather than there being a single widely utilized upgrade mechanism within the
network (such as soft forks for Bitcoin), there exist several possible upgrade mechanisms within the Lightning Network. In this
section, we enumerate these upgrade mechanisms and
provide a real-world example of their use in the past.
===== Internal network upgrades
((("upgrades","internal network")))We start with the upgrade type that requires the most protocol-level
coordination: internal network upgrades. An internal network upgrade is
characterized by one that requires _every single node_ within a prospective payment path to understand the new feature. Such an upgrade is similar to any
upgrade within the internet that requires hardware-level upgrades within
the core-relay portion of the upgrade. In the context of the Lightning Network, however, we deal
with pure software, so such upgrades are easier to deploy, yet they still
require much more coordination than any other upgrade mechanism in the
network.
One example of such an upgrade within the network was the introduction of a TLV
encoding for the routing information encoded within the onion
packets. The prior format used a hardcoded fixed-length message
format to communicate information such as the next hop.
Because this format was fixed, it meant that new protocol-level upgrades weren't possible. The move to the more flexible TLV
format meant that after this upgrade, any sort of feature that
modified the type of information communicated at each hop could be rolled out at will.
It's worth mentioning that the TLV onion upgrade was a sort of "soft" internal
network upgrade, in that if a payment wasn't using any new feature beyond
that new routing information encoding, then a payment could be transmitted
using a mixed set of nodes.
===== End-to-end upgrades
((("upgrades","end-to-end")))To contrast the internal network upgrade, in this section we describe the
_end-to-end_ network upgrade. This upgrade mechanism differs from the internal
network upgrade in that it only requires the "ends" of the payment, the sender
and recipient, to upgrade.
This type of upgrade allows
for a wide array of unrestricted innovation within the network. Because of the
onion encrypted nature of payments within the network, those forwarding HTLCs
within the center of the network may not even know that new features are being
utilized.
One example of an end-to-end upgrade within the network was the rollout of multipart payments (MPP). MPP is a protocol-level feature that enables a
single payment to be split into multiple parts or paths, to be assembled at the
recipient for settlement. The rollout of MPP was coupled with a new
`node_announcement` level feature bit that indicates that the recipient knows
how to handle partial payments. Assuming a sender and recipient know about each
other (possibly via a BOLT #11 invoice), then they're able to use the new
feature without any further negotiation.
Another example of an end-to-end upgrade are the various types of
_spontaneous_ payments deployed within the network. One early type of
spontaneous payments called _keysend_ worked by simply placing the preimage of a payment within the encrypted onion. Upon receipt, the destination would decrypt the
preimage, then use that to settle the payment. Because the entire packet is end-to-end encrypted, this payment type was safe, since none of the intermediate nodes
are able to fully unwrap the onion to uncover the payment preimage.
==== Channel Construction Level Updates
((("wire protocol","channel construction level updates")))The final broad category of updates are those that happen at
the channel construction level, but which don't modify the structure of the HTLC used widely within the network. When we say channel construction, we mean
how the channel is funded or created. As an example, the eltoo channel type
can be rolled out within the network using a new `node_announcement` level
feature bit as well as a `channel_announcement` level feature bit. Only the two
peers on the sides of the channels need to understand and advertise these new
features. This channel pair can then be used to forward any payment type
granted the channel supports it.
Another is the _anchor outputs_ channel format which allows the commitment fee to be
bumped via Bitcoin's Child-Pays-For-Parent (CPFP) fee management mechanism(((range="endofrange", startref="ix_13_wire_protocol-asciidoc10")))(((range="endofrange", startref="ix_13_wire_protocol-asciidoc9")))(((range="endofrange", startref="ix_13_wire_protocol-asciidoc8"))).(((range="endofrange", startref="ix_13_wire_protocol-asciidoc7")))(((range="endofrange", startref="ix_13_wire_protocol-asciidoc6")))
=== Conclusion
Lightning's wire protocol is incredibly flexible and allows for rapid innovation and interoperability without strict consensus. It is one of the reasons that the Lightning Network is experiencing much faster development and is attractive to many developers, who might otherwise find Bitcoin's development style too conservative and slow.(((range="endofrange", startref="ix_13_wire_protocol-asciidoc0")))

1228
14_encrypted_transport.asciidoc
View File

File diff suppressed because it is too large Load Diff

436
15_payment_requests.asciidoc
Unescape Escape View File

@ -1,218 +1,218 @@
[[invoices]]
== Lightning Payment Requests
In this chapter we will look at _Lightning payment requests_ or as they are more commonly known _Lightning invoices_.
=== Invoices in the Lightning Protocol Suite
_Payment requests_, aka _invoices_, are part of the payment layer and are show in the upper left of <<LN_payment_request_highlight>>.
[[LN_payment_request_highlight]]
.The Lightning Network protocol suite
image::images/mtln_1501.png["The Lightning Network Protocol Suite"]
=== Introduction
As we've learned throughout the book, minimally two pieces of data are required
to complete a Lightning payment: a payment hash and a destination. As
`SHA-256` is used in the Lightning Network to implement HTLCs, this information
requires 32 bytes to communicate. Destinations, on the other hand, are
simply the `secp256k1` public key of the node that wishes to receive a payment.
The purpose of a payment request in the context of the Lightning Network is to
communicate these two pieces of information from sender to receiver. The QR-code–friendly format for communicating the information required
to complete a payment from receiver to sender is described in https://github.com/lightningnetwork/lightning-rfc/blob/master/11-payment-encoding.md[BOLT #11: Invoice Protocol for Lightning Payments]. In practice, more than just the
payment hash and destination are communicated in a payment request to
make the encoding more fully featured.
=== Lightning Payment Requests versus Bitcoin Addresses
A commonly asked question when people first encounter a Lightning Payment
request is: why can't a normal static address format be used instead?
To answer this question, you must first internalize how Lightning
differs from base layer Bitcoin as a payment method. Compared to a Bitcoin
address which may be used to make a potentially unbounded number of payments
(though reusing a Bitcoin address may degrade one's privacy), a Lightning
payment request should only ever be used _once_. This is due to the fact that
sending a payment to a Bitcoin address essentially uses a public key
cryptosystem to "encode" the payment in a manner that only the true "owner" of
that Bitcoin address can redeem it.
In contrast, to complete a Lightning payment, the recipient must
reveal a "secret" to the entire payment route including the sender. This can be
interpreted as usage of a kind of domain-specific symmetric cryptography because
the payment preimage is for practical purposes a nonce (number only used
once). If the sender attempts to make another payment using that identical
payment hash, then they risk losing funds because the payment may not actually be
delivered to the destination. It's safe to assume that after a preimage has
been revealed, all nodes in the path will keep it around forever, then rather
than forward the HTLC to collect a routing fee if the payment is
completed, they can simply settle the payment at that instance and gain the
entire payment amount in return. As a result, it's unsafe to ever use a payment
request more than once.
New variants of the original Lightning payment request exist that allow the sender to reuse them as many times as they want. These variants flip the normal payment flow as the sender transmits a preimage within the encrypted onion payload to the receiver, who is the only
one that is able to decrypt it and settle the payment. Alternatively, assuming
a mechanism that allows a sender to typically request a new payment request
from the receiver, then an interactive protocol can be used to allow a
degree of payment request reuse.
=== BOLT #11: Lightning Payment Request Serialization and Interpretation
In this section, we'll describe the mechanism used to encode the set of
information required to complete a payment on the Lightning Network. As
mentioned earlier, the payment hash and destination is the minimum amount of
information required to complete a payment. However, in practice, more
information such as time-lock information, payment request expiration, and
possibly an on-chain fallback address are also communicated. The full specification document is https://github.com/lightningnetwork/lightning-rfc/blob/master/11-payment-encoding.md[BOLT #11: Invoice Protocol for Lightning Payments].
==== Payment Request Encoding in Practice
First, let's examine what a real payment request looks like in practice. The
following is a valid payment request that could have been used to complete a
payment on the mainnet Lightning Network at the time it was created:
```
lnbc2500u1pvjluezpp5qqqsyqcyq5rqwzqfqqqsyqcyq5rqwzqfqqqsyqcyq5rqwzqfqypqdq5xysxxatsyp3k7enxv4jsxqzpuaztrnwngzn3kdzw5hydlzf03qdgm2hdq27cqv3agm2awhz5se903vruatfhq77w3ls4evs3ch9zw97j25emudupq63nyw24cg27h2rspfj9srp
```
==== The Human-Readable Prefix
Looking at the string, we can tease out a portion that we can parse with our
eyes, while the rest of it just looks like a random set of strings. The part
that is somewhat parsable by a human is referred to as the _human-readable prefix_. It allows a human to quickly extract some relevant information from a
payment request at a glance. In this case, we can see that this payment is for
the mainnet instance of the Lightning Network (`lnbc`), and is requesting 2,500
uBTC (microbitcoin), or `25,000,000` satoshis. The latter potion is referred
to as the data portion and uses an extensible format to encode the
information required to complete a payment.
Each version of instance of the Lightning Network (mainnet, testnet, etc.) has
its own human-readable prefix (see <<table1501>>). This allows client software and also humans to
quickly determine if a payment request can be satisfied by their node or not.
[[table1501]]
.BOLT #11 network prefixes
[options="header"]
|=============================
|Network |BOLT #11 prefix
|mainnet |`lnbc`
|testnet |`lntb`
|simnet/regtest|`lnbcrt`
|=============================
The first portion of the human-readable prefix is a _compact_ expression of the
amount of the payment request. The compact amount is encoded in two parts:
first, an integer is used as the _base_ amount. This is then followed by a
`multiplier` that allows us to specify distinct order of magnitude increases
offset by the base amount. If we return to our initial example, then we can
take the `2500u` portion and decrease it by a factor of 1,000 to instead use
`2500m` or (2,500 `mBTC`). As a rule of thumb, to ascertain the amount
of an invoice at a glance, take the base factor and multiply it by the
`multiplier`.
A full list of the currently defined multipliers is given in <<table1502>>.
[[table1502]]
.BOLT #11 amount multipliers
[options="header"]
|==============================================
|Multiplier|Bitcoin unit|Multiplication factor
|`m`|milli|0.001
|`u`|micro|0.000001
|`n`|nano|0.000000001
|`p`|pico|0.000000000001
|==============================================
==== Bech32 and the Data Segment
If the "unreadable" portion looks familiar, then that's because it uses the
very same encoding scheme as SegWit compatible Bitcoin addresses use today,
namely bech32. Describing the bech32 encoding scheme is outside the scope
of this chapter. In brief, it's a sophisticated way to encode short strings
that has very good error correction as well as detection properties.
The data portion can be separated into three sections:
* The timestamp
* Zero or more tagged key-value pairs
* The signature of the entire invoice
The timestamp is expressed in seconds since the year 1970, or the Unix Epoch. This
timestamp allows the sender to gauge how old the invoice is, and as we'll see
later, allows the receiver to force an invoice to only be valid for a period of
time if they wish.
Similar to the TLV format we learned about in <<tlv>>, the BOLT #11 invoice
format uses a series of extensible key-value pairs to encode information
needed to satisfy a payment. Because key-value pairs are used, it's easy to add
new values in the future if a new payment type or additional
requirement/functionality is introduced.
Finally, a signature is included that covers the entire invoice signed by the
destination of the payment. This signature allows the sender to verify that the
payment request was indeed created by the destination of the payment. Unlike
Bitcoin payment requests which aren't signed, this allows us to ensure that a
particular entity signed the payment request. The signature itself is encoded
using a recovery ID, which allows a more compact signature to be used that
allows public key extraction. When verifying the signature, the recovery ID
extracts the public key, then verifies that against the public key included in
the invoice.
===== Tagged invoice fields
The tagged invoice fields are encoded in the main body of the invoice. These
fields represent different key-value pairs that express either additional
information that may help complete the payment or information which is
_required_ to complete the payment. Because a slight variant of bech32 is
utilized, each of these fields are actually in the "base 5" domain.
A given tag field is comprised of three components:
* The `type` of the field (5 bits)
* The `length` of the data of the field (10 bits
* The `data` itself, which is `length * 5 bytes` in size
A full list of all the currently defined tagged fields is given in <<table1503>>.
[[table1503]]
.BOLT #11 tagged invoice fields
[options="header"]
|===
|Field tag|Data length|Usage
|`p`|`52`|The `SHA-256` payment hash.
|`s`|`52`|A `256-bit` secret that increases the end-to-end privacy of a payment by mitigating probing by intermediate nodes.
|`d`|Variable|The description, a short UTF-8 string of the purpose of the payment.
|`n`|`53`|The public key of the destination node.
|`h`|`52`|A hash that represents a description of the payment itself. This can be used to commit to a description that's over 639 bytes in length.
|`x`|Variable|The expiry time in seconds of the payment. The default is 1 hour (3,600) if not specified.
|`c`|Variable|The `min_cltv_expiry` to use for the final hop in the route. The default is 9 if not specified.
|`f`|Variable|A fallback on-chain address to be used to complete the payment if the payment cannot be completed over the Lightning Network.
|`r`|Variable|One or more entries that allow a receiver to give the sender additional ephemeral edges to complete the payment.
|`9`|Variable|A set of 5-bit values that contain the feature bits that are required in order to complete the payment.
|===
The elements contained in the field `r` are commonly referred to as _routing hints_. They allow the receiver to communicate an extra set of edges that may
help the sender complete their payment. The hints are usually used when the
receiver has some/all private channels, and they wish to guide the sender into
this "unmapped" portion of the channel graph. A routing hint encodes
effectively the same information that a normal `channel_update` message does.
The update is itself packed into a single value with the following fields:
* The `pubkey` of the outgoing node in the edge (264 bits)
* The `short_channel_id` of the "virtual" edge (64 bits)
* The base fee (`fee_base_msat`) of the edge (32 bits)
* The proportional fee (`fee_proportional_millionths`) (32 bits)
* The CLTV expiry delta (`cltv_expiry_delta`) (16 bits)
The final portion of the data segment is the set of feature bits that
communicate to the sender the functionality needed to complete a
payment. As an example, if a new payment type is added in the future that isn't
backward compatible with the original payment type, then the receiver can set
a _required_ feature bit to communicate that the payer needs to
understand that feature to complete the payment.
=== Conclusion
As we have seen, invoices are a lot more than just a request for an amount. They contain critical information about _how_ to make the payment, such as routing hints, the destination node's public key, ephemeral keys to increase security, and much more.
[[invoices]]
== Lightning Payment Requests
((("Lightning invoices", id="ix_15_payment_requests-asciidoc0", range="startofrange")))In this chapter we will look at _Lightning payment requests_ or as they are more commonly known _Lightning invoices_.
=== Invoices in the Lightning Protocol Suite
((("Lightning invoices","Lightning Protocol suite and")))((("Lightning Network Protocol","Lightning invoices in")))_Payment requests_, aka _invoices_, are part of the payment layer and are show in the upper left of <<LN_payment_request_highlight>>.
[[LN_payment_request_highlight]]
.The Lightning Network protocol suite
image::images/mtln_1501.png["The Lightning Network Protocol Suite"]
=== Introduction
As we've learned throughout the book, minimally two pieces of data are required
to complete a Lightning payment: a payment hash and a destination. As
`SHA-256` is used in the Lightning Network to implement HTLCs, this information
requires 32 bytes to communicate. Destinations, on the other hand, are
simply the `secp256k1` public key of the node that wishes to receive a payment.
The purpose of a payment request in the context of the Lightning Network is to
communicate these two pieces of information from sender to receiver. The QR-code–friendly format for communicating the information required
to complete a payment from receiver to sender is described in https://github.com/lightningnetwork/lightning-rfc/blob/master/11-payment-encoding.md[BOLT #11: Invoice Protocol for Lightning Payments]. In practice, more than just the
payment hash and destination are communicated in a payment request to
make the encoding more fully featured.
=== Lightning Payment Requests versus Bitcoin Addresses
((("Bitcoin addresses, Lightning invoices versus")))((("Lightning invoices","Bitcoin addresses versus")))A commonly asked question when people first encounter a Lightning Payment
request is: why can't a normal static address format be used instead?
To answer this question, you must first internalize how Lightning
differs from base layer Bitcoin as a payment method. Compared to a Bitcoin
address which may be used to make a potentially unbounded number of payments
(though reusing a Bitcoin address may degrade one's privacy), a Lightning
payment request should only ever be used _once_. This is due to the fact that
sending a payment to a Bitcoin address essentially uses a public key
cryptosystem to "encode" the payment in a manner that only the true "owner" of
that Bitcoin address can redeem it.
In contrast, to complete a Lightning payment, the recipient must
reveal a "secret" to the entire payment route including the sender. This can be
interpreted as usage of a kind of domain-specific symmetric cryptography because
the payment preimage is for practical purposes a nonce (number only used
once). If the sender attempts to make another payment using that identical
payment hash, then they risk losing funds because the payment may not actually be
delivered to the destination. It's safe to assume that after a preimage has
been revealed, all nodes in the path will keep it around forever, then rather
than forward the HTLC to collect a routing fee if the payment is
completed, they can simply settle the payment at that instance and gain the
entire payment amount in return. As a result, it's unsafe to ever use a payment
request more than once.
New variants of the original Lightning payment request exist that allow the sender to reuse them as many times as they want. These variants flip the normal payment flow as the sender transmits a preimage within the encrypted onion payload to the receiver, who is the only
one that is able to decrypt it and settle the payment. Alternatively, assuming
a mechanism that allows a sender to typically request a new payment request
from the receiver, then an interactive protocol can be used to allow a
degree of payment request reuse.
=== BOLT #11: Lightning Payment Request Serialization and Interpretation
((("BOLT (Basis of Lightning Technology) standards documents","Lightning payment request serialization/interpretation")))((("Lightning invoices","payment request serialization/interpretation")))In this section, we'll describe the mechanism used to encode the set of
information required to complete a payment on the Lightning Network. As
mentioned earlier, the payment hash and destination is the minimum amount of
information required to complete a payment. However, in practice, more
information such as time-lock information, payment request expiration, and
possibly an on-chain fallback address are also communicated. The full specification document is https://github.com/lightningnetwork/lightning-rfc/blob/master/11-payment-encoding.md[BOLT #11: Invoice Protocol for Lightning Payments].
==== Payment Request Encoding in Practice
((("Lightning invoices","payment request encoding in practice")))First, let's examine what a real payment request looks like in practice. The
following is a valid payment request that could have been used to complete a
payment on the mainnet Lightning Network at the time it was created:
```
lnbc2500u1pvjluezpp5qqqsyqcyq5rqwzqfqqqsyqcyq5rqwzqfqqqsyqcyq5rqwzqfqypqdq5xysxxatsyp3k7enxv4jsxqzpuaztrnwngzn3kdzw5hydlzf03qdgm2hdq27cqv3agm2awhz5se903vruatfhq77w3ls4evs3ch9zw97j25emudupq63nyw24cg27h2rspfj9srp
```
==== The Human-Readable Prefix
((("human-readable prefixes")))((("Lightning invoices","human-readable prefix")))Looking at the string, we can tease out a portion that we can parse with our
eyes, while the rest of it just looks like a random set of strings. The part
that is somewhat parsable by a human is referred to as the _human-readable prefix_. It allows a human to quickly extract some relevant information from a
payment request at a glance. In this case, we can see that this payment is for
the mainnet instance of the Lightning Network (`lnbc`), and is requesting 2,500
uBTC (microbitcoin), or `25,000,000` satoshis. The latter potion is referred
to as the data portion and uses an extensible format to encode the
information required to complete a payment.
Each version of instance of the Lightning Network (mainnet, testnet, etc.) has
its own human-readable prefix (see <<table1501>>). This allows client software and also humans to
quickly determine if a payment request can be satisfied by their node or not.
[[table1501]]
.BOLT #11 network prefixes
[options="header"]
|=============================
|Network |BOLT #11 prefix
|mainnet |`lnbc`
|testnet |`lntb`
|simnet/regtest|`lnbcrt`
|=============================
The first portion of the human-readable prefix is a _compact_ expression of the
amount of the payment request. The compact amount is encoded in two parts:
first, an integer is used as the _base_ amount. This is then followed by a
`multiplier` that allows us to specify distinct order of magnitude increases
offset by the base amount. If we return to our initial example, then we can
take the `2500u` portion and decrease it by a factor of 1,000 to instead use
`2500m` or (2,500 `mBTC`). As a rule of thumb, to ascertain the amount
of an invoice at a glance, take the base factor and multiply it by the
`multiplier`.
A full list of the currently defined multipliers is given in <<table1502>>.
[[table1502]]
.BOLT #11 amount multipliers
[options="header"]
|==============================================
|Multiplier|Bitcoin unit|Multiplication factor
|`m`|milli|0.001
|`u`|micro|0.000001
|`n`|nano|0.000000001
|`p`|pico|0.000000000001
|==============================================
==== Bech32 and the Data Segment
((("bech32, Lightning invoices and")))((("Lightning invoices","bech32 and data segment")))If the "unreadable" portion looks familiar, then that's because it uses the
savery same encoding scheme as SegWit compatible Bitcoin addresses use today,
namely bech32. Describing the bech32 encoding scheme is outside the scope
of this chapter. In brief, it's a sophisticated way to encode short strings
that has very good error correction as well as detection properties.
The data portion can be separated into three sections:
* The timestamp
* Zero or more tagged key-value pairs
* The signature of the entire invoice
The timestamp is expressed in seconds since the year 1970, or the Unix Epoch. This
timestamp allows the sender to gauge how old the invoice is, and as we'll see
later, allows the receiver to force an invoice to only be valid for a period of
time if they wish.
Similar to the TLV format we learned about in <<tlv>>, the BOLT #11 invoice
format uses a series of extensible key-value pairs to encode information
needed to satisfy a payment. Because key-value pairs are used, it's easy to add
new values in the future if a new payment type or additional
requirement/functionality is introduced.
Finally, a signature is included that covers the entire invoice signed by the
destination of the payment. This signature allows the sender to verify that the
payment request was indeed created by the destination of the payment. Unlike
Bitcoin payment requests which aren't signed, this allows us to ensure that a
particular entity signed the payment request. The signature itself is encoded
using a recovery ID, which allows a more compact signature to be used that
allows public key extraction. When verifying the signature, the recovery ID
extracts the public key, then verifies that against the public key included in
the invoice.
===== Tagged invoice fields
((("Lightning invoices","tagged invoice fields")))The tagged invoice fields are encoded in the main body of the invoice. These
fields represent different key-value pairs that express either additional
information that may help complete the payment or information which is
_required_ to complete the payment. Because a slight variant of bech32 is
utilized, each of these fields are actually in the "base 5" domain.
A given tag field is comprised of three components:
* The `type` of the field (5 bits)
* The `length` of the data of the field (10 bits
* The `data` itself, which is `length * 5 bytes` in size
A full list of all the currently defined tagged fields is given in <<table1503>>.
[[table1503]]
.BOLT #11 tagged invoice fields
[options="header"]
|===
|Field tag|Data length|Usage
|`p`|`52`|The `SHA-256` payment hash.
|`s`|`52`|A `256-bit` secret that increases the end-to-end privacy of a payment by mitigating probing by intermediate nodes.
|`d`|Variable|The description, a short UTF-8 string of the purpose of the payment.
|`n`|`53`|The public key of the destination node.
|`h`|`52`|A hash that represents a description of the payment itself. This can be used to commit to a description that's over 639 bytes in length.
|`x`|Variable|The expiry time in seconds of the payment. The default is 1 hour (3,600) if not specified.
|`c`|Variable|The `min_cltv_expiry` to use for the final hop in the route. The default is 9 if not specified.
|`f`|Variable|A fallback on-chain address to be used to complete the payment if the payment cannot be completed over the Lightning Network.
|`r`|Variable|One or more entries that allow a receiver to give the sender additional ephemeral edges to complete the payment.
|`9`|Variable|A set of 5-bit values that contain the feature bits that are required in order to complete the payment.
|===
The elements contained in the field `r` are commonly referred to as _routing hints_. They allow the receiver to communicate an extra set of edges that may
help the sender complete their payment. The hints are usually used when the
receiver has some/all private channels, and they wish to guide the sender into
this "unmapped" portion of the channel graph. A routing hint encodes
effectively the same information that a normal `channel_update` message does.
The update is itself packed into a single value with the following fields:
* The `pubkey` of the outgoing node in the edge (264 bits)
* The `short_channel_id` of the "virtual" edge (64 bits)
* The base fee (`fee_base_msat`) of the edge (32 bits)
* The proportional fee (`fee_proportional_millionths`) (32 bits)
* The CLTV expiry delta (`cltv_expiry_delta`) (16 bits)
The final portion of the data segment is the set of feature bits that
communicate to the sender the functionality needed to complete a
payment. As an example, if a new payment type is added in the future that isn't
backward compatible with the original payment type, then the receiver can set
a _required_ feature bit to communicate that the payer needs to
understand that feature to complete the payment.
=== Conclusion
As we have seen, invoices are a lot more than just a request for an amount. They contain critical information about _how_ to make the payment, such as routing hints, the destination node's public key, ephemeral keys to increase security, and much more.(((range="endofrange", startref="ix_15_payment_requests-asciidoc0")))

1242
16_security_privacy_ln.asciidoc
View File

File diff suppressed because it is too large Load Diff

162
17_conclusion.asciidoc
Unescape Escape View File

@ -1,81 +1,81 @@
[[conclusion_chapter]]
== Conclusion
In just a few years, the Lightning Network has gone from a whitepaper to a rapidly growing global network. As Bitcoin's second layer it has delivered on the promise of fast, inexpensive, and private payments. Additionally, it has started a tsunami of innovation, as it unleashes developers from the constraints of lock-step consensus that exist in Bitcoin development.
Innovation in the Lightning Network is happening in several different levels:
* At Bitcoin's Core protocol, providing use and demand for new Bitcoin Script opcodes, signing algorithms, and optimizations
* At the Lightning protocol level with new features deployed rapidly network-wide
* At the payment channel level, with new channel constructs and enhancements
* As distinct opt-in features deployed end-to-end by independent implementations that senders and recipients can use if they want
* With new and exciting Lightning Applications (LApps) build on top of the clients and protocols
Let's look at how these innovations are changing Lightning now and in the near future.
=== Decentralized and Asynchronous Innovation
Lightning isn't bound by lock-step consensus, as is the case with Bitcoin. That means that different Lightning clients can implement different features and negotiate their interactions (see <<feature_bits>>). As a result, innovation in the Lightning Network is occurring at a much faster rate than in Bitcoin.
Not only is Lightning advancing rapidly, but it is creating demand for new features in the Bitcoin system. Many recent and planned innovations in Bitcoin are both motivated and justified by their use in the Lightning Network. In fact, the Lightning Network is often mentioned as an example use-case for many of the new features.
[[bitcoin_prot_17]]
==== Bitcoin Protocol and Bitcoin Script Innovation
The Bitcoin system is, by necessity, a conservative system that has to preserve compatibility with consensus rules to avoid unplanned forks of the blockchain or partitions of the P2P network. As a result, new features require a lot of coordination and testing before they are implemented in mainnet, the live production system.
Here are some of the current or proposed innovations in Bitcoin that are motivated by use cases in the Lightning Network:
Neutrino:: A lightweight client protocol with improved privacy features over the legacy SPV protocol. Neutrino is mostly used by Lightning clients to access the Bitcoin blockchain.
Schnorr signatures:: Introduced as part of the _Taproot_ soft fork, Schnorr signatures will enable flexible Point Time-Locked Contracts (PTLC) for channel construction in Lightning. This might in particular make use of revocable signatures instead of revocable transactions.
Taproot:: Also part of the November 2021 soft fork that introduces Schnorr signatures, Taproot allows complex scripts to appear as single-payer, single-payee payments, and indistinguishable from the most common type of payment on Bitcoin. This will allow Lightning channel cooperative (mutual) closure transactions to appear indistinguishable from simple payments and increase privacy for LN users.
Input rebinding:: Also known by the names SIGHAS_NOINPUT or SIGHASH_ANYPREVOUT, this planned upgrade to the Bitcoin Script language is primarily motivated by advanced smart contracts such as the eltoo channel protocol.
Covenants:: Currently in the early stages of research, coventants allow transactions to create outputs that constrain future transactions which spend them. This mechanism could increase security for Lightning channels by making it possible to enforce address whitelisting in commitment transactions.
==== Lightning Protocol Innovation
The Lightning P2P protocol is highly extensible and has undergone a lot of change since its inception. The "It's OK to be odd" rule used in feature bits (see <<feature_bits>>) ensures that nodes can negotiate the features they support, enabling multiple independent upgrades to the protocol.
==== TLV Extensibility
The Type-Length-Value (see <<tlv>>) mechanism for extending the messaging protocol is extremely powerful and has already enabled the introduction of several new capabilities in Lightning while maintaining both forward and backward compatibility.
A prominent example, which is currently being developed and makes use of this, is path blinding and trampoline payments. This allows a recipient to hide itself from the sender, but also allows mobile clients to send payments without the necessity of storing the full channel graph on their devices by using a third party to which they don't need to reveal the final recipient.
==== Payment Channel Construction
Payment channels are an abstraction that is operated by two channel partners. As long as those two are willing to run new code, they can implement a variety of channel mechanisms simultaneously. In fact, recent research suggests that channels could even be upgraded to a new mechanism dynamically, without closing the old channel and opening a new channel type.
eltoo:: A proposed channel mechanism that uses input-rebinding to significantly simplify the operation of payment channels and remove the need for the penalty mechanism. It needs a new Bitcoin signature type before it can be implemented
==== Opt-In End-to-End Features
Point Time-Locked Contracts (PTLCs):: A different approach to HTLCs, PTLCs can increase privacy, reduce information leaked to intermediary nodes, and operate more efficiently than HTLC-based channels.
Large channels:: Large or _Wumbo_ channels were introduced in a dynamic way to the network without requiring coordination. Channels that support large payments are advertized as part of the channel announcement messages and can be used in an opt-in manner.
Multipart payments (MPP):: MPP was also introduced in an opt-in manner, but even better only requires the sender and recipient of a payment to be able to do MPP. The rest of the network simply routes HTLCs as if they are single-part payments.
JIT-Routing:: An optional method that can be used by routing nodes to increase their reliability and to protect themselves from being spammed.
Keysend:: An upgrade introduced independently by Lightning client implementations, it allows the sender to send money in an "unsolicited" and asynchronous way without requiring an invoice first.
HODL invoicesfootnote:[The word _HODL_ comes from an excited misspelling of the word "HOLD" shouted in a forum to encourage people not to sell bitcoin in a panic.]:: Payments where the final HTLC is not collected, committing the sender to the payment, but allowing the recipient to delay collection until some other condition is satisfied, or cancel the invoice without collection. This was also implemented independently by different Lightning clients and can be used in an opt-in manner.
Onion routed message services:: The onion routing mechanism and the underlying public key databse of nodes can be used to send data that is unrelated to payments, such as text messages or forum posts. The use of Lightning to enable paid messaging as a solution to spam posts and Sybil attacks (spam) is another innovation that was implemented independently of the core protocol.
Offers:: Currently proposed as BOLT #12 but already implemented by some nodes, this is a communication protocol to request (recurring) invoices from remote nodes via Onion messages.
[[lapps]]
=== Lightning Applications (LApps)
While still in their infancy, we are already seeing the emergence of interesting Lightning Applications. Broadly defined as an application that uses the Lightning Protocol or a Lightning client as a component, LApps are the application layer of Lightning.
A prominent example is LNURL, which provides a similar functionality as BOLT #12 Offers, but just over http and Lightning addresses. It works on top of Offers to provide users with an email-style address to which others can send funds while the software in the background requests an invoice against the LNURL endpoint of the node.
Further LApps are being built for simple games, messaging applications, microservices, payable APIs, paid dispensers (e.g., fuel pumps), derivative trading systems, and much more.
=== Ready, Set, Go!
The future is looking bright. The Lightning Network is taking Bitcoin to new unexplored markets and applications. Equipped with the knowledge in this book, you can explore this new frontier or maybe even join as a pioneer and forge a new path.
[[conclusion_chapter]]
== Conclusion
((("innovations in Lightning", id="ix_17_conclusion-asciidoc0", range="startofrange")))In just a few years, the Lightning Network has gone from a whitepaper to a rapidly growing global network. As Bitcoin's second layer it has delivered on the promise of fast, inexpensive, and private payments. Additionally, it has started a tsunami of innovation, as it unleashes developers from the constraints of lock-step consensus that exist in Bitcoin development.
Innovation in the Lightning Network is happening in several different levels:
* At Bitcoin's Core protocol, providing use and demand for new Bitcoin Script opcodes, signing algorithms, and optimizations
* At the Lightning protocol level with new features deployed rapidly network-wide
* At the payment channel level, with new channel constructs and enhancements
* As distinct opt-in features deployed end-to-end by independent implementations that senders and recipients can use if they want
* With new and exciting Lightning Applications (LApps) build on top of the clients and protocols
Let's look at how these innovations are changing Lightning now and in the near future.
=== Decentralized and Asynchronous Innovation
((("innovations in Lightning","decentralized/asynchronous nature of")))Lightning isn't bound by lock-step consensus, as is the case with Bitcoin. That means that different Lightning clients can implement different features and negotiate their interactions (see <<feature_bits>>). As a result, innovation in the Lightning Network is occurring at a much faster rate than in Bitcoin.
Not only is Lightning advancing rapidly, but it is creating demand for new features in the Bitcoin system. Many recent and planned innovations in Bitcoin are both motivated and justified by their use in the Lightning Network. In fact, the Lightning Network is often mentioned as an example use-case for many of the new features.
[[bitcoin_prot_17]]
==== Bitcoin Protocol and Bitcoin Script Innovation
((("Bitcoin (system)","innovations motivated by Lightning Network use cases")))((("Bitcoin script","innovations motivated by Lightning Network use cases")))((("innovations in Lightning","Bitcoin innovations motivated by Lightning Network use cases")))The Bitcoin system is, by necessity, a conservative system that has to preserve compatibility with consensus rules to avoid unplanned forks of the blockchain or partitions of the P2P network. As a result, new features require a lot of coordination and testing before they are implemented in mainnet, the live production system.
Here are some of the current or proposed innovations in Bitcoin that are motivated by use cases in the Lightning Network:
Neutrino:: A lightweight client protocol with improved privacy features over the legacy SPV protocol. Neutrino is mostly used by Lightning clients to access the Bitcoin blockchain.
Schnorr signatures:: Introduced as part of the _Taproot_ soft fork, Schnorr signatures will enable flexible Point Time-Locked Contracts (PTLC) for channel construction in Lightning. This might in particular make use of revocable signatures instead of revocable transactions.
Taproot:: Also part of the November 2021 soft fork that introduces Schnorr signatures, Taproot allows complex scripts to appear as single-payer, single-payee payments, and indistinguishable from the most common type of payment on Bitcoin. This will allow Lightning channel cooperative (mutual) closure transactions to appear indistinguishable from simple payments and increase privacy for LN users.
Input rebinding:: Also known by the names SIGHAS_NOINPUT or SIGHASH_ANYPREVOUT, this planned upgrade to the Bitcoin Script language is primarily motivated by advanced smart contracts such as the eltoo channel protocol.
Covenants:: Currently in the early stages of research, coventants allow transactions to create outputs that constrain future transactions which spend them. This mechanism could increase security for Lightning channels by making it possible to enforce address whitelisting in commitment transactions.
==== Lightning Protocol Innovation
((("innovations in Lightning","Lightning P2P protocol")))((("Lightning Network Protocol","innovations in")))The Lightning P2P protocol is highly extensible and has undergone a lot of change since its inception. The "It's OK to be odd" rule used in feature bits (see <<feature_bits>>) ensures that nodes can negotiate the features they support, enabling multiple independent upgrades to the protocol.
==== TLV Extensibility
((("innovations in Lightning","TLV extensibility")))((("Type-Length-Value (TLV) format","innovations in")))The Type-Length-Value (see <<tlv>>) mechanism for extending the messaging protocol is extremely powerful and has already enabled the introduction of several new capabilities in Lightning while maintaining both forward and backward compatibility.
A prominent example, which is currently being developed and makes use of this, is path blinding and trampoline payments. This allows a recipient to hide itself from the sender, but also allows mobile clients to send payments without the necessity of storing the full channel graph on their devices by using a third party to which they don't need to reveal the final recipient.
==== Payment Channel Construction
((("innovations in Lightning","payment channel construction")))((("payment channel","innovations in construction")))Payment channels are an abstraction that is operated by two channel partners. As long as those two are willing to run new code, they can implement a variety of channel mechanisms simultaneously. In fact, recent research suggests that channels could even be upgraded to a new mechanism dynamically, without closing the old channel and opening a new channel type.
eltoo:: A proposed channel mechanism that uses input-rebinding to significantly simplify the operation of payment channels and remove the need for the penalty mechanism. It needs a new Bitcoin signature type before it can be implemented
==== Opt-In End-to-End Features
((("innovations in Lightning","opt-in end-to-end features")))Point Time-Locked Contracts (PTLCs):: A different approach to HTLCs, PTLCs can increase privacy, reduce information leaked to intermediary nodes, and operate more efficiently than HTLC-based channels.
Large channels:: Large or _Wumbo_ channels were introduced in a dynamic way to the network without requiring coordination. Channels that support large payments are advertized as part of the channel announcement messages and can be used in an opt-in manner.
Multipart payments (MPP):: MPP was also introduced in an opt-in manner, but even better only requires the sender and recipient of a payment to be able to do MPP. The rest of the network simply routes HTLCs as if they are single-part payments.
JIT-Routing:: An optional method that can be used by routing nodes to increase their reliability and to protect themselves from being spammed.
Keysend:: An upgrade introduced independently by Lightning client implementations, it allows the sender to send money in an "unsolicited" and asynchronous way without requiring an invoice first.
HODL invoicesfootnote:[The word _HODL_ comes from an excited misspelling of the word "HOLD" shouted in a forum to encourage people not to sell bitcoin in a panic.]:: Payments where the final HTLC is not collected, committing the sender to the payment, but allowing the recipient to delay collection until some other condition is satisfied, or cancel the invoice without collection. This was also implemented independently by different Lightning clients and can be used in an opt-in manner.
Onion routed message services:: The onion routing mechanism and the underlying public key databse of nodes can be used to send data that is unrelated to payments, such as text messages or forum posts. The use of Lightning to enable paid messaging as a solution to spam posts and Sybil attacks (spam) is another innovation that was implemented independently of the core protocol.
Offers:: Currently proposed as BOLT #12 but already implemented by some nodes, this is a communication protocol to request (recurring) invoices from remote nodes via Onion messages.
[[lapps]]
=== Lightning Applications (LApps)
((("innovations in Lightning","Lightning Applications")))((("Lightning Applications (LApps)")))While still in their infancy, we are already seeing the emergence of interesting Lightning Applications. Broadly defined as an application that uses the Lightning Protocol or a Lightning client as a component, LApps are the application layer of Lightning.
A prominent example is LNURL, which provides a similar functionality as BOLT #12 Offers, but just over http and Lightning addresses. It works on top of Offers to provide users with an email-style address to which others can send funds while the software in the background requests an invoice against the LNURL endpoint of the node.
Further LApps are being built for simple games, messaging applications, microservices, payable APIs, paid dispensers (e.g., fuel pumps), derivative trading systems, and much more.
=== Ready, Set, Go!
The future is looking bright. The Lightning Network is taking Bitcoin to new unexplored markets and applications. Equipped with the knowledge in this book, you can explore this new frontier or maybe even join as a pioneer and forge a new path.(((range="endofrange", startref="ix_17_conclusion-asciidoc0")))

1034
appendix-bitcoin-fundamentals-review.asciidoc
View File

File diff suppressed because it is too large Load Diff

222
appendix_docker_basics.asciidoc
Unescape Escape View File

@ -1,111 +1,111 @@
[appendix]
[[appendix_docker]]
== Docker Basic Installation and Use
This book contains a number of examples that run inside Docker containers for standardization across different operating systems.
This section will help you install Docker and familiarize yourself with some of the most commonly used Docker commands, so that you can run the book's example containers.
=== Installing Docker
Before we begin, you should install the Docker container system on your computer. Docker is an open system that is distributed for free as a _Community Edition_ for many different operating systems including Windows, macOS, and Linux. The Windows and Mac versions are called _Docker Desktop_ and consist of a GUI desktop application and command-line tools. The Linux version is called _Docker Engine_ and is comprised of a server daemon and command-line tools. We will be using the command-line tools, which are identical across all platforms.
Go ahead and install Docker for your operating system by following the instructions to "Get Docker" from the https://docs.docker.com/get-docker[Docker website].
Select your operating system from the list and follow the installation instructions.
[TIP]
====
If you install on Linux, follow the post-installation instructions to ensure you can run Docker as a regular user instead of user root. Otherwise, you will need to prefix all +docker+ commands with +sudo+, running them as root like: +sudo docker+.
====
Once you have Docker installed, you can test your installation by running the demo container +hello-world+ like this:
[[docker-hello-world]]
----
$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
[...]
----
=== Basic Docker Commands
In this appendix, we use Docker quite extensively. We will be using the following Docker commands and arguments.
==== Building a Container
++++
<pre data-type="programlisting">docker build [-t <em>tag</em>] [<em>directory</em>]</pre>
++++
++__tag__++ is how we identify the container we are building, and ++__directory__++ is where the container's context (folders and files) and definition file (+Dockerfile+) are found.
==== Running a Container
++++
<pre data-type="programlisting">docker run -it [--network <em>netname</em>] [--name <em>cname</em>] <em>tag</em></pre>
++++
++__netname__++ is the name of a Docker network, ++__cname__++ is the name we choose for this container instance, and ++__tag__++ is the name tag we gave the container when we built it.
==== Executing a Command in a Container
++++
<pre data-type="programlisting">docker exec <em>cname command</em></pre>
++++
++__cname__++ is the name we gave the container in the +run+ command, and ++__command__++ is an executable or script that we want to run inside the container.
==== Stopping and Starting a Container
In most cases, if we are running a container in an _interactive_ as well as _terminal_ mode, i.e., with the +i+ and +t+ flags (combined as +-it+) set, the container can be stopped by simply pressing +CTRL-C+ or by exiting the shell with +exit+ or +CTRL-D+. If a container does not terminate, you can stop it from another terminal like this:
++++
<pre data-type="programlisting">docker stop <em>cname</em></pre>
++++
To resume an already existing container, use the `start` command like so:
++++
<pre data-type="programlisting">docker start <em>cname</em></pre>
++++
==== Deleting a Container by Name
If you name a container instead of letting Docker name it randomly, you cannot reuse that name until the container is deleted. Docker will return an error like this:
[source,bash]
----
docker: Error response from daemon: Conflict. The container name "/bitcoind" is already in use...
----
To fix this, delete the existing instance of the container:
++++
<pre data-type="programlisting">docker rm <em>cname</em></pre>
++++
++__cname__++ is the name assigned to the container (+bitcoind+ in the example error message).
==== List Running Containers
----
docker ps
----
This command shows the current running containers and their names.
==== List Docker Images
----
docker image ls
----
This command shows the Docker images that have been built or downloaded on your computer.
=== Conclusion
These basic Docker commands will be enough to get you started and will allow you to run all the examples in this book.
[appendix]
[[appendix_docker]]
== Docker Basic Installation and Use
((("Docker","basic installation and use", id="ix_appendix_docker_basics-asciidoc0", range="startofrange")))This book contains a number of examples that run inside Docker containers for standardization across different operating systems.
This section will help you install Docker and familiarize yourself with some of the most commonly used Docker commands, so that you can run the book's example containers.
=== Installing Docker
((("Docker","installing")))Before we begin, you should install the Docker container system on your computer. Docker is an open system that is distributed for free as a _Community Edition_ for many different operating systems including Windows, macOS, and Linux. The Windows and Mac versions are called _Docker Desktop_ and consist of a GUI desktop application and command-line tools. The Linux version is called _Docker Engine_ and is comprised of a server daemon and command-line tools. We will be using the command-line tools, which are identical across all platforms.
Go ahead and install Docker for your operating system by following the instructions to "Get Docker" from the https://docs.docker.com/get-docker[Docker website].
Select your operating system from the list and follow the installation instructions.
[TIP]
====
If you install on Linux, follow the post-installation instructions to ensure you can run Docker as a regular user instead of user root. Otherwise, you will need to prefix all +docker+ commands with +sudo+, running them as root like: +sudo docker+.
====
Once you have Docker installed, you can test your installation by running the demo container +hello-world+ like this:
[[docker-hello-world]]
----
$ docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
[...]
----
=== Basic Docker Commands
((("Docker","basic commands")))In this appendix, we use Docker quite extensively. We will be using the following Docker commands and arguments.
==== Building a Container
++++
<pre data-type="programlisting">docker build [-t <em>tag</em>] [<em>directory</em>]</pre>
++++
((("Docker","building a container")))((("Docker containers","building a container")))++__tag__++ is how we identify the container we are building, and ++__directory__++ is where the container's context (folders and files) and definition file (+Dockerfile+) are found.
==== Running a Container
++++
<pre data-type="programlisting">docker run -it [--network <em>netname</em>] [--name <em>cname</em>] <em>tag</em></pre>
++++
((("Docker containers","running a container")))++__netname__++ is the name of a Docker network, ++__cname__++ is the name we choose for this container instance, and ++__tag__++ is the name tag we gave the container when we built it.
==== Executing a Command in a Container
++++
<pre data-type="programlisting">docker exec <em>cname command</em></pre>
++++
((("Docker containers","executing a command in a container")))++__cname__++ is the name we gave the container in the +run+ command, and ++__command__++ is an executable or script that we want to run inside the container.
==== Stopping and Starting a Container
((("Docker containers","stopping/starting a container")))In most cases, if we are running a container in an _interactive_ as well as _terminal_ mode, i.e., with the +i+ and +t+ flags (combined as +-it+) set, the container can be stopped by simply pressing +CTRL-C+ or by exiting the shell with +exit+ or +CTRL-D+. If a container does not terminate, you can stop it from another terminal like this:
++++
<pre data-type="programlisting">docker stop <em>cname</em></pre>
++++
To resume an already existing container, use the `start` command like so:
++++
<pre data-type="programlisting">docker start <em>cname</em></pre>
++++
==== Deleting a Container by Name
((("Docker containers","deleting a container by name")))If you name a container instead of letting Docker name it randomly, you cannot reuse that name until the container is deleted. Docker will return an error like this:
[source,bash]
----
docker: Error response from daemon: Conflict. The container name "/bitcoind" is already in use...
----
To fix this, delete the existing instance of the container:
++++
<pre data-type="programlisting">docker rm <em>cname</em></pre>
++++
++__cname__++ is the name assigned to the container (+bitcoind+ in the example error message).
==== List Running Containers
----
docker ps
----
((("Docker containers","list running containers")))This command shows the current running containers and their names.
==== List Docker Images
----
docker image ls
----
((("Docker containers","list Docker images")))This command shows the Docker images that have been built or downloaded on your computer.
=== Conclusion
These basic Docker commands will be enough to get you started and will allow you to run all the examples in this book.(((range="endofrange", startref="ix_appendix_docker_basics-asciidoc0")))

1446
appendix_protocol_messages.asciidoc
View File

File diff suppressed because it is too large Load Diff
Write Preview
Loading…
Cancel
Save