lnbook/ch03.asciidoc

[role="pagenumrestart"]
[[ch03_How_Lightning_Works]]
== How the Lightning Network Works

Now that we've followed Alice as she's set up a Lightning Wallet and purchased a coffee from Bob, we'll look under the hood and unpack the different components of the Lightning Network involved in that process.
This chapter will give a high level overview and will not delve into all the technical details.
The goal is rather to help you to become aware of the most important concepts and building blocks of the Lightning Network.

If you have experience in computer science, cryptography, Bitcoin, and protocol development, then this chapter should be enough for you to be able to fill out the connecting details by yourself.
If you are less experienced, this chapter shall give you a good enough overview so that you will have an easier time reading through the formal protocol specifications, known as BOLTs (Basis of Lightning Technology).
In case you are a beginner, this chapter will help you better understand the technical chapters of the book.
However, we are aware that some readers may have a hard time following high level descriptions without explaining all the details.
If you are one of those readers you may want to skip this chapter.

We'll start with a one sentence definition of what the Lightning Network (LN) is and break it down in the remainder of this chapter.

**The Lightning Network (LN) is both a peer-to-peer network of _payment channels_ on top of the _Bitcoin protocol_ as well as a communication protocol that defines how participants set up and execute the smart contracts on the Bitcoin network**

We will see that a payment channel is simply a 2-out-of-2 multisignature address on the Bitcoin network for which you hold one key and your channel partner holds the other key.
This multisignature address comes with a cryptographic protocol that is established by creating a sequence of transactions that spend from this address.
Each new transaction is negotiated between you and your channel partner and are attempts to spend from the multisignature address.
The latest transaction in the sequence encodes the balance of the channel and defines how the funds locked into the multisignature address are to be divided between you and your channel partner.
Thus adding a new transaction to this sequence is equivalent to moving ownership of funds in the channel, without the Bitcoin network being aware of it.
Each transaction in the sequence makes use of Bitcoin's scripting language, and thus the negotiation of funds between you and your channel partner is managed by a Bitcoin smart contract.
The contract is set up such as to penalize a channel member if it tries to steal funds by submitting an earlier invalidated state of the channel.

[NOTE]
====
When someone says they 'own' bitcoin they typically mean that they know the private key of a bitcoin address that has some unspent transaction outputs (UTXOs).
The private key allows the owner to produce a signature for a transaction that spends the bitcoin by sending it to a different address.
Thus 'ownership' of bitcoin can be defined as the ability to spend that bitcoin.

If you have an unpublished but signed transaction from a 2-out-of-2 multisignature address, where some outputs are sent to an address you own, and additionally you own one of the private keys of the multisignature address, then you effectively own the bitcoin of that output.
Without your help no other transaction from the 2-out-of-2 multisignature address can be produced.
However, without the help of anybody else you can transfer the funds to an address which you control.
On the Lightning Network ownership of your funds is almost always encoded with you having a pre-signed transaction spending from a 2-out-of-2 multisignature address.
====

Additionally, these contracts have the option to connect channels in a way that one can forward payments from one channel to another.
For example, Alice can send money to Bob if Alice had a channel with Mallory and Mallory had a channel with Bob.
By the design of the Lightning Network, it is possible to extend the smart contracts which operate the channel so that Mallory has no way of stealing the funds that are being routed through her.
Not only does the construction of the payment channel work for the partners without the necessity to trust each other but the entire network works without the necessity to trust any other participant.
Since the channels are funds on multisignature addresses and as the contracts are unpublished but presigned Bitcoin transactions, all the trust that is needed to operate the Lightning Network comes from the trust in the decentralized Bitcoin network!

The aforementioned innovations are certainly the major breakthrough that allow the creation of the Lightning Network.
However, the Lightning Network is so much more than the cryptographic protocols on top of the Bitcoin scripting language.
It is a full communication protocol that allows peers to communicate with each other via Lightning messages to achieve the transfer of bitcoin.
The communication protocol also defines how Lightning messages are being encrypted and exchanged.
A gossip protocol is part of the specification that is used to share public information about the topology of the network with the other participants.
This information is needed for Alice to be aware of the fact that Mallory has a channel with Bob so that she can decide to send a payment indirectly via Mallory to Bob.

Last but not least, it is important to understand that the Lightning Network is nothing but Bitcoin.
We emphasize this as you might find people who will try to spread misinformation and create a false barrier between the "real" Bitcoin and the Lightning Network or even use terms like the Lightning Network coin. At any point in time, any value held in the Lightning Network is exclusively held as bitcoin and exclusively secured by the Bitcoin network. 
We hope that as soon as you have studied the remainder of this book you do not need to believe what we just mentioned.
We rather hope that this book enables you to critically do your own research to understand and verify the following (instead of trusting us):
Besides all the technical primitives, the Lightning Network protocol is a creative way to get more benefits out of Bitcoin by allowing an arbitrary amount of instant payments with instant settlements without the necessity to trust anyone else but the Bitcoin network.

=== Payment channels

As you have seen in the last chapter, in order to use the Lightning Network, Alice had to use her wallet software to create a payment channel with another LN participant.
From a computer science perspective a payment channel is a cryptographic communication protocol between you and your channel partner.
It allows both of the channel partners to send funds back and forth as often as they wish.
A channel is only limited by two things:
First, the time it takes for the internet to transfer the few bytes of data that the protocol requires to make a payment between you and your partner.
Second, the capacity of the channel, which is the term for the monetary funds, i.e. bitcoin, that have been assigned to and locked up by the channel.
Payment channels have a couple of very interesting and useful properties.

* If the channel is open, making a payment does not require the confirmation of Bitcoin blocks. In fact - as long as you and your channel partner follow the protocol - it does not require any interaction with the Bitcoin network or anyone else other than with your channel partner.
* The cryptographic protocol is constructed in a way that there is little to no trust involved between you and your channel partner. If your partner becomes unresponsive or tries to cheat you, you can ask the Bitcoin network to act as a court system resolving the conflict according to the rules that you and your partner have previously agreed upon.
* The capacity of the channel will be split between you and your partner. In that sense at that level you already will have gained privacy compared to the Bitcoin network where every transaction is public. Within payment channels the amounts and values of payments are only known to you and your partner. Only the final balance which is the aggregate of all payments in that channel will become visible on the Bitcoin blockchain if and when the payment channel is closed.
* As the time to update the channel is primarily bound by the communication speed of the internet, making a payment on a payment channel is almost instant.

Bitcoin was about 5 years old when talented developers first figured out how payment channels could be constructed and by now there are at least 3 different methods known.
This chapter will only focus on the channel construction method proposed by Joseph Poon and Thaddeus Dryja in 2015 since it is actually being used in the Lightning Network and was first described in the Lightning Network whitepaper.
The two other proposed methods are the "Duplex Micropayment" channels which have been introduced by Christian Decker around the same time as the "Poon and Dryja" channels and the "eltoo" channels which have been introduced in 2018 by Christian Decker, Rusty Russel and our co-author Olaoluwa Osuntokun.
The later seem much easier to implement and have a couple of nicer properties.
However, these "eltoo" channels require a new OP_CODE in the Bitcoin scripting language and can therefore currently not be implemented on top of the Bitcoin mainnet.

==== Multisig addresses
Payment channels are smart contracts that build on top of 2-out-of-2 multisignature addresses which will be spent by Segwit transactions.
By knowing the properties of multisignature addresses you will be able to understand the exact construction of payment channels.
However, at this point we will not review all the technical details about multisignature addresses and Segwit but just stick to the basic properties.
If you know Bitcoin well enough to be familiar with those topics feel free to skip this section.

[TIP]
====
A deep dive into the topic discussed here is presented in chapter 7 of Mastering Bitcoin which can be found at: https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch07.asciidoc.
Also, in case you are not familiar with P2PKH addresses and the basic format and scripting language of Bitcoin we encourage you to study chapter 6 of Mastering Bitcoin which can be found at: https://github.com/bitcoinbook/bitcoinbook/blob/develop/ch06.asciidoc.
There is also a video on Rene's YouTube channel which dissects the bits and bytes of a transaction spending from a P2PKH output at: https://youtu.be/1n4g3eYX1UI
====

To allow escrow services and complex ownership configurations between several stakeholders, the Bitcoin scripting language provides multisignature addresses.
The general form of a locking script setting an M--out-of-N multisignature condition is:

----
M <Public Key 1> <Public Key 2> ... <Public Key N> N CHECKMULTISIG
----

where N is the total number of listed public keys and M is the threshold of required signatures to spend the output.

A special case which is particularly interesting for us is a locking script setting a 2-out-of-2 multisignature condition.
It would look like this:

----
2 <Public Key A> <Public Key B> 2 CHECKMULTISIG
----

The preceding locking script can be satisfied with an unlocking script containing pairs of signatures:

----
0 <Signature A> <Signature B>
----
The two scripts together would form the combined validation script:

----
0 <Signature A> <Signature B> 2 <Public Key A> <Public Key B> 2 CHECKMULTISIG
----

==== Funding Transaction
We have already seen that one of the important building blocks of a payment channel is a 2-out-of-2 multisignature address.
To open a payment channel one must send bitcoin to that address.
The Bitcoin transaction that sends bitcoin to that 2-out-of-2 multisignature address is called the funding transaction. 
It is included in the Bitcoin blockchain.
While the payment channel opened by two participants of the Lightning Network can be private, the funding transaction will always be publicly visible to the Bitcoin network.
The amount of bitcoin sent to the multisignature address forms an upper limit on how much Bitcoin can be transacted using the channel, and is called the capacity of the channel.
Two channel partners will never be able to conduct payments on a channel that are larger than its channel capacity.
While the Bitcoin network can see that funds have been committed to a channel using a funding transaction, it is unable to determine how those funds are distributed between the two channel partners.

[Note]
====
You might hear people falsely complaining about bitcoin being locked up by the Lightning Network preventing their movement.
This is incorrect.
One can use the Bitcoin network to send bitcoin from a P2PKH address as well as from a 2-out-of-2 multisignature address with a P2WSH transaction.
In both cases transfer of ownership incurs the corresponding bitcoin fees for processing the "on-chain" transaction on the Bitcoin network.
However, once the bitcoin are used to open a payment channel they can freely flow within the Lightning Network from one participant to another one.
If a channel partner should not respond, one will always have the chance to fall back to the on-chain transactions without the necessity for the channel partner to help to do so.
On-chain fees and Bitcoin confirmation times make moving bitcoin on the Bitcoin network more expensive and slower than moving bitcoin on the Lightning Network.
====

===== Example of a poor channel opening procedure
Alice wants to open a payment channel with Mallory but she is not yet fully aware of the Lightning Network Protocol.
She creates a new private and corresponding public key and talks to Mallory who does the same and sends her public key to Alice.
Now Alice prepares a Bitcoin Transaction sending a few mBTC to the multisignature address that was created from Alice's and Mallory's key.
As Alice wasn't aware of the protocol to open the channel she now has to trust that Mallory will provide her signature if Alice wants to spend from the multisignature address.
Mallory on the other side has the chance to execute a blackmail attack on Alice by holding back her signature and denying Alice access to her funds.

In order to prevent Mallory from performing such an attack Alice will need to create a spend transaction from the funding transaction and have that transaction signed by Mallory before she broadcasts her funding transaction to the Bitcoin network.
This transaction that protects Alice is called Commitment transaction and we will study it now.

==== Commitment Transaction

You have just learnt that a payment channel needs to be opened by preparing a funding transaction which sends the capacity of the payment channel to a 2-2 multisignature address.
From the example in the last section you learnt that more ingredients are necessary to open and operate a payment channel that does not rely on trusting the channel partner.
These ingredients are the commitment transactions.
They are used to make sure that everyone on the channel is able to get their own funds back in case the channel partner becomes unresponsive or, even worse, if the channel partner deliberately or by accident tries to cheat with the execution of the protocol.
The commitment transactions also encode the balance of the payment channel.
The balance of the payment channel is an agreement of the channel partners about how the capacity is split among the partners.
Let us assume Alice opens a channel with a capacity of 10 mBTC with Bob.
Naturally one would assume that Alice should still be in the possession of the 10 mBTC.
This can actually be easily achieved with the following construction:

. Alice creates a new private / public key pair and informs Bob that she wishes to open a channel via the `open_channel` message.
. Bob also creates a new private / public key pair and agrees to accept a channel from Alice while sending his public key to Alice via the `accept_channel` message.
. Alice now creates a funding transaction from her wallet that sends 10 mBTC to the multisignature address with a locking script `2 <Public Key A> <Public Key B> 2 CHECKMULTISIG`.
. Alice does not broadcast the funding transaction but informs Bob about the transaction id of the funding transaction by sending a `funding_created` message.
. Both Alice and Bob create their version of a commitment transaction. This Transaction will spend from the funding transaction and send all the bitcoin back to an address controlled by Alice.
. Alice provides a signature for Bob's Commitment Transaction. This signature was already included in the `funding_created` message.
. Bob provides a signature for Alice's Commitment Transaction and sends this back to Alice via the `funding_signed` message.
. Only after signatures have been exchanged Alice will broadcast the funding transaction to the Bitcoin network.

With this protocol Alice did not give up ownership of her 10 mBTC even though the funds have been sent to a 2-2 multisignature wallet for which Alice controls only one key.
If Bob stops responding to Alice she will be able to broadcast her commitment transaction and receive her funds back.
She will only have lost the fees for the two on-chain transactions.
As long as she follows the protocol and has her node secured this is her only risk when opening a channel.

The commitment transactions will not only serve the purpose of allowing Alice to withdraw her funds directly after opening the channel in case Bob does not answer.
More commitment transactions are created during the lifetime of the channel to encode the balance between Alice and Bob.
If Alice wanted to send 3 mBTC to Bob to pay him for a service he offered, both would create a new version of their commitment transaction which would now send 7mBTC to Alice and 3 mBTC to Bob and share signatures with each other.
However you will probably have realized that there is a major flaw with this particular design.

**Do you see any way how Alice could cheat on Bob?**

We hope you recognize that with the so far described system nothing could stop Alice from publishing her old or even initial commitment transaction which grants her 10 mBTC.
Since that commitment transaction has previously been signed by Bob he can't prevent Alice from doing so.
Obviously Alice could tell Bob that she has deleted the old commitment transaction but as we mentioned several times the Lightning Network does operate without trust so a smarter mechanism is needed to prevent Alice from publishing an old commitment transaction.
As Bitcoin is censorship resistant no one can prevent a participant from the Lightning Network to publish an old commitment transaction.
However the commitment transactions can be slightly modified so that publishing an outdated commitment transaction is discouraged by a rather high punishment.
The penalty for broadcasting an old commitment transaction is to give the other channel partner the ability to claim the funds that belonged to the broadcaster of the transaction.
This means that Bob would have the ability to claim 10 mBTC from the output that belonged to Alice in her original Commitment transaction if she publishes it after she has agreed to a second commitment transaction in which she would only own 7 mBTC and Bob would own 3 mBTC.
With such a strong penalty mechanism in place Alice should never purposely publish an old state as she would almost always lose her remaining funds in the channel.

[Note]
====
In Mastering Bitcoin Andreas states it very elegantly:
A key characteristic of bitcoin is that once a transaction is valid, it remains valid and does not expire. The only way to cancel a transaction is by double-spending its inputs with another transaction before it was mined. That's why we used timelocks [...] to ensure that more recent commitments could be spent before older commitments were valid.
====

We will now see how such a penalty mechanism can be included to the above construction of the commitment transactions.
Usually the commitment transaction has at least two outputs, one for each partner.
However a channel partner will encumber their own output with a timelock and a revocation secret.
The timelock prevents the owner of the output to spend it directly once the commitment transaction was included in a block.
The timelock is usually measured in blocktime and can be up to 2016 which is statistically speaking two weeks (assuming a blocktime of 10 minutes which is the target for the Bitcoin network).
Within the timelock anyone who knows the revocation secret can spend the output even well before the timelock was over.
Alice and Bob know only one half of the revocation secret but if they share their half with the other party, the other party knows the full secret.
In order to update the balance and receive a signature from Bob, Alice will have to share her half of the revocation secret of the current commitment transaction with Bob.

Obviously for every new update of the channel balance new revocation secrets have to be created and the old ones all need to be saved.
Luckily the secrets are rather small and it is only the channel partners who need to do that, not the entire network.
Still managing the revocation secrets is one of the more tricky parts of Lightning nodes that hinders node operators to maintain backups.
Watchtower services or switching to the aforementioned eltoo channels might be future strategies to mitigate these problems.

With the timelocks in place Alice can close the channel at any time if Bob does not respond.
She has to wait for the time lock until she can spend her funds from the commitment transaction after the commitment transaction has a confirmation.
Other than that and the fees for opening and closing a channel there comes no additional burden with maintaining a payment channel.

==== Announcing the channel
Channel partners can agree to publicly announce the channel over the gossip protocol that comes with the Lightning Network.
This is useful as other nodes will then be able to utilize this channel to route payments to other participants of the network.
Note that even for private channels which are not announced over the gossip protocol, the funding transaction is always publicly stored in the Bitcoin blockchain.
However as it is just a regular transaction to a 2-2 multisignature address, participants of the Bitcoin network do not know if this particular transaction is used to maintain a payment channel.
Private channel can still be used in routing payments but only by the subset of nodes which are aware of their existence.
If a channel and its capacity is publicly announced on the gossip protocol, the channel partners will also be able to announce some meta data about the channel.
This meta data includes the routing fees a node charges to forward payments on that channel, information about what kind and how many Hash Time-Locked Contracts (HTLCs) will be accepted.
As we have not discussed HTLCs yet we will just mention that they are additional conditional outputs in the commitment transactions used for routing payments and for updating the channel balance.
We will later investigate HTLCs in more detail.
When new participants join the Lightning Network they will be able to download the information propagated via the gossip protocol from their peers.
Peers can only omit messages but as every message is signed by the node that originally sent out the message the information on the gossip protocol cannot be modified to trick other participants.

==== Closing the channel
The main goal of people using the Lightning Network is to keep their channels open as long as possible.
Opening and closing payment channels will result in bitcoin fees and in transactions that need to be stored in the Bitcoin Blockchain.
An open channel on the other side allows you to make an arbitrary amount of payments on the Lightning Network (as long as you have funds and they are liquid).
However sometimes there is the necessity that you have to close a channel. For example:

* You have become aware of the fact that your computer got compromised and you want to secure your funds by sending them to cold storage.
* Your channel partner might be offline for too much time so that you cannot utilize the funds in that channel.
* After analyzing your routing statistics, as well as the network topology, you might have come to the conclusion that it might be better to close some channels and open some new ones.
* Your channel partner might have breached the protocol - due to a software bug or on purpose and you have to protect your funds.

There are 3 ways to close a payment channel:

* the good way - mutual close
* the bad way - force close
* the ugly way - protocol breach

Not all ways could be chosen for each of the above mentioned reasons.
For example if your channel partner is offline you will not be able to engage in the good way to do a mutual close.
The good news for you is that your Lightning Network software will most likely automatically select the best closing mechanism that can currently be used if you ask the software to close the channel or if the software discovers an issue with your channel partner and follows the protocol specification which in most of such cases state that the channel shall be closed.

===== Examining the good way - mutual close
The preferred and good way to close a channel is the mutual close.
When you decide that you want to close the channel, your Lightning Network node will inform your channel partner about your intention.
Now the channel will be prepared for shutting down.
No new routing attempts will be accepted from either channel partner and the ongoing routing attempts will be settled or removed after they timed out.
Once no further routing attempts are pending, the closing transaction is prepared.
This transaction is similar to the commitment transaction.
It has the same balance as the commitment transaction but no outputs are encumbered with a time lock.
As the finish up of the routing attempts could take some time, a mutual close can also take some time.
The on-chain transaction fees of the shutdown transaction for closing the channel in a mutual way are being paid by the party who opened the channel and not as many people think by the person who initiated the closing procedure.
As both nodes sign the shutdown transaction they have the chance to pay small fees for the Bitcoin transaction by using their on-chain fee estimator.
Even though there is a potential waiting time, this type of channel close is usually faster than the bad way.

===== Examining the bad way - force close
In case your node cannot engage in a mutual close (most likely because your channel partner is either offline or not responding) you will have to do a force close.
This is done by publishing the latest commitment transaction that your node has.
As discussed before the Bitcoin network has no way of knowing if this was the most recent commitment transaction or an old one which you might have published for a financial gain.
Thus after that transaction was mined you will have to wait for the timelock of your output to expire until you can spend your own funds.
The output of your channel partner is not encumbered with a time lock and can be spent directly.
The on-chain fees will be much higher than in the good way of the mutual close for several reasons:

* The most obvious reason is that when the commitment transaction was negotiated you and your channel partner didn't know how high the on-chain fees might be at the time the force close is taking place.
As the fees cannot be changed without reasigning outputs of the commitment transaction which needs two signatures and as the force close usually should happen in an urgent situation the protocol developers decided to be very generous with the fee rate for the commitment transactions. It can be up to 5 times higher than the fee estimators would suggest at the time the commitment transaction is negotiated.
* The pending routing attempts in the commitment transaction are encoded as additional outputs which take up more space and will also hit the chain.
* In particular those routing attempts will have to be resolved on-chain by additional spends. These additional spends don't have to overestimate the fees but it still adds to the bill.

In general you should not do a force close unless it is absolutely necessary.
Your funds will be locked for a longer time and the person who opened the channel will have to pay higher fees. Also you might have to pay on-chain fees to abort or settle routing attempts - even if you haven't opened the channel.

===== Examining the ugly way - protocol breach
In case your channel partner tries to cheat you - whether deliberate or not - by publishing an outdated commitment transaction, you will be able to use the timelock to catch this cheating attempt and collect on the outputs by using the revocation secret you had previously received to negotiate a newer state of the channel.
This close can actually go in two ways.
First if you catch your partner in time you will claim their funds. In that case the closing will be rather fast. Also you will have to pay the on-chain fees which could be really high if there is a lot of demand for transactions at that time.
This should not bother you as you just gained the entire channel capacity.
Second if you did not catch the cheating attempt then your channel partner will be able to collect their outputs after the time lock expired.
In that case the fees of the commitment transaction are again paid by the partner who opened the channel and the fees for collecting the outputs are paid by the person controlling the output that is being collected.
Also all the routing attempts will have to be resolved just as during the force close.
While this method could be fully executed faster than the good and the bad way to close the channel, it is obviously never recommended to engage in this channel closing protocol.

=== Invoices

Every payment on the Lightning Network starts with a person who wants to receive bitcoins issuing an invoice.
The main reason for this process is that it helps to make the payment process over a path of payement channels atomic.
Atomic means that no node on the path can decide to take the money that is being routed and stop the routing process.
The payment will either be transferred successfully through the path of nodes or will not be delivered.
There are no such things as a partial payment or a half successful payment.
While Lightning Nodes usually use the encrypted communication channels over the peer to peer network to exchange information, invoices are being transferred via a second communication channel.
This could be via a Webservice or Email.
Invoices are usually encoded either as long bech32 strings or as QR codes which enables them to be easily scanned by smartphones.
Obviously the invoices contains the amount of bitcoin that is requested and a signature of the payee.
The later is used to extract the address of the payee so that the payer knows whom to send the money.
Besides some other meta data the most important but not quite obvious data in the invoice is a Payment Hash.

==== Payment Hash
The payee will choose a truly random number `r` and produces the `sha256` of that number which we call the Payment Hash `H(r)`.
Note that an adversary should have no means of guessing or predicting `r`.
Using a customer id or the hash of entries of the shopping cart together with a timestamp is not truly random and yields a security risk.
The payment process of the Lightning Network is only secure if `r` is chosen completely randomly and is not predictable and as long as the Hash function cannot be inverted.
We note that this is not an additional security assumption for Bitcoin as the security of the Hash function is currently what Bitcoin mining is built upon.

==== Additional Meta Data
Invoices can encode some other useful meta data.
For example a short description.
In case a user has several invoices to pay, the user can read the description and make sure what the invoice was about.
As payment channels do not need to be publicly announced, the payee can also provide some private channels as routing hints to the invoice.
These hints can also be used for public channels to point to those channels on which the payee has enough inbound liquidity to actually receive the amount.
In case the payer's Lightning node is not able to send the payment over the Lightning Network, invoices can also include a fallback address.
We would however always recommend to open a new payment channel instead of doing an on-chain transaction that does not add an additional payment channel.
Invoices also have an expiry time so that the payee can delete the preimage after some time to free up space.

=== Delivering the payment

You have already learnt that payments start with the payee creating an invoice which includes a Payment Hash to make sure that payments are atomic and that no one on the path of payment channels can withhold the transferred money to their benefit.
In this section we will dive into the ideas and methods that are being used to deliver a payment over the Lightning Network and utilize everything that we have used so far.
We need to introduce one missing protocol of the Lightning Network which is the gossip protocol.

==== The peer 2 peer gossip protocol

As mentioned payment channels can be announced to the network after they are established.
This is happening via a peer to peer gossip protocol.
After opening a channel a node will send out an announcement of the channel via the `channel_announcement` message to its peers.
Every peer can validate the information from the `channel_announcement` message and verify that the funding transaction was indeed confirmed by the Bitcoin network.
Upon verification the node will forward the gossip message to its peers.
In order to avoid spam this will of course only be done if the message was not forwarded by that node yet.
Similarly nodes can be announced via the gossip protocol with the `node_announcment` message.
For this message to be forwarded a node has to have at least one public channel that was announced on the gossip protocol.
This again is a spam measure of the gossip protocol.
Payment channels have various meta data that are useful for other participants of the network to know.
This meta data is mainly used for making routing decisions.
As nodes might want to change the meta data of their channel once in a while this information is shared in a `channel_update` message.
These messages will only be forwarded about 4 times per day for every channel to prevent spam.
The gossip protocol also comes with a variety of queries and tools to initially synchronize a node with the view of the network or to update the nodes view after being offline for a while.
We will investigate the details of the gossip protocol in a later chapter.
For now it is only important to know that it exists and that it is used to share topology information of the Lightning Network with its participants.
This topology information is crucually needed for delivering payments through the network of payment channels.

[NOTE]
====
A major challenge for the participants of the Lightning Network is that the topology information that is being shared by the gossip protocol is only partial.
For example the capacity of the payment channels is shared on the gossip protocol via the `channel_announcement` message.
However this information is not as useful as the actual distribution of the capacity into a local balance between the two channel partners.
This is obvious as a node can only forward the amount of bitcoin via a particular payment channel that it actually owns within that channel.
While Lightning could have been designed to share balance information of channels and a precise topology this has not been done for two reasons:
1. To protect the privacy of the users and not shout out every financial transaction and payment that is being conducted.
2. To be able to scale the amount of payments that can be conducted with the Lightning Network. Remember the Lightning Network was created in the first place because notifying every participant about every payment does not scale well. Thus for simple technical reasons the Lightning Network cannot be designed in a way that the current balance updates of channels are being shared among participants.
====

==== Finding a path

Payments on the Lightning Network are forwarded along a path of channels from one participant to another one.
Thus a path of payment channels has to be selected.
If we knew the exact channel balances of every channel we would easily be able to compute one of the standard path finding algorithms taught in any computer science program to choose a path for the payment.
This could even be done in a way to optimize the fees that would have to be paid by the payer to the nodes that kindly forward the payment.
However as discussed the balance information of all channels is and cannot be available to all participants of the network.
Thus we need to have some path finding strategy.
This strategy relates closely to the routing algorithm that is used.
As we will see in the next section on the Lightning Network we use a source based onion routing protocol for routing payments.
This means in particular that the sender of the payment has to find a path through the network.
As mentioned before with only partial information about the network topology this is a real challenge and active research is still being conducted into optimizing this part of the Lightning Network implementations.
The fact that the pathfinding problem is not fully solved for the case of the Lightning Network is a major point of criticism towards the technology.
On the good side we emphasize that the certainly not optimal strategy - that is currently being used - of probing paths until one has enough liquidity to forward the payment works still rather well.
Also it is worthwhile to note that this probing is done by the LN node or wallet and is not directly seen by the user of the software.
The user might only realize that probing is taking place if the payment is not going though instantly as it usually would.

[NOTE]
====
On the Internet we use the internet protocol and the IP forwarding algorithm to forward internet packages from the sender to the destination.
While the TCP/IP protocol stack allows reliable communication by resending packages that are not acknowledged this mechanism could not be reused directly in the Lightning Network.
A payment that is not being forwarded would effectively mean that the money was stolen by a router and the sender cannot just send out another payment.
While the routing protocol together with the Border Gateway Protocol which are used for data and information transport on the internet have the really nice property of allowing the internet hosts to collaboratively find a path for the information through the internet, we cannot reuse and adopt this protocol for forwarding payments on the Lightning Network.
====

Of course path finding is trivial if we want to pay our direct channel partner and we have enough balance on our side of the channel to do so.
In all other cases information from the gossip protocol is used to help with path finding.
This includes which public payment channels are connecting nodes, which capacity the channels have and what fee policies the owners require.

==== Onion routing

If the sending node of a payment has selected a path that is supposed to be used to make the payment, the Lightning Network uses an onion routing scheme similar to the famous TOR-network.
The routing scheme is called the SPHINX mixformat and will be explained in detail in a later chapter.
For now we want to focus on its properties for the transport of payments which we also call onions.

1. The most important property is that a routing node can only see on which channels it received an onion and on which channel to set up an HTLCs and thus to which peer to forward the onion. This means that no routing node can know who initiated the payment and for whom the payment is supposed to be. The exception of course would be if the node is the recipient. In that case it would know that it was the final destination.
2. The onions are small enough to fit into a single TCP/IP package and actually even a link layer frame. This will make traffic analysis for intruding the privacy of the payments almost impossible.
3. The onions are constructed in a way that they will always have the same length independent of the position of the processing node along the path.
4. Onions can have up to 20 hops included allowing for sufficiently long paths.
5. The encryption of the onion for every hop uses different ephemeral encryption keys with every single onion. Should a key (in particular the private key of the public node key) leak at some point in time an attacker who collected onions cannot decrypt the other onions that have been stored.
6. Errors can be sent back from the erring node in an encrypted way to the original sender. This is particularly useful as we have seen that Lightning nodes who initiate the onions select a path without knowing whether every node has enough liquidity along their channels to forward the payment.

As mentioned we will discuss the details of the Onion Format later but we note already that the `Payment Hash`, while needed to set up the HTLCs for the payment, is not transported within the onions.
The `Payment Hash` is rather included in the Lightning Message that also transports the onion.

==== Payment Forwarding Algorithm

* receive an incoming HTLC
* forward an HTLC
* BOLT 02 channel update protocol
* sending back errors

=== Missing bits
From a computer science perspective the Lightning Network protocol is mainly a peer to peer protocol between its participants.
All communication between participants is sent via so called Lightning Messages.
Most importantly communication is needed to open and close payment channels, to send and receive onions, to set up and settle or fail HTLCs and for exchanging gossip information.
The Lightning messages are sent in an encrypted way after a peer connection has been established.
Establishing the peer connection follows a cryptographic handshake following the Noise Protocol Framework.
The Noise Protocol Framework is a collection of templates for cryptographic handshakes and is also used by WhatsApp and Wireguard.
Using the Noise Protocol Framework makes sure that every message that is sent via Lightning is encrypted and authenticated.
This makes development a little bit tricky as one cannot easily monitor one's own traffic on a tool like wireshark for debugging. footnote:[Luckily tools exist to make developer's lives easier: https://github.com/nayutaco/lightning-dissector]

=== some thoughts that were supposed to be at other places but didn't make it to the text yet
* network of payment channels
* different scope of the network
** global path finding (entire knowledge of the network necessary)
** multihop routing (onion necessary only a subset of nodes involved)
** locally setting up and settling HTLCs (only peers involved)

=== Thoughts about Trust
As long as a person follows the protocol and has their node secured, there is no principle risk of losing funds when participating in the Lightning Network.
However there is the risk of paying fees when opening a channel.
Any risk should come with a potential reward.
In our case the reward is that she can send and receive payments of bitcoin on the Lightning Network at any time and that she can earn bitcoin by forwarding other payments.
Alice decides that the reward is enough for her to take on the fee risk.
Since Bob can directly close the channel, which costs fees paid by Alice, she will need to have a little trust in Bob.

=== Comparison with Bitcoin

While the Lightning Network is built on top of Bitcoin, and inherits many of its features and properties, there are important differences that users of both need to be aware of.

==== Selecting Outputs vs Finding a Path

In order to make a payment on the Bitcoin network, a sender needs to consume one or more Unspent Transaction Outputs (UTXOs).
If a user has multiple UTXOs, they need to select which one to send.
For instance, a user making a payment of 1 BTC can use a single output with value 1 BTC, two outputs with value 0.25 BTC and 0.75 BTC, or four outputs with value 0.25 BTC each.

On Lightning, payments do not require inputs to be consumed but rather for the channel balance to be updated.
This is done by finding a path of channels with sufficient capacity from the sender to the receiver.
As many paths may exist, the choice of path to the Lightning Network payer is somewhat analogous to the choice of UTXO to the Bitcoin payer.

==== Change Outputs on Bitcoin vs No Change on Lightning

In order to make a payment on the Bitcoin network, a sender needs to consume one or more Unspent Transaction Outputs (UTXOs).
The entire UTXO needs to be spent, so if a user wishes to spend 0.8 BTC, but only has a 1 BTC UTXO, then they need to send 0.8 BTC to the receiver, and 0.2 BTC back to themselves.
This 0.2 BTC creates a new UTXO called a 'change output'.

On Lightning, the UTXO is consumed during the Funding Transaction, which leads to the creation of a channel.
Once the bitcoin is locked within that channel, portions of it can be sent back and forth within the channel, without the need to create any change.
This is because the channel partners simply update the channel balance, and only create a new UTXO when the channel is eventually closed using the Bitcoin network.

==== Mining Fees vs Routing Fees

On the Bitcoin network, users pay fees to miners to have their transactions included in a block.
These fees are paid to the miners who mine that particular block, and are based on the _size_ of the transaction in _bytes_ that the transaction is using in a block, as well as how quickly the user wants that transaction mined.
As miners will typically mine the most profitable transactions first, a user who wants their transaction mined immediately will pay a _higher_ fee-per-byte, while a user who is not in a hurry will pay a _lower_ fee-per-byte.

On the Lightning Network, users pay fees to other users to route payments through their channels.
In order to route a payment, a routing user will have to move the funds in two or more channels they own, as well as transmit the data for the sender's payment.
Typically, the routing user will charge the sender based on the _value_ of the payment, as well as setting their own fees they require to route the payment.
Higher value payments will thus cost more to route, and a market for capacity will exist, where different users will charge different fees for routing through their channels.

==== Varying Fees Depending Traffic vs Announced Fees

On the Bitcoin network, miners are profit-seeking, and so will typically include as many transactions in a block as possible, while staying within the block size limit (actually, a modified form called the block weight limit).
If there are more transactions in the queue (called the mempool) than can fit in a block, they will begin by mining the transactions that pay the highest fees per byte (highest fee per weight).
Thus, if there are many transactions in the queue, users will have to pay a higher fee to be included in the next block, or they will have to wait until there are fewer transactions in the queue.
This naturally leads to the concept of 'traffic' and the creation of a fee market where users pay based on how urgently they need their transaction included in the next block.

On the Lightning Network, traffic does not exist since users are not competing for block space outside of the Funding or Closing transactions.
Instead, they are paying fees to the users routing their payments, and different routers will charge different fees for routing through their channels.
Naturally, routers who are charging lower fees for the same capacity will be more attractive to route through.
Thus a fee market exists where routers are in competition with each other over the fees they charge to route payments through their channels.

==== Public Transactions on the Blockchain vs Secret payments

On the Bitcoin network, every payment is publicly visible on the ledger.
While the addresses involved are pseudonymous and are not typically tied to identity, they will still be collected and validated by every other user on the network.

Lightning Network payments, on the other hand, are completely private.
Typically only the sender and the receiver will be fully aware of the source, destination, and amount of bitcoin transacted in a particular transaction.
However, it is not required for the receiver to even know the source of the payment.
As payments are onion-routed, the users who route the payment will only be aware of the amount of the payment, but not the source or the destination.
This is because the security model of Lightning only requires users to be aware of the balances in their channels.
They are not required to be aware of the state of the entire network.

==== Waiting for Confirmations vs Instant settlement

On the Bitcoin network, transactions are only 'settled' once they have been included in a block, in which case they are 'confirmed' in that block.
As more blocks are mined on the Bitcoin network after this, the transaction acquires more 'confirmations' and is considered more secure.

On the Lightning Network, confirmation only matters for opening and closing channels.
Once a Funding Transaction has reached a suitable number of confirmations (e.g. 3), the channel partners consider the channel open.
As the bitcoin in the channel is secured by the smart contract that manages that channel, payments settle instantly once received by the receiver and are not reversible.
When the channel is closed, a transaction will be made on the Bitcoin network and, only once that transaction is confirmed will the channel be considered closed.

==== Sending arbitrary amounts vs capacity restrictions

On the Bitcoin network, a user can send any amount of bitcoin that they own to another user, without capacity restrictions.

On the Lightning Network, a user can only send as much bitcoin as currently exists on their side of a particular channel.
For instance, if a user owns one channel with 0.4 BTC on their side, and another channel with 0.2 BTC on their side, then the maximum they can send with one payment is 0.4 BTC.
This is true regardless of how much bitcoin the user currently has in their Bitcoin wallet.
At the time of writing, Atomic Multi-Path Payments (AMPs) are in development, which, in the above example, would allow the user to combine their 0.4 BTC and 0.2 BTC channels to be able to send a maximum of 0.6 BTC with one payment.

==== Using the Blockchain as a Ledger vs Using the Blockchain as a Court system

On the Bitcoin Network, every transaction is eventually recorded in a block on the blockchain.
The blockchain thus forms of a complete history of every transaction since Bitcoin's creation, and a way to fully audit every bitcoin in existence.
Once a transaction is included in the blockchain it is final.
Thus no disputes can arise and it is unambiguous how much bitcoin is controlled by a particular address at a particular point in the blockchain.
(The only possible dispute is if the blockchain forks into two or more different blockchains)

On the Lightning Network, the balance in a channel at a particular time is known only to the two channel partners, and is only made visible to the rest of the network when the channel is closed.
When the channel is closed, the final balance of the channel is submitted to the Bitcoin blockchain, and each partner receives their share of the bitcoin in that channel.
For instance: if the opening balance was 1 BTC to Alice, and Alice made a payment of 0.3 BTC to Bob, then the final balance of the channel is 0.7 BTC to Alice and 0.3 BTC to Bob.
If Alice tries to cheat by submitting the opening state of the channel to the Bitcoin blockchain, with 1 BTC to Alice and 0 BTC to Bob, then Bob can retaliate by submitting the true final state of the channel, as well as penalty transaction that gives him all Bitcoin in the channel.
In this context, the Bitcoin blockchain acts as a court system; recording the initial and final balances of each channel, and approving penalties if one of the parties tries to cheat.