From c7b6de9b9e19d46a07b65ca3ce573abfc2c75251 Mon Sep 17 00:00:00 2001
From: Stephan de Wit <stephan.de.wit@deciso.com>
Date: Fri, 12 Apr 2024 13:30:28 +0200
Subject: [PATCH] carp: clarify on the concept of a VHID and the CARP
 implementation. Also add the procedure for safely adding a Virtual IP to a
 running CARP cluster

---
 source/manual/firewall_vip.rst | 34 +++++++++++++++++++++++++++++++++-
 source/manual/how-tos/carp.rst | 25 ++++++++++++++++++++++---
 2 files changed, 55 insertions(+), 4 deletions(-)

diff --git a/source/manual/firewall_vip.rst b/source/manual/firewall_vip.rst
index afb9b498..f84cb584 100644
--- a/source/manual/firewall_vip.rst
+++ b/source/manual/firewall_vip.rst
@@ -32,7 +32,7 @@ it will respond to ICMP ping requests and will generate ARP traffic
 (OSI layer 2).
 
 Additionally you can add an alias into an existing CARP group
-(by setting its VHID).
+(by setting its VHID). See the CARP VIP type below for more information.
 
 Usually the subnet mask should match the interfaces or be defined as a single address (/32 or /128).
 
@@ -44,6 +44,12 @@ CARP
 Specifies an address for use in a high availability cluster, acts like a
 regular address when the node is in MASTER state.
 
+A VHID Group number must be specified. The "Select an unassigned VHID" button allows you to
+automatically select an available VHID number. The usual approach to selecting a VHID is to use a different
+number per interface, but this is not a strict requirement, since the underlying protocol only
+requires a VHID to be unique within the broadcast domain of the specified interface. However, to ease
+management and debugging it is recommended to keep a separate VHID per interface.
+
 Internally a custom mac address is generated needed for the protocol.
 More information about CARP can be found in our :doc:`high availability </manual/hacarp>` section.
 
@@ -55,6 +61,32 @@ More information about CARP can be found in our :doc:`high availability </manual
     CARP uses IP protocol number 112 (0x70), to detect priority it will send out advertisements using
     :code:`224.0.0.18` or :code:`FF02::12`.
 
+**Combining CARP virtual IP types with IP aliases**
+
+In cases where there is a need for multiple IP aliases on a single interface which should be shared by a CARP cluster,
+you can assign a single CARP VIP with a specific VHID in combination with regular IP alias types,
+setting the VHID field to the same number as the initial CARP VIP VHID:
+
+- The entire set of configured virtual IP addresses are now considered a single host (VHID).
+- Only this VHID will send out advertisement packets.
+- The set of IP addresses for this VHID are hashed and inserted in the advertisement packets.
+  This hash is compared to the same VHID hash on the peer on reception of CARP advertisements. If they do not match,
+  the peer will assume the master role as the configuration is out of sync.
+
+.. Note::
+    See `Adding multiple CARP IPs <how-tos/carp.html#adding-multiple-carp-ips>`__ for more information and the
+    proper procedure to add IP aliases to a running CARP cluster.
+
+.. Warning::
+    While technically it is possible to assign multiple CARP VIPs on the same interface, but with separate VHIDs,
+    this has no benefit and is not recommended. The CARP traffic and system procedures for failover will increase
+    linearly in noise per virtual IP. Since the primary purpose of CARP is to react to link state changes, a single
+    VHID acting for a single interface is the most efficient way to use the protocol.
+
+.. Tip::
+    If you're debugging a CARP setup, consider raising the CARP system logging verbosity. This can be done by
+    adding the :code:`net.inet.carp.log` with value :code:`2` tunable in System -> Settings -> Tunables.
+    The logs can be seen in System -> Log Files -> General (kernel process) or by using :code:`dmesg`.
 
 ..................
 Proxy ARP
diff --git a/source/manual/how-tos/carp.rst b/source/manual/how-tos/carp.rst
index 69934ec5..ed123e78 100644
--- a/source/manual/how-tos/carp.rst
+++ b/source/manual/how-tos/carp.rst
@@ -282,11 +282,19 @@ want to expose them for NAT or different services running on your
 Firewall, you will also have to add them to your HA setup. 
 Since adding a VHID for every IP would make the CARP traffic very noisy,
 you can also add a new IP Alias and choose the correct VHID where the
-first CARP IP is configured. 
+first CARP IP is configured. See `CARP Virtual IP type <../firewall_vip.html#carp>`__ for more information
+on the concept.
 
 .. Note::
-   IP Alias is not synchronized to slave, be sure to also add it to your
-   second machine.
+   IP Aliases are not synchronized to the backup firewall during a configuration sync,
+   be sure to also add it to your second machine when setting up CARP.
+
+.. Attention::
+    Adding an IP alias to a running CARP system requires some consideration. Since adding a new IP Alias
+    to an existing VHID on a single machine will invalidate the VHID hash for both sides, both machines will
+    react by switching to the master state, triggering a split-brain scenario. To avoid this, CARP must
+    explicitly be disabled on one of the machines before adding the new IP Alias.
+    For an exact procedure, refer to `the example <carp.html#example-adding-a-virtual-ip-to-a-carp-ha-cluster>`__
 
 -----------------------------------
 Example: Updating a CARP HA Cluster
@@ -304,6 +312,17 @@ these steps:
 With these steps you will not lose too many packets and your existing connection will be transferred as well.
 Also note that entering persistent mode survives a reboot.
 
+-------------------------------------------------
+Example: Adding a virtual IP to a CARP HA Cluster
+-------------------------------------------------
+
+- Disable CARP (not maintenance mode) on either the primary or secondary unit. When disabling it on the master,
+  the backup should take over.
+- Add the virtual IP alias to the machine where CARP is disabled.
+- While keeping CARP disabled on this machine, add the same IP alias to the other machine. This may interrupt
+  traffic briefly at worst, but this is acceptable in a failover scenario.
+- Re-enable CARP on the previous machine. Normal operation should resume.
+
 .. _configuring-carp-with-ipv6:
 
 --------------------------