Development/how-to's add a simple doc about remote kernel debugging.

source/development/how-tos/kernel_debugging.rst

@@ -0,0 +1,109 @@
+==============================
+Remote debugging the kernel
+==============================
+
+FreeBSD/HardenedBSD supports remote debugging using a serial interface.
+Since most virtual solutions support serial interfaces it can be quite convenient to deploy a kernel and start a debug session
+on another machine.
+
+The setup assumes two (virtual) machines with a serial (rs232) connection in between.
+
+.. blockdiag::
+    :scale: 100%
+
+    diagram init {
+        build_vm [label = "Build VM"];
+        test_vm [label = "Test VM"];
+        build_vm -> test_vm;
+    }
+
+
+
+------------------------------
+Configure and build a kernel
+------------------------------
+
+Use our toolchain described in detail `here <https://github.com/opnsense/tools>`__ and add the following options
+to your :code:`SMP` file in :code:`/usr/tools/config/[VERSION]/SMP`
+
+
+.. code-block:: c
+
+    #nomakeoptions DEBUG
+    options        GDB                     # Support remote GDB.
+    makeoptions    DEBUG=-g
+
+
+Next clean and build a kernel
+
+.. code-block:: sh
+
+    make clean-obj,kernel kernel
+
+
+-------------------------------------
+Install the kernel on test vm
+-------------------------------------
+
+
+With the build finished, we should have a kernel package on the build machine available at the following location:
+
+    :code:`/usr/local/opnsense/build/[Version]/[Architecture]/sets/kernel*.txz`
+
+
+Copy this package to the test vm and install it using our :code:`opnsense-update` command:
+
+.. code-block::
+
+      opnsense-update -ikfr [version] -l /location/from/
+
+
+Where [version] is the version part of the kernel package, such as :code:`19.1.2`
+
+
+-------------------------------------------
+Configure the test vm
+-------------------------------------------
+
+To be able to connect to the test machine using :code:`kgdb`, you need to make sure some settings are set.
+
+
+Edit :code:`/boot/device.hints` and change or add :code:`hint.uart.0.flags` to the following:
+
+.. code-block::
+
+    hint.uart.0.flags="0xc0"
+
+
+Also /boot/loader.conf.local should have a baud-rate configured for the serial device:
+
+.. code-block::
+
+    comconsole_speed="115200"
+
+
+
+-----------------------
+Test your setup
+-----------------------
+
+Login to the test machine and force a debug session in gdb, using the following commands:
+
+.. code-block:: sh
+
+      # sysctl debug.kdb.current=gdb
+      # sysctl debug.kdb.enter=1
+      db> gdb
+      Step to enter the remote GDB backend.
+      db> c (continue)
+
+
+The go to the build machine, make sure gdb is installed (:code:`pkg install gdb`) and go to the directory where
+the debug symbols are and start a session, ask a backtrace :code:`bt` and continue normal operation :code:`c`:
+
+.. code-block: sh:
+
+    # cd /usr/obj/usr/src/sys/SMP/
+    # kgdb -r /dev/cuau0 ./kernel.debug
+    (kgdb) bt
+    (kgdb) c