rust-raspberrypi-OS-tutorials/16_virtual_mem_part4_higher_half_kernel

Tutorial 16 - Virtual Memory Part 4: Higher-Half Kernel

tl;dr

• The time has come: We map and run the kernel from the top of the 64 bit virtual address space! 🥳

Table of Contents

• Introduction
• Implementation
  • Linking Changes
  • Position-Independent Boot Code
• Test it
• Diff to previous

Introduction

A long time in the making, in this tutorial we finally map the kernel to the most significant area (alternatively: higher-half) of the 64 bit virtual address space. This makes room for future applications to use the whole of the least significant area of the virtual memory space.

As has been teased since tutorial 14, we will make use of the AArch64's TTBR1. Since the kernel's virtual address space size currently is 1 GiB (defined in bsp/__board_name__/memory/mmu.rs), TTBR1 will cover the range from 0xffff_ffff_ffff_ffff down to ffff_ffff_c000_0000 (both inclusive).

Implementation

In src/memory/mmu.rs, we extend the AssociatedTranslationTable trait with a TableStartFromTop associated type:

/// Intended to be implemented for [`AddressSpace`].
pub trait AssociatedTranslationTable {
    /// A translation table whose address range is:
    ///
    /// [u64::MAX, (u64::MAX - AS_SIZE) + 1]
    type TableStartFromTop;

    /// A translation table whose address range is:
    ///
    /// [AS_SIZE - 1, 0]
    type TableStartFromBottom;
}

Architecture specific code in _arch/aarch64/memory/mmu/translation_table.rs populates both types now by making use of a new generic that is added to FixedSizeTranslationTable, which defines whether the covered address space starts from the top or the bottom:

pub struct FixedSizeTranslationTable<const NUM_TABLES: usize, const START_FROM_TOP: bool> {
    ...

impl<const AS_SIZE: usize> memory::mmu::AssociatedTranslationTable
    for memory::mmu::AddressSpace<AS_SIZE>
where
    [u8; Self::SIZE >> Granule512MiB::SHIFT]: Sized,
{
    type TableStartFromTop =
        FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, true>;

    type TableStartFromBottom =
        FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, false>;
}

Thanks to this infrastructure, BSP Rust code in bsp/__board_name__/memory/mmu.rs only needs to change to this newly introduced type in order to switch from lower half to higher half translation tables for the kernel:

type KernelTranslationTable =
    <KernelVirtAddrSpace as AssociatedTranslationTable>::TableStartFromTop;

Linking Changes

In the kernel.ld linker script, we define a new symbol __kernel_virt_start_addr now, which is the start address of the kernel's virtual address space, calculated as (u64::MAX - __kernel_virt_addr_space_size) + 1. Before the first section definition, we set the linker script's location counter to this address:

SECTIONS
{
    . =  __kernel_virt_start_addr;

    ASSERT((. & PAGE_MASK) == 0, "Start of address space is not page aligned")

    /***********************************************************************************************
    * Code + RO Data + Global Offset Table
    ***********************************************************************************************/

Since we are not identity mapping anymore, we start to make use of the AT keyword in the output section specification:

/* The physical address at which the the kernel binary will be loaded by the Raspberry's firmware */
__rpi_phys_binary_load_addr = 0x80000;

/* omitted */

SECTIONS
{
    . =  __kernel_virt_start_addr;

    /* omitted */

    __code_start = .;
    .text : AT(__rpi_phys_binary_load_addr)

This will manifest in the kernel ELF segment attributes, as can be inspected using the make readelf command:

$ make readelf

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000010000 0xffffffffc0000000 0x0000000000080000
                 0x000000000000cb08 0x000000000000cb08  R E    0x10000
  LOAD           0x0000000000020000 0xffffffffc0010000 0x0000000000090000
                 0x0000000000030dc0 0x0000000000030de0  RW     0x10000
  LOAD           0x0000000000060000 0xffffffffc0860000 0x0000000000000000
                 0x0000000000000000 0x0000000000080000  RW     0x10000

 Section to Segment mapping:
  Segment Sections...
   00     .text .rodata
   01     .data .bss
   02     .boot_core_stack

As you can see, VirtAddr and PhysAddr are different now, as compared to all the previous tutorials where they were identical. This information from the ELF file will eventually be parsed by the translation table tool and incorporated when compiling the precomputed translation tables.

You might have noticed that .text .rodata and .boot_core_stack exchanged places as compared to previous tutorials. The reason this was done is that with a remapped kernel, this is trivial to do without affecting the physical layout. This allows us to place an unmapped guard page between the boot core stack and the mmio remap region in the VA space, which nicely protects the kernel from stack overflows now:

/***********************************************************************************************
* MMIO Remap Reserved
***********************************************************************************************/
__mmio_remap_start = .;
. += 8 * 1024 * 1024;
__mmio_remap_end_exclusive = .;

ASSERT((. & PAGE_MASK) == 0, "MMIO remap reservation is not page aligned")

/***********************************************************************************************
* Guard Page
***********************************************************************************************/
. += PAGE_SIZE;

/***********************************************************************************************
* Boot Core Stack
***********************************************************************************************/
.boot_core_stack (NOLOAD) : AT(__rpi_phys_dram_start_addr)
{
    __boot_core_stack_start = .;         /*   ^             */
                                         /*   | stack       */
    . += __rpi_phys_binary_load_addr;    /*   | growth      */
                                         /*   | direction   */
    __boot_core_stack_end_exclusive = .; /*   |             */
} :segment_boot_core_stack

ASSERT((. & PAGE_MASK) == 0, "End of boot core stack is not page aligned")

Changes in the _arch MMU driver are minimal, and mostly concerned with configuring TCR_EL1 for use with TTBR1_EL1 now. And of course, setting TTBR1_EL1 in fn enable_mmu_and_caching(...).

Position-Independent Boot Code

Remember all the fuss that we made about position-independent code that will be needed until the MMU is enabled. Let's quickly check what it means for us in reality now:

In _arch/aarch64/cpu/boot.rs, we turn on the MMU just before returning from EL2 to EL1. So by the time the CPU enters EL1, virtual memory will be active, and the CPU must therefore use the new higher-half virtual addresses for everything it does.

Specifically, this means the address from which the CPU should execute upon entering EL1 (function kernel_init()) must be a valid virtual address, same as the stack pointer's address. Both of them are programmed in function fn prepare_el2_to_el1_transition(...), so we must ensure now that link-time addresses are used here. For this reason, retrieval of these addresses happens in assembly in boot.s, where we can explicitly enforce generation of absolute addresses:

// Load the _absolute_ addresses of the following symbols. Since the kernel is linked at
// the top of the 64 bit address space, these are effectively virtual addresses.
ADR_ABS	x1, __boot_core_stack_end_exclusive
ADR_ABS	x2, kernel_init

Both values are forwarded to the Rust entry point function _start_rust(), which in turn forwards them to fn prepare_el2_to_el1_transition(...).

One more thing to consider is that we keep on programming the boot core's stack address for EL2 using an address that is calculated PC-relative, because all the EL2 code will still run while virtual memory is disabled. As such, we need the "physical" address of the stack, so to speak.

The previous tutorial also explained that it is not easily possible to compile select files using -fpic in Rust. Still, we are doing some function calls in Rust before virtual memory is enabled, so theoretically, there is room for failure. However, branches to local code in AArch64 are usually generated PC-relative. So it is a small risk worth taking. Should it still fail someday, at least our automated CI pipeline would give notice when the tests start to fail.

Test it

That's it! We are ready for a higher-half kernel now. Power up your Raspberrys and marvel at those beautiful (virtual) addresses:

Raspberry Pi 3:

$ make chainboot
[...]

Precomputing kernel translation tables and patching kernel ELF
             ------------------------------------------------------------------------------------
                 Sections          Virt Start Addr         Phys Start Addr       Size      Attr
             ------------------------------------------------------------------------------------
  Generating .text .rodata    | 0xffff_ffff_c000_0000 | 0x0000_0000_0008_0000 |  64 KiB | C RO X
  Generating .data .bss       | 0xffff_ffff_c001_0000 | 0x0000_0000_0009_0000 | 256 KiB | C RW XN
  Generating .boot_core_stack | 0xffff_ffff_c086_0000 | 0x0000_0000_0000_0000 | 512 KiB | C RW XN
             ------------------------------------------------------------------------------------
    Patching Kernel table struct at ELF file offset 0x2_0000
    Patching Kernel tables physical base address start argument to value 0xb_0000 at ELF file offset 0x1_0088
    Finished in 0.14s

Minipush 1.0

[MP] ⏳ Waiting for /dev/ttyUSB0
[MP] ✅ Serial connected
[MP] 🔌 Please power the target now

 __  __ _      _ _                 _
|  \/  (_)_ _ (_) |   ___  __ _ __| |
| |\/| | | ' \| | |__/ _ \/ _` / _` |
|_|  |_|_|_||_|_|____\___/\__,_\__,_|

           Raspberry Pi 3

[ML] Requesting binary
[MP] ⏩ Pushing 257 KiB ======================================🦀 100% 128 KiB/s Time: 00:00:02
[ML] Loaded! Executing the payload now

[    2.870248] mingo version 0.16.0
[    2.870456] Booting on: Raspberry Pi 3
[    2.870911] MMU online:
[    2.871203]       -------------------------------------------------------------------------------------------------------------------------------------------
[    2.872947]                         Virtual                                   Physical               Size       Attr                    Entity
[    2.874691]       -------------------------------------------------------------------------------------------------------------------------------------------
[    2.876436]       0xffff_ffff_c000_0000..0xffff_ffff_c000_ffff --> 0x00_0008_0000..0x00_0008_ffff |  64 KiB | C   RO X  | Kernel code and RO data
[    2.878050]       0xffff_ffff_c001_0000..0xffff_ffff_c004_ffff --> 0x00_0009_0000..0x00_000c_ffff | 256 KiB | C   RW XN | Kernel data and bss
[    2.879621]       0xffff_ffff_c005_0000..0xffff_ffff_c005_ffff --> 0x00_3f20_0000..0x00_3f20_ffff |  64 KiB | Dev RW XN | BCM PL011 UART
[    2.881137]                                                                                                             | BCM GPIO
[    2.882589]       0xffff_ffff_c006_0000..0xffff_ffff_c006_ffff --> 0x00_3f00_0000..0x00_3f00_ffff |  64 KiB | Dev RW XN | BCM Interrupt Controller
[    2.884214]       0xffff_ffff_c086_0000..0xffff_ffff_c08d_ffff --> 0x00_0000_0000..0x00_0007_ffff | 512 KiB | C   RW XN | Kernel boot-core stack
[    2.885818]       -------------------------------------------------------------------------------------------------------------------------------------------

Raspberry Pi 4:

$ BSP=rpi4 make chainboot
[...]

Precomputing kernel translation tables and patching kernel ELF
             ------------------------------------------------------------------------------------
                 Sections          Virt Start Addr         Phys Start Addr       Size      Attr
             ------------------------------------------------------------------------------------
  Generating .text .rodata    | 0xffff_ffff_c000_0000 | 0x0000_0000_0008_0000 |  64 KiB | C RO X
  Generating .data .bss       | 0xffff_ffff_c001_0000 | 0x0000_0000_0009_0000 | 256 KiB | C RW XN
  Generating .boot_core_stack | 0xffff_ffff_c086_0000 | 0x0000_0000_0000_0000 | 512 KiB | C RW XN
             ------------------------------------------------------------------------------------
    Patching Kernel table struct at ELF file offset 0x2_0000
    Patching Kernel tables physical base address start argument to value 0xb_0000 at ELF file offset 0x1_0080
    Finished in 0.13s

Minipush 1.0

[MP] ⏳ Waiting for /dev/ttyUSB0
[MP] ✅ Serial connected
[MP] 🔌 Please power the target now

 __  __ _      _ _                 _
|  \/  (_)_ _ (_) |   ___  __ _ __| |
| |\/| | | ' \| | |__/ _ \/ _` / _` |
|_|  |_|_|_||_|_|____\___/\__,_\__,_|

           Raspberry Pi 4

[ML] Requesting binary
[MP] ⏩ Pushing 257 KiB ======================================🦀 100% 128 KiB/s Time: 00:00:02
[ML] Loaded! Executing the payload now

[    2.871960] mingo version 0.16.0
[    2.871994] Booting on: Raspberry Pi 4
[    2.872449] MMU online:
[    2.872742]       -------------------------------------------------------------------------------------------------------------------------------------------
[    2.874486]                         Virtual                                   Physical               Size       Attr                    Entity
[    2.876230]       -------------------------------------------------------------------------------------------------------------------------------------------
[    2.877975]       0xffff_ffff_c000_0000..0xffff_ffff_c000_ffff --> 0x00_0008_0000..0x00_0008_ffff |  64 KiB | C   RO X  | Kernel code and RO data
[    2.879589]       0xffff_ffff_c001_0000..0xffff_ffff_c004_ffff --> 0x00_0009_0000..0x00_000c_ffff | 256 KiB | C   RW XN | Kernel data and bss
[    2.881159]       0xffff_ffff_c005_0000..0xffff_ffff_c005_ffff --> 0x00_fe20_0000..0x00_fe20_ffff |  64 KiB | Dev RW XN | BCM PL011 UART
[    2.882676]                                                                                                             | BCM GPIO
[    2.884128]       0xffff_ffff_c006_0000..0xffff_ffff_c006_ffff --> 0x00_ff84_0000..0x00_ff84_ffff |  64 KiB | Dev RW XN | GICv2 GICD
[    2.885601]                                                                                                             | GICV2 GICC
[    2.887074]       0xffff_ffff_c086_0000..0xffff_ffff_c08d_ffff --> 0x00_0000_0000..0x00_0007_ffff | 512 KiB | C   RW XN | Kernel boot-core stack
[    2.888678]       -------------------------------------------------------------------------------------------------------------------------------------------

Diff to previous

diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/Cargo.toml 16_virtual_mem_part4_higher_half_kernel/kernel/Cargo.toml
--- 15_virtual_mem_part3_precomputed_tables/kernel/Cargo.toml
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "mingo"
-version = "0.15.0"
+version = "0.16.0"
 authors = ["Andre Richter <andre.o.richter@gmail.com>"]
 edition = "2021"


diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/cpu/boot.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/cpu/boot.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/cpu/boot.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/cpu/boot.rs
@@ -34,7 +34,10 @@
 /// - The `bss` section is not initialized yet. The code must not use or reference it in any way.
 /// - The HW state of EL1 must be prepared in a sound way.
 #[inline(always)]
-unsafe fn prepare_el2_to_el1_transition(phys_boot_core_stack_end_exclusive_addr: u64) {
+unsafe fn prepare_el2_to_el1_transition(
+    virt_boot_core_stack_end_exclusive_addr: u64,
+    virt_kernel_init_addr: u64,
+) {
     // Enable timer counter registers for EL1.
     CNTHCTL_EL2.write(CNTHCTL_EL2::EL1PCEN::SET + CNTHCTL_EL2::EL1PCTEN::SET);

@@ -57,11 +60,11 @@
     );

     // Second, let the link register point to kernel_init().
-    ELR_EL2.set(crate::kernel_init as *const () as u64);
+    ELR_EL2.set(virt_kernel_init_addr);

     // Set up SP_EL1 (stack pointer), which will be used by EL1 once we "return" to it. Since there
     // are no plans to ever return to EL2, just re-use the same stack.
-    SP_EL1.set(phys_boot_core_stack_end_exclusive_addr);
+    SP_EL1.set(virt_boot_core_stack_end_exclusive_addr);
 }

 //--------------------------------------------------------------------------------------------------
@@ -78,14 +81,19 @@
 #[no_mangle]
 pub unsafe extern "C" fn _start_rust(
     phys_kernel_tables_base_addr: u64,
-    phys_boot_core_stack_end_exclusive_addr: u64,
+    virt_boot_core_stack_end_exclusive_addr: u64,
+    virt_kernel_init_addr: u64,
 ) -> ! {
-    prepare_el2_to_el1_transition(phys_boot_core_stack_end_exclusive_addr);
+    prepare_el2_to_el1_transition(
+        virt_boot_core_stack_end_exclusive_addr,
+        virt_kernel_init_addr,
+    );

     // Turn on the MMU for EL1.
     let addr = Address::new(phys_kernel_tables_base_addr as usize);
     memory::mmu::enable_mmu_and_caching(addr).unwrap();

-    // Use `eret` to "return" to EL1. This results in execution of kernel_init() in EL1.
+    // Use `eret` to "return" to EL1. Since virtual memory will already be enabled, this results in
+    // execution of kernel_init() in EL1 from its _virtual address_.
     asm::eret()
 }

diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/cpu/boot.s 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/cpu/boot.s
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/cpu/boot.s
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/cpu/boot.s
@@ -18,6 +18,18 @@
 	add	\register, \register, #:lo12:\symbol
 .endm

+// Load the address of a symbol into a register, absolute.
+//
+// # Resources
+//
+// - https://sourceware.org/binutils/docs-2.36/as/AArch64_002dRelocations.html
+.macro ADR_ABS register, symbol
+	movz	\register, #:abs_g3:\symbol
+	movk	\register, #:abs_g2_nc:\symbol
+	movk	\register, #:abs_g1_nc:\symbol
+	movk	\register, #:abs_g0_nc:\symbol
+.endm
+
 //--------------------------------------------------------------------------------------------------
 // Public Code
 //--------------------------------------------------------------------------------------------------
@@ -56,19 +68,31 @@
 	// Load the base address of the kernel's translation tables.
 	ldr	x0, PHYS_KERNEL_TABLES_BASE_ADDR // provided by bsp/__board_name__/memory/mmu.rs

-	// Set the stack pointer. This ensures that any code in EL2 that needs the stack will work.
-	ADR_REL	x1, __boot_core_stack_end_exclusive
-	mov	sp, x1
+	// Load the _absolute_ addresses of the following symbols. Since the kernel is linked at
+	// the top of the 64 bit address space, these are effectively virtual addresses.
+	ADR_ABS	x1, __boot_core_stack_end_exclusive
+	ADR_ABS	x2, kernel_init
+
+	// Load the PC-relative address of the stack and set the stack pointer.
+	//
+	// Since _start() is the first function that runs after the firmware has loaded the kernel
+	// into memory, retrieving this symbol PC-relative returns the "physical" address.
+	//
+	// Setting the stack pointer to this value ensures that anything that still runs in EL2,
+	// until the kernel returns to EL1 with the MMU enabled, works as well. After the return to
+	// EL1, the virtual address of the stack retrieved above will be used.
+	ADR_REL	x3, __boot_core_stack_end_exclusive
+	mov	sp, x3

 	// Read the CPU's timer counter frequency and store it in ARCH_TIMER_COUNTER_FREQUENCY.
 	// Abort if the frequency read back as 0.
-	ADR_REL	x2, ARCH_TIMER_COUNTER_FREQUENCY // provided by aarch64/time.rs
-	mrs	x3, CNTFRQ_EL0
-	cmp	x3, xzr
+	ADR_REL	x4, ARCH_TIMER_COUNTER_FREQUENCY // provided by aarch64/time.rs
+	mrs	x5, CNTFRQ_EL0
+	cmp	x5, xzr
 	b.eq	.L_parking_loop
-	str	w3, [x2]
+	str	w5, [x4]

-	// Jump to Rust code. x0 and x1 hold the function arguments provided to _start_rust().
+	// Jump to Rust code. x0, x1 and x2 hold the function arguments provided to _start_rust().
 	b	_start_rust

 	// Infinitely wait for events (aka "park the core").

diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs
@@ -136,7 +136,7 @@
 /// aligned, so the lvl3 is put first.
 #[repr(C)]
 #[repr(align(65536))]
-pub struct FixedSizeTranslationTable<const NUM_TABLES: usize> {
+pub struct FixedSizeTranslationTable<const NUM_TABLES: usize, const START_FROM_TOP: bool> {
     /// Page descriptors, covering 64 KiB windows per entry.
     lvl3: [[PageDescriptor; 8192]; NUM_TABLES],

@@ -302,10 +302,19 @@
 where
     [u8; Self::SIZE >> Granule512MiB::SHIFT]: Sized,
 {
-    type TableStartFromBottom = FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }>;
+    type TableStartFromTop =
+        FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, true>;
+
+    type TableStartFromBottom =
+        FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, false>;
 }

-impl<const NUM_TABLES: usize> FixedSizeTranslationTable<NUM_TABLES> {
+impl<const NUM_TABLES: usize, const START_FROM_TOP: bool>
+    FixedSizeTranslationTable<NUM_TABLES, START_FROM_TOP>
+{
+    const START_FROM_TOP_OFFSET: Address<Virtual> =
+        Address::new((usize::MAX - (Granule512MiB::SIZE * NUM_TABLES)) + 1);
+
     /// Create an instance.
     #[allow(clippy::assertions_on_constants)]
     const fn _new(for_precompute: bool) -> Self {
@@ -336,9 +345,14 @@
         &self,
         virt_page_addr: PageAddress<Virtual>,
     ) -> Result<(usize, usize), &'static str> {
-        let addr = virt_page_addr.into_inner().as_usize();
-        let lvl2_index = addr >> Granule512MiB::SHIFT;
-        let lvl3_index = (addr & Granule512MiB::MASK) >> Granule64KiB::SHIFT;
+        let mut addr = virt_page_addr.into_inner();
+
+        if START_FROM_TOP {
+            addr = addr - Self::START_FROM_TOP_OFFSET;
+        }
+
+        let lvl2_index = addr.as_usize() >> Granule512MiB::SHIFT;
+        let lvl3_index = (addr.as_usize() & Granule512MiB::MASK) >> Granule64KiB::SHIFT;

         if lvl2_index > (NUM_TABLES - 1) {
             return Err("Virtual page is out of bounds of translation table");
@@ -384,8 +398,9 @@
 // OS Interface Code
 //------------------------------------------------------------------------------

-impl<const NUM_TABLES: usize> memory::mmu::translation_table::interface::TranslationTable
-    for FixedSizeTranslationTable<NUM_TABLES>
+impl<const NUM_TABLES: usize, const START_FROM_TOP: bool>
+    memory::mmu::translation_table::interface::TranslationTable
+    for FixedSizeTranslationTable<NUM_TABLES, START_FROM_TOP>
 {
     fn init(&mut self) -> Result<(), &'static str> {
         if self.initialized {
@@ -479,7 +494,7 @@
 //--------------------------------------------------------------------------------------------------

 #[cfg(test)]
-pub type MinSizeTranslationTable = FixedSizeTranslationTable<1>;
+pub type MinSizeTranslationTable = FixedSizeTranslationTable<1, true>;

 #[cfg(test)]
 mod tests {

diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/memory/mmu.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/memory/mmu.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/memory/mmu.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/memory/mmu.rs
@@ -66,6 +66,7 @@

 impl MemoryManagementUnit {
     /// Setup function for the MAIR_EL1 register.
+    #[inline(always)]
     fn set_up_mair(&self) {
         // Define the memory types being mapped.
         MAIR_EL1.write(
@@ -79,20 +80,21 @@
     }

     /// Configure various settings of stage 1 of the EL1 translation regime.
+    #[inline(always)]
     fn configure_translation_control(&self) {
-        let t0sz = (64 - bsp::memory::mmu::KernelVirtAddrSpace::SIZE_SHIFT) as u64;
+        let t1sz = (64 - bsp::memory::mmu::KernelVirtAddrSpace::SIZE_SHIFT) as u64;

         TCR_EL1.write(
-            TCR_EL1::TBI0::Used
+            TCR_EL1::TBI1::Used
                 + TCR_EL1::IPS::Bits_40
-                + TCR_EL1::TG0::KiB_64
-                + TCR_EL1::SH0::Inner
-                + TCR_EL1::ORGN0::WriteBack_ReadAlloc_WriteAlloc_Cacheable
-                + TCR_EL1::IRGN0::WriteBack_ReadAlloc_WriteAlloc_Cacheable
-                + TCR_EL1::EPD0::EnableTTBR0Walks
-                + TCR_EL1::A1::TTBR0
-                + TCR_EL1::T0SZ.val(t0sz)
-                + TCR_EL1::EPD1::DisableTTBR1Walks,
+                + TCR_EL1::TG1::KiB_64
+                + TCR_EL1::SH1::Inner
+                + TCR_EL1::ORGN1::WriteBack_ReadAlloc_WriteAlloc_Cacheable
+                + TCR_EL1::IRGN1::WriteBack_ReadAlloc_WriteAlloc_Cacheable
+                + TCR_EL1::EPD1::EnableTTBR1Walks
+                + TCR_EL1::A1::TTBR1
+                + TCR_EL1::T1SZ.val(t1sz)
+                + TCR_EL1::EPD0::DisableTTBR0Walks,
         );
     }
 }
@@ -131,7 +133,7 @@
         self.set_up_mair();

         // Set the "Translation Table Base Register".
-        TTBR0_EL1.set_baddr(phys_tables_base_addr.as_usize() as u64);
+        TTBR1_EL1.set_baddr(phys_tables_base_addr.as_usize() as u64);

         self.configure_translation_control();


diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/kernel.ld 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/kernel.ld
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/kernel.ld
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/kernel.ld
@@ -8,6 +8,13 @@
 PAGE_SIZE = 64K;
 PAGE_MASK = PAGE_SIZE - 1;

+/* The kernel's virtual address range will be:
+ *
+ * [END_ADDRESS_INCLUSIVE, START_ADDRESS]
+ * [u64::MAX             , (u64::MAX - __kernel_virt_addr_space_size) + 1]
+ */
+__kernel_virt_start_addr = ((0xffffffffffffffff - __kernel_virt_addr_space_size) + 1);
+
 __rpi_phys_dram_start_addr = 0;

 /* The physical address at which the the kernel binary will be loaded by the Raspberry's firmware */
@@ -26,34 +33,22 @@
  */
 PHDRS
 {
-    segment_boot_core_stack PT_LOAD FLAGS(6);
     segment_code            PT_LOAD FLAGS(5);
     segment_data            PT_LOAD FLAGS(6);
+    segment_boot_core_stack PT_LOAD FLAGS(6);
 }

 SECTIONS
 {
-    . =  __rpi_phys_dram_start_addr;
-
-    /***********************************************************************************************
-    * Boot Core Stack
-    ***********************************************************************************************/
-    .boot_core_stack (NOLOAD) :
-    {
-        __boot_core_stack_start = .;         /*   ^             */
-                                             /*   | stack       */
-        . += __rpi_phys_binary_load_addr;    /*   | growth      */
-                                             /*   | direction   */
-        __boot_core_stack_end_exclusive = .; /*   |             */
-    } :segment_boot_core_stack
+    . =  __kernel_virt_start_addr;

-    ASSERT((. & PAGE_MASK) == 0, "End of boot core stack is not page aligned")
+    ASSERT((. & PAGE_MASK) == 0, "Start of address space is not page aligned")

     /***********************************************************************************************
     * Code + RO Data + Global Offset Table
     ***********************************************************************************************/
     __code_start = .;
-    .text :
+    .text : AT(__rpi_phys_binary_load_addr)
     {
         KEEP(*(.text._start))
         *(.text._start_arguments) /* Constants (or statics in Rust speak) read by _start(). */
@@ -91,6 +86,25 @@
     . += 8 * 1024 * 1024;
     __mmio_remap_end_exclusive = .;

+    ASSERT((. & PAGE_MASK) == 0, "MMIO remap reservation is not page aligned")
+
+    /***********************************************************************************************
+    * Guard Page
+    ***********************************************************************************************/
+    . += PAGE_SIZE;
+
+    /***********************************************************************************************
+    * Boot Core Stack
+    ***********************************************************************************************/
+    .boot_core_stack (NOLOAD) : AT(__rpi_phys_dram_start_addr)
+    {
+        __boot_core_stack_start = .;         /*   ^             */
+                                             /*   | stack       */
+        . += __rpi_phys_binary_load_addr;    /*   | growth      */
+                                             /*   | direction   */
+        __boot_core_stack_end_exclusive = .; /*   |             */
+    } :segment_boot_core_stack
+
     ASSERT((. & PAGE_MASK) == 0, "End of boot core stack is not page aligned")

     /***********************************************************************************************

diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/memory/mmu.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/memory/mmu.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/memory/mmu.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/memory/mmu.rs
@@ -20,7 +20,7 @@
 //--------------------------------------------------------------------------------------------------

 type KernelTranslationTable =
-    <KernelVirtAddrSpace as AssociatedTranslationTable>::TableStartFromBottom;
+    <KernelVirtAddrSpace as AssociatedTranslationTable>::TableStartFromTop;

 //--------------------------------------------------------------------------------------------------
 // Public Definitions
@@ -153,14 +153,6 @@
 /// `translation table tool` and patched into the kernel binary. This function just adds the mapping
 /// record entries.
 pub fn kernel_add_mapping_records_for_precomputed() {
-    let virt_boot_core_stack_region = virt_boot_core_stack_region();
-    generic_mmu::kernel_add_mapping_record(
-        "Kernel boot-core stack",
-        &virt_boot_core_stack_region,
-        &kernel_virt_to_phys_region(virt_boot_core_stack_region),
-        &kernel_page_attributes(virt_boot_core_stack_region.start_page_addr()),
-    );
-
     let virt_code_region = virt_code_region();
     generic_mmu::kernel_add_mapping_record(
         "Kernel code and RO data",
@@ -176,4 +168,12 @@
         &kernel_virt_to_phys_region(virt_data_region),
         &kernel_page_attributes(virt_data_region.start_page_addr()),
     );
+
+    let virt_boot_core_stack_region = virt_boot_core_stack_region();
+    generic_mmu::kernel_add_mapping_record(
+        "Kernel boot-core stack",
+        &virt_boot_core_stack_region,
+        &kernel_virt_to_phys_region(virt_boot_core_stack_region),
+        &kernel_page_attributes(virt_boot_core_stack_region.start_page_addr()),
+    );
 }

diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/memory.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/memory.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/memory.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/memory.rs
@@ -37,13 +37,7 @@
 //! The virtual memory layout is as follows:
 //!
 //! +---------------------------------------+
-//! |                                       | boot_core_stack_start @ 0x0
-//! |                                       |                                ^
-//! | Boot-core Stack                       |                                | stack
-//! |                                       |                                | growth
-//! |                                       |                                | direction
-//! +---------------------------------------+
-//! |                                       | code_start @ 0x8_0000 == boot_core_stack_end_exclusive
+//! |                                       | code_start @ __kernel_virt_start_addr
 //! | .text                                 |
 //! | .rodata                               |
 //! | .got                                  |
@@ -59,6 +53,16 @@
 //! |                                       |
 //! +---------------------------------------+
 //! |                                       |  mmio_remap_end_exclusive
+//! | Unmapped guard page                   |
+//! |                                       |
+//! +---------------------------------------+
+//! |                                       | boot_core_stack_start
+//! |                                       |                                ^
+//! | Boot-core Stack                       |                                | stack
+//! |                                       |                                | growth
+//! |                                       |                                | direction
+//! +---------------------------------------+
+//! |                                       | boot_core_stack_end_exclusive
 //! |                                       |
 pub mod mmu;


diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/lib.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/lib.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/lib.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/lib.rs
@@ -157,11 +157,6 @@
     )
 }

-#[cfg(not(test))]
-extern "Rust" {
-    fn kernel_init() -> !;
-}
-
 //--------------------------------------------------------------------------------------------------
 // Testing
 //--------------------------------------------------------------------------------------------------

diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu/translation_table.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu/translation_table.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu/translation_table.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu/translation_table.rs
@@ -99,9 +99,9 @@

         assert_eq!(tables.init(), Ok(()));

-        let virt_start_page_addr: PageAddress<Virtual> = PageAddress::from(0);
-        let virt_end_exclusive_page_addr: PageAddress<Virtual> =
-            virt_start_page_addr.checked_offset(5).unwrap();
+        let virt_end_exclusive_page_addr: PageAddress<Virtual> = PageAddress::MAX;
+        let virt_start_page_addr: PageAddress<Virtual> =
+            virt_end_exclusive_page_addr.checked_offset(-5).unwrap();

         let phys_start_page_addr: PageAddress<Physical> = PageAddress::from(0);
         let phys_end_exclusive_page_addr: PageAddress<Physical> =
@@ -124,7 +124,7 @@
         );

         assert_eq!(
-            tables.try_page_attributes(virt_start_page_addr.checked_offset(6).unwrap()),
+            tables.try_page_attributes(virt_start_page_addr.checked_offset(-1).unwrap()),
             Err("Page marked invalid")
         );


diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu/types.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu/types.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu/types.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu/types.rs
@@ -67,6 +67,11 @@
 // PageAddress
 //------------------------------------------------------------------------------
 impl<ATYPE: AddressType> PageAddress<ATYPE> {
+    /// The largest value that can be represented by this type.
+    pub const MAX: Self = PageAddress {
+        inner: Address::new(usize::MAX).align_down_page(),
+    };
+
     /// Unwraps the value.
     pub fn into_inner(self) -> Address<ATYPE> {
         self.inner

diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu.rs
@@ -66,6 +66,11 @@
 pub trait AssociatedTranslationTable {
     /// A translation table whose address range is:
     ///
+    /// [u64::MAX, (u64::MAX - AS_SIZE) + 1]
+    type TableStartFromTop;
+
+    /// A translation table whose address range is:
+    ///
     /// [AS_SIZE - 1, 0]
     type TableStartFromBottom;
 }

diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/tests/02_exception_sync_page_fault.rs 16_virtual_mem_part4_higher_half_kernel/kernel/tests/02_exception_sync_page_fault.rs
--- 15_virtual_mem_part3_precomputed_tables/kernel/tests/02_exception_sync_page_fault.rs
+++ 16_virtual_mem_part4_higher_half_kernel/kernel/tests/02_exception_sync_page_fault.rs
@@ -28,8 +28,8 @@
     // This line will be printed as the test header.
     println!("Testing synchronous exception handling by causing a page fault");

-    info!("Writing beyond mapped area to address 9 GiB...");
-    let big_addr: u64 = 9 * 1024 * 1024 * 1024;
+    info!("Writing to bottom of address space to address 1 GiB...");
+    let big_addr: u64 = 1024 * 1024 * 1024;
     core::ptr::read_volatile(big_addr as *mut u64);

     // If execution reaches here, the memory access above did not cause a page fault exception.

diff -uNr 15_virtual_mem_part3_precomputed_tables/tools/translation_table_tool/arch.rb 16_virtual_mem_part4_higher_half_kernel/tools/translation_table_tool/arch.rb
--- 15_virtual_mem_part3_precomputed_tables/tools/translation_table_tool/arch.rb
+++ 16_virtual_mem_part4_higher_half_kernel/tools/translation_table_tool/arch.rb
@@ -255,6 +255,8 @@
     end

     def lvl2_lvl3_index_from(addr)
+        addr -= BSP.kernel_virt_start_addr
+
         lvl2_index = addr >> Granule512MiB::SHIFT
         lvl3_index = (addr & Granule512MiB::MASK) >> Granule64KiB::SHIFT


diff -uNr 15_virtual_mem_part3_precomputed_tables/tools/translation_table_tool/bsp.rb 16_virtual_mem_part4_higher_half_kernel/tools/translation_table_tool/bsp.rb
--- 15_virtual_mem_part3_precomputed_tables/tools/translation_table_tool/bsp.rb
+++ 16_virtual_mem_part4_higher_half_kernel/tools/translation_table_tool/bsp.rb
@@ -6,7 +6,7 @@

 # Raspberry Pi 3 + 4
 class RaspberryPi
-    attr_reader :kernel_granule, :kernel_virt_addr_space_size
+    attr_reader :kernel_granule, :kernel_virt_addr_space_size, :kernel_virt_start_addr

     MEMORY_SRC = File.read('kernel/src/bsp/raspberrypi/memory.rs').split("\n")

@@ -14,6 +14,7 @@
         @kernel_granule = Granule64KiB

         @kernel_virt_addr_space_size = KERNEL_ELF.symbol_value('__kernel_virt_addr_space_size')
+        @kernel_virt_start_addr = KERNEL_ELF.symbol_value('__kernel_virt_start_addr')

         @virt_addr_of_kernel_tables = KERNEL_ELF.symbol_value('KERNEL_TABLES')
         @virt_addr_of_phys_kernel_tables_base_addr = KERNEL_ELF.symbol_value(