# Tutorial 16 - Virtual Memory Part 4: Higher-Half Kernel ## tl;dr - The time has come: We map and run the kernel from the top of the 64 bit virtual address space! 🥳 ## Table of Contents - [Introduction](#introduction) - [Implementation](#implementation) - [Linking Changes](#linking-changes) - [Position-Independent Boot Code](#position-independent-boot-code) - [Test it](#test-it) - [Diff to previous](#diff-to-previous) ## Introduction A long time in the making, in this tutorial we finally map the kernel to the most significant area (alternatively: higher-half) of the 64 bit virtual address space. This makes room for future applications to use the whole of the least significant area of the virtual memory space. As has been teased since `tutorial 14`, we will make use of the `AArch64`'s `TTBR1`. Since the kernel's virtual address space size currently is `1 GiB` (defined in `bsp/__board_name__/memory/mmu.rs`), `TTBR1` will cover the range from `0xffff_ffff_ffff_ffff` down to `ffff_ffff_c000_0000` (both inclusive). ## Implementation In `src/memory/mmu.rs`, we extend the `AssociatedTranslationTable` trait with a `TableStartFromTop` associated type: ```rust /// Intended to be implemented for [`AddressSpace`]. pub trait AssociatedTranslationTable { /// A translation table whose address range is: /// /// [u64::MAX, (u64::MAX - AS_SIZE) + 1] type TableStartFromTop; /// A translation table whose address range is: /// /// [AS_SIZE - 1, 0] type TableStartFromBottom; } ``` Architecture specific code in `_arch/aarch64/memory/mmu/translation_table.rs` populates both types now by making use of a new generic that is added to `FixedSizeTranslationTable`, which defines whether the covered address space starts from the top or the bottom: ```rust pub struct FixedSizeTranslationTable { ... ``` ```rust impl memory::mmu::AssociatedTranslationTable for memory::mmu::AddressSpace where [u8; Self::SIZE >> Granule512MiB::SHIFT]: Sized, { type TableStartFromTop = FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, true>; type TableStartFromBottom = FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, false>; } ``` Thanks to this infrastructure, `BSP` Rust code in `bsp/__board_name__/memory/mmu.rs` only needs to change to this newly introduced type in order to switch from lower half to higher half translation tables for the kernel: ```rust type KernelTranslationTable = ::TableStartFromTop; ``` ### Linking Changes In the `kernel.ld` linker script, we define a new symbol `__kernel_virt_start_addr` now, which is the start address of the kernel's virtual address space, calculated as `(u64::MAX - __kernel_virt_addr_space_size) + 1`. Before the first section definition, we set the linker script's location counter to this address: ```ld.s SECTIONS { . = __kernel_virt_start_addr; ASSERT((. & PAGE_MASK) == 0, "Start of address space is not page aligned") /*********************************************************************************************** * Code + RO Data + Global Offset Table ***********************************************************************************************/ ``` Since we are not identity mapping anymore, we start to make use of the `AT` keyword in the output section specification: ```ld.s /* The physical address at which the the kernel binary will be loaded by the Raspberry's firmware */ __rpi_phys_binary_load_addr = 0x80000; /* omitted */ SECTIONS { . = __kernel_virt_start_addr; /* omitted */ __code_start = .; .text : AT(__rpi_phys_binary_load_addr) ``` This will manifest in the kernel ELF `segment` attributes, as can be inspected using the `make readelf` command: ```console $ make readelf Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flags Align LOAD 0x0000000000010000 0xffffffffc0000000 0x0000000000080000 0x000000000000cb08 0x000000000000cb08 R E 0x10000 LOAD 0x0000000000020000 0xffffffffc0010000 0x0000000000090000 0x0000000000030dc0 0x0000000000030de0 RW 0x10000 LOAD 0x0000000000060000 0xffffffffc0860000 0x0000000000000000 0x0000000000000000 0x0000000000080000 RW 0x10000 Section to Segment mapping: Segment Sections... 00 .text .rodata 01 .data .bss 02 .boot_core_stack ``` As you can see, `VirtAddr` and `PhysAddr` are different now, as compared to all the previous tutorials where they were identical. This information from the `ELF` file will eventually be parsed by the `translation table tool` and incorporated when compiling the precomputed translation tables. You might have noticed that `.text .rodata` and `.boot_core_stack` exchanged places as compared to previous tutorials. The reason this was done is that with a remapped kernel, this is trivial to do without affecting the physical layout. This allows us to place an unmapped `guard page` between the `boot core stack` and the `mmio remap region` in the VA space, which nicely protects the kernel from stack overflows now: ```ld.s /*********************************************************************************************** * MMIO Remap Reserved ***********************************************************************************************/ __mmio_remap_start = .; . += 8 * 1024 * 1024; __mmio_remap_end_exclusive = .; ASSERT((. & PAGE_MASK) == 0, "MMIO remap reservation is not page aligned") /*********************************************************************************************** * Guard Page ***********************************************************************************************/ . += PAGE_SIZE; /*********************************************************************************************** * Boot Core Stack ***********************************************************************************************/ .boot_core_stack (NOLOAD) : AT(__rpi_phys_dram_start_addr) { __boot_core_stack_start = .; /* ^ */ /* | stack */ . += __rpi_phys_binary_load_addr; /* | growth */ /* | direction */ __boot_core_stack_end_exclusive = .; /* | */ } :segment_boot_core_stack ASSERT((. & PAGE_MASK) == 0, "End of boot core stack is not page aligned") ``` Changes in the `_arch` `MMU` driver are minimal, and mostly concerned with configuring `TCR_EL1` for use with `TTBR1_EL1` now. And of course, setting `TTBR1_EL1` in `fn enable_mmu_and_caching(...)`. ### Position-Independent Boot Code Remember all the fuss that we made about `position-independent code` that will be needed until the `MMU` is enabled. Let's quickly check what it means for us in reality now: In `_arch/aarch64/cpu/boot.rs`, we turn on the `MMU` just before returning from `EL2` to `EL1`. So by the time the CPU enters `EL1`, virtual memory will be active, and the CPU must therefore use the new higher-half `virtual addresses` for everything it does. Specifically, this means the address from which the CPU should execute upon entering `EL1` (function `kernel_init()`) must be a valid _virtual address_, same as the stack pointer's address. Both of them are programmed in function `fn prepare_el2_to_el1_transition(...)`, so we must ensure now that _link-time_ addresses are used here. For this reason, retrieval of these addresses happens in `assembly` in `boot.s`, where we can explicitly enforce generation of **absolute** addresses: ```asm // Load the _absolute_ addresses of the following symbols. Since the kernel is linked at // the top of the 64 bit address space, these are effectively virtual addresses. ADR_ABS x1, __boot_core_stack_end_exclusive ADR_ABS x2, kernel_init ``` Both values are forwarded to the Rust entry point function `_start_rust()`, which in turn forwards them to `fn prepare_el2_to_el1_transition(...)`. One more thing to consider is that we keep on programming the boot core's stack address for `EL2` using an address that is calculated `PC-relative`, because all the `EL2` code will still run while virtual memory _is disabled_. As such, we need the "physical" address of the stack, so to speak. The previous tutorial also explained that it is not easily possible to compile select files using `-fpic` in `Rust`. Still, we are doing some function calls in `Rust` before virtual memory is enabled, so _theoretically_, there is room for failure. However, branches to local code in `AArch64` are usually generated PC-relative. So it is a small risk worth taking. Should it still fail someday, at least our automated CI pipeline would give notice when the tests start to fail. ## Test it That's it! We are ready for a higher-half kernel now. Power up your Raspberrys and marvel at those beautiful (virtual) addresses: Raspberry Pi 3: ```console $ make chainboot [...] Precomputing kernel translation tables and patching kernel ELF ------------------------------------------------------------------------------------ Sections Virt Start Addr Phys Start Addr Size Attr ------------------------------------------------------------------------------------ Generating .text .rodata | 0xffff_ffff_c000_0000 | 0x0000_0000_0008_0000 | 64 KiB | C RO X Generating .data .bss | 0xffff_ffff_c001_0000 | 0x0000_0000_0009_0000 | 256 KiB | C RW XN Generating .boot_core_stack | 0xffff_ffff_c086_0000 | 0x0000_0000_0000_0000 | 512 KiB | C RW XN ------------------------------------------------------------------------------------ Patching Kernel table struct at ELF file offset 0x2_0000 Patching Kernel tables physical base address start argument to value 0xb_0000 at ELF file offset 0x1_0088 Finished in 0.14s Minipush 1.0 [MP] ⏳ Waiting for /dev/ttyUSB0 [MP] ✅ Serial connected [MP] 🔌 Please power the target now __ __ _ _ _ _ | \/ (_)_ _ (_) | ___ __ _ __| | | |\/| | | ' \| | |__/ _ \/ _` / _` | |_| |_|_|_||_|_|____\___/\__,_\__,_| Raspberry Pi 3 [ML] Requesting binary [MP] ⏩ Pushing 257 KiB ======================================🦀 100% 128 KiB/s Time: 00:00:02 [ML] Loaded! Executing the payload now [ 2.870248] mingo version 0.16.0 [ 2.870456] Booting on: Raspberry Pi 3 [ 2.870911] MMU online: [ 2.871203] ------------------------------------------------------------------------------------------------------------------------------------------- [ 2.872947] Virtual Physical Size Attr Entity [ 2.874691] ------------------------------------------------------------------------------------------------------------------------------------------- [ 2.876436] 0xffff_ffff_c000_0000..0xffff_ffff_c000_ffff --> 0x00_0008_0000..0x00_0008_ffff | 64 KiB | C RO X | Kernel code and RO data [ 2.878050] 0xffff_ffff_c001_0000..0xffff_ffff_c004_ffff --> 0x00_0009_0000..0x00_000c_ffff | 256 KiB | C RW XN | Kernel data and bss [ 2.879621] 0xffff_ffff_c005_0000..0xffff_ffff_c005_ffff --> 0x00_3f20_0000..0x00_3f20_ffff | 64 KiB | Dev RW XN | BCM PL011 UART [ 2.881137] | BCM GPIO [ 2.882589] 0xffff_ffff_c006_0000..0xffff_ffff_c006_ffff --> 0x00_3f00_0000..0x00_3f00_ffff | 64 KiB | Dev RW XN | BCM Interrupt Controller [ 2.884214] 0xffff_ffff_c086_0000..0xffff_ffff_c08d_ffff --> 0x00_0000_0000..0x00_0007_ffff | 512 KiB | C RW XN | Kernel boot-core stack [ 2.885818] ------------------------------------------------------------------------------------------------------------------------------------------- ``` Raspberry Pi 4: ```console $ BSP=rpi4 make chainboot [...] Precomputing kernel translation tables and patching kernel ELF ------------------------------------------------------------------------------------ Sections Virt Start Addr Phys Start Addr Size Attr ------------------------------------------------------------------------------------ Generating .text .rodata | 0xffff_ffff_c000_0000 | 0x0000_0000_0008_0000 | 64 KiB | C RO X Generating .data .bss | 0xffff_ffff_c001_0000 | 0x0000_0000_0009_0000 | 256 KiB | C RW XN Generating .boot_core_stack | 0xffff_ffff_c086_0000 | 0x0000_0000_0000_0000 | 512 KiB | C RW XN ------------------------------------------------------------------------------------ Patching Kernel table struct at ELF file offset 0x2_0000 Patching Kernel tables physical base address start argument to value 0xb_0000 at ELF file offset 0x1_0080 Finished in 0.13s Minipush 1.0 [MP] ⏳ Waiting for /dev/ttyUSB0 [MP] ✅ Serial connected [MP] 🔌 Please power the target now __ __ _ _ _ _ | \/ (_)_ _ (_) | ___ __ _ __| | | |\/| | | ' \| | |__/ _ \/ _` / _` | |_| |_|_|_||_|_|____\___/\__,_\__,_| Raspberry Pi 4 [ML] Requesting binary [MP] ⏩ Pushing 257 KiB ======================================🦀 100% 128 KiB/s Time: 00:00:02 [ML] Loaded! Executing the payload now [ 2.871960] mingo version 0.16.0 [ 2.871994] Booting on: Raspberry Pi 4 [ 2.872449] MMU online: [ 2.872742] ------------------------------------------------------------------------------------------------------------------------------------------- [ 2.874486] Virtual Physical Size Attr Entity [ 2.876230] ------------------------------------------------------------------------------------------------------------------------------------------- [ 2.877975] 0xffff_ffff_c000_0000..0xffff_ffff_c000_ffff --> 0x00_0008_0000..0x00_0008_ffff | 64 KiB | C RO X | Kernel code and RO data [ 2.879589] 0xffff_ffff_c001_0000..0xffff_ffff_c004_ffff --> 0x00_0009_0000..0x00_000c_ffff | 256 KiB | C RW XN | Kernel data and bss [ 2.881159] 0xffff_ffff_c005_0000..0xffff_ffff_c005_ffff --> 0x00_fe20_0000..0x00_fe20_ffff | 64 KiB | Dev RW XN | BCM PL011 UART [ 2.882676] | BCM GPIO [ 2.884128] 0xffff_ffff_c006_0000..0xffff_ffff_c006_ffff --> 0x00_ff84_0000..0x00_ff84_ffff | 64 KiB | Dev RW XN | GICv2 GICD [ 2.885601] | GICV2 GICC [ 2.887074] 0xffff_ffff_c086_0000..0xffff_ffff_c08d_ffff --> 0x00_0000_0000..0x00_0007_ffff | 512 KiB | C RW XN | Kernel boot-core stack [ 2.888678] ------------------------------------------------------------------------------------------------------------------------------------------- ``` ## Diff to previous ```diff diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/Cargo.toml 16_virtual_mem_part4_higher_half_kernel/kernel/Cargo.toml --- 15_virtual_mem_part3_precomputed_tables/kernel/Cargo.toml +++ 16_virtual_mem_part4_higher_half_kernel/kernel/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "mingo" -version = "0.15.0" +version = "0.16.0" authors = ["Andre Richter "] edition = "2021" diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/cpu/boot.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/cpu/boot.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/cpu/boot.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/cpu/boot.rs @@ -34,7 +34,10 @@ /// - The `bss` section is not initialized yet. The code must not use or reference it in any way. /// - The HW state of EL1 must be prepared in a sound way. #[inline(always)] -unsafe fn prepare_el2_to_el1_transition(phys_boot_core_stack_end_exclusive_addr: u64) { +unsafe fn prepare_el2_to_el1_transition( + virt_boot_core_stack_end_exclusive_addr: u64, + virt_kernel_init_addr: u64, +) { // Enable timer counter registers for EL1. CNTHCTL_EL2.write(CNTHCTL_EL2::EL1PCEN::SET + CNTHCTL_EL2::EL1PCTEN::SET); @@ -57,11 +60,11 @@ ); // Second, let the link register point to kernel_init(). - ELR_EL2.set(crate::kernel_init as *const () as u64); + ELR_EL2.set(virt_kernel_init_addr); // Set up SP_EL1 (stack pointer), which will be used by EL1 once we "return" to it. Since there // are no plans to ever return to EL2, just re-use the same stack. - SP_EL1.set(phys_boot_core_stack_end_exclusive_addr); + SP_EL1.set(virt_boot_core_stack_end_exclusive_addr); } //-------------------------------------------------------------------------------------------------- @@ -78,14 +81,19 @@ #[no_mangle] pub unsafe extern "C" fn _start_rust( phys_kernel_tables_base_addr: u64, - phys_boot_core_stack_end_exclusive_addr: u64, + virt_boot_core_stack_end_exclusive_addr: u64, + virt_kernel_init_addr: u64, ) -> ! { - prepare_el2_to_el1_transition(phys_boot_core_stack_end_exclusive_addr); + prepare_el2_to_el1_transition( + virt_boot_core_stack_end_exclusive_addr, + virt_kernel_init_addr, + ); // Turn on the MMU for EL1. let addr = Address::new(phys_kernel_tables_base_addr as usize); memory::mmu::enable_mmu_and_caching(addr).unwrap(); - // Use `eret` to "return" to EL1. This results in execution of kernel_init() in EL1. + // Use `eret` to "return" to EL1. Since virtual memory will already be enabled, this results in + // execution of kernel_init() in EL1 from its _virtual address_. asm::eret() } diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/cpu/boot.s 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/cpu/boot.s --- 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/cpu/boot.s +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/cpu/boot.s @@ -18,6 +18,18 @@ add \register, \register, #:lo12:\symbol .endm +// Load the address of a symbol into a register, absolute. +// +// # Resources +// +// - https://sourceware.org/binutils/docs-2.36/as/AArch64_002dRelocations.html +.macro ADR_ABS register, symbol + movz \register, #:abs_g3:\symbol + movk \register, #:abs_g2_nc:\symbol + movk \register, #:abs_g1_nc:\symbol + movk \register, #:abs_g0_nc:\symbol +.endm + //-------------------------------------------------------------------------------------------------- // Public Code //-------------------------------------------------------------------------------------------------- @@ -56,19 +68,31 @@ // Load the base address of the kernel's translation tables. ldr x0, PHYS_KERNEL_TABLES_BASE_ADDR // provided by bsp/__board_name__/memory/mmu.rs - // Set the stack pointer. This ensures that any code in EL2 that needs the stack will work. - ADR_REL x1, __boot_core_stack_end_exclusive - mov sp, x1 + // Load the _absolute_ addresses of the following symbols. Since the kernel is linked at + // the top of the 64 bit address space, these are effectively virtual addresses. + ADR_ABS x1, __boot_core_stack_end_exclusive + ADR_ABS x2, kernel_init + + // Load the PC-relative address of the stack and set the stack pointer. + // + // Since _start() is the first function that runs after the firmware has loaded the kernel + // into memory, retrieving this symbol PC-relative returns the "physical" address. + // + // Setting the stack pointer to this value ensures that anything that still runs in EL2, + // until the kernel returns to EL1 with the MMU enabled, works as well. After the return to + // EL1, the virtual address of the stack retrieved above will be used. + ADR_REL x3, __boot_core_stack_end_exclusive + mov sp, x3 // Read the CPU's timer counter frequency and store it in ARCH_TIMER_COUNTER_FREQUENCY. // Abort if the frequency read back as 0. - ADR_REL x2, ARCH_TIMER_COUNTER_FREQUENCY // provided by aarch64/time.rs - mrs x3, CNTFRQ_EL0 - cmp x3, xzr + ADR_REL x4, ARCH_TIMER_COUNTER_FREQUENCY // provided by aarch64/time.rs + mrs x5, CNTFRQ_EL0 + cmp x5, xzr b.eq .L_parking_loop - str w3, [x2] + str w5, [x4] - // Jump to Rust code. x0 and x1 hold the function arguments provided to _start_rust(). + // Jump to Rust code. x0, x1 and x2 hold the function arguments provided to _start_rust(). b _start_rust // Infinitely wait for events (aka "park the core"). diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/memory/mmu/translation_table.rs @@ -136,7 +136,7 @@ /// aligned, so the lvl3 is put first. #[repr(C)] #[repr(align(65536))] -pub struct FixedSizeTranslationTable { +pub struct FixedSizeTranslationTable { /// Page descriptors, covering 64 KiB windows per entry. lvl3: [[PageDescriptor; 8192]; NUM_TABLES], @@ -302,10 +302,19 @@ where [u8; Self::SIZE >> Granule512MiB::SHIFT]: Sized, { - type TableStartFromBottom = FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }>; + type TableStartFromTop = + FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, true>; + + type TableStartFromBottom = + FixedSizeTranslationTable<{ Self::SIZE >> Granule512MiB::SHIFT }, false>; } -impl FixedSizeTranslationTable { +impl + FixedSizeTranslationTable +{ + const START_FROM_TOP_OFFSET: Address = + Address::new((usize::MAX - (Granule512MiB::SIZE * NUM_TABLES)) + 1); + /// Create an instance. #[allow(clippy::assertions_on_constants)] const fn _new(for_precompute: bool) -> Self { @@ -336,9 +345,14 @@ &self, virt_page_addr: PageAddress, ) -> Result<(usize, usize), &'static str> { - let addr = virt_page_addr.into_inner().as_usize(); - let lvl2_index = addr >> Granule512MiB::SHIFT; - let lvl3_index = (addr & Granule512MiB::MASK) >> Granule64KiB::SHIFT; + let mut addr = virt_page_addr.into_inner(); + + if START_FROM_TOP { + addr = addr - Self::START_FROM_TOP_OFFSET; + } + + let lvl2_index = addr.as_usize() >> Granule512MiB::SHIFT; + let lvl3_index = (addr.as_usize() & Granule512MiB::MASK) >> Granule64KiB::SHIFT; if lvl2_index > (NUM_TABLES - 1) { return Err("Virtual page is out of bounds of translation table"); @@ -384,8 +398,9 @@ // OS Interface Code //------------------------------------------------------------------------------ -impl memory::mmu::translation_table::interface::TranslationTable - for FixedSizeTranslationTable +impl + memory::mmu::translation_table::interface::TranslationTable + for FixedSizeTranslationTable { fn init(&mut self) -> Result<(), &'static str> { if self.initialized { @@ -479,7 +494,7 @@ //-------------------------------------------------------------------------------------------------- #[cfg(test)] -pub type MinSizeTranslationTable = FixedSizeTranslationTable<1>; +pub type MinSizeTranslationTable = FixedSizeTranslationTable<1, true>; #[cfg(test)] mod tests { diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/memory/mmu.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/memory/mmu.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/src/_arch/aarch64/memory/mmu.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/_arch/aarch64/memory/mmu.rs @@ -66,6 +66,7 @@ impl MemoryManagementUnit { /// Setup function for the MAIR_EL1 register. + #[inline(always)] fn set_up_mair(&self) { // Define the memory types being mapped. MAIR_EL1.write( @@ -79,20 +80,21 @@ } /// Configure various settings of stage 1 of the EL1 translation regime. + #[inline(always)] fn configure_translation_control(&self) { - let t0sz = (64 - bsp::memory::mmu::KernelVirtAddrSpace::SIZE_SHIFT) as u64; + let t1sz = (64 - bsp::memory::mmu::KernelVirtAddrSpace::SIZE_SHIFT) as u64; TCR_EL1.write( - TCR_EL1::TBI0::Used + TCR_EL1::TBI1::Used + TCR_EL1::IPS::Bits_40 - + TCR_EL1::TG0::KiB_64 - + TCR_EL1::SH0::Inner - + TCR_EL1::ORGN0::WriteBack_ReadAlloc_WriteAlloc_Cacheable - + TCR_EL1::IRGN0::WriteBack_ReadAlloc_WriteAlloc_Cacheable - + TCR_EL1::EPD0::EnableTTBR0Walks - + TCR_EL1::A1::TTBR0 - + TCR_EL1::T0SZ.val(t0sz) - + TCR_EL1::EPD1::DisableTTBR1Walks, + + TCR_EL1::TG1::KiB_64 + + TCR_EL1::SH1::Inner + + TCR_EL1::ORGN1::WriteBack_ReadAlloc_WriteAlloc_Cacheable + + TCR_EL1::IRGN1::WriteBack_ReadAlloc_WriteAlloc_Cacheable + + TCR_EL1::EPD1::EnableTTBR1Walks + + TCR_EL1::A1::TTBR1 + + TCR_EL1::T1SZ.val(t1sz) + + TCR_EL1::EPD0::DisableTTBR0Walks, ); } } @@ -131,7 +133,7 @@ self.set_up_mair(); // Set the "Translation Table Base Register". - TTBR0_EL1.set_baddr(phys_tables_base_addr.as_usize() as u64); + TTBR1_EL1.set_baddr(phys_tables_base_addr.as_usize() as u64); self.configure_translation_control(); diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/kernel.ld 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/kernel.ld --- 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/kernel.ld +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/kernel.ld @@ -8,6 +8,13 @@ PAGE_SIZE = 64K; PAGE_MASK = PAGE_SIZE - 1; +/* The kernel's virtual address range will be: + * + * [END_ADDRESS_INCLUSIVE, START_ADDRESS] + * [u64::MAX , (u64::MAX - __kernel_virt_addr_space_size) + 1] + */ +__kernel_virt_start_addr = ((0xffffffffffffffff - __kernel_virt_addr_space_size) + 1); + __rpi_phys_dram_start_addr = 0; /* The physical address at which the the kernel binary will be loaded by the Raspberry's firmware */ @@ -26,34 +33,22 @@ */ PHDRS { - segment_boot_core_stack PT_LOAD FLAGS(6); segment_code PT_LOAD FLAGS(5); segment_data PT_LOAD FLAGS(6); + segment_boot_core_stack PT_LOAD FLAGS(6); } SECTIONS { - . = __rpi_phys_dram_start_addr; - - /*********************************************************************************************** - * Boot Core Stack - ***********************************************************************************************/ - .boot_core_stack (NOLOAD) : - { - __boot_core_stack_start = .; /* ^ */ - /* | stack */ - . += __rpi_phys_binary_load_addr; /* | growth */ - /* | direction */ - __boot_core_stack_end_exclusive = .; /* | */ - } :segment_boot_core_stack + . = __kernel_virt_start_addr; - ASSERT((. & PAGE_MASK) == 0, "End of boot core stack is not page aligned") + ASSERT((. & PAGE_MASK) == 0, "Start of address space is not page aligned") /*********************************************************************************************** * Code + RO Data + Global Offset Table ***********************************************************************************************/ __code_start = .; - .text : + .text : AT(__rpi_phys_binary_load_addr) { KEEP(*(.text._start)) *(.text._start_arguments) /* Constants (or statics in Rust speak) read by _start(). */ @@ -91,6 +86,25 @@ . += 8 * 1024 * 1024; __mmio_remap_end_exclusive = .; + ASSERT((. & PAGE_MASK) == 0, "MMIO remap reservation is not page aligned") + + /*********************************************************************************************** + * Guard Page + ***********************************************************************************************/ + . += PAGE_SIZE; + + /*********************************************************************************************** + * Boot Core Stack + ***********************************************************************************************/ + .boot_core_stack (NOLOAD) : AT(__rpi_phys_dram_start_addr) + { + __boot_core_stack_start = .; /* ^ */ + /* | stack */ + . += __rpi_phys_binary_load_addr; /* | growth */ + /* | direction */ + __boot_core_stack_end_exclusive = .; /* | */ + } :segment_boot_core_stack + ASSERT((. & PAGE_MASK) == 0, "End of boot core stack is not page aligned") /*********************************************************************************************** diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/memory/mmu.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/memory/mmu.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/memory/mmu.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/memory/mmu.rs @@ -20,7 +20,7 @@ //-------------------------------------------------------------------------------------------------- type KernelTranslationTable = - ::TableStartFromBottom; + ::TableStartFromTop; //-------------------------------------------------------------------------------------------------- // Public Definitions @@ -153,14 +153,6 @@ /// `translation table tool` and patched into the kernel binary. This function just adds the mapping /// record entries. pub fn kernel_add_mapping_records_for_precomputed() { - let virt_boot_core_stack_region = virt_boot_core_stack_region(); - generic_mmu::kernel_add_mapping_record( - "Kernel boot-core stack", - &virt_boot_core_stack_region, - &kernel_virt_to_phys_region(virt_boot_core_stack_region), - &kernel_page_attributes(virt_boot_core_stack_region.start_page_addr()), - ); - let virt_code_region = virt_code_region(); generic_mmu::kernel_add_mapping_record( "Kernel code and RO data", @@ -176,4 +168,12 @@ &kernel_virt_to_phys_region(virt_data_region), &kernel_page_attributes(virt_data_region.start_page_addr()), ); + + let virt_boot_core_stack_region = virt_boot_core_stack_region(); + generic_mmu::kernel_add_mapping_record( + "Kernel boot-core stack", + &virt_boot_core_stack_region, + &kernel_virt_to_phys_region(virt_boot_core_stack_region), + &kernel_page_attributes(virt_boot_core_stack_region.start_page_addr()), + ); } diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/memory.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/memory.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/src/bsp/raspberrypi/memory.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/bsp/raspberrypi/memory.rs @@ -37,13 +37,7 @@ //! The virtual memory layout is as follows: //! //! +---------------------------------------+ -//! | | boot_core_stack_start @ 0x0 -//! | | ^ -//! | Boot-core Stack | | stack -//! | | | growth -//! | | | direction -//! +---------------------------------------+ -//! | | code_start @ 0x8_0000 == boot_core_stack_end_exclusive +//! | | code_start @ __kernel_virt_start_addr //! | .text | //! | .rodata | //! | .got | @@ -59,6 +53,16 @@ //! | | //! +---------------------------------------+ //! | | mmio_remap_end_exclusive +//! | Unmapped guard page | +//! | | +//! +---------------------------------------+ +//! | | boot_core_stack_start +//! | | ^ +//! | Boot-core Stack | | stack +//! | | | growth +//! | | | direction +//! +---------------------------------------+ +//! | | boot_core_stack_end_exclusive //! | | pub mod mmu; diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/lib.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/lib.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/src/lib.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/lib.rs @@ -157,11 +157,6 @@ ) } -#[cfg(not(test))] -extern "Rust" { - fn kernel_init() -> !; -} - //-------------------------------------------------------------------------------------------------- // Testing //-------------------------------------------------------------------------------------------------- diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu/translation_table.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu/translation_table.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu/translation_table.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu/translation_table.rs @@ -99,9 +99,9 @@ assert_eq!(tables.init(), Ok(())); - let virt_start_page_addr: PageAddress = PageAddress::from(0); - let virt_end_exclusive_page_addr: PageAddress = - virt_start_page_addr.checked_offset(5).unwrap(); + let virt_end_exclusive_page_addr: PageAddress = PageAddress::MAX; + let virt_start_page_addr: PageAddress = + virt_end_exclusive_page_addr.checked_offset(-5).unwrap(); let phys_start_page_addr: PageAddress = PageAddress::from(0); let phys_end_exclusive_page_addr: PageAddress = @@ -124,7 +124,7 @@ ); assert_eq!( - tables.try_page_attributes(virt_start_page_addr.checked_offset(6).unwrap()), + tables.try_page_attributes(virt_start_page_addr.checked_offset(-1).unwrap()), Err("Page marked invalid") ); diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu/types.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu/types.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu/types.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu/types.rs @@ -67,6 +67,11 @@ // PageAddress //------------------------------------------------------------------------------ impl PageAddress { + /// The largest value that can be represented by this type. + pub const MAX: Self = PageAddress { + inner: Address::new(usize::MAX).align_down_page(), + }; + /// Unwraps the value. pub fn into_inner(self) -> Address { self.inner diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu.rs 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/src/memory/mmu.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/src/memory/mmu.rs @@ -66,6 +66,11 @@ pub trait AssociatedTranslationTable { /// A translation table whose address range is: /// + /// [u64::MAX, (u64::MAX - AS_SIZE) + 1] + type TableStartFromTop; + + /// A translation table whose address range is: + /// /// [AS_SIZE - 1, 0] type TableStartFromBottom; } diff -uNr 15_virtual_mem_part3_precomputed_tables/kernel/tests/02_exception_sync_page_fault.rs 16_virtual_mem_part4_higher_half_kernel/kernel/tests/02_exception_sync_page_fault.rs --- 15_virtual_mem_part3_precomputed_tables/kernel/tests/02_exception_sync_page_fault.rs +++ 16_virtual_mem_part4_higher_half_kernel/kernel/tests/02_exception_sync_page_fault.rs @@ -28,8 +28,8 @@ // This line will be printed as the test header. println!("Testing synchronous exception handling by causing a page fault"); - info!("Writing beyond mapped area to address 9 GiB..."); - let big_addr: u64 = 9 * 1024 * 1024 * 1024; + info!("Writing to bottom of address space to address 1 GiB..."); + let big_addr: u64 = 1024 * 1024 * 1024; core::ptr::read_volatile(big_addr as *mut u64); // If execution reaches here, the memory access above did not cause a page fault exception. diff -uNr 15_virtual_mem_part3_precomputed_tables/tools/translation_table_tool/arch.rb 16_virtual_mem_part4_higher_half_kernel/tools/translation_table_tool/arch.rb --- 15_virtual_mem_part3_precomputed_tables/tools/translation_table_tool/arch.rb +++ 16_virtual_mem_part4_higher_half_kernel/tools/translation_table_tool/arch.rb @@ -255,6 +255,8 @@ end def lvl2_lvl3_index_from(addr) + addr -= BSP.kernel_virt_start_addr + lvl2_index = addr >> Granule512MiB::SHIFT lvl3_index = (addr & Granule512MiB::MASK) >> Granule64KiB::SHIFT diff -uNr 15_virtual_mem_part3_precomputed_tables/tools/translation_table_tool/bsp.rb 16_virtual_mem_part4_higher_half_kernel/tools/translation_table_tool/bsp.rb --- 15_virtual_mem_part3_precomputed_tables/tools/translation_table_tool/bsp.rb +++ 16_virtual_mem_part4_higher_half_kernel/tools/translation_table_tool/bsp.rb @@ -6,7 +6,7 @@ # Raspberry Pi 3 + 4 class RaspberryPi - attr_reader :kernel_granule, :kernel_virt_addr_space_size + attr_reader :kernel_granule, :kernel_virt_addr_space_size, :kernel_virt_start_addr MEMORY_SRC = File.read('kernel/src/bsp/raspberrypi/memory.rs').split("\n") @@ -14,6 +14,7 @@ @kernel_granule = Granule64KiB @kernel_virt_addr_space_size = KERNEL_ELF.symbol_value('__kernel_virt_addr_space_size') + @kernel_virt_start_addr = KERNEL_ELF.symbol_value('__kernel_virt_start_addr') @virt_addr_of_kernel_tables = KERNEL_ELF.symbol_value('KERNEL_TABLES') @virt_addr_of_phys_kernel_tables_base_addr = KERNEL_ELF.symbol_value( ```