Nikolay GeorgievSoftware Engineer

Case study 05 · Operating system written from scratch in Rust

Malrynth

An independent x86-64 operating system with its own loader and kernel. A Rust UEFI loader hands a validated boot contract to a dependency-free kernel that owns physical memory, and CI boot-tests it under QEMU and OVMF.

PrototypeBoot milestone complete, physical memory merged
Role
Founder and sole developer
Timeline
Aug 2026 to present
Context
Long-term personal systems project, private repository
Layers
System software
Links
Private repository

01Overview

Malrynth is my long-term operating-system project, an independent x86-64 OS with its own loader and kernel, written in Rust. It is not a Linux distribution and does not run on Linux underneath.

Why it exists. To understand the machine all the way down, and to design a platform on my own terms, with capability-based security, local accounts as first-class citizens and any AI integration optional and never in charge of security.

Where it is. Early, and stated plainly. The first milestone boots and reports diagnostics, and the kernel now owns physical memory. Paging, interrupts, scheduling and userspace are the next milestones.

Specification

Language
Rust 1.97 stable, pinned; no nightly features
Targets
x86_64-unknown-uefi (loader), x86_64-unknown-none (kernel)
Crates
3: boot ABI, UEFI loader, kernel; the kernel has zero dependencies
Loader
42,496-byte PE32+ application, byte-reproducible
Handoff
BootInfo v1: repr(C), magic, version and size checked
Memory
Canonical ownership map and a two-bitmap frame allocator
Verification
Repeated QEMU and OVMF boots, panic path, reproducibility
Tests
22 Rust unit tests, 67 tooling tests, 2 CI workflows
Design
6 ADRs, an M0 to M7 roadmap, 107 design documents
  • Not a Linux distribution. Malrynth has its own loader and kernel, and Linux is only a development host.
  • The kernel trusts nothing the firmware hands it. The memory map is re-validated and canonicalised before use.
  • A boot passes only on an exact serial marker. CI also boots a build that must panic, and fault-injection runs are documented.
  • Builds are byte-reproducible across paths and hosts, and a test enforces it.

02What I built

Boot

  • malboot, a UEFI application with its own bounds-checked ELF64 loader that accepts only a static x86-64 executable.
  • The kernel is placed at an exact physical address. If the firmware cannot provide it, the loader refuses rather than relocating.
  • Framebuffer and ACPI RSDP discovery, handoff memory allocated in advance, and a memory-map pre-flight with headroom before ExitBootServices.

Kernel

  • A no_std, no_main kernel with zero external crates, a custom linker script and separate executable, read-only and writable segments.
  • Handoff validation (pointer, magic, version, size, load address) and a polled 16550 serial console.
  • Physical memory ownership. The firmware map is treated as untrusted, canonicalised with explicit reservations and managed by a two-bitmap frame allocator that checks itself on every boot.
  • A panic handler with a validated frame-pointer backtrace, symbolised on the host so no symbol parser runs in the kernel.

Tooling and verification

  • Build, run and test tooling for Linux (Bash) and native Windows (Python), with a contract test that fails when their compiler flags drift apart.
  • QEMU and OVMF boot tests that pass only on an exact success marker, plus panic-path and byte-reproducibility checks, all in CI.
  • Secret scanning, SHA-pinned CI actions, a committed lockfile and a ledger for every third-party dependency.

Architecture and governance

  • A constitution, 6 ADRs that must record rejected alternatives, an M0 to M7 roadmap and a threat model.
  • Design specifications for capabilities, IPC, drivers, storage, UI and compatibility, each labelled with its real status.

03Architecture

Today the system has three layers, firmware, a Rust UEFI loader and a kernel that owns physical memory. Everything above that (paging, interrupts, scheduling, userspace, drivers, desktop) is specified in milestones and not built yet.

Fig. 05.1

Boot path, and what comes next

Firmware and loader

  • ExternalUEFI firmware

    OVMF on QEMU q35

    • malboot

  • Stagemalboot

    Rust UEFI app, own ELF64 parser

    • Place kernel

  • StagePlace kernel

    exact physical address; refuse, never relocate

    • ExitBootServices

  • StageExitBootServices

    map pre-flighted with headroom; no way back

    • BootInfo v1

Handoff and kernel

  • StoreBootInfo v1

    repr(C), magic, version, size, memory regions

    • Kernel _start · rdi

  • StageKernel _start

    no_std, zero crates; validates the handoff

    • Physical memory (M1A)
    • COM1 serial · diagnostics

  • ServicePhysical memory (M1A)

    untrusted map canonicalised, bitmap frame allocator

    • Page tables, higher half

  • HardwareCOM1 serial

    polled 16550 UART

    • Boot test harness · markers

  • ServiceBoot test harness

    exact success marker, repeated boots in CI

Next milestones

  • Next milestonePage tables, higher half

    M1B: W^X, guarded stacks, heap

    • Interrupts and time

  • PlannedInterrupts and time

    M1C-F: IDT, APIC, timer, SMP

    • Scheduler, userspace

  • PlannedScheduler, userspace

    M2+: processes, syscalls, capabilities, IPC

What exists today: UEFI firmware loads malboot, which validates and places the kernel, prepares the handoff and exits boot services; the kernel validates the handoff, canonicalises the untrusted memory map and takes ownership of physical memory. The boot test passes only on an exact serial marker. The dashed row is the milestone plan and is not built yet.

04Hard problems

  1. Failing well after ExitBootServices

    Problem

    After ExitBootServices there is no console, allocator or firmware to return to, so a late failure cannot even be reported.

    What I did

    The loader allocates all handoff memory in advance and checks that the final memory map will fit, with headroom, while it can still print a reason. After the exit it halts rather than returning to firmware that no longer exists.

  2. Trusting the memory map

    Problem

    A firmware memory map can overlap, arrive unsorted or describe memory that must never be handed out.

    What I did

    The kernel recomputes every range with checked arithmetic, rounds usable RAM inward and reservations outward, lets reservations win overlaps, and sorts and merges the result without an allocator, using fixed scratch arrays.

  3. Sparse memory and high MMIO

    Problem

    Holes in physical memory and device ranges far above RAM would inflate the allocator's bitmap and its accounting.

    What I did

    Frame numbers are compacted across holes, and host tests cover sparse RAM and high MMIO explicitly.

  4. Reproducible binaries across hosts

    Problem

    The loader embedded absolute paths that path remapping could not reach, so builds differed between machines and directories.

    What I did

    Loader debug info was dropped and deterministic linker options were set. A structural PE differ found the remaining cross-host differences, and a test now enforces identical hashes.

  5. Backtraces you can trust

    Problem

    A corrupted stack can produce a backtrace that looks plausible but is false.

    What I did

    Every frame is validated for alignment, direction and distance, the walk reports why it stopped, and a dedicated panic build proves the path in CI.

05Decisions

Its own kernel, never Linux underneath.
Linux is a development host and a future compatibility target, never the kernel Malrynth runs on.
Use uefi-rs, but only in the loader.
A wrong firmware structure offset silently corrupts boot. The dependency stops running at ExitBootServices, so it never becomes part of the kernel's trusted base.
Stable, pinned Rust with no nightly features.
The build should not depend on unstable compiler features, and both targets ship precompiled core libraries.
Refuse an oversized memory map rather than truncate it.
A truncated map could let the allocator hand out firmware or device memory.
Symbolise panics on the host.
It keeps a symbol parser, which handles untrusted input, out of ring 0.
The next milestone's direct map will cover classified RAM only.
The boot contract carries no cacheability attributes, so mapping all physical space could mis-cache device memory.

06Status

Works today

  • Boots under QEMU and OVMF through loader, handoff, kernel entry, serial diagnostics and physical memory ownership.
  • CI runs formatting, lints, 22 unit tests, repeated boots, a panic-path check and a reproducibility check.
  • Development tooling runs natively on both Windows and Linux.

Not built yet

  • M1B, with Malrynth-owned page tables, a higher-half kernel, W^X, guarded stacks and a heap.
  • M1C to M1F, covering exceptions, the APIC and timer, ACPI tables and multiple cores.
  • M2 and later, bringing processes, a scheduler, system calls and userspace, then capabilities, IPC and drivers.
  • Moving from QEMU to a physical reference machine, which is researched on a separate branch.

The long-term vision (a desktop, identity, security, applications, device management, a developer platform and optional, provider-neutral AI) is written down as specifications. None of it is presented here as built.

07What I learned

  • A boot loader teaches you to decide early, while you can still report a failure.
  • Treat every input as untrusted, including the firmware.
  • A long vision needs short, testable milestones, or it stays a document.

08Stack

  • Rust
  • UEFI
  • x86-64
  • QEMU and OVMF
  • Python
  • GitHub Actions
  • Linux
  • Windows