How a VM Actually Boots in 125 Milliseconds

If you are looking for wtf is hypervisor, this ain’t for you. Ask chatgpt you will get everything about that including VirtualBox screenshots and the same QEMU command you’ve copy-pasted a hundred times. This is about what’s actually happening underneath, the full chain from a broken 1980s CPU design decision to a fully isolated Linux VM booting in less time than your average HTTP request.

If you’ve ever wondered why AWS Lambda feels instant, or why E2B can spin up sandboxes like candy, this is where that speed comes from.

x86 Had No Concept of Multiple Operating System

The original x86 model assumes one OS owns the machine:

Hardware
   ↑
Kernel (Ring 0)  → full privilege, controls everything
   ↑
Applications (Ring 3)  → can't touch hardware without asking the kernel

Ring 0 is God mode. The kernel sitting there can manage memory, control interrupts, talk directly to hardware. Applications in Ring 3 can’t do any of that without going through the kernel.

Virtualization breaks this model in a very specific way: guest kernels also want Ring 0. You want to run five Linux VMs on one machine, each thinking they own the CPU. But there’s only one physical machine.

The obvious fix: put the host kernel in Ring 0 and push guest kernels to Ring 1.

Ring 0 → host kernel
Ring 1 → guest kernel
Ring 3 → guest apps

This doesn’t work. x86 has a nasty property where some privileged instructions trap correctly when executed from Ring 1, some fail silently, and some behave differently without ever notifying the hypervisor. popf loads a value directly into EFLAGS. EFLAGS is the register that controls whether hardware interrupts are enabled on the CPU. The interrupt flag inside it is what allows the OS to be preempted, to be tapped on the shoulder by the hardware timer and told “your turn is over.” If a guest kernel executes popf from Ring 1 and it clears the interrupt flag on the real CPU, the hypervisor loses its ability to preempt that guest. No interrupt fires. No trap. No VMEXIT. The guest holds the CPU indefinitely. The hypervisor has no way to take it back. It never knew it happened.

The CPU gave hypervisors no reliable way to catch everything a guest kernel might do. For about years, engineers worked around this with increasingly creative hacks.

The Two Hacks That Kept Virtualization Alive

Binary translation (VMware)

VMware’s approach: scan guest code for dangerous instructions before execution and rewrite them.

Guest wants:

sgdt [mem]   ; store GDT register to memory — no trap fires, real GDT address leaks

VMware rewrites it to:

call hypervisor_handle_sgdt()   ; returns a fake GDT address, guest stays fooled

The rewritten version calls into the hypervisor instead of touching real hardware. It worked. Guest OSes ran unmodified. The catch: every block of guest code had to be scanned and potentially rewritten before the CPU touched it. That’s overhead on top of overhead, constantly, for the entire lifetime of the VM.

Paravirtualization (Xen)

Xen took the opposite approach. Instead of tricking the guest, tell it the truth. Modify the guest OS to know it’s inside a VM and replace dangerous instructions with explicit hypercalls.

; old way, not safe inside a VM
hlt

; Xen way, guest cooperates explicitly
hypercall(HALT)

Faster than binary translation because there’s no scanning or rewriting at runtime. But it required modifying every guest OS you wanted to run, which made Windows support a recurring headache.

Xen’s design was intentionally minimal. It only knew how to virtualize CPUs, manage memory, schedule VMs, handle VM exits, and isolate guests. No network drivers, no disk drivers, no USB, no GPU. All of that got delegated to Dom0.

Dom0 is a privileged Linux VM that Xen boots automatically at startup. It gets direct access to real hardware:

Hardware
   ↓
Xen (the only software running on bare metal)
   ↓
Dom0 (first VM Xen starts, direct NIC/SSD/PCI access)
   ↓
DomU #1, DomU #2 (normal VMs, no hardware access)

Normal guest VMs (DomU) have zero hardware privileges. A guest reading from disk wasn’t talking to a disk. It was sending a request to Dom0, which then talked to the actual disk and sent data back. Every single I/O operation from every guest went through Dom0.

That architecture worked until it didn’t. Dom0 became a bottleneck under heavy I/O. Worse, it became a security boundary problem: if Dom0 gets compromised, every guest VM on that host is exposed. You built isolation into the system and then put a single privileged VM in the middle that could break all of it.

Binary translation and paravirtualization both had ceilings. One was too slow, the other had a structural single point of failure and required OS modifications that vendors wouldn’t always make. Both were software working around a hardware problem.

Intel decided to fix the hardware.

VT-x: When the CPU Finally Cooperated

Intel shipped VT-x in 2005. AMD shipped AMD-V in 2006. This changed everything.

The CPU became virtualization-aware. Intel added instructions specifically for managing VMs:

VMXON      ; enable virtualization mode
VMLAUNCH   ; start a VM
VMRESUME   ; resume a paused VM
VMXOFF     ; disable virtualization

More importantly, it added a second dimension to privilege that did not exist before. VMX root mode and VMX non-root mode. Before VT-x, privilege was one axis. Ring 0 meant the CPU would execute anything, including instructions that touch memory, interrupts, and hardware directly. Ring 3 meant the CPU would trap if you tried. That was the whole model. VT-x adds a layer that sits outside the Ring system entirely:

VMX root?      Ring?
Host kernel       YES            0     full trust, real hardware
Guest kernel      NO             0     Ring 0 inside the VM only
Guest app         NO             3     same as always

Non-root mode Ring 0 is not the same thing as root mode Ring 0. The guest kernel has Ring 0 privilege within its own VM. It can manage its own memory, schedule its own processes, do everything a kernel normally does. But when it tries something that would touch the real machine like reading a hardware register, issuing an interrupt, checking the CPU identity, hardware stops it cold. Not because the instruction is privileged in the Ring sense. Because the CPU is now watching for it specifically. Root mode is the exit hatch that did not exist before 2005. The guest can be Ring 0 without being able to escape its box. That distinction is the entire foundation of modern virtualization.

VMEXIT: How the Hypervisor Actually Gets Control

VMEXIT fires when the guest does something the hardware knows the hypervisor needs to see. Not every instruction causes one. Most guest code runs directly on the CPU at full speed, no hypervisor involved. VMEXIT is reserved for specific events: the guest executes CPUID to figure out what CPU it’s on, writes to a control register, pokes an I/O port, reads the timestamp counter with RDTSC, or tries to manage hardware interrupts. The exact list is configurable per-VM in the VMCS, but those are the common ones. When it fires, three things happen automatically in hardware. The CPU saves the complete guest state, every register, every flag, the instruction pointer into a data structure called the VMCS (VM Control Structure). It switches from non-root to root mode. Then it jumps to a fixed address in the hypervisor. The hypervisor wakes up, reads an exit reason code out of the VMCS, and handles whatever the guest tried to do.

Guest executes CPUID
        ↓
VMEXIT: CPU writes guest state to VMCS, switches to root mode
        ↓
Hypervisor reads exit reason: "guest asked for CPUID"
        ↓
Hypervisor decides what value to return
(could be real, could be fake, it has full control)
        ↓
VMRESUME: CPU reads guest state from VMCS, resumes guest
        ↓
Guest gets the value. No idea any of this happened.

The VMCS is the piece worth naming. It is a hardware-managed memory region. The CPU writes to it on exit and reads from it on resume, automatically, every single time. The guest’s execution state is preserved perfectly across the entire round trip. From inside the VM, CPUID executed and returned a number. The fact that a hypervisor ran a function call in between is completely invisible. This is also why VMEXIT frequency matters so much for performance later. Each exit is: write state to VMCS, mode switch, hypervisor runs, mode switch back, restore state from VMCS. Sub-microsecond individually. Hundreds of them per second from unnecessary I/O emulation and the cost stacks up fast.

KVM: The Kernel Already Had Root, Someone Just Had to Ask

VT-x is sitting right there in the hardware. The CPU supports virtualization natively. You’d think you could just… use it.

int main() {
    asm("VMXON");  // enable virtualization
}

Nope. CPU rejects it instantly. VMX instructions require Ring 0. Userspace is Ring 3. That gap might as well be a concrete wall.

Here’s the thing though. The Linux kernel is already running at Ring 0. It has the privilege. It just wasn’t using it for virtualization. KVM is a kernel module that fixes exactly that. Load it, and Linux quietly becomes a hypervisor. The kernel can now call:

vmxon();
vmlaunch();
vmresume();

Think of it like this. VT-x is Satoru Gojo locked in the prison realm, raw power sitting dormant in the hardware. Userspace can see it, can’t touch it. KVM is what gives that power a user. Once it’s loaded, the kernel isn’t just an OS anymore, it’s an OS with hypervisor capability bolted on.

KVM exposes all of this through one device file:

/dev/kvm

Any userspace program that opens it gets access to the virtualization machinery through ioctls:

open("/dev/kvm")
ioctl(KVM_CREATE_VM)
ioctl(KVM_CREATE_VCPU)
ioctl(KVM_RUN)

That program is the VMM (Virtual Machine Monitor). The split is clean:

VMX hardware   = the raw capability
KVM            = kernel interface to that capability  
VMM (QEMU etc) = the userspace program that orchestrates everything

KVM handles guest CPU execution, VMEXIT processing, and vCPU state management. The VMM handles everything else. KVM doesn’t care what device model you want, what kernel the guest runs, or what the VM is for. It’s a thin interface between userspace and the hardware. Intentionally boring and minimal like me. That’s the point.

QEMU: The Greatest Liar in Systems Programming (And Why That’s a Problem)

KVM runs guest code on real hardware at near-native speed. But a guest OS needs more than a CPU to function. Booting Linux inside a VM means the guest expects to find:

CPU, RAM, Disk controller, Network card, BIOS/UEFI, Interrupt controller

KVM gives you exactly one of those. QEMU provides the rest by lying about all of them.

QEMU is a device emulator. It creates fake hardware so convincingly that the guest has no idea it’s talking to software. When a guest does a disk read:

read("/etc/passwd")

Here’s what actually happens:

Guest issues disk I/O
        ↓
VMX triggers VMEXIT (I/O port access)
        ↓
KVM sees the exit type
        ↓
Returns control to QEMU in userspace
        ↓
QEMU emulates the disk controller
        ↓
Reads from host disk image (.qcow2)
        ↓
Returns data back through KVM
        ↓
VMRESUME → guest continues

The guest gets its data. It has no idea that its “disk controller” is a function call in a userspace process. QEMU is Itachi’s genjutsu as a systems program: the guest is fully convinced it’s talking to real hardware. It never is.

This works. It also has a cost.

Every single I/O operation triggers a VMEXIT, a round trip through KVM, a jump into QEMU userspace, device emulation, and a VMRESUME back to the guest. Five hops per I/O operation, every time.

But that’s not even the main issue. The real problem is what QEMU emulates by default:

Motherboard, PCI bus, USB controller, BIOS/UEFI,
Network card, Graphics card, Sound card, Legacy ISA devices

The guest OS initializes all of this at boot. Each device initialization involves I/O instructions. Each I/O instruction triggers a VMEXIT. You’re paying the five-hop cost hundreds of times before the VM is even running a workload, for hardware the guest will never actually use.

That’s where the 30-second to 2-minute boot times came from. It wasn’t slow hardware. It was the guest kernel spending its entire boot cycle initializing a museum of emulated devices from 1998 that nobody asked for. The BIOS alone eats 2 to 5 seconds before the kernel even starts.

QEMU’s flexibility is its own worst enemy. It can emulate anything, so it emulates everything. And in a world where you need a VM to boot in milliseconds, emulating everything is a bad tradeoff.

VirtIO: Cut the VMEXITs

Every I/O operation in QEMU goes through a five-hop round trip. Guest pokes an I/O port, VMEXIT fires, KVM hands off to QEMU, QEMU emulates a device, sends data back, VMRESUME. Every. Single. Time.

Instead of emulating real hardware with I/O port instructions, VirtIO uses a shared memory ring buffer between the guest and the hypervisor. The guest driver drops a request into the ring. The hypervisor reads it and processes it. No I/O port poking, no unnecessary VMEXITs.

virtio-net     → network
virtio-blk     → block device
virtio-scsi    → storage
virtio-vsock   → VM to host communication
virtio-rng     → random number generator

Sending a network packet with VirtIO:

Guest driver:
Packet → descriptor into virtqueue → single kick notification to hypervisor

Hypervisor:
Reads descriptor → reads packet from shared memory → sends it

The data moves through shared memory. One VMEXIT per batch of operations instead of one per operation. The guest and hypervisor are sharing the buffer directly, no emulated hardware in the middle, no fiction to maintain.

VirtIO is a standard, not an implementation. Every major VMM supports it. The guest needs the virtio kernel drivers, which have been in mainline Linux since 2.6.24, and it works with any compliant hypervisor. Write the driver once, run it everywhere.

The deeper point: VirtIO didn’t just make I/O faster. It made the case that you don’t need to emulate real hardware at all. If the guest is going to use a VirtIO driver anyway, why is QEMU spending time pretending to be a physical NIC? That question is what leads to MicroVMs.

MicroVMs: The Millennium Falcon Was Always Faster Than a Star Destroyer

The Star Destroyer has everything. Hangar bays, thousands of crew members, turbolaser batteries, a cafeteria, probably an HR department. It’s impressive. It also takes forever to do anything.

The Millennium Falcon has a cockpit, an engine, guns, and Han Solo. That’s it. That’s enough.

QEMU is the Star Destroyer. It emulates a full PC: PCI bus, USB controller, BIOS, UEFI, sound card, graphics card, legacy ISA devices. The guest initializes all of it at boot. Most of it will never be used. You’re paying the boot cost of a full 1990s PC for a workload that just needs to run Python scripts.

MicroVMs are the Millennium Falcon. Strip out everything except what the workload actually needs:

vCPU
Memory
virtio-net
virtio-block
virtio-vsock
Serial port

Six things. That’s the entire device model. The guest initializes six things instead of hundreds. Boot time drops off a cliff.

The MicroVM space is bigger than most people realize:

Every single one of them sits on the same three-layer foundation: KVM at the bottom, a VMM in userspace, a guest kernel on top. What differs is how minimal the device model is above KVM.

They also share a massive amount of code. In 2018, AWS and Google started rust-vmm: a collection of shared Rust crates so that nobody has to keep reimplementing the same low-level primitives from scratch.

kvm-ioctls    → KVM userspace interface
vm-memory     → guest memory abstractions
virtio-queue     → virtio device logic
vhost         → device acceleration
vmm-sys-util  → low-level utilities

Firecracker, crosvm, Cloud Hypervisor, libkrun all build on these. A security fix in vm-memory benefits every VMM in the group at once. One patch, five products hardened. That’s the actual value of rust-vmm.

Firecracker: Dwight Schrute as a VMM

Dwight doesn’t waste time. Doesn’t make small talk. Doesn’t do anything that isn’t directly required by the task. Extremely efficient, slightly terrifying, gets results. ( Assistant TO the Regional Manager.)

Firecracker has that energy. AWS open-sourced it in 2018. It runs Lambda, Fargate, and is the foundation under E2B’s AI agent sandboxes. Its device list is four I/O items:

vCPU
Memory
virtio-net    (network)
virtio-block  (disk)
virtio-vsock  (host to guest communication)
Serial port

No PCI bus. No USB. No sound card. No BIOS. No GRUB.

That last part is where a lot of time gets recovered. Traditional VMs run BIOS initialization, then hand off to a bootloader like GRUB, which then loads the kernel. That sequence alone eats 2 to 5 seconds before the guest kernel even starts executing. Firecracker skips it entirely with direct kernel boot.

Instead of the BIOS and bootloader dance, Firecracker loads a pre-built Linux kernel image directly into guest memory and sets the CPU registers to point at the kernel entry point. The startup sequence looks like:

Firecracker opens /dev/kvm via ioctl
Creates VM, allocates guest memory via mmap()
Creates vCPUs (each one is a kernel thread on the host)
Loads Linux kernel image directly into guest memory
Sets instruction pointer to kernel entry point
Configures the four virtio devices
KVM_RUN → VMLAUNCH → guest kernel starts executing immediately

No BIOS. No GRUB. The kernel wakes up and starts running.

Cut the BIOS phase (saves 2 to 5 seconds). Strip the device model to six things (massively fewer VMEXITs during initialization). The result is a running Linux VM with its own kernel, filesystem, and network namespace in about 125ms.

Dwight would approve. No unnecessary steps. No redundant hardware. Bears. Beets. Battlestar Galactica.

Snapshots and UFFD: Why 125ms Isn’t the End of the Story

125ms is fast. But if you’re spinning up hundreds or thousands of sandboxes per second, even 125ms becomes overhead. More importantly, booting a fresh Linux kernel every time means repeating the exact same initialization work over and over again.

The obvious optimization is: don’t boot twice.

Boot the VM once. Initialize everything you need. Install dependencies, start the runtime, load libraries, warm caches. Once the VM reaches its “ready” state, freeze it into a snapshot. Every future sandbox resumes from that point instead of starting from scratch.

A Firecracker snapshot captures the complete execution state of a running VM:

• Guest RAM
• vCPU register state
• Virtual device state
• VM metadata

It’s everything required to resume execution as if the VM had only been paused for an instant.

The workflow becomes:

Boot VM once
      ↓
Initialize runtime
Install dependencies
Reach ready state
      ↓
Take snapshot
────────────────────────────────────
New request arrives
      ↓
Restore snapshot
      ↓
VM resumes exactly where it left off

The guest kernel doesn’t reboot. The runtime doesn’t restart. Initialization code doesn’t run again. From inside the VM, time simply continues. Published Firecracker benchmarks have shown snapshot restore latencies as low as ~4ms.

There’s one obvious problem.

A snapshot may contain gigabytes of guest memory. If restoring a snapshot meant copying every byte of RAM before the guest could execute, you’d simply replace kernel boot time with a massive memory copy.

Instead, Firecracker restores memory lazily.

Rather than eagerly loading every page into RAM, the guest’s memory is mapped, and pages are populated only when they’re actually accessed.

This is where Linux’s userfaultfd (UFFD) becomes useful.

Normally, a page fault looks like this:

CPU accesses unmapped page
        ↓
Hardware raises page fault
        ↓
Kernel resolves the fault
        ↓
Execution continues

With UFFD, userspace can participate in handling those faults.

A process registers a region of guest memory with the kernel. When the guest touches a page that hasn’t been populated yet, the kernel pauses the faulting thread and notifies the userspace handler instead of immediately resolving the fault itself.

The flow becomes:

Guest accesses page
        ↓
Hardware raises page fault
        ↓
Kernel pauses the guest thread
        ↓
UFFD handler receives notification
        ↓
Handler retrieves the required page
(from the snapshot or another backing store)
        ↓
Handler copies the page into guest memory
        ↓
Kernel resumes guest execution

Instead of paying the cost of restoring an entire 2GB snapshot up front, the VM only loads the pages it actually touches.

If a workload accesses just 20MB during its first few seconds, only those pages need to be fetched immediately. The rest of the snapshot can remain untouched until the guest eventually needs.

The result is a VM that starts executing almost immediately while memory fills in transparently behind the scenes.

Some platforms optimize this even further with warm pools.

Instead of restoring a snapshot after a request arrives, they keep a pool of already-restored microVMs sitting idle, waiting for work. When a request comes in, the scheduler simply assigns one of those pre-restored VMs to it, eliminating even the snapshot restore from the request’s critical path.

The Full Chain

The path from “VMs take 2 minutes” to “VMs boot in 125ms” is a straight line of specific problems and specific fixes.

x86 couldn’t be virtualized cleanly, so VMware rewrote dangerous instructions at runtime. VT-x moved that interception into CPU hardware. KVM exposed hardware virtualization to Linux userspace through /dev/kvm. QEMU emulated everything a guest could want, which made it flexible but slow. VirtIO replaced I/O port emulation with shared memory ring buffers. MicroVMs stripped the device model down to the four VirtIO devices that matter and killed the BIOS. Firecracker built a production VMM on top of rust-vmm using exactly that design. Snapshots with UFFD removed even the 125ms kernel boot from the critical path.

Each step solved the bottleneck the previous step introduced. The result is that you can spin up a fully isolated VM with its own kernel, filesystem, and network namespace in less time than most DNS lookups take.

That’s not magic. That’s 30 years of engineers fixing one broken assumption at a time.