Skip to content

rust-os-design.md

Latest commit

 

History

History
480 lines (386 loc) · 18.5 KB

rust-os-design.md

File metadata and controls

480 lines (386 loc) · 18.5 KB

RustOS: A Hypervisor-Native Operating System

Table of Contents

  1. Overview and Goals
  2. Architecture Overview
  3. Platform Support
  4. Hypervisor Integration
  5. Kernel/User Mode Interaction
  6. Memory Management
  7. Process and Thread Management
  8. I/O and Device Management
  9. Security Model
  10. Development and Build System
  11. Future Considerations

Overview and Goals

RustOS is a modern, hypervisor-native operating system written entirely in Rust, designed from the ground up to leverage Rust's memory safety guarantees and zero-cost abstractions. The primary goals are:

Core Objectives

  • Memory Safety: Eliminate entire classes of vulnerabilities through Rust's ownership system
  • Performance: Achieve near-native performance with zero-cost abstractions
  • Hypervisor-First Design: Optimize for virtualized environments rather than bare metal
  • Modern Architecture: Clean, modular design unconstrained by legacy compatibility
  • Developer Experience: Provide excellent tooling and debugging capabilities

Design Principles

  • Capability-Based Security: Replace traditional access control with capability-based security
  • Microkernel Philosophy: Minimize kernel complexity while maintaining performance
  • Async-First: Built around Rust's async/await ecosystem for optimal concurrency
  • Type-Safe Interfaces: Leverage Rust's type system for compile-time correctness
  • Resource Efficiency: Optimize for cloud and edge computing environments

Architecture Overview

RustOS follows a hybrid microkernel architecture with the following key components:

┌─────────────────────────────────────────────────────────┐
│                    User Applications                    │
├─────────────────────────────────────────────────────────┤
│                  System Services                       │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐    │
│  │ File System  │ │   Network    │ │   Display    │    │
│  │   Service    │ │   Service    │ │   Service    │    │
│  └──────────────┘ └──────────────┘ └──────────────┘    │
├─────────────────────────────────────────────────────────┤
│                 Capability Runtime                     │
├─────────────────────────────────────────────────────────┤
│                    RustOS Kernel                       │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐    │
│  │   Memory     │ │   Process    │ │     IPC      │    │
│  │  Manager     │ │  Scheduler   │ │   Manager    │    │
│  └──────────────┘ └──────────────┘ └──────────────┘    │
├─────────────────────────────────────────────────────────┤
│                Hardware Abstraction Layer              │
├─────────────────────────────────────────────────────────┤
│                   Hypervisor Interface                 │
│         (KVM, Xen, VMware, Hyper-V, etc.)             │
└─────────────────────────────────────────────────────────┘

Key Architectural Decisions

  1. Hypervisor Abstraction: Direct integration with hypervisor APIs rather than hardware
  2. Capability-Based IPC: All inter-process communication uses typed capabilities
  3. Async Runtime: Built on top of a custom async executor optimized for system programming
  4. Zero-Copy Networking: Direct buffer sharing between network stack and applications
  5. Compile-Time Resource Allocation: Static analysis for memory and CPU resource planning

Platform Support

x86_64 Target Architecture

RustOS initially targets the x86_64 architecture with the following considerations:

CPU Features

  • Required: SSE2, RDTSC, CPUID, NX bit support
  • Recommended: AVX2, RDRAND, SMAP/SMEP, Intel CET
  • Future: AVX-512, Intel MPX, ARM64 compatibility layer

Memory Model

  • Virtual Memory: 4-level paging with 1GB huge pages support
  • Address Space: 48-bit virtual addresses (256TB user space)
  • NUMA Awareness: First-class support for NUMA topology
  • Memory Protection: Leveraging Intel MPK (Memory Protection Keys)

Boot Process

  1. Hypervisor Boot: Direct kernel loading via hypervisor
  2. EFI Compatibility: Optional EFI boot for bare metal testing
  3. Multiboot2: Support for advanced boot loaders
  4. Hot Plug: Runtime CPU and memory hot-plug support

Hypervisor Integration

Hypervisor-Native Design

RustOS is designed specifically for virtualized environments:

Paravirtualization Strategy

pub trait HypervisorInterface {
    async fn allocate_memory(&self, size: usize, flags: MemoryFlags) -> Result<VirtAddr>;
    async fn deallocate_memory(&self, addr: VirtAddr, size: usize) -> Result<()>;
    async fn create_vcpu(&self, config: VcpuConfig) -> Result<VcpuHandle>;
    async fn setup_interrupt(&self, vector: u8, handler: InterruptHandler) -> Result<()>;
    async fn hypercall(&self, call: HypercallRequest) -> Result<HypercallResponse>;
}

Supported Hypervisors

  • KVM/QEMU: Primary development and testing platform
  • Xen: Paravirtualized and HVM modes
  • VMware vSphere: Production deployment target
  • Microsoft Hyper-V: Azure cloud compatibility
  • AWS Nitro: Direct integration for EC2 instances

Performance Optimizations

  • Enlightened Page Tables: Direct hypervisor memory management
  • SR-IOV Integration: Hardware-accelerated I/O virtualization
  • VFIO Passthrough: Direct device assignment capabilities
  • Time Synchronization: Hypervisor-aware timekeeping
  • Balloon Driver: Dynamic memory management

Kernel/User Mode Interaction

Revolutionary Approach: Capability Channels

Instead of traditional syscalls, RustOS implements a capability-based communication system:

Traditional Syscall Problems

  • Context switching overhead
  • Validation complexity
  • Security boundary confusion
  • Synchronous operation blocking

Capability Channel Solution

pub struct CapabilityChannel<T: CapabilityType> {
    sender: async_channel::Sender<T>,
    receiver: async_channel::Receiver<T>,
    capability: Capability,
}

// Example: File system access
pub enum FileSystemCapability {
    Read(FileHandle, Buffer) -> Result<BytesRead>,
    Write(FileHandle, Buffer) -> Result<BytesWritten>,
    Open(Path, OpenFlags) -> Result<FileHandle>,
    Close(FileHandle) -> Result<()>,
}

Three-Tier Interaction Model

  1. Direct Capability Invocation: Zero-copy for trusted code
  2. Async Message Passing: For untrusted or complex operations
  3. Emergency Syscalls: Minimal set for bootstrapping and debugging

Performance Benefits

  • Zero-Copy Operations: Direct memory sharing between kernel and user space
  • Batch Processing: Multiple operations in single context switch
  • Async by Default: Non-blocking operations with back-pressure
  • Type Safety: Compile-time verification of capability usage

Alternative Designs Considered

1. Traditional Syscall Interface

Pros: Familiar to developers, well-understood semantics Cons: High overhead, synchronous blocking, security complexity Verdict: Rejected due to performance and security concerns

2. Shared Memory + Signals

Pros: High performance, established patterns Cons: Complex synchronization, error-prone, limited composability Verdict: Used internally but not exposed to applications

3. eBPF-style Sandboxing

Pros: Safe code execution in kernel context Cons: Limited expressiveness, complex toolchain Verdict: Considered for future extension mechanism

4. Microkernel Message Passing

Pros: Clean separation, well-studied approach Cons: High message passing overhead Verdict: Inspiration for capability channels but with modern async approach

Memory Management

Rust-Aware Memory Management

Core Principles

  • Ownership-Based Allocation: Memory allocator understands Rust ownership
  • Zero-Copy Philosophy: Minimize data copying across boundaries
  • NUMA-Aware Allocation: Automatic NUMA node-aware memory placement
  • Predictable Performance: Avoid garbage collection, prefer deterministic allocation

Memory Allocation Strategy

pub struct RustOSAllocator {
    // Per-CPU allocation pools
    per_cpu_pools: [AllocationPool; MAX_CPUS],
    // Large object allocator
    large_object_allocator: SlabAllocator,
    // Hypervisor memory interface
    hypervisor: Arc<dyn HypervisorInterface>,
}

pub trait MemoryCapability {
    fn allocate_pages(&self, order: u8, flags: PageFlags) -> Result<PageRange>;
    fn map_device_memory(&self, phys_addr: PhysAddr, size: usize) -> Result<VirtAddr>;
    fn create_shared_mapping(&self, size: usize) -> Result<SharedMapping>;
}

Advanced Features

  • Memory Tagging: Hardware-assisted memory safety (ARM MTE/Intel LAM)
  • Lazy Allocation: Commit memory only on first access
  • Memory Compression: Transparent memory compression for inactive pages
  • Hot-Cold Separation: Automatic hot/cold data separation

Virtual Memory Management

Address Space Layout

0x0000_0000_0000_0000 - 0x0000_7FFF_FFFF_FFFF: User Space (128TB)
0x0000_8000_0000_0000 - 0x0000_FFFF_FFFF_FFFF: Capability Space (128TB)
0xFFFF_8000_0000_0000 - 0xFFFF_FFFF_FFFF_FFFF: Kernel Space (128TB)

Page Table Management

  • Recursive Page Tables: Efficient page table traversal
  • Copy-on-Write: Efficient process forking and memory sharing
  • Demand Paging: Load pages only when accessed
  • PCID Support: Process Context ID for TLB efficiency

Process and Thread Management

Modern Process Model

Process Capabilities

pub struct Process {
    // Unique process identifier
    pid: ProcessId,
    // Capability set for this process
    capabilities: CapabilitySet,
    // Address space
    address_space: AddressSpace,
    // Async runtime
    async_runtime: AsyncRuntime,
    // Resource limits
    resource_limits: ResourceLimits,
}

Thread Architecture

  • Green Threads: M:N threading model with async/await integration
  • Work-Stealing Scheduler: Efficient load balancing across cores
  • Priority Inheritance: Prevent priority inversion
  • CPU Affinity: NUMA-aware thread placement

Scheduler Design

Multi-Level Feedback Queue with Rust-Specific Optimizations

pub struct RustOSScheduler {
    // Real-time tasks
    rt_queue: PriorityQueue<Task>,
    // Interactive tasks
    interactive_queue: CfsQueue<Task>,
    // Background tasks
    background_queue: FairQueue<Task>,
    // Async task scheduler
    async_executor: AsyncExecutor,
}

Scheduling Policies

  • Real-Time: FIFO and Round-Robin for hard real-time tasks
  • Completely Fair Scheduler (CFS): For interactive applications
  • Batch: For CPU-intensive background tasks
  • Idle: For low-priority maintenance tasks

I/O and Device Management

Async-First I/O Architecture

Device Driver Model

pub trait DeviceDriver: Send + Sync {
    type Error: std::error::Error + Send + Sync;
    
    async fn initialize(&self) -> Result<(), Self::Error>;
    async fn read(&self, buffer: &mut [u8]) -> Result<usize, Self::Error>;
    async fn write(&self, buffer: &[u8]) -> Result<usize, Self::Error>;
    async fn ioctl(&self, cmd: u32, arg: usize) -> Result<usize, Self::Error>;
}

I/O Subsystems

Network Stack
  • User-Space TCP/IP: High-performance user-space networking
  • Zero-Copy Sockets: Direct buffer sharing with network cards
  • eBPF Integration: Programmable packet processing
  • RDMA Support: Remote Direct Memory Access for high-performance computing
Storage Stack
  • NVMe-First Design: Optimized for modern NVMe SSDs
  • Async Block Layer: Non-blocking I/O operations
  • Copy-on-Write File System: Built-in CoW file system
  • Distributed Storage: Native support for distributed storage protocols
Display and GPU
  • Vulkan API: Direct Vulkan support for GPU compute and graphics
  • Wayland Compositor: Modern display server protocol
  • Hardware Acceleration: Direct GPU access for compute workloads

Security Model

Capability-Based Security

Core Security Principles

  • Principle of Least Privilege: Processes receive only necessary capabilities
  • Fail-Safe Defaults: Secure by default configuration
  • Complete Mediation: All access goes through capability system
  • Defense in Depth: Multiple layers of security mechanisms

Capability System

pub struct Capability {
    // Unique capability identifier
    id: CapabilityId,
    // What operations are allowed
    permissions: PermissionSet,
    // Resource being protected
    resource: ResourceHandle,
    // Expiration time (optional)
    expires_at: Option<Instant>,
}

pub enum Permission {
    Read,
    Write,
    Execute,
    Delete,
    Grant,
    Delegate,
}

Security Features

  • Hardware Security Modules: Integration with TPM/HSM for key management
  • Mandatory Access Control: SELinux-inspired mandatory access control
  • Control Flow Integrity: Hardware-assisted CFI (Intel CET/ARM Pointer Authentication)
  • Stack Protection: Stack canaries and shadow stacks
  • Address Space Layout Randomization: Enhanced ASLR with entropy

Cryptographic Integration

Built-in Cryptography

  • Hardware Acceleration: AES-NI, SHA extensions, CRC32 instructions
  • Post-Quantum Cryptography: Preparation for quantum-resistant algorithms
  • Secure Boot: Verified boot chain with capability-based trust
  • Encrypted Storage: Transparent disk encryption

Development and Build System

Rust-Specific Toolchain

Build System

[package]
name = "rustos-kernel"
version = "0.1.0"
edition = "2021"

[dependencies]
# Core dependencies
no-std-compat = "0.4"
linked_list_allocator = "0.10"
x86_64 = "0.14"
futures = { version = "0.3", default-features = false }

# Hypervisor interfaces
kvm-bindings = "0.6"
xen-bindings = "0.2"

[profile.kernel]
inherits = "release"
panic = "abort"
lto = true
codegen-units = 1

Development Workflow

  1. Cross-Compilation: Custom target specification for kernel development
  2. Testing: Unit tests, integration tests, and hypervisor-based testing
  3. Debugging: GDB integration with QEMU for kernel debugging
  4. Profiling: Built-in profiling support for performance analysis
  5. Documentation: Comprehensive documentation generation with rustdoc

Quality Assurance

  • Static Analysis: Clippy, Miri, and custom lints for kernel code
  • Dynamic Analysis: AddressSanitizer and ThreadSanitizer ports
  • Formal Verification: Integration with verification tools like Prusti
  • Continuous Integration: Automated testing across multiple hypervisors

Distribution and Deployment

Container Integration

  • OCI Compatibility: OS images can be distributed as OCI containers
  • Immutable Infrastructure: Read-only root filesystem with overlay
  • A/B Updates: Atomic system updates with rollback capability
  • Configuration Management: Declarative system configuration

Future Considerations

Roadmap

Phase 1: Core Kernel (Months 1-12)

  • Basic hypervisor integration
  • Memory management subsystem
  • Process and thread management
  • Basic I/O and networking
  • Capability system foundation

Phase 2: System Services (Months 12-24)

  • File system service
  • Network stack
  • Display server
  • Audio subsystem
  • Device driver framework

Phase 3: Developer Experience (Months 24-36)

  • Comprehensive tooling
  • Performance profiling
  • Debugging infrastructure
  • Application framework
  • Package management

Phase 4: Advanced Features (Months 36+)

  • Real-time capabilities
  • GPU compute integration
  • Distributed system features
  • Container orchestration
  • Machine learning acceleration

Research Areas

Formal Verification

  • Rust Verification: Collaborate with Rust verification research
  • Capability Correctness: Formal verification of capability system
  • Memory Safety Proofs: Mathematical proofs of memory safety properties

Performance Research

  • Zero-Copy Everything: Minimize data copying throughout the system
  • Predictable Performance: Bounded execution time for critical operations
  • Energy Efficiency: Power-aware scheduling and resource management

Security Innovations

  • Hardware-Software Co-design: Leverage emerging hardware security features
  • Quantum-Resistant Security: Prepare for post-quantum cryptography
  • Privacy-Preserving Computing: Built-in support for confidential computing

Long-term Vision

RustOS aims to become the foundation for next-generation computing infrastructure:

  • Cloud-Native OS: Designed specifically for cloud and edge environments
  • Developer-Friendly: Excellent tooling and debugging experience
  • Security-First: Secure by design with formal verification
  • Performance-Oriented: Competitive with traditional operating systems
  • Ecosystem Integration: Seamless integration with Rust ecosystem

This document is a living specification that will evolve as RustOS development progresses. Contributions and feedback are welcome through the project's issue tracker and discussion forums.