core: Add CPU, I/O, and network resource limits to run-service

[joaoantoniocardoso]

Helps #3356

---

Extend the service resource limitation system beyond memory to include:

• CPU limits via cgroups v2 cpu.max (percentage of cores)
• I/O bandwidth limits via cgroups v2 io.max (read/write MB/s)

Service tuple format updated to:
NAME,MEMORY_MB,CPU_PERCENT,IO_READ_MBPS,IO_WRITE_MBPS,COMMAND

Environment variables to disable limits:

• BLUEOS_DISABLE_RESOURCE_LIMITS: disables all limits
• BLUEOS_DISABLE_MEMORY_LIMIT: disables memory limit
• BLUEOS_DISABLE_CPU_LIMIT: disables CPU limit
• BLUEOS_DISABLE_IO_LIMIT: disables I/O limits

---

• All features tested as below:

Testing

Setup

# SSH into the Pi
ssh pi@blueos.local

# Disable swap for accurate memory limit testing (run on host, not in container)
sudo swapoff -a
free -h  # Verify Swap shows 0B

# Access the BlueOS container
docker exec -it blueos-core bash

# Install stress tool for testing
apt-get update && apt-get install -y stress

Test 1: Memory Limit

# Test 1a: Usage BELOW limit (should succeed)
run-service test_mem "stress --vm 1 --vm-bytes 200M --timeout 10" 256 0 0 0
# Expected: completes successfully after 10s

# Cleanup
rmdir /sys/fs/cgroup/$DOCKER_CGROUP/test_mem 2>/dev/null

# Test 1b: Usage ABOVE limit (should be OOM killed)
# --vm-keep holds memory instead of alloc/free loop, guaranteeing OOM
run-service test_mem_oom "stress --vm 1 --vm-bytes 300M --vm-keep --timeout 30" 256 0 0 0 &

# Wait and observe - should keep getting killed and restarting
sleep 5
# Expected: "Killed" messages, "worker got signal 9", "failed run completed in 1s"
# Process keeps restarting because 300MB can never fit in 256MB limit

# Cleanup
pkill -9 stress 2>/dev/null; sleep 1
rmdir /sys/fs/cgroup/$DOCKER_CGROUP/test_mem_oom 2>/dev/null

Test 2: CPU Limit

# Start stress with 50% CPU limit (in background)
run-service test_cpu "stress --cpu 1 --timeout 20" 0 50 0 0 &
sleep 2

# Verify CPU limit is set (should show "50000 100000")
cat /sys/fs/cgroup/$DOCKER_CGROUP/test_cpu/cpu.max

# In another terminal, check CPU usage - should be ~50%
top -b -n 3 -d 2 | grep stress
# Expected: stress process showing ~50% CPU (not 100%)

# Wait for completion
wait

# Cleanup
rmdir /sys/fs/cgroup/$DOCKER_CGROUP/test_cpu 2>/dev/null

Test 3: I/O Write Limit

# First, test baseline speed (no limit) - use 100MB to bypass cache
time dd if=/dev/zero of=/var/logs/blueos/test_baseline bs=1M count=100 oflag=direct
# Expected: completes in a few seconds at full disk speed

rm -f /var/logs/blueos/test_baseline

# Now test with 5 MB/s limit - should take ~20 seconds
run-service test_io "dd if=/dev/zero of=/var/logs/blueos/test_io bs=1M count=100 oflag=direct" 0 0 0 5
# Expected: takes ~20 seconds (100MB at 5MB/s)

# Verify I/O limit was set
cat /sys/fs/cgroup/$DOCKER_CGROUP/test_io/io.max
# Expected: 179:0 ... wbps=5242880

# Cleanup
rm -f /var/logs/blueos/test_io
rmdir /sys/fs/cgroup/$DOCKER_CGROUP/test_io 2>/dev/null

Test 4: Combined Limits

# Test with all limits: 256MB memory, 50% CPU, 2MB/s I/O write
run-service test_all "stress --cpu 1 --vm 1 --vm-bytes 200M --timeout 20" 256 50 0 2 &
sleep 2

# Verify all limits are set
echo "Memory:" && cat /sys/fs/cgroup/$DOCKER_CGROUP/test_all/memory.max
echo "CPU:" && cat /sys/fs/cgroup/$DOCKER_CGROUP/test_all/cpu.max
echo "I/O:" && cat /sys/fs/cgroup/$DOCKER_CGROUP/test_all/io.max

# In another terminal, check CPU is limited to ~50%
top -b -n 3 -d 2 | grep stress

# Wait for completion
wait

# Cleanup
rmdir /sys/fs/cgroup/$DOCKER_CGROUP/test_all 2>/dev/null

Test 5: Verify Existing Services

# Check that beacon service has its 250MB memory limit
cat /sys/fs/cgroup/$DOCKER_CGROUP/beacon/memory.max
# Expected: 262144000 (250MB)

# Verify service cgroups exist
ls /sys/fs/cgroup/$DOCKER_CGROUP/ | grep -E "autopilot|beacon|nginx"

Expected Results

Test	File	Expected Value
Memory 256MB	memory.max	268435456
CPU 50%	cpu.max	50000 100000
I/O 5MB/s write	io.max	179:0 ... wbps=5242880
Beacon 250MB	memory.max	262144000

Verifying Real Services

To verify limits work for actual services (with DOCKER_CGROUP properly set):

# Check beacon service (configured with 250MB memory limit)
cat /sys/fs/cgroup/$DOCKER_CGROUP/beacon/memory.max
# Expected: 262144000

# Check processes are in the cgroup
cat /sys/fs/cgroup/$DOCKER_CGROUP/beacon/cgroup.procs

Cleanup

# Re-enable swap on host after testing
exit  # Exit container
sudo swapon -a

Summary by Sourcery

Extend run-service resource management to support optional CPU and disk I/O limits alongside memory, and improve how processes are attached to cgroups.

New Features:

• Allow configuring per-service CPU usage limits via cgroups v2 cpu.max based on a percentage parameter.
• Allow configuring per-service disk I/O read and write bandwidth limits via cgroups v2 io.max derived from MB/s parameters.
• Support running services without resource limits by treating zero-valued limit parameters as unlimited.

Enhancements:

• Attach the parent shell and all child processes to the service cgroup whenever any resource limit is set, ensuring limits are inherited consistently.
• Improve service startup logging to include a concise description of the active resource limits and clarify restart messages to refer to generic resource limit breaches instead of only memory.

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• The description mentions BLUEOS_DISABLE_* environment variables to selectively disable limits, but the script changes don’t appear to read or honor these variables yet; consider wiring them into has_any_limit and the individual limit sections so behavior matches the documented interface.
• The I/O limiting logic relies on a hardcoded list of block devices (/dev/mmcblk0, /dev/sda, /dev/nvme0n1); consider deriving the underlying block device dynamically (e.g., via findmnt/lsblk/stat on the root filesystem) so this works correctly on a wider range of platforms and configurations.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The description mentions BLUEOS_DISABLE_* environment variables to selectively disable limits, but the script changes don’t appear to read or honor these variables yet; consider wiring them into has_any_limit and the individual limit sections so behavior matches the documented interface.
- The I/O limiting logic relies on a hardcoded list of block devices (/dev/mmcblk0, /dev/sda, /dev/nvme0n1); consider deriving the underlying block device dynamically (e.g., via findmnt/lsblk/stat on the root filesystem) so this works correctly on a wider range of platforms and configurations.

## Individual Comments

### Comment 1
<location> `core/run-service.sh:39-54` </location>
<code_context>
+    ROOT_MINOR=""
+    
+    # Try whole block devices (not partitions) - order matters for Raspberry Pi
+    for DEV in /dev/mmcblk0 /dev/sda /dev/nvme0n1; do
+        if [ -b "$DEV" ]; then
+            ROOT_MAJOR=$(stat -c '%t' "$DEV" 2>/dev/null)
+            ROOT_MINOR=$(stat -c '%T' "$DEV" 2>/dev/null)
+            if [ -n "$ROOT_MAJOR" ] && [ -n "$ROOT_MINOR" ]; then
+                # Convert from hex to decimal
+                ROOT_MAJOR=$((16#$ROOT_MAJOR))
+                ROOT_MINOR=$((16#$ROOT_MINOR))
+                break
+            fi
+        fi
+    done
+    
+    # Skip I/O limiting if no valid block device found
</code_context>

<issue_to_address>
**suggestion:** I/O limiting only considers a very small fixed set of block device names, which may miss valid root devices on other platforms or configurations.

The logic only checks `/dev/mmcblk0`, `/dev/sda`, and `/dev/nvme0n1`, so I/O limiting silently won’t apply on systems with other root devices (e.g. `/dev/mmcblk1`, `/dev/vda`, other NVMe indices, custom mappings). It would be more robust to (1) derive the backing block device for `/` programmatically (e.g. via `stat -c %d /` and mapping to `/dev`), or (2) at least expand the candidate set (e.g. wildcard matches, filtered to whole devices only) so this works across more environments.

```suggestion
    ROOT_MAJOR=""
    ROOT_MINOR=""
    ROOT_DEV=""

    # Try to derive the backing block device for "/" programmatically.
    # Prefer findmnt, fall back to /proc/self/mounts.
    if command -v findmnt >/dev/null 2>&1; then
        ROOT_SOURCE=$(findmnt -n -o SOURCE / 2>/dev/null || true)
    else
        ROOT_SOURCE=$(awk '$2=="/"{print $1; exit}' /proc/self/mounts 2>/dev/null || true)
    fi

    # Map mount source to a /dev/... node when possible
    if [ -n "$ROOT_SOURCE" ]; then
        case "$ROOT_SOURCE" in
            /dev/*)
                ROOT_DEV="$ROOT_SOURCE"
                ;;
            UUID=*|LABEL=*)
                if command -v blkid >/dev/null 2>&1; then
                    ROOT_DEV=$(blkid -o device -t "$ROOT_SOURCE" 2>/dev/null | head -n1 || true)
                fi
                ;;
        esac
    fi

    # Normalise to a whole-disk device (not a partition) if possible
    if [ -n "$ROOT_DEV" ] && [ -b "$ROOT_DEV" ]; then
        BASENAME=$(basename "$ROOT_DEV")
        case "$BASENAME" in
            nvme*n*p[0-9]*|mmcblk*p[0-9]*)
                WHOLE_DEV="/dev/${BASENAME%p*[0-9]}"
                ;;
            *[0-9])
                WHOLE_DEV="/dev/${BASENAME%[0-9]*}"
                ;;
            *)
                WHOLE_DEV="$ROOT_DEV"
                ;;
        esac
        if [ -b "$WHOLE_DEV" ]; then
            ROOT_DEV="$WHOLE_DEV"
        fi
    fi

    # If we successfully resolved a device, get its major/minor
    if [ -n "$ROOT_DEV" ] && [ -b "$ROOT_DEV" ]; then
        ROOT_MAJOR=$(stat -c '%t' "$ROOT_DEV" 2>/dev/null)
        ROOT_MINOR=$(stat -c '%T' "$ROOT_DEV" 2>/dev/null)
    fi

    # Fallback: scan a broader set of common whole block devices across platforms
    if [ -z "$ROOT_MAJOR" ] || [ -z "$ROOT_MINOR" ]; then
        for DEV in /dev/mmcblk* /dev/sd? /dev/vd? /dev/nvme*n1; do
            [ -b "$DEV" ] || continue
            ROOT_MAJOR=$(stat -c '%t' "$DEV" 2>/dev/null)
            ROOT_MINOR=$(stat -c '%T' "$DEV" 2>/dev/null)
            if [ -n "$ROOT_MAJOR" ] && [ -n "$ROOT_MINOR" ]; then
                break
            fi
        done
    fi

    # Convert from hex to decimal if we obtained values
    if [ -n "$ROOT_MAJOR" ] && [ -n "$ROOT_MINOR" ]; then
        ROOT_MAJOR=$((16#$ROOT_MAJOR))
        ROOT_MINOR=$((16#$ROOT_MINOR))
    fi
```
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}