`docker run --cpus 2 ...` doesn't work

[kvinwang]

When running docker run --rm --cpus 2 ubuntu in dstack CVM, it fails with an error:

docker: Error response from daemon: Range of CPUs is from 0.01 to 1.00, as there are only 1 CPUs available.

This is because we have pinned dockerd to only one CPU in Dstack-TEE/meta-dstack#13.
It is unclear what the better solution might be.

Workaround (v0.5.6+)

Add the following to your pre_launch_script to enable --cpus N:

mkdir -p /etc/systemd/system/docker.service.d/
cat > /etc/systemd/system/docker.service.d/override.conf << 'EOF'
[Service]
CPUAffinity=
EOF
systemctl daemon-reload
systemctl restart docker

Note: This removes the CPU pinning optimization, which may make docker pull ~2.7x slower due to TDX IPI overhead. See comments below for a dynamic cpuset switching approach that preserves fast pulls.

[OGNodeFather]

@kvinwang you say "help wanted" and I'm curious, how I might be of help here? I reported this issue because i need to be able to set cpus to more than 1 for performance of an application we deployed. If there is any way I can help, I'd be interested.

[kvinwang]

@kvinwang you say "help wanted" and I'm curious, how I might be of help here? I reported this issue because i need to be able to set cpus to more than 1 for performance of an application we deployed. If there is any way I can help, I'd be interested.

Reverting the pin to cpu0 change in Dstack-TEE/meta-dstack#13 would fix the CPU setting issue but reintroduce slow docker image extraction.

I've marked this "help wanted" as no clear solution in my mind for achieving both goals simultaneously. I would really appreciate it if you have any good ideas.

[h4x3rotab]

Is it possible to make pinning feature configurable in the pre-launch script?

[kvinwang]

Analysis

After investigating, I found the root cause:

Why --cpus 2 fails

Docker's CPU validation in daemon/daemon_unix.go uses sysinfo.NumCPU() which relies on sched_getaffinity(2) system call. When systemd's CPUAffinity=0 is set, sched_getaffinity only sees 1 CPU, causing Docker to reject --cpus values > 1.

if resources.NanoCPUs > int64(sysinfo.NumCPU())*1e9 {
    return fmt.Errorf("Range of CPUs is from 0.01 to %d.00...", sysinfo.NumCPU())
}

Potential Solutions

1. User workaround via pre_launch_script (immediate)

# Remove CPU pinning for apps that need --cpus
rm -f /usr/lib/systemd/system/docker.service.d/override.conf
systemctl daemon-reload
systemctl restart docker

2. Use cgroup cpuset instead of CPUAffinity (needs testing)

• cgroup cpuset.cpus limits don't affect sched_getaffinity
• Docker containers can use --cpus beyond daemon's cpuset
• Need to verify if this still provides the pull acceleration benefit

3. Make it configurable at image build time

• Add an option to exclude the CPU pinning override file
• Users who need --cpus can build images without it

4. Patch Docker to use cgroup-aware CPU count

• Use uber-go/automaxprocs approach to read from cgroup instead of sched_getaffinity
• This would be an upstream Docker change

Recommended Next Steps

1. Test if cgroup cpuset (instead of CPUAffinity) provides similar pull acceleration
2. If yes, switch from CPUAffinity=0 to cgroup-based limiting
3. Document the pre_launch_script workaround for users who need multi-CPU containers immediately

[kvinwang]

Test Results on 8-core 16GB VM

Tested on teepod2.kvin.wang with dstack-0.5.6 image:

Test Scenario	docker pull python:latest	--cpus 2 works?
CPUAffinity=0 (current)	28.6s	❌ No
No CPU restriction (8 CPUs)	77.5s (2.7x slower)	✅ Yes
cpu.max bandwidth limit	2m13s (no help)	N/A
Dynamic cpuset switching	28.6s	✅ Yes

Solution: Dynamic cgroup cpuset switching

The key discovery is that cgroup cpuset.cpus can be changed dynamically without restarting dockerd:

# 1. Pin to CPU 0 before pull (fast mode)
echo "0" > /sys/fs/cgroup/system.slice/docker.service/cpuset.cpus

# 2. Pull image - only 28.6 seconds
docker pull python:latest

# 3. Restore all CPUs after pull
echo "0-7" > /sys/fs/cgroup/system.slice/docker.service/cpuset.cpus

# 4. Now --cpus 2 works!
docker run --rm --cpus 2 busybox echo "SUCCESS: --cpus 2 works!"

Why this works

1. cpuset.cpus vs CPUAffinity: Unlike CPUAffinity which affects sched_getaffinity(), cgroup cpuset.cpus doesn't. Docker's sysinfo.NumCPU() still sees all system CPUs.

2. Dynamic switching: cgroup cpuset can be modified at runtime without service restart. We can pin dockerd during pulls and restore after.

3. Both benefits achieved: Fast docker pulls (28.6s) + --cpus N container limits work.

Proposed Implementation

Replace CPUAffinity=0 in docker.service override with a wrapper script that:

1. Monitors for docker pull commands
2. Temporarily pins dockerd to CPU 0 during pulls via cpuset
3. Restores cpuset after pull completes

Or, simpler approach:

• Remove CPUAffinity=0 from docker.service
• Add cpuset pinning to critical commands (like image pull during boot)
• Let user containers have full CPU access by default

Would like feedback on which approach is preferred before implementing.

[kvinwang]

Deep Dive: Root Cause Analysis

TL;DR

The slowdown is caused by TDX IPI (Inter-Processor Interrupt) overhead. In TDX CVM, cross-vCPU communication is ~5.6x slower than on bare metal, and dockerd (Go) generates massive cross-core IPI traffic during image pulls.

Evidence

1. The problem is in dockerd, not containerd

docker cpuset	containerd cpuset	Pull time
0 (pinned)	0-7 (free)	28s
0-7 (free)	0 (pinned)	79s
0 (pinned)	0 (pinned)	24s
0-7 (free)	0-7 (free)	80s

Only dockerd's CPU affinity matters.

2. Cross-vCPU lock latency is 5.6x worse in TDX

Lock ping-pong test (Python threading.Lock, 50000 iterations):

CPU config	TDX CVM	Host	Ratio
Same CPU (0,0)	6.38 μs	5.96 μs	1.07x
Different CPUs (0,1)	60.84 μs	10.89 μs	5.6x

3. strace confirms: 68% time spent in futex

Multi-CPU pull (slow):

• futex: 23.5s total, 867 μs/call average, 27,120 calls

Single-CPU pull (fast):

• futex: 13.0s total, 410 μs/call average, 31,779 calls

4. IPI counts prove cross-core overhead

Mode	Pull time	RES (reschedule IPI)	CAL (function call IPI)
Multi CPU	12.2s	+582	+77,937
Single CPU	4.8s	+1,326	+1,941

Multi-CPU mode generates 40x more function call IPIs.

5. Context switches also doubled

Mode	Pull time	Context switches
Single CPU	4.7s	104,976
Multi CPU	14.7s	190,976

6. Host shows NO difference

docker save/load test with 300MB image on host: 6.4s pinned vs 6.4s unpinned - identical.

Root Cause

In TDX architecture, IPIs between vCPUs must go through the TDX Module (guest → TD Module → guest). This adds significant latency to every cross-vCPU synchronization operation.

Go programs like dockerd use many goroutines with frequent channel operations and mutex acquisitions, generating heavy futex/IPI traffic. When goroutines are spread across multiple vCPUs, each synchronization point pays the TDX IPI tax.

Pinning to a single CPU eliminates cross-vCPU IPIs entirely, making all synchronization local and fast.

Conclusion

CPUAffinity=0 is a valid and necessary workaround for a TDX architectural limitation. The --cpus incompatibility is a tradeoff. Possible mitigations:

1. Docker-side fix: Patch sysinfo.NumCPU() to read from /sys/devices/system/cpu/online instead of sched_getaffinity (upstream change)
2. Wait for TDX improvements: Future Intel TDX versions may optimize IPI handling
3. User workaround: Remove CPU pinning via pre_launch_script for apps that need --cpus