Skip to content

fix: per-network static IP assignment for multi-network containers#4758

Open
tariromukute wants to merge 4 commits intocontainerd:mainfrom
tariromukute:fix/scope-static-ip-per-network-in-cni-args
Open

fix: per-network static IP assignment for multi-network containers#4758
tariromukute wants to merge 4 commits intocontainerd:mainfrom
tariromukute:fix/scope-static-ip-per-network-in-cni-args

Conversation

@tariromukute
Copy link

Summary

When using nerdctl compose with services connected to multiple networks that each have a static IPv4 address (ipv4_address), all static IPs were passed to every CNI plugin via a shared CNI_ARGS (IP=<addr>). This caused failures because each bridge plugin would attempt to allocate an IP address that belongs to a different network's subnet.

This PR fixes multi-network static IP assignment by:

  1. Collecting per-network IPs and passing them as a JSON-encoded annotation (nerdctl/ip-per-network) instead of the single-value --ip= flag (which can only hold one IP).
  2. Using cnilibrary directly (instead of go-cni's Setup()) to set up each network individually with:
    • The correct container-side interface name (eth0, eth1, eth2, ...) — go-cni's WithConfListBytes always assigned eth0 to separate CNI instances.
    • Per-network CNI_ARGS containing only the IP for that specific network.
  3. Maintaining backward compatibility: single-network containers with a static IP still use the legacy --ip= flag and the existing go-cni Setup() path.

Changelog KeepA

[Fixed]:

  • Fixed nerdctl compose up failing for services with static IPs (ipv4_address) on multiple networks due to all IPs being passed to every CNI plugin via shared CNI_ARGS.
  • Fixed container-side interface name collision (eth0 already exists) when setting up multiple networks individually, by using cnilibrary.AddNetworkList/DelNetworkList directly with correct per-network IfName.

[Added]:

  • New nerdctl/ip-per-network annotation (JSON map of network name → static IPv4 address) for multi-network static IP propagation from compose to the OCI hook.
  • Direct cnilibrary based per-network CNI setup and teardown functions (perNetworkAdd, perNetworkDel) with correct interface naming.

Steps to Reproduce

  1. Create a compose file with a service attached to two networks, each with a static IP:
version: '3.8'
services:
  web:
    image: nginx:alpine
    ports:
      - "8080:80"
    networks:
      public_net_access:
        ipv4_address: 192.168.72.130
      public_net:
        ipv4_address: 192.168.70.130
    environment:
      - NGINX_HOST=web.local
      - NGINX_PORT=80

networks:
  public_net:
    driver: bridge
    name: demo-oai-public-net
    ipam:
      config:
        - subnet: 192.168.70.128/26
  public_net_access:
    name: oai-public-access
    ipam:
      config:
        - subnet: 192.168.72.128/26
  1. Run nerdctl compose up -d

Expected behaviour

Each network should only receive IP= and interface in it's corresponding network. Creation should be successful.

Actual behaviour

Both networks receive request to create the same interface and IP. See error below

332164/cid -l=com.docker.compose.project=examples -l=com.docker.compose.service=web -l=com.docker.compose.config-hash=b43a23c3c83aac9237234aee90ad8aa99bbf51c418d20d8f62941a24dc5a21d3 -d --name=examples-web-1 --pull=never -e=NGINX_HOST=web.local -e=NGINX_PORT=80 --net=demo-oai-public-net --ip=192.168.70.130 --net=oai-public-access --ip=192.168.72.130 --hostname=web -p=8080:80/tcp --restart=no nginx:alpine] 
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running createRuntime hook #0: exit status 1, stdout: , stderr: time="2026-02-27T10:02:07+01:00" level=warning msg="Container failed starting. Removing allocated network configuration."
time="2026-02-27T10:02:07+01:00" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): failed to allocate all requested IPs: 192.168.72.130" 
FATA[0000] error while creating container examples-web-1: error while creating container examples-web-1: exit status 1 

@AkihiroSuda
Copy link
Member

Thanks, but please add DCO sign
https://wiki.linuxfoundation.org/dco

dependabot bot and others added 4 commits February 27, 2026 23:59
Bumps [github.com/containerd/cgroups/v3](https://github.com/containerd/cgroups) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/containerd/cgroups/releases)
- [Commits](containerd/cgroups@v3.1.2...v3.1.3)

---
updated-dependencies:
- dependency-name: github.com/containerd/cgroups/v3
  dependency-version: 3.1.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Tariro Mukute <18515926+tariromukute@users.noreply.github.com>
Fix issue 4753

NOTE: used Claude Code

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Signed-off-by: Tariro Mukute <18515926+tariromukute@users.noreply.github.com>
@tariromukute tariromukute force-pushed the fix/scope-static-ip-per-network-in-cni-args branch from f7ee528 to 38d88d8 Compare February 27, 2026 23:01
@tariromukute
Copy link
Author

tariromukute commented Feb 27, 2026

Thanks, but please add DCO sign https://wiki.linuxfoundation.org/dco

I have done so, thanks.

Copy link
Member

@haytok haytok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for creating this PR! Would it be possible to have e2e tests using Tigron for this feature ?

Comment on lines +607 to 616
commonOpts := []cni.NamespaceOpts{}
commonOpts = append(commonOpts, portMapOpts...)
commonOpts = append(commonOpts, macAddressOpts...)
commonOpts = append(commonOpts, ip6AddressOpts...)
commonOpts = append(commonOpts,
cni.WithLabels(map[string]string{
"IgnoreUnknown": "1",
}),
cni.WithArgs("NERDCTL_CNI_DHCP_HOSTNAME", opts.state.Annotations[labels.Hostname]),
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT:
commonOpts is used for opts.cni.Setup, so these can be defined directly under // Legacy path: single IP (or no IP) shared across all networks.

if len(portMappings) > 0 {
rt.CapabilityArgs["portMappings"] = portMappings
}
result, err := cniConfig.AddNetworkList(ctx, confList, rt)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the new method, since AddNetworkList is called instead of Setup, the ips capability (for IPv6 static addresses) is not added to rt.CapabilityArgs, so it seems necessary to add them.

ips | Dynamically allocate IPs for container interface. Runtime which has the ability of address allocation can pass these to plugins. | ips | A list of IP (string entries). [ “10.10.0.1/24”, “3ffe:ffff:0:01ff::1/64” ] | none | CNI static plugin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants