feat(system-tests): make the local backend's ic-gateway reachable by name - #11270
Draft
basvandijk wants to merge 4 commits into
Draft
feat(system-tests): make the local backend's ic-gateway reachable by name#11270basvandijk wants to merge 4 commits into
basvandijk wants to merge 4 commits into
Conversation
…name
On the local backend the ic-gateway served a self-signed certificate for
`<vm name>.local`. Both halves are unusable from inside the group, so every
client papered over them with a `reqwest` resolve override plus
`danger_accept_invalid_certs`. That works for driver-side clients and for
nothing else -- a *node* has no such knobs, which is why a nested node
registers straight against a replica's `:8080` on this backend instead of
going through the gateway as it does on Farm.
Both halves turn out to be fixable without touching any node.
The domain moves to `<vm name>.ic.net` and is registered with the group's
`dnsmasq` through `add_dns_record`, the same mechanism
`setup_api_bn_local_playnet` already uses for `apibn-{idx}.ic.net`; the suffix
becomes one shared constant. It could not have stayed under `.local`: GuestOS
and HostOS resolve through systemd-resolved, which routes `*.local` to mDNS and
never to the unicast `DNS=` servers `dnsmasq` answers on, so no amount of
dnsmasq configuration would make such a name resolve in a guest.
The certificate is now issued by the dev root CA instead of being self-signed.
That CA is already in every *dev* IC-OS image's trust store --
`ic-os/{guestos,hostos}/context/Dockerfile` installs
`canister_http_test_ca.cert` and runs `update-ca-certificates` in the
`output_dev` stage -- and its private key is checked in beside it, which is how
`canister_http` and `ckbtc` already serve HTTPS that nodes accept. So a node
can reach the gateway with no node-side configuration at all, and no production
code and no IC-OS config has to change.
This deliberately does not extend to the API boundary nodes, whose ephemeral CA
stays as it is: `nns_delegation_manager` builds its root store from the
compiled-in `webpki_roots` and never consults `/etc/ssl/certs`, which is
precisely why `extra_api_boundary_node_trust_anchors_pem` exists.
Driver-side, `uses_self_signed_cert` gives way to `root_certificate()`, since
`reqwest` adds an extra root to the platform's own rather than replacing them:
clients now verify the chain instead of skipping verification.
`resolve_override_for_url` stays. The driver sits in the group's network
namespace but reads the host's `/etc/resolv.conf`, whose nameserver is
unreachable from there, so driver-side DNS is dead regardless of what `dnsmasq`
knows; fixing that means giving the driver a mount namespace, and is left for
its own change.
Adding `x509-parser` to `rcgen`'s features is what makes
`CertificateParams::from_ca_cert_pem` available, and the new `rsa` dependency
re-encodes the checked-in PKCS#1 CA key as the PKCS#8 that `rcgen`'s `ring`
backend requires.
Verified on the local backend: `dfx_smoke_test_local` and `nns_dapp_test_local`
both pass, fetching canister assets over `https://<canister
id>.ic-gateway.ic.net` with the chain verified rather than accepted blindly.
`dfx_smoke_test` and `nns_dapp_test` still pass on Farm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Enables locally hosted IC gateways to use in-group DNS and CA-verified TLS.
Changes:
- Registers local gateway domains through
dnsmasq. - Issues gateway certificates from the dev root CA.
- Updates clients and Rust/Bazel dependencies for certificate verification.
Reviewed changes
Copilot reviewed 10 out of 13 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
rs/tests/sdk/src/asset.rs |
Verifies gateway TLS certificates. |
rs/tests/nns/nns_dapp_test.rs |
Verifies TLS for dapp requests. |
rs/tests/driver/src/driver/local_backend.rs |
Defines the shared in-group domain. |
rs/tests/driver/src/driver/ic.rs |
Reuses the shared domain suffix. |
rs/tests/driver/src/driver/ic_gateway_vm.rs |
Adds CA signing, DNS registration, and trust handling. |
rs/tests/driver/Cargo.toml |
Adds RSA support. |
rs/tests/driver/BUILD.bazel |
Adds the Bazel RSA dependency. |
rs/tests/common.bzl |
Provides CA certificate and key runfiles. |
Cargo.toml |
Enables rcgen X.509 parsing. |
Cargo.lock |
Updates Cargo dependency resolution. |
Cargo.Bazel.toml.lock |
Updates Bazel’s Cargo lock state. |
Cargo.Bazel.json.lock |
Updates generated Bazel crate metadata. |
bazel/rust.MODULE.bazel |
Enables rcgen X.509 parsing for Bazel. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…cate A wildcard SAN matches a single label, so `*.<domain>` does not cover `<canister id>.raw.<domain>` -- the canonical raw hosts, which the Farm path gives their own `*.raw` CNAME. Add the matching SAN so the local certificate covers what Farm's does. Nothing is broken today: the only three places that build a raw host are two Prometheus scraping-target files, consumed inside a universal VM with its own trust store, and one nns-dapp metadata string for a canister the driver does not deploy. But the gap is a trap for the next caller, and more so once the driver-side resolve overrides go away. Note the certificate is now ahead of DNS here: the group's `dnsmasq` is served a hosts file, which has no wildcards, so only the apex is registered and a subdomain still has to be resolved by the client. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dev root CA was added to `IC_GATEWAY_RUNTIME_DEPS`, which feeds *every* variant of a gateway-using target -- Farm and `_colocate` included -- even though only the local backend ever reads it. For a colocated target that is worse than dead weight: `colocate_test.rs` tars the runtime-deps directory and copies it to the driver's universal VM, so around thirty Farm targets were shipping a CA signing key to a VM with no use for it. The key is public, so nothing is compromised, but there is no reason to send it. Move both files to `_local_only_deps` in `system_tests.bzl`, which is plumbed into the `_local` variant alone, and rename the variables to the `ENV_DEPS__` prefix that channel uses. Verified through the runfiles manifests: `dfx_smoke_test_local` has both entries, `dfx_smoke_test` has neither, and both still pass -- the Farm path never touches them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five claims in the comments added with this feature were wrong. None changes behaviour; all of them would mislead the next reader. - The PKCS#1 re-encode was justified by `rcgen`'s "default `ring` backend". `aws_lc_rs` is in fact enabled workspace-wide (`rustls-acme` -> `ic-bn-lib` -> `ic-boundary` -> `pocket-ic-server`) and `rcgen` prefers it when both features are on, so the backend in use loads PKCS#1 directly and the conversion is not strictly needed. Keep it -- correctness should not hinge on a transitive feature of an unrelated crate, where a dependency edit surfaces as a runtime failure in every local gateway test -- but say that instead. - `signed_by` was said to write the CA's subject key identifier into the leaf's authority key identifier. `rcgen` emits an AKI only when `use_authority_key_identifier_extension` is set, which `CertificateParams::new` leaves false; the leaf's only extension is its SAN. The issuer name and the signature are what make the chain verify. - Serving the root in `chain_pem` was justified as sparing the client an AIA fetch. There is no intermediate to fetch, and RFC 8446 4.4.2 lets the root be omitted, so a trusting client gains nothing. It is kept because it makes the served chain self-describing while debugging a handshake. - `root_certificate` claimed no CA vouches for an API boundary node's certificate. Under `with_api_boundary_nodes_playnet` one does. The reason those clients still need `danger_accept_invalid_certs` is that `IcNodeSnapshot::get_public_url` gives them an IP-literal URL, which no name in the certificate can match. - Apex-only DNS registration was justified with "nothing inside a VM resolves a canister subdomain today". `prometheus_vm` writes `<canister id>.raw.<domain>` scrape targets that the Prometheus VM resolves through this very `dnsmasq`. They have never resolved on this backend -- the domain was an equally unresolvable `.local` name before -- so nothing regressed, but the comment asserted the opposite of the code it was justifying. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the local backend the ic-gateway served a self-signed certificate for
<vm name>.local. Neither half is usable from inside the group, so every client papered over both — areqwestresolve override plusdanger_accept_invalid_certs.That works for driver-side clients and for nothing else. A node has no such knobs, which is why a nested node on this backend registers straight against a replica's
:8080instead of going through the gateway as it does on Farm — and why it needswith_group_wide_firewall_whitelist()to get there.Both halves turn out to be fixable without touching any node.
The domain
<vm name>.ic.net, registered with the group'sdnsmasqthroughadd_dns_record— the same mechanismsetup_api_bn_local_playnetalready uses forapibn-{idx}.ic.net. The suffix becomes one shared constant.It could not have stayed under
.local: GuestOS and HostOS resolve through systemd-resolved, which routes*.localto mDNS and never to the unicastDNS=serversdnsmasqanswers on. No amount of dnsmasq configuration would make such a name resolve in a guest.Only the apex is registered — a hosts file has no wildcards, and nothing inside a VM resolves a canister subdomain today. A wildcard would mean
--address=/<domain>/<addr>on dnsmasq's command line, whichSIGHUPdoes not re-read.The certificate
Issued by the dev root CA instead of self-signed. That CA is already in every dev IC-OS image's trust store —
ic-os/{guestos,hostos}/context/Dockerfileinstallscanister_http_test_ca.certand runsupdate-ca-certificates, in theoutput_devstage only — and its private key is checked in beside it. This is howcanister_httpandckbtcalready serve HTTPS that nodes accept, and thedev-certs/README.mddocuments exactly this use.So a node reaches the gateway with no node side configuration: no config-image plumbing, no
CONFIG_VERSIONbump, no production code change. Production images are unaffected — the CA lives in theoutput_devstage.This deliberately does not extend to the API boundary nodes, whose ephemeral CA stays as it is.
nns_delegation_managerbuilds its root store from the compiled-inwebpki_rootsand never consults/etc/ssl/certs, which is precisely whyextra_api_boundary_node_trust_anchors_pemexists; the client that reaches the gateway is a different one (the orchestrator'sic-agent, which goes throughrustls-platform-verifier→ the OS store).Driver side
uses_self_signed_certgives way toroot_certificate().reqwestadds an extra root to the platform's own rather than replacing them, so clients now verify the chain instead of skipping verification.resolve_override_for_urlstays: the driver sits in the group's network namespace but reads the host's/etc/resolv.conf, whose nameserver is unreachable from there, so driver-side DNS is dead regardless of whatdnsmasqknows. Fixing that means giving the driver a mount namespace, which is left for its own change.Two mechanical additions:
x509-parserinrcgen's features, forCertificateParams::from_ca_cert_pem; andrsa, to re-encode the checked-in PKCS#1 CA key as the PKCS#8 thatrcgen'sringbackend requires.Verification
dfx_smoke_test_localandnns_dapp_test_localboth pass, fetching canister assets overhttps://<canister id>.ic-gateway.ic.netwith the chain verified rather than accepted blindly.dfx_smoke_testandnns_dapp_teststill pass on Farm.#11265 will be restacked on top of this, dropping its plain-HTTP nested path and the group-wide firewall whitelist that goes with it.
🤖 Generated with Claude Code