learn: HTTP/HTTPS observation via transparent proxy

[ghazariann]

Covers the HTTP observation item from the RFC #72 (http.allow: observed method+host+path rules, gated by --learn-http).

New arguments

Flag	Effect
--learn-http	Required gate. Spawns proxy in passthrough mode, writes [http] section. Port 80 always intercepted.
--http-inject-ca <path>	Adds HTTPS (port 443) via ephemeral CA injection into the named trust bundle.
--http-port <port>	Additional port to intercept (repeatable).

--learn-http must be passed explicitly: --http-inject-ca and --http-port alone do not activate observation.

What this does

sandlock learn now records HTTP traffic (method + host + path) and writes it as [http].allow rules in the profile. The --http-inject-ca path is also written to [config].http_inject_ca so sandlock run picks it up automatically.

[config]
http_inject_ca = ["/etc/ssl/certs/ca-certificates.crt"]

[http]
ports = [80, 443]
allow = ["GET api.openai.com/v1/models"]

How it works

An optional log_fn callback was added to AclService in the transparent proxy. In learn mode the closure pushes "METHOD host/path" into LearnObserver::http_requests. The proxy previously only spawned when ACL rules were present, the condition was extended to also spawn when log_fn is set (passthrough, no rules enforced).

CLI

# Plain HTTP (port 80)
sandlock learn --learn-http -o profile.toml -- \
    curl -s http://google.com/

# HTTPS (port 443, ephemeral CA injected into system bundle)
sandlock learn --learn-http \
    --http-inject-ca /etc/ssl/certs/ca-certificates.crt \
    -o profile.toml -- \
    curl -s https://google.com/

# Non-standard port
sandlock learn --learn-http --http-port 8080 -o profile.toml -- \
    curl -s http://localhost:8080/healthz

Discussion

Follows the design stance in original RFC #72: keep learn loose, minimal new arguments.

Port learning: at connect() time only IP+port are visible. Port 80 is always included; port 443 requires --http-inject-ca; anything else requires --http-port. The proxy can only intercept ports it is told to listen on upfront. Can --http-port be removed by intercepting all TCP ports and detecting HTTP dynamically (learning HTTP ports)?

CA: --http-inject-ca <bundle> is the preferred mode. --http-ca/--http-key (BYO CA) is not wired in LearnArgs: the inject approach covers the same use case with less friction. Is this acceptable?

Other config-like arguments not wired in learn: --http-ca-out (write ephemeral CA to a path) and --http-auth (credential injection) are sandlock run flags with no learn equivalent. For Node.js workloads, HTTPS interception also requires NODE_EXTRA_CA_CERTS pointing to the sandlock CA via --env. Should these arguments be added to LearnArgs?

Tests

• test_learn_captures_http_request: single GET recorded with correct path
• test_learn_captures_http_multiple_paths: multiple paths deduplicated
• test_learn_captures_https_request: --http-inject-ca written to [config]
• test_learn_then_run_http (integration): learned path allowed; unlearned path blocked (403)
• test_learn_then_run_https (integration): HTTPS round-trip, [config] picked up by run automatically

[congwang-mk]

Thanks for the PR! One quick question: since this is only for sandlock learn, can we remove --learn-http and maybe use --http-port or --http-inject-ca as the gate?

[ghazariann]

Thanks for the PR! One quick question: since this is only for sandlock learn, can we remove --learn-http and maybe use --http-port or --http-inject-ca as the gate?

Agreed on keeping args minimal, but plain HTTP (port 80, no CA, no extra ports) needs neither --http-inject-ca nor --http-port, so there's nothing left to gate on. Do we really need the gate in the first place? Does the proxy add enough overhead to justify an explicit opt-in, or can we spawn it by default?

[ghazariann]

Thanks for the PR! One quick question: since this is only for sandlock learn, can we remove --learn-http and maybe use --http-port or --http-inject-ca as the gate?

Agreed on keeping args minimal, but plain HTTP (port 80, no CA, no extra ports) needs neither --http-inject-ca nor --http-port, so there's nothing left to gate on. Do we really need the gate in the first place? Does the proxy add enough overhead to justify an explicit opt-in, or can we spawn it by default?

Another argument for making it default: without HTTP learning, sandlock learn records exact IPs in [network].allow (e.g. curling google resolves to tcp://142.251.46.78:443), which might break on IP rotation. With HTTP default, extend_net_allow_for_http generates hostname-based rules re-resolved fresh on every learned sandlock run. And since sandlock run only enables the HTTP proxy when [http].allow and [http].ports are non-empty in the profile, workloads that don't use HTTP pay no cost.

[congwang-mk]

Thanks for the update! A few issues below:

1. Learned [http].ports can disagree with what was actually intercepted

crates/sandlock-cli/src/learn.rs:684 recomputes the port list by hand as [80] + 443 (if --http-inject-ca) + every --http-port. The builder does not merge those: crates/sandlock-core/src/sandbox/builder.rs:962 synthesizes the [80, 443] default only when self.http_ports is empty, and otherwise takes the user list verbatim.

So sandlock learn --learn-http --http-port 8080 -- app intercepts port 8080 only. Port 80 traffic is never proxied and never recorded, yet the emitted profile says ports = [80, 8080] with a non-empty allow. At sandlock run time port 80 is now intercepted, and http_acl_check is default-deny once allow is non-empty (crates/sandlock-core/src/http.rs:147), so a port-80 request that succeeded during learn gets a 403. Same for 443 when --http-inject-ca is combined with --http-port.

Suggested fix: record the effective ports from the built policy (policy.http_ports, which is what profile::to_profile already uses at crates/sandlock-core/src/profile.rs:526) instead of recomputing them. The new line in docs/learn.md ("Port 80 is always intercepted") is wrong for the same reason.

2. --merge silently discards everything HTTP that was just learned

The merge block starts from profile_out = existing.clone() (learn.rs:719) and then unions back only filesystem.read/write, network.allow, network.allow_bind, and limits. http.allow, http.ports, and config.http_inject_ca are never carried over, so sandlock learn --learn-http --merge base.toml writes a profile with no learned HTTP rules and reports no error. test_learn_then_run_merge has no HTTP variant, so nothing catches it.

3. The new flags' help text claims an implication the code does not implement

--http-inject-ca and --http-port are documented in main.rs as "(implies --learn-http)", while the PR description says the opposite and the code agrees with the description. Both failure modes are poor:

• --http-inject-ca without --learn-http fails in builder.rs:869 with --http-inject-ca requires --http-allow or --http-deny, naming two flags that do not exist on sandlock learn.
• --http-port without --learn-http is a silent no-op: http_acl_addr is None, so network/connect.rs:91 never redirects.

Adding requires = "learn_http" to the three new flags fixes both, and the builder message should mention the log-callback case now that it is a legitimate way to satisfy the check.

4. Neither "HTTPS" test makes an HTTPS request

test_learn_captures_https_request and test_learn_then_run_https both build http://127.0.0.1:{port}/... URLs against the plaintext spawn_http_server, so they assert only that the http_inject_ca path is echoed into [config]. The PR description lists the latter as an "HTTPS round-trip". CA injection, 443 interception, and MITM signing in learn mode are therefore untested, which is the part of the feature most likely to break. Both tests also return silently when no system CA bundle is found, so on such a host they pass without testing anything.

5. The log_fn call site does not match its own doc comment

service.rs:45 says the callback fires "after the host is validated, before the ACL check", but the call at service.rs:185 happens before verify_host at line 189. Effect: a request the proxy then rejects with "Host header does not match connection destination" still gets baked into the learned profile as an allow rule. Moving the call below verify_host matches both the comment and the intent.

Smaller things

• In learn_test.rs the three per-test use std::io::Read; statements are redundant with the new module-level import and will warn as unused.
• The doc comment on spawn_http_server says it "accepts one connection then exits" while the body loops over incoming(), and the inline comment two lines down says the opposite.
• --env is applied to the observed process but never written into the profile, so a workload needing NODE_EXTRA_CA_CERTS at learn time will not get it at run time. Worth at least a doc line, since the PR raises the question itself.

On the --learn-http gate discussion

The argument that HTTP learning yields hostname rules instead of the rotation-fragile literal IPs currently written to [network].allow is a strong one. If HTTP observation becomes the default, note that issues 1 and 3 largely dissolve, since there would no longer be two independent notions of "which ports are intercepted".

[ghazariann]

Addressed all review points:

1. 38aee48 - learn.rs now reads policy.http_ports from the built policy instead of recomputing by hand.

2. 798012f - merge block now unions http.allow, http.ports, and config.http_inject_ca from the observed run into the merged profile. ec65195 extends this further: program.env is unioned (observed wins on conflict) and config.http_ca_out is carried when the existing profile has none.

3. 33a5351 - dropped --learn-http entirely; HTTP observation is now always on so there is no gate to imply.

4. 0dae1d3 - test_learn_captures_https_request and test_learn_then_run_https rewritten to curl https://example.com/ through the MITM proxy, exercising real TLS, cert signing, CA injection, and proxy forwarding. Also fixed a bug uncovered while writing these tests: in the test harness curl resolved its CA bundle to /usr/lib/ssl/cert.pem (a symlink to /etc/ssl/certs/ca-certificates.crt), while in a normal shell it uses the canonical path directly. ca_inject::path_matches did exact-string comparison, so the symlink path never matched the inject path, injection was silently skipped, and curl exited 60. Fixed by canonicalizing both resolved and each inject path before comparing.

5. 2772bf4 - log_fn call moved below verify_host.

Smaller things (redundant use std::io::Read;, wrong spawn_http_server doc comment, --env not written to profile) - fixed in ec65195 and 0dae1d3.