feat(supervisor-middleware): add network egress middleware#2027
feat(supervisor-middleware): add network egress middleware#2027pimlock wants to merge 28 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
595191e to
97b750f
Compare
358906a to
1fbcdbc
Compare
|
🌿 Preview your docs: https://nvidia-preview-pr-2027.docs.buildwithfern.com/openshell |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
2 similar comments
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
/ok to test c4b0dcf |
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
…are outages An unreachable operator-registered middleware service previously aborted sandbox startup via a hard error in load_policy, contradicting the per-request on_error contract and the resilient live-reload path. Retry the initial connect and, on failure, degrade to the built-in registry so matched requests are governed by each config's on_error (deny for fail_closed, allow for fail_open) instead of blocking the whole sandbox. The policy poll loop now reconciles the registry on every poll while an install is pending, so a recovered service is adopted without waiting for a config change; a failed reconcile also no longer blocks unrelated policy updates. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
…limit A chain entry whose binding did not resolve reported a zero body limit, which dragged the whole chain's buffer cap to zero and spuriously failed body-bearing requests over capacity even when a resolved middleware could have processed them. Exclude unresolved entries from the limit via a new DescribedChainEntry::is_resolved(); when no entry resolves, skip buffering and apply each entry's on_error directly. Also fix two parallel-test flakes found while validating the change: - Build middleware OCSF events into a Vec and assert on it directly instead of capturing through the global tracing pipeline, whose callsite-interest cache is process-global and raced under parallel runs. - Accumulate the websocket deny response until the reason marker arrives rather than assuming a single read returns the full body. Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
c4b0dcf to
2b7cf4e
Compare
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
|
Label |
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
| Plan startup and updates around these boundaries: | ||
|
|
||
| - Start registered services before the gateway. The gateway validates every registration during startup. | ||
| - Keep service endpoints reachable from both the gateway and sandbox supervisors. The supervisors call operator-run services directly on the request path. |
There was a problem hiding this comment.
Clarifying question - What is the behavior when a service endpoint is unexpectedly or intermittently unavailable? This would be in the case where a on gateway and supervisor startups, the service was healthy, but then becomes unavailable later. What would happen?
There was a problem hiding this comment.
This depends on the on_error setting. If it's fail_open, the request will be sent upstream, if it's fail_closed (default), it will be rejected.
| // HttpRequestTarget describes the admitted HTTP destination and request target. | ||
| message HttpRequestTarget { |
There was a problem hiding this comment.
Again just out of curiosity - are generic headers in scope here or part of the target?
There was a problem hiding this comment.
The headers are currently modelled as part of the request itself, this HttpRequestTarget is supposed to represent "where" and the rest (the "what") is in the HttpRequestEvaluation.
The reason for that split was for HttpRequestTarget to represent what the network policy operates on when making its allow/deny decision.
| /// | ||
| /// Matching is case-insensitive. Invalid or empty patterns return an error | ||
| /// instead of silently becoming a non-match. | ||
| pub fn host_matches(pattern: &str, host: &str) -> std::result::Result<bool, String> { |
There was a problem hiding this comment.
Nit: Although we're using this to match a host, this is matching a general pattern to a string should we rename the function?
I'm also wondering if it make sense to introduce a Pattern type that wraps glob::Pattern and performs pattern validation on new?
| use miette::{Result, miette}; | ||
|
|
||
| /// Binding identifier for the built-in secret redaction middleware. | ||
| pub const BUILTIN_SECRETS: &str = "openshell/secrets"; |
There was a problem hiding this comment.
If this were go, I would consider implementing this as a type that implements some "Middleware" interface. Is something similar possible / idomatic in Rust?
| fn validate_secrets_config(config: &prost_types::Struct) -> Result<()> { | ||
| let mode = config | ||
| .fields | ||
| .get("secrets") | ||
| .and_then(|value| match value.kind.as_ref() { | ||
| Some(prost_types::value::Kind::StringValue(value)) => Some(value.as_str()), | ||
| _ => None, | ||
| }) | ||
| .unwrap_or("redact"); | ||
| if mode != "redact" { | ||
| return Err(miette!( | ||
| "{BUILTIN_SECRETS} only supports config.secrets: redact in phase 1" | ||
| )); | ||
| } | ||
| Ok(()) | ||
| } |
There was a problem hiding this comment.
It is not clear from this function which fields the openshell/secrets config actually has. Does this further justify a concrete type? (Something like what we do for the driver-config structs?)
| Some(prost_types::value::Kind::StringValue(value)) => Some(value.as_str()), | ||
| _ => None, | ||
| }) | ||
| .unwrap_or("redact"); |
There was a problem hiding this comment.
So the default value is redact even if no secrets field is present?
|
|
||
| #[test] | ||
| fn secrets_config_defaults_to_redact() { | ||
| validate_builtin_config(BUILTIN_SECRETS, &prost_types::Struct::default()).unwrap(); |
There was a problem hiding this comment.
I don't actually see redact mentioned in this test at all.
| if endpoint.tls == "skip" | ||
| && selector_matches_host(middleware, &endpoint.host).unwrap_or(false) | ||
| { |
There was a problem hiding this comment.
[P1] Detect tls-skip conflicts against wildcard endpoints
When a tls: skip endpoint uses a wildcard host such as *.example.com and a middleware selector includes a concrete host such as api.example.com, this compares the selector against the literal endpoint pattern, so validation passes even though real traffic for api.example.com is admitted without TLS inspection and the required middleware cannot run. The conflict check needs to reason about selector/endpoint pattern overlap, not only whether the selector matches the endpoint string.
| let chain = engine.query_middleware_chain(&middleware_network_input(ctx))?; | ||
| let req = match apply_middleware_chain( | ||
| req, | ||
| client, | ||
| ctx, | ||
| chain, | ||
| engine.middleware_runner(), | ||
| engine.generation_guard(), | ||
| ) |
There was a problem hiding this comment.
It looks like L7 policy is evaluated before middleware. So I'd imagine that the middleware could modify or replace an allowed operation with one that would otherwise be denied by policy. Is this intended? If so we should clearly document this.
There was a problem hiding this comment.
Great point.
As discussed - we want the policy to be guaranteed by openshell, so we need to add a check after any middleware hook (at least if the body/headers were modified by the middleware). I will make this change.
| &mut tls_upstream, | ||
| &ctx, | ||
| &generation_guard, | ||
| Some(&opa_engine), |
There was a problem hiding this comment.
[P1] Matching middleware still does not make CONNECT inspection mandatory. should_inspect_l7 above is derived only from the endpoint L7 route. For an L4-only endpoint with a matching fail-closed chain, an unsupported HTTP method or h2c preface can take the raw-copy path without invoking middleware. Please include the selected middleware chain when deciding whether inspection is required and fail closed for uninspectable protocols.
|
|
||
| if allowed || (config.enforcement == EnforcementMode::Audit && !force_deny) { | ||
| let chain = engine.query_middleware_chain(&middleware_network_input(ctx))?; | ||
| let req = match apply_middleware_chain( |
There was a problem hiding this comment.
[P1] Body-aware policy has already evaluated the original JSON-RPC request before this middleware call. An ALLOW result can replace that body with a locally denied operation, and the replacement is forwarded below without reparsing or reevaluation. Please run transformations before body-aware policy evaluation, or reparse and reevaluate the final body.
| }; | ||
| for endpoint in &rule.endpoints { | ||
| if endpoint.tls == "skip" | ||
| && selector_matches_host(middleware, &endpoint.host).unwrap_or(false) |
There was a problem hiding this comment.
[P1] endpoint.host may itself be a wildcard pattern, so treating it as the concrete host argument does not detect selector-pattern intersection. For example, tls: skip on *.example.com and middleware selecting api.example.com pass validation even though runtime traffic is uninspectable. Please conservatively detect pattern overlap; the corresponding runtime validation needs the same fix.
| // operator service is picked up without waiting for a policy change. | ||
| // Failure preserves the last-known-good registry and does not block | ||
| // the remaining config updates below. | ||
| reconcile_middleware_registry( |
There was a problem hiding this comment.
[P1] The registry is reconciled before the new OPA policy is committed. Concurrent requests can therefore evaluate the old policy against the new registry. If policy reload fails, the old policy remains while the hash is still advanced below, making the mismatch persistent and allowing an unresolved fail-open guard to be skipped. Please commit policy and registry atomically, or roll back and retry both together.
| return Err("host pattern must not contain whitespace".to_string()); | ||
| } | ||
|
|
||
| let pattern = glob::Pattern::new(&pattern.to_ascii_lowercase()) |
There was a problem hiding this comment.
[P1] glob::Pattern allows * to cross dots, but the documented network endpoint semantics use * for one DNS label and ** for multiple labels. This can make an exclusion such as *.example.com unexpectedly suppress middleware for deep.api.example.com. Please reuse the label-aware DNS matcher and add multi-label exclusion coverage.
| if name.is_empty() | ||
| || matches!( | ||
| name.as_str(), | ||
| "authorization" | "cookie" | "host" | "content-length" | "transfer-encoding" |
There was a problem hiding this comment.
[P1] Proxy-Authorization is not filtered here. For non-CONNECT forward-proxy requests, remote middleware can receive Basic proxy credentials before a later forwarding step strips the header. Please remove proxy credential headers before constructing the middleware evaluation.
There was a problem hiding this comment.
Fixed in f4c0206. Proxy-Authorization is now filtered before constructing the middleware request, with regression coverage for both origin and proxy credentials.
| // Destination and HTTP request target. | ||
| HttpRequestTarget target = 6; | ||
| // HTTP request headers before OpenShell injects credentials. | ||
| map<string, string> headers = 7; |
There was a problem hiding this comment.
[P1] A map cannot represent repeated HTTP headers. The current parser keeps only the last value for middleware, while request rebuilding preserves all original values upstream. That creates a first-value-versus-last-value authorization differential. Please preserve repeated header values in the contract or reject ambiguous duplicates before evaluation.
| .iter() | ||
| .filter(|entry| entry.is_resolved()) | ||
| .map(openshell_supervisor_middleware::DescribedChainEntry::max_body_bytes) | ||
| .min() |
There was a problem hiding this comment.
[P2] Taking the smallest cap for the entire chain loses per-stage on_error behavior. A 2 KiB request with a 1 KiB fail-open stage followed by a 1 MiB fail-closed stage is denied before the second stage runs, even though only the first stage should be skipped. Please buffer for eligible later stages and apply capacity failures per stage.
| "request body credential rewrite buffers at most {MAX_REWRITE_BODY_BYTES} bytes" | ||
| )); | ||
| } | ||
| if body.len().saturating_add(chunk_size) > MAX_REWRITE_BODY_BYTES { |
There was a problem hiding this comment.
[P2] This fixed credential-rewrite ceiling also applies to middleware chunked-body collection. A binding advertising a 1 MiB limit still rejects a 300 KiB chunked payload here. Please separate these limits or pass the negotiated middleware limit through the collector.
There was a problem hiding this comment.
Fixed in f4c0206. Chunked middleware collection now uses the negotiated middleware body limit instead of the 256 KiB credential-rewrite ceiling; a 300 KiB payload under a 1 MiB binding cap is covered.
| serde_json::Value::Bool(value) => Kind::BoolValue(value), | ||
| serde_json::Value::Number(value) => Kind::NumberValue( | ||
| value | ||
| .as_f64() |
There was a problem hiding this comment.
[P2] as_f64 accepts integers that cannot be represented exactly. For example, 9007199254740993 is delivered as 9007199254740992, which can change tenant IDs or allow/deny thresholds. Please reject values that do not round-trip exactly or use a lossless config representation.
There was a problem hiding this comment.
Fixed in f4c0206. Integer config values that cannot round-trip exactly through protobuf double storage are now rejected, with coverage around the 2^53 boundary.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
| // MiddlewareManifest describes one service and the bindings it exposes. | ||
| message MiddlewareManifest { | ||
| // Middleware protocol version implemented by the service. | ||
| string api_version = 1; |
There was a problem hiding this comment.
this probably isn't needed since the proto itself is versioned
| // Human-readable service name used for diagnostics. | ||
| string name = 2; | ||
| // Service-defined version string used for diagnostics. | ||
| string service_version = 3; |
There was a problem hiding this comment.
what is service in this context?
| // ValidateConfigRequest contains one policy configuration to validate. | ||
| message ValidateConfigRequest { | ||
| // Middleware protocol version selected by OpenShell. | ||
| string api_version = 1; |
There was a problem hiding this comment.
i dont think this is necessary if we're using protos
| // HttpRequestEvaluation contains one buffered HTTP request to evaluate. | ||
| message HttpRequestEvaluation { | ||
| // Middleware protocol version selected by OpenShell. | ||
| string api_version = 1; |
There was a problem hiding this comment.
same as above, the proto already has a version
| use miette::{Result, miette}; | ||
|
|
||
| /// Binding identifier for the built-in secret redaction middleware. | ||
| pub const BUILTIN_SECRETS: &str = "openshell/secrets"; |
There was a problem hiding this comment.
is there a way to decouple this so that we don't hardcode middleware into openshell-core? this winds up on the client and other places i think.
maybe a redaction middleware crate that is configured into the server and supervisor?
| .iter() | ||
| .filter(|entry| entry.is_resolved()) | ||
| .map(openshell_supervisor_middleware::DescribedChainEntry::max_body_bytes) | ||
| .min() |
There was a problem hiding this comment.
(agent finding)
[P2] Do not cap pre-buffering at the smallest middleware limit
When a chain mixes middleware with different max_body_bytes values, using the minimum here makes the relay treat any body larger than the smallest later middleware as unbufferable before ChainRunner can run earlier middleware that can handle it. For example, a 100 KB request with a first 256 KB redactor and a later 4 KB guard will either be denied or passed through unprocessed depending on on_error, instead of running the redactor and then applying the guard's per-entry capacity handling.
There was a problem hiding this comment.
nit: consider calling this supervisor_middleware.proto.
| // gRPC endpoint reachable from the sandbox supervisor. | ||
| string grpc_endpoint = 2; | ||
| // Operator-owned body limit applied to every binding exposed by the service. | ||
| uint64 max_body_bytes = 4; |
There was a problem hiding this comment.
Agent review: SupervisorMiddlewareService is introduced in this PR and doesn't exist on main, so there are no deployed supervisors to maintain wire compatibility with. Renumber max_body_bytes from 4 to 3 to close the gap before this ships — a future contributor will otherwise reuse field 3 and silently break compatibility with any supervisors running this version.
| { | ||
| return false; | ||
| } | ||
| name.starts_with("x-openshell-middleware-") |
There was a problem hiding this comment.
Agent review: is_safe_append_header uses a default-deny allowlist (correct security posture), but the required x-openshell-middleware-* prefix is not documented in the proto comment on add_headers, in the operator docs, or in the error message. An operator whose service returns x-correlation-id will get unsafe_response_headers routed through on_error with no explanation of what went wrong. The error at the validate_header_mutations call site should name the required prefix, and the proto comment and extensibility docs should document this constraint explicitly.
| @@ -903,6 +949,42 @@ where | |||
| let _ = &eval_target; | |||
|
|
|||
| if allowed || config.enforcement == EnforcementMode::Audit { | |||
There was a problem hiding this comment.
Agent review: when enforcement == Audit, middleware still runs and a Denied result sends a hard 403 to the client. Audit mode is defined as log-but-not-block — a middleware deny should be recorded as an OCSF finding and the request allowed through, not surfaced to the client as a rejection. Either the MiddlewareApplyResult::Denied arm should be treated as an audit finding here (and at the equivalent sites at lines 462, 1226, and 1456), or this intentional override of audit semantics needs to be explicitly documented and covered by a test.
| use miette::{Result, miette}; | ||
|
|
||
| /// Binding identifier for the built-in secret redaction middleware. | ||
| pub const BUILTIN_SECRETS: &str = "openshell/secrets"; |
There was a problem hiding this comment.
Non-blocking / follow-up suggestion.
Built-ins are currently treated structurally differently from external services: they bypass connect_services, are hardwired into IN_PROCESS_SERVICE, and require special-case branches in ensure_policy_bindings_registered and policy validation (== BUILTIN_SECRETS). Adding a second built-in also requires touching match arms in two separate places across two crates (openshell-core::middleware::validate_builtin_config and builtins/mod.rs).
A trait-based approach would address this: a BuiltinMiddleware trait (covering binding_id, describe, validate_config, evaluate) with a factory function instantiate(id) -> Option<&'static dyn BuiltinMiddleware> as the single registration point. Built-ins would self-register through the same connect_services path as externals, seeding the binding ID dedup set naturally and removing the special-case branches. Adding a new built-in would then only require implementing the trait and adding one arm to instantiate.
Not a blocker for this PR, but worth tracking before a second built-in lands.
Summary
Implements the first usable RFC 0009 supervisor middleware slice: proto-backed, host-selected HTTP egress middleware for
HttpRequest/pre_credentials, with both in-process built-ins and statically registered operator-run gRPC services.The implementation covers RFC 0009 Phase 1 and adds basic external-service support from Phase 2. It establishes the contract, policy plumbing, ordered chain execution, built-in secret redaction, static gateway registration, relay integration, validation before policy persistence, body limits, audit events, and user-facing configuration and operations documentation.
Tip
See example middleware implementation: #2169
Related Issue
Closes #2010
Part of #1733
Design/RFC: #1738
Changes
openshell/secretsredactor and statically registered operator-run gRPC services.network_middlewarespolicy configuration and validation, independent of the network policy rule that admits a request.Testing
mise run pre-commitpassesChecklist