PR #262 bounds how much any single request may buffer before its signature is checked. It does not bound how many such requests may be in flight, or how long each may take to send its body. Those two gaps are what remain, and together they still let an unauthenticated caller exhaust a node's memory.
What is left after #262
Each request is now capped at its route group's limit, and a body exactly at the limit is still read in full before rejection. That is asserted by a test #262 ships (empty_and_exactly_at_limit_bodies_behave_as_before, which checks polled == LIMIT and a 401). So a caller with no DID, no signature, and no existing repo can still make the node hold the full group limit, per connection, for as long as it keeps the socket dripping.
Measured with a probe (16 concurrent unsigned POSTs to /{owner}/{repo}/git-receive-pack, each parked on a shared barrier just before its last body frame, so all 16 were provably in flight at once):
N=16 limit=32MiB baseline_rss=8MiB peak_rss=287MiB delta=279MiB
status=401 Unauthorized (x16)
279 MiB resident from sixteen requests that were all then rejected. The memory is spent entirely before authentication.
Against the deployed configuration this is small numbers, not large ones:
- Production machines are
memory = 1024 MB (infra/fly/fly.toml and both siblings).
- Production sets
GITLAWB_MAX_PACK_BYTES = 524288000 (500 MB), so two concurrent unsigned requests to a git write route reach the machine's memory.
- Other signed groups are capped at
SIGNED_BODY_LIMIT (2 MB). Fly's hard_limit = 500 connections permits 500 x 2 MB = 1000 MB, which is also the whole machine.
The connection cap exists, but it is sized above what the box can hold, so it does not act as a mitigation on either path.
Why the existing controls do not cover it
rate_limit_by_ip counts arrivals in a rolling window and never decrements on completion (crates/gitlawb-node/src/rate_limit.rs), so it bounds request rate, not concurrency. The push bucket is 600 per hour; this needs about three.
idle_timeout = 120 in the Fly configs is an idle timeout. A connection sending one byte every few seconds is never idle.
There is no request-duration or body-read timeout in process. tower-http is compiled with features ["cors", "trace", "request-id", "limit"], so tower_http::timeout is not built, and axum::serve is called with no hyper builder configuration. The one tokio::time::timeout in git/smart_http.rs bounds the git child process, which is downstream of buffering and after auth.
Note for anyone triaging against #174: that PR's read/write semaphores and per-caller caps are all acquired inside handlers, and require_signature is a middleware layer that runs before any handler, so they sit downstream of this buffer and do not close it.
What would close it
A request-duration or body-read timeout on the routes that buffer, so a slow sender cannot hold a buffer indefinitely, and a concurrency bound sized against actual machine memory rather than above it. Both want sizing against real traffic, which is why they were left out of #262 rather than guessed at.
Streaming the receive-pack body to disk instead of buffering would remove the git-path term entirely, but that is a larger change: the body is currently parsed for ref updates before the authorization checks, scanned for a side-band capability, and written to the child's stdin.
PR #262 bounds how much any single request may buffer before its signature is checked. It does not bound how many such requests may be in flight, or how long each may take to send its body. Those two gaps are what remain, and together they still let an unauthenticated caller exhaust a node's memory.
What is left after #262
Each request is now capped at its route group's limit, and a body exactly at the limit is still read in full before rejection. That is asserted by a test #262 ships (
empty_and_exactly_at_limit_bodies_behave_as_before, which checkspolled == LIMITand a 401). So a caller with no DID, no signature, and no existing repo can still make the node hold the full group limit, per connection, for as long as it keeps the socket dripping.Measured with a probe (16 concurrent unsigned POSTs to
/{owner}/{repo}/git-receive-pack, each parked on a shared barrier just before its last body frame, so all 16 were provably in flight at once):279 MiB resident from sixteen requests that were all then rejected. The memory is spent entirely before authentication.
Against the deployed configuration this is small numbers, not large ones:
memory = 1024MB (infra/fly/fly.tomland both siblings).GITLAWB_MAX_PACK_BYTES = 524288000(500 MB), so two concurrent unsigned requests to a git write route reach the machine's memory.SIGNED_BODY_LIMIT(2 MB). Fly'shard_limit = 500connections permits 500 x 2 MB = 1000 MB, which is also the whole machine.The connection cap exists, but it is sized above what the box can hold, so it does not act as a mitigation on either path.
Why the existing controls do not cover it
rate_limit_by_ipcounts arrivals in a rolling window and never decrements on completion (crates/gitlawb-node/src/rate_limit.rs), so it bounds request rate, not concurrency. The push bucket is 600 per hour; this needs about three.idle_timeout = 120in the Fly configs is an idle timeout. A connection sending one byte every few seconds is never idle.There is no request-duration or body-read timeout in process.
tower-httpis compiled with features["cors", "trace", "request-id", "limit"], sotower_http::timeoutis not built, andaxum::serveis called with no hyper builder configuration. The onetokio::time::timeoutingit/smart_http.rsbounds the git child process, which is downstream of buffering and after auth.Note for anyone triaging against #174: that PR's read/write semaphores and per-caller caps are all acquired inside handlers, and
require_signatureis a middleware layer that runs before any handler, so they sit downstream of this buffer and do not close it.What would close it
A request-duration or body-read timeout on the routes that buffer, so a slow sender cannot hold a buffer indefinitely, and a concurrency bound sized against actual machine memory rather than above it. Both want sizing against real traffic, which is why they were left out of #262 rather than guessed at.
Streaming the receive-pack body to disk instead of buffering would remove the git-path term entirely, but that is a larger change: the body is currently parsed for ref updates before the authorization checks, scanned for a side-band capability, and written to the child's stdin.