Skip to content

Fix DTLS half-open peer exhaustion and ClientHello RSA signing DoS#72

Open
NikoCat233 wants to merge 2 commits into
willardf:mainfrom
NikoCat233:fix/dtls-dos
Open

Fix DTLS half-open peer exhaustion and ClientHello RSA signing DoS#72
NikoCat233 wants to merge 2 commits into
willardf:mainfrom
NikoCat233:fix/dtls-dos

Conversation

@NikoCat233

Copy link
Copy Markdown

This PR fixes two denial-of-service issues in the DTLS listener:

  1. Half-open DTLS peers could remain in existingPeers indefinitely when no application-level UDP connection had been created.
  2. Retransmitted cookie-verified ClientHello messages caused the server to generate a new RSA signature for every retransmission.

Half-open peer cleanup

A peer is added to existingPeers after a cookie-verified ClientHello, while its entry in allConnections is only created after the DTLS handshake completes and an application Hello is received. The stale connection path previously queued cleanup only when allConnections already contained the peer's ConnectionId. As a result, a client could stop during the handshake and leave its cryptographic peer state allocated permanently.

The cleanup path now removes and disposes stale peers directly when no application connection exists. Cleanup uses both the ConnectionId and the exact PeerData instance to avoid removing a replacement session that reused the same endpoint.

The timeout covers every pre-connection state, including:

  • stopping after the cookie-verified ClientHello;
  • sending ClientKeyExchange and ChangeCipherSpec but omitting Finished;
  • completing DTLS but never sending the application Hello.

Application data forwarding now holds a small processing lease while it creates or accesses the application-level connection. This prevents the stale timer from removing a peer in the short interval between decrypting the application Hello and registering its connection.

ServerKeyExchange signature caching

The listener previously rebuilt the complete server flight for every accepted ClientHello retransmission. This called EncodeServerKeyExchangeMessage each time and performed a new RSA-SHA256 signature, allowing one cookie-verified peer to consume disproportionate CPU.

The listener now caches the encoded ServerKeyExchange payload for the current peer and handshake. Retransmissions still generate fresh DTLS record sequence numbers and rebuild the records, but copy the already signed handshake payload instead of signing it again. The cache is never shared across peers or handshake epochs.

Certificate bytes and the ServerKeyExchange signature are captured as one handshake-specific certificate generation. SetCertificate uses a reader/writer lock so that an in-progress handshake continues using a matching certificate and private key while newly started handshakes use the replacement certificate. This avoids producing a retransmission containing a new certificate with a signature from the old key.

The cached data contains only the public certificate, ECDHE parameters, public key, algorithms, and RSA signature. It does not contain the ECDHE private key, shared secret, or master secret. The public cache is released when the epoch transition is completed.

Tests

Added regression coverage for:

  • eviction after a cookie-verified ClientHello is left half-open;
  • eviction after ClientKeyExchange and ChangeCipherSpec are accepted but Finished is omitted;
  • reuse of exactly one RSA signature across ClientHello retransmissions within the same half-open handshake;
  • normal DTLS connection and retransmission behavior.

Validation performed:

  • net472 and netstandard2.0 builds pass;
  • all 46 DTLS tests pass;
  • the three security regression tests pass across five repeated runs (15/15);
  • the first commit builds and its cleanup tests pass when checked out independently;
  • the full suite passes 165/166 tests. The remaining UdpReliableMessageAckWithDropTest failure is an existing timing-sensitive UDP ACK test and reproduces on the unchanged baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant