Fix DTLS half-open peer exhaustion and ClientHello RSA signing DoS#72
Open
NikoCat233 wants to merge 2 commits into
Open
Fix DTLS half-open peer exhaustion and ClientHello RSA signing DoS#72NikoCat233 wants to merge 2 commits into
NikoCat233 wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes two denial-of-service issues in the DTLS listener:
existingPeersindefinitely when no application-level UDP connection had been created.Half-open peer cleanup
A peer is added to
existingPeersafter a cookie-verified ClientHello, while its entry inallConnectionsis only created after the DTLS handshake completes and an application Hello is received. The stale connection path previously queued cleanup only whenallConnectionsalready contained the peer's ConnectionId. As a result, a client could stop during the handshake and leave its cryptographic peer state allocated permanently.The cleanup path now removes and disposes stale peers directly when no application connection exists. Cleanup uses both the ConnectionId and the exact PeerData instance to avoid removing a replacement session that reused the same endpoint.
The timeout covers every pre-connection state, including:
Application data forwarding now holds a small processing lease while it creates or accesses the application-level connection. This prevents the stale timer from removing a peer in the short interval between decrypting the application Hello and registering its connection.
ServerKeyExchange signature caching
The listener previously rebuilt the complete server flight for every accepted ClientHello retransmission. This called
EncodeServerKeyExchangeMessageeach time and performed a new RSA-SHA256 signature, allowing one cookie-verified peer to consume disproportionate CPU.The listener now caches the encoded ServerKeyExchange payload for the current peer and handshake. Retransmissions still generate fresh DTLS record sequence numbers and rebuild the records, but copy the already signed handshake payload instead of signing it again. The cache is never shared across peers or handshake epochs.
Certificate bytes and the ServerKeyExchange signature are captured as one handshake-specific certificate generation.
SetCertificateuses a reader/writer lock so that an in-progress handshake continues using a matching certificate and private key while newly started handshakes use the replacement certificate. This avoids producing a retransmission containing a new certificate with a signature from the old key.The cached data contains only the public certificate, ECDHE parameters, public key, algorithms, and RSA signature. It does not contain the ECDHE private key, shared secret, or master secret. The public cache is released when the epoch transition is completed.
Tests
Added regression coverage for:
Validation performed:
net472andnetstandard2.0builds pass;UdpReliableMessageAckWithDropTestfailure is an existing timing-sensitive UDP ACK test and reproduces on the unchanged baseline.