Fix doubling of rental for PKCS#8 exports in PQC

[vcsjones]

The MLKem, MLDsa, and SlhDsa had a small quirk with the way their PKCS#8 export work.

It would rent from the CryptoPool to create a buffer to populate, based on size. Rent is only guaranteed to return a buffer that is at least large enough.

If the TryExportPkcs8PrivateKeyCore implementation returned false, indicating the buffer was too small, we would double the size and rent again. However, we double based on how much we last asked for, not how much we actually got back.

For example, CryptoPool.Rent would ask for 2592 bytes for ML-DSA-44, and sometimes CryptoPool would give it back an array that is 8192 bytes in size. The Core implementation would return false, indicating that the buffer is too small. The loop would then double it to 5184, and get 8192 back again. So the TryExportPkcs8PrivateKeyCore would be called with the same size multiple times in a row.

A unit test ExportPkcs8PrivateKey_Resizes exists to ensure that does not happen and that subsequent calls to TryExportPkcs8PrivateKeyCore are always given a larger buffer on the next call - and that test would sometimes fail.

Interestingly, CompositeMLDsa already has the fix

runtime/src/libraries/Common/src/System/Security/Cryptography/CompositeMLDsa.cs

Line 1890 in 214a9a4

size = buffer.Length;

So it needed no changes.

Fixes #128110

[dotnet-policy-service]

Tagging subscribers to this area: @bartonjs, @vcsjones, @dotnet/area-system-security
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR updates the PKCS#8 private key export retry loop for PQC key types so that when a rented buffer is still too small, the next retry request is based on the actual rented buffer length (not the previously requested size). This ensures each subsequent TryExportPkcs8PrivateKeyCore attempt is made with a strictly larger buffer, preventing repeated calls with the same buffer size when CryptoPool.Rent returns an oversized array.

Changes:

• In MLKem, MLDsa, and SlhDsa, update the resize loop to set size = buffer.Length before doubling and re-renting.
• Align behavior with the already-correct pattern used in CompositeMLDsa.

Show a summary per file

File	Description
src/libraries/Common/src/System/Security/Cryptography/SlhDsa.cs	Fix PKCS#8 export buffer growth to double based on the actual rented buffer length.
src/libraries/Common/src/System/Security/Cryptography/MLKem.cs	Same buffer growth fix for ML-KEM PKCS#8 exports.
src/libraries/Common/src/System/Security/Cryptography/MLDsa.cs	Same buffer growth fix for ML-DSA PKCS#8 exports.

Copilot's findings

• Files reviewed: 3/3 changed files
• Comments generated: 1