X509 cold-start impacting latency sensitive CLI use-cases

[avin3sh]

We were looking into OpenSslCachedSystemStoreProvider cold-start latency on Ubuntu 24.04 w/ libssl 3.0.1x - as we noticed our .NET 10 CLIs that were making HTTP request almost always were incurring 80-100ms latency even though actual network latency was expected to be negligible.

On squinting it looked like ~80 ms alone was spent in LoadMachineStores parsing + deduplicating system roots. Most of which was dominated by ~ 350 d2i_X509 calls against libssl decoder.

From benchmarks it looks like the cost of the above is ~120 µs per d2i_X509 call plus PEM decoding, BIO setup, etc.

Also worth noting is that there is no .NET-way to avoid this path, every chain build unconditionally requests the cached store - thhis meant ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust OR ChainPolicy.CustomTrustStore do NOT bypass the above mechanics. The system store load is on the critical path for every cold chain build, which kinda makes this things worse if you are building CLIs, and it's even more worse if you are in a typical corporate setup with additional CAs to deal with.

[All of this is further complicated by general perf issues noted with OpenSSL 3.0 - openssl/openssl#19119, openssl/openssl#17950, openssl/openssl#20698 but there looks like some room for improvement on .NET side of things]

---

When I was first reading about this problem, there was an assumption that other languages weren't eagerly loading all the certs but when I looked at other ecosystems, from a cursory glance it looks like it is not unusual to load /etc/ssl/certs/ca-certificates.crt and walk /etc/ssl/certs/*, but some of these languages used their own managed parser instead of deferring to libssl and are likely gaining single digit µs performance as a result. [1] [2]

---

In .NET's case d2i_X509 currently serves two roles in the load path:

1. Produce the X509 handle that finally lives in the SafeX509StackHandle consumed by X509_verify_cert during chain build - this needs every cert to be d2i'd up front
2. Produce managed X509Certificate2 so that cert.Subject, cert.Issuer, and Thumbprint bbased deduplication can run

(2) doesn't strictly require d2i_X509 and (1) is only needed for certs involved in the chain build, and not for all the certs.

Another thing to note is the d2i calls happen BEFORE deduplication, so even if keep everything aside, that's something that should have been optimized, and is safer. (If we ever do it, I hope to see it backported to .NET 10 and not just done in .NET 11).

I am not a crypto expert ... but had access LLM (so, same thing ? /s), so I tried experimenting with parsing the DER bytes with .NET's own System.Formats.Asn1.AsnReader and seemed like that alone can take care of (2) above, without needing to d2i. Even content hash deduping might work ? It took 1.2 ms for me vs. 40x more on the openssl path but I am likely overlooking a lot of details but still... #118613 suggests we are not averse to these kind of improvements 😉

Now that leaves us with the most difficult part, which is (1) as it requires deferring d2i_X509 to chain-build time, which probably means

(a) looking at other X509_LOOKUP methods (my mentor flagged X509_LOOKUP_hash_dir)

(b) a custom lookup callback with the X509_STORE. OpenSSL's chain processor calls into managed code, which finds the matching index entry, calls d2i_X509 lazily, returns the handle.

---

[akoeplinger]

cc @vcsjones