[improve][pip] PIP-489: FIPS 140-3 compliance mode for Apache Pulsar

[david-streamlio]

PIP: this PR

Motivation

Pulsar has no supported FIPS 140-3 deployment path today, despite being most of the way there at the JCA layer (SecurityUtility already resolves BC vs. BC-FIPS reflectively). A full audit of master found the gaps documented in the proposal: the BC/BC-FIPS swap lost its packaging story in the Gradle migration; the broker binary listener cannot be steered off BoringSSL (tlsProvider is declared but never wired); Conscrypt is the shipped web-TLS default; several security paths use non-approved algorithms (SHA-1 OAEP and ECIES in MessageCryptoBc, MD5-crypt/DES-crypt in Basic auth, a non-HMAC construction in SaslRoleTokenSigner); and there is no FIPS documentation or meaningful FIPS test coverage.

Modifications

Adds pip/pip-489.md, proposing a supported, tested, documented FIPS mode deployment profile: TLS provider wiring for every listener (matching the proxy's existing pattern), a FIPS distribution variant / documented jar swap, a fipsMode fail-fast startup validator, metadata-negotiated migration to RSA_OAEP_SHA256 key wrapping (with ECDH_AES_KW replacing ECIES in a later phase), SHA-2-crypt Basic auth, a dual-verify HMAC migration for the SASL role-token signer, a name-stable fix for the SHA-1-derived Kubernetes resource names, and a real FIPS TLS integration test group.

Companion quick-win PRs already open (intentionally outside the PIP): #26152, #26153, #26154.

A DISCUSS thread has been started on dev@pulsar.apache.org referencing this PR.

[lhotari]

I wanted to flag some overlap so we can keep the efforts aligned. PIP-478 has been in progress and, besides the asynchronous authentication interfaces on the client side, it covers the TLS transport aspects of Pulsar 5.0 on both the client and the server side. The dependency between the auth and the TLS parts is explained in the PIP document (PIP-478: #25890, discussion thread: https://lists.apache.org/thread/s9n9jksr9vqgn9o982zmnnkcxdcncy3f).

On the TLS transport side there are more concerns than configuring the JCA provider. Stricter security compliance -- for example FIPS 140-3 Level 3 -- requires an HSM: the key material is stored in a hardware device and kept outside the application, so it never crosses into the Pulsar process. This can be achieved with the PulsarTlsFactory plugin interface introduced in PIP-478: a plugin implementation can integrate with a security solution that meets such requirements (for instance building the TLS context against an HSM via a PKCS#11 token, so the private key never leaves the device). For the simpler FIPS 140-3 Level 1 case, PIP-478 also lets you configure the JCA provider directly (for example a FIPS-validated provider such as BC-FIPS on the JDK TLS engine), and it wires the engine/provider selection through every server component and the client.

PIP-478 also adds a PulsarHttpClient API. The Pulsar client today uses several HTTP clients -- for example authentication plugins such as OAuth2 that call a token endpoint -- which currently don't share a centralized configuration. The PulsarHttpClient API is added for authentication plugin implementations so that the TLS configuration of those HTTP clients can be controlled by the PulsarTlsFactory when there are special requirements, or handled through the Pulsar v5 client's TlsPolicy configuration when there are none. This keeps the TLS transport (and FIPS) configuration consistent across the client's outbound TLS, not only the binary protocol and the web/admin listeners.

Given that overlap, I'd suggest that PIP-489 builds upon PIP-478 for the TLS transport aspects rather than defining a separate TLS transport configuration path. That would let PIP-489 concentrate on the broader FIPS-compliance concerns that PIP-478 intentionally leaves out of scope -- FIPS-approved algorithms in message encryption and authentication, the FIPS distribution/packaging, and a fail-fast FIPS-mode validation switch -- while reusing the TLS transport foundation.

On the packaging point specifically: I don't think the Gradle migration actually lost anything essential there. BC and BC-FIPS can't co-exist on the classpath, so switching between them is really a matter of replacing the non-FIPS BC jars with the FIPS ones. The old bouncy-castle/bcfips module wasn't especially useful for that swap in practice -- a client can simply exclude the non-FIPS BC dependencies and add the FIPS ones, and on the server side it can be handled by keeping the non-FIPS and FIPS jars in separate directories and choosing which one to put on the classpath based on configuration. So the FIPS packaging story seems fairly tractable without reintroducing a dedicated swappable module.

[lhotari]

One more bit of history that may help scope PIP-489: FIPS / BouncyCastle-FIPS support in Pulsar has been partial and handled case by case in the past, rather than as a coherent, tested capability -- which is a big part of why a proper end-to-end FIPS design is worth doing.

Two concrete examples:

1. Pulsar has never actually used BouncyCastle's JSSE (TLS) provider. A pulsar-site correction (Remove TLS/JSSE statement in BouncyCastle intro pulsar-site#974) removed a docs statement that implied otherwise, noting "Pulsar does not implement bouncy castle jsse; there is no dependency on bc jsse in Pulsar." So even with the BC-FIPS jars on the classpath, the TLS transport did not route through a BouncyCastle/FIPS TLS provider -- the FIPS story for the transport itself was never wired.

2. On the message-encryption side, [improve][misc] Improve AES-GCM cipher performance #23122 switched AES-GCM in MessageCryptoBc from the BouncyCastle provider to SunJCE for performance. In reviewing it ([improve][misc] Improve AES-GCM cipher performance #23122 (review)) I raised the FIPS concern -- when the FIPS library is enabled, SunJCE should not become the default -- and the finding was that the existing code already ignored the presence of BouncyCastleFipsProvider. In other words FIPS compliance was already a separate, not-fully-handled concern rather than something the crypto paths consistently respected.

The takeaway is that FIPS support in Pulsar has been incomplete and piecemeal: the TLS transport was never routed through a FIPS provider, and the crypto paths didn't consistently honor the FIPS provider when it was present. That's exactly the gap worth closing, and it splits naturally: PIP-478 covers the TLS-transport slice (a configurable engine plus any JCA provider via TlsPolicy.jcaProvider, and the PulsarTlsFactory plugin for HSM-backed / FIPS 140-3 Level 3 cases), and PIP-489 can cover the broader FIPS-mode concerns -- FIPS-approved algorithms in message encryption and authentication, packaging, and a fail-fast validation switch -- so the crypto paths consistently respect FIPS rather than silently falling back to non-validated providers.

[david-streamlio]

Thanks @lhotari — this is exactly the alignment I was hoping to get before the VOTE, and I agree with the split you propose. I've updated the PIP (95ac5af) accordingly:

• TLS transport now builds on PIP-478. Design section (a) is rewritten: PIP-489 no longer defines its own TLS provider wiring. FIPS mode consumes PIP-478's TlsPolicy (JDK engine + JCA provider), PulsarTlsFactory covers the HSM/PKCS#11 (140-3 Level 3) case, and PulsarHttpClient brings the auth plugins' outbound HTTPS under the same configuration — that last one closes a gap I hadn't addressed (OAuth2/OIDC token-endpoint calls going through the default SSLContext). The per-listener one-line .tlsProvider(...) pass-throughs are kept only as a documented interim option for maintenance branches, explicitly not a deliverable here. PIP-489 concentrates on what PIP-478 leaves out of scope: approved algorithms in message crypto/auth, packaging, and the fail-fast fipsMode validator.
• Packaging stays a jar-swap story. I've noted in General Notes that flavor selection is a classpath-assembly concern (client-side dependency exclusion/replacement; server-side separate directories or the fipsRuntimeClasspath variant) rather than reintroducing the Maven-era swappable modules.
• Your history points are captured in Background — no BC JSSE dependency ever existed (pulsar-site#974), and the crypto paths haven't consistently honored an installed FIPS provider ([improve][misc] Improve AES-GCM cipher performance #23122) — as motivation for making FIPS a coherent, validated capability instead of case-by-case fixes.

While revising I also tightened the fipsMode validator to cover deployment posture, not just provider selection: keystore/truststore types (JKS rejected; PKCS12 documented unsupported — FIPS-conformant PKCS#12 needs the RFC 9879 KDF/PBMAC profile), a TLS 1.2 floor, cipher-suite and certificate-algorithm screening, and the java.security / BCFKS-converted-cacerts guidance in the deployment docs.

One sequencing question for you: given PIP-489's phase 1 depends on PIP-478's TlsPolicy landing, do you see any issue with the two PIPs proceeding through DISCUSS/VOTE in parallel, with 489's TLS-dependent items gated on 478's implementation?