[SPARK-38954][CORE] Cloud Credential Refresh and Distribution Without Kerberos

[parthchandra]

What changes were proposed in this pull request?

This PR enables Spark's existing credential distribution mechanism (driver obtains tokens, pushes them to executors, refreshes before expiry) to work without Kerberos. Today this mechanism is unconditionally gated
on Kerberos being enabled. This change adds a single configuration switch (spark.security.credentials.directProviders.enabled) that lifts the Kerberos gate on token collection.

Changes:

• HadoopDelegationTokenManager.renewalEnabled now also returns true when the new config is enabled.
• obtainDelegationTokens(creds) and obtainTokensAndScheduleRenewal() branch on whether Kerberos credentials are present: if yes, call providers inside doLogin()/doAs() as before; if not (but the switch is
  on), call providers directly. Providers that require Kerberos self-gate via delegationTokensRequired returning false.
• Per-provider try/catch for failure isolation — one provider throwing does not block others.
• SupportsDelegationToken gains an overridable tokenManagerRequired() hook so scheduler backends can activate the token manager based on the new config.
• Hard startup failure if spark.network.crypto.enabled is not true when the switch is active (prevents bearer tokens over plaintext RPC).
• WARN log when the config is set but no providers are discovered via ServiceLoader.

Design document: SPIP: Cloud Credential Refresh and Distribution Without Kerberos

Why are the changes needed?

In non-Kerberos deployments (cloud, internal IdP, cross-account access on YARN), each executor independently authenticates against identity services, causing thundering-herd load amplification, inconsistent
credential state, and no centralized control over credential rotation. The existing distribution infrastructure is already provider-agnostic (Kafka proves this) — only the activation gates prevent it from running
without Kerberos.

Does this PR introduce any user-facing change?

Yes. New configuration:

Config	Default	Description
spark.security.credentials.directProviders.enabled	false	Enables delegation token collection and renewal without Kerberos. When true, the manager starts even if Hadoop security is not enabled, calling all
providers whose delegationTokensRequired returns true. Requires spark.network.crypto.enabled=true.

Existing deployments are unaffected (config defaults to false). When Kerberos IS present, all code paths are unchanged.

How was this patch tested?

New unit test suite NonKerberosCredentialsSuite (7 tests):

• renewalEnabled returns true/false based on config
• Providers are called without Kerberos when config is enabled
• Providers with delegationTokensRequired=false are skipped
• A failing provider does not prevent other providers from running
• Individual provider can be disabled via spark.security.credentials.<service>.enabled
• Startup fails if spark.network.crypto.enabled is not true

Existing HadoopDelegationTokenManagerSuite (4 tests) passes unmodified — verifies no regression in Kerberos deployments.

Was this patch authored or co-authored using generative AI tooling?

Yes, Co-authored by Claude Code (opus 4.6)

[dongjoon-hyun]

Thank you, @parthchandra .

[parthchandra]

@pan3793, @steveloughran, @attilapiros (since you reviewed the SPIP). PTAL.

[peter-toth]

Thanks for the PR, @parthchandra!

This lifts the Kerberos gate on Spark's existing delegation-token machinery (driver collects tokens, pushes them to executors, refreshes before expiry) so it can run for non-Kerberos providers, behind a new off-by-default switch spark.security.credentials.directProviders.enabled that also hard-requires RPC encryption. The manager now branches: with Kerberos creds it uses doLogin()/doAs() as before, otherwise (switch on) it calls providers directly; providers self-gate via delegationTokensRequired. The shape is reasonable and the existing Kerberos paths look preserved. Main things I found: the new config isn't documented; the encryption gate only accepts network.crypto (not ssl.rpc or SASL encryption); and the Kerberos-detection logic is duplicated with a subtly inconsistent ccache branch.

Blocking

1. Missing documentation. The new user-facing config spark.security.credentials.directProviders.enabled (and its RPC-encryption requirement) isn't documented in docs/security.md, which already lists the sibling spark.security.credentials.${service}.enabled. Please add an entry there.

Non-blocking

2. Confirm the config version. .version("5.0.0") is master's version; a backportable new feature would ship first in branch-4.x (4.3.0). Confirm the intended target release.
3. Encryption gate is too narrow. spark.ssl.rpc.enabled and spark.authenticate.enableSaslEncryption also encrypt RPC; the hard require rejects otherwise-valid deployments. Accept any of the three.
4. Kerberos detection duplicated & inconsistent. renewalEnabled and the new obtainTokensAndScheduleRenewal compute "has Kerberos" differently for ccache, so ccache + no creds + switch-on wrongly routes through doLogin/doAs. Extract one shared helper.
5. Per-provider try/catch also changes the Kerberos path. Swallowing a provider failure (vs the old propagate-and-retry) applies to existing Kerberos deployments too. Scope it or call it out.
6. Renewal path untested. Tests don't cover start()/updateTokensTask/obtainTokensAndScheduleRenewal -- where finding 4 lives. A mock RpcEndpointRef would let you test it.

[peter-toth]

Re-checked through 3100606 — findings 3, 4, 5, 6 resolved (encryption gate now accepts network-crypto / SASL / SSL-RPC; Kerberos detection unified via hasKerberosCredentials; try/catch scoped to isolateFailures, so the Kerberos path still propagates and retries; start() now covered), nothing regressed.

Blocking

• 1. Docs still missing (round 1): The new spark.security.credentials.directProviders.enabled and its RPC-encryption requirement still aren't in docs/security.md, alongside the sibling spark.security.credentials.${service}.enabled (~line 974). A new user-facing config needs a doc entry before merge.

Non-blocking

• 2. Version — target release (round 1): Answering your question on the package.scala:1689 thread: per project policy a backportable feature takes the version of the branch it first ships in, so if you backport it (which fits an opt-in feature) it's 4.3.0, not 5.0.0. Still .version("5.0.0") at this head — flagging the edit once you decide.
• 7. Direct-provider failures never retried (new): On the renewal loop the direct path isolates every provider failure, so if all providers fail transiently the fetch "succeeds" with empty creds and nextRenewal = Long.MaxValue — the next renewal is scheduled ~forever out and updateTokensTask's retry-after-CREDENTIALS_RENEWAL_RETRY_WAIT never fires. Executors get empty credentials with no recovery short of a restart. [inline: HadoopDelegationTokenManager.scala:278]

Minor

• 8. Stale start() doc (new): Its scaladoc still says "Requires a principal and keytab" / "requires that a keytab has been provided to Spark", which no longer holds on the direct path.

[peter-toth]

Finding 7. On this renewal path, obtainDelegationTokens(isolateFailures = true) catches every provider exception (line 199) and returns None for the failed provider. If all providers fail (e.g. the IdP is transiently unreachable at startup), nextRenewal folds to Long.MaxValue, so the delay computed just below (line 286) is effectively infinite and scheduleRenewal won't re-run for ~centuries. And since nothing throws, updateTokensTask's catch — which would otherwise reschedule after CREDENTIALS_RENEWAL_RETRY_WAIT — never fires. Net: executors receive empty credentials and there's no recovery short of a driver restart. The Kerberos path avoids this because it doesn't isolate, so a failure propagates and gets retried.

Note nextRenewal == Long.MaxValue is also the legitimate "providers succeeded but reported no expiry" case, so keying a retry off the delay alone would over-fire. Cleaner to have the isolate path signal that a failure actually occurred (e.g. count the caught exceptions) and, when that count is > 0, schedule CREDENTIALS_RENEWAL_RETRY_WAIT instead of the computed delay.

[parthchandra]

@peter-toth addressed items, 1, 2, 7, and 8 as well.
Also, opened PR for spark 4.3 - #57473

[dongjoon-hyun]

This seems to be removed mistakenly. Please recover this comment.

[dongjoon-hyun]

Let's keep SupportsDelegationToken trait concisely. I don't think Logging is required or used in this trait.

[dongjoon-hyun]

This looks insufficient (unsafe) to me. Technically, we need to check NETWORK_AUTH_ENABLED because spark.network.crypto.enabled and spark.authenticate.enableSaslEncryption only take effect when spark.authenticate=true.

Suggested change

	val hasEncryption = sparkConf.get(NETWORK_CRYPTO_ENABLED) ||
	sparkConf.get(SASL_ENCRYPTION_ENABLED) ||
	sparkConf.getBoolean("spark.ssl.rpc.enabled", false)
	require(hasEncryption,
	"RPC channel encryption (spark.network.crypto.enabled, " +
	"spark.authenticate.enableSaslEncryption, or spark.ssl.rpc.enabled) must be enabled when " +
	"spark.security.credentials.directProviders.enabled is true. " +
	"Credential tokens must not be transmitted over unencrypted channels.")
	val hasEncryption = sparkConf.getBoolean("spark.ssl.rpc.enabled", false) ||
	(sparkConf.get(NETWORK_AUTH_ENABLED) &&
	(sparkConf.get(NETWORK_CRYPTO_ENABLED) || sparkConf.get(SASL_ENCRYPTION_ENABLED)))
	require(hasEncryption,
	"RPC channel encryption must be enabled when " +
	"spark.security.credentials.directProviders.enabled is true: either " +
	"spark.ssl.rpc.enabled=true, or spark.authenticate=true together with " +
	"spark.network.crypto.enabled or spark.authenticate.enableSaslEncryption. " +
	"Credential tokens must not be transmitted over unencrypted channels.")

[dongjoon-hyun]

Is this valid?

This is adjacent to but distinct from @peter-toth 's finding 7: that fix keys off failureCount, which is 0 here because no provider is ever invoked. This fallback branch silently stops renewal. It returns (new Credentials(), Long.MaxValue, 0), so the code below sends an empty UpdateDelegationTokens to the executors and — since failureCount == 0, the retry-wait path from finding 7 doesn't trigger — schedules the next renewal at 0.75 * (Long.MaxValue - now), i.e. effectively never. There is no recovery short of a driver restart.

The branch is reachable: with spark.kerberos.renewal.credentials=ccache and directProviders.enabled=false, renewalEnabled is true at start(), but if the ticket cache expires before a later renewal fires, hasKerberosCredentials becomes false and this branch is taken. Previously this scenario threw from doLogin()/the providers, and updateTokensTask's catch rescheduled after CREDENTIALS_RENEWAL_RETRY_WAIT — so a user who refreshed their ticket cache would recover on the next retry. That retry loop is lost here, which is a regression for existing ccache deployments.

Suggest throwing instead, so the existing catch in updateTokensTask handles it and keeps retrying:

Suggested change

	(new Credentials(), Long.MaxValue, 0)
	// Reachable e.g. when ccache-based Kerberos credentials expired after start().
	// Throw so that updateTokensTask() reschedules a retry instead of silently
	// distributing empty credentials and never renewing again.
	throw new IllegalStateException("Cannot obtain delegation tokens: no Kerberos " +
	"credentials are available and direct credential providers are not enabled.")

[dongjoon-hyun]

This makes tokenManagerRequired() return true for every CoarseGrainedSchedulerBackend subclass, but only YARN, Kubernetes, and local mode actually override createTokenManager(). On standalone, StandaloneSchedulerBackend inherits the default createTokenManager(): None, so enabling spark.security.credentials.directProviders.enabled there passes the encryption require, flips tokenManagerRequired() to true — and then setupTokenManager() silently does nothing. No tokens are collected, no warning is logged, and the user has no signal that the feature is inert on their cluster manager.

1. Please document in the PR description (and documentation) that StandaloneCluster is not supported (or out of scope).
2. Log a warning in setupTokenManager() (or here) when the config is enabled but createTokenManager() returns None, so make the misconfiguration visible.

[dongjoon-hyun]

This looks too much for me. Let's avoid a wild-card import.

[dongjoon-hyun]

Suggested change

	super.tokenManagerRequired() || conf.get(CREDENTIALS_DIRECT_PROVIDERS_ENABLED)
	super.tokenManagerRequired() || conf.get(config.CREDENTIALS_DIRECT_PROVIDERS_ENABLED)

[dongjoon-hyun]

If the switch is on but no providers were discovered (the WARN case above), renewalEnabled is still true, so start() sends an empty UpdateDelegationTokens and schedules the next renewal ~never (nextRenewal == Long.MaxValue, failureCount == 0). Simpler to not start the renewer at all like the following.

Suggested change

	hasKerberosCredentials || sparkConf.get(CREDENTIALS_DIRECT_PROVIDERS_ENABLED)
	hasKerberosCredentials ||
	(sparkConf.get(CREDENTIALS_DIRECT_PROVIDERS_ENABLED) && delegationTokenProviders.nonEmpty)

[peter-toth]

Re-checked through c2dabb5f — findings 1, 2, 7, 8 resolved (docs added, version → 4.3.0, all-providers-fail now reschedules via the retry-wait, start() docstring updated), nothing new from my end. I agree with @dongjoon-hyun's findings, which should be addressed before merge.