RANGER-5655: Dynamic unified ingestor registry for audit partition routing and service allowlists

[ramackri]

Summary

Implements RANGER-5655: a dynamic unified ingestor registry for audit-ingestor so operators can change Kafka partition routing and per-repo service allowlists at runtime — without restarting ingestor pods.

The plan lives in compacted Kafka topic ranger_audit_partition_plan (1 partition). All ingestor replicas converge via a background watcher; AuditPartitioner reads in-memory state on the hot path.

Feature flag (default off): ranger.audit.ingestor.kafka.partition.plan.dynamic.enabled=false
When off, behavior matches today (static XML routing/allowlists; plan REST returns 503).

What this PR adds

Area	Change
Registry	Versioned JSON plan (plugins, buffer, services) in Kafka; bootstrap, validate, watch, apply
Routing	Dynamic AuditPartitioner — dedicated plugin lanes + shared buffer pool; append-only layout
Allowlists	Registry-first per-repo services map; XML fallback; auth_to_local union recomposed on plan updates
REST (v2)	GET / PATCH /api/audit/partition-plan — no Kerberos (public control plane)
Auto-onboard	Unknown appId + allowlisted principal on POST /api/audit/access registers plugin/repo without manual PATCH
Multi-repo	Same plugin appId can serve multiple Ranger repos (e.g. dev_trino, dev_trino2) in one plan version

API (v2)

Method	Path	Auth	Purpose
GET	/api/audit/partition-plan	None	Read current plan
PATCH	/api/audit/partition-plan	None	Merge update (expectedVersion, optional plugins, services, pluginScales)
POST	/api/audit/access?serviceName=&appId=	Kerberos	Ingest audits; auto-onboard when needed

Request body: PartitionPlanReplacement with optimistic locking (expectedVersion) and append-only merge semantics.

Scope

• In scope: audit-server/audit-ingestor (+ audit-common constants), unit tests, ranger-audit-ingestor-site.xml
• Out of scope: Docker/dev-support harness, plugin packaging (separate PRs)

~50 files; review by layer (REST → service/mutations → registry/watcher → partitioner/allowlist → tests).

Testing

mvn verify -pl audit-server/audit-ingestor -Drat.skip=true   -Dtest='PartitionPlan*Test,ServiceAllowlist*Test,AuthToLocalRuleComposerTest'

Partition-plan + allowlist unit tests pass; checkstyle/PMD pass.

Manual E2E (Kerberos, Kafka, ingestor, Solr, HDFS/Hive plugins): static mode regression, PATCH promote/scale, auto-onboard, multi-repo allowlists, partition routing verified end-to-end.

Full details (PDF)

Architecture, plan document schema, REST examples, manual test matrix, and operational notes:

Dynamic Ingestor Registry Guide (Ranger Audit Ingestor).pdf

Checklist

• Feature disabled by default — no behavior change unless flag enabled
• Static mode regression verified
• Unit tests for registry, REST validation, routing, allowlists
• Manual Docker audit-lab validation documented in PDF

QA

• What problem does this solve?
  Today, if you onboard a new plugin repo or change who can post audits, you edit ingestor XML and restart every ingestor pod. Same for giving a hot plugin its own Kafka partitions — restart + careful ordering.

• What does this PR do?
  Adds an optional dynamic registry (off by default) so partition routing and per-repo allowlists can change at runtime without ingestor restarts.

• How it works (simple version)
  • Plan JSON lives in a compacted Kafka topic: ranger_audit_partition_plan (1 partition)
  • Every ingestor watches that topic and keeps the plan in memory (PartitionPlanWatcher → PartitionPlanHolder) • AuditPartitioner uses the plan when writing to ranger_audits
  • ServiceAllowlistResolver checks who can post audits per repo — plan first (services[repo].allowedUsers), XML fallback if repo isn’t in the plan yet
  • AuthToLocalRuleComposer rebuilds auth_to_local from the union of allowlisted users when the plan changes

• How operators change things
  • GET /api/audit/partition-plan — read current layout (no Kerberos)
  • PATCH /api/audit/partition-plan — promote/scale plugins, update allowlists (no Kerberos)
  • Or do nothing manually: first audit from an unknown plugin on POST /api/audit/access can auto-onboard it (Kerberos as today); ServiceAllowlistResolver enforces allowlist on that same path
  • One plugin (appId) can serve multiple Ranger repos (e.g. dev_trino + dev_trino2) — each with its own allowlist via ServiceAllowlistResolver • Scaling a plugin adds partitions append-only (doesn’t reshuffle existing plugins)
  • Flag: ranger.audit.ingestor.kafka.partition.plan.dynamic.enabled=false (default)
  • When off → today’s behavior unchanged; plan REST returns 503; allowlists stay XML-only

Operators / automation          Plugins (normal path)
        |                                |
   GET/PATCH plan REST              POST /access
        |                                |
        v                                v
   mergePartitionPlan()          ensurePluginOnboarded()
        |                                |
        +--------> Kafka topic ranger_audit_partition_plan
                          |
                   PartitionPlanWatcher (all pods)
                          |
                   PartitionPlanHolder (memory)
                    /              \
         AuditPartitioner      ServiceAllowlistResolver
         (Kafka produce)      (+ AuthToLocalRuleComposer)

[Copilot]

Pull request overview

This PR implements the “dynamic unified ingestor registry” for audit-ingestor by introducing a Kafka-compacted, versioned partition-plan document (including per-repo service allowlists) and a simplified REST control plane to onboard/update plugins at runtime without restarting ingestor pods.

Changes:

• Adds Kafka-backed partition-plan registry plumbing (bootstrap, watcher, validator, update applier, registry client) with in-memory hot-path state via PartitionPlanHolder.
• Simplifies/introduces partition-plan REST endpoints in AuditREST for GET /partition-plan, POST /partition-plan/plugins, and PATCH /partition-plan/plugins/{pluginId} with request validation.
• Introduces dynamic-mode allowlist behavior (registry-first with XML fallback) and composes auth_to_local rules from the global allowlist union when dynamic mode is enabled.

Reviewed changes

Copilot reviewed 50 out of 50 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/ServiceAllowlistResolverTest.java	Unit tests for registry-first allowlist resolution with XML fallback.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/ServiceAllowlistBootstrapTest.java	Unit tests for loading/merging allowlists from XML properties into plan services.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanValidatorTest.java	Unit tests for plan shape validation and append-only constraints.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanUpdateApplierTest.java	Unit tests for applying compacted Kafka plan records into memory.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanServiceTest.java	Unit tests for dynamic-enabled flag and in-memory plan reads.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanServiceMutationTest.java	Unit tests for onboard/update flows, optimistic locking, and topic-grow failure handling.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanRequestValidatorTest.java	Unit tests for REST request model validation (mandatory services on onboard).
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanKafkaConfigTest.java	Unit tests for plan-topic config resolution and dynamic flag parsing.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanHolderTest.java	Unit tests for plan holder allowlist access and install validation.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanBootstrapTest.java	Unit tests for bootstrap plan layout and overrides.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanBootstrapSupportTest.java	Unit tests for empty-registry bootstrap and peer-publish adoption.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanAllocatorTest.java	Unit tests for onboard/update allocation behavior and service ownership rules.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/model/PartitionPlanJsonTest.java	Unit tests for JSON round-trip and semantic equality.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/partition/AuthToLocalRuleComposerTest.java	Unit tests for composing/applying auth_to_local rules based on allowlists and dynamic mode.
audit-server/audit-ingestor/src/test/java/org/apache/ranger/audit/producer/kafka/AuditPartitionerDynamicTest.java	Unit tests for dynamic partition routing behavior and concurrency.
audit-server/audit-ingestor/src/main/resources/conf/ranger-audit-ingestor-site.xml	Updates config docs and adds dynamic partition-plan properties.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/server/AuditServerConfig.java	Allows overriding ingestor config path via -Daudit.config.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/rest/AuditREST.java	Adds simplified partition-plan REST endpoints and integrates dynamic allowlist enforcement.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/ServiceAllowlistResolver.java	Implements registry-first per-repo allowlist authorization with XML fallback.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/ServiceAllowlistBootstrap.java	Loads allowlists from XML properties and merges into plans when services are missing.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PrimaryCatalogRule.java	Holds parsed auth_to_local catalog rules and mapped short names.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanWatcher.java	Background watcher that bootstraps and refreshes in-memory plan from Kafka.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanValidator.java	Validates plan structure, services, and append-only update semantics.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanUpdateApplier.java	Applies newer plan versions from compacted Kafka records into PartitionPlanHolder.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanService.java	Service layer for REST reads/mutations with optimistic locking and topic growth.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanRequestValidator.java	Validates OnboardPlugin / UpdatePlugin request bodies.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanRegistryFactory.java	Factory for opening Kafka-backed plan registries for REST mutations.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanRegistry.java	Interface for durable partition-plan storage (Kafka compacted topic).
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanKafkaConfig.java	Centralizes partition-plan Kafka config resolution and security settings.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanHolder.java	Atomic in-memory plan holder used by hot-path routing and allowlist resolution.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanBootstrapConfig.java	Represents bootstrap inputs derived from legacy producer/XML config.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/PartitionPlanBootstrap.java	Bootstraps v1 plan from legacy config and seeds Kafka registry when empty.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/model/UpdatePlugin.java	REST DTO for plugin updates (scale + allowlist mutations).
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/model/ServiceAllowlistEntry.java	Plan DTO for per-repo allowlists with optional plugin ownership.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/model/PluginPartitionAssignment.java	DTO for explicit/contiguous partition assignments.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/model/PartitionPlan.java	Versioned plan DTO with JSON serialization/deserialization + validation.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/model/OnboardPlugin.java	REST DTO for onboarding a plugin (mandatory services).
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/KafkaPartitionPlanRegistry.java	Kafka implementation of the compacted plan registry.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/KafkaAuditTopicPartitionGrower.java	Grows audit topic partitions before plans reference new tail IDs.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/exception/PartitionPlanException.java	Base exception for plan validation and mutation errors.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/exception/PartitionPlanConflictException.java	Optimistic-lock conflict exception carrying the current plan (HTTP 409).
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/constants/PartitionPlanConstants.java	Constants for initial plan version and consumer group ids.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/AuthToLocalRuleComposer.java	Composes and applies auth_to_local rules based on allowlist union in dynamic mode.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/partition/AuthToLocalRuleCatalog.java	Parses the auth_to_local catalog and composes a reduced active ruleset.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/AuditPartitioner.java	Adds dynamic plan routing path using PartitionPlanHolder.
audit-server/audit-ingestor/src/main/java/org/apache/ranger/audit/producer/kafka/AuditMessageQueue.java	Starts/stops PartitionPlanWatcher when dynamic mode is enabled.
audit-server/audit-common/src/test/java/org/apache/ranger/audit/utils/AuditMessageQueueUtilsTest.java	Adds tests for building Kafka AdminClient config.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/utils/AuditMessageQueueUtils.java	Adds plan-topic creation, topic-exists probing, admin config helper, and topic grow helper.
audit-server/audit-common/src/main/java/org/apache/ranger/audit/server/AuditServerConstants.java	Adds dynamic partition-plan constants and changes default configured plugins to empty.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ramackri]

Dynamic Ingestor Registry Guide (Ranger Audit Ingestor).pdf