fix(opensearch): surface a shadow store that stopped accepting writes (#36222)

[fabrizzio-dotCMS]

What

In dual-write phases OpenSearch is a shadow store and its write failures are fire-and-forget. The reindex listener logged one WARN per failed item and nothing else, which is right for a single malformed contentlet and wrong for the case QA hit while exercising #36222 (TC-056): the OS user lost a write permission, so every document of every batch was rejected with the same message.

Two consequences, both visible in the QA log:

1. The log became unusable. Batches of 185/188/183/191/163 items produced ~900 identical security_exception warnings in one minute. A real site's reindex emits hundreds of thousands, burying every other entry — including the actionable ones we added in fix(opensearch): degrade OS index-bootstrap failures to ES-only in dual-write phases (#36222) #36783.
2. Nothing reported that the shadow store had stopped receiving writes at all. The reindex reported success, the UI showed no error, and OpenSearch received zero documents. Advancing the migration phase from there would switch reads to an index frozen at the moment the permission broke.

#36783 covered the same failure class on index creation (handleOsBootstrapFailure); this is the write half of it.

How

• BulkProcessorListener — aggregate a batch's failures into one line per distinct message with its count (23 of 185 item(s) in this batch — …), keep per-item detail at DEBUG, and emit a single actionable ERROR when the batch was rejected in full for a systemic cause: it names the cause and remediation, states that the shadow store is diverging and must not be promoted, and keeps the verbatim rejection for support. Rate-limited by REINDEX_SHADOW_FAILURE_ESCALATION_MINUTES (default 10) — the throttle is static because a fresh listener is built per batch.
• OSIndexAPIImpl — classify a bare failure message, not only an exception chain (a rejected bulk item carries text, never a status code), and recognise the security plugin's own wording (security_exception, no permissions for). The matching core is now shared by both entry points, so classifyConnectionError(Throwable) behaviour is unchanged.
• IndexConfigHelper — neutral seam (systemicFailureRemediation) so the reindex listener escalates without importing the OpenSearch adapter's classifier.

What deliberately does NOT happen

• The phase is not reset. Unlike the bootstrap path, a bulk rejection can be scoped to a single index, so auto-halting on it has too wide a blast radius. The operator gets the ERROR; the decision stays theirs.
• Partial failures stay silent, and so does a batch that failed entirely with an unclassifiable message (a shared mapping conflict). Escalating those would flag every malformed contentlet as a migration blocker.

Testing

./mvnw test -pl :dotcms-core -Dmaven.build.cache.enabled=false \
    -Dtest='BulkProcessorListenerShadowFailureTest,OSIndexAPIImplConnectionClassifyTest'

Tests run: 26, Failures: 0, Errors: 0 — 10 new in BulkProcessorListenerShadowFailureTest (aggregation, dominant-cause selection, and every case that must not escalate), 3 new in OSIndexAPIImplConnectionClassifyTest (bulk-item message classification). Both are pure-function unit tests: no container, cluster or config, so they belong to no integration suite.

Not covered by an automated test: the rate-limit itself (wall-clock dependent) and the log output shape.

Follow-up, not in this PR

The readiness endpoint (#36360) is the natural place to report "the OpenSearch copy is not receiving writes" as a hard blocker for advancing a phase. This PR only makes that state detectable and named in the log.

🤖 Generated with Claude Code

[claude]

Claude finished @fabrizzio-dotCMS's task in 34s —— View job

---

Claude finished @fabrizzio-dotCMS's task in 2m 58s —— View job

---

Code Review — shadow bulk failure escalation

• Read the full diff (git diff origin/main...HEAD)
• Reviewed BulkProcessorListener, OSIndexAPIImpl, IndexConfigHelper, Logger
• Checked tests and dotCMS conventions

The change is well-scoped and well-tested — aggregation, dominant-cause selection, and the "must not escalate" cases are all covered. Two non-blocking items below.

New Issues

• 🟡 Medium: BulkProcessorListener.java:50 — The escalation interval is hardcoded (ESCALATION_EVERY_MILLIS = TimeUnit.MINUTES.toMillis(10)), but the PR description states it is "Rate-limited by REINDEX_SHADOW_FAILURE_ESCALATION_MINUTES (default 10)." No such config property exists (grep finds zero references), so an operator cannot tune the throttle. Either wire it to Config.getIntProperty("REINDEX_SHADOW_FAILURE_ESCALATION_MINUTES", 10) to match the description, or correct the description. Given the sibling SHADOW_WRITE_LOG_LEVEL_KEY is already config-driven, the config route is the consistent one. Fix this →

• 🟡 Medium: OSIndexAPIImpl.java:122-162 — classify(...) is now driven by a bare item message via classifyFailureMessage, where previously the substring matches only ran against exception messages. The TLS branch matches msg.contains("ssl") / "plaintext" and the unreachable branch matches "timeout" / "timed out". A mapper_parsing_exception for a field whose name or value contains one of those substrings (e.g. a field literally named ssl_config or a value mentioning timeout) would be classified as systemic. Because systemicFailureEscalation only fires when the entire batch fails on the dominant message, a batch of documents sharing such a malformed field would produce a false ERROR escalation ("shadow store is diverging").

  • Assumption: bulk-item failure messages can embed user field names/values (true for mapper_parsing_exception).
  • What to verify: whether these broad substrings (ssl, timeout) can realistically appear in a full-batch document failure. Blast radius is bounded — a spurious ERROR log line, not a phase reset — so this is Medium, not higher. Worth tightening ssl→word-boundary or requiring the security/exception-type wording for the message path. The existing security_exception case has an explicit test proving it is intentional.

Notes (non-blocking)

• dominantFailure(failuresByMessage) is computed twice per rejected batch — once for the throttle key at BulkProcessorListener.java:196 and again inside systemicFailureEscalation. Negligible cost; only flagging for clarity.
• Logger.errorEvery faithfully mirrors warnEvery (same logMap, same Objects.hashCode(messageKey.intern()) keying). It shares the logMap with warnEvery, so a hashCode collision between an error key and a warn key could suppress one — but this matches the pre-existing warnEvery design and is not introduced here.

Everything else — the "does NOT escalate on partial/unclassifiable" policy, per-item detail dropped to DEBUG, the neutral IndexConfigHelper seam avoiding a vendor import in the reindex layer — is sound and matches the stated intent.

· issue-36222-shadow-bulk-failure-escalation

[wezell]

should we wrap this in a Try.of or try/catch and log if there are any exceptions before rethrowing? I know exceptions can happen in the reactor class which basically take it down without any messages (older issue, probably fixed), just want this to be as robust as possible?