fix(opensearch): run the full reindex after an absorbed OS halt (#36222)

[fabrizzio-dotCMS]

What

In a dual-write phase, an OpenSearch permission failure silently downgraded a full reindex into a no-op — on Elasticsearch too.

Found while reproducing QA's third TC-056 report on #36222. QA described it as "the reindex runs, shows progress, and when it finishes nothing happens". That report was right, and the cause is not the OpenSearch half.

Why it happens

OSIndexAPIImpl.indexExists returns false for any failure, so a 403 on the exists probe is indistinguishable from a missing index:

1. indexReadyOS() → false (the OS user may not even probe those names)
2. indexReady() → false (phases 1/2 require both engines)
3. fullReindexStart() takes its else branch → initIndex() instead of the reindex

The request returns HTTP 200, the progress bar completes, and no index is created in either engine. The Full reindex started line never appears.

Pressing Reindex a second time works — but only because the first press halted the migration to phase 0. Elasticsearch then advances to a new timestamp while OpenSearch stays behind, and the two engines diverge. The readiness endpoint (#36360) flags that aftermath as MISSING_COUNTERPART / safeToAdvance: false.

Same button, same build	newIndexPath	New ES index
Phase 1, OS role narrowed (403)	[,] — empty	none
Phase 0 (control)	[working_…, live_…]	created

How

• fullReindexStart re-evaluates once after initIndex() when the migration was halted during it, and then runs the reindex the caller asked for. Gated on the halt rather than a bare indexReady() re-check, so a fresh install — where initIndex() legitimately turns it true — keeps its current behaviour. The reindex body moves to startFullReindex(), shared by both paths.
• MigrationHaltReport (new) keeps the classified cause and its remediation so the REST layer can report why the migration stopped without importing the OpenSearch adapter. Held per JVM, deliberately matching the scope of the runtime-only phase reset.
• POST /api/v1/esindex/reindex compares the phase across the call and returns the reason in messages.

What deliberately does NOT change

• The degradation to Elasticsearch-only stays. fix(opensearch): degrade OS index-bootstrap failures to ES-only in dual-write phases (#36222) #36783's policy is right; this only completes the operation the user asked for afterwards. Being silent about it was the defect.
• indexExists still swallows 403 as false. Distinguishing it would push an OpenSearch permission problem into paths that serve Elasticsearch — exactly what fix(opensearch): degrade OS index-bootstrap failures to ES-only in dual-write phases (#36222) #36783 set out to avoid, for a much wider blast radius.
• The "both engines ready" rule in phases 1/2 is intentional and untouched.

Testing

Reproduced against a real OpenSearch 3.x with the security plugin and a role scoped to a non-matching index pattern (the cluster-id mismatch TC-056 describes, induced through the role because the cluster id is immutable after first boot). One click now yields:

Migration phase reset to PHASE_0 (was PHASE_1)
The OpenSearch migration was halted while preparing the full reindex. Continuing on Elasticsearch only…
Full reindex started by user: admin@dotcms.com          ← 32 ms later, same click
New indicies: [working_20260806224715, live_20260806224715]

and the response carries:

The OpenSearch migration was switched off (was PHASE_1_DUAL_WRITE_ES_READS) and dotCMS is serving from Elasticsearch only. Cause: AUTH_FORBIDDEN (…). Rejected index names: … Fix the cause and re-enable the migration phase when ready.

./mvnw verify -pl :dotcms-integration -Dcoreit.test.skip=false \
    -Dopensearch.upgrade.test=true -Dit.test=OsBootstrapForbiddenIndexTest

Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 — one new case in OsBootstrapForbiddenIndexTest (already registered in OpenSearchUpgradeSuite) asserting the ES suffix is non-empty and the halt reason is recorded. It fails before this change.

Not covered automatically: the REST messages payload, verified by hand against the stack above.

Follow-up, not in this PR

The operator still learns nothing from the UI — the message rides in the response envelope, and nothing renders it yet. And the readiness endpoint reports the consequence (MISSING_COUNTERPART) but recommends "run a full reindex", which cannot work while the role is wrong; teaching it to probe OS permissions would close that loop.

🤖 Generated with Claude Code

[claude]

Claude finished @fabrizzio-dotCMS's task in 6m 51s —— View job

---

Code Review: full reindex after an absorbed OS halt

I read the diff and the surrounding index/REST layers (ContentletIndexAPIImpl.fullReindexStart / initIndex / handleOsBootstrapFailure, MigrationHaltReport, ESIndexResource.startReindex, and the two other haltMigration() call sites).

The core fix is sound: fullReindexStart() now re-runs startFullReindex() after an absorbed halt, gated on the halt transition rather than a bare indexReady() re-check, so the fresh-install path is preserved. The refactor into startFullReindex() is clean and the transaction/@WrapInTransaction semantics still hold (Config-based halt survives even if the ES reindex rolls back). Good test coverage for the primary scenario.

One correctness concern worth addressing before merge:

New Issues

• 🟡 Medium: dotCMS/src/main/java/com/dotcms/rest/api/v1/index/ESIndexResource.java:385-394 — migrationHaltMessage can report a stale/mis-attributed cause. The method fires whenever the phase went started → not-started during the request (for any reason) and then reads MigrationHaltReport.last(). But only handleOsBootstrapFailure (ContentletIndexAPIImpl.java:1182) records a report — the other two halt paths, checkAndInitializeIndex (:560) and the endpoint-overlap gate in bootstrapAndPointOS (:1061), call haltMigration() without recording one. The :1061 gate is reachable from this exact call chain (fullReindexStart → initIndex → bootstrapAndPoint/initOSCatchup → bootstrapAndPointOS). MigrationHaltReport.LAST is a JVM-global AtomicReference that is only ever set, never cleared.
  • Failure scenario: an AUTH_FORBIDDEN halt records a report earlier in the JVM's life → operator re-enables the phase → a later reindex halts via the endpoint-overlap gate (:1061, no new report). The REST response (and the Logger.warn at ContentletIndexAPIImpl.java:1368-1371) then reports the stale AUTH_FORBIDDEN cause and its rejected index names instead of the actual endpoint-overlap cause — actively misleading the operator toward the wrong fix.
  • Assumption: the :1061 gate is the only non-recording halt reachable mid-reindex; checkAndInitializeIndex is startup-only. What to verify: whether any other runtime path can reset the phase during a reindex.
  • Suggested fix: either record a MigrationHaltReport (or clear LAST) at the :1061 and other haltMigration() sites so the last report always matches the current halt, or correlate the report to this request (e.g. clear LAST before the call and only trust a report recorded after). Fix this →

Notes (non-blocking)

• MigrationHaltReport.LAST is a static AtomicReference never reset between integration tests in the same JVM. Only this new test asserts on it today, so no current flakiness — but a future test relying on last() being empty could see leftover state. Consider a @Before/@After clear if more tests start reading it.
• The startReindex degraded response (ESIndexResource.java:365) returns HTTP 200 with the message in messages — consistent with the PR's stated "the reindex succeeds on ES" contract. No @Schema change needed since the method returns raw Response.

Everything else in the diff — the startFullReindex() extraction, the halt-gated re-evaluation, and the indexSuffixOS invariant — looks correct.
· branch issue-36222-full-reindex-after-os-halt