fix(sso): bound the Entra ID parent group walk when Microsoft Graph stops answering - #3261
Open
marevol wants to merge 1 commit into
Open
fix(sso): bound the Entra ID parent group walk when Microsoft Graph stops answering#3261marevol wants to merge 1 commit into
marevol wants to merge 1 commit into
Conversation
…tops answering
The asynchronous parent group walk looped over every direct group id and
called processParentGroup for each one. getParentGroup caught the cache
loader's failure, logged a full stack trace and handed back an empty pair,
so the loop always ran to completion.
When Graph answers /me/memberOf but then fails on
POST /groups/{id}/getMemberGroups with anything other than 429/503,
Authorization_RequestDenied or Request_ResourceNotFound - a 500/502/504, or
a transport failure such as DNS, connection refused, or the connect/read
timeouts - the walk costs one request and one stack trace per direct group
on every login, each waiting out the timeouts. It runs on corelib's shared
TimeoutManager pool, whose overflow policy is CallerRunsPolicy, so a wide
enough walk stalls the timer thread and every other timed task with it.
15.7 was bounded here only by accident: CurlException is unchecked, Guava
wraps it in UncheckedExecutionException, and the catch covered only
ExecutionException, so the exception escaped and the task-level
catch (Exception) aborted the walk at the first group. Widening that catch
was right, but it removed the bound.
The walk now stops after maxConsecutiveGroupLookupFailures (default 3)
consecutive lookups that Graph did not answer, logs one WARN naming how
many group ids were skipped, and still applies everything collected so far.
Consecutive rather than total is deliberate: one permanently broken group id
must not stop the rest of the walk, while an unreachable Graph trips the
bound at once.
- getParentGroup raises ParentGroupLookupException instead of flattening a
failure into an empty pair, so "not answered" is distinguishable from
"this group has no parents". An empty pair stays an answer: the depth
limit, a group with no parents and the throttle skip all keep returning
one, and none of them consumes the bound.
- processParentGroup returns boolean, mirroring what processDirectMemberOf
already does, and catches the exception.
- Only the first failure of a walk is logged with its stack trace; the rest
are one line each. The bound alone does not bound the log volume, because
a walk that alternates success and failure never reaches the consecutive
limit.
- The recursive getParentGroup call inside loadParentGroup now propagates
out of the cache loader, so the outer group id is left uncached when a
nested lookup fails. That is the existing rule: a transient failure is
never pinned in groupCache for the whole TTL.
- maxConsecutiveGroupLookupFailures is listed in the commented property
block of fess_sso++.xml alongside maxGroupDepth.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
EntraIdAuthenticator.scheduleParentGroupLookuplooped over every direct group id and calledprocessParentGroupfor each one.getParentGroupcaught the cache loader's failure, logged a full stack trace and handed back an empty pair, so the loop always ran to completion.When Microsoft Graph answers
/me/memberOfbut then fails onPOST /groups/{id}/getMemberGroupswith anything other than 429/503,Authorization_RequestDeniedorRequest_ResourceNotFound— a 500/502/504, or a transport failure such as DNS, connection refused, or the 10s connect / 30s read timeouts added in 15.8 — the walk costs one request and one stack trace per direct group, on every login, each waiting out the timeouts.It runs on corelib's shared
TimeoutManagerpool:availableProcessors()/2threads, aLinkedBlockingQueueof the same size andThreadPoolExecutor.CallerRunsPolicy. Once that overflows, the walk runs on the TimeoutManager timer thread itself and stalls every other Fess timed task.15.7 was bounded here only by accident:
CurlExceptionis unchecked, Guava wraps it inUncheckedExecutionException, and 15.7 caught onlyExecutionException, so the exception escaped and the task-levelcatch (Exception)aborted the whole walk at the first group. Widening that catch was correct, but it removed the bound.Fix
Stop the walk after
maxConsecutiveGroupLookupFailures(default 3) consecutive lookups that Graph did not answer, log one WARN naming how many group ids were skipped, and still apply everything collected so far.Consecutive rather than total is deliberate: one permanently broken group id must not stop the rest of the walk, while an unreachable Graph trips the bound immediately.
getParentGroupraises a newParentGroupLookupExceptioninstead of flattening a failure into an empty pair, so "Graph did not answer" is distinguishable from "this group has no parents". An empty pair stays an answer: the depth limit, a group with no parents, and the throttle short-circuit all keep returning one, and none of them consumes the bound.processParentGroupreturnsbooleanand catches the exception — the same shapeprocessDirectMemberOfalready uses.TimeoutManagerlambda intorunParentGroupLookup, so it is reachable from a test without a timer thread.maxConsecutiveGroupLookupFailuresis listed in the commented<property>block offess_sso++.xmlalongsidemaxGroupDepth.Known consequence
The recursive
getParentGroupcall insideloadParentGroupnow propagates the new exception out of the Guava loader, so the outer group id is no longer cached when a nested lookup fails. That is the existing rule — a transient failure is never pinned ingroupCachefor the whole TTL — and the recursion is only reachable whenprocessGroupthrew, because on every other path it adds the id it was given and the!groupList.contains(value)guard is false.Tests
Five new tests in the authenticator test class, all built on the existing scripted-Graph helper:
Every new test was mutation-checked: with the production change reverted (and with seven other targeted mutations — no bound, no counter reset, an empty answer counted as a failure, an always-on stack trace, the depth limit counted as a failure, a wrong skipped count, no permission notification) each one fails, and passes again once restored.
Five existing tests changed:
test_getParentGroup_doesNotCacheAFailedLookup→test_getParentGroup_reportsAFailedLookupAndDoesNotCacheIt: still asserts nothing is cached and that recovery works, but now asserts the failure is raised rather than returned as an empty pair.test_getParentGroup_survivesAnUncheckedFailureFromTheLoader→test_getParentGroup_reportsAnUncheckedFailureFromTheLoader: the Guava-wrapped unchecked failure has to reach the same reported-and-counted path as a checked one.test_processParentGroup_callsOverloadWithDepth,test_processParentGroup_respectsDepthLimit,test_processParentGroup_nullUser_depthExceeded: assert the new boolean result, pinning that the depth limit reports success — counting it as a failure would let a misconfiguredmaxGroupDepthstop the whole walk.mvn -o clean test -Dtest='org.codelibs.fess.sso.**'→ Tests run: 248, Failures: 0, Errors: 0, Skipped: 0.mvn -o clean javadoc:jar→ BUILD SUCCESS.