search: attribute permit-wait latency to the blocking resource#6589
Draft
loutPhilipps wants to merge 3 commits into
Draft
search: attribute permit-wait latency to the blocking resource#6589loutPhilipps wants to merge 3 commits into
loutPhilipps wants to merge 3 commits into
Conversation
Per-split leaf work was spawned without propagating the current span, starting a new trace instead of nesting under multi_index_leaf_search. Also prunes collinear gRPC-adapter spans, adds spans for distinct phases (plan_splits_for_root_search, list_relevant_splits, fetch_docs_in_split, leaf_search_single_split_wrapper), splits the CPU-pool queue wait out of tantivy_search, and surfaces search_partial_hits_phase at INFO. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds query AST, split and doc count, time range to the root_search span attributes, and warmup_mb/downloaded_mb attributes on the warmup span. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
waiting_for_leaf_search_split_semaphore was a bare timer, so a long wait couldn't be attributed to the two things a split-search permit can block on: warmup/download slots or the memory budget. The actor now tracks the resource blocking the head of the queue in a shared BlockReasonHandle (an atomic). Because permits are served in order, whatever blocks the head blocks every waiter, so this queue-level reason is correct for a waiter at any position. Each SearchPermitFuture reads the handle, and a drop guard (WaitBlockReasonRecorder) on the wait span records blocked_on when the wait ends -- granted or cancelled -- gated on the wait being long (>=1ms) so instant grants stay unlabeled. Attributing on the wait side (not from the granted permit) is what lets the long, deadline-cancelled waits -- the ones we care about -- carry a reason even though they never get a permit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
loutPhilipps
force-pushed
the
lout.philipps/span-attributes
branch
from
July 21, 2026 12:46
248f437 to
ecb2499
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
waiting_for_leaf_search_split_semaphorewas a bare timer, so a long wait couldn't be attributed to the two things a split-search permit can block on: warmup/download slots or the memory budget.The actor now tracks the resource blocking the head of the queue in a shared
BlockReasonHandle(an atomic). Because permits are served in order, whatever blocks the head blocks every waiter, so this queue-level reason is correct for a waiter at any position. EachSearchPermitFuturereads the handle, and a drop guard (WaitBlockReasonRecorder) on the wait span recordsblocked_onwhen the wait ends — granted or cancelled — gated on the wait being long (≥1ms) so instant grants stay unlabeled. Attributing on the wait side (not from the granted permit) is what lets the long, deadline-cancelled waits — the ones we care about — carry a reason even though they never get a permit.Stacked on #6588.
🤖 Generated with Claude Code