api: emit canonical heading path on treewalk citations (HAL-70)

[hallelx2]

What

The engine now emits a real structural heading path (title_path) on every TreeWalk citation. This is the fix for path_correct@1 = 0% (root-cause analysis on HAL-70).

Before: a citation was {start_page, end_page, section_ids, quote}. The "structure-aware citation" differentiator was not in the response payload — the consumer had to reverse-engineer a heading path from the parser's chunk titles, which are the wrong vocabulary, so the bench's path metric was structurally 0% (while span_in_top1 was 20% — content sometimes right, label always wrong).

How

buildTreeWalkCitations resolves the document's LLM TOC tree (through the strategy's existing TOC provider) into a section-ID → heading-path map via tree.BuildHeadingPaths (HAL-109, page-range containment), then attaches the primary (first/earliest-page) section's heading path as title_path on each citation — exactly the field the path-correctness metric reads.

Both the non-streaming and SSE paths build citations through this one chokepoint, so both gain the field.

Degradation

headingPathsForDoc returns nil on every failure mode (no provider, ErrNoTOC, fetch error, unparseable TOC). A nil map is safe to index → title_path is simply omitted and the citation is byte-for-byte what it was before. Never worse than today.

Tests

TestBuildTreeWalkCitations_EmitsHeadingPath (citation carries ["Setup","Install"] for its primary section) and ..._NoTOCOmitsHeadingPath (graceful omission on ErrNoTOC). go build ./..., go test ./internal/api/... ./pkg/tree/..., gofmt, go vet all clean.

Stacked on HAL-109

Base branch is the HAL-109 PR (#39) because this consumes tree.BuildHeadingPaths. The companion bench change (point the retriever at /v1/answer/treewalk and consume the engine title_path) ships in vectorless-bench. The before/after fixture-bench numbers are pending a live run against an engine carrying both PRs.

Closes HAL-70

Summary by Sourcery

Emit canonical structural heading paths on TreeWalk citations using the document TOC while preserving previous behavior when TOC data is unavailable.

New Features:

• Add a title_path field to TreeWalk citations populated from the document's canonical heading path for the primary cited section.

Bug Fixes:

• Ensure TreeWalk citations expose the correct structural heading path used by path-correctness metrics instead of requiring consumers to infer it from chunk titles.

Enhancements:

• Introduce a helper to resolve section-ID-to-heading-path mappings from persisted TOC data with safe degradation when TOC retrieval or parsing fails.

Tests:

• Add tests verifying citations include the expected heading path when TOC data exists and gracefully omit title_path when TOC data is missing.

[sourcery-ai]

Reviewer's Guide

Adds canonical structural heading paths to TreeWalk citations by resolving TOC-derived section heading paths and emitting them as title_path on citations, with safe degradation when TOC data is unavailable and tests covering both presence and omission cases.

Sequence diagram for TreeWalk citation building with title_path

sequenceDiagram
    actor User
    participant API
    participant Deps
    participant TOCProvider as TOCProvider
    participant Tree as tree

    User->>API: /v1/answer/treewalk
    API->>Deps: buildTreeWalkCitations(ctx, tree, res)
    activate Deps
    Deps->>Deps: headingPathsForDoc(ctx, tree)
    alt TOC available
        Deps->>TOCProvider: GetTOC(ctx, tree.DocumentID)
        TOCProvider-->>Deps: raw TOC JSON
        Deps->>Tree: BuildHeadingPaths(tree.Root, nodes)
        Tree-->>Deps: headingPaths (sectionID→path)
    else TOC unavailable or invalid
        Deps-->>Deps: headingPaths = nil
    end

    loop for each source in CitationSources(res)
        Deps->>Deps: build citation map
        alt len(sectionIDs) > 0 and headingPaths[primary] exists
            Deps-->>Deps: set c[title_path] = headingPaths[sectionIDs[0]]
        else no heading path
            Deps-->>Deps: omit title_path
        end
    end
    Deps-->>API: citations
    deactivate Deps
    API-->>User: response with citations (possibly with title_path)

Loading

File-Level Changes

Change	Details	Files
Emit canonical heading paths (title_path) on TreeWalk citations based on primary section IDs.	Call a new helper to precompute a sectionID-to-heading-path map before iterating citation sources. When building each citation, look up the primary (first) section’s heading path and, if present, attach it as the title_path field. Preserve existing citation shape when no heading path is available so responses remain backward compatible.	internal/api/treewalk.go
Introduce headingPathsForDoc helper to resolve section heading paths from persisted TOC data with safe failure modes.	Check for nil tree, missing strategy, or missing TOC provider and return nil if any are absent. Fetch TOC bytes via the strategy’s TOC provider and return nil on errors or empty payloads, including ErrNoTOC. Unmarshal TOC JSON into TOCNode structures, logging a warning and returning nil on parse failure. Use tree.BuildHeadingPaths to construct and return the sectionID-to-heading-path map.	internal/api/treewalk.go
Add tests verifying title_path emission and graceful omission when TOC is unavailable.	Create a stub TOC provider and helper to build a canonical TOC matching the existing test tree layout. Add TestBuildTreeWalkCitations_EmitsHeadingPath to assert that a citation for pages 1-2 gets section_ids[0] == sec_a1 and title_path ["Setup","Install"]. Add TestBuildTreeWalkCitations_NoTOCOmitsHeadingPath to assert that when the TOC provider returns ErrNoTOC, citations still carry section_ids but omit title_path.	internal/api/treewalk_citations_test.go

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 993e1c8c-9fab-443a-94e0-4ca708cf0aa9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch halleluyaholudele/hal-70-engine-fix-citation-path-correctness-in-retrieval

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• In headingPathsForDoc, all TOC fetch errors (including transient or unexpected ones) are silently treated as nil paths; consider differentiating retrieval.ErrNoTOC from other errors so non-"no TOC" failures are logged or surfaced for easier diagnosis while still degrading gracefully for the expected missing-TOC case.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- In headingPathsForDoc, all TOC fetch errors (including transient or unexpected ones) are silently treated as nil paths; consider differentiating retrieval.ErrNoTOC from other errors so non-"no TOC" failures are logged or surfaced for easier diagnosis while still degrading gracefully for the expected missing-TOC case.

## Individual Comments

### Comment 1
<location path="internal/api/treewalk.go" line_range="390-391" />
<code_context>
+	if t == nil || t.Root == nil || d.TreeWalkStrategy == nil || d.TreeWalkStrategy.TOC == nil {
+		return nil
+	}
+	raw, err := d.TreeWalkStrategy.TOC.GetTOC(ctx, t.DocumentID)
+	if err != nil || len(raw) == 0 {
+		return nil
+	}
</code_context>
<issue_to_address>
**suggestion (bug_risk):** Distinguish between "no TOC" and other GetTOC errors to avoid silently hiding operational issues.

Currently any `GetTOC` error (including infra/transport issues) is treated as "no TOC" and dropped with no logging, which can mask real problems. It’d be better to special‑case a sentinel like `retrieval.ErrNoTOC` (if available) and log other errors, e.g. at debug level:

```go
raw, err := d.TreeWalkStrategy.TOC.GetTOC(ctx, t.DocumentID)
if err != nil {
    if !errors.Is(err, retrieval.ErrNoTOC) {
        d.Logger.Debug("answer/treewalk: TOC fetch failed; citations omit heading paths",
            "err", err, "document_id", t.DocumentID)
    }
    return nil
}
if len(raw) == 0 {
    return nil
}
```

This keeps graceful degradation while exposing unexpected failures for observability.

Suggested implementation:

```golang
	raw, err := d.TreeWalkStrategy.TOC.GetTOC(ctx, t.DocumentID)
	if err != nil {
		// Distinguish between "no TOC" and other errors so we don't silently hide
		// operational issues. retrieval.ErrNoTOC is expected; everything else is logged.
		if !errors.Is(err, retrieval.ErrNoTOC) {
			d.Logger.Debug("answer/treewalk: TOC fetch failed; citations omit heading paths",
				"err", err,
				"document_id", t.DocumentID,
			)
		}
		return nil
	}
	if len(raw) == 0 {
		return nil
	}

```

To make this compile, ensure:
1. `errors` is imported in this file:
   - Add `import "errors"` to the import block if it's not already present.
2. The `retrieval` package (which defines `ErrNoTOC`) is imported:
   - Add the correct import path for `retrieval` (for example `github.com/yourorg/yourrepo/internal/retrieval` or whatever is used elsewhere in the codebase).
3. If the sentinel error is named slightly differently (e.g. `retrieval.ErrNoToc` or similar), adjust the constant name in the `errors.Is` call accordingly.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[hallelx2]

Thanks @sourcery-ai — fixed in 44c8753: headingPathsForDoc now special-cases retrieval.ErrNoTOC (silent, expected) and logs any other GetTOC error at debug, while still degrading to no heading paths. Good catch on the masked-failure risk.