fix(core): skip non-hole remnants in OGC extraction

[Filyus]

Problem

extract_ogc can panic on a self-touching / near-degenerate subject shape when using OGC output.

A minimized standalone reproducer is available here:

https://github.com/Filyus/ioverlay-repro

With i_overlay = 7.0.0:

cargo run
# assertion failed: overlay_rule.is_fill_top(link.fill)

cargo run --release
# index out of bounds: the len is 40 but the index is 9223372036854775807

The release panic is a downstream symptom. The debug assertion shows the earlier invariant break.

Root Cause

The first OGC pass classifies contours and skips hole contours for a second pass. In a self-touching sliver case, skip_contour can mark a larger traversal as HoleVisited. That traversal may include non-hole remnants whose fill does not satisfy:

overlay_rule.is_fill_top(link.fill)

During the second hole pass, the actual hole contour is extracted first. Some non-hole remnants can remain unvisited in the hole-only buffer. The second pass then tries to treat those remnants as hole starts.

In debug this hits:

debug_assert!(overlay_rule.is_fill_top(link.fill));

In release the invalid pseudo-hole can reach join_sorted_holes, where parent lookup returns ContourIndex::EMPTY; EMPTY.index() becomes usize::MAX >> 1, producing the 9223372036854775807 panic.

Fix

When the second OGC pass encounters a HoleVisited remnant whose fill is not valid for the current overlay_rule, consume that contour and continue instead of adding it to holes / anchors.

This keeps the invariant local to OGC extraction:

debug_assert!(overlay_rule.is_fill_top(link.fill));

Only contours that actually satisfy the hole-start condition are passed to hole binding.

This is intentionally not a ShapeBinder fallback: the binder was receiving invalid pseudo-hole input. The fix is applied at the earlier stage where that input is produced.

Regression Test

Adds a minimized 4-contour / 13-point float regression case derived from the standalone repro. The test uses:

• OverlayRule::Subject
• FillRule::NonZero
• fixed adapter scale 50000
• clean_result = true
• ogc = true
• preserve_output_collinear = true

Verification

cargo test --test crash_tests test_05
cargo test --release --test crash_tests test_05
cargo test --test ocg_tests
cargo test --release --test ocg_tests
cargo test
cargo test --release

The standalone repro also passes when patched to this local branch:

OK: produced 17 shape(s) (no panic)

Supersedes

This supersedes #77. That PR handled the downstream ContourIndex::EMPTY panic in ShapeBinder; this PR fixes the earlier OGC extraction invariant violation that produces the invalid pseudo-hole.

Note

Prepared with LLM assistance and validated with a minimized standalone reproducer plus debug/release test suites.

[NailxSharipov]

Thank you for your time and research. This now looks like a real problem. Let me look into it, and I'll give you my feedback.

[codecov]

Codecov Report

❌ Patch coverage is 50.00000% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
iOverlay/src/core/extract_ogc.rs	50.00%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

[NailxSharipov]

I've uncovered a real problem.

When traversing a contour, I use a node as an end detector, but when an end node has multiple edges, like in the example below, this is a bad idea. The correct solution is to use the start edge as the end mark.

If you still want to contribute, you can leave the test case without solution; otherwise, I'll close this pull request.

Anyway thanks for the help

[Filyus]

You're welcome! I'll update the PR or create a new one today.

[Filyus]

Closing this implementation PR in favor of a separate test-only PR as requested. The minimized regression case is now in #79 without a proposed solution.