Fix overlap filter to use segmentation masks when available (#1987)

[Nishant-ZFYII]

What does this PR do?

This PR fixes the Overlap Filter workflow block (roboflow_core/overlap@v1) so that instance segmentation predictions use masks for overlap checks, rather than implicitly falling back to bounding-box-only overlap.

Before this change, OverlapBlockV1.run() only used predictions.xyxy, which could produce false positives for segmented objects with irregular shapes (their bounding boxes overlap even when the masks do not).

This PR:

• Uses pixel-level mask overlap when segmentation masks are present (and consistent with detection length)
• Keeps the existing bbox overlap behavior as a fallback when masks are not present
• Makes output ordering deterministic by sorting indices before indexing into sv.Detections
• Adds/updates unit tests covering the regression and both mask/bbox code paths

Related Issue(s): Fixes #1987

Type of Change

• Bug fix (non-breaking change that fixes an issue)

Testing

• I have tested this change locally
• I have added/updated tests for this change

Test details:

• Updated/expanded unit tests for the overlap block to include:
  • A regression test demonstrating the original false-positive case for instance segmentation (bbox overlap but masks do not overlap)
  • Direct tests for the new mask overlap helper (Center Overlap and Any Overlap)
  • A fallback test verifying bbox-only behavior still works when masks are absent
  • Existing sanity tests for true-positive overlap and no-overlap scenarios

Example command run locally:

• pytest tests/workflows/unit_tests/core_steps/analytics/test_coords_overlap_v1.py -v

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where necessary, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have updated the documentation accordingly (if applicable)

Additional Context

I’m new to the repo, so please let me know if you’d prefer this behavior behind a new version (e.g. @v2) or if the current approach (fixing @v1 since it already accepts instance segmentation predictions) is aligned with the project’s versioning expectations. Happy to adjust based on maintainer feedback.

[CLAassistant]

All committers have signed the CLA.

[Nishant-ZFYII]

Good Day,
Can you please look into the PR and let me know if any other changes are required.

Thanks and Regards.
Nishant.

[grzegorz-roboflow]

(Claude)
Performance: full-image np.logical_and for "Any Overlap" (minor)

np.logical_and(overlap_mask, other_mask).any() allocates a full (H, W) intermediate boolean
array on every comparison. For large images (e.g., 4K = 3840x2160) with many detections, this can be
costly. A more efficient approach would be to restrict the comparison to the intersection of the two
bounding boxes:

Suggested change

	return bool(np.logical_and(overlap_mask, other_mask).any())
	x1 = max(overlap_bbox[0], other_bbox[0])
	y1 = max(overlap_bbox[1], other_bbox[1])
	x2 = min(overlap_bbox[2], other_bbox[2])
	y2 = min(overlap_bbox[3], other_bbox[3])
	if x1 >= x2 or y1 >= y2:
	return False
	return bool(np.logical_and(overlap_mask[y1:y2, x1:x2], other_mask[y1:y2, x1:x2]).any())

This will also require to pass extra param to this method, so signature will need to receive: overlap_bbox: list[int]
Then, it will also need predictions.xyxy[oi] passed when OverlapBlockV1.masks_overlap is called and some test assertions fixed
Would be nice to use named params when masks_overlap is called (though it's up to you)

[grzegorz-roboflow]

hi @Nishant-ZFYII, thank you for this contribution! I have ran your PR through Claude and got one suggestion that I think makes sense in terms of performance, please let me know what are your thoughts about this.