HBASE-30256 FuzzyRowFilter mishandles the fuzzy-mask encoding on the …

[liuxiaocs7]

…… (#8412)

(cherry picked from commit 69d5102)

[Copilot]

Pull request overview

This PR addresses HBASE-30256 by fixing FuzzyRowFilter mask/key preprocessing so that the “no-unsafe” path correctly interprets and normalizes fuzzy-mask encodings (including legacy/v1 and unsafe peer encodings), and so that serialization/deserialization does not leak internal mask representations across platforms.

Changes:

• Normalize incoming masks on the no-unsafe path (including legacy unsafe encodings) and ensure non-fixed key bytes are cleared so hinting produces the smallest matching row.
• Make serialization (toByteArray), getFuzzyKeys, equals, and hashCode consistently use the public {0 (fixed), 1 (non-fixed)} mask form.
• Add end-to-end and wire-level tests covering unsafe/no-unsafe behavior, legacy/v1/v2 encodings, and round-trip/equality semantics.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FuzzyRowFilter.java	Fixes mask normalization and preprocessing across unsafe/no-unsafe paths; ensures public-form masks are exposed/serialized and equality/hash semantics are consistent.
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFuzzyRowFilterWoUnsafe.java	Adds comprehensive “no-unsafe” path coverage, including hinting and wire-format normalization cases.
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFuzzyRowFilter.java	Adds unsafe-path regression tests for serialization leakage, equality/hash stability, and v1/v2 wire behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.