JIT: Fix correctness issues in RangeOps

[EgorBo]

Three correctness bugs in src/coreclr/jit/rangecheck.h:

1. RangeOps::Or — unsound upper bound

Or used ApplyRangeOp, which pairs (r1.lo, r2.lo) and (r1.hi, r2.hi). That works for monotonic operators but is wrong for bitwise OR:

[3..5] | [4..7]  ->  lo = 3|4 = 7, hi = 5|7 = 7   (claims [7..7])

The actual minimum is max(3, 4) = 4 (achieved at x=4, y=4), so the computed lower bound is too large and the range is too narrow. Downstream folding could then mis-prove relops, e.g. (x|y) != 7 was folded to false in:

static int Test(int x, int y)
{
    if (x < 3 || x > 5) return -1;
    if (y < 4 || y > 7) return -1;
    int r = x | y;
    if (r != 7) return 1;   // folded to false -> Test(4, 4) returned 0
    return 0;
}

Replaced with a direct constant-range computation for non-negative ranges:

• lo = max(a, c) — OR can only set more bits, so result >= max(x, y)
• hi = bit-fill mask of max(b, d), i.e. (1 << (top_bit + 1)) - 1 — bits in result can be at most as high as the top bit of max(b, d)

Note that a naive b | d would also be unsound: e.g. [3..4] | [4..4] would give upper = 4|4 = 4, but 3|4 = 7 is also reachable.

2. RangeOps::Multiply — signed-overflow UB on the unsigned path

When unsignedMul = true, the four corner products use plain signed int multiplication after only an unsigned overflow check passes. For example [0..50000] * [0..50000] with unsignedMul = true: all four MulOverflows(_, _, /*unsigned*/ true) checks pass (2.5e9 fits unsigned), but 50000 * 50000 as int*int is signed-overflow UB. Added a second overflow check using signed semantics whenever unsignedMul is set.

3. RangeOps::ConvertShiftToMultiply — 1 << 31

The guard allowed shift count = 31, producing 1 << 31. That is defined to INT_MIN in C++20, undefined behavior pre-C++20, and either way the subsequent Multiply cannot use it usefully. Tightened the upper bound to 30.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR updates CoreCLR JIT range analysis (RangeOps in rangecheck.h) to avoid unsound range computations and undefined behavior in a few integer operations that can feed downstream assertion propagation and folding.

Changes:

• Fix RangeOps::Or to compute a conservative, sound result range for non-negative constant ranges (instead of applying a monotonic-style pairing that is invalid for OR).
• Prevent signed-overflow UB in RangeOps::Multiply when the operation is being analyzed with unsignedMul=true by re-checking with signed overflow semantics before performing int multiplications.
• Tighten ConvertShiftToMultiply to disallow shift count 31 to avoid problematic 1 << 31 behavior and keep subsequent multiplication reasoning safe.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[BoyBaykiller]

Just noting that I am already adding a similar function here. I already use it for the existing sub->xor transform in morph.
In particular BitsetFromRange(0, max(b, d)) has the same effect.

[EgorBo]

@BoyBaykiller yeah, my copilot even tried to add XOR along the way there. I'll take a look at your PR tomorrow, I'm not sure I like the XOR impl