Skip to content

[LAPACK][rocSOLVER] Fix incorrect QR results on repeated runs (#626) - #758

Open
zjin-lcf wants to merge 7 commits into
uxlfoundation:developfrom
zjin-lcf:fix/rocsolver-qr-stream-sync-626
Open

[LAPACK][rocSOLVER] Fix incorrect QR results on repeated runs (#626)#758
zjin-lcf wants to merge 7 commits into
uxlfoundation:developfrom
zjin-lcf:fix/rocsolver-qr-stream-sync-626

Conversation

@zjin-lcf

Copy link
Copy Markdown
Contributor

Follow-up to #748: the same #626 failure happens on the rocSOLVER backend, and this PR fixes it there.

Stacked on #748 — it is branched off that PR's head, so until #748 merges the diff here also contains its commits. The rocSOLVER-specific work is the last three commits.

The failure on ROCm

The geqrf + orgqr diagonal regression test added in #748 is backend agnostic, and it fails out of the box on an AMD Instinct MI100 (gfx908, ROCm 7.1.1), with exactly the #626 symptom — for a diagonal input the diagonal of Q stays at the input value on some runs:

[3]: 1024 1024 6 # FAIL
        run 5: Q(863, 863) = 2, expected 1

Root cause

rocSOLVER calls only enqueue work on the HIP stream of the surrounding host task / native command. When ext_codeplay_enqueue_native_command is available, this backend skips the hipStreamSynchronize it otherwise performs, so the call returns with the factorization still in flight.

That is only safe if a dependent submission is ordered against that stream, and it is not: urEnqueueNativeCommandExp picks the stream through getNextComputeStream(NumEventsInWaitList, EventWaitList, Guard), which merely tries to reuse the stream of a dependency and enqueues no hipStreamWaitEvent for the wait list. When the heuristic declines, orgqr runs on a second stream that never waits for geqrf.

Unlike getrf/potrf/..., the Householder routines have no info output in rocSOLVER, so they also do not get the implicit host-side wait that lapack_info_check performs — which is what makes the cuSOLVER fix in #748 work. There is no equivalent argument to pass here.

Reduced to plain SYCL + rocSOLVER (no oneMath), on the MI100:

variant wrong results
two dependent native commands, relying on the SYCL event dependency 5/15 runs
same, plus an explicit hipStreamWaitEvent between them 0/15 runs
same, on an in-order queue (single stream) 0/10 runs

The same reduction with plain hipMemsetAsync instead of rocSOLVER does not fail, because a single stream is used often enough for the heuristic to hold.

Changes

  • Wait for the stream before returning from a rocSOLVER call. This is what the backend already did on toolchains without ext_codeplay_enqueue_native_command; the unsynchronized path is dropped rather than kept for a case where it is not sound. Fixing the ordering in the UR HIP adapter (both it and the CUDA one look affected) would let this be relaxed again.
  • Take the HIP stream from the interop handle, as [LAPACK][cuSOLVER] Fix incorrect QR results on repeated runs (#626) #748 does for cuSOLVER. Both spellings resolve to the same stream inside a native command today, so this is an alignment with the other backends, not the fix.
  • Check the devInfo allocation, mirroring the create_devinfo helper [LAPACK][cuSOLVER] Fix incorrect QR results on repeated runs (#626) #748 adds to cuSOLVER.

Test plan — AMD Instinct MI100, ROCm 7.1.1, oneMath built with -DENABLE_ROCSOLVER_BACKEND=ON -DTARGET_DOMAINS=lapack and an open-source DPC++ (intel/llvm, HIP backend)

geqrf + orgqr regression test from #748, n = 256 / 512 / 1024, 6 runs each, buffer and USM, RT and CT APIs:

build result
#748 head, unpatched FAIL — 3 of 4 test cases, from run 0 to run 5 depending on the run
this PR PASS — 4 of 4, repeated invocations

Full LAPACK suite against Netlib reference (ILP64 LAPACKE), run one suite per process, unpatched vs. patched:

  • 172 passed with this PR vs. 169 unpatched; the difference is exactly the QR regression tests.

  • 112 failures remain, all in the *BatchGroup suites, which throw unimplemented for the group-batch APIs. Identical on the unpatched build.

  • Getrf/Getrs USM abort in both builds inside the DPC++ runtime (ProgramManager::getDeviceKernelInfo assertion) before reaching any oneMath code; unrelated to this change.

  • Reproduced [cuSOLVER] Incorrect results in QR decomposition on CUDA #626 on ROCm and confirmed the fix.

  • No change in the rest of the rocSOLVER LAPACK suite.

  • CI: rocSOLVER LAPACK functional tests.

The rocBLAS, cuBLAS and cuSOLVER backends share the same #ifdef SYCL_EXT_ONEAPI_ENQUEUE_NATIVE_COMMAND pattern and are exposed to the same hazard; I have left them alone here.

zjin-lcf and others added 7 commits July 27, 2026 21:53
The Householder-family cuSOLVER routines (geqrf, orgqr, ormqr, gebrd,
orgbr, orgtr, ormtr and the complex ung*/unm* variants) passed nullptr
for cuSOLVER's devInfo argument and skipped the lapack_info_check that
every other routine (getrf, potrf, ...) performs.

Besides losing all error reporting, this omitted the implicit
synchronization that lapack_info_check performs (it reads devInfo back
via a blocking queue.wait()). Without it, the SYCL event returned from
the native-command submission could signal before the cuSOLVER kernels
had finished, so a subsequent memcpy read partially-computed data. This
produced nondeterministic, size-dependent wrong results - e.g. QR of a
diagonal matrix returning Q diagonals stuck at the input value on the
second and later runs for n >= 256 (issue uxlfoundation#626).

Allocate a real devInfo and call lapack_info_check in all of these
routines (buffer and USM paths), matching the established pattern. This
both restores error checking and removes the race.

Also align CusolverScopedContextHandler::get_stream with the cuBLAS
backend by returning the interop handle's native queue
(ih.get_native_queue()) instead of the queue's default stream, so
cuSOLVER work is enqueued on the stream the SYCL runtime tracks for
native-command completion.

Fixes uxlfoundation#626.
Adds a functional test that repeatedly runs geqrf + orgqr on a known
diagonal matrix (whose Q is the identity) and verifies Q's diagonal is 1
on every run. This guards against a regression of the cuSOLVER
synchronization bug from uxlfoundation#626, where run 2+ returned corrupted results.

Co-authored-by: Cursor <cursoragent@cursor.com>
…LVER

A failed malloc_device silently degraded into passing a null devInfo to
cuSOLVER, which disables the routine's error reporting - exactly the
situation that hid the bug from uxlfoundation#626. Route every USM devInfo allocation
through a create_devinfo helper that throws device_bad_alloc instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
…sion test

The test creates a default, out-of-order queue, so the USM orgqr call was
not ordered after geqrf. Pass the geqrf event as a dependency; the buffer
path keeps relying on accessor ordering.

The test needs no reference implementation, so stop dropping the whole
LAPACK domain when Netlib LAPACKE is absent - only the tests that compare
against a reference are skipped now.

Co-authored-by: Cursor <cursoragent@cursor.com>
rocSOLVER calls only enqueue work on the HIP stream of the surrounding
host task or native command. A submission that depends on that work may
be scheduled on a different stream of the queue's stream pool without
being ordered against this one, so it can read a factorization that has
not finished yet. The Householder routines (geqrf, orgqr, ...) have no
`info` output, so unlike getrf/potrf/... they do not get an implicit
host-side wait from `lapack_info_check` that hides the problem.

Wait for the stream before returning from the rocSOLVER call, which is
what this backend already did on toolchains without
`ext_codeplay_enqueue_native_command`.

On an MI100 this turns the geqrf+orgqr diagonal regression test for uxlfoundation#626
from failing on repeated runs into passing.

Co-authored-by: Cursor <cursoragent@cursor.com>
`ext_codeplay_enqueue_native_command` requires work to be enqueued on the
stream of the interop handle, which is the stream the SYCL runtime records
the submission's completion event on. Query it the same way the cuBLAS and
cuSOLVER backends do instead of asking the queue for its native stream.

Both currently resolve to the same stream inside a native command, so this
is an alignment with the other backends rather than a behavioural change.

Co-authored-by: Cursor <cursoragent@cursor.com>
…SOLVER

A failed `malloc_device` for the `devInfo` output silently disabled the
routine's error reporting, since rocSOLVER treats the resulting null
pointer as "do not report". Throw `device_bad_alloc` instead, matching the
cuSOLVER backend.

Co-authored-by: Cursor <cursoragent@cursor.com>
@zjin-lcf
zjin-lcf requested review from a team as code owners August 12, 2026 20:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant