[Bug]: HEVC AMF 10-bit fails to initialize with QVBR / HQVBR (Init() error 10)

[DummyTitan]

Description:

HEVC encoding using AMF fails to initialize when combining 10-bit (P010 / Main10) with QVBR or HQVBR rate control modes. The encoder returns Init() failed with error 10 and no frames are encoded.

This behavior is reproducible with a minimal FFmpeg test and does not depend on input complexity.

Environment:

FFmpeg: 2026-03-30-git-e54e117998-full_build (Gyan.dev)
OS: Windows
GPU: AMD Radeon RX 7900 XT
CPU: AMD Ryzen 9 7950X
RAM: 32GB DDR5 @ 5200 MT/s (XMP enabled)
Encoder: hevc_amf

Reproduction Command (QVBR):

ffmpeg -f lavfi -i testsrc2=size=1280x720:rate=30 -t 5 ^
-vf format=p010le ^
-c:v hevc_amf ^
-profile:v main10 -bitdepth 10 ^
-rc qvbr -qvbr_quality_level 23 ^
-y test_qvbr_10bit.mkv

Reproduction Command (HQVBR):

ffmpeg -f lavfi -i testsrc2=size=1280x720:rate=30 -t 5 ^
-vf format=p010le ^
-c:v hevc_amf ^
-profile:v main10 -bitdepth 10 ^
-rc hqvbr -qvbr_quality_level 23 ^
-y test_hqvbr_10bit.mkv

Observed Behavior:

Encoder fails during initialization:

encoder->Init() failed with error 10
Error while opening encoder
Could not open encoder before EOF
No output frames are produced

Expected Behavior:

Encoder should initialize and encode normally using QVBR / HQVBR in 10-bit mode, as both features are exposed and documented.

Validation / Control Tests:

QVBR / HQVBR + 8-bit (nv12) → Works correctly
10-bit (p010le) + VBR/CBR → Works correctly
10-bit + QVBR / HQVBR → Fails consistently

This isolates the issue specifically to:

HEVC + 10-bit + (QVBR / HQVBR)

Notes:

FFmpeg reports support for:
p010le
main10
rc=qvbr, rc=hqvbr
Therefore, this appears to be a backend/AMF issue rather than misuse of parameters.
Both QVBR and HQVBR likely trigger internal pre-analysis paths that are incompatible with HEVC 10-bit.

Request:

Confirm whether QVBR and HQVBR are officially supported for HEVC 10-bit in current AMF versions
Clarify if this is a known limitation or regression
Provide guidance or timeline for a fix if applicable

Additional Info:
Issue is fully reproducible with synthetic input (testsrc2), eliminating source-related variables.

Conclusion:
QVBR and HQVBR are currently unusable with HEVC 10-bit via AMF despite being exposed as supported, causing a hard failure during encoder initialization.

[DummyTitan]

just incase all the screenshots are from different combinations for conversion using HQVBR and QVBR but the same error appears in all of them no mater what.

[MikhailAMD]

QVBR and HQVBR both use PreAnalysis (PA) component which supports 8-bit only input. https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_Video_PreAnalysis_API.md section 1.2.

[DummyTitan]

Follow-up / Clarification on QVBR/HQVBR 10-bit limitation

Thank you for the clarification regarding pre-analysis and its relation to QVBR/HQVBR.

However, there is still an important inconsistency that needs to be addressed more clearly.

1. Documentation vs actual behavior

The current documentation suggests that:

• HEVC 10-bit (P010 / Main10) is supported
• QVBR / HQVBR are supported

However, in practice:

«QVBR / HQVBR fail when combined with 10-bit encoding»

The documentation references pre-analysis, but it does not explicitly state that:

«QVBR / HQVBR are effectively limited to 8-bit due to the pre-analysis pipeline»

This creates a misleading expectation that these features can be used together.

2. Capability exposure is misleading

From an API and tooling perspective:

• Capabilities report support for both:
  • 10-bit encoding
  • QVBR / HQVBR

But there is no indication that:

«these features are mutually incompatible in practice»

This leads developers to assume the combination is valid.

3. Comparison with other stacks (including VAAPI)

From a broader ecosystem perspective:

• Other vendor implementations (e.g. NVENC, Quick Sync) support:

  • quality-based rate control in 10-bit

• Additionally, alternative Linux paths such as VAAPI expose:

  • QVBR-like controls depending on the driver and platform

While behavior may differ, this raises an important question:

«Is the limitation in AMF due to hardware, or due to the current implementation?»

4. Hardware vs implementation clarification (critical)

At this point, it is unclear whether:

• This is a hardware limitation, or
• This is an AMF implementation limitation (e.g. pre-analysis restricted to 8-bit)

This distinction is extremely important.

5. Request for explicit clarification

Could you please confirm:

1. Is QVBR / HQVBR officially unsupported for HEVC 10-bit on current AMD hardware?

2. If so, is this due to:

   • hardware limitations, or
   • software / pipeline design (pre-analysis constraints)?

3. If this is an implementation limitation:

   • are there plans to support 10-bit + quality-based rate control in future AMF versions?

4. Why this matters

This limitation forces a trade-off:

• Use 10-bit → lose advanced quality-based rate control
• Use HQVBR/QVBR → restricted to 8-bit

For modern workflows (HDR, high-efficiency encoding, consistent quality), this is a significant limitation.

7. Suggestion

To avoid confusion, it would help to:

• Explicitly document:

  «“QVBR/HQVBR require 8-bit pre-analysis and are not supported with 10-bit encode paths”»

• Or restrict invalid combinations at the API level

Conclusion

This is not only a documentation issue, but a functional limitation that affects real-world usage.

Clarifying whether this is a hardware constraint or an implementation limitation would be extremely valuable for developers and users making platform decisions.

[MikhailAMD]

1. I agree that documentation is not clear. Will ask to update.
2. Capabilities in AMF give basic "static" information. It does not take an account combination of parameters set on component. Ultimately, Init() function is the final validator.
3. This is not HW limitation. Note, that VAAPI doesn't have PA at all.
4. I cannot comment on future plans but agree that 10-bit support is highly desired.

[DummyTitan]

Thank you for the clarification, I really appreciate the detailed response.

It is especially helpful to know that this is not a hardware limitation, but related to the current implementation and pre-analysis pipeline.

Documentation

I’m glad to hear that the documentation will be updated.
That will definitely help, as the current behavior is not obvious from the exposed capabilities and can lead to incorrect assumptions.

Additional request

Given that this is not a hardware limitation, I would like to kindly ask if you could help bring this topic to the relevant internal teams.

From a user perspective, the lack of support for:

10-bit encoding combined with QVBR/HQVBR

creates a significant limitation in real-world workflows.

Why this matters

Currently, users are forced to choose between:

• 10-bit encoding (modern formats, HDR workflows)
• or advanced quality-based rate control

This trade-off is not obvious from the API capabilities and impacts encoding decisions in practice.

Final note

I understand that you may not be able to comment on future plans, but even raising awareness internally would already be very valuable.

I would really like to see AMD continue improving in this area, as the hardware itself is clearly capable.

Thank you again for your time and for looking into this.

[MikhailAMD]

Given that this is not a hardware limitation, I would like to kindly ask if you could help bring this topic to the relevant internal teams.
Already did.

[DummyTitan]

Thank you, I really appreciate you bringing this to the relevant team.

That already helps a lot, especially knowing that it is not a hardware limitation.

Looking forward to any updates when available.