DSC RC parameter tables deviate from VESA DSC 1.1, causing block artifacts on compliant sinks

[triple-groove]

Summary

The DSC Rate Control (RC) parameter lookup tables minqp444_8b and maxqp444_8b in nvt_dsc_pps.c contain non-standard values for the 8bpc / 8.0 bpp operating point (column index 4). Specifically, ranges 9-14 have more constrained max_qp values than the VESA DSC 1.1 specification mandates. Additionally, the ofs_und8 bpg_offset table has an incorrect value at range index 11.

These deviations prevent the DSC encoder from quantizing aggressively enough when the RC buffer approaches overflow on high-detail content, resulting in visible block/tile artifacts (commonly described as "rainbow static") on DSC sinks that implement strict VESA-compliant decompression.

Root Cause

The VESA DSC 1.1 specification (Table E-5, "RC Parameters for 8 bpc, 4:4:4") defines specific (min_qp, max_qp, bpg_offset) triples for each of the 15 RC buffer ranges. NVIDIA's tables differ from the standard in ranges 9-14:

Range	NV min_qp	VESA min_qp	NV max_qp	VESA max_qp	NV bpg_ofs	VESA bpg_ofs
0-8	(match)	(match)	(match)	(match)	(match)	(match)
9	4	3	10	10	-10	-10
10	5	5	10	11	-10	-10
11	5	5	11	12	-10	-12
12	5	5	11	13	-12	-12
13	8	7	12	13	-12	-12
14	12	13	13	15	-12	-12

The critical difference is in the max_qp values for ranges 10-14. The VESA standard allows max_qp up to 15 (range 14), while NVIDIA's tables cap it at 13. This means the encoder cannot quantize as aggressively when the RC buffer is near overflow, which can cause the RC model to exceed its buffer capacity on complex content. When the RC buffer overflows, the compressed output for that DSC block becomes corrupted, producing visible artifacts at block/tile boundaries.

Range 14 (max_qp 13 vs 15) is the most critical — this is the highest buffer fullness range where the encoder needs maximum quantization headroom to avoid overflow.

Impact

This affects all DP and HDMI DSC output at 8bpc / 8.0 bpp on all NVIDIA GPUs using the open kernel modules. The artifacts manifest as:

• Block-aligned corruption ("rainbow static") on high-detail content
• Artifacts appear at DSC block/slice boundaries
• More visible on sinks with strict VESA-compliant DSC decoders

Confirmed affected hardware:

• GPU: NVIDIA RTX 5090 (Blackwell)
• Sink: Bigscreen Beyond VR headset (DSC 1.1, 8bpc, 8.0 bpp, RGB 4:4:4, 3840x1920@90Hz)
• Works correctly on Windows (which presumably uses VESA-standard RC parameters)

Affected Code

• src/common/modeset/timing/nvt_dsc_pps.c
  • minqp444_8b[15][37] — column index 4 (rows 9, 13, 14)
  • maxqp444_8b[15][37] — column index 4 (rows 10-14)
  • ofs_und8[] — local array in RC parameter selection block, index 11

The column index is computed as:

idx = (2 * (bitsPerPixel - 6 * BPP_UNIT)) / BPP_UNIT;
// For 8.0 bpp: idx = (2 * (128 - 96)) / 16 = 4

Fix

minqp444_8b — rows 9, 13, 14, column 4

-       ,{ 6, 5, 5, 4, 4, 4, 3, 3, ...    // row 9
+       ,{ 6, 5, 5, 4, 3, 4, 3, 3, ...    // row 9: col4 4->3

-       ,{ 9, 9, 9, 9, 8, 8, 7, 7, ...    // row 13
+       ,{ 9, 9, 9, 9, 7, 8, 7, 7, ...    // row 13: col4 8->7

-       ,{14,14,13,13,12,12,12,12, ...    // row 14
+       ,{14,14,13,13,13,12,12,12, ...    // row 14: col4 12->13

maxqp444_8b — rows 10-14, column 4

-       ,{12,12,11,11,10,10,10,10, ...    // row 10
+       ,{12,12,11,11,11,10,10,10, ...    // row 10: col4 10->11

-       ,{12,12,12,11,11,11,10,10, ...    // row 11
+       ,{12,12,12,11,12,11,10,10, ...    // row 11: col4 11->12

-       ,{12,12,12,12,11,11,11,11, ...    // row 12
+       ,{12,12,12,12,13,11,11,11, ...    // row 12: col4 11->13

-       ,{13,13,13,13,12,12,11,11, ...    // row 13
+       ,{13,13,13,13,13,12,11,11, ...    // row 13: col4 12->13

-       ,{15,15,14,14,13,13,13,13, ...    // row 14
+       ,{15,15,14,14,15,13,13,13, ...    // row 14: col4 13->15

ofs_und8 bpg_offset — index 11

- const NvU32 ofs_und8[] = { 2, 0, 0, -2, -4, -6, -8, -8, -8, -10, -10, -10, -12, -12, -12 };
+ const NvU32 ofs_und8[] = { 2, 0, 0, -2, -4, -6, -8, -8, -8, -10, -10, -12, -12, -12, -12 };

All corrected values match the VESA DSC 1.1 specification Table E-5.

Verification

Tested on RTX 5090 (Blackwell) with Bigscreen Beyond VR headset (DSC 1.1, 8bpc, 8.0 bpp, RGB 4:4:4):

• Before fix: Persistent block-aligned "rainbow static" artifacts on high-detail VR content
• After fix: Artifacts completely eliminated; display output matches Windows quality

A patch is available at: https://github.com/triple-groove/nvidia-bsb-dsc-fix

Additional Notes

Other BPP columns may also be affected

This report focuses on column index 4 (8.0 bpp) because that is the operating point used by the Bigscreen Beyond. Other columns in the same tables may also deviate from the VESA standard — a full audit of all BPP columns against the specification is recommended.

Potential impact on other tables

The minqp444_10b, maxqp444_10b, minqp444_12b, maxqp444_12b, and the 4:2:2 variants should also be audited against their respective VESA specification tables.

Environment

• GPU: NVIDIA RTX 5090 (Blackwell)
• Driver: open-gpu-kernel-modules 580.119.02
• OS: Fedora 43, Linux 6.18.9
• Sink: Bigscreen Beyond VR headset (DSC 1.1, 8bpc, 8.0bpp, RGB 4:4:4, 3840x1920@90Hz)