decompress: keep the repeat-offset triple in registers on x86-64

[matevz-kovacic]

Summary

ZSTD_decodeSequence() keeps the three recent offsets in
seqState->prevOffset[3] and updates them through variable
subscripts — prevOffset[ll0], prevOffset[!ll0],
prevOffset[offset]. A variable index is what stops the compiler
promoting the array to registers, so the triple stays in memory; and
because it is carried from one sequence to the next, the
sequence-decoding loop ends up with a store→load dependency chain
running through memory across iterations, in the hottest loop of the
decoder.

The scalar form of this state is already in the tree, inside
#if defined(__aarch64__). That arm holds two independent things: the
hoisting of the triple into locals, which carries no comment, and a
separate ZSTD_seqSymbol ZSTD_memcpy hack whose comment describes an
aarch64 ldr/ldrb/ldrh codegen problem. Nothing in the tree states why
the first of those is aarch64-only. This patch widens the fence for the
hoisting and leaves the ZSTD_seqSymbol hack fenced to aarch64 where it
is. No new logic is introduced.

Decode throughput rises +2.7% to +3.4% (gcc) and +5.5% to +9.3%
(clang) across levels 1/3/9/19 on Zen 5, with the same sign and
compiler ordering on Raptor Lake. The loop issues 21–25% fewer stores
under gcc and 38–42% fewer under clang, with L1d loads down 9–14%.
Compressed output is byte-identical at every level under both compilers,
and aarch64 code paths are untouched.

Is the decoded output affected?

No. RFC 8878 §3.1.1.5 defines the repeat-offset update as a total
function on the triple — seven cases, each a constant rearrangement plus
at most one new value:

offBase	ll	offset	new (r0,r1,r2)
> 3	any	offBase − 3	(offset, r0, r1)
1	≠ 0	r0	unchanged
1	= 0	r1	(r1, r0, r2)
2	≠ 0	r1	(r1, r0, r2)
2	= 0	r2	(r2, r0, r1)
3	≠ 0	r2	(r2, r0, r1)
3	= 0	r0 − 1	(r0 − 1, r0, r1)

Case analysis over those seven rows shows the existing aarch64 arm and
the existing x86-64 arm compute the same function. Every remaining
difference between them sits under MEM_32bits(), which is a
compile-time 0 on any 64-bit target, so those branches are already
dead there.

Every index used is statically confined to {0, 1, 2}:

• ll0 is a C comparison result, so ll0 ∈ {0,1} and !ll0 ∈ {0,1}.
• prevOffset[offset] is reached only on the ofBits == 1 path. Since
  OF_bits[] is the identity map, ofBits == 1 means offset code 1,
  and OF_base[1] == 1. So
  offset = ofBase + ll0 + readBits(1) = 1 + ll0 + bit with both
  addends in {0,1}, giving offset ∈ {1,2,3} — it can never be 0.
  offset == 3 is then peeled off into prevOffset[0] − 1 by the
  preceding conditional, leaving offset ∈ {1,2} at the subscript.
• The shift-down updates use the literal indices 0, 1 and 2.

Nothing about the array is load-bearing.

Why !defined(__ILP32__)

The scalar arm omits every MEM_32bits() path, so it is equivalent only
on LP64/LLP64. x32 is excluded explicitly rather than by assumption.

Generated code

Hardware counters for this patch (the counter run was made against
the same single-patch build measured below), per decode pass over
Silesia, AMD Zen 5. Counted as
perf stat(--reps N) − perf stat(--reps 0) so that corpus loading,
warm-up and image setup cancel exactly and only the timed passes remain.

Change relative to baseline, negative = fewer:

	gcc L1	gcc L3	gcc L9	gcc L19	clang L1	clang L3	clang L9	clang L19
store dispatches	−20.6%	−24.1%	−24.5%	−24.6%	−37.6%	−41.6%	−42.2%	−42.2%
store→load forwards	−19.7%	−20.6%	−21.3%	−20.0%	+3.6%	+3.6%	+2.9%	+4.2%
L1d loads	−8.9%	−11.4%	−11.8%	−11.3%	−10.5%	−13.4%	−13.2%	−13.7%
load dispatches	−7.0%	−8.5%	−8.9%	−7.6%	+8.9%	+8.9%	+9.5%	+10.9%
instructions	−6.0%	−7.1%	−6.9%	−7.5%	−4.8%	−6.1%	−5.6%	−6.2%
cycles	−2.8%	−3.1%	−4.2%	−3.2%	−4.7%	−8.9%	−8.2%	−5.6%

Absolute, gcc level 9, per pass:

store dispatches      2.5428e8 -> 1.9198e8    -62.3 million
store->load forwards  2.2236e8 -> 1.7509e8    -47.3 million
L1d loads             8.6149e8 -> 7.5978e8   -101.7 million
instructions          2.1640e9 -> 2.0145e9   -149.5 million

Both compilers lose stores; they differ in what else moves. gcc drops
store→load forwards by ~21% and loads by ~8% as well. clang drops stores
harder (−42%) but issues ~9% more load dispatches and marginally more
forwards, and still ends up with the larger cycle win. That asymmetry is
consistent with the two compilers having materialised the array
differently in the first place.

L1d load misses are unchanged everywhere (−1.9% to +0.7%), i.e. this
removes accesses rather than improving locality — which is what a
register-promotion change should look like.

Effect on compressed output: none

The patch touches only lib/decompress. Total compressed size over
Silesia is byte-identical to baseline at every level under both
compilers, on both test machines:

L1 73,229,468    L3 66,137,723    L9 59,081,628    L19 52,891,946

The two compilers also agree with each other exactly, as they must.

Effect on aarch64 and 32-bit targets: none

The aarch64 arm is entered on exactly the same condition as before, and
the ZSTD_seqSymbol ZSTD_memcpy hack inside it remains fenced to
aarch64. x32 is excluded by !defined(__ILP32__) and keeps the previous
arm.

Effect on compression speed: small, sign-inconsistent movement

Not none, so stated plainly. Encode throughput moves by under ±1% in
both directions. Three of the eight cells are statistically
distinguishable from baseline, and they disagree on sign:

	L1	L3	L9	L19
gcc	−0.12%	−0.01%	+0.06%	+0.66%
clang	−0.52%	+0.90%	−0.14%	+0.16%

(bold = interquartile ranges disjoint from baseline; positive is faster)

The encoder source is byte-identical, so none of this can be semantic.
Movement of this size that does not agree on sign across cells is the
signature of code layout. This file already acknowledges layout
sensitivity of comparable magnitude: the .p2align block ~140 lines
below exists because decompression-loop alignment moved throughput by
about 10% via DSB/MITE residency. Its comment reports the effect as
reproduced on Kabylake and Coffeelake and not reproducible on
Haswell, Broadwell or Skylake — i.e. layout sensitivity here is already
known to be strongly machine-dependent.

These layout effects are stable per binary rather than run-to-run noise.
On the two-patch variant described in the notes, clang encode at levels
1 and 3 reproduced −0.4% to −0.6% across three independent trials of 16
reps each. So the effect is real; where it lands depends on the binary
rather than on the change.

A gate that forbids any statistically detectable regression anywhere
rejects on the single −0.52% cell. Whether that is the right call for a
decoder change worth +3% to +9% is a judgement for the maintainers, and
the numbers are here rather than omitted.

Correctness

At levels 1, 3, 9, 19 under both gcc and clang, over all 12 Silesia
files, on both machines:

• round-trip, compared by SHA-256 of the decoded stream rather than by
  size;
• forward compatibility — patched output decodes with an unmodified
  upstream binary;
• backward compatibility — patched binary decodes unmodified upstream
  output;
• tests/fuzzer -i2000 -s1 and tests/zstreamtest -i1500 -s1.

All pass.

Benchmarks

Decode, Silesia (12 files, 211,938,580 B), single-threaded, -O3, no
-march. Median of 22 reps, pinned to one core, baseline and patched
runs alternated and order-counterbalanced, first rep discarded.

level	Zen 5 gcc	Zen 5 clang	Raptor Lake gcc	Raptor Lake clang
1	+2.74%	+5.48%	+1.41%	+5.60%
3	+3.42%	+9.31%	+1.95%	+9.83%
9	+3.19%	+8.58%	+2.28%	+8.55%
19	+3.18%	+6.58%	+1.46%	+6.04%

All eight Zen 5 cells have disjoint interquartile ranges.

Noise floor. Two independently built copies of the unpatched tree
raced against each other through the identical measurement path give a
worst-cell apparent difference of 0.44% on the Zen 5 box, so the
figures above are 6–21× the floor. (.text sections of the two builds
hash identically, so that 0.44% is pure timing noise.) The Raptor Lake
box measured 3.40% under the same procedure, which is why its numbers
are quoted only for sign and ordering.

Cycle cross-check. Cycles are frequency-invariant, so they cannot be
moved by clock drift. They agree with wall-clock throughout — clang L3
−8.9% cycles against +9.31% throughput, gcc L19 −3.2% against +3.18% —
which rules out a frequency artefact.

Notes

• A related change was evaluated and is deliberately not included.
  Widening the PREFETCH_L1(match) in ZSTD_execSequence() from
  aarch64 to x86-64 looks like the natural companion to this patch. It
  is not: on Zen 5 it makes things worse. Decode with this patch alone
  is +5.48/+9.31/+8.58/+6.58% under clang; adding the prefetch drops it
  to +4.01/+7.48/+7.89/+5.15%. Measured standalone against baseline on
  Raptor Lake it is negative (−1.6% to −0.3% under gcc). It helps only
  in combination, only on Raptor Lake, and hurts on Zen 5.

• A second related change was evaluated and rejected. Widening the
  usePrefetchDecoder heuristic so ZSTD_decompressSequencesLong is
  selected more often is tempting, because this patch changes the
  tradeoff that heuristic is implicitly calibrated against. Forcing it
  on costs 11–13% under clang. The existing heuristic is right to
  decline it.

• Environment.

  	AMD	Intel
  CPU	Ryzen 7 9700X (Zen 5), bare metal	Core i5-13400 (Raptor Lake)
  OS	Ubuntu 24.04, kernel 6.17.0-23	Windows 11
  gcc	16.1.0 (built from source)	16.1.0 (MinGW-W64 UCRT)
  clang	22.1.8, LLVM ca7933e47d3a	22.1.8, LLVM ca7933e47d3a
  isolation	governor performance, turbo off, SMT sibling offline, pinned core	pinned P-core, raised priority
  noise floor	0.44%	3.40%

  Identical compiler versions on both, so the two machines differ only
  in CPU vendor and OS.

• Not tested: Zen 3/4, server-class Xeon or EPYC, or any non-x86-64
  target beyond confirming aarch64 is untouched. The mechanism —
  registers instead of memory for a loop-carried tuple — is
  architecture-general; the magnitude will not be.