[XNNPACK] Add the off-graph flat KV-cache

[kiymetakdemir]

Summary

Adds the byte layer for the off-graph KV cache on XNNPACK: XnnSequenceCache stores K/V in per-layer host pools behind the neutral SequenceCache, which keeps the length bookkeeping and step planning. A step's tokens are copied to the slots the planner names, and the window comes back as an AttendSpec: base pointers plus the pool's extent (slots) and the live prefix (valid_len), pointer-typed so the consumer wraps it at the call site instead of the cache owning tensor state.

Files

• backends/xnnpack/runtime/XnnCache.h — AttendSpec (K/V base pointers, slots, valid_len, and the mask kind
  None/Causal/Explicit with an optional additive mask) plus the XnnCache op face, kept separate from the neutral
  tensor-free face so one cache exposes both without a diamond. Masking is the cache's to declare, not the caller's, which is what lets a windowed or tree layer slot in behind the same op; Explicit has no producer until a ring layer needs one.
• backends/xnnpack/runtime/XnnSequenceCache.h — Pool (per-layer host buffer, BHSD-major [1, H, slots, D], one copy per head at the slots the planner names, lazy doubling growth that re-lays-out every head because the row stride widens) and XnnSequenceCache : cache::SequenceCache, XnnCache. Construction goes through a static create() returning Result, since a constructor cannot reject a bad config without exceptions. A ring step can span two runs or start mid-pool, and update_and_fetch rejects both rather than half-serving them.
• backends/xnnpack/test/runtime/test_xnn_sequence_cache.cpp, backends/xnnpack/test/CMakeLists.txt — the tests.

Testing

20 GTest cases. The centrepiece drives the cache through a realistic step sequence — prefill, chunked prefill, decodes, crossing four pool doublings — while an independent naive model appends each step to a flat per-head history, and compares the pool read as [1, H, slots, D] over [0, valid_len) after every step.

cmake . -DEXECUTORCH_BUILD_XNNPACK=ON -DEXECUTORCH_BUILD_TESTS=ON -Bcmake-out
cmake --build cmake-out --target backends_xnnpack_test
ctest --test-dir cmake-out -R backends_xnnpack_test --output-on-failure

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21822

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 4 Unrelated Failures, 1 Unclassified Failure

As of commit 601c570 with merge base 564e655 ():

NEW FAILURES - The following jobs have failed:

• pull / test-vulkan-operators-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 988e6d2e80d4f900851e940c3721eb8317c73468f16c090030e8e4fcee49c427 /exec failed with exit code 2
• pull / unittest / linux / linux-job (gh)
  RuntimeError: Command docker exec -t 231774b7668bc73a3492651ebc01daf6e2192d9251658a3e071c3a2c0c6f5973 /exec failed with exit code 2
• pull / unittest / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 2
• pull / unittest-buck / linux / linux-job (gh)
  RuntimeError: Command docker exec -t 80d664e499ede326ef56435b02ea427c0410f819fe544b8e501ae6fc07c66fb7 /exec failed with exit code 3
• pull / unittest-editable / linux / linux-job (gh)
  RuntimeError: Command docker exec -t 4be06df3eeaefdf2aa66faf94afbb1fc752795645091886d9b0f8ff4264d16f3 /exec failed with exit code 2
• pull / unittest-editable / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 2

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

• pull / test-arduino-library / run / arduino-library (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 6ad4f13f364669a974a55d0be653571319e62cbabc3d7813769e841954e84a70 /exec failed with exit code 2

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, models) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• pull / unittest / windows / windows-job (gh) (detected as infra flaky with no log or failing log classifier)
• pull / unittest-editable / windows / windows-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest-buck / macos / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 3

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.