Skip to content

Add chess.chesstb: pure-Python prober for chesstb endgame tablebases - #1194

Open
noobpwnftw wants to merge 10 commits into
niklasf:masterfrom
noobpwnftw:add-chesstb-tablebases
Open

Add chess.chesstb: pure-Python prober for chesstb endgame tablebases#1194
noobpwnftw wants to merge 10 commits into
niklasf:masterfrom
noobpwnftw:add-chesstb-tablebases

Conversation

@noobpwnftw

Copy link
Copy Markdown

Add chess.chesstb: pure-Python prober for chesstb endgame tablebases

What this adds

A new module, chess.chesstb, that probes the chesstb endgame
tablebase format directly from chess.Board positions — in the same spirit as
the existing chess.syzygy and chess.gaviota modules.

chesstb ships three table types per material:

Table Extension Answer
WDL .lzw 50-move-rule-aware win/draw/loss with cursed/blessed classes
DTC .lzdtc distance-to-conversion (plies to the next zeroing move)
DTM50 .lzdtm50 one pack giving both the unbounded DTM and the exact 50MR DTM at any halfmove clock

API

import chess, chess.chesstb

with chess.chesstb.open_tablebase("/path/to/chesstb") as tb:
    board = chess.Board("8/8/8/5k2/8/8/1Q6/K7 w - - 0 1")
    tb.probe_wdl(board)        # 2  (+2 win .. -2 loss, like syzygy)
    tb.probe_dtz(board)        # 19 (signed distance-to-conversion)
    tb.probe_dtm(board)        # 19 (signed distance-to-mate, ignoring 50MR)
    tb.probe_dtm50(board)      # (2, 19): rule-true (wdl, plies) at the board's clock

get_wdl / get_dtz / get_dtm are non-raising variants returning a default
(None) when no table is available. probe(board, rule50=0) returns the full
structured result (all fields at once).

Why pure Python

chess.syzygy is pure Python; this follows suit. The module depends only on
python-chess and the standard library:

  • LZMA (DTC / DTM50 blocks) via stdlib lzma with FORMAT_RAW (the C++
    side uses the LZMA SDK with props appended at each block tail).
  • LZ4 (WDL blocks) via a small bundled pure-Python LZ4-block decoder
    (~30 lines) supporting the optional LZ4 dictionary. No new dependency.

The position index (symmetry canonicalization, king/pawn slice managers, the
binomial piece-group ranking, the radix-composed board index and the
index-permutation layout) and the probe orchestration (dropped-frame one-ply
minimax reconstruction for shrunk files, the en-passant overlay, and the DTM50
halfmove-clock layer selection) are faithful re-implementations of the C++
src/probe library. Square numbering already matches python-chess exactly
(a1=0 … h8=63), so boards are consumed directly.

Validation

Every value is validated bit-for-bit against the reference C++ prober
(tests/probe_fen) by enumerating positions and comparing WDL, DTC/dtz, DTM
and DTM50 in lockstep:

  • All 145 shipped ≤5-man materials, ~72k positions at halfmove clock 0 —
    0 mismatches.
  • Layered DTM50 at halfmove clocks 1, 30, 40, 80, 98, 99 across all
    materials (exercising the CONST/SINGLE/DOUBLE/MULTI changepoint state machine,
    the draw-end hint, and recover_mate_at_hmc) — 0 mismatches.
  • Dropped-frame reconstruction: both the symmetric-mirror path (e.g. KRKR)
    and the asymmetric one-ply-minimax derive (109 of 145 materials ship a dropped
    frame) — 0 mismatches.
  • En-passant overlay and color-mirrored material (stronger side is
    Black) — verified against the oracle.
  • The pure-Python LZ4 decoder is cross-checked block-for-block against the
    reference lz4 C library on every shipped WDL table.

A self-contained test_chesstb.py pins representative expected values (no C++
build required) and skips when no tables are present.

Scope / future work

  • probe_root_* (Fathom-style root move ranking) is not yet ported.
  • The standalone .lzdtm table is intentionally not read: DTM is served from
    the .lzdtm50 pack (layer 0), which supersedes it.
  • Performance is pure-Python (block-cached); a future pass could add mmap and
    a native-accelerated LZ4 path. Correctness, not speed, is the goal here.

noobpwnftw and others added 3 commits June 9, 2026 03:29
chesstb (https://github.com/noobpwnftw/chesstb) ships three table types per
material: 50-move-rule-aware WDL (with cursed/blessed classes), DTC
(distance-to-conversion), and a DTM50 pack that answers both the unbounded DTM
and the exact 50MR DTM at any halfmove clock.

This adds chess/chesstb.py, a faithful pure-Python re-implementation of the
reference C++ prober (no native extension; depends only on the standard
library's lzma plus a small bundled LZ4-block decoder). It covers the position
index (symmetry canonicalization, king/pawn slice managers, binomial
piece-group ranking, index-permutation layout), all three decoders, and the
probe orchestration (dropped-frame one-ply-minimax reconstruction for shrunk
files, the en-passant overlay, and DTM50 halfmove-clock layer selection).

API mirrors chess.syzygy: open_tablebase(); probe_wdl/probe_dtz/probe_dtm
return signed values; probe_dtm50 returns (wdl, plies); get_* are non-raising.

Validated bit-for-bit against the reference C++ prober across all 145 shipped
<=5-man materials and every halfmove-clock layer. A small fixture set
(data/chesstb) drives ChesstbTestCase.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-color _blocks dicts only ever grew: every decoded WDL/DTC/DTM50
block stayed resident until Tablebase.close(), so a long probing session
accumulated all blocks of every opened table.

Add _BlockCache, a thread-safe LRU keyed by (per_color, block_id) that
tracks an approximate resident-byte total and evicts least-recently-used
blocks (dropping them from their owning _blocks dict) once the budget is
exceeded. Each table file routes _get_block* through it; Tablebase owns a
single shared cache so the budget bounds the whole session rather than
each file. New block_cache_bytes kwarg on Tablebase/open_tablebase tunes
the budget (default 64 MiB); close() clears it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PieceConfig canonicalizes side ordering by total strength (stronger
side -> White), but only swapped when bs > ws, leaving the bs == ws
case with no tiebreak. For equal-strength asymmetric material (e.g.
KQP vs KRR, both 1000), the canonical orientation then depended on
which color held which pieces on the probed board, so cfg.name()
produced "KQPKRR" or "KRRKQP" depending on the input.

Since tables are opened by cfg.name(), half the orientations of every
such material computed a filename that doesn't exist on disk and
probing wrongly reported table-not-found.

Add a deterministic tiebreak mirroring Piece_Config::sort_pieces in
the C++ generator: when totals are equal, the side with more
high-value pieces (more queens, then rooks, ...) becomes White, so
cfg.name() reproduces the generator's on-disk filename.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@noobpwnftw
noobpwnftw force-pushed the add-chesstb-tablebases branch 7 times, most recently from a252d4c to 228bac6 Compare July 1, 2026 01:08
@noobpwnftw
noobpwnftw force-pushed the add-chesstb-tablebases branch 3 times, most recently from df7a960 to 8055aaf Compare August 8, 2026 19:14
Mirror the generator/probe additions for the lowercase-'p' material: an
opposing pawn pair (white below black on one file) indexed jointly so a
large table shrinks to a partial 'p' variant.

- PairGroup: the 120-placement same-file opposing enumeration, with
  index_of / find_canonical matching src/egtb/pair_group.h exactly (the
  on-disk pawn-slice ids depend on the enumeration order).
- PawnSliceManager: 3-way pair x free_white x free_black product.
- PositionIndexConfig: force file-mirror symmetry when a pair is present,
  collect the pair pawns even with no free-pawn class, and split the
  canonical opposing pair out of the all-pawns placement at index time.
- PieceConfig: has_pair flag; min_key stays pair-stripped (on-disk header),
  cache_key adds the pair bit so a 'p'-material and its free-pieces twin
  (e.g. KQpKp vs KQK) don't collide in the in-memory caches; name() emits
  'p' on both sides (KQpKp).
- probe(): prefer the 'p' table for an opposing-pair board when it is on
  disk, else fall back to the full material.

Cross-checked 500 random KQpKp/KpKp positions against the C++ probe_fen
(WDL + DTC) with zero mismatches.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@noobpwnftw
noobpwnftw force-pushed the add-chesstb-tablebases branch 2 times, most recently from 9e36bf4 to d17b765 Compare August 10, 2026 01:53
A cell's draw-end changepoint pins DTC: a W/L position still decisive at
hmc = 100 - dtc turns DRAW one tick later, so its flip layer is h = 102 - dtc.
Decoding that flip beside the value out of the same record lets the pack answer
a third metric with one extra rank load and no second lookup.

- DTM50File.read returns (value, draw_flip); dtc_from_draw_flip turns the flip
  into a distance. Both ends stay blind: h == 1 is a cell already drawn at a
  fresh clock, naming the cursed band without pinning a distance, and never
  flipping means dtc <= 1, which only a mate splits.
- Conversion is clock-free, so only the layer-0 probe prices DTC; a layered
  probe returns the mate distance alone. dtc/ is read only for the cursed band.
- A frame the file dropped has no record to read, but is already rebuilt by
  one-ply minimax, so _derive_dtm50_flat minimaxes DTC over those same children
  in its own accumulator (_DTCMinimax) -- conversion distance ranks moves its
  own way. _derive_dtm50 carries none.
- The distance derives take the class the dropped-frame read would have decoded
  with, rather than minimaxing it back out of the children. A DRAW answers 0
  without a walk, as does anything the clock folds to DRAW, and a child whose
  best offer ranks below the pinned class is never probed for a distance.
  Outranked for the mate distance implies outranked for conversion, since
  rank(invert(cw)) <= rank(dtc_lift(invert(cw))), so one screen feeds both
  accumulators exactly the children that can win them.
- Every deriver reports nothing rather than a best taken over a partial move
  set. A skipped child's class bounds what it could have offered; an unknown
  class bounds nothing and unpins outright; for DTC the >100 transition lifts a
  LOSE bound to BLESSED_LOSS.
- An ep capture zeroes, so it converts at dtc 1 whatever the child holds. The
  base value is consulted only to break a tie in class, which is how a cursed
  base the pack cannot pin still reports a dtc once ep lifts it. A DRAW
  selection stores 0, as everything else does.
- _require() raises MissingTableError when a derive cannot resolve a cell,
  instead of letting probe_wdl() KeyError on an ILLEGAL class -- the stricter
  guards make that reachable on a partial table set.

Cross-checked against data/chesstb by probing wdl+dtc against wdl+dtm50: 10639
classified cells over all seven materials agree, dtz 1..64, including 136
ep-bearing cells. The fixtures drop black, so black-to-move cells put the two
independent minimaxes against each other. mypy --strict clean; full suite
passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noobpwnftw
noobpwnftw force-pushed the add-chesstb-tablebases branch from d17b765 to 5b161fe Compare August 10, 2026 02:09
noobpwnftw and others added 2 commits August 10, 2026 19:52
Table files were read whole into a bytes object, so every material a probe
touched stayed fully resident for the life of the Tablebase. Mapping them
instead, as chess.syzygy.Table.init_mmap does, bounds memory by the pages
actually touched and makes opening a table a syscall rather than a multi-GB
read -- which is also what makes it cheap enough to open under a lock, so the
same change is what lets concurrent probing work.

- _TableFile centralizes the mmap lifecycle for all three kinds: a read-only
  ACCESS_READ mapping, fd closed immediately, MADV_RANDOM where available
  (probing jumps between blocks, it does not stream), and the size check and
  header parse behind one _open() that unmaps again if either fails.
- A live memoryview over a mapping makes mmap.close() raise BufferError, so
  _Serial hands out tracked views (r.view()) for the slices that outlive
  parsing -- the offset vectors and the compressed stream -- and close()
  releases them before unmapping. BufferError is still caught, so a stray
  slice held elsewhere leaves the unmap to refcounting rather than raising.
- The 64-bit bit-window reads in MonoUintVec/Min0UintVec go through a
  precompiled struct.Struct("<Q"), mirroring syzygy's module-level Structs,
  instead of int.from_bytes over a fresh memoryview slice per call: 3.5x
  faster on the hottest read in a probe. Fewer than 8 bytes remain near the end
  of a mapping, which unpack_from rejects and the slicing form tolerates.

Probing concurrently then needs three things guarded. Unlike syzygy, which
registers every Table up front in add_directory and so never mutates its dicts
while probing, this module resolves and opens on first use, and it keeps a
decoded-block cache syzygy has no equivalent of.

- One lock per kind guards that kind's open cache and the dirs entry its _find
  walks. Double-checked, so a warm probe never acquires it, and a table is
  published only once fully constructed. add_directory extends each list under
  the same lock, so no probe sees a partly extended search path.
- Block decodes are guarded per (color, block), not per table, via a shared
  _PerColor base: two threads wanting the same cold block decode it once, while
  threads wanting different cold blocks never queue behind each other. That
  buys real parallelism for DTC/DTM50, where lzma releases the GIL (measured
  2.07x on four threads); for WDL, whose LZ4 decoder here is pure Python and
  holds the GIL, it only avoids the duplicate work. _BlockCache eviction drops
  a block's lock along with the block, so that dict stays bounded by the
  budget instead of growing once per block ever decoded.
- close() unmaps files and releases the views into them, so it drains in-flight
  probes first through a read-count/condition pair, as
  chess.syzygy.Table.close does. probe() is where the reader registers: it is
  the one entry point every other probe method funnels through, so a single
  registration covers the whole walk, including tables opened along the way.
  Without the drain, six probing threads fail immediately with "operation
  forbidden on released memoryview object".
- The module-level KingSliceManager and PositionIndexConfig caches are
  double-checked under their own locks; both are expensive to build and were
  racing to build duplicates. Lock order is read condition -> open lock ->
  {index config -> king slices, block cache}, and block lock -> block cache;
  nothing acquires them in reverse.

Verified against data/chesstb with a stress harness: 120 random positions over
seven materials, answers compared against a single-threaded reference with 8
threads sharing a Tablebase, with a 1-byte block budget forcing eviction on
every probe, with 12 threads racing the first open of each table, with close()
called under live probes, and from cold module caches with 16 threads -- all
match, no deadlock -- the tiny-budget stage takes minutes, since every probe
re-decodes there, so it is progress and not elapsed time that distinguishes a
hang. Three regression tests cover the mapping lifecycle, the shared-probe path
and close-vs-probe; the last asserts the drain directly (blocked, then
released), which catches a missing drain in 14 ms rather than racing for it.
mypy --strict clean; full suite 301 tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A LOSE pin fixes every child's class. The mover takes the best offer on
hand, so a position the WDL companion calls a clean loss has no child
offering better than LOSE -- every one of them is a clean WIN for the
opponent, and a child that were only a cursed win would have made the pin
BLESSED_LOSS. At the bottom rank below_pinned_class prunes nothing either,
so the per-child WDL probe returns a constant and buys nothing.

Drop it in the three derives that minimax over children, which spares one
WDL read and one index computation per child on the one path that cannot
prune. A child missing from disk still surfaces as a failed distance read
-- except for a zeroing child in the DTC derive, whose conversion distance
is 1 whatever the child holds, so that case now answers where it used to
unpin the minimax.

All four sites test the exact five-class pin. The flat DTM50 derive in
particular cannot relax that: the DTC minimax running beside the mate one
classifies the 50MR boundary itself, off a class a fold would erase.

Mirrors the same change in the C++ prober.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noobpwnftw
noobpwnftw force-pushed the add-chesstb-tablebases branch from cfde6e0 to 8abfafa Compare August 10, 2026 18:21
A pair config counts only its FREE pieces, so KpKp sits at num_pieces == 2
while physically holding four. _make_child's bare-kings test was a bare
`num_pieces <= 2`, so a child routed into a 'p' table was returned as an
unconditional draw without its table ever being consulted -- in both the
derivation and en-passant paths.

Route every "is it just kings?" test through PieceConfig.is_bare_kings,
mirroring Piece_Config::is_bare_kings on the C++ side, and drop the cached
is_kk element from _make_child's return: with ten consumers all holding the
config anyway, caching put the derivation far from every use, where nothing
could contradict it.

Also reject castling rights in probe() with MissingTableError, as
chess.syzygy and chess.gaviota do for the same input. The tables are built
without them, and probing anyway is not merely approximate but internally
inconsistent: a direct read ignores the extra castling move while a
dropped-frame derivation minimaxes over it.

Finally, make legality an explicit caller precondition -- _is_legal and the
"illegal_pos" status it was the only producer of are gone, along with the
ValueError _require raised from it. This matches the C++ prober, where the
generator and the verification tools screen positions themselves and the
walk only ever builds children from legal move generation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noobpwnftw
noobpwnftw force-pushed the add-chesstb-tablebases branch from 8abfafa to f6ee418 Compare August 10, 2026 18:51
…tride

_TableFile._open_source is now the one place the transport is decided: set
_data to what close() releases, return the buffer to read through. The
default still maps the file. Nothing above it needs the buffer protocol,
and no span wider than one block is requested any more -- a color's data
section and the offset vectors are addressed as (buffer, base) instead of
sliced out, so a source that fetches on slice no longer pulls the whole
section at open, under that kind's lock.

DTM50 blocks carry a prefix index every 256 positions and walk at most one
stride at read time, as src/probe/dtm50_file.cpp does, rather than
materializing a state and an index per position: 13.6 ms and 151 KiB for a
1,048,576-position block against 114-153 ms and some 37 MiB, of which the
cache was told 9 MiB. That understatement made the block budget meaningless
and turned a working set slightly over it into a re-decode per probe --
eight threads at a 4 MiB budget got 265 of 1600 probes done in 90s, now
12s. A warm read pays a walk instead of a list index for it: 6.0s -> 8.0s
over resident blocks, against 2.52s -> 1.98s where decodes dominate.

Verified against the old path on 3,930,528 positions and 1,512,206 hint
prefixes, and on 4000 probes with every ProbeResult field unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noobpwnftw
noobpwnftw force-pushed the add-chesstb-tablebases branch from a7b51f6 to 0cf9e8b Compare August 11, 2026 00:25
`_open_source` was already the seam for reading a table through something
other than a mapping, but `_open_wdl`/`_open_dtc`/`_open_dtm50` named the
three concrete classes directly, so reaching it meant reimplementing all
three -- double-checked locking and cache bookkeeping included. Name them
as class attributes instead, so an override is three subclasses and a
`_find`.

`_find`'s result now only has to be something `_open_source` understands,
which the `TableSource` alias says and the three constructors accept; the
default source still wants a path. Resolution is `is not None` rather than
truthiness, since a handle's `__bool__` is its own business.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant