Add chess.chesstb: pure-Python prober for chesstb endgame tablebases by noobpwnftw · Pull Request #1194 · niklasf/python-chess

noobpwnftw · 2026-06-08T19:38:31Z

Add `chess.chesstb`: pure-Python prober for chesstb endgame tablebases

What this adds

A new module, chess.chesstb, that probes the chesstb endgame
tablebase format directly from chess.Board positions — in the same spirit as
the existing chess.syzygy and chess.gaviota modules.

chesstb ships three table types per material:

Table	Extension	Answer
WDL	`.lzw`	50-move-rule-aware win/draw/loss with cursed/blessed classes
DTC	`.lzdtc`	distance-to-conversion (plies to the next zeroing move)
DTM50	`.lzdtm50`	one pack giving both the unbounded DTM and the exact 50MR DTM at any halfmove clock

API

import chess, chess.chesstb

with chess.chesstb.open_tablebase("/path/to/chesstb") as tb:
    board = chess.Board("8/8/8/5k2/8/8/1Q6/K7 w - - 0 1")
    tb.probe_wdl(board)        # 2  (+2 win .. -2 loss, like syzygy)
    tb.probe_dtz(board)        # 19 (signed distance-to-conversion)
    tb.probe_dtm(board)        # 19 (signed distance-to-mate, ignoring 50MR)
    tb.probe_dtm50(board)      # (2, 19): rule-true (wdl, plies) at the board's clock

get_wdl / get_dtz / get_dtm are non-raising variants returning a default
(None) when no table is available. probe(board, rule50=0) returns the full
structured result (all fields at once).

Why pure Python

chess.syzygy is pure Python; this follows suit. The module depends only on
python-chess and the standard library:

LZMA (DTC / DTM50 blocks) via stdlib lzma with FORMAT_RAW (the C++
side uses the LZMA SDK with props appended at each block tail).
LZ4 (WDL blocks) via a small bundled pure-Python LZ4-block decoder
(~30 lines) supporting the optional LZ4 dictionary. No new dependency.

The position index (symmetry canonicalization, king/pawn slice managers, the
binomial piece-group ranking, the radix-composed board index and the
index-permutation layout) and the probe orchestration (dropped-frame one-ply
minimax reconstruction for shrunk files, the en-passant overlay, and the DTM50
halfmove-clock layer selection) are faithful re-implementations of the C++
src/probe library. Square numbering already matches python-chess exactly
(a1=0 … h8=63), so boards are consumed directly.

Validation

Every value is validated bit-for-bit against the reference C++ prober
(tests/probe_fen) by enumerating positions and comparing WDL, DTC/dtz, DTM
and DTM50 in lockstep:

All 145 shipped ≤5-man materials, ~72k positions at halfmove clock 0 —
0 mismatches.
Layered DTM50 at halfmove clocks 1, 30, 40, 80, 98, 99 across all
materials (exercising the CONST/SINGLE/DOUBLE/MULTI changepoint state machine,
the draw-end hint, and recover_mate_at_hmc) — 0 mismatches.
Dropped-frame reconstruction: both the symmetric-mirror path (e.g. KRKR)
and the asymmetric one-ply-minimax derive (109 of 145 materials ship a dropped
frame) — 0 mismatches.
En-passant overlay and color-mirrored material (stronger side is
Black) — verified against the oracle.
The pure-Python LZ4 decoder is cross-checked block-for-block against the
reference lz4 C library on every shipped WDL table.

A self-contained test_chesstb.py pins representative expected values (no C++
build required) and skips when no tables are present.

Scope / future work

probe_root_* (Fathom-style root move ranking) is not yet ported.
The standalone .lzdtm table is intentionally not read: DTM is served from
the .lzdtm50 pack (layer 0), which supersedes it.
Performance is pure-Python (block-cached); a future pass could add mmap and
a native-accelerated LZ4 path. Correctness, not speed, is the goal here.

chesstb (https://github.com/noobpwnftw/chesstb) ships three table types per material: 50-move-rule-aware WDL (with cursed/blessed classes), DTC (distance-to-conversion), and a DTM50 pack that answers both the unbounded DTM and the exact 50MR DTM at any halfmove clock. This adds chess/chesstb.py, a faithful pure-Python re-implementation of the reference C++ prober (no native extension; depends only on the standard library's lzma plus a small bundled LZ4-block decoder). It covers the position index (symmetry canonicalization, king/pawn slice managers, binomial piece-group ranking, index-permutation layout), all three decoders, and the probe orchestration (dropped-frame one-ply-minimax reconstruction for shrunk files, the en-passant overlay, and DTM50 halfmove-clock layer selection). API mirrors chess.syzygy: open_tablebase(); probe_wdl/probe_dtz/probe_dtm return signed values; probe_dtm50 returns (wdl, plies); get_* are non-raising. Validated bit-for-bit against the reference C++ prober across all 145 shipped <=5-man materials and every halfmove-clock layer. A small fixture set (data/chesstb) drives ChesstbTestCase. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The per-color _blocks dicts only ever grew: every decoded WDL/DTC/DTM50 block stayed resident until Tablebase.close(), so a long probing session accumulated all blocks of every opened table. Add _BlockCache, a thread-safe LRU keyed by (per_color, block_id) that tracks an approximate resident-byte total and evicts least-recently-used blocks (dropping them from their owning _blocks dict) once the budget is exceeded. Each table file routes _get_block* through it; Tablebase owns a single shared cache so the budget bounds the whole session rather than each file. New block_cache_bytes kwarg on Tablebase/open_tablebase tunes the budget (default 64 MiB); close() clears it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

PieceConfig canonicalizes side ordering by total strength (stronger side -> White), but only swapped when bs > ws, leaving the bs == ws case with no tiebreak. For equal-strength asymmetric material (e.g. KQP vs KRR, both 1000), the canonical orientation then depended on which color held which pieces on the probed board, so cfg.name() produced "KQPKRR" or "KRRKQP" depending on the input. Since tables are opened by cfg.name(), half the orientations of every such material computed a filename that doesn't exist on disk and probing wrongly reported table-not-found. Add a deterministic tiebreak mirroring Piece_Config::sort_pieces in the C++ generator: when totals are equal, the side with more high-value pieces (more queens, then rooks, ...) becomes White, so cfg.name() reproduces the generator's on-disk filename. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Mirror the generator/probe additions for the lowercase-'p' material: an opposing pawn pair (white below black on one file) indexed jointly so a large table shrinks to a partial 'p' variant. - PairGroup: the 120-placement same-file opposing enumeration, with index_of / find_canonical matching src/egtb/pair_group.h exactly (the on-disk pawn-slice ids depend on the enumeration order). - PawnSliceManager: 3-way pair x free_white x free_black product. - PositionIndexConfig: force file-mirror symmetry when a pair is present, collect the pair pawns even with no free-pawn class, and split the canonical opposing pair out of the all-pawns placement at index time. - PieceConfig: has_pair flag; min_key stays pair-stripped (on-disk header), cache_key adds the pair bit so a 'p'-material and its free-pieces twin (e.g. KQpKp vs KQK) don't collide in the in-memory caches; name() emits 'p' on both sides (KQpKp). - probe(): prefer the 'p' table for an opposing-pair board when it is on disk, else fall back to the full material. Cross-checked 500 random KQpKp/KpKp positions against the C++ probe_fen (WDL + DTC) with zero mismatches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

A cell's draw-end changepoint pins DTC: a W/L position still decisive at hmc = 100 - dtc turns DRAW one tick later, so its flip layer is h = 102 - dtc. Decoding that flip beside the value out of the same record lets the pack answer a third metric with one extra rank load and no second lookup. - DTM50File.read returns (value, draw_flip); dtc_from_draw_flip turns the flip into a distance. Both ends stay blind: h == 1 is a cell already drawn at a fresh clock, naming the cursed band without pinning a distance, and never flipping means dtc <= 1, which only a mate splits. - Conversion is clock-free, so only the layer-0 probe prices DTC; a layered probe returns the mate distance alone. dtc/ is read only for the cursed band. - A frame the file dropped has no record to read, but is already rebuilt by one-ply minimax, so _derive_dtm50_flat minimaxes DTC over those same children in its own accumulator (_DTCMinimax) -- conversion distance ranks moves its own way. _derive_dtm50 carries none. - The distance derives take the class the dropped-frame read would have decoded with, rather than minimaxing it back out of the children. A DRAW answers 0 without a walk, as does anything the clock folds to DRAW, and a child whose best offer ranks below the pinned class is never probed for a distance. Outranked for the mate distance implies outranked for conversion, since rank(invert(cw)) <= rank(dtc_lift(invert(cw))), so one screen feeds both accumulators exactly the children that can win them. - Every deriver reports nothing rather than a best taken over a partial move set. A skipped child's class bounds what it could have offered; an unknown class bounds nothing and unpins outright; for DTC the >100 transition lifts a LOSE bound to BLESSED_LOSS. - An ep capture zeroes, so it converts at dtc 1 whatever the child holds. The base value is consulted only to break a tie in class, which is how a cursed base the pack cannot pin still reports a dtc once ep lifts it. A DRAW selection stores 0, as everything else does. - _require() raises MissingTableError when a derive cannot resolve a cell, instead of letting probe_wdl() KeyError on an ILLEGAL class -- the stricter guards make that reachable on a partial table set. Cross-checked against data/chesstb by probing wdl+dtc against wdl+dtm50: 10639 classified cells over all seven materials agree, dtz 1..64, including 136 ep-bearing cells. The fixtures drop black, so black-to-move cells put the two independent minimaxes against each other. mypy --strict clean; full suite passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Table files were read whole into a bytes object, so every material a probe touched stayed fully resident for the life of the Tablebase. Mapping them instead, as chess.syzygy.Table.init_mmap does, bounds memory by the pages actually touched and makes opening a table a syscall rather than a multi-GB read -- which is also what makes it cheap enough to open under a lock, so the same change is what lets concurrent probing work. - _TableFile centralizes the mmap lifecycle for all three kinds: a read-only ACCESS_READ mapping, fd closed immediately, MADV_RANDOM where available (probing jumps between blocks, it does not stream), and the size check and header parse behind one _open() that unmaps again if either fails. - A live memoryview over a mapping makes mmap.close() raise BufferError, so _Serial hands out tracked views (r.view()) for the slices that outlive parsing -- the offset vectors and the compressed stream -- and close() releases them before unmapping. BufferError is still caught, so a stray slice held elsewhere leaves the unmap to refcounting rather than raising. - The 64-bit bit-window reads in MonoUintVec/Min0UintVec go through a precompiled struct.Struct("<Q"), mirroring syzygy's module-level Structs, instead of int.from_bytes over a fresh memoryview slice per call: 3.5x faster on the hottest read in a probe. Fewer than 8 bytes remain near the end of a mapping, which unpack_from rejects and the slicing form tolerates. Probing concurrently then needs three things guarded. Unlike syzygy, which registers every Table up front in add_directory and so never mutates its dicts while probing, this module resolves and opens on first use, and it keeps a decoded-block cache syzygy has no equivalent of. - One lock per kind guards that kind's open cache and the dirs entry its _find walks. Double-checked, so a warm probe never acquires it, and a table is published only once fully constructed. add_directory extends each list under the same lock, so no probe sees a partly extended search path. - Block decodes are guarded per (color, block), not per table, via a shared _PerColor base: two threads wanting the same cold block decode it once, while threads wanting different cold blocks never queue behind each other. That buys real parallelism for DTC/DTM50, where lzma releases the GIL (measured 2.07x on four threads); for WDL, whose LZ4 decoder here is pure Python and holds the GIL, it only avoids the duplicate work. _BlockCache eviction drops a block's lock along with the block, so that dict stays bounded by the budget instead of growing once per block ever decoded. - close() unmaps files and releases the views into them, so it drains in-flight probes first through a read-count/condition pair, as chess.syzygy.Table.close does. probe() is where the reader registers: it is the one entry point every other probe method funnels through, so a single registration covers the whole walk, including tables opened along the way. Without the drain, six probing threads fail immediately with "operation forbidden on released memoryview object". - The module-level KingSliceManager and PositionIndexConfig caches are double-checked under their own locks; both are expensive to build and were racing to build duplicates. Lock order is read condition -> open lock -> {index config -> king slices, block cache}, and block lock -> block cache; nothing acquires them in reverse. Verified against data/chesstb with a stress harness: 120 random positions over seven materials, answers compared against a single-threaded reference with 8 threads sharing a Tablebase, with a 1-byte block budget forcing eviction on every probe, with 12 threads racing the first open of each table, with close() called under live probes, and from cold module caches with 16 threads -- all match, no deadlock -- the tiny-budget stage takes minutes, since every probe re-decodes there, so it is progress and not elapsed time that distinguishes a hang. Three regression tests cover the mapping lifecycle, the shared-probe path and close-vs-probe; the last asserts the drain directly (blocked, then released), which catches a missing drain in 14 ms rather than racing for it. mypy --strict clean; full suite 301 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A LOSE pin fixes every child's class. The mover takes the best offer on hand, so a position the WDL companion calls a clean loss has no child offering better than LOSE -- every one of them is a clean WIN for the opponent, and a child that were only a cursed win would have made the pin BLESSED_LOSS. At the bottom rank below_pinned_class prunes nothing either, so the per-child WDL probe returns a constant and buys nothing. Drop it in the three derives that minimax over children, which spares one WDL read and one index computation per child on the one path that cannot prune. A child missing from disk still surfaces as a failed distance read -- except for a zeroing child in the DTC derive, whose conversion distance is 1 whatever the child holds, so that case now answers where it used to unpin the minimax. All four sites test the exact five-class pin. The flat DTM50 derive in particular cannot relax that: the DTC minimax running beside the mate one classifies the 50MR boundary itself, off a class a fold would erase. Mirrors the same change in the C++ prober. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A pair config counts only its FREE pieces, so KpKp sits at num_pieces == 2 while physically holding four. _make_child's bare-kings test was a bare `num_pieces <= 2`, so a child routed into a 'p' table was returned as an unconditional draw without its table ever being consulted -- in both the derivation and en-passant paths. Route every "is it just kings?" test through PieceConfig.is_bare_kings, mirroring Piece_Config::is_bare_kings on the C++ side, and drop the cached is_kk element from _make_child's return: with ten consumers all holding the config anyway, caching put the derivation far from every use, where nothing could contradict it. Also reject castling rights in probe() with MissingTableError, as chess.syzygy and chess.gaviota do for the same input. The tables are built without them, and probing anyway is not merely approximate but internally inconsistent: a direct read ignores the extra castling move while a dropped-frame derivation minimaxes over it. Finally, make legality an explicit caller precondition -- _is_legal and the "illegal_pos" status it was the only producer of are gone, along with the ValueError _require raised from it. This matches the C++ prober, where the generator and the verification tools screen positions themselves and the walk only ever builds children from legal move generation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…tride _TableFile._open_source is now the one place the transport is decided: set _data to what close() releases, return the buffer to read through. The default still maps the file. Nothing above it needs the buffer protocol, and no span wider than one block is requested any more -- a color's data section and the offset vectors are addressed as (buffer, base) instead of sliced out, so a source that fetches on slice no longer pulls the whole section at open, under that kind's lock. DTM50 blocks carry a prefix index every 256 positions and walk at most one stride at read time, as src/probe/dtm50_file.cpp does, rather than materializing a state and an index per position: 13.6 ms and 151 KiB for a 1,048,576-position block against 114-153 ms and some 37 MiB, of which the cache was told 9 MiB. That understatement made the block budget meaningless and turned a working set slightly over it into a re-decode per probe -- eight threads at a 4 MiB budget got 265 of 1600 probes done in 90s, now 12s. A warm read pays a walk instead of a list index for it: 6.0s -> 8.0s over resident blocks, against 2.52s -> 1.98s where decodes dominate. Verified against the old path on 3,930,528 positions and 1,512,206 hint prefixes, and on 4000 probes with every ProbeResult field unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`_open_source` was already the seam for reading a table through something other than a mapping, but `_open_wdl`/`_open_dtc`/`_open_dtm50` named the three concrete classes directly, so reaching it meant reimplementing all three -- double-checked locking and cache bookkeeping included. Name them as class attributes instead, so an override is three subclasses and a `_find`. `_find`'s result now only has to be something `_open_source` understands, which the `TableSource` alias says and the three constructors accept; the default source still wants a path. Resolution is `is not None` rather than truthiness, since a handle's `__bool__` is its own business. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

noobpwnftw and others added 3 commits June 9, 2026 03:29

noobpwnftw force-pushed the add-chesstb-tablebases branch 7 times, most recently from a252d4c to 228bac6 Compare July 1, 2026 01:08

noobpwnftw force-pushed the add-chesstb-tablebases branch 3 times, most recently from df7a960 to 8055aaf Compare August 8, 2026 19:14

noobpwnftw force-pushed the add-chesstb-tablebases branch 2 times, most recently from 9e36bf4 to d17b765 Compare August 10, 2026 01:53

noobpwnftw force-pushed the add-chesstb-tablebases branch from d17b765 to 5b161fe Compare August 10, 2026 02:09

noobpwnftw and others added 2 commits August 10, 2026 19:52

noobpwnftw force-pushed the add-chesstb-tablebases branch from cfde6e0 to 8abfafa Compare August 10, 2026 18:21

noobpwnftw force-pushed the add-chesstb-tablebases branch from 8abfafa to f6ee418 Compare August 10, 2026 18:51

noobpwnftw force-pushed the add-chesstb-tablebases branch from a7b51f6 to 0cf9e8b Compare August 11, 2026 00:25

Navigation Menu

Uh oh!

Add chess.chesstb: pure-Python prober for chesstb endgame tablebases - #1194

Add chess.chesstb: pure-Python prober for chesstb endgame tablebases#1194
noobpwnftw wants to merge 10 commits into
niklasf:masterfrom
noobpwnftw:add-chesstb-tablebases

noobpwnftw commented Jun 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

noobpwnftw commented Jun 8, 2026

Add chess.chesstb: pure-Python prober for chesstb endgame tablebases

What this adds

API

Why pure Python

Validation

Scope / future work

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Add `chess.chesstb`: pure-Python prober for chesstb endgame tablebases