From 0e1be59a105feafcd871afb1ca6cdeb3bc19059f Mon Sep 17 00:00:00 2001 From: Peter Tos <38345014+bmxcode@users.noreply.github.com> Date: Sat, 22 Aug 2026 15:27:31 +1000 Subject: [PATCH 1/2] D20: find the partitions eight short AKAI images displaced Eight of the 44 AKAI images are short of the disc they were made from: whole 32 KB blocks are missing from the .mdx file, so every partition after a gap sits that much nearer the front than the disk's own table places it. 39 such partitions are now found and read, carrying 432 volumes, 17 180 files and 15 808 samples that nothing could reach. The search is anchored, quantised and floored, which is what separates it from the scan ADR-0023 refused: it walks backwards from the position the table gives that partition, steps in the unit the container states it stores the disc in, and stops at the end of the partition already accepted. Partition 1 therefore cannot move on any disc. Closes #25. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 29 +++ README.md | 24 +- docs/README.md | 9 +- ...s-come-from-the-table-the-disc-declares.md | 2 +- ...ition-is-anchored-quantised-and-floored.md | 109 +++++++++ docs/adr/README.md | 1 + docs/formats/akai-fs.md | 95 ++++---- src/samplerdisc/batch.py | 9 + src/samplerdisc/cli.py | 12 +- src/samplerdisc/container/base.py | 18 ++ src/samplerdisc/container/mdx.py | 17 ++ src/samplerdisc/fs/akai.py | 157 ++++++++++--- src/samplerdisc/fs/base.py | 11 + src/samplerdisc/sample/akai.py | 26 ++- tests/fixtures.py | 17 ++ tests/test_akai_fs.py | 219 +++++++++++++++++- tests/test_containers.py | 27 +++ tests/test_discs.py | 204 +++++++++++++++- 18 files changed, 869 insertions(+), 117 deletions(-) create mode 100644 docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 869eb32..84db4c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,35 @@ Notable changes to `samplerdisc`. Format-level findings live in [docs/formats/]( ## Unreleased +### Added + +- **Eight AKAI images are short of the disc they were made from, and their displaced partitions are now read.** Whole 32 KB blocks are missing from these `.mdx` files — the container decodes every block it holds, and the file is not a complete copy of the disc — so every partition after a gap sits that much nearer the front than the disk's own partition table places it. **39 partitions are found there**, carrying **432 volumes, 17 180 files and 15 808 samples** that nothing could reach before. The collection goes from 89 156 samples to **104 921**, and AKAI from 56 425 to **72 190**. ([docs/formats/akai-fs.md](docs/formats/akai-fs.md), [ADR-0028](docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md), [#25](https://github.com/bmxcode/samplerdisc/issues/25)) + + | Disc | Recovered | Volumes | Files | Samples | Written | + |---|---:|---:|---:|---:|---:| + | `AMG - Kickin' Lunatic Beats 2 CD1` | 7 | 121 | 7 723 | 7 308 | 7 293 | + | `AMG - Kickin' Lunatic Beats 2 CD2` | 7 | 100 | 6 203 | 5 912 | 5 912 | + | `AKAI.S3000.Sound.Library.6` | 7 | 84 | 1 054 | 955 | 955 | + | `Back In Time Records - Elektra Vox` | 5 | 33 | 661 | 463 | 463 | + | `Audio Factory - Classical Wild Takes` | 4 | 18 | 189 | 80 | 80 | + | `AKAI.S3000.Sound.Library.5` | 3 | 34 | 473 | 400 | 377 | + | `AKAI.S3000.Sound.Library.7` | 3 | 20 | 660 | 546 | 546 | + | `AMG - Global Trance Mission 2` | 3 | 22 | 217 | 144 | 139 | + + **The audio verifies, and that is what made this safe to do at all.** 15 765 of the 15 808 recovered samples — 99.7 % — carry a payload header whose id, valid flag and name agree with the directory entry that placed them, checked by a structure the search never consults. A gap removes whole blocks, so everything past one moves by a *constant*: inside a displaced partition the directory and its audio moved together and stay consistent. Five of the eight discs recover thousands of files with **not one refusal between them**. + + **The search is anchored, quantised and floored, which is what separates it from the scan [ADR-0023](docs/adr/0023-partitions-come-from-the-table-the-disc-declares.md) refused.** It walks *backwards from the position the table gives that partition* — which also settles which partition it is, since sizes repeat across a disk; it steps in *the unit the container states it stores the disc in*, new as `SectorImage.granularity`, 32 768 bytes on these images; and it *stops at the end of the partition already accepted*, so nothing is ever found inside a partition already being read. Partition 1 therefore cannot move on any disc — its declared position is 0 — and every count pinned since D15 is unchanged on the other 36 discs. + + **70 declared partitions stay unread and 60 of them are not absent**, which is the deliberate cost. A header is there, inside a partition already being read: 31 would overlap one, 29 land exactly on one. `Best Service - Alpha Dance I` is the whole disc's worth of that and recovers nothing. The remaining 10 have no header at any position the search may look at. + + **ADR-0023 was right on the evidence it had, and one piece of that evidence turns out to be something else.** It refused a signature scan partly because the header's constant field is a sawtooth that audio reproduces — 374 matching blocks on `Global Trance Mission 2`, 153 on `ProSamples vol.14`. Digested, `vol.14`'s 153 are its five real partition headers plus **one 8192-byte block repeated 148 times, byte for byte**; audio does not do that. They are complete stale partition headers sitting in free space, with volume directories that parse. There is no byte test that separates those from a real header, because they are real headers — so the refusal stands and is firmer than it was. + + **The payload name check still has no unique positives, and this was the deliverable expected to give it one.** It fires 104 times across the collection now and never once without the id and valid tests firing too — the 15 808 recovered samples included. A displacement landing exactly on another sample's header is the case it exists for, and 15 808 more displaced samples produced none of them. ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md)) + + **`AKAI.S3000.Sound.Library.6`'s recovered partition 5 brings 18 stereo pairs whose halves declare different rates** — `PROPHE FX1` at 44 597 against 44 100, and the like. The joiner refuses to fuse a pair the disc disagrees with itself about and writes both mono halves, so nothing is lost; it is why the run's "skipped (damage)" line goes from 5 to 23 while no audio went missing. + +- **A short image says so.** `list` prints the displacement under each partition heading and in its layout line — *"11 partitions declared, 8 present in this image (7 of them displaced -- this image is short of the disc it was made from)"* — and `batch` records that line per disc in the manifest, as `layout`. Before this, a disc missing ten of its eleven partitions said only how many were present. ([ADR-0028](docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)) + ### Fixed - **S3000 AKAI samples had 42 bytes of header at the front of their audio.** The S3000 family writes a **192-byte** header where the S1000 family writes 150, and every sample was read at 150. That does not fail — the frame count comes out right, the WAV opens, the length is within 0.1 % — so what shipped was a file beginning with a burst of roughly ±20 000 lasting 0.24 ms in place of the attack, missing the last 21 frames of the sound, with every loop point 21 frames out of alignment. **13 451 of the collection's 56 490 AKAI samples**, on nine discs: `AKAI.S3000.Sound.Library.1`–`7` (4 455, 3 086, 1 990, 1 010, 601, 168 and 218), `East Connexion Piano` (730) and `AMG - Now CD-Rom for (AKAI)` (1 193). **Anyone who extracted those discs should do it again.** ([docs/formats/akai-fs.md](docs/formats/akai-fs.md), [ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md)) diff --git a/README.md b/README.md index 9537475..dc80dd9 100644 --- a/README.md +++ b/README.md @@ -78,32 +78,36 @@ Compressed `.mdx` is the piece no other open-source tool reads today. The format | | | |---|---| | Discs converted | 72 of 79 | -| Samples | 89 156 | -| Stereo pairs rejoined | 15 808 | +| Samples | 104 921 | +| Stereo pairs rejoined | 17 309 | | Audio CD tracks | 161 | | Duplicate audio suppressed | 5 719 | -| Entries not the file their entry placed | 61 | -| Entries skipped (damage) | 5 | -| Time | 48 s | +| Entries not the file their entry placed | 104 | +| Entries skipped (damage) | 23 | +| Time | 72 s | By filesystem: | | Discs | Samples | Stereo pairs | Skipped | |---|---:|---:|---:|---:| -| AKAI | 44 | 56 425 | 14 461 | 66 | +| AKAI | 44 | 72 190 | 15 962 | 127 | | E-mu `EMU3` | 7 | 14 738 | 6 | 0 | | ISO 9660 | 15 | 11 601 | — | 0 | | Roland `S770 MR25A` | 5 | 6 392 | 1 341 | 0 | "Stereo pairs" counts files joined from an `-L`/`-R` pair by name. E-mu's six are the only ones on those discs, and they are a different and much rarer thing than the 2 656 samples whose record declares two channels. -Every WAV was checked against the disc it came from — **70 of 70 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 89 156 payloads, zero mismatches. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all seven discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against. +Every WAV was checked against the disc it came from — **70 of 70 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 104 921 payloads, zero mismatches. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all seven discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against. -105 125 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **275 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, six are on a Proteus library that ships `Dead Air` as a sample, and two are on a Roland disc. That is what the discs hold, not something the decoder did. +122 391 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **275 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, six are on a Proteus library that ships `Dead Air` as a sample, and two are on a Roland disc. That is what the discs hold, not something the decoder did. -Sample rates run from 6 000 to 49 999 Hz across 1 047 distinct values. The odd ones are real — E-mu writes rates like 24 444 and 27 778, and AKAI uses 33 075 (¾ of 44 100) and 29 400 (⅔) to trade bandwidth for memory. They are carried through exactly as the disc states them and never rounded. +Sample rates run from 6 000 to 49 999 Hz across 1 071 distinct values. The odd ones are real — E-mu writes rates like 24 444 and 27 778, and AKAI uses 33 075 (¾ of 44 100) and 29 400 (⅔) to trade bandwidth for memory. They are carried through exactly as the disc states them and never rounded. -The 66 AKAI entries not written are two different faults. **61 are payloads that are not the file the directory placed there** — their header carries another file's id, valid flag or name — and 60 of those 61 are a run to the end of one volume, on ten discs; the other 34 AKAI discs have none. That is what a rip losing a run of blocks looks like from inside a directory, and it does not need a partition to go missing: `Best Service - Alpha Dance II` declares six partitions and holds all six, and still loses 21 of `AC.DRUMLOOPS`'s 22 samples. Each is refused with a line naming every field that disagrees and the entry that placed it, rather than being written out under a name that is not its own ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). The remaining five are damage of a different kind: four samples whose header is otherwise perfect and whose rate field reads 0, 519, 519 or 1280, and one stereo pair whose halves declare 44 033 and 44 100, so the joiner refuses to fuse them and writes both mono halves instead. +The 127 AKAI entries not written are two different faults. **104 are payloads that are not the file the directory placed there** — their header carries another file's id, valid flag or name — and 103 of those 104 are a run to the end of one volume, on nine discs; the other 35 AKAI discs have none. That is what a rip losing a run of blocks looks like from inside a directory, and it does not need a partition to go missing: `Best Service - Alpha Dance II` declares six partitions and holds all six, and still loses 21 of `AC.DRUMLOOPS`'s 22 samples. Each is refused with a line naming every field that disagrees and the entry that placed it, rather than being written out under a name that is not its own ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). Every one of those 103 is sitting intact, under its own name, a whole number of container blocks earlier in the image — the same fault as the displaced partitions below, one level down, and not yet recovered ([#35](https://github.com/bmxcode/samplerdisc/issues/35)). + +The other 23 are damage of a different kind, and 19 of them are not damage at all: four samples whose header is otherwise perfect and whose rate field reads 0, 519, 519 or 1280, and **19 `-L`/`-R` pairs whose two halves declare different sample rates** — 18 of them synth FX on `AKAI.S3000.Sound.Library.6`, at 44 597 against 44 100 and the like. The joiner refuses to fuse a pair the disc disagrees with itself about, and writes both mono halves instead, so no audio is lost. + +**Eight of the 44 AKAI images are short of the disc they were made from.** Whole 32 KB blocks are missing from the file, so every partition after a gap sits that much nearer the front than the disk's own table says — and 39 of them are found there and read, carrying 17 180 files and 15 808 samples that nothing could reach before. `list` and the manifest say so per disc: *"11 partitions declared, 8 present in this image (7 of them displaced — this image is short of the disc it was made from)"*. 70 declared partitions are still unread, and 60 of those are refused rather than absent: a header is there, inside a partition already being read, and reading it would put one run of bytes under two partitions at once ([ADR-0028](docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)). The seven that do not convert are accounted for: one S-550 disc present as both `.iso` and `.nrg` — a different format from the S-7xx and not yet read ([ADR-0014](docs/adr/0014-one-backend-per-on-disc-format.md)) — two Digidesign SampleCell discs, one audio CD with no cue sheet present as both `.mdx` and `.cdr`, and one ISO 9660 disc holding E-mu `.EBL` banks rather than audio. diff --git a/docs/README.md b/docs/README.md index ad008aa..90b4a42 100644 --- a/docs/README.md +++ b/docs/README.md @@ -59,13 +59,16 @@ All of that vanishes into a working parser. Six months on, the code says *what* | D17 | E-mu loop points in the WAV `smpl` chunk ([ADR-0025](adr/0025-the-loop-is-decoded-the-root-key-is-not.md)) | done | | D18 | E-mu stereo samples decoded from the record's channel count ([ADR-0026](adr/0026-the-record-declares-the-channel-count.md)) | done | | D19 | An AKAI payload must be the file its entry placed, and the S3000 header is 192 bytes ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)) | done | +| D20 | The partitions eight short AKAI images displaced, found by an anchored search ([ADR-0028](adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)) | done | -Across the local collection, by listing: 71 of 79 images claimed, 2 578 volumes, 110 989 files, 77 620 of them samples. The AKAI discs are 44 of those images and 68 997 of those files, read across 275 partitions — before D15 they were 14 670 files, because only the partition at the origin was read. +Across the local collection, by listing: 71 of 79 images claimed, 3 010 volumes, 128 169 files, 93 428 of them samples. The AKAI discs are 44 of those images and 86 177 of those files, read across 314 partitions — 275 where the disk's table puts them and 39 displaced by blocks the rip lost. Before D15 they were 14 670 files, because only the partition at the origin was read. ## What is not done -- **25 AKAI files are displaced inside a partition the table calls complete.** `Best Service - Alpha Dance II` declares six partitions and holds all six, and 21 of `AC.DRUMLOOPS`'s 22 samples are somebody else's audio; `AKAI.S3000.Sound.Library.1` and `.3` lose three and one the same way. The rip dropped a run of blocks *inside* a partition rather than the blocks a header sat on, so nothing in the partition table registers a gap and the declared-against-present count reads clean. They are refused and named rather than written ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)); recovering them would mean locating a volume's blocks by something other than the chain the allocation map declares, which is the search [ADR-0022](adr/0022-a-volume-is-explained-by-the-allocation-map.md) and [ADR-0023](adr/0023-partitions-come-from-the-table-the-disc-declares.md) both refused. Sibling of [#25](https://github.com/bmxcode/samplerdisc/issues/25) and invisible to it. -- **The AKAI payload name check has no positives on real data.** It is the test that separates *"this payload is a sample"* from *"this payload is **this** sample"*, and across all 44 discs it fires 60 times and never once without the id and valid tests having fired too — because on the short images the displacement lands mid-audio, which does not look like a header. It is exercised only synthetically. A displacement landing exactly on another sample's header is the case it exists for and the collection has none, which is worth knowing before [#25](https://github.com/bmxcode/samplerdisc/issues/25) recovers those partitions ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). +- **70 declared AKAI partitions are still unread, and 60 of them are not absent.** A header is there and it is inside a partition already being read — 31 would overlap one, 29 land exactly on one. `Best Service - Alpha Dance I` is the whole disc's worth of that: its one displaced partition sits four blocks inside partition 4, so the disc recovers nothing at all. Reading them would mean putting one run of bytes under two partitions' bookkeeping, and this project has no field that says how much shorter than its declared size a partition preceding a gap really is ([ADR-0028](adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md), [#25](https://github.com/bmxcode/samplerdisc/issues/25)). The other 10 have no header at any position the search may look at. +- **104 AKAI files are displaced inside a partition the table calls complete.** `Best Service - Alpha Dance II` declares six partitions and holds all six, and 21 of `AC.DRUMLOOPS`'s 22 samples are somebody else's audio; `AKAI.S3000.Sound.Library.5`'s recovered partitions lose 23 the same way. The rip dropped a run of blocks *inside* a partition rather than the blocks a header sat on, so nothing in the partition table registers a gap. **The mechanism is now measured rather than supposed: 103 of the 104 are sitting intact, carrying their entry's own name, a whole number of container blocks earlier.** They are still refused and named rather than written ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)); recovering them would mean locating a volume's blocks by something other than the chain the allocation map declares, which is the search [ADR-0022](adr/0022-a-volume-is-explained-by-the-allocation-map.md) and [ADR-0023](adr/0023-partitions-come-from-the-table-the-disc-declares.md) both refused, and unlike a partition a file has no declared position to search back from ([#35](https://github.com/bmxcode/samplerdisc/issues/35), [ADR-0028](adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)). +- **The AKAI payload name check still has no positives on real data.** It is the test that separates *"this payload is a sample"* from *"this payload is **this** sample"*, and across all 44 discs it now fires 104 times and never once without the id and valid tests having fired too. D20 was the deliverable expected to change that — a displacement landing exactly on another sample's header is the case it exists for, and recovering 15 808 more samples from displaced partitions produced none. It remains exercised only synthetically ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). +- **What wrote the stale partition headers in free space is unestablished.** `ProSamples vol.14` carries 148 byte-identical copies of one, `vol.12` 86, and `Global Trance Mission 2` three distinct ones repeated 288, 58 and 19 times, every one in a block the allocation map calls free. Filler from the mastering and a header from an earlier state of the disk both fit. It matters only as the reason a partition header may never be scanned for, and for that the reading does not need settling ([ADR-0028](adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)). - **What the AKAI valid byte's low bits mean is unestablished.** `0x81` on 29 samples and `0x9c` on two, against `0x80` on 56 397. The `0x80` flag is the sample-is-valid bit and the rest is unread; three combinations on two discs is not enough to read them from. - **Roland S-550.** `Roland LCD1.iso`/`.nrg` opens `* ROLAND S-550 *` and is a different format from the S-7xx entirely ([ADR-0014](adr/0014-one-backend-per-on-disc-format.md)). Neither archive holds a second specimen, so it stays deferred rather than being reverse-engineered from one disc. diff --git a/docs/adr/0023-partitions-come-from-the-table-the-disc-declares.md b/docs/adr/0023-partitions-come-from-the-table-the-disc-declares.md index 732cfad..cfcfbe7 100644 --- a/docs/adr/0023-partitions-come-from-the-table-the-disc-declares.md +++ b/docs/adr/0023-partitions-come-from-the-table-the-disc-declares.md @@ -1,6 +1,6 @@ # ADR-0023 · An AKAI disc's partitions come from the table it declares -**Status:** accepted · 2026-08-21 +**Status:** accepted · 2026-08-21 · *amended by [ADR-0028](0028-a-displaced-partition-is-anchored-quantised-and-floored.md): "a declared position with no header is skipped, never searched for" becomes "searched for backwards from the declared position, in the container's storage unit, for a header restating this partition's size, and never below the end of the partition already accepted". The refusal of a **scan** stands — and the 374 blocks below turn out to be complete stale partition headers rather than audio, which makes it firmer.* ## Context diff --git a/docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md b/docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md new file mode 100644 index 0000000..1551af7 --- /dev/null +++ b/docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md @@ -0,0 +1,109 @@ +# ADR-0028 · A displaced AKAI partition is searched for from its declared position, in the container's unit, and never inside one already read + +**Status:** accepted · 2026-08-22 · *amends [ADR-0023](0023-partitions-come-from-the-table-the-disc-declares.md): "a declared position with no header is skipped, never searched for" becomes "searched for backwards, under three constraints". What ADR-0023 refused was a free signature scan, and it was right to; the evidence that changed is below.* + +## Context + +Eight of the 44 AKAI images are **short of the disc they were made from**. The container is faithful to the file — every MDX block decodes, every one emits exactly the bytes it should — and the file is not a complete copy of the disc: whole 32 KB blocks are missing, so everything after a gap sits that much nearer the front. [ADR-0022](0022-a-volume-is-explained-by-the-allocation-map.md) found it from inside a volume directory, [ADR-0023](0023-partitions-come-from-the-table-the-disc-declares.md) measured it across the collection, and [issue #25](https://github.com/bmxcode/samplerdisc/issues/25) records what it costs: **109 of the 384 declared partitions have no header where the table puts them**, and on these eight the header is not absent, it is displaced. + +ADR-0023 declined to go and find them, on three grounds, each of which it stated as sufficient: + +- **the signature is a sawtooth and audio reproduces it** — 374 blocks of `Global Trance Mission 2`'s free space matched, and 153 of `ProSamples vol.14`; +- **filtering the matches by whether they yield files lost real partitions**, 94 volumes down to 72 on `Advanced Media Trax 3`; +- **the gain came almost entirely from short images**, where the audio is displaced with nothing reporting it — 7 723 of the recovered files being on `Kickin' Lunatic Beats 2 CD1`, the disc whose partition 1 already extracted nine samples that were not their own. + +The third is answered, and by a different deliverable: [ADR-0027](0027-a-payload-must-be-the-file-its-entry-placed.md) made every AKAI sample prove it is the file its directory entry placed, by four fields the payload repeats. Displaced audio is no longer something that ships silently; it is refused and named. So the question ADR-0023 could not ask is now askable — *does the recovered audio verify?* — and the answer is measured below. + +The first two are answered by the search being **anchored, size-matched and floored** rather than being a scan. That is a claim, and this record has to show it. + +### What the header-shaped blocks in free space actually are + +**ADR-0023's first ground was right about the danger and wrong about its cause, and the truth is worse.** Those 374 blocks are not audio that happens to reproduce a rising sawtooth. Digested, `ProSamples vol.14`'s 153 matches are **one 8 KB block repeated 148 times**, byte for byte, plus its five real partition headers. `Global Trance Mission 2`'s 374 are three distinct blocks repeated 288, 58 and 19 times, plus its partitions and one singleton. Audio does not repeat 8192 bytes exactly 148 times. + +They are **partition headers**, complete ones. `vol.14`'s carries the pristine formatted volume directory — `VOLUME 001` … `VOLUME 100`, type 0, start block 0 — and `Global Trance Mission 2`'s names real volumes: `R+R KIT 1`, `REGGAE KIT 1`, `RAP KIT 1`. Every one sits in a block the partition's own allocation map calls free. Whether they are filler the mastering wrote or headers from an earlier state of the disk is not established and does not matter here: **they pass every test the header offers**, including the two size fields, including a volume directory that parses. A scan cannot tell them from a real header on the bytes, at all, and there is no stricter byte test that would. + +That is why the constraints below are about *where a header may be found* and not about *what a header looks like*. + +### What the constrained search recovers + +Searching backward from each declared position in the container's own storage unit, requiring `partition_header()` to restate the size the table gave **that** partition, and stopping at the end of the partition already accepted: + +| Disc | Recovered | Volumes | Files | Samples | Passing ADR-0027 | +|---|---:|---:|---:|---:|---:| +| `AMG - Kickin' Lunatic Beats 2 CD1` | 7 | 121 | 7 723 | 7 308 | 7 293 | +| `AMG - Kickin' Lunatic Beats 2 CD2` | 7 | 100 | 6 203 | 5 912 | 5 912 | +| `AKAI.S3000.Sound.Library.6` | 7 | 84 | 1 054 | 955 | 955 | +| `Back In Time Records - Elektra Vox` | 5 | 33 | 661 | 463 | 463 | +| `Audio Factory - Classical Wild Takes` | 4 | 18 | 189 | 80 | 80 | +| `AKAI.S3000.Sound.Library.5` | 3 | 34 | 473 | 400 | 377 | +| `AKAI.S3000.Sound.Library.7` | 3 | 20 | 660 | 546 | 546 | +| `AMG - Global Trance Mission 2` | 3 | 22 | 217 | 144 | 139 | +| **Total** | **39** | **432** | **17 180** | **15 808** | **15 765 (99.7 %)** | + +**The 99.7 % is the finding this rests on, and it is not a coincidence of the search.** The gaps are whole blocks missing from the image, so everything past one shifts by a constant: inside a recovered partition the directory and its audio moved *together* and stay consistent with each other. Displacement only breaks payload-versus-directory agreement where a gap falls **inside** a partition read at its declared position — which is exactly ADR-0027's 61 mismatches, all of them a run to the end of one volume just past a gap. + +Three things corroborate the recovery and none of them is the search: + +- **The displacements reproduce the table in [the format doc](../formats/akai-fs.md)**, measured a deliverable ago by a different method and for a different purpose. +- **`Kickin' CD1` yields 7 723 files**, which is precisely the number ADR-0023's rejected signature scan measured for that disc independently. +- **No recovered partition repeats another's header block**, on any of the eight — which is what separates a partition from the stale copies above. + +## Decision + +**A declared partition with no header at its declared position is searched for backwards, in the unit the container stores the disc in, for a header restating the size the table gave that partition — and never below the end of the partition already accepted.** + +Five parts. + +**Backwards from the declared position, and anchored to it.** A short image has *lost* bytes; nothing has moved away from the front. The search starts where the table puts partition *k* and walks back, so what it finds is partition *k* or nothing. **That is what settles the index**, which is otherwise unanswerable: sizes repeat — eight of `Loop Soup`'s nine are 7680 blocks — so a header restating size N does not say which N it is, and a free scan that found 72 headers could name only five of them. Here the index is the table's and the search never has to choose. The output path `out/partition-N/` is therefore the table's N, on a displaced partition exactly as on a present one, and there is no case where it cannot be known. + +**In the unit the container states.** `SectorImage.granularity` is new: cooked bytes per unit this container stores the disc in — the thing an incomplete image is missing whole numbers of. Flat containers hold the sectors literally and report one sector; `MdxImage` reports its measured block size scaled to the cooked stride, 32 768 on all eight of these and 30 720 on a subchannel image. The filesystem asks and does not name it: a `32768` in `fs/akai.py` would be the container's business asserted by the layer that cannot know it, and an AKAI check in `container/` would be the same leak the other way ([ADR-0003](0003-brand-neutral-pluggable-backends.md)). A container that reports nothing usable gets no search rather than a default step chosen in `fs/`. + +**Confirmed by the size the table gave that partition.** The same confirmation a declared position gets — the constant field, the size echo at `0xC6`, the tail at `0xC8` — plus the block count this index was given. Placement by one structure, confirmation by another, which is the shape [ADR-0020](0020-read-e-iv-through-its-sample-directory.md), [ADR-0021](0021-a-bank-owns-the-run-its-header-declares.md) and ADR-0023 already use. + +**Never below the end of the partition already accepted.** This is the safety argument and it does all of the rejecting. It is a **bound on the search**, not a filter on its results: nothing below the floor is examined, so an overlap cannot arise and cannot need resolving. Two consequences fall out of it rather than being added. Partition 1 can never move, on any disc, because its declared position is 0 and its window is empty — which is what protects every count pinned since #22. And accepted partitions never overlap in either direction: a recovered partition ends *before* its declared end, so it cannot reach into a later one either. + +**A displaced partition says so, and says it where a listing shows it.** `Partition.displaced` and `Volume.displaced` carry the byte distance; `list` prints it under the partition heading and in the layout line — *"11 partitions declared, 8 present in this image (7 of them displaced — this image is short of the disc it was made from)"* — and `batch` records that line per disc in the manifest. Not on the `File` or on an `Extracted`: the provenance is a property of the partition, constant across the 7 723 files of one, and a per-file flag would repeat it 7 723 times to say one thing. + +## Alternatives rejected + +**Keep ADR-0023's rule and leave the fifteen thousand files unread.** The conservative position, and it was the right one for exactly as long as a recovered payload could not be checked. ADR-0027 is what changed: 99.7 % of the 15 808 recovered samples prove they are the files their entries placed, against a structure the search knows nothing about, and the 0.3 % that do not are refused and named like any other. Refusing readable audio that verifies, on the strength of a rule whose stated reason has been answered, is not conservatism. + +**Scan the image for the header signature**, ADR-0023's own framing of what recovery would mean. Rejected, and more firmly than ADR-0023 could: the free blocks of these discs hold *complete partition headers*, 148 identical copies on one disc, so the scan's false positives are not weak matches to be filtered — they are indistinguishable from the real thing by construction. It also cannot name what it finds: 67 of 72 first-pass hits restated a size several partitions of the disc share. + +**Filter a scan's matches by whether the partition yields files.** ADR-0023 measured this and it *lost* real partitions — 94 volumes down to 72 on `Advanced Media Trax 3`. Rejected again, on the same measurement, and worth restating because it is the tempting repair: a partition that legitimately holds nothing is not a false positive, and a test that cannot tell those apart trades a wrong answer for a missing one. `Advanced Media Trax 3` is pinned in `tests/test_discs.py` at 9 partitions, 94 volumes, 2 938 files, **nothing displaced**, precisely so a future search that touches it fails. + +**Search in AKAI blocks — a unit the filesystem can state on its own.** No container API, one layer touched instead of two, and *measured identical on this collection*: at 32 768, 8 192 and 2 048 bytes the search recovers the same 39 partitions on the same eight discs, with the same displacements. Rejected on what it claims rather than on what it produced. A step of 8 192 examines four positions for every one a lost container block could have put a header at, and three of the four are places the fault cannot reach; looseness that the data does not currently punish is still looseness, and it is what ADR-0023 refused. The measurement is the reason to be relaxed about the choice, not a reason to make the weaker one. + +**Read an overlapping recovery, truncated at the clash.** 24 of the 70 refusals are a header that is really there and really is that partition's, sitting inside a partition already being read — `Kickin' CD2`'s partition 2, four AKAI blocks inside partition 1, is the clean specimen. Truncating would yield most of the partition. Rejected because the clash is not a boundary dispute to be split: it means the image holds one run of bytes that two partitions' bookkeeping both describe, and the audio can only belong to one of them. Reading both would put the same blocks under two names — the failure `Protozoa` taught in [ADR-0021](0021-a-bank-owns-the-run-its-header-declares.md), and the one thing worse than not reading a partition. + +**Move the *present* partition instead, on the grounds that the displaced one is where the disc really says.** The most interesting alternative, because on a clash the present partition's declared extent is provably wrong: its last blocks are not in the image, which is why its neighbour's header sits inside them. Rejected on the gate this deliverable is judged by. The 275 partitions read today are the collection's entire established baseline — 2 154 volumes and 68 997 files, pinned per disc — and re-placing any of them to gain a neighbour would move counts that four deliverables have verified, in exchange for a partition whose extent still could not be read whole. What is actually true is that a partition preceding a gap is *shorter in the image than it declares*, and this project has no field that states that. Saying so in the layout line is the honest half; acting on it is not. + +**Refuse every disc that has any overlapping recovery.** All-or-nothing per disc, which would be simple to explain. Rejected on the data: `Kickin' CD2`'s partition 2 clashes and its partitions 3 to 9 do not, and refusing them costs 6 203 files to punish a fault they do not share. The rule is per partition because the damage is. + +**Recover `Alpha Dance I` by relaxing the floor by the size of one container block.** Its single missing partition is displaced by four AKAI blocks — the same gap as `Kickin' CD2` — and one byte of tolerance would take it. Rejected as a threshold with one specimen behind it, and the argument does not stop there: an overlap is an overlap at any size, and the reason to refuse a 4-block clash is the reason to refuse a 60-block one. `Alpha Dance I` recovers nothing, and the record says so plainly rather than tuning until it does. + +**Search forward as well as backward.** Symmetry, and it costs nothing to write. Rejected because it is not symmetric: a rip that lost blocks makes an image *short*, and there is no mechanism on the shelf that inserts them. Every one of the 39 displacements is backwards, as are all 70 refusals. A forward search would double the candidate positions in exchange for a fault nothing has produced. + +**Report the provenance per file, on `Extracted` and in the manifest's skip list.** The strongest signal for a user grepping a run. Rejected for repeating one fact 7 723 times: displacement is a property of the partition, `Volume.displaced` is where a consumer can already see it, and the manifest carries the backend's layout line per disc, which says it once and says it for the discs that wrote nothing too. + +**Fix issue [#35](https://github.com/bmxcode/samplerdisc/issues/35) here — the run of blocks lost *inside* a partition.** The mechanism found here explains it exactly, and that is now measured rather than supposed: **103 of the 104 refused payloads across the collection are found intact, carrying their entry's own name, a whole number of container blocks earlier** — one block back on `Alpha Dance II`'s 21 and `Library.1`'s 3, 134 on `Library.3`'s one, one to four on the rest. The single exception is `Loop Soup`'s known directory record that lands mid-sample. Rejected as a different decision, not a bigger version of this one: a partition has a *declared position* to anchor a search to, and a file has only a chain the allocation map states. Placing a file's audio somewhere the map does not put it is the search [ADR-0022](0022-a-volume-is-explained-by-the-allocation-map.md) and ADR-0023 both refused, and it would need its own answer to "which file is this?" — which is a name comparison, on payloads whose name does not decode on 54 of 60. + +## Consequences + +**Good, and the headline.** The 44 AKAI discs go from 275 partitions to **314**, from 2 154 volumes to **2 586**, and from 68 997 files to **86 177**. 15 765 samples that were unreadable are now written, all of them on the eight images the collection had written off. The collection goes from 89 156 samples to **104 921**. + +**Good.** The audio verifies against a structure the search does not consult. 15 765 of the 15 808 recovered samples carry a payload header whose id, valid flag and name agree with the directory entry that placed them — 99.7 %, against 99.85 % over all 72 298 AKAI samples — and every accepted payload's word count agrees with its entry's declared size. `Kickin' CD2`, `Library.6`, `Library.7`, `Elektra Vox` and `Classical Wild Takes` recover **6 203, 1 054, 660, 661 and 189 files with not one refusal between them**. + +**Good.** The eight images now *say* they are short, in `list` and in the manifest, per partition and per disc. Before this they said "11 partitions declared, 1 present" and the other ten were an absence with no explanation attached. + +**Bad, and stated plainly.** 70 declared partitions stay unread and 31 of them are not absent — a header is there, inside a partition already being read. `Best Service - Alpha Dance I` gains nothing at all: its one displaced partition clashes, and this deliverable's answer for that disc is no answer. 10 more have no header at any position the search may look at, and 29 land exactly on a partition already read. + +**Bad.** Extraction on the eight discs writes about 15 800 files that were not there before, under `out/partition-N/` paths that did not exist. Anyone who extracted them should do it again, and anyone who diffed a previous run will see additions on those eight and nowhere else. + +**Bad.** `list` and `batch` are slower on a disc with missing partitions, because the search walks from a declared position back to the last accepted one and that window can be most of an image. Measured over the whole collection the walk goes from 26 to 45 seconds and a full `batch` from 48 to 72; on a disc with nothing missing the cost is zero, since the search never runs. + +**Watch for.** A disc where a *present* partition's header is a stale copy. The floor makes the first partition of a run authoritative over everything after it, so a stale header accepted at a declared position would push the real one below the floor and cost every partition behind it. Nothing on the shelf does this — the 275 present partitions were verified before this deliverable and none moved — but the failure would present as partitions quietly not recovering rather than as an error. + +**Watch for.** ADR-0023's statement that a partition header must never be scanned for, which stands and is now better argued. The line to hold is that a position must be *placed* by a structure the disc states — here the table, with the loss the container declares subtracted from it — and never chosen because the bytes there look right. The free blocks of these discs are full of bytes that look exactly right. + +**Watch for.** `granularity` growing meanings. It is what an incomplete image is missing whole numbers of, and nothing else: not a read size, not an alignment, not a hint about compression. The temptation will be to use it as a chunk size somewhere in `container/`, and then to change it for a reason that has nothing to do with damage. diff --git a/docs/adr/README.md b/docs/adr/README.md index b907ce1..cb0b7a1 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -33,3 +33,4 @@ If you find yourself writing an ADR with no rejected alternative, you are writin | [0025](0025-the-loop-is-decoded-the-root-key-is-not.md) | The E-mu loop is decoded from the record; the root key is not there to decode | Deriving the root key from the sample name; no `smpl` chunk without a root key; clamping the loop end as AKAI and Roland do; emitting the second channel's loop as well; trusting the declared extent over the record length | | [0026](0026-the-record-declares-the-channel-count.md) | The E-mu record declares the channel count, and its own extents confirm it | Splitting on the channel count alone; writing the mono halves alongside; filing it under `stereo/`; deciding stereo from the audio; taking high-correlation one-channel records as stereo; splitting at `end_L`; writing two mono files | | [0027](0027-a-payload-must-be-the-file-its-entry-placed.md) | An AKAI payload must be the file its entry placed, and its header length is the generation the entry declares | Closing #23 as not reproducing; dropping the name comparison; solving for the header length at runtime; branching on the id and valid bytes; sniffing the header pattern; an allow-list of valid bytes; never relaxing the flag; writing a mismatch under the payload's name; a `--force` flag; a third result shape; fixing the mid-partition damage here | +| [0028](0028-a-displaced-partition-is-anchored-quantised-and-floored.md) | A displaced AKAI partition is searched for from its declared position, in the container's unit, and never inside one already read | Leaving them unread; scanning for the signature; filtering a scan by file yield; searching in AKAI blocks; truncating an overlapping recovery; moving the present partition instead; refusing a whole disc for one clash; relaxing the floor for `Alpha Dance I`; searching forward; per-file provenance; fixing #35 here | diff --git a/docs/formats/akai-fs.md b/docs/formats/akai-fs.md index eac2bf0..e73ea09 100644 --- a/docs/formats/akai-fs.md +++ b/docs/formats/akai-fs.md @@ -30,9 +30,13 @@ Getting index 10 wrong is the classic failure and it is not obvious: `KICKIN B0- | `0x70A` | **block allocation map** — one u16 per block, as many as `0x00` declares | | `0x4500` | **partition table** — the disk's own list of its partitions, in the first partition only | -`0x02`, `0xC6` and `0xC8` all hold on all **275 partitions** of the 44 AKAI discs, and together they are what identifies a partition header. The two size fields matter most: a block count with no echo behind it is not a partition's, which is a firmer test than anything about the image's length. +`0x02`, `0xC6` and `0xC8` all hold on all **314 partitions** read across the 44 AKAI discs, and together they are what identifies a partition header. The two size fields matter most: a block count with no echo behind it is not a partition's, which is a firmer test than anything about the image's length. -**The pattern at `0x02` is a rising sawtooth, and sample data reproduces it.** As 16-bit PCM, `3333 × i` is a saw wave, so audio does match it — 374 blocks of one disc's free space carry a complete header prefix, every one in a block the allocation map calls free. That is why a partition header is *confirmed* where the table says one is, and never scanned for ([ADR-0023](../adr/0023-partitions-come-from-the-table-the-disc-declares.md)). +**Free blocks carry complete copies of a partition header, and a scan cannot tell them from the real thing.** 374 blocks of `AMG - Global Trance Mission 2`'s free space satisfy all three fields *and* restate a size the disk's table declares; `ProSamples vol.14` has 153 and `vol.12` 91. They are not audio that happens to fit the pattern, which is what an earlier note here said: digested, `vol.14`'s 153 are its five real partition headers plus **one 8192-byte block repeated 148 times, byte for byte**, and `Global Trance Mission 2`'s are three distinct blocks repeated 288, 58 and 19 times. Audio does not do that. + +Each copy is a whole header with a volume directory that parses. `vol.14`'s holds the pristine formatted state — `VOLUME 001` … `VOLUME 100`, type 0, start block 0 — and `Global Trance Mission 2`'s names real volumes (`R+R KIT 1`, `REGGAE KIT 1`, `RAP KIT 1`). What wrote them is not established: filler from the mastering, or headers from an earlier state of the disk. Every one is in a block the partition's own allocation map calls free. + +The consequence is the same either way and stronger than the sawtooth reading was: **there is no byte test that separates these from a partition header, because they are partition headers.** A position must be placed by something the disc states — the table, with the container's lost blocks subtracted from it — and confirmed there ([ADR-0023](../adr/0023-partitions-come-from-the-table-the-disc-declares.md), [ADR-0028](../adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)). The size at `0x00` is what bounds the allocation map, and it is much smaller than the disc: a partition of 7680 blocks is 62.9 MB inside an image of 500 MB or more. A disc carries **several partitions** — see *More than one partition* below. @@ -47,10 +51,12 @@ Volume entry, 16 bytes: **The type is a byte, and 13 is a different field.** Reading the pair as one u16 was harmless for as long as nothing used the value — `start` at 14 is unaffected, so every volume on every disc was still found — and it produced nonsense the moment anything did: the one disc that sets byte 13 reported volume types of 513, 769 and 1025, an inflation of 256 per volume. -What byte 13 is has not been established. It is zero on 4373 of the 4400 non-empty slots across the collection. The exception is `OMI … Universe Of Sounds Vol.1 (Roland S-770,S-750)`, where it runs 2, 3, 4 … 28 over the disc's 27 live volumes in slot order and is 0 on the unused slot at the end — an incrementing per-volume index of some kind, on one disc, with no second specimen to check it against. +What byte 13 is has not been established. It is zero on 4373 of the 4400 non-empty slots of the 44 discs' **first** partitions, and the exception is `OMI … Universe Of Sounds Vol.1 (Roland S-770,S-750)`, where it runs 2, 3, 4 … 28 over the disc's 27 live volumes in slot order and is 0 on the unused slot at the end — an incrementing per-volume index of some kind, on one disc, with no second specimen to check it against. Over all 314 partitions read it is zero on 31 333 of 31 400; where the other 40 are has not been looked into. ### Volume type +Counted over the **first** partitions, which is where the correspondence below was established: + | Byte | Volumes | Reading | Evidence | |---|---|---|---| | 1 | 338 | S1000 | no file in any of them sets the type byte's high bit | @@ -58,6 +64,8 @@ What byte 13 is has not been established. It is zero on 4373 of the 4400 non-emp | 7 | 91 | CD3000 | 57 of 91 hold nothing but high-bit files; the rest are mixed | | 0 | 10 | not a live volume | — | +Over all 314 partitions the shape is the same at ten times the size — 1 733 type 1, 711 type 7, 107 type 3 and 35 type 0 — with every type-1 volume that holds files holding only low-bit ones, and type 7 and type 3 running 449 of 682 and 79 of 107 all-high with the rest mixed. + The generations line up with the high bit on the *file* type byte, which is the same signal that names a kept original `.s3p` rather than `.s1p`. A volume can hold files of both generations, so the correspondence is a strong tendency and not a rule. **Type 0 is not an allocation flag, and must not be used as one.** Six of the ten type-0 volumes hold nothing, and the other **four hold 63 files between them** — real audio on `Big Bang` and ProSamples `vol.01`, `vol.19` and `vol.24`. Rejecting type 0 is a one-line change that silently discards all of it. What separates the two groups is the allocation map, not the type. @@ -76,7 +84,7 @@ At `0x70A`, immediately after the volume directory's hundred slots: **one u16 pe A file's extent is the chain from its start block to `0xC000`. **That is what verifies the map rather than merely making it plausible**: the chain length and the file size are stated by two different structures, and across all 44 AKAI discs they agree for **14 607 of 14 607 files** in the first partitions — exactly, with no disc disagreeing anywhere. -Read across all 275 partitions the figure is **68 267 of 68 284**, and the seventeen exceptions are one thing rather than a scatter: every one is a `MULTI FILE` — type `m`, all on `AKAI.S3000.Sound.Library.1` — whose chain runs exactly one block past what its size needs. A multi appears to be allocated a spare block. It is the only kind that disagrees anywhere. +Read across all 314 partitions the figure is **85 338 of 85 355**, and the seventeen exceptions are one thing rather than a scatter: every one is a `MULTI FILE` — type `m`, all on `AKAI.S3000.Sound.Library.1` — whose chain runs exactly one block past what its size needs. A multi appears to be allocated a spare block. It is the only kind that disagrees anywhere. The exclusion behind that figure is itself the finding. Five volumes sit on blocks the map calls free, and their files' chains are gone with them; four of those five hold 63 files that read perfectly. These are **deleted volumes**: the blocks went back to the free list, and because the medium is a mastered CD-ROM nothing ever reused them, so the directory and the audio are still there to be read. They are listed with their files like any other volume. @@ -117,9 +125,9 @@ That last entry is why the table is worth reading rather than multiplying the fi A partition's own blocks are numbered from **its** start, so the same block number means a different place in each one, and a file read with the partition term dropped returns another partition's audio rather than an error. -Where the image holds no header at a position the table declares, it is **skipped and not searched for**. On the discs where that happens the header turns up displaced by a whole number of the container's 32 KB blocks — the image is short of the disc it was made from, which is the fault below and not a filesystem to go hunting through. +Where the image holds no header at a position the table declares, the partition is either absent from the disc or **displaced** — moved towards the front by blocks the container lost. The two are told apart by where a header turns up, and the rule for that is in *An image can be short* below. -Across the 44 discs the table declares 384 partitions and 275 are present in the images, holding **2 154 volumes and 68 997 files**, against the 448 volumes and 14 670 files of the first partitions alone. +Across the 44 discs the table declares 384 partitions. **275 sit where the table puts them and 39 more are displaced**, holding **2 586 volumes and 86 177 files** between them, against the 448 volumes and 14 670 files of the first partitions alone. ## An image can be short @@ -132,25 +140,34 @@ Two structures agree on the displacement and neither knows about the other: 131 072 bytes is exactly four MDX blocks, which is a quantity of the *container* and one the AKAI filesystem knows nothing about — that is what identifies the layer at fault. `CD2` of the same pair is short by one such block (its partition 2 is at `size − 4`), and its partition 1 happens to survive it intact. -**The partition table turns that into a measurement anyone can repeat**, and it explains the three discs that looked like a different layout. Where a declared partition has no header at its declared position, searching the image finds one *earlier*, always by a whole number of 32 KB container blocks: +**The partition table turns that into a measurement anyone can repeat**, and it explains the discs that looked like a different layout. Where a declared partition has no header at its declared position, searching the image finds one *earlier*, always by a whole number of 32 KB container blocks. These are the eight images short of their disc, with the displacement of every partition recovered from them, in AKAI blocks: -| Disc | Declared | Present | Displacement, first to last | -|---|---|---|---| -| `AMG - Kickin' Lunatic Beats 2 CD2` | 9 | 1 | 4 blocks, on all eight | -| `Best Service - Alpha Dance I` | 5 | 4 | 4 | -| `AKAI.S3000.Sound.Library.5` | 9 | 3 | 8 … 36 | -| `AMG - Global Trance Mission 2` | 9 | 4 | 8 … 32 | -| `Audio Factory - Classical Wild Takes` | 11 | 6 | 16, on all five | -| `AMG - Kickin' Lunatic Beats 2 CD1` | 11 | 1 | 16 … 336 | -| `AKAI.S3000.Sound.Library.6` | 9 | 1 | 60 … 68 | -| `Back In Time Rrcords - Elektra Vox` | 13 | 1 | 424 … 2888 | -| `AKAI.S3000.Sound.Library.7` | 11 | 1 | 1456 … 7288 | +| Disc | Declared | Read | Displacements recovered | +|---|---:|---:|---| +| `AMG - Kickin' Lunatic Beats 2 CD2` | 9 | 8 | 4 on partitions 3–9 | +| `Audio Factory - Classical Wild Takes` | 11 | 10 | 16 on partitions 8–11 | +| `AKAI.S3000.Sound.Library.6` | 9 | 8 | 68 on partitions 3–9 | +| `AMG - Kickin' Lunatic Beats 2 CD1` | 11 | 8 | 52 on partitions 3–8, 200 on 10 | +| `AKAI.S3000.Sound.Library.5` | 9 | 6 | 12, 32, 32 on partitions 5, 7, 8 | +| `AMG - Global Trance Mission 2` | 9 | 7 | 8, 8, 32 on partitions 6, 7, 9 | +| `AKAI.S3000.Sound.Library.7` | 11 | 4 | 5508, 2028, 512 on partitions 3, 4, 5 | +| `Back In Time Rrcords - Elektra Vox` | 13 | 6 | 488, 964, 1844, 2664, 1928 on partitions 3, 5, 7, 9, 11 | + +Every displacement is a multiple of 4 blocks — 32 768 bytes, one MDX block — and every one of these images is `.mdx`. They also **accumulate**: `Elektra Vox` slips by 424 blocks, then another 64, then 156, and so on down the disc, which is what a rip losing blocks here and there looks like from inside the filesystem. These are not discs laid out differently; they are incomplete rips. -Every displacement is a multiple of 4 blocks — 32 768 bytes, one MDX block — and every one of these images is `.mdx`. They also **accumulate**: `Elektra Vox` slips by 424 blocks, then another 64, then 156, and so on down the disc, which is what a rip losing blocks here and there looks like from inside the filesystem. These are not discs laid out differently; they are incomplete rips. Recovering their partitions would mean locating headers by search, which [ADR-0023](../adr/0023-partitions-come-from-the-table-the-disc-declares.md) declines to do and [issue #25](https://github.com/bmxcode/samplerdisc/issues/25) records. +The 39 recovered partitions hold **432 volumes, 17 180 files and 15 808 samples**, and 15 765 of those samples — 99.7 % — carry a payload header that agrees with the directory entry that placed it. That is the point about a gap: it removes whole blocks, so everything after it moves by a *constant*, and inside a displaced partition the directory and its audio move together and stay consistent. A displacement only breaks that agreement where the gap falls **inside** a partition read at its declared position. -A missing partition is not always damage, and the two are distinguishable: on the ProSamples discs the declared partitions that are absent have **no header anywhere near** them, because the CD carries only the front of a larger disk or the mastering never wrote them. Displacement is the tell, not absence. +**Where a search may look, and where it may not.** A header is confirmed at a found position exactly as at a declared one, and three constraints keep the search from being a scan ([ADR-0028](../adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)): -A short image also shows up in the table's arithmetic alone: `Kickin' Lunatic Beats 2 CD1` declares eleven partitions where the image holds one, and `ProSamples vol.54` declares nine of a 63 488-block disk on a CD of 30 720 blocks — the second of those is not damage, it is a CD carrying only the front of a larger disk. Declared against present is worth printing for that reason: it names the gap without diagnosing it. +- it walks **backwards from the position the table gives that partition**, because a short image has lost bytes and nothing moved away from the front — which also settles the index, since a header restating size N does not say which N it is; +- it steps in the **unit the container stores the disc in**, 32 768 cooked bytes for these `.mdx` images, because that is what the rip lost whole numbers of; +- it stops at the **end of the partition already accepted**, so nothing is ever found inside a partition already being read. + +The last is what refuses the rest. **70 declared partitions stay unread**: 10 have no header at any position the search may look at, 29 land exactly on a partition already read, and 31 would overlap one. `Best Service - Alpha Dance I` is entirely in the third group — its one displaced partition sits four blocks inside partition 4, so the disc recovers nothing. + +A missing partition is not always damage, and the two are distinguishable. On the ProSamples discs the declared partitions that are absent were never written — the CD carries only the front of a larger disk — and the nearest header-shaped block is either exactly one partition back (the previous partition's own header, seen through a size the disk repeats) or a stale copy sitting in free space, 70 blocks back on `vol.12` and 21 on `vol.14`. Both are inside a partition already read, and both are refused for that and not for being those discs. + +A short image also shows up in the table's arithmetic alone: `Kickin' Lunatic Beats 2 CD1` declares eleven partitions and three of them cannot be read from this image at all, and `ProSamples vol.54` declares nine of a 63 488-block disk on a CD of 30 720 blocks — the second of those is not damage, it is a CD carrying only the front of a larger disk. Declared against read is worth printing for that reason: it names the gap without diagnosing it. The visible consequence inside partition 1 is not only the four empty volumes. **Nine files in `13-TRACK 06` no longer hold their own audio**: everything past the first gap has slid, so their payload is mid-PCM rather than a header. They are refused and named, one line each saying which fields disagree and which entry placed them ([ADR-0027](../adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). [Issue #23](https://github.com/bmxcode/samplerdisc/issues/23) proposed that check believing they were being written out; they were already being refused, with a message that said neither which test failed nor that a directory entry was involved. @@ -240,13 +257,13 @@ Sample data is **signed 16-bit little-endian mono PCM**. That is already exactly Three structures agree, none of them aware of the others: -- **The type byte.** 13 451 of the 44 discs' 56 490 samples have it set and every one of them is 192; the other 42 989 are 150. No disc mixes the two rules and there is not one exception. -- **The word count.** The directory's declared size is `words × 2 + header_len` on **56 430 of 56 430** payloads readable at all — 13 441 at 192, 42 989 at 150. The 60 that fail it are the damaged ones, and every one of those also fails an identity test below. +- **The type byte.** 15 352 of the 44 discs' 72 298 samples have it set and every one of them is 192; the other 56 946 are 150. No disc mixes the two rules and there is not one exception. +- **The word count.** The directory's declared size is `words × 2 + header_len` on **72 190 of 72 190** payloads readable at all — 15 314 at 192, 56 876 at 150. The 108 that fail it are the damaged ones, and every one of those also fails an identity test below. - **The bytes.** On `AKAI.S3000.Sound.Library.2`'s `NPF E0`, offsets 150–170 are zero and 171–191 are `0a ff ff 22 a8 00 aa ff ff 00 8c ff ff 00 aa ff ff 00 88 ff ff` — the same shape on every sample of the disc, which audio is not. Waveform starts at 192. -Nine of the 44 discs are affected: `AKAI.S3000.Sound.Library.1`–`7` (4 455, 3 086, 1 990, 1 010, 601, 168 and 218 samples), `East Connexion Piano` (730) and `AMG - Now CD-Rom for (AKAI)` (1 193). +Nine of the 44 discs are affected: `AKAI.S3000.Sound.Library.1`–`7` (4 451, 3 083, 1 989, 1 010, 971, 1 123 and 764 samples written), `East Connexion Piano` (730) and `AMG - Now CD-Rom for (AKAI)` (1 193). `Library.5`, `.6` and `.7` are much larger figures than D19 measured, because three quarters of their partitions were unread until the displaced ones were recovered. -**Getting this wrong does not fail, which is why it lasted four deliverables.** Reading a 192 at 150 yields the right frame count, a WAV that opens, and a length within 0.1 % — with 42 bytes of header in place of the attack (a burst of roughly ±20 000 lasting 0.24 ms, an audible click), the last 21 frames of the sound gone, and every loop point 21 frames out. An earlier note here said S3000 discs *may* use a 192-byte variant and advised branching on the id and valid bytes. Both halves were wrong: the variant is not conditional, and those two bytes do not carry the answer — `0x80` appears on 42 989 samples at 150 and 13 410 at 192, and the id is `3` on both. See [ADR-0027](../adr/0027-a-payload-must-be-the-file-its-entry-placed.md). +**Getting this wrong does not fail, which is why it lasted four deliverables.** Reading a 192 at 150 yields the right frame count, a WAV that opens, and a length within 0.1 % — with 42 bytes of header in place of the attack (a burst of roughly ±20 000 lasting 0.24 ms, an audible click), the last 21 frames of the sound gone, and every loop point 21 frames out. An earlier note here said S3000 discs *may* use a 192-byte variant and advised branching on the id and valid bytes. Both halves were wrong: the variant is not conditional, and those two bytes do not carry the answer — `0x80` appears on 56 876 samples at 150 and 15 314 at 192, and the id is `3` on both. See [ADR-0027](../adr/0027-a-payload-must-be-the-file-its-entry-placed.md). ### The valid byte is a flag, not a value @@ -256,33 +273,25 @@ Nine of the 44 discs are affected: `AKAI.S3000.Sound.Library.1`–`7` (4 455, 3 Every sample payload restates the file's id, its valid flag and its name, and the directory entry states the name and the size independently. Where they disagree, the payload is not the file the entry placed and it is refused rather than written under that entry's name — a WAV that opens, plays, and is somebody else's audio is the worst failure this format offers ([ADR-0027](../adr/0027-a-payload-must-be-the-file-its-entry-placed.md), [issue #23](https://github.com/bmxcode/samplerdisc/issues/23)). -**65 of the 56 490 samples disagree**, on ten discs; the other 34 have none. What each test catches: - -| Test | Fires | Fires alone | -|---|---:|---:| -| id is not `3` | 61 | 1 | -| valid byte has no `0x80` | 60 | 0 | -| name is not the entry's | 60 | **0** | -| rate outside 4 000–50 000 | 58 | 4 | - -**The name comparison has no unique catch anywhere in the collection.** Every payload whose name disagrees also has a wrong id and a cleared valid flag, because on these images the displacement lands mid-audio and mid-audio does not look like a header. It is kept anyway: the other three ask whether the payload is *a* sample, and only this one asks whether it is *this* sample. +**104 of the 72 298 samples disagree**, on nine discs; the other 35 have none. The name test's unique catch is still **zero** — every payload whose name disagrees also has a wrong id and a cleared valid flag, because a displacement lands mid-audio and mid-audio does not look like a header. It is kept anyway: the other three ask whether the payload is *a* sample, and only this one asks whether it is *this* sample. Where they are: -| Disc | Mismatches | Where | Partitions declared / present | +| Disc | Mismatches | Where | Partitions declared / read | |---|---:|---|---| +| `AKAI.S3000.Sound.Library.5` | 30 | `HIT NOISE` last 17 of 20, `SURDO` last 7 of 13, `BELL TREE` last 6 of 13 | 9 / 6 | +| `Kickin' Lunatic Beats 2 CD1` | 24 | `09-TRACK 37` last 15 of 26, `13-TRACK 06` last 9 of 20 | 11 / 8 | | `Best Service - Alpha Dance II` | 21 | `AC.DRUMLOOPS`, last 21 of 22 | **6 / 6** | | `Best Service - Alpha Dance I` | 15 | `ATTACK BANK2`, last 15 of 18 | 5 / 4 | -| `Kickin' Lunatic Beats 2 CD1` | 9 | `13-TRACK 06`, last 9 of 20 | 11 / 1 | -| `AKAI.S3000.Sound.Library.5` | 7 | `SURDO`, last 7 of 13 | 9 / 3 | -| `AKAI.S3000.Sound.Library.1` | 4 | `3084 B.BEAT6`, last 3 of 8; one rate | 13 / 13 | -| `AMG - Global Trance Mission 2` | 3 | `AMBIENT PAD2`, last 3 of 6 | 9 / 4 | -| `AKAI.S3000.Sound.Library.2` | 3 | three isolated rate bytes — 0, 519, 519 | 13 / 13 | +| `AMG - Global Trance Mission 2` | 8 | `SYNTH 10` last 5 of 6, `AMBIENT PAD2` last 3 of 6 | 9 / 7 | +| `AKAI.S3000.Sound.Library.1` | 3 | `3084 B.BEAT6`, last 3 of 8 | 13 / 13 | | `AKAI.S3000.Sound.Library.3` | 1 | `VOLUME 001`, its only file | 13 / 13 | -| `Audio Factory - Classical Wild Takes` | 1 | `VOLUME 002`, last of 2 | 11 / 6 | +| `Audio Factory - Classical Wild Takes` | 1 | `VOLUME 002`, last of 2 | 11 / 10 | | `AMG - Loop Soup` | 1 | `SOUP 101-103`, entry 27 of 39 | 9 / 9 | -**60 of the 65 are a run to the end of one volume**, which is what a lost run of blocks looks like from inside a directory. The remaining five are single files: four with a corrupt rate byte and an otherwise perfect header — `EG 2MUTE` at 0 Hz, `M.VOICE A1` and `SYN 1` at 519, `HOUSE BASS` at 1280 — and `Loop Soup`'s one directory record whose start block lands mid-sample. Those four *are* the files their entries placed, with one field unusable, which is a different fault from the other 61 and is counted apart. +Four more samples are refused for a corrupt rate byte with an otherwise perfect header — `EG 2MUTE` at 0 Hz, `M.VOICE A1` and `SYN 1` at 519, `HOUSE BASS` at 1280. Those *are* the files their entries placed, with one field unusable, and are counted apart. + +**103 of the 104 are a run to the end of one volume**, which is what a lost run of blocks looks like from inside a directory; the exception is `Loop Soup`'s one directory record whose start block lands mid-sample. And the run is a *displacement*, now measured rather than inferred: **103 of the 104 refused payloads are sitting intact, carrying their entry's own name, a whole number of container blocks earlier** — one block back for `Alpha Dance II`'s 21, 134 for `Library.3`'s one, one to four for the rest. Recovering them is [issue #35](https://github.com/bmxcode/samplerdisc/issues/35) and is not done: a partition has a declared position to search back from and a file has only the chain its allocation map states ([ADR-0028](../adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)). **A tail run does not require a missing partition.** `Alpha Dance II` declares six partitions and holds all six; `Library.1` and `Library.3` likewise. Their damage is a run of blocks lost *inside* a partition, so no header goes missing and the table's declared-against-present arithmetic sees nothing. Declared equalling present is not a clean bill of health. diff --git a/src/samplerdisc/batch.py b/src/samplerdisc/batch.py index 649a096..da814ff 100644 --- a/src/samplerdisc/batch.py +++ b/src/samplerdisc/batch.py @@ -34,6 +34,13 @@ class DiscReport: container: str | None = None filesystem: str | None = None origin: int | None = None + #: The backend's one line on how the disc is divided, verbatim -- the same + #: line ``list`` prints, or None where the filesystem has no structure + #: above the volume. It is here because it is the only place a run over a + #: collection records that an image is **short of the disc it was made + #: from**: its samples extract and verify, and the disc has more on it than + #: the file does (ADR-0028). + layout: str | None = None volumes: list[dict[str, Any]] = field(default_factory=list) samples: int = 0 #: Samples that were stereo on the disc, of ``samples``. Counted apart @@ -100,6 +107,8 @@ def convert_disc( return report report.filesystem = origin.backend.name report.origin = origin.offset + describe = getattr(origin.backend, "layout", None) + report.layout = describe(image, origin.offset) if describe is not None else None out_dir = os.path.join(out_root, safe_name(os.path.splitext(os.path.basename(path))[0])) # Keyed by partition *and* name: nearly every partition of an AKAI diff --git a/src/samplerdisc/cli.py b/src/samplerdisc/cli.py index d3627a6..a7c1dd5 100644 --- a/src/samplerdisc/cli.py +++ b/src/samplerdisc/cli.py @@ -49,7 +49,17 @@ def cmd_list(args: argparse.Namespace) -> int: if volume.partition and volume.partition != current: current = volume.partition partitions += 1 - print(f"\npartition {volume.partition}") + # Where the partition was not where the disc's table put it, + # say so here rather than only in the summary line: these + # volumes read perfectly and are still evidence that the image + # is short of the disc (ADR-0028). + short = ( + f" ({volume.displaced} bytes before its declared position " + f"-- this image is short of the disc)" + if volume.displaced + else "" + ) + print(f"\npartition {volume.partition}{short}") # Volumes sit under their partition where there is one, because # their names repeat across partitions (ADR-0023). indent = " " if volume.partition else "" diff --git a/src/samplerdisc/container/base.py b/src/samplerdisc/container/base.py index 1431a9f..ca5325e 100644 --- a/src/samplerdisc/container/base.py +++ b/src/samplerdisc/container/base.py @@ -41,6 +41,24 @@ def read(self, offset: int, length: int) -> bytes: def sectors(self) -> int: return self.size // SECTOR_SIZE + @property + def granularity(self) -> int: + """Cooked bytes per unit this container stores the disc in. + + The unit an *incomplete* image is missing whole numbers of. A container + that holds the sectors literally can only be short by sectors, so the + default is one; a container that packs the disc into larger blocks -- + compressed MDX -- loses a whole block at a time, and every byte after + the gap moves forward by that much. + + This is a fact about the container and nothing else: it says what the + file is made of, not what is inside it. A filesystem that finds a + structure displaced from where the disc's own table puts it needs the + step to search in, and the step is a quantity of this layer -- which is + precisely why the filesystem may not name it (ADR-0003, ADR-0028). + """ + return SECTOR_SIZE + @property def origin(self) -> int: """Cooked-stream offset at which the data track begins.""" diff --git a/src/samplerdisc/container/mdx.py b/src/samplerdisc/container/mdx.py index 1abb87b..9e6886e 100644 --- a/src/samplerdisc/container/mdx.py +++ b/src/samplerdisc/container/mdx.py @@ -206,6 +206,23 @@ def _build_index(self) -> list[Block]: def size(self) -> int: return self._size + @property + def granularity(self) -> int: + """One block of the chain, in cooked bytes -- 32 768 on most images. + + The chain carries no index, so nothing in the file says which block is + which: a rip that lost one produces a file that decodes perfectly and + is short of the disc by exactly this much, with everything after the + gap moved forward. That is what makes the number worth publishing + rather than keeping to this module. + + Measured, never assumed. It is ``block_size`` scaled from the stored + stride to the cooked one, so an image holding 2144-byte sectors with + subchannel reports the 30 720 cooked bytes its 32 160-byte block + carries, not 32 160. See docs/formats/mdx.md. + """ + return self.block_size // self.stride * SECTOR_SIZE + @property def compressed_blocks(self) -> int: return sum(1 for block in self.blocks if not block.stored) diff --git a/src/samplerdisc/fs/akai.py b/src/samplerdisc/fs/akai.py index 552b33c..765dec2 100644 --- a/src/samplerdisc/fs/akai.py +++ b/src/samplerdisc/fs/akai.py @@ -37,9 +37,10 @@ VOLUME_START_OFFSET = 14 #: Volume type byte: which sampler owns the volume, or 0 for one the sampler -#: will not load. Counted across 44 discs: 338 type 1, 91 type 7, 9 type 3, -#: 10 type 0. It is NOT an allocation flag -- four type-0 volumes carry 63 -#: files between them, which is why the allocation map below does that job. +#: will not load. Counted over the 44 discs' first partitions: 338 type 1, 91 +#: type 7, 9 type 3, 10 type 0; over all 314 partitions read, 1733/711/107/35. +#: It is NOT an allocation flag -- four type-0 volumes carry 63 files between +#: them, which is why the allocation map below does that job. VOLUME_TYPE_INACTIVE = 0 VOLUME_TYPES = {0: "inactive", 1: "S1000", 3: "S3000", 7: "CD3000"} @@ -88,10 +89,12 @@ #: after the volume directory's hundred slots. This is the disc's own record of #: what each block holds, and it is what tells a live volume from a slot that #: was formatted and never used -- a question the volume entry itself cannot -#: answer. Verified across all 44 AKAI discs: every one of 14 607 files has a -#: chain of exactly ceil(size / BLOCK_SIZE) blocks, counting the files of the -#: five deleted volumes out -- their blocks went back to the free list, so -#: their chains no longer describe the audio still sitting in them. +#: answer. Verified across all 44 AKAI discs: 85 338 of 85 355 files have a +#: chain of exactly ceil(size / BLOCK_SIZE) blocks -- 14 607 of 14 607 in the +#: first partitions, and 17 `MULTI FILE` entries on one disc are the only +#: exception anywhere. That counts the files of the five deleted volumes out: +#: their blocks went back to the free list, so their chains no longer describe +#: the audio still sitting in them. #: See docs/formats/akai-fs.md and ADR-0022. FAT_OFFSET = VOLUME_DIR_OFFSET + _MAX_VOLUMES * VOLUME_ENTRY_LEN @@ -120,19 +123,23 @@ #: See docs/formats/akai-fs.md and ADR-0023. PARTITION_TABLE_OFFSET = 0x4500 -#: 196 bytes of constant at 0x02, byte-identical across all 275 partitions of -#: the 44 discs: 3333 x i as u16 LE, i = 0..97, wrapping. Nothing is known to +#: 196 bytes of constant at 0x02, byte-identical across all 314 partitions read +#: on the 44 discs: 3333 x i as u16 LE, i = 0..97, wrapping. Nothing is known to #: read it and nothing on any disc varies with it, but a partition header #: carries it, which is what confirms a partition is where the table says. #: -#: **It must never be scanned for.** As 16-bit PCM it is a rising sawtooth, and -#: sample data reproduces it: 374 blocks of one disc's free space match, every -#: one in a block the allocation map calls free (ADR-0023). +#: **It must never be scanned for**, and the reason is stronger than it looks: +#: the free blocks of these discs hold complete *copies* of a partition header +#: -- 148 byte-identical ones on `ProSamples vol.14`, 374 matches on `Global +#: Trance Mission 2`, every one in a block the allocation map calls free. No +#: byte test separates those from a real header, because they are real headers. +#: A position is placed by what the disc states and confirmed there, never +#: chosen because the bytes look right (ADR-0023, ADR-0028). HEADER_PATTERN_OFFSET = 0x02 HEADER_PATTERN = struct.pack("<98H", *(3333 * step & 0xFFFF for step in range(98))) #: The header restates its own size. The u16 at 0xC6 is the block count at 0x00 -#: plus this bias, and the u16 at 0xC8 is 47; both hold on all 275 partitions +#: plus this bias, and the u16 at 0xC8 is 47; both hold on all 314 partitions #: measured. Two fields of the header agreeing is what lets a candidate be #: confirmed by the disc rather than by the arithmetic that placed it. SIZE_ECHO_OFFSET = 0xC6 @@ -166,6 +173,12 @@ class Partition(NamedTuple): index: int offset: int blocks: int + #: Bytes between where the table puts this partition and where its header + #: actually is, 0 for a partition sitting where it was declared. Non-zero + #: only on an image short of the disc it was made from: the table places + #: the partition on the *disk*, and the image has lost whole container + #: blocks in front of it (ADR-0028). + displaced: int = 0 def partition_table(image: SectorImage, origin: int) -> list[int]: @@ -210,18 +223,74 @@ def partition_header(image: SectorImage, offset: int) -> int | None: return blocks +def displaced_header( + image: SectorImage, origin: int, declared: int, size: int, floor: int +) -> int | None: + """Where a declared partition's header is, when the image is short of it. + + Searches **backwards from the position the table gave this partition**, in + steps of the container's own storage unit, for a header restating the size + the table gave *this* partition -- and stops at ``floor``, the end of the + partition already accepted before it. Returns the offset or None. + + Every one of those four words is load-bearing, because a free search for + this header is what ADR-0023 refused and was right to refuse: the constant + at 0x02 is a sawtooth and sample data reproduces it. + + **Backwards from the declared position.** A short image has *lost* bytes, + so everything after a gap has moved towards the front and nothing has moved + away from it. Searching forward would be searching where the fault cannot + have put anything. + + **In the container's unit.** The displacement is a whole number of the + blocks the container stores, because that is what the rip dropped; the + filesystem does not know what that unit is and must not guess, so it asks + (ADR-0003). On the collection the answer is the same at 32 768, 8 192 and + 2 048 bytes, so this is not what makes the search safe -- it is what keeps + it honest about the fault it is repairing, and what bounds its cost. + + **Restating this partition's size.** The header confirms the position, as + it does at a declared one: the same three fields, plus the size the table + gave this index. Sizes repeat within a disk, which is why the anchor + matters -- see below. + + **Stopping at the floor.** A partition may not be found inside one already + being read. This is the whole safety argument and it does all the rejecting + on the ProSamples discs, whose missing partitions are genuinely absent: on + seven of them the nearest match is exactly one partition back, which is the + *previous* partition's header seen through a size that repeats, and on + `vol.12` and `vol.14` it is 70 and 21 blocks back, inside a partition's + audio, which is the sawtooth ADR-0023 warned about. Both are below the + floor and neither is accepted. It also makes partition 1 unmovable on every + disc: its declared position is 0, so its window is empty. + """ + step = image.granularity + if step <= 0: + return None + position = declared - step + while position >= floor: + if partition_header(image, origin + position) == size: + return position + position -= step + return None + + def partitions(image: SectorImage, origin: int) -> Iterator[Partition]: """Every partition of the disk that this image actually holds. - The table says where each one begins; the header there confirms it. A - declared position the image has no header at is **skipped rather than - searched for**: on the discs where that happens the image is short of the - disc it was made from, and the header it is missing turns up displaced by a - whole number of the container's own 32 KB blocks -- a fault of the rip, not - a filesystem to go hunting through (ADR-0022, ADR-0023, issue #17). - - Because the table gives absolute positions, one missing header costs its - own partition and no other: the walk carries on to the next declared start. + The table says where each one begins and the header there confirms it. A + declared position with no header is not the end of the story on an image + that is **short of the disc it was made from**: the rip lost whole blocks + of the container, so everything past a gap sits that much nearer the front, + and the partition is there to be found rather than absent. Searching for it + is ``displaced_header`` above, and what makes that search a confirmation + rather than a scan is written there (ADR-0028). + + Because the table gives absolute positions, one partition that cannot be + found costs its own and no other: the walk carries on to the next declared + start. Accepted partitions never overlap -- the search stops at the end of + the last one accepted, and a recovered partition always ends before its + declared end, so it cannot reach into a later one either. """ sizes = partition_table(image, origin) if not sizes: @@ -233,14 +302,20 @@ def partitions(image: SectorImage, origin: int) -> Iterator[Partition]: yield Partition(1, 0, blocks) return at = 0 + #: Bytes past the end of the last partition accepted. Nothing may be found + #: below it, and it is the far end of every search. + floor = 0 for index, size in enumerate(sizes, start=1): - offset = at * BLOCK_SIZE + declared = at * BLOCK_SIZE at += size - if origin + offset >= image.size: - continue - if partition_header(image, origin + offset) != size: - continue - yield Partition(index, offset, size) + offset = declared + if partition_header(image, origin + declared) != size: + found = displaced_header(image, origin, declared, size, floor) + if found is None: + continue + offset = found + floor = offset + size * BLOCK_SIZE + yield Partition(index, offset, size, declared - offset) def decode_name(raw: bytes) -> str: @@ -271,7 +346,7 @@ def allocation_map(image: SectorImage, offset: int) -> list[int]: from one, instead of a map invented out of whatever follows the header. What makes a count sane is the header restating it: the u16 at 0xC6 is the - block count plus a fixed bias on all 275 partitions measured, so a count + block count plus a fixed bias on all 314 partitions read, so a count with no echo behind it is not a partition's and gets no map. That is a firmer test than the image's length, which was the previous one and refused the map to a partition the image merely *ends inside* -- leaving two volumes @@ -458,6 +533,7 @@ def _volumes(self, image: SectorImage, offset: int, partition: Partition) -> Ite start_block=start, origin=partition.offset, partition=partition.index, + displaced=partition.displaced, ) volume.files = list(self._files(image, at, partition.offset, start, max_block)) if not volume.files: @@ -531,18 +607,31 @@ def layout(self, image: SectorImage, offset: int) -> str: """One line on how the disk is divided, for ``list``. Declared against present is the interesting pair: `Kickin' Lunatic - Beats 2 CD1` declares eleven partitions and the image holds one, and + Beats 2 CD1` declares eleven partitions and the image holds eight, and that gap is the image being short of the disc rather than the disc being small (issue #17). Saying it is the difference between a fact and an absence nobody sees. + + A displaced count says the same thing from the other side. Where it is + non-zero the image has lost blocks and the partitions after each gap + were found before the table puts them, so *this image is not the disc* + -- which is worth a user knowing before they wonder why a volume is + missing (ADR-0028). """ declared = partition_table(image, offset) - present = sum(1 for _ in partitions(image, offset)) + found = list(partitions(image, offset)) + displaced = sum(1 for part in found if part.displaced) if not declared: return "no partition table -- reading the partition at the origin" if len(declared) == 1: return "1 partition" - return f"{len(declared)} partitions declared, {present} present in this image" + line = f"{len(declared)} partitions declared, {len(found)} present in this image" + if displaced: + line += ( + f" ({displaced} of them displaced -- this image is short of the disc " + f"it was made from)" + ) + return line def parse_sample(self, entry: File, payload: bytes): """Parse one sample, telling the parser what the directory declared. @@ -561,8 +650,8 @@ def parse_sample(self, entry: File, payload: bytes): ``.s1s``. S3000 samples put 192 bytes in front of the audio and S1000 ones 150; reading a 192 at 150 puts 42 bytes of header into the WAV as PCM, drops the last 21 frames and leaves every loop point 21 frames - out. That was happening to 13 451 of the collection's 56 490 AKAI - samples. See docs/formats/akai-fs.md and ADR-0027. + out. That was happening to 13 451 of the 56 490 AKAI samples the + collection read before D19. See docs/formats/akai-fs.md and ADR-0027. Imported here rather than at module scope: ``sample.akai`` imports the charset and the name helpers from this module, so a top-level import diff --git a/src/samplerdisc/fs/base.py b/src/samplerdisc/fs/base.py index e426ea6..1407a25 100644 --- a/src/samplerdisc/fs/base.py +++ b/src/samplerdisc/fs/base.py @@ -71,6 +71,17 @@ class Volume: #: AKAI disc's partitions -- nearly every one has a ``VOLUME 001`` -- so #: this is what keeps two of them apart in a listing and on disk. partition: int = 0 + #: Bytes between where the disc's own bookkeeping puts this volume's + #: partition and where it was found, 0 -- and meaningless -- where the + #: filesystem has no partitions. + #: + #: Non-zero says the image is short of the disc it was made from: whole + #: units of the container are missing in front of this volume, so it and + #: everything after it sit nearer the front than the disc's table declares. + #: The audio is intact and internally consistent, and *the image is not the + #: disc* -- which a listing should say rather than leave to be inferred + #: from a partition count (ADR-0028). + displaced: int = 0 #: Why this volume has no files, when that is expected rather than wrong. #: A volume with no files and no note is the signature of a probe that #: matched something it should not have (ADR-0012), so the two cases must diff --git a/src/samplerdisc/sample/akai.py b/src/samplerdisc/sample/akai.py index 239e1f7..f9e8688 100644 --- a/src/samplerdisc/sample/akai.py +++ b/src/samplerdisc/sample/akai.py @@ -17,13 +17,13 @@ #: Two header lengths, and **which one applies is declared, not sniffed**: the #: S3000 family writes 192 bytes and the S1000 family 150. The directory's type #: byte carries the generation in its high bit -- the same bit that already -#: names a kept original `.s3s` rather than `.s1s` -- and it splits the 56 490 -#: samples of the 44 discs perfectly, 13 451 at 192 and 42 989 at 150, with no +#: names a kept original `.s3s` rather than `.s1s` -- and it splits the 72 298 +#: samples of the 44 discs perfectly, 15 352 at 192 and 56 946 at 150, with no #: disc mixing the two rules. #: #: The payload confirms it from the other side: the directory's declared size -#: is ``words * 2 + header_len`` on **56 430 of 56 430** readable payloads, and -#: the 60 that fail that identity are the damaged ones. Placed by one +#: is ``words * 2 + header_len`` on **72 190 of 72 190** readable payloads, and +#: the 108 that fail that identity are the damaged ones. Placed by one #: structure, confirmed by another, the shape ADR-0020, ADR-0021 and ADR-0023 #: already use. See docs/formats/akai-fs.md and ADR-0027. HEADER_LEN_S1000 = 150 @@ -112,21 +112,23 @@ def _check_identity(payload: bytes, declared_name: str | None) -> str: is true of a mid-PCM payload and of a wrong-but-valid one alike and they are not the same news. - **What the name test is for.** Across the 44 AKAI discs it fires 60 times + **What the name test is for.** Across the 44 AKAI discs it fires 104 times and never once on its own: every payload whose name disagrees also has a wrong id and a cleared valid flag, because on these images the displacement - lands mid-audio rather than on another header. It is kept regardless. The - other three ask whether the payload is *a* sample; only this one asks - whether it is *this* sample, which is the failure issue #23 named and the - one a short image's recovered partitions would raise (#25). It is also what - makes the valid flag safe to read as a bit rather than as a byte. - See ADR-0027. + lands mid-audio rather than on another header. D20 was expected to change + that -- recovering the partitions eight short images had displaced added + 15 808 samples, all of them from exactly the kind of image this test was + kept for, and produced no unique catch either (#25). It is kept regardless. + The other three ask whether the payload is *a* sample; only this one asks + whether it is *this* sample, which is the failure issue #23 named. It is + also what makes the valid flag safe to read as a bit rather than as a byte. + See ADR-0027 and ADR-0028. """ if len(payload) <= OFF_RATE + 2: raise NotASample(f"payload is {len(payload)} bytes, too short for a sample header") # Every disagreement is collected rather than the first one raised. On the - # 61 real mismatches the id is wrong on all of them, so stopping at the + # 104 real mismatches the id is wrong on all of them, so stopping at the # first would report "id is 179" every time and never once mention the name # -- which is the test that says the payload belongs to a different file # rather than to none. What the fields disagree about *together* is also diff --git a/tests/fixtures.py b/tests/fixtures.py index c203ecf..5430d8f 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -322,6 +322,23 @@ def akai_disc(partitions, *, declared=None, flag: int = 0) -> bytes: return bytes(image) +def short_image(data: bytes, at: int, unit: int, count: int = 1) -> bytes: + """Drop ``count`` whole ``unit``-sized blocks starting at ``at``. + + What an incomplete rip produces, and the reason it is invisible: the + container's chain carries no index, so the file that comes out decodes + perfectly and is short of the disc. Everything past the gap sits + ``count * unit`` bytes nearer the front than the disc's own bookkeeping + puts it -- a partition header included, which is what makes the damage + findable at all (ADR-0028). + + ``at`` must be a multiple of ``unit``: a rip loses whole blocks of whatever + the container stores, never part of one. + """ + assert at % unit == 0, f"a container loses whole blocks; {at} is not a multiple of {unit}" + return data[:at] + data[at + count * unit :] + + def akai_sample( name: str, rate: int = 44100, diff --git a/tests/test_akai_fs.py b/tests/test_akai_fs.py index de6d530..f8ea816 100644 --- a/tests/test_akai_fs.py +++ b/tests/test_akai_fs.py @@ -532,11 +532,10 @@ def test_block_numbers_are_relative_to_their_own_partition(tmp_path): def test_a_declared_partition_the_image_lacks_is_skipped_and_the_walk_goes_on(tmp_path): """An image short of the disk it was made from loses a partition, not the rest. - The table gives absolute positions, so a missing header costs its own - partition and nothing after it. Where the header is missing the walk stops - rather than searching: on the discs that do this the header turns up - displaced by a whole number of the container's blocks, which is the rip - being incomplete (ADR-0023, issue #17). + The table gives absolute positions, so a partition that cannot be found + costs its own and nothing after it. Here there is nothing to find -- the + partition's blocks are zeros, and the backward search reaches no header at + any position it may look at (ADR-0023, ADR-0028, issue #17). """ blank = bytes(512 * 8192) data = fixtures.akai_disc( @@ -606,3 +605,213 @@ def test_a_disc_with_no_partition_table_reads_the_partition_at_the_origin(tmp_pa volumes = list(BACKEND.volumes(image, 0)) assert [v.partition for v in volumes] == [1, 1] assert BACKEND.layout(image, 0) == "no partition table -- reading the partition at the origin" + + +# --- an image short of the disc it was made from (ADR-0028) -------------- + +#: Blocks per partition in the short-image fixtures below. Small enough that a +#: three-partition disc is a few megabytes and large enough that the partition +#: table at 0x4500 fits inside the first one. +_SHORT_BLOCKS = 192 + +#: What the MDX container stores a disc in, and therefore what a rip of one +#: loses. The fixtures delete whole units of this and nothing else. +_UNIT = 32768 + + +def _short_disc(count: int, gap_at_block: int, units: int = 1) -> bytes: + """A disk of ``count`` partitions with a run of container blocks missing. + + ``gap_at_block`` is where the gap starts, in AKAI blocks from the origin. + Everything after it moves ``units * 32768`` bytes towards the front, the + table still declaring the positions the *disk* had. + """ + partitions = [ + fixtures.akai_partition( + [ + ( + f"VOLUME 00{index}", + [(f"KICK {index}", 0x73, 278, fixtures.akai_sample(f"KICK {index}"))], + ) + ], + blocks_total=_SHORT_BLOCKS, + ) + for index in range(1, count + 1) + ] + return fixtures.short_image(fixtures.akai_disc(partitions), gap_at_block * 8192, _UNIT, units) + + +def test_a_partition_the_rip_moved_is_found_where_the_lost_blocks_put_it(tmp_path): + """The deliverable, in one fixture (ADR-0028). + + A gap inside partition 1 pulls every partition after it towards the front + by exactly what was lost. Partition 2's header then sits *inside* partition + 1's declared extent and is refused -- that is `Alpha Dance I` and + `Kickin' Lunatic Beats 2 CD2`'s partition 2, and refusing it is what keeps + the 275 partitions read before this from moving. Partition 3 is clear of + everything already accepted and is read, with its volume and its sample + intact, because inside a displaced partition the directory and the audio + moved together. + """ + from samplerdisc.fs.akai import partitions + + image = image_of(tmp_path, _short_disc(3, gap_at_block=100)) + found = [(p.index, p.offset, p.displaced) for p in partitions(image, 0)] + assert found == [ + (1, 0, 0), + (3, 2 * _SHORT_BLOCKS * 8192 - _UNIT, _UNIT), + ] + volumes = list(BACKEND.volumes(image, 0)) + assert [(v.partition, v.name, v.displaced) for v in volumes] == [ + (1, "VOLUME 001", 0), + (3, "VOLUME 003", _UNIT), + ] + # The recovered partition's audio is the file its directory placed: the + # payload header carries the entry's own name, which is what says the + # partition was found and not merely something header-shaped (ADR-0027). + entry = volumes[1].files[0] + payload = BACKEND.read_file(image, 0, entry) + assert decode_name(payload[3:15]) == entry.name == "KICK 3" + assert BACKEND.layout(image, 0) == ( + "3 partitions declared, 2 present in this image (1 of them displaced -- " + "this image is short of the disc it was made from)" + ) + + +def test_a_partition_is_never_found_inside_one_already_being_read(tmp_path): + """The rule that does all the rejecting, and the whole safety argument. + + Partition 2's header is really there and really is partition 2's -- and + taking it would mean reading two partitions over the same bytes, so it is + refused. The search stops at the end of the last partition accepted, which + is why this is a bound rather than a filter: nothing below the floor is + ever examined. + + On the shelf this is what refuses the ProSamples discs, whose missing + partitions are genuinely absent and whose nearest match is a stale header + sitting in a partition already read (ADR-0028). + """ + from samplerdisc.fs.akai import displaced_header, partition_header, partitions + + image = image_of(tmp_path, _short_disc(2, gap_at_block=100)) + at = _SHORT_BLOCKS * 8192 - _UNIT + # The header is there, and reading it is not the question. + assert partition_header(image, at) == _SHORT_BLOCKS + # It is inside partition 1, so the search may not reach it ... + assert ( + displaced_header(image, 0, _SHORT_BLOCKS * 8192, _SHORT_BLOCKS, _SHORT_BLOCKS * 8192) + is None + ) + # ... and with no floor in the way, the same call finds it. The rule is the + # floor and nothing else about the bytes. + assert displaced_header(image, 0, _SHORT_BLOCKS * 8192, _SHORT_BLOCKS, 0) == at + assert [(p.index, p.displaced) for p in partitions(image, 0)] == [(1, 0)] + assert BACKEND.layout(image, 0) == "2 partitions declared, 1 present in this image" + + +def test_partition_one_cannot_move_on_any_disc(tmp_path): + """Its declared position is 0, so its window is empty by construction. + + Partition 1 is where the table itself lives and it is the control on every + count this project has pinned since #22. Nothing may search in front of it, + and there is nowhere in front of it to search. + """ + from samplerdisc.fs.akai import displaced_header + + image = image_of(tmp_path, _short_disc(3, gap_at_block=100)) + assert displaced_header(image, 0, 0, _SHORT_BLOCKS, 0) is None + + +def test_the_search_steps_in_the_unit_the_container_states(tmp_path): + """The step is the container's word, not the filesystem's guess (ADR-0003). + + A displacement of one 32 KB block is invisible to a search stepping in + 64 KB ones. The filesystem has no way to know which is right -- the number + belongs to the layer that stores the disc -- so it asks, and this is that + seam under test rather than the value 32768. + """ + from samplerdisc.fs.akai import displaced_header + + class _Coarser: + """The same image, telling the filesystem a different storage unit.""" + + def __init__(self, inner, granularity): + self._inner, self.granularity = inner, granularity + + size = property(lambda self: self._inner.size) + + def read(self, offset, length): + return self._inner.read(offset, length) + + image = image_of(tmp_path, _short_disc(3, gap_at_block=100)) + declared = 2 * _SHORT_BLOCKS * 8192 + floor = _SHORT_BLOCKS * 8192 + assert displaced_header(image, 0, declared, _SHORT_BLOCKS, floor) == declared - _UNIT + assert displaced_header(_Coarser(image, 2 * _UNIT), 0, declared, _SHORT_BLOCKS, floor) is None + # A container with nothing to say gets no search at all rather than a + # default step chosen here. + assert displaced_header(_Coarser(image, 0), 0, declared, _SHORT_BLOCKS, floor) is None + + +def test_a_partition_never_written_is_not_recovered_from_another_ones_header(tmp_path): + """The ProSamples case: absent is not displaced, and the two must not merge. + + These CDs carry the front of a larger disk -- `vol.54` declares nine + partitions of a 63 488-block disk on a CD of 30 720 blocks -- so the + partitions they are missing were never written and there is nothing to + find. What makes that dangerous is that a *size* repeats: with no floor the + search walks back over the unwritten region and reaches partition 1's + header, which restates the same block count and would be read a second time + under partition 3's index. The second assertion is that failure, and the + first is the rule refusing it (ADR-0028). + """ + from samplerdisc.fs.akai import displaced_header, partitions + + blank = bytes(_SHORT_BLOCKS * 8192) + data = fixtures.akai_disc( + [ + fixtures.akai_partition( + [("VOLUME 001", [("KICK 1", 0x73, 278, fixtures.akai_sample("KICK 1"))])], + blocks_total=_SHORT_BLOCKS, + ), + blank, + blank, + ] + ) + image = image_of(tmp_path, data) + declared = 2 * _SHORT_BLOCKS * 8192 + assert displaced_header(image, 0, declared, _SHORT_BLOCKS, _SHORT_BLOCKS * 8192) is None + assert displaced_header(image, 0, declared, _SHORT_BLOCKS, 0) == 0 + assert [(p.index, p.displaced) for p in partitions(image, 0)] == [(1, 0)] + assert BACKEND.layout(image, 0) == "3 partitions declared, 1 present in this image" + + +def test_a_displaced_partition_must_restate_the_size_the_table_gave_it(tmp_path): + """The header confirms a found position exactly as it does a declared one. + + Three fields and the block count the table gave *this* index. A header that + is one block out is not this partition's, and is refused rather than read + with the size the table wanted (ADR-0023, ADR-0028). + """ + import struct + + from samplerdisc.fs.akai import ( + PARTITION_BLOCKS_OFFSET, + SIZE_ECHO_BIAS, + SIZE_ECHO_OFFSET, + displaced_header, + partitions, + ) + + declared = 2 * _SHORT_BLOCKS * 8192 + floor = _SHORT_BLOCKS * 8192 + data = bytearray(_short_disc(3, gap_at_block=100)) + at = declared - _UNIT + # Partition 3's header, restating one block less than the table gives it. + struct.pack_into(" set[int]: *(size for size, _, _, _, _, _ in _EMU3.values()), *(size for size, _, _, _, _, _ in _AKAI.values()), *(size for size, _, _, _, _ in _AKAI_PAYLOAD.values()), + *(size for size, *_ in _AKAI_SHORT.values()), + _AKAI_NOTHING_RECOVERED[1], } @@ -700,15 +702,28 @@ def test_protozoa_gives_each_bank_its_own_records() -> None: #: #: The two partition columns are the pin on #22, and they differ on purpose: #: `Kickin' Lunatic Beats 2 CD1` declares eleven partitions and the image holds -#: one, which is the image being short of the disc rather than the disc being -#: small (ADR-0023, issue #17). A present count that climbs to the declared one -#: would mean the walk had started accepting positions with no header at them. +#: eight, three of which no rule may recover -- the image is short of the disc +#: rather than the disc being small (ADR-0023, ADR-0028, issue #17). A present +#: count that climbs to the declared one would mean the walk had started +#: accepting positions with no header at them. +#: +#: Three of the nine are the discs a recovery must **not** touch, and they are +#: pinned here for that and nothing else. `Advanced Media Trax 3` is not short +#: -- nine declared, nine present -- and an earlier signature search cost it 22 +#: of its 94 volumes, so it is the falsifying case (ADR-0023). `ProSamples +#: vol.14` and `vol.54` are the second: their missing partitions were never +#: written, `vol.54` declaring nine partitions of a 63 488-block disk on a CD of +#: 30 720 blocks, and `vol.14`'s free space carries 148 byte-identical copies of +#: a partition header that a search must not take (ADR-0028). #: ``label: (size in bytes, volumes, files, noted, partitions declared, present)``. _AKAI = { "AKAI Advance Orchestra Upgrade 97 Vol.1": (545_720_320, 88, 2669, 4, 9, 9), "AMG - Loop Soup AKAI": (542_419_100, 60, 4689, 0, 9, 9), - "AMG - Kickin' Lunatic Beats 2 AKAI CD1": (378_443_564, 18, 669, 5, 11, 1), - "AMG - Kickin' Lunatic Beats 2 AKAI CD2": (371_768_845, 20, 1346, 0, 9, 1), + "AMG - Kickin' Lunatic Beats 2 AKAI CD1": (378_443_564, 139, 8392, 13, 11, 8), + "AMG - Kickin' Lunatic Beats 2 AKAI CD2": (371_768_845, 120, 7549, 0, 9, 8), + "Advanced Media Trax 3 - Modern Composer": (759_441_984, 94, 2938, 0, 9, 9), + "Best Service ProSamples vol.14 - World Grooves": (553_357_312, 15, 757, 0, 9, 5), + "Best Service ProSamples vol.54 - Techno 138 BPM": (251_658_240, 32, 1212, 0, 9, 4), "OMI Universe Of Sounds Vol.1 (Roland S-770,S-750)": (295_837_696, 69, 1653, 1, 5, 5), "Back in Time Records - Big Bang": (269_979_648, 137, 2689, 0, 9, 9), "Best Service ProSamples vol.01 - Hip Hop and R&B Drumloops": (314_882_048, 41, 492, 1, 9, 5), @@ -726,6 +741,9 @@ def test_protozoa_gives_each_bank_its_own_records() -> None: "AMG - Loop Soup AKAI": (7, 490, 0), "AMG - Kickin' Lunatic Beats 2 AKAI CD1": (18, 669, 5), "AMG - Kickin' Lunatic Beats 2 AKAI CD2": (20, 1346, 0), + "Advanced Media Trax 3 - Modern Composer": (4, 281, 0), + "Best Service ProSamples vol.14 - World Grooves": (4, 221, 0), + "Best Service ProSamples vol.54 - Techno 138 BPM": (10, 198, 0), "OMI Universe Of Sounds Vol.1 (Roland S-770,S-750)": (28, 900, 1), "Back in Time Records - Big Bang": (8, 380, 0), "Best Service ProSamples vol.01 - Hip Hop and R&B Drumloops": (7, 82, 0), @@ -762,6 +780,176 @@ def test_akai_discs_list_their_volumes_and_files(label: str) -> None: assert all(v.partition for v in volumes) +#: The eight images short of the disc they were made from, and what searching +#: back for their displaced partition headers recovers (ADR-0028, issue #25). +#: +#: The displacements are the finding and are pinned per partition, in AKAI +#: blocks: they are what says the rip lost whole 32 KB container blocks, and +#: they accumulate down the disc because it lost them in several places. Every +#: one is a multiple of 4 blocks -- 32 768 bytes, one MDX block -- and a +#: displacement that stopped being one would mean the search had begun stepping +#: in something that is not the container's unit. +#: +#: The last five columns are for the **displaced partitions alone**, not the +#: whole disc, which is the same reasoning as `_AKAI_FIRST_PARTITION` the other +#: way round: pinned apart, they say what recovery contributed rather than +#: leaving it to be subtracted. `refused` counts payloads that are not the file +#: their entry placed, and it is pinned as tightly as the rest -- 43 of the +#: 15 808 recovered samples, 99.7 % passing, is what says these partitions are +#: the disc's own and not something header-shaped (ADR-0027). +#: ``label: (size, declared, present, {index: displacement in blocks}, volumes, +#: files, samples, written, refused)``. +_AKAI_SHORT = { + "AMG - Kickin' Lunatic Beats 2 AKAI CD1": ( + 378_443_564, 11, 8, {3: 52, 4: 52, 5: 52, 6: 52, 7: 52, 8: 52, 10: 200}, + 121, 7723, 7308, 7293, 15, + ), + "AMG - Kickin' Lunatic Beats 2 AKAI CD2": ( + 371_768_845, 9, 8, {3: 4, 4: 4, 5: 4, 6: 4, 7: 4, 8: 4, 9: 4}, + 100, 6203, 5912, 5912, 0, + ), + "AKAI.S3000.Sound.Library.5": ( + 294_252_089, 9, 6, {5: 12, 7: 32, 8: 32}, 34, 473, 400, 377, 23, + ), + "AKAI.S3000.Sound.Library.6": ( + 320_291_524, 9, 8, {3: 68, 4: 68, 5: 68, 6: 68, 7: 68, 8: 68, 9: 68}, + 84, 1054, 955, 955, 0, + ), + "AKAI.S3000.Sound.Library.7": ( + 221_665_577, 11, 4, {3: 5508, 4: 2028, 5: 512}, 20, 660, 546, 546, 0, + ), + "Back In Time Rrcords - Elektra Vox AKAI": ( + 353_568_222, 13, 6, {3: 488, 5: 964, 7: 1844, 9: 2664, 11: 1928}, + 33, 661, 463, 463, 0, + ), + "AMG - Global Trance Mission 2 AKAI": ( + 392_438_329, 9, 7, {6: 8, 7: 8, 9: 32}, 22, 217, 144, 139, 5, + ), + "Audio Factory - Classical Wild Takes AKAI": ( + 226_074_906, 11, 10, {8: 16, 9: 16, 10: 16, 11: 16}, 18, 189, 80, 80, 0, + ), +} # fmt: skip + +#: The disc that recovers nothing, and the reason it is here rather than absent +#: from the table above. `Alpha Dance I` is short by one container block, and +#: its one missing partition's header sits four AKAI blocks inside partition 4, +#: which is already being read. Refusing it costs the disc everything a +#: recovery could have given it, and that is the conservative half of ADR-0028 +#: shown on the one disc where it is the whole answer. +_AKAI_NOTHING_RECOVERED = ("Best Service - Alpha Dance I AKAI", 193_592_710, 5, 4) + + +@pytest.mark.parametrize("label", sorted(_AKAI_SHORT)) +def test_a_short_akai_image_yields_the_partitions_the_lost_blocks_moved(label: str) -> None: + """The deliverable, per disc (ADR-0028, issue #25). + + Each of these images decodes cleanly and is missing whole 32 KB blocks of + the disc it was made from, so every partition after a gap sits that much + nearer the front than the disk's own table puts it. Searching back from the + declared position in the container's unit, for a header restating the size + the table gave *that* partition, and never reaching into a partition already + read, finds 39 of them across the eight. + + Four things are asserted together and each would fail differently. The + **displacement per partition** is the finding. The **volume and file counts** + are the yield. The **written and refused counts** are what says the audio is + the disc's own: a displaced partition's directory and its audio moved + together, so the payload check that condemns a misplaced file passes on + 99.7 % of what this recovers. And **every recovered partition's header block + is distinct** from every other partition's on the disc, which is what + separates a partition from the stale copies of a header that sit in these + discs' free space -- 148 byte-identical ones on `ProSamples vol.14`. + """ + from samplerdisc.fs.akai import BLOCK_SIZE, partition_table, partitions + from samplerdisc.sample import NotASample, PayloadMismatch + + size, declared, present, displacements, volumes, files, samples, written, refused = _AKAI_SHORT[ + label + ] + with open_image(_pinned_disc(label, size)) as image: + origin = find_origin(image) + assert origin is not None and origin.backend.name == "akai" + found = list(partitions(image, origin.offset)) + assert len(partition_table(image, origin.offset)) == declared + assert len(found) == present + assert {p.index: p.displaced // BLOCK_SIZE for p in found if p.displaced} == displacements + # Every displacement is a whole number of what the container stores. + assert all(p.displaced % image.granularity == 0 for p in found) + # Partition 1 is where the table lives, and it can never move: its + # declared position is 0, so there is nowhere in front of it to search. + assert found[0].index == 1 and found[0].displaced == 0 + + # No two partitions read may overlap, and none may repeat another's + # header block -- a stale copy of one would do both. + heads: dict[bytes, int] = {} + end = 0 + for part in found: + assert part.offset >= end, f"{label}: partition {part.index} overlaps the one before" + end = part.offset + part.blocks * BLOCK_SIZE + head = image.read(origin.offset + part.offset, BLOCK_SIZE) + assert head not in heads, ( + f"{label}: partition {part.index} has the same header block as {heads[head]}" + ) + heads[head] = part.index + + moved = {index for index, _ in displacements.items()} + seen = kept = mismatched = 0 + volumes_seen = files_seen = 0 + for volume in origin.backend.volumes(image, origin.offset): + if volume.partition not in moved: + continue + assert volume.displaced == displacements[volume.partition] * BLOCK_SIZE + volumes_seen += 1 + files_seen += len(volume.files) + for entry in volume.samples(): + seen += 1 + payload = origin.backend.read_file(image, origin.offset, entry) + try: + origin.backend.parse_sample(entry, payload) + except PayloadMismatch: + mismatched += 1 + except NotASample: + pass + else: + kept += 1 + assert (volumes_seen, files_seen, seen, kept, mismatched) == ( + volumes, + files, + samples, + written, + refused, + ) + + +def test_a_displaced_partition_that_would_overlap_a_present_one_is_refused() -> None: + """`Alpha Dance I`: the conservative half of the rule, costing a whole disc. + + Its partition 5 is displaced by four AKAI blocks -- one 32 KB container + block, the same gap as `Kickin' Lunatic Beats 2 CD2` -- and unlike CD2 there + is no later partition clear of the clash to recover. The header is really + there and really is partition 5's; reading it would mean two partitions over + the same bytes, so it stays unread and the disc gains nothing (ADR-0028). + """ + from samplerdisc.fs.akai import BLOCK_SIZE, displaced_header, partition_header, partitions + + label, size, declared, present = _AKAI_NOTHING_RECOVERED + with open_image(_pinned_disc(label, size)) as image: + origin = find_origin(image) + assert origin is not None and origin.backend.name == "akai" + found = list(partitions(image, origin.offset)) + assert len(found) == present + assert not any(p.displaced for p in found) + + last = found[-1] + assert last.index == declared - 1 + at = last.offset + last.blocks * BLOCK_SIZE + blocks = partition_header(image, origin.offset + at - 4 * BLOCK_SIZE) + # The header is there, four blocks inside the partition already read ... + assert blocks is not None + # ... and the search stops at that partition's end, so it is never seen. + assert displaced_header(image, origin.offset, at, blocks, at) is None + + @pytest.mark.parametrize("label", sorted(_AKAI_FIRST_PARTITION)) def test_akai_partition_one_holds_exactly_what_it_did_before(label: str) -> None: """#22 adds partitions; it must not move a single volume of the first one. @@ -933,7 +1121,7 @@ def test_akai_keeps_the_files_of_a_volume_the_allocation_map_calls_free() -> Non "East Connexion Piano": (277_092_352, 730, 730, 0, 0), "AMG - Now CD-Rom for (AKAI)": (521_322_496, 1193, 1193, 0, 0), "Best Service - Alpha Dance II AKAI": (309_865_547, 1740, 0, 21, 0), - "AMG - Kickin' Lunatic Beats 2 AKAI CD1": (378_443_564, 624, 0, 9, 0), + "AMG - Kickin' Lunatic Beats 2 AKAI CD1": (378_443_564, 7932, 0, 24, 0), "AMG - Loop Soup AKAI": (542_419_100, 3434, 0, 1, 0), "AKAI Advance Orchestra Upgrade 97 Vol.1": (545_720_320, 2236, 0, 0, 0), } @@ -951,7 +1139,7 @@ def test_akai_payloads_are_the_files_their_directory_entries_placed(label: str) high bit. Reading a 192 at 150 does not fail -- it writes a WAV that opens, holding 42 bytes of header as PCM at the front, 21 frames short at the end, with every loop point 21 frames out. That was happening to **13 451 of the - collection's 56 490** AKAI samples, on nine discs, and four of them are + 56 490** AKAI samples read before D20, on nine discs, and four of them are pinned here whole. **The identity.** 65 payloads across the 44 discs are not the file their @@ -989,7 +1177,7 @@ def test_akai_payloads_are_the_files_their_directory_entries_placed(label: str) # The generation bit chose the length; the payload's own word # count against the directory's declared size is what confirms # it, and the two are written by different structures. Across - # the collection this holds for 56 425 of 56 425 accepted + # the collection this holds for 72 190 of 72 190 accepted # payloads -- which is what makes the rule a finding rather # than a reading that happens to fit (ADR-0020, ADR-0027). (words,) = struct.unpack_from(" Date: Sat, 22 Aug 2026 15:34:57 +1000 Subject: [PATCH 2/2] D20: correct two stale sawtooth claims in the search docstring The header-shaped blocks in free space are complete stale partition headers, not audio reproducing the constant field's sawtooth. Corrected in the format doc and the ADR already; these two sentences were missed. Co-Authored-By: Claude Opus 5 --- src/samplerdisc/fs/akai.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/samplerdisc/fs/akai.py b/src/samplerdisc/fs/akai.py index 765dec2..5666393 100644 --- a/src/samplerdisc/fs/akai.py +++ b/src/samplerdisc/fs/akai.py @@ -234,8 +234,9 @@ def displaced_header( partition already accepted before it. Returns the offset or None. Every one of those four words is load-bearing, because a free search for - this header is what ADR-0023 refused and was right to refuse: the constant - at 0x02 is a sawtooth and sample data reproduces it. + this header is what ADR-0023 refused and was right to refuse: the free + blocks of these discs hold complete stale *copies* of a partition header, + which no byte test can tell from the real thing. **Backwards from the declared position.** A short image has *lost* bytes, so everything after a gap has moved towards the front and nothing has moved @@ -259,9 +260,9 @@ def displaced_header( on the ProSamples discs, whose missing partitions are genuinely absent: on seven of them the nearest match is exactly one partition back, which is the *previous* partition's header seen through a size that repeats, and on - `vol.12` and `vol.14` it is 70 and 21 blocks back, inside a partition's - audio, which is the sawtooth ADR-0023 warned about. Both are below the - floor and neither is accepted. It also makes partition 1 unmovable on every + `vol.12` and `vol.14` it is 70 and 21 blocks back, on one of those stale + copies sitting in a read partition's free space. Both are below the floor + and neither is examined. It also makes partition 1 unmovable on every disc: its declared position is 0, so its window is empty. """ step = image.granularity