D19: the S3000 header is 192 bytes, and an AKAI payload must be the file its entry placed

CHANGELOG.md

@@ -4,8 +4,26 @@ Notable changes to `samplerdisc`. Format-level findings live in [docs/formats/](
 
 ## Unreleased
 
+### Fixed
+
+- **S3000 AKAI samples had 42 bytes of header at the front of their audio.** The S3000 family writes a **192-byte** header where the S1000 family writes 150, and every sample was read at 150. That does not fail — the frame count comes out right, the WAV opens, the length is within 0.1 % — so what shipped was a file beginning with a burst of roughly ±20 000 lasting 0.24 ms in place of the attack, missing the last 21 frames of the sound, with every loop point 21 frames out of alignment. **13 451 of the collection's 56 490 AKAI samples**, on nine discs: `AKAI.S3000.Sound.Library.1`–`7` (4 455, 3 086, 1 990, 1 010, 601, 168 and 218), `East Connexion Piano` (730) and `AMG - Now CD-Rom for (AKAI)` (1 193). **Anyone who extracted those discs should do it again.** ([docs/formats/akai-fs.md](docs/formats/akai-fs.md), [ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md))
+
+  **Which length applies is declared, not sniffed.** The directory entry's type byte carries the generation in its high bit — the same bit that already named a kept original `.s3s` rather than `.s1s` — and it splits the 44 discs' samples perfectly, 13 451 at 192 and 42 989 at 150, with no disc mixing the two rules. The payload confirms it from the other side: the directory's declared size is `words × 2 + header_len` on **56 430 of 56 430** readable payloads. The format doc's old advice — that S3000 discs *may* use a 192-byte variant, and to branch on the id and valid bytes — was wrong twice: the variant is not conditional, and `0x80` appears on 42 989 samples at 150 and 13 410 at 192.
+
+  **The suite had no AKAI payload check and now does.** The E-mu and Roland paths have been asserting their audio against the disc for two deliverables; AKAI's was verified against the same wrong offset it was written from, which is how a 42-byte slip survived four releases in a green suite. Every accepted payload is now checked byte for byte against the disc from the chosen offset, and the word-count identity is asserted over every AKAI disc rather than the pinned ones.
+
+- **31 healthy S3000 samples were being thrown away as damage.** Their valid byte is `0x81` (29 on `Library.2`) or `0x9c` (2 on `Library.1`) rather than exactly `0x80`, with a correct id, a name matching their directory entry, a normal rate and a word count the declared size agrees with. `0x80` is a flag inside the byte and was being tested as the whole byte. The collection goes from 89 125 samples to **89 156**, AKAI from 56 394 to **56 425**, and AKAI stereo joins from 14 449 to **14 461** — the extra pairs completing because both halves are now present. ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md))
+
 ### Added
 
+- **An AKAI payload is checked against the directory entry that placed it.** The payload repeats the file's id, valid flag and name, and nothing compared them ([#23](https://github.com/bmxcode/samplerdisc/issues/23)). Where they disagree the file is refused rather than written under that entry's name, and the reason names **every** field that disagrees and the entry involved — `payload header carries id 224 not 3, a name that does not decode in the AKAI charset, placed by an entry named 'PADY CHORD06'` — instead of the old `payload does not start with an AKAI sample header`, which was true of a payload that is mid-audio and of one that is a perfectly good sample under the wrong name alike. Counted apart from damage in the summary and the manifest, as duplicates already were. ([docs/formats/akai-fs.md](docs/formats/akai-fs.md), [ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md))
+
+  **#23's bug did not reproduce, and saying so is the finding.** All 96 payloads that disagreed with their entry were already being refused — the nine named files on `Kickin' Lunatic Beats 2 CD1` included — and they were exactly the 96 skips the README already reported, not a disjoint set. Three and a half of the four proposed tests had been there since D3. What was missing was that the refusal said nothing useful, so nobody checked whether it existed. Ruling out the false-positive case the issue itself pointed at — the S3000 header — is what found the defect above.
+
+  **The name comparison ships with zero positives on real data**, and that is stated rather than buried. Across all 44 discs it fires 60 times and never once without the id and valid tests firing too, because on these images the displacement lands mid-audio and mid-audio does not look like a header. It is kept because it is the only test that asks whether a payload is *this* sample rather than *a* sample — the case [#25](https://github.com/bmxcode/samplerdisc/issues/25) will raise — and it is exercised synthetically alone.
+
+  **The 65 remaining refusals cluster, which is what says the check is right.** 60 of them are a run to the end of one volume, on ten discs; the other 34 AKAI discs have none. Five are single files — four with a corrupt rate byte and an otherwise perfect header, which are counted as damage rather than as mismatches because they *are* the files their entries placed. `Best Service - Alpha Dance II` is the one to note: it declares six partitions and holds all six, and still loses 21 of one volume's 22 samples, so a rip can drop a run of blocks inside a partition without the partition table registering anything.
+
 - **E-mu stereo samples come out as stereo.** The sample record's pointer block declares a channel count, and where it declares two the payload is a **block** split — all of the left channel, then all of the right. Read as one mono stream that is a file twice as long as the sound, with the right channel playing after the left instead of alongside it, and that is what this project wrote for **2 656 of the 14 738** E-mu samples: 28, 601, 592, 8, 279, 320 and 828 across the seven reference discs. `Miroslav Vitous String Ensembles` is all of it — 828 of 828. They are now interleaved into one stereo WAV each. ([docs/formats/emu3.md](docs/formats/emu3.md), [ADR-0026](docs/adr/0026-the-record-declares-the-channel-count.md), [#32](https://github.com/bmxcode/samplerdisc/issues/32))
 
   **A channel count is not enough on its own.** 2 721 records declare `start_R` half a payload on, and **65 of them close `end_L` somewhere else** — 19 on `protozoa`, 40 on `eiiix-1`, 6 on `eiiix-2`. Those 65 are not stereo: their halves correlate at 0.01 on fine structure and 0.01 at best lag, which is the score of two unrelated records, against 0.40 and 0.53 for the known-true stereo pairs on the same shelf. On `protozoa` six of them can be identified exactly: the first half of each is, byte for byte, the whole of a one-channel record of the same name in another bank, nothing matches the second half, and `end_L` closes 8 bytes past the halfway point rather than on it. Requiring the left block to close exactly where the right one opens is what separates them, and those 65 stay mono.

README.md

@@ -78,31 +78,32 @@ Compressed `.mdx` is the piece no other open-source tool reads today. The format
 | | |
 |---|---|
 | Discs converted | 72 of 79 |
-| Samples | 89 125 |
-| Stereo pairs rejoined | 15 796 |
+| Samples | 89 156 |
+| Stereo pairs rejoined | 15 808 |
 | Audio CD tracks | 161 |
 | Duplicate audio suppressed | 5 719 |
-| Entries skipped (damage) | 97 |
-| Time | 46 s |
+| Entries not the file their entry placed | 61 |
+| Entries skipped (damage) | 5 |
+| Time | 48 s |
 
 By filesystem:
 
 | | Discs | Samples | Stereo pairs | Skipped |
 |---|---:|---:|---:|---:|
-| AKAI | 44 | 56 394 | 14 449 | 97 |
+| AKAI | 44 | 56 425 | 14 461 | 66 |
 | E-mu `EMU3` | 7 | 14 738 | 6 | 0 |
 | ISO 9660 | 15 | 11 601 | — | 0 |
 | Roland `S770 MR25A` | 5 | 6 392 | 1 341 | 0 |
 
 "Stereo pairs" counts files joined from an `-L`/`-R` pair by name. E-mu's six are the only ones on those discs, and they are a different and much rarer thing than the 2 656 samples whose record declares two channels.
 
-Every WAV was checked against the disc it came from — **70 of 70 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 89 125 payloads, zero mismatches. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all seven discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against.
+Every WAV was checked against the disc it came from — **70 of 70 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 89 156 payloads, zero mismatches. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all seven discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against.
 
-105 082 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **275 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, six are on a Proteus library that ships `Dead Air` as a sample, and two are on a Roland disc. That is what the discs hold, not something the decoder did.
+105 125 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **275 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, six are on a Proteus library that ships `Dead Air` as a sample, and two are on a Roland disc. That is what the discs hold, not something the decoder did.
 
 Sample rates run from 6 000 to 49 999 Hz across 1 047 distinct values. The odd ones are real — E-mu writes rates like 24 444 and 27 778, and AKAI uses 33 075 (¾ of 44 100) and 29 400 (⅔) to trade bandwidth for memory. They are carried through exactly as the disc states them and never rounded.
 
-The 97 damage skips are almost all one thing: 92 payloads that do not begin with an AKAI sample header, concentrated on seven `.mdx` images. The rest are four implausible sample rates and one stereo pair whose halves declare 44 033 and 44 100, so the joiner refuses to fuse them and writes both mono halves instead.
+The 66 AKAI entries not written are two different faults. **61 are payloads that are not the file the directory placed there** — their header carries another file's id, valid flag or name — and 60 of those 61 are a run to the end of one volume, on ten discs; the other 34 AKAI discs have none. That is what a rip losing a run of blocks looks like from inside a directory, and it does not need a partition to go missing: `Best Service - Alpha Dance II` declares six partitions and holds all six, and still loses 21 of `AC.DRUMLOOPS`'s 22 samples. Each is refused with a line naming every field that disagrees and the entry that placed it, rather than being written out under a name that is not its own ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). The remaining five are damage of a different kind: four samples whose header is otherwise perfect and whose rate field reads 0, 519, 519 or 1280, and one stereo pair whose halves declare 44 033 and 44 100, so the joiner refuses to fuse them and writes both mono halves instead.
 
 The seven that do not convert are accounted for: one S-550 disc present as both `.iso` and `.nrg` — a different format from the S-7xx and not yet read ([ADR-0014](docs/adr/0014-one-backend-per-on-disc-format.md)) — two Digidesign SampleCell discs, one audio CD with no cue sheet present as both `.mdx` and `.cdr`, and one ISO 9660 disc holding E-mu `.EBL` banks rather than audio.
 

docs/README.md

@@ -58,11 +58,16 @@ All of that vanishes into a working parser. Six months on, the code says *what*
 | D16 | AIFF payloads converted, deduplicated against their WAV twin, and EXS24/HALion instruments kept ([ADR-0024](adr/0024-the-aiff-twin-is-converted-and-deduplicated.md)) | done |
 | D17 | E-mu loop points in the WAV `smpl` chunk ([ADR-0025](adr/0025-the-loop-is-decoded-the-root-key-is-not.md)) | done |
 | D18 | E-mu stereo samples decoded from the record's channel count ([ADR-0026](adr/0026-the-record-declares-the-channel-count.md)) | done |
+| D19 | An AKAI payload must be the file its entry placed, and the S3000 header is 192 bytes ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)) | done |
 
 Across the local collection, by listing: 71 of 79 images claimed, 2 578 volumes, 110 989 files, 77 620 of them samples. The AKAI discs are 44 of those images and 68 997 of those files, read across 275 partitions — before D15 they were 14 670 files, because only the partition at the origin was read.
 
 ## What is not done
 
+- **25 AKAI files are displaced inside a partition the table calls complete.** `Best Service - Alpha Dance II` declares six partitions and holds all six, and 21 of `AC.DRUMLOOPS`'s 22 samples are somebody else's audio; `AKAI.S3000.Sound.Library.1` and `.3` lose three and one the same way. The rip dropped a run of blocks *inside* a partition rather than the blocks a header sat on, so nothing in the partition table registers a gap and the declared-against-present count reads clean. They are refused and named rather than written ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)); recovering them would mean locating a volume's blocks by something other than the chain the allocation map declares, which is the search [ADR-0022](adr/0022-a-volume-is-explained-by-the-allocation-map.md) and [ADR-0023](adr/0023-partitions-come-from-the-table-the-disc-declares.md) both refused. Sibling of [#25](https://github.com/bmxcode/samplerdisc/issues/25) and invisible to it.
+- **The AKAI payload name check has no positives on real data.** It is the test that separates *"this payload is a sample"* from *"this payload is **this** sample"*, and across all 44 discs it fires 60 times and never once without the id and valid tests having fired too — because on the short images the displacement lands mid-audio, which does not look like a header. It is exercised only synthetically. A displacement landing exactly on another sample's header is the case it exists for and the collection has none, which is worth knowing before [#25](https://github.com/bmxcode/samplerdisc/issues/25) recovers those partitions ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)).
+- **What the AKAI valid byte's low bits mean is unestablished.** `0x81` on 29 samples and `0x9c` on two, against `0x80` on 56 397. The `0x80` flag is the sample-is-valid bit and the rest is unread; three combinations on two discs is not enough to read them from.
+
 - **Roland S-550.** `Roland LCD1.iso`/`.nrg` opens `* ROLAND S-550 *` and is a different format from the S-7xx entirely ([ADR-0014](adr/0014-one-backend-per-on-disc-format.md)). Neither archive holds a second specimen, so it stays deferred rather than being reverse-engineered from one disc.
 - **Ensoniq and Kurzweil backends.** The archives are full of these discs and the containers already open them; each needs a module in `fs/` and nothing else ([ADR-0003](adr/0003-brand-neutral-pluggable-backends.md)).
 - **65 records declare two channels and are not stereo.** They put `start_R` half a payload on and then close `end_L` somewhere else, and their halves measure as unrelated audio. Six of `protozoa`'s are explained — their first half is byte for byte a whole one-channel record of the same name in another bank, so the payload is twice the sound — and the rest are not. They come out mono, which is right, but *what they hold* is unestablished on `eiiix-1`'s 40, `eiiix-2`'s 6 and thirteen of `protozoa`'s ([formats/emu3.md](formats/emu3.md), [ADR-0026](adr/0026-the-record-declares-the-channel-count.md)).