D21: an E-mu record is closed by the channel it declares, and found by either one

CHANGELOG.md

@@ -4,8 +4,40 @@ Notable changes to `samplerdisc`. Format-level findings live in [docs/formats/](
 
 ## Unreleased
 
+### Fixed
+
+- **An E-mu record was being closed by the wrong channel, and most EIII/ESI samples were 46 frames short.** The 92-byte sample record carries a start, an end, a loop start and a loop end **per channel**, and this project took the record's length from `+34` — which is the *right* channel's end pointer. It closes the record only where the right-hand set is what describes the audio, and on 10 274 of the 15 272 EIII/ESI records here the left-hand set is. **Anyone who extracted an EIII or ESI disc before this should do it again.** ([docs/formats/emu3.md](docs/formats/emu3.md), [ADR-0029](docs/adr/0029-a-record-is-closed-by-the-channel-it-declares.md), [#39](https://github.com/bmxcode/samplerdisc/issues/39))
+
+  Four shapes the right-hand set takes when it is not describing the record, each of which loses something different:
+
+  | Shape | What `+34` gives | Records |
+  |---|---|---:|
+  | `start_R = 0`, `end_R = end_L − 92` — the same channel counted from the payload's start | an extent **92 bytes short**, so the sample loses its last 46 frames | 7 005 |
+  | `start_R = end_R = 0` — the unused side zeroed | an extent of 2, shorter than the header, so the record is dropped | 872 |
+  | a fixed memory frame — 1 MiB on `E-mu Classics`, 2 MiB on `EIIIX Vol. 1` | an address past the whole bank, so the record is dropped | 95 |
+  | `start_R = 92` with the left set not opening the audio | nothing: the record is never found | 1 429 |
+
+  **Every EIII/ESI count moves, and the four reference discs move most:**
+
+  | Disc | Samples | With a loop |
+  |---|---|---|
+  | `ESI-32 General Midi` | 2 265 → **2 635** | 107 → **1 778** |
+  | `Protozoa` | 5 852 → **6 595** | 1 689 → **5 244** |
+  | `EIIIX Vol. 1` | 1 189 → **1 248** | 1 157 → **1 215** |
+  | `EIIIX Vol. 2` | 1 333 → **1 337** | 1 260 → **1 264** |
+
+  **The loop counts move because a loop end that used to run past the audio now fits.** [ADR-0025](docs/adr/0025-the-loop-is-decoded-the-root-key-is-not.md) refused those rather than clamping them back, and refusing is what made the wrong extent visible as a missing loop instead of a wrong one. The 838 loops that newly appear on `ESI-32 General Midi` score **+0.87** on the splice shape test against a control of +0.01, and the newly *found* records score +0.90 — a false hit inside PCM does not carry a loop that splices.
+
+  **Two independent fields say the new extent is right.** The stride to the next record follows `end_L + 2` on 2 093 of `ESI-32 General Midi`'s records and `end_R + 2` on 30. And the bank header's own declared run — a different structure entirely — now ends exactly on the last record of **173 of 186 banks**, against 86 before, with **no bank left one 92-byte header short** on any disc; [ADR-0021](docs/adr/0021-a-bank-owns-the-run-its-header-declares.md) had recorded that population as a loose fit.
+
+  **The three E-IV discs are byte-for-byte unchanged** — 449, 2 822 and 828 samples, same payload digests. They size a record from their own big-endian sample directory and never read `+34`, so they are the control that says nothing shared was disturbed. The stereo counts are unchanged on every disc that had them, too.
+
+  **`Protozoa`'s trombones resolve.** [ADR-0026](docs/adr/0026-the-record-declares-the-channel-count.md) found six records whose first half was byte for byte a whole one-channel record of the same name in another bank, with nothing matching the second half. There was no second half: read at its declared length, `Proteus1PresetsX`'s `Trom B2` is byte-identical to `Vintage PresetsX`'s. The 65 records that "declared two channels and were not stereo" were all of them sized at twice their length, and that population is now **zero** across ten discs.
+
 ### Added
 
+- **Three EIIIX/ESI discs join the reference set, and twelve banks that read as empty now extract.** `E-mu Classics`, `Vintage` and `Ditto Drums` each claimed a bank and returned nothing with no explanation, which is the [ADR-0012](docs/adr/0012-a-probe-must-confirm-a-file.md) signature this project treats as a bug rather than a quirk. None was an index bank; all twelve are the record-extent fault above. `Ditto Drums` shows the whole of it on one disc — **74 samples before, 948 after** — and `Vintage`'s `Juno Synths` is a bank whose every record declares its one channel on the right, so nothing found any of it. ([#39](https://github.com/bmxcode/samplerdisc/issues/39))
+
 - **Eight AKAI images are short of the disc they were made from, and their displaced partitions are now read.** Whole 32 KB blocks are missing from these `.mdx` files — the container decodes every block it holds, and the file is not a complete copy of the disc — so every partition after a gap sits that much nearer the front than the disk's own partition table places it. **39 partitions are found there**, carrying **432 volumes, 17 180 files and 15 808 samples** that nothing could reach before. The collection goes from 89 156 samples to **104 921**, and AKAI from 56 425 to **72 190**. ([docs/formats/akai-fs.md](docs/formats/akai-fs.md), [ADR-0028](docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md), [#25](https://github.com/bmxcode/samplerdisc/issues/25))
 
   | Disc | Recovered | Volumes | Files | Samples | Written |

README.md

@@ -49,7 +49,7 @@ out/AKAI.S3000.Sound.Library.1/
 
 Extraction writes one WAV per sample, grouped by volume. Where a disc stores stereo as split mono files — the AKAI `-L` / `-R` convention — you also get a joined stereo WAV, and the mono originals are kept alongside it rather than replaced.
 
-Some samples are stereo on the disc itself rather than paired by name: an E-mu record declares its own channel count, and 2 656 of the 14 738 E-mu samples here declare two. Those come out as one stereo WAV under the sample's own name — not in `stereo/`, which means "rebuilt from two files", and with no mono halves to keep, because nothing was guessed at ([ADR-0026](docs/adr/0026-the-record-declares-the-channel-count.md)).
+Some samples are stereo on the disc itself rather than paired by name: an E-mu record declares its own channel count, and 2 843 of the 19 371 E-mu samples here declare two. Those come out as one stereo WAV under the sample's own name — not in `stereo/`, which means "rebuilt from two files", and with no mono halves to keep, because nothing was guessed at ([ADR-0026](docs/adr/0026-the-record-declares-the-channel-count.md)).
 
 The audio is a byte-for-byte copy: AKAI stores signed 16-bit little-endian PCM and so does WAV, so there is no resampling, no bit-depth change and no dithering anywhere in the process. Loop points, root key and tuning from the disc are written into the WAV's standard `smpl` chunk, so a DAW that understands them picks them up and one that doesn't sees an ordinary WAV.
 
@@ -73,35 +73,35 @@ Compressed `.mdx` is the piece no other open-source tool reads today. The format
 
 ## Tested against
 
-79 disc images from three archive.org collections — [retro-sample-cds](https://archive.org/details/retro-sample-cds), [archive-oldschoolscds](https://archive.org/details/archive-oldschoolscds) and [Best Service ProSamples](https://archive.org/details/best-service-pro-samples-vol.-12-dance-vocals-akai-1-cd). Fifty-two flat `.iso`/`.bin`, seventeen compressed `.mdx`, five raw CD images, two `.nrg`, one `.mds`/`.mdf` pair, two audio CDs.
+82 disc images from three archive.org collections — [retro-sample-cds](https://archive.org/details/retro-sample-cds), [archive-oldschoolscds](https://archive.org/details/archive-oldschoolscds) and [Best Service ProSamples](https://archive.org/details/best-service-pro-samples-vol.-12-dance-vocals-akai-1-cd). Fifty-five flat `.iso`/`.bin`, seventeen compressed `.mdx`, five raw CD images, two `.nrg`, one `.mds`/`.mdf` pair, two audio CDs.
 
 | | |
 |---|---|
-| Discs converted | 72 of 79 |
-| Samples | 104 921 |
-| Stereo pairs rejoined | 17 309 |
+| Discs converted | 75 of 82 |
+| Samples | 109 554 |
+| Stereo pairs rejoined | 17 310 |
 | Audio CD tracks | 161 |
 | Duplicate audio suppressed | 5 719 |
 | Entries not the file their entry placed | 104 |
 | Entries skipped (damage) | 23 |
-| Time | 72 s |
+| Time | 70 s |
 
 By filesystem:
 
 | | Discs | Samples | Stereo pairs | Skipped |
 |---|---:|---:|---:|---:|
 | AKAI | 44 | 72 190 | 15 962 | 127 |
-| E-mu `EMU3` | 7 | 14 738 | 6 | 0 |
+| E-mu `EMU3` | 10 | 19 371 | 7 | 0 |
 | ISO 9660 | 15 | 11 601 | — | 0 |
 | Roland `S770 MR25A` | 5 | 6 392 | 1 341 | 0 |
 
-"Stereo pairs" counts files joined from an `-L`/`-R` pair by name. E-mu's six are the only ones on those discs, and they are a different and much rarer thing than the 2 656 samples whose record declares two channels.
+"Stereo pairs" counts files joined from an `-L`/`-R` pair by name. E-mu's seven are the only ones on those discs, and they are a different and much rarer thing than the 2 843 samples whose record declares two channels.
 
-Every WAV was checked against the disc it came from — **70 of 70 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 104 921 payloads, zero mismatches. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all seven discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against.
+Every WAV was checked against the disc it came from — **73 of 73 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 109 554 payloads, zero mismatches; the ten E-mu discs were re-measured for D21 and match on all 19 371. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all ten discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against.
 
-122 391 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **275 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, six are on a Proteus library that ships `Dead Air` as a sample, and two are on a Roland disc. That is what the discs hold, not something the decoder did.
+127 025 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **301 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, 24 are unused slots on the `Ditto Drums` ESI-32 disc, six are on a Proteus library that ships `Dead Air` as a sample, two are on a Roland disc, and one each on `E-mu Classics` and `Vintage`. That is what the discs hold, not something the decoder did.
 
-Sample rates run from 6 000 to 49 999 Hz across 1 071 distinct values. The odd ones are real — E-mu writes rates like 24 444 and 27 778, and AKAI uses 33 075 (¾ of 44 100) and 29 400 (⅔) to trade bandwidth for memory. They are carried through exactly as the disc states them and never rounded.
+Sample rates run from 6 000 to 50 000 Hz across 1 145 distinct values. The odd ones are real — E-mu writes rates like 24 444 and 27 778, and AKAI uses 33 075 (¾ of 44 100) and 29 400 (⅔) to trade bandwidth for memory. They are carried through exactly as the disc states them and never rounded.
 
 The 127 AKAI entries not written are two different faults. **104 are payloads that are not the file the directory placed there** — their header carries another file's id, valid flag or name — and 103 of those 104 are a run to the end of one volume, on nine discs; the other 35 AKAI discs have none. That is what a rip losing a run of blocks looks like from inside a directory, and it does not need a partition to go missing: `Best Service - Alpha Dance II` declares six partitions and holds all six, and still loses 21 of `AC.DRUMLOOPS`'s 22 samples. Each is refused with a line naming every field that disagrees and the entry that placed it, rather than being written out under a name that is not its own ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). Every one of those 103 is sitting intact, under its own name, a whole number of container blocks earlier in the image — the same fault as the displaced partitions below, one level down, and not yet recovered ([#35](https://github.com/bmxcode/samplerdisc/issues/35)).
 
@@ -127,6 +127,7 @@ These discs are also the only place in this project where the correct output is
 - **`CUES` chunks in NRG** are not parsed; only `CUEX`. No disc using the older form was to hand to check the layout against.
 - **An audio CD with no cue sheet cannot be split into tracks.** `samplerdisc info` tells you when a disc's content looks like Red Book audio, and `extract --assume-audio-cd` writes the whole stream as one WAV, but the track boundaries live in a cue, not in the bytes.
 - **AIFF-C, and 8-bit AIFF, are refused rather than read.** AIFF-C may be compressed, and compressed data written out as PCM plays as noise while reporting nothing wrong. 8-bit AIFF is signed where 8-bit WAV is unsigned, so carrying it would change every sample value rather than reorder its bytes — which is the one thing this tool does not do ([ADR-0024](docs/adr/0024-the-aiff-twin-is-converted-and-deduplicated.md)).
+- **An E-mu loop that spans the whole sample is only refused where it starts at frame 0.** The format writes the sample's own bounds into the loop pointers when nothing set them, and several discs write them with a small inset instead — frame 6 to seven frames from the end. The `Ditto Drums` ESI-32 disc does it on **934 of its 948** samples, so those WAVs carry a `smpl` chunk telling a DAW to loop the entire file. A DAW that ignores `smpl` is unaffected, and the audio is right either way.
 - **EXS24 and HALion instruments are kept, not read.** `--keep-originals` writes the `.exs` and `.fxp` files out byte for byte; turning them into a playable instrument is ConvertWithMoss's job.
 - **AKAI S900, floppy images and the DD partition.** Use [akaiutil](https://sourceforge.net/projects/akaiutil/).
 

docs/README.md

@@ -60,8 +60,9 @@ All of that vanishes into a working parser. Six months on, the code says *what*
 | D18 | E-mu stereo samples decoded from the record's channel count ([ADR-0026](adr/0026-the-record-declares-the-channel-count.md)) | done |
 | D19 | An AKAI payload must be the file its entry placed, and the S3000 header is 192 bytes ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)) | done |
 | D20 | The partitions eight short AKAI images displaced, found by an anchored search ([ADR-0028](adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)) | done |
+| D21 | An E-mu record is closed by the channel it declares, and found by either one ([ADR-0029](adr/0029-a-record-is-closed-by-the-channel-it-declares.md)) | done |
 
-Across the local collection, by listing: 71 of 79 images claimed, 3 010 volumes, 128 169 files, 93 428 of them samples. The AKAI discs are 44 of those images and 86 177 of those files, read across 314 partitions — 275 where the disk's table puts them and 39 displaced by blocks the rip lost. Before D15 they were 14 670 files, because only the partition at the origin was read.
+Across the local collection, by listing: 74 of 82 images claimed, 3 096 volumes, 132 802 files, 98 061 of them samples. The AKAI discs are 44 of those images and 86 177 of those files, read across 314 partitions — 275 where the disk's table puts them and 39 displaced by blocks the rip lost. Before D15 they were 14 670 files, because only the partition at the origin was read.
 
 ## What is not done
 
@@ -73,8 +74,7 @@ Across the local collection, by listing: 71 of 79 images claimed, 3 010 volumes,
 
 - **Roland S-550.** `Roland LCD1.iso`/`.nrg` opens `* ROLAND S-550 *` and is a different format from the S-7xx entirely ([ADR-0014](adr/0014-one-backend-per-on-disc-format.md)). Neither archive holds a second specimen, so it stays deferred rather than being reverse-engineered from one disc.
 - **Ensoniq and Kurzweil backends.** The archives are full of these discs and the containers already open them; each needs a module in `fs/` and nothing else ([ADR-0003](adr/0003-brand-neutral-pluggable-backends.md)).
-- **65 records declare two channels and are not stereo.** They put `start_R` half a payload on and then close `end_L` somewhere else, and their halves measure as unrelated audio. Six of `protozoa`'s are explained — their first half is byte for byte a whole one-channel record of the same name in another bank, so the payload is twice the sound — and the rest are not. They come out mono, which is right, but *what they hold* is unestablished on `eiiix-1`'s 40, `eiiix-2`'s 6 and thirteen of `protozoa`'s ([formats/emu3.md](formats/emu3.md), [ADR-0026](adr/0026-the-record-declares-the-channel-count.md)).
-- **`esi32-gm` and `protozoa` declare a longer extent than their record length gives.** On about 2 200 records of one and 3 800 of the other, `end_L` runs some 45 frames past the payload `+34` produces. Either the reader is 90 bytes short on those samples or the extent field means something else; the splice test does not settle it — a scan for the true loop end peaks within ±2 frames of the declared one on 10% and 20% of records, against about 3% for chance. Those records are refused a loop rather than given a clamped one, and nothing is changed on the strength of an unsettled measurement.
+- **An E-mu whole-extent loop is only refused where it starts at frame 0.** The format writes the sample's own bounds into the loop pointers when nothing set them, and on several discs it writes them with a small inset instead — frame 6 to seven frames from the end. `ditto-drums` does it on **934 of its 948** samples, `eiiix-1` on 460 and `eiiix-2` on 320, and those come out as a WAV telling a DAW to loop the whole file. Pre-dates D21 and was left out of it deliberately, so that the loop counts move for one reason at a time ([formats/emu3.md](formats/emu3.md), [ADR-0029](adr/0029-a-record-is-closed-by-the-channel-it-declares.md)).
 - **No root key for E-mu.** It is not in the sample record: no byte of the 92 tracks the note in the sample's own name above chance, over 1 741 named records of `esi32-gm` and 917 of `eiiix-1`. The E3 keeps it in the preset, and presets are not read — see the `E4P1` entry below. E-mu WAVs carry their loop with the RIFF neutral root key of 60 ([ADR-0025](adr/0025-the-loop-is-decoded-the-root-key-is-not.md)).
 - **`E4P1` presets are not read.** The three E-IV discs carry 916, 901 and 284 of them. On `eiv-studio` 100 of 230 banks have no sample directory and are listed with a note; preset-only banks are the likely explanation, and it is not established.
 - **The `.mds` track table is unread.** One real pair now reads end to end, but geometry is sniffed from the `.mdf` rather than taken from the descriptor, so a multi-track or offset image would be read from byte 0. What the one specimen's descriptor holds is written down in [formats/mdx.md](formats/mdx.md) without being relied on.