diff --git a/CHANGELOG.md b/CHANGELOG.md index 84db4c7..b3a25d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,8 +4,40 @@ Notable changes to `samplerdisc`. Format-level findings live in [docs/formats/]( ## Unreleased +### Fixed + +- **An E-mu record was being closed by the wrong channel, and most EIII/ESI samples were 46 frames short.** The 92-byte sample record carries a start, an end, a loop start and a loop end **per channel**, and this project took the record's length from `+34` — which is the *right* channel's end pointer. It closes the record only where the right-hand set is what describes the audio, and on 10 274 of the 15 272 EIII/ESI records here the left-hand set is. **Anyone who extracted an EIII or ESI disc before this should do it again.** ([docs/formats/emu3.md](docs/formats/emu3.md), [ADR-0029](docs/adr/0029-a-record-is-closed-by-the-channel-it-declares.md), [#39](https://github.com/bmxcode/samplerdisc/issues/39)) + + Four shapes the right-hand set takes when it is not describing the record, each of which loses something different: + + | Shape | What `+34` gives | Records | + |---|---|---:| + | `start_R = 0`, `end_R = end_L − 92` — the same channel counted from the payload's start | an extent **92 bytes short**, so the sample loses its last 46 frames | 7 005 | + | `start_R = end_R = 0` — the unused side zeroed | an extent of 2, shorter than the header, so the record is dropped | 872 | + | a fixed memory frame — 1 MiB on `E-mu Classics`, 2 MiB on `EIIIX Vol. 1` | an address past the whole bank, so the record is dropped | 95 | + | `start_R = 92` with the left set not opening the audio | nothing: the record is never found | 1 429 | + + **Every EIII/ESI count moves, and the four reference discs move most:** + + | Disc | Samples | With a loop | + |---|---|---| + | `ESI-32 General Midi` | 2 265 → **2 635** | 107 → **1 778** | + | `Protozoa` | 5 852 → **6 595** | 1 689 → **5 244** | + | `EIIIX Vol. 1` | 1 189 → **1 248** | 1 157 → **1 215** | + | `EIIIX Vol. 2` | 1 333 → **1 337** | 1 260 → **1 264** | + + **The loop counts move because a loop end that used to run past the audio now fits.** [ADR-0025](docs/adr/0025-the-loop-is-decoded-the-root-key-is-not.md) refused those rather than clamping them back, and refusing is what made the wrong extent visible as a missing loop instead of a wrong one. The 838 loops that newly appear on `ESI-32 General Midi` score **+0.87** on the splice shape test against a control of +0.01, and the newly *found* records score +0.90 — a false hit inside PCM does not carry a loop that splices. + + **Two independent fields say the new extent is right.** The stride to the next record follows `end_L + 2` on 2 093 of `ESI-32 General Midi`'s records and `end_R + 2` on 30. And the bank header's own declared run — a different structure entirely — now ends exactly on the last record of **173 of 186 banks**, against 86 before, with **no bank left one 92-byte header short** on any disc; [ADR-0021](docs/adr/0021-a-bank-owns-the-run-its-header-declares.md) had recorded that population as a loose fit. + + **The three E-IV discs are byte-for-byte unchanged** — 449, 2 822 and 828 samples, same payload digests. They size a record from their own big-endian sample directory and never read `+34`, so they are the control that says nothing shared was disturbed. The stereo counts are unchanged on every disc that had them, too. + + **`Protozoa`'s trombones resolve.** [ADR-0026](docs/adr/0026-the-record-declares-the-channel-count.md) found six records whose first half was byte for byte a whole one-channel record of the same name in another bank, with nothing matching the second half. There was no second half: read at its declared length, `Proteus1PresetsX`'s `Trom B2` is byte-identical to `Vintage PresetsX`'s. The 65 records that "declared two channels and were not stereo" were all of them sized at twice their length, and that population is now **zero** across ten discs. + ### Added +- **Three EIIIX/ESI discs join the reference set, and twelve banks that read as empty now extract.** `E-mu Classics`, `Vintage` and `Ditto Drums` each claimed a bank and returned nothing with no explanation, which is the [ADR-0012](docs/adr/0012-a-probe-must-confirm-a-file.md) signature this project treats as a bug rather than a quirk. None was an index bank; all twelve are the record-extent fault above. `Ditto Drums` shows the whole of it on one disc — **74 samples before, 948 after** — and `Vintage`'s `Juno Synths` is a bank whose every record declares its one channel on the right, so nothing found any of it. ([#39](https://github.com/bmxcode/samplerdisc/issues/39)) + - **Eight AKAI images are short of the disc they were made from, and their displaced partitions are now read.** Whole 32 KB blocks are missing from these `.mdx` files — the container decodes every block it holds, and the file is not a complete copy of the disc — so every partition after a gap sits that much nearer the front than the disk's own partition table places it. **39 partitions are found there**, carrying **432 volumes, 17 180 files and 15 808 samples** that nothing could reach before. The collection goes from 89 156 samples to **104 921**, and AKAI from 56 425 to **72 190**. ([docs/formats/akai-fs.md](docs/formats/akai-fs.md), [ADR-0028](docs/adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md), [#25](https://github.com/bmxcode/samplerdisc/issues/25)) | Disc | Recovered | Volumes | Files | Samples | Written | diff --git a/README.md b/README.md index dc80dd9..a9ac962 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ out/AKAI.S3000.Sound.Library.1/ Extraction writes one WAV per sample, grouped by volume. Where a disc stores stereo as split mono files — the AKAI `-L` / `-R` convention — you also get a joined stereo WAV, and the mono originals are kept alongside it rather than replaced. -Some samples are stereo on the disc itself rather than paired by name: an E-mu record declares its own channel count, and 2 656 of the 14 738 E-mu samples here declare two. Those come out as one stereo WAV under the sample's own name — not in `stereo/`, which means "rebuilt from two files", and with no mono halves to keep, because nothing was guessed at ([ADR-0026](docs/adr/0026-the-record-declares-the-channel-count.md)). +Some samples are stereo on the disc itself rather than paired by name: an E-mu record declares its own channel count, and 2 843 of the 19 371 E-mu samples here declare two. Those come out as one stereo WAV under the sample's own name — not in `stereo/`, which means "rebuilt from two files", and with no mono halves to keep, because nothing was guessed at ([ADR-0026](docs/adr/0026-the-record-declares-the-channel-count.md)). The audio is a byte-for-byte copy: AKAI stores signed 16-bit little-endian PCM and so does WAV, so there is no resampling, no bit-depth change and no dithering anywhere in the process. Loop points, root key and tuning from the disc are written into the WAV's standard `smpl` chunk, so a DAW that understands them picks them up and one that doesn't sees an ordinary WAV. @@ -73,35 +73,35 @@ Compressed `.mdx` is the piece no other open-source tool reads today. The format ## Tested against -79 disc images from three archive.org collections — [retro-sample-cds](https://archive.org/details/retro-sample-cds), [archive-oldschoolscds](https://archive.org/details/archive-oldschoolscds) and [Best Service ProSamples](https://archive.org/details/best-service-pro-samples-vol.-12-dance-vocals-akai-1-cd). Fifty-two flat `.iso`/`.bin`, seventeen compressed `.mdx`, five raw CD images, two `.nrg`, one `.mds`/`.mdf` pair, two audio CDs. +82 disc images from three archive.org collections — [retro-sample-cds](https://archive.org/details/retro-sample-cds), [archive-oldschoolscds](https://archive.org/details/archive-oldschoolscds) and [Best Service ProSamples](https://archive.org/details/best-service-pro-samples-vol.-12-dance-vocals-akai-1-cd). Fifty-five flat `.iso`/`.bin`, seventeen compressed `.mdx`, five raw CD images, two `.nrg`, one `.mds`/`.mdf` pair, two audio CDs. | | | |---|---| -| Discs converted | 72 of 79 | -| Samples | 104 921 | -| Stereo pairs rejoined | 17 309 | +| Discs converted | 75 of 82 | +| Samples | 109 554 | +| Stereo pairs rejoined | 17 310 | | Audio CD tracks | 161 | | Duplicate audio suppressed | 5 719 | | Entries not the file their entry placed | 104 | | Entries skipped (damage) | 23 | -| Time | 72 s | +| Time | 70 s | By filesystem: | | Discs | Samples | Stereo pairs | Skipped | |---|---:|---:|---:|---:| | AKAI | 44 | 72 190 | 15 962 | 127 | -| E-mu `EMU3` | 7 | 14 738 | 6 | 0 | +| E-mu `EMU3` | 10 | 19 371 | 7 | 0 | | ISO 9660 | 15 | 11 601 | — | 0 | | Roland `S770 MR25A` | 5 | 6 392 | 1 341 | 0 | -"Stereo pairs" counts files joined from an `-L`/`-R` pair by name. E-mu's six are the only ones on those discs, and they are a different and much rarer thing than the 2 656 samples whose record declares two channels. +"Stereo pairs" counts files joined from an `-L`/`-R` pair by name. E-mu's seven are the only ones on those discs, and they are a different and much rarer thing than the 2 843 samples whose record declares two channels. -Every WAV was checked against the disc it came from — **70 of 70 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 104 921 payloads, zero mismatches. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all seven discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against. +Every WAV was checked against the disc it came from — **73 of 73 discs match exactly**, comparing multisets of SHA-256 over the PCM per disc rather than going via filenames, so duplicate names cannot mask a mismatch and no path is guessed. 109 554 payloads, zero mismatches; the ten E-mu discs were re-measured for D21 and match on all 19 371. The E-mu stereo samples are compared with their channels put back the way the disc stored them, since their WAV holds the same bytes interleaved; `tests/test_discs.py` asserts that de-interleaving reproduces the disc's two blocks exactly, per sample, on all ten discs. The two audio CDs are not in that count: their tracks are cut from a stream by a cue, so there is no run of bytes on the disc to compare a track against. -122 391 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **275 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, six are on a Proteus library that ships `Dead Air` as a sample, and two are on a Roland disc. That is what the discs hold, not something the decoder did. +127 025 WAV files were written in all — the samples, the stereo joins and the audio CD tracks. None is unreadable and none is zero-length. **301 are silent for their whole length, and every one of them matches the disc exactly**: 267 are the blank `15G-KIT…Z` slots on `ProSamples vol.15`, 24 are unused slots on the `Ditto Drums` ESI-32 disc, six are on a Proteus library that ships `Dead Air` as a sample, two are on a Roland disc, and one each on `E-mu Classics` and `Vintage`. That is what the discs hold, not something the decoder did. -Sample rates run from 6 000 to 49 999 Hz across 1 071 distinct values. The odd ones are real — E-mu writes rates like 24 444 and 27 778, and AKAI uses 33 075 (¾ of 44 100) and 29 400 (⅔) to trade bandwidth for memory. They are carried through exactly as the disc states them and never rounded. +Sample rates run from 6 000 to 50 000 Hz across 1 145 distinct values. The odd ones are real — E-mu writes rates like 24 444 and 27 778, and AKAI uses 33 075 (¾ of 44 100) and 29 400 (⅔) to trade bandwidth for memory. They are carried through exactly as the disc states them and never rounded. The 127 AKAI entries not written are two different faults. **104 are payloads that are not the file the directory placed there** — their header carries another file's id, valid flag or name — and 103 of those 104 are a run to the end of one volume, on nine discs; the other 35 AKAI discs have none. That is what a rip losing a run of blocks looks like from inside a directory, and it does not need a partition to go missing: `Best Service - Alpha Dance II` declares six partitions and holds all six, and still loses 21 of `AC.DRUMLOOPS`'s 22 samples. Each is refused with a line naming every field that disagrees and the entry that placed it, rather than being written out under a name that is not its own ([ADR-0027](docs/adr/0027-a-payload-must-be-the-file-its-entry-placed.md)). Every one of those 103 is sitting intact, under its own name, a whole number of container blocks earlier in the image — the same fault as the displaced partitions below, one level down, and not yet recovered ([#35](https://github.com/bmxcode/samplerdisc/issues/35)). @@ -127,6 +127,7 @@ These discs are also the only place in this project where the correct output is - **`CUES` chunks in NRG** are not parsed; only `CUEX`. No disc using the older form was to hand to check the layout against. - **An audio CD with no cue sheet cannot be split into tracks.** `samplerdisc info` tells you when a disc's content looks like Red Book audio, and `extract --assume-audio-cd` writes the whole stream as one WAV, but the track boundaries live in a cue, not in the bytes. - **AIFF-C, and 8-bit AIFF, are refused rather than read.** AIFF-C may be compressed, and compressed data written out as PCM plays as noise while reporting nothing wrong. 8-bit AIFF is signed where 8-bit WAV is unsigned, so carrying it would change every sample value rather than reorder its bytes — which is the one thing this tool does not do ([ADR-0024](docs/adr/0024-the-aiff-twin-is-converted-and-deduplicated.md)). +- **An E-mu loop that spans the whole sample is only refused where it starts at frame 0.** The format writes the sample's own bounds into the loop pointers when nothing set them, and several discs write them with a small inset instead — frame 6 to seven frames from the end. The `Ditto Drums` ESI-32 disc does it on **934 of its 948** samples, so those WAVs carry a `smpl` chunk telling a DAW to loop the entire file. A DAW that ignores `smpl` is unaffected, and the audio is right either way. - **EXS24 and HALion instruments are kept, not read.** `--keep-originals` writes the `.exs` and `.fxp` files out byte for byte; turning them into a playable instrument is ConvertWithMoss's job. - **AKAI S900, floppy images and the DD partition.** Use [akaiutil](https://sourceforge.net/projects/akaiutil/). diff --git a/docs/README.md b/docs/README.md index 90b4a42..4a95067 100644 --- a/docs/README.md +++ b/docs/README.md @@ -60,8 +60,9 @@ All of that vanishes into a working parser. Six months on, the code says *what* | D18 | E-mu stereo samples decoded from the record's channel count ([ADR-0026](adr/0026-the-record-declares-the-channel-count.md)) | done | | D19 | An AKAI payload must be the file its entry placed, and the S3000 header is 192 bytes ([ADR-0027](adr/0027-a-payload-must-be-the-file-its-entry-placed.md)) | done | | D20 | The partitions eight short AKAI images displaced, found by an anchored search ([ADR-0028](adr/0028-a-displaced-partition-is-anchored-quantised-and-floored.md)) | done | +| D21 | An E-mu record is closed by the channel it declares, and found by either one ([ADR-0029](adr/0029-a-record-is-closed-by-the-channel-it-declares.md)) | done | -Across the local collection, by listing: 71 of 79 images claimed, 3 010 volumes, 128 169 files, 93 428 of them samples. The AKAI discs are 44 of those images and 86 177 of those files, read across 314 partitions — 275 where the disk's table puts them and 39 displaced by blocks the rip lost. Before D15 they were 14 670 files, because only the partition at the origin was read. +Across the local collection, by listing: 74 of 82 images claimed, 3 096 volumes, 132 802 files, 98 061 of them samples. The AKAI discs are 44 of those images and 86 177 of those files, read across 314 partitions — 275 where the disk's table puts them and 39 displaced by blocks the rip lost. Before D15 they were 14 670 files, because only the partition at the origin was read. ## What is not done @@ -73,8 +74,7 @@ Across the local collection, by listing: 71 of 79 images claimed, 3 010 volumes, - **Roland S-550.** `Roland LCD1.iso`/`.nrg` opens `* ROLAND S-550 *` and is a different format from the S-7xx entirely ([ADR-0014](adr/0014-one-backend-per-on-disc-format.md)). Neither archive holds a second specimen, so it stays deferred rather than being reverse-engineered from one disc. - **Ensoniq and Kurzweil backends.** The archives are full of these discs and the containers already open them; each needs a module in `fs/` and nothing else ([ADR-0003](adr/0003-brand-neutral-pluggable-backends.md)). -- **65 records declare two channels and are not stereo.** They put `start_R` half a payload on and then close `end_L` somewhere else, and their halves measure as unrelated audio. Six of `protozoa`'s are explained — their first half is byte for byte a whole one-channel record of the same name in another bank, so the payload is twice the sound — and the rest are not. They come out mono, which is right, but *what they hold* is unestablished on `eiiix-1`'s 40, `eiiix-2`'s 6 and thirteen of `protozoa`'s ([formats/emu3.md](formats/emu3.md), [ADR-0026](adr/0026-the-record-declares-the-channel-count.md)). -- **`esi32-gm` and `protozoa` declare a longer extent than their record length gives.** On about 2 200 records of one and 3 800 of the other, `end_L` runs some 45 frames past the payload `+34` produces. Either the reader is 90 bytes short on those samples or the extent field means something else; the splice test does not settle it — a scan for the true loop end peaks within ±2 frames of the declared one on 10% and 20% of records, against about 3% for chance. Those records are refused a loop rather than given a clamped one, and nothing is changed on the strength of an unsettled measurement. +- **An E-mu whole-extent loop is only refused where it starts at frame 0.** The format writes the sample's own bounds into the loop pointers when nothing set them, and on several discs it writes them with a small inset instead — frame 6 to seven frames from the end. `ditto-drums` does it on **934 of its 948** samples, `eiiix-1` on 460 and `eiiix-2` on 320, and those come out as a WAV telling a DAW to loop the whole file. Pre-dates D21 and was left out of it deliberately, so that the loop counts move for one reason at a time ([formats/emu3.md](formats/emu3.md), [ADR-0029](adr/0029-a-record-is-closed-by-the-channel-it-declares.md)). - **No root key for E-mu.** It is not in the sample record: no byte of the 92 tracks the note in the sample's own name above chance, over 1 741 named records of `esi32-gm` and 917 of `eiiix-1`. The E3 keeps it in the preset, and presets are not read — see the `E4P1` entry below. E-mu WAVs carry their loop with the RIFF neutral root key of 60 ([ADR-0025](adr/0025-the-loop-is-decoded-the-root-key-is-not.md)). - **`E4P1` presets are not read.** The three E-IV discs carry 916, 901 and 284 of them. On `eiv-studio` 100 of 230 banks have no sample directory and are listed with a note; preset-only banks are the likely explanation, and it is not established. - **The `.mds` track table is unread.** One real pair now reads end to end, but geometry is sniffed from the `.mdf` rather than taken from the descriptor, so a multi-track or offset image would be read from byte 0. What the one specimen's descriptor holds is written down in [formats/mdx.md](formats/mdx.md) without being relied on. diff --git a/docs/adr/0021-a-bank-owns-the-run-its-header-declares.md b/docs/adr/0021-a-bank-owns-the-run-its-header-declares.md index 8211698..4d15d08 100644 --- a/docs/adr/0021-a-bank-owns-the-run-its-header-declares.md +++ b/docs/adr/0021-a-bank-owns-the-run-its-header-declares.md @@ -65,3 +65,5 @@ What survives is the half that matters: a bank that declares sample data and yie **Bad, and the headline.** `esi32-gm` moves from 2 424 to 2 265 and `protozoa` from 6 788 to 5 852. Both old numbers counted another bank's records; `esi32-gm`'s were not previously suspected, and the issue records it as unaffected. It is not — its last bank ran to the end of the image and was credited with 193 records that belong to the two banks before it. Both counts are now pinned by a disc-backed test rather than by a table in a document, which is why the doc's baseline was able to be wrong for a release. **Watch for.** A disc whose `0x34` is damaged or zero on a bank that really holds audio. It will list empty with a note saying the header declares no sample area, and the note will be true about the header and wrong about the bank. Before this record that disc would have listed its neighbour's samples instead, which is not better, but it is a different wrong and someone may meet it. + +> **D21 read the other half of the run.** This record measured the last record of a bank ending exactly at `0x30 + 74 + 0x34` on 72 banks and "exactly 92 bytes — one sample header — short of it on 19 more", and took the second population as a loose fit. It is not: those banks were sized from the right channel's end pointer on a record that declares its audio on the left, which is 92 bytes short by construction. Under [ADR-0029](0029-a-record-is-closed-by-the-channel-it-declares.md) the fit is exact on 173 of 186 banks across ten discs and **no bank is 92 short on any of them**. Nothing here changes: the run is still the bound, it still gates where a record starts and never where its audio ends, and the four sample counts this record moved move again for a different reason. diff --git a/docs/adr/0025-the-loop-is-decoded-the-root-key-is-not.md b/docs/adr/0025-the-loop-is-decoded-the-root-key-is-not.md index 2e63620..92c3953 100644 --- a/docs/adr/0025-the-loop-is-decoded-the-root-key-is-not.md +++ b/docs/adr/0025-the-loop-is-decoded-the-root-key-is-not.md @@ -101,3 +101,5 @@ Those samples are stereo, and this project currently writes each as a double-len Nothing in D17 changes that — fixing it moves audio, which is the one thing this deliverable must not do. It is written up in [formats/emu3.md](../formats/emu3.md) and left open for a deliverable of its own. The loop frames decoded here survive it unchanged: `(pointer − start) / 2` is a per-channel frame index either way. > **D18 acted on this in [ADR-0026](0026-the-record-declares-the-channel-count.md), and every loop count above survived it.** One correction to the paragraph above: the channel count alone selects 2 721 records and 65 of those are not stereo — their own `end_L` contradicts the split — so 2 656 are written as stereo. `protozoa`'s weak 0.55 is those false positives, 19 of its 27. The two records anywhere whose loop end lies past its own channel are both among the 65, which is why 107, 1 157, 1 260, 1 689, 449, 2 551 and 826 are unchanged rather than nearly unchanged. + +> **D21 settled the question this record left open.** *"Which of the two is right is not established here and is not guessed at"* — the extent, or the record length. It was the record length, taken from `+34`, which is the right channel's end pointer and is 92 bytes short on a record that declares its audio on the left ([ADR-0029](0029-a-record-is-closed-by-the-channel-it-declares.md)). With the corrected extent the declared loop end fits inside the payload without being clamped, and `esi32-gm` goes from **107 loops of 2 265 samples to 1 778 of 2 635**, `protozoa` from 1 689 to 5 244. The refusal this record decided on is untouched and is the reason the correction is visible at all: a loop end past the audio is still refused rather than clamped, so a wrong extent showed up as a missing loop instead of as a wrong one. The 838 newly admitted loops on `esi32-gm` score **+0.87** on the shape test against a control of +0.01. diff --git a/docs/adr/0026-the-record-declares-the-channel-count.md b/docs/adr/0026-the-record-declares-the-channel-count.md index 4c23cbb..923a26e 100644 --- a/docs/adr/0026-the-record-declares-the-channel-count.md +++ b/docs/adr/0026-the-record-declares-the-channel-count.md @@ -91,3 +91,5 @@ The inverse error — a record that is stereo and declares one channel — was l **Bad, accepted.** Four records across `eiv-analogia` and `eiv-studio` have byte-identical halves. They are written as stereo, because the record declares two channels and this decision is that the record is the authority; a dual-mono file is a real thing a sampler can hold, and second-guessing it would mean deciding channel counts by comparing audio, which is the alternative rejected above. **Watch for.** A generation that splits into blocks and writes a *third* record shape — one where `end_L` genuinely means something other than the left block's close. It would be refused silently and come out double-length mono, which is exactly the failure this deliverable fixed and exactly as quiet as it was before. The per-disc stereo counts in `tests/test_discs.py` are the tripwire: they are pinned as tightly as the sample counts, so a shape arriving or leaving has something to fail against. + +> **D21 explained the 65.** The gate's third condition — `end_L + 2 == start_R` — caught 65 records across three discs that declared a two-channel shape and were not stereo, six of which were identified here as a payload holding the sound twice. They were records sized from the wrong pointer: with the extent taken from the channel the record declares, `Proteus1PresetsX`'s `Trom B2` is byte-identical to `Vintage PresetsX`'s rather than twice its length, and **the population is zero on all ten discs** ([ADR-0029](0029-a-record-is-closed-by-the-channel-it-declares.md)). The condition stays — it is what makes D21's two-channel test exact — but it is now an unexercised gate rather than one firing 65 times, and the stereo counts it selects are unchanged on every disc that had them: 28, 8, 601, 592. diff --git a/docs/adr/0029-a-record-is-closed-by-the-channel-it-declares.md b/docs/adr/0029-a-record-is-closed-by-the-channel-it-declares.md new file mode 100644 index 0000000..85a41c5 --- /dev/null +++ b/docs/adr/0029-a-record-is-closed-by-the-channel-it-declares.md @@ -0,0 +1,102 @@ +# ADR-0029 · An E-mu record is closed by the channel it declares + +**Status:** accepted · 2026-08-24 + +## Context + +[Issue #39](https://github.com/bmxcode/samplerdisc/issues/39): three EIIIX/ESI discs new to the collection each claim a bank, return no files and give no reason. Twelve banks across `emu-classics`, `vintage` and `ditto-drums` — ten on `ditto-drums` alone. That is the [ADR-0012](0012-a-probe-must-confirm-a-file.md) signature, and none of them is the case [ADR-0021](0021-a-bank-owns-the-run-its-header-declares.md) already explains: every one declares a non-zero sample area, so `0x34` has nothing to say about any of them. + +The cause is one field read as something it is not, and it reaches a long way past those three discs. + +`+34` carried two names in this module — `OFF_SAMPLE_END_R` and `OFF_SAMPLE_RECORD_LEN` — and the walk took the record's extent from it unconditionally. **It is the right channel's end pointer.** It closes the record only where the right-hand set is what describes the record's audio, and on 10 274 of the 15 272 EIII/ESI records across seven discs the left-hand set is. Four ways it goes wrong, each of them a whole bank or a whole disc: + +| Shape | What the right-hand set holds | What the old walk did | +|---|---|---| +| **mirror-92** | `start_R = 0`, `end_R = end_L − 92` — the same channel written from the payload's start instead of the record's | extent **92 bytes short**: 2 127 records on `esi32-gm`, 3 965 on `protozoa`, 7 005 in all | +| **zeroed** | `start_R = end_R = 0` | extent of 2, shorter than the header, so the record is rejected outright — 872 records, and nine of `ditto-drums`'s ten silent banks | +| **fixed frame** | `start_R = 92 + F`, `end_R = 92 + 2F − 2` for a constant allocation frame `F` — 1 MiB on `emu-classics`, 2 MiB on `eiiix-1` | extent past the whole bank region, so the record is dropped as unreadable — `Vox Haunt X`'s 14, and 95 records in all | +| **right-declared** | `start_R = 92` with the left set not opening the audio — `start_L = 0` on 1 371 of them | invisible: the walk's signature is `92` at `+22`, so it never looks. All of `vintage`'s `Juno Synths`, and **1 429 records** including 353 on `esi32-gm` and 607 on `protozoa` that were never listed | + +The last one is the sharpest, because [formats/emu3.md](../formats/emu3.md) already recorded it. *"Either set can be the single one — 542 records on `eiv-studio` … declare their one channel on the **right**, with the left zeroed"*, and *"Read the set whose start is 92 and use it."* That rule was applied to picking a loop and to nothing else. The EIII walk both **located** and **sized** a record from the left-hand pointer alone. + +The mirror-92 shape is the answer to a question [docs/README.md](../README.md) had open since D17: *"`esi32-gm` and `protozoa` declare a longer extent than their record length gives … Either the reader is 90 bytes short on those samples or the extent field means something else."* It is the reader, by 92 bytes, and the splice test could not settle it because the splice test was being asked the wrong question. + +Two measurements settle it, and neither is the pointer block arguing with itself. + +**The stride to the next record.** `end_L + 2` equals the distance to the next record on 2 093 of `esi32-gm`'s records where `end_R + 2` equals it on 30. + +**The bank header's own run length — a different field, in a different structure.** ADR-0021 measured that a bank's last record ends exactly at `0x30 + 74 + 0x34` on 72 banks and *"exactly 92 bytes — one sample header — short of it on 19 more"*, and recorded the second population as a loose fit. It is not a loose fit; it is this bug, seen from the bank header. Under the corrected extent it disappears: + +| Disc | banks with records | last record ends exactly at the run's end, now | before | +|---|---:|---:|---:| +| `esi32-gm` | 7 | **7** | 2 | +| `protozoa` | 15 | **15** | 0 | +| `eiiix-1` | 44 | **39** | 35 | +| `eiiix-2` | 44 | **39** | 37 | +| `emu-classics` | 19 | **16** | 8 | +| `vintage` | 13 | **13** | 4 | +| `ditto-drums` | 44 | **44** | 0 | +| **total** | **186** | **173** | **86** | + +Not one bank on any disc is left in the "92 bytes short" bucket. The reference bank the format doc pins says the same thing on its own: `8M GeneralMidi X` now yields **531 records totalling 8 248 316 bytes**, which is its declared run to the byte, where it previously yielded 452 records and 7 345 200 bytes inside a run of 8 248 316 that nothing filled. + +## Decision + +**A record's extent is closed by the end pointer of the set that opens its audio, and a record is found by either set opening it.** + +Three parts, all in `fs/emu3.py`: + +**The signature is `92` at `+22` or at `+26`.** Whichever set opens the audio identifies the record; the two anchors are deduplicated by address and the result is yielded in address order, so which anchor found a record cannot change what is written. + +**The extent is `end + 2` of the set whose start is 92** — the larger of the two where both do, which is what the stride follows on every disc measured. + +**Except on a confirmed two-channel record, where it is `end_R + 2`** and covers both blocks. Stated from the pointers alone: `start_L == 92`, `start_R == end_L + 2`, and `end_R − start_R == end_L − start_L`. That is algebraically [ADR-0026](0026-the-record-declares-the-channel-count.md)'s three conditions, without reference to the payload size — which matters, because the payload size is what is being computed. `sample/emu3.py`'s `_is_block_split` is the same statement given the payload; the two must agree, and the per-disc stereo counts in the disc-backed suite are what holds them together. + +## What the evidence says about the audio + +The extent decides how much audio each sample is, so the counts moving is not by itself a reason to believe them. Two independent things say the new audio is right. + +**The loops splice.** 838 of `esi32-gm`'s loops and 1 082 of `protozoa`'s are newly admitted — they are exactly the loops [ADR-0025](0025-the-loop-is-decoded-the-root-key-is-not.md) refused because the declared end ran past the payload, and with the corrected extent they fit **without being clamped**, which is the move that record showed destroys a loop. Scored by the shape and join tests of [formats/emu3.md](../formats/emu3.md), with its controls — a wrong start at the same end, a 64-frame floor, a 15 % RMS floor at both ends: + +| Disc | group | scored | shape *r* | control | join | seamless | control seamless | +|---|---|---:|---:|---:|---:|---:|---:| +| `esi32-gm` | loop newly admitted | 838 | **+0.87** | +0.01 | 1.25 | 87 % | 36 % | +| `esi32-gm` | record newly found | 130 | **+0.90** | −0.03 | 1.36 | 80 % | 29 % | +| `protozoa` | loop already produced | 905 | +0.85 | +0.02 | 1.35 | 83 % | 38 % | +| `protozoa` | loop newly admitted | 1 082 | **+0.82** | −0.01 | 1.07 | 90 % | 32 % | +| `protozoa` | record newly found | 330 | **+0.77** | −0.06 | 0.90 | 87 % | 34 % | +| `eiiix-1` | loop already produced | 413 | +0.96 | +0.02 | 1.03 | 90 % | 36 % | +| `eiiix-1` | record newly found | 133 | **+0.99** | −0.10 | 0.84 | 94 % | 23 % | +| `vintage` | record newly found | 250 | **+0.94** | −0.03 | 1.50 | 85 % | 23 % | + +The newly admitted loops score with the loops this project already shipped, against a control at zero. The **records newly found** — the right-declared and zeroed-right-set ones — score the same way, which is the answer to the obvious worry about widening a signature that scans through megabytes of audio: a false hit inside PCM does not carry a loop that splices. + +**`protozoa`'s trombones resolve.** [ADR-0026](0026-the-record-declares-the-channel-count.md) identified six records whose *"first half is byte for byte the whole of a one-channel record of the same name in another bank"* — `Trom B2`, 16 756 bytes inside a 33 512-byte payload, with nothing on the disc matching the second half. With the extent taken from the left channel, `Proteus1PresetsX`'s `Trom B2` is 16 764 bytes and is **byte-identical to `Vintage PresetsX`'s and `Vintage InstrmtX`'s**, as are `Trom E3`, `Trom C5`, `Trom D4` and `Trom G4`. The unexplained second half was never part of the record. + +That has a consequence for ADR-0026 worth stating plainly. Its gate's third condition — `end_L + 2 == start_R` — caught 65 records across three discs that declared a two-channel shape and were not stereo. Under the corrected extent that population is **zero on all ten discs**: those records only ever looked like a split because they were sized at twice their length. The condition stays, because it is what makes the two-channel test above exact and because a gate is not removed for going quiet, but it is now an unexercised gate and that is recorded rather than glossed. + +## Alternatives rejected + +**Fall back to `end_L` only where `end_R` yields no usable extent.** The minimal fix. It closes #39 and moves none of the four pinned baselines. Rejected: it is a rule that says *use whichever number works*, which is the kind of thing this project's format docs exist to replace. It leaves 6 092 records on `esi32-gm` and `protozoa` 92 bytes short, it cannot explain the bank-run fit, and it would have to be undone the first time a record's wrong extent happened to be usable. + +**Take `max(end_L, end_R) + 2` unconditionally.** Simpler than the rule adopted and right on three of the four shapes. Rejected on the fixed-frame shape: there `end_R` names the far end of a 1 MiB or 2 MiB allocation frame, is larger than `end_L`, and has nothing to do with this record. It also loses the two-channel case's meaning — it would get the right number for the wrong reason, and the reason is what a later reader needs. + +**Widen the signature and leave the extent alone.** Fixes `vintage` and nothing else: nine of `ditto-drums`'s ten banks stay silent, `Vox Haunt X` stays silent, and the 92-byte shortfall stays. + +**Bound the extent by the next record found instead of by the pointers.** Attractive because the stride is the evidence. Rejected: records are *found, not chained* on these discs and the runs have gaps, so the next hit is not the next record — the format doc records that as the reason the walk is a scan in the first place. Using it to size a record would make the last record of every run unbounded and would put a measurement in place of a declaration. + +**Fix the whole-extent loop guard at the same time.** `ditto-drums` writes a loop spanning the entire sample on 934 of its 948 records, starting at frame 6 rather than frame 0, and `sample/emu3.py` only refuses a whole-extent loop that starts at exactly 0. Rejected as a separate change: the same pattern is already shipping on `eiiix-1` (460) and `eiiix-2` (320) and predates this record, so folding it in would move the loop counts for two unrelated reasons at once and make neither attributable. Filed instead. + +## Consequences + +**Good.** All twelve banks of #39 yield files, and the only volumes left empty on the three discs are index banks and the sampler's own code banks, both of which already carry their note. `ditto-drums` goes from 74 samples to **948**. + +**Good.** The three E-IV discs are **byte-identical** across the change — 449, 2 822 and 828, same digests. They size a record from their own big-endian sample directory and never read `+34` ([ADR-0020](0020-read-e-iv-through-its-sample-directory.md)), so they are the control that says the shared parts were not disturbed. + +**Good.** Payload overlaps into the following record fall rather than rise — 439 to 238 on `eiiix-1`, 236 to 188 on `eiiix-2` — and the newly found right-declared records land in the gaps between existing records rather than inside them: 0 of 353 on `esi32-gm`, 0 of 607 on `protozoa`, 1 of 18 on `eiiix-1`. + +**Bad, and the headline.** All four EIII/ESI reference discs move, and every pinned payload digest with them. `esi32-gm` 2 265 → **2 635** samples and 107 → **1 778** loops; `protozoa` 5 852 → **6 595** and 1 689 → **5 244**; `eiiix-1` 1 189 → **1 248**; `eiiix-2` 1 333 → **1 337**. **Anyone who extracted an EIII or ESI disc before this should do it again**: on `esi32-gm` and `protozoa` most samples were 46 frames short at the end, and on every disc some samples were missing entirely. This is the same shape of consequence ADR-0021 took, for the same reason — the old number was wrong and nothing asserted the right one. + +**Bad.** Two pinned discs turned out to share a file size with another image in the collection — `emu-classics` with `Vol. 03 – Orchestral`, `eiv-studio` with `Producer Series Vol. 2 – More Studio Essentials` — which the disc suite's size-based pin treats as one disc filed twice. A whole publisher's series arriving at once is exactly when that assumption breaks. The pin now falls back to a digest of the image's first megabyte where two share a size, which keeps it a property of the disc rather than of its filename ([ADR-0004](0004-detect-by-signature.md)) and leaves every other pin untouched. + +**Watch for.** A fifth shape of right-hand set. The rule refuses a record where *neither* set opens the audio at 92, which is a silent drop of exactly the kind this record exists to remove — it will present as a bank yielding fewer records than its run declares, and the bank-run test above is what would catch it. diff --git a/docs/adr/README.md b/docs/adr/README.md index cb0b7a1..dac792b 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -34,3 +34,4 @@ If you find yourself writing an ADR with no rejected alternative, you are writin | [0026](0026-the-record-declares-the-channel-count.md) | The E-mu record declares the channel count, and its own extents confirm it | Splitting on the channel count alone; writing the mono halves alongside; filing it under `stereo/`; deciding stereo from the audio; taking high-correlation one-channel records as stereo; splitting at `end_L`; writing two mono files | | [0027](0027-a-payload-must-be-the-file-its-entry-placed.md) | An AKAI payload must be the file its entry placed, and its header length is the generation the entry declares | Closing #23 as not reproducing; dropping the name comparison; solving for the header length at runtime; branching on the id and valid bytes; sniffing the header pattern; an allow-list of valid bytes; never relaxing the flag; writing a mismatch under the payload's name; a `--force` flag; a third result shape; fixing the mid-partition damage here | | [0028](0028-a-displaced-partition-is-anchored-quantised-and-floored.md) | A displaced AKAI partition is searched for from its declared position, in the container's unit, and never inside one already read | Leaving them unread; scanning for the signature; filtering a scan by file yield; searching in AKAI blocks; truncating an overlapping recovery; moving the present partition instead; refusing a whole disc for one clash; relaxing the floor for `Alpha Dance I`; searching forward; per-file provenance; fixing #35 here | +| [0029](0029-a-record-is-closed-by-the-channel-it-declares.md) | An E-mu record is closed by the channel it declares, and found by either one | Falling back to `end_L` only where `end_R` fails; `max(end_L, end_R)` unconditionally; widening the signature alone; bounding the extent by the next record found; fixing the whole-extent loop guard here | diff --git a/docs/formats/emu3.md b/docs/formats/emu3.md index 5beba02..ff28982 100644 --- a/docs/formats/emu3.md +++ b/docs/formats/emu3.md @@ -1,10 +1,10 @@ # E-mu `EMU3` -The filesystem E-mu wrote on CD-ROMs for the Emulator III, EIIIX, ESI-32, ESI-4000 and Emulator IV. The archives file these as separate generations; the disc does not. All seven reference discs write `EMU3` at byte 0 and share one directory format. +The filesystem E-mu wrote on CD-ROMs for the Emulator III, EIIIX, ESI-32, ESI-4000 and Emulator IV. The archives file these as separate generations; the disc does not. All ten reference discs write `EMU3` at byte 0 and share one directory format. The *bank interior* is not shared. EIII/ESI banks carry a bank header and are located by it. E-IV banks carry none at all — not one `EMULATOR` string on any of the three E-IV discs — and reach their samples through an `E3S1` sample directory instead ([ADR-0020](../adr/0020-read-e-iv-through-its-sample-directory.md)). -Verified against seven discs: +Verified against ten discs: | Short name | File | Size | |---|---|---| @@ -12,12 +12,19 @@ Verified against seven discs: | `protozoa` | `E-MU Formula 4000 Series Vol. 5 – Protozoa.iso` | 131 690 496 | | `eiiix-1` | `E-MU - EIIIX Sound Library Vol. 1 – Emulator Standards (EIIIX CD-ROM).iso` | 304 128 000 | | `eiiix-2` | `E-MU - EIIIX Sound Library Vol. 2 – More Emulator Standards (EIIIX CD-ROM).iso` | 304 435 200 | +| `emu-classics` | `Vol. 07 – E-mu Classics.iso` | 526 723 072 | +| `vintage` | `Vol. 08 – Vintage.iso` | 527 030 272 | +| `ditto-drums` | `Vol. 16 – Twenty Six Studio Drum Kits and Percussion ESI-32 (aka Ditto Drums).iso` | 308 121 600 | | `eiv-analogia` | `Producer Series Vol. 6 – Analogia Project (CD 2) (E-MU E-IV CD-ROM).iso` | 293 912 576 | | `eiv-studio` | `Producer Series Vol. 1 – Studio Essentials (E-MU E-IV CD-ROM).iso` | 399 077 376 | | `eiv-vitous` | `Miroslav Vitous … String Ensembles (EMU E-IV CD-ROM).iso` | 532 443 136 | `eiv-analogia` and `eiv-studio` are both Producer Series and may share a mastering run; `eiv-vitous` is a different publisher and is the independence check. Where a constant holds on Producer Series and not on Vitous, Vitous is right. +The last three arrived with [issue #39](https://github.com/bmxcode/samplerdisc/issues/39) and are the discs the record extent below was established against. They are the same series as `esi32-gm` — `Vol. NN` of one publisher's ESI/EIIIX library — and `ditto-drums` is the one that shows the whole of the record-extent bug on a single disc: 74 samples read before it, 948 after. + +**Two of these discs share a file size with another image in the archive**, which the disc-backed suite's size-based pin used to treat as one disc filed twice: `emu-classics` with `Vol. 03 – Orchestral` at 526 723 072, and `eiv-studio` with `Producer Series Vol. 2 – More Studio Essentials` at 399 077 376. A whole publisher's series mastered in one run is where "sizes are distinct" stops holding. The pin falls back to a digest of the image's first megabyte, which covers the `EMU3` header, the folder table and the first bank directory — the parts that actually differ. + Addressing is in **512-byte blocks**, not the 2048-byte cooked sector. ## Header @@ -186,7 +193,7 @@ A record starts **two bytes before its name**; those two bytes are `00 00` on ev | 22 | 4 | u32 LE `start_L` — **92 on every record measured**, and the signature the walk scans for | | 26 | 4 | u32 LE `start_R` | | 30 | 4 | u32 LE `end_L` | -| 34 | 4 | u32 LE `end_R` — the same field as the record length, **two short** of the distance to the next (EIII only) | +| 34 | 4 | u32 LE `end_R` | | 38 | 4 | u32 LE `loop_start_L` | | 42 | 4 | u32 LE `loop_start_R` | | 46 | 4 | u32 LE `loop_end_L` | @@ -194,7 +201,9 @@ A record starts **two bytes before its name**; those two bytes are `00 00` on ev | 54 | 4 | u32 LE **sample rate** | | 92 | … | sample data | -The signature — 92 at `+22`, a plausible rate, sixteen printable name bytes — is specific enough to scan megabytes of audio without false hits. On `esi32-gm`'s `8M GeneralMidi X` bank it yields **452 records with 452 distinct names** totalling 7.00 MiB inside a bank declaring 8 MiB. +The signature — 92 at `+22` **or** at `+26`, a plausible rate, sixteen printable name bytes — is specific enough to scan megabytes of audio without false hits. On `esi32-gm`'s `8M GeneralMidi X` bank it yields **531 records with 531 distinct names** totalling 8 248 316 bytes, which is the bank's declared run to the byte. + +Either start, because either set may be the one that opens the audio — see below. **76 of that bank's 531 records declare their one channel on the right**, and scanning for `+22` alone leaves every one of them, and all of `vintage`'s `Juno Synths`, invisible. ### The eight pointers @@ -214,16 +223,76 @@ Counts per disc, over every sample the walk yields: | Disc | two channels, confirmed | two channels, contradicted | one channel, left set | one channel, right set | neither | |---|---|---|---|---|---| -| `esi32-gm` | 28 | 0 | 2 230 | 0 | 7 | -| `eiiix-1` | 601 | 40 | 380 | 0 | 168 | -| `eiiix-2` | 592 | 6 | 680 | 0 | 55 | -| `protozoa` | 8 | 19 | 5 791 | 0 | 34 | +| `esi32-gm` | 28 | 0 | 2 247 | 353 | 7 | +| `protozoa` | 8 | 0 | 5 927 | 595 | 65 | +| `eiiix-1` | 601 | 0 | 380 | 0 | 267 | +| `eiiix-2` | 592 | 0 | 680 | 0 | 65 | +| `emu-classics` | 185 | 0 | 1 179 | 73 | 79 | +| `vintage` | 2 | 0 | 633 | 350 | 8 | +| `ditto-drums` | 0 | 0 | 941 | 0 | 7 | | `eiv-analogia` | 279 | 0 | 146 | 8 | 16 | | `eiv-studio` | 320 | 0 | 1 882 | 542 | 78 | | `eiv-vitous` | 828 | 0 | 0 | 0 | 0 | -| **total** | **2 656** | **65** | **11 109** | **550** | **358** | +| **total** | **2 843** | **0** | **14 015** | **1 921** | **592** | + +**"Contradicted" is a shape of its own and it must be rejected**, not counted as stereo: the record declares `start_R` half a payload on and then closes its left channel somewhere else. **It occurs on none of the ten discs.** An earlier revision of this table gave 65 — 40 on `eiiix-1`, 19 on `protozoa`, 6 on `eiiix-2` — and every one of them was a record sized at twice its length, because the extent came from the wrong pointer. Given the right extent they are not half-payload splits at all. The gate's third condition is kept, and is now exercised only synthetically; see "Stereo" below ([ADR-0029](../adr/0029-a-record-is-closed-by-the-channel-it-declares.md)). + +The "neither" column is records whose `start_R` is none of the three values — not a right-hand single channel, which is an ordinary record and is counted as one here. An earlier revision of this table folded those 550 into "neither", giving 620 for `eiv-studio` and 24 for `eiv-analogia`. + +### The set that opens the audio is the set that closes it + +**This is the whole rule, and it applies to locating a record, sizing it and reading its loop.** The audio begins immediately after the 92-byte header, so the set that describes it opens at 92; the record runs to that set's own end pointer, two bytes on. Only a confirmed two-channel record runs to `end_R`, because there the payload is both blocks. + +`+34` was read as the record's length for four deliverables, and it is the **right channel's end**. It closes the record only where the right-hand set is what describes the record, and on 10 274 of the 15 272 EIII/ESI records here the left-hand set is. Four shapes the right-hand set takes when it is not describing this record, each of which breaks a reader that takes `+34` as a length: + +| Shape | What the right-hand set holds | Reading `+34` as the length gives | Count | +|---|---|---|---| +| **mirror-92** | `start_R = 0`, `end_R = end_L − 92` — the same channel counted from the payload's start rather than the record's | an extent **92 bytes short**, so every sample loses its last 46 frames | 7 005 | +| **zeroed** | `start_R = end_R = 0` | an extent of 2, shorter than the header, so the record is dropped entirely | 872 | +| **fixed frame** | `start_R = 92 + F`, `end_R = 92 + 2F − 2` for a constant allocation frame `F` — 1 MiB on `emu-classics`, 2 MiB on `eiiix-1` | an address past the whole bank region, so the record is dropped as unreadable | 95 | +| **right-declared** | `start_R = 92` with the left set not opening the audio — `start_L = 0` on 1 371 of them | nothing: the record is never found, because the left-hand signature does not match it | 1 429 | + +Per disc, by which set opens the audio: + +| Disc | left only | both | right only | of which two-channel | +|---|---:|---:|---:|---:| +| `esi32-gm` | 2 280 | 2 | 353 | 28 | +| `protozoa` | 4 303 | 1 685 | 607 | 8 | +| `eiiix-1` | 850 | 380 | 18 | 601 | +| `eiiix-2` | 655 | 680 | 2 | 592 | +| `emu-classics` | 669 | 748 | 99 | 185 | +| `vintage` | 569 | 74 | 350 | 2 | +| `ditto-drums` | 948 | 0 | 0 | 0 | +| **total** | **10 274** | **3 569** | **1 429** | **1 416** | + +Where **both** sets open at 92 the two ends disagree by a few bytes in either direction, and the larger is the one the stride follows — 920 records on `protozoa` where `end_L` is 8 bytes past `end_R`, 90 where it is 8 bytes short of it. The mirror is not always exactly 92 either: on 886 left-only records it is within a few bytes of it or arbitrary. Neither matters, because the rule never reads the set that did not open the audio. + +**The two-channel exception, stated from the pointers alone:** + +``` +start_L == 92 and start_R == end_L + 2 and end_R - start_R == end_L - start_L +``` -**"Contradicted" is a shape of its own and it must be rejected**, not counted as stereo: the record declares `start_R` half a payload on and then closes its left channel somewhere else. See "Stereo" below for what those 65 records turn out to hold. The "neither" column is records whose `start_R` is none of the three values — not a right-hand single channel, which is an ordinary record and is counted as one here. An earlier revision of this table folded those 550 into "neither", giving 620 for `eiv-studio` and 24 for `eiv-analogia`. +which is [ADR-0026](../adr/0026-the-record-declares-the-channel-count.md)'s three conditions without reference to the payload size — necessary, because the payload size is what is being computed from it. + +#### Two measurements settle this, and neither is the pointer block agreeing with itself + +**The stride to the next record.** `end_L + 2` equals the distance to the next record on 2 093 of `esi32-gm`'s records; `end_R + 2` equals it on 30. + +**The bank header's declared run — a different field in a different structure.** [ADR-0021](../adr/0021-a-bank-owns-the-run-its-header-declares.md) measured a bank's last record ending exactly at `0x30 + 74 + 0x34` on 72 banks, and *"exactly 92 bytes — one sample header — short of it"* on 19 more, and took the second population for a loose fit. It is this, seen from the bank header: + +| Disc | banks with records | last record ends exactly at the run's end | reading `+34` as the length | +|---|---:|---:|---:| +| `esi32-gm` | 7 | **7** | 2 | +| `protozoa` | 15 | **15** | 0 | +| `eiiix-1` | 44 | **39** | 35 | +| `eiiix-2` | 44 | **39** | 37 | +| `emu-classics` | 19 | **16** | 8 | +| `vintage` | 13 | **13** | 4 | +| `ditto-drums` | 44 | **44** | 0 | +| **total** | **186** | **173** | **86** | + +No bank on any disc is left 92 bytes short. `tests/test_discs.py` asserts these counts, because they are the independent half of the evidence and are exactly the sort of thing a later simplification removes without noticing. **Do not derive one channel's pointers from the other's.** `loop_start_R == loop_start_L + P/2` holds exactly on `eiv-vitous`'s 828 records and on 1.2% of `esi32-gm`'s. Read the set whose start is 92 and use it. @@ -262,9 +331,28 @@ The **end** cannot be isolated the same way: `+46` and `+30` sit six frames apar Same disc, same shape, separated only by whether the end fits. A clamped end is a loop point the disc did not state, so a record whose loop end lies past its audio yields **no** loop ([ADR-0025](../adr/0025-the-loop-is-decoded-the-root-key-is-not.md)). -That is most of why the reference disc yields fewest. On about 2 200 of `esi32-gm`'s records `end_L` runs roughly 45 frames past the payload its own length field produces, and 95% of the disc is refused on it. +That used to be most of why the reference disc yielded fewest, and the reason was not the loop rule. **The reader was 92 bytes short.** On about 2 200 of `esi32-gm`'s records `end_L` ran roughly 45 frames past the payload its own length field produced, and 95% of the disc was refused on it — because that length field is `end_R`, the right channel's end, and those records declare their audio on the left. An earlier revision of this doc left it open: *"Either the reader is 90 bytes short on those samples or the extent field means something else."* It was the reader ([ADR-0029](../adr/0029-a-record-is-closed-by-the-channel-it-declares.md)). + +With the extent taken from the channel the record declares, the same loop ends fit inside the payload **without being clamped**, and `esi32-gm` goes from 107 loops of 2 265 samples to 1 778 of 2 635, `protozoa` from 1 689 to 5 244. That the counts move is not evidence. The newly admitted loops were scored the same way as the rest, with the same controls, and they splice: + +| Disc | group | scored | shape *r* | control | join | seamless | control seamless | +|---|---|---|---|---|---|---|---| +| `esi32-gm` | loop newly admitted | 838 | **+0.87** | +0.01 | 1.25 | 87% | 36% | +| `esi32-gm` | record newly found | 130 | **+0.90** | −0.03 | 1.36 | 80% | 29% | +| `protozoa` | loop already produced | 905 | +0.85 | +0.02 | 1.35 | 83% | 38% | +| `protozoa` | loop newly admitted | 1 082 | **+0.82** | −0.01 | 1.07 | 90% | 32% | +| `protozoa` | record newly found | 330 | **+0.77** | −0.06 | 0.90 | 87% | 34% | +| `eiiix-1` | loop already produced | 413 | +0.96 | +0.02 | 1.03 | 90% | 36% | +| `eiiix-1` | record newly found | 133 | **+0.99** | −0.10 | 0.84 | 94% | 23% | +| `vintage` | record newly found | 250 | **+0.94** | −0.03 | 1.50 | 85% | 23% | + +The window is the 256 frames *before* the loop start and before the loop end rather than after them: a loop of period `P` makes `x[t] ≈ x[t − P]`, and unlike the forward pairing that one still exists on a loop running to the last frame of the sample — which here is most of them. + +The **record newly found** rows are the right-declared and zeroed-right-set records, and they answer the obvious worry about widening a signature that scans through megabytes of audio. A false hit inside PCM does not carry a loop that splices at +0.9 against a control at zero. -**Which of the two is right is open.** If the record's extent is correct, the reader is 90 bytes short on those samples. Scanning candidate ends across ±96 frames and taking the shape-test peak does *not* confirm it: the peak lands within ±2 frames of the declared end on **10%** of `esi32-gm`'s records and 20% of `protozoa`'s, where a uniform peak would give about 3%. Better than chance, nowhere near an answer. Nothing is changed on the strength of it. +The scan for a better end that an earlier revision reported — a peak within ±2 frames of the declared end on 10% of `esi32-gm`'s records against about 3% for chance — was measuring the 92-byte shortfall, and is not repeated. + +**934 of `ditto-drums`'s 948 loops span the whole sample**, starting at frame 6 and ending seven frames from the end. That is the format's "no loop" — the sample's own bounds, written with a small inset — and the guard here refuses a whole-extent loop only where it starts at exactly 0. It is not new and not this deliverable's to fix: `eiiix-1` ships 460 of them and `eiiix-2` 320, and folding a second loop rule in would move the loop counts for two unrelated reasons at once. ### There is no root key @@ -366,22 +454,29 @@ end_L + 2 == start_R `P` is the payload the record's own length field produces, and `end_L + 2` is where the left channel closes — the pointer names the first byte of the last word, so the extent runs two bytes past it. -**The third condition is not decoration.** 2 721 records across the seven discs satisfy the first two and **65 fail the third**: 19 on `protozoa`, 40 on `eiiix-1`, 6 on `eiiix-2`. They declare a left channel that overlaps the right block — `protozoa` writes `end_L` exactly 8 bytes past `start_R` on 18 records — or one that stops well short of it, as `eiiix-1`'s `LP Up Stroke` does at 24 766 bytes of a 36 402-byte half. +**The third condition used to reject 65 records, and it now rejects none — because those 65 were a symptom of a different bug.** An earlier revision recorded that 2 721 records across seven discs satisfied the first two conditions and 65 failed the third: 19 on `protozoa`, 40 on `eiiix-1`, 6 on `eiiix-2`, each declaring a left channel that overlapped the right block or stopped well short of it. Every one of them was sized from `end_R` — the right channel's end — on a record that declares its audio on the left, so `P` came out at twice the record's length and `start_R` landed on `start_L + P/2` by arithmetic. Given the extent the record actually declares, none of the ten discs holds a contradicted record ([ADR-0029](../adr/0029-a-record-is-closed-by-the-channel-it-declares.md)). + +`protozoa`'s trombones are the case that closes it. `Trom B2`, `Trom E3` and `Trom A3` are each written in two banks, and this doc recorded that the **first half of each is byte for byte a whole one-channel record of the same name** in `Vintage PresetsX` — 16 756 bytes against a 33 512-byte payload — with nothing on the disc matching the second half. With the right extent, `Proteus1PresetsX`'s `Trom B2` is 16 764 bytes and is **byte-identical to `Vintage PresetsX`'s and `Vintage InstrmtX`'s**, as are `Trom E3`, `Trom C5`, `Trom D4` and `Trom G4`. There was never a second half; the record was being read twice its length. -Those 65 are not stereo, and `protozoa` says what six of them are. `Trom B2`, `Trom E3` and `Trom A3` are each written in two banks, and in all six records the **first half is byte for byte the whole of a one-channel record of the same name** in `Vintage PresetsX` — 16 756 bytes of `Trom B2` against a 33 512-byte payload. Nothing on the disc matches the second half. The payload is twice the sound, so `start_R` lands on `start_L + P/2` by arithmetic rather than by declaration, and `end_L` is what gives it away: it closes the audio 8 bytes *past* the halfway point instead of on it. +**The condition stays.** It is what makes the two-channel test exact when the extent is computed from the pointers alone — that test is this one restated — and a gate is not deleted for going quiet. It is now exercised synthetically only, and that is recorded rather than glossed. -**2 656 records pass all three**, which is what this project writes as stereo: +**2 843 records pass all three**, which is what this project writes as stereo: | Disc | stereo | of samples | |---|---|---| -| `esi32-gm` | 28 | 2 265 | -| `eiiix-1` | 601 | 1 189 | -| `eiiix-2` | 592 | 1 333 | -| `protozoa` | 8 | 5 852 | +| `esi32-gm` | 28 | 2 635 | +| `protozoa` | 8 | 6 595 | +| `eiiix-1` | 601 | 1 248 | +| `eiiix-2` | 592 | 1 337 | +| `emu-classics` | 185 | 1 516 | +| `vintage` | 2 | 993 | +| `ditto-drums` | 0 | 948 | | `eiv-analogia` | 279 | 449 | | `eiv-studio` | 320 | 2 822 | | `eiv-vitous` | 828 | 828 | +The four discs that had stereo before have exactly the numbers they had, which is the check that changing how long a record is did not change what shape it is. + ### Measured directly, the selected halves are one performance The instrument matters here, and the obvious one is confounded. Median RMS-envelope correlation between the two halves separates the populations — @@ -411,6 +506,8 @@ The instrument matters here, and the obvious one is confounded. Median RMS-envel The confirmed records score with the positive control on all seven discs; the 65 the third condition rejects score with the negative control. `protozoa`'s eight are too few to establish anything on their own and rest on the other six discs. +Those last two rows were measured against the extent `+34` produces, and that population no longer exists — with the record sized from the channel it declares, no disc holds a contradicted record ([ADR-0029](../adr/0029-a-record-is-closed-by-the-channel-it-declares.md)). They are kept because they are what the third condition was established on: the halves it rejected were unrelated audio, measured, and that is why the condition is still there. + ### The first block is the left channel Structural: the pointer block is ordered `(start_L, start_R)` and `start_L` addresses the first block. @@ -419,7 +516,7 @@ The only content evidence is weak and agrees with it. Of `eiv-analogia`'s twelve ### A one-channel record that is really stereo does not occur here -The inverse error was looked for. **Not one of the 12 017 records that do not declare the two-channel shape declares an extent of half its payload**, which is the structural signature it would leave. The 439 whose halves correlate above 0.9 by envelope show a midpoint z-jump of −0.27 to −0.41 — no discontinuity where a block join would be — and they are the decaying single notes above. +The inverse error was looked for. **Not one of the 12 017 records that did not declare the two-channel shape declared an extent of half its payload**, which is the structural signature it would leave. The 439 whose halves correlate above 0.9 by envelope show a midpoint z-jump of −0.27 to −0.41 — no discontinuity where a block join would be — and they are the decaying single notes above. `eiiix-2` is the disc to check, because an earlier revision of this section gave it the weakest separation in the table at 0.59 for one-channel records. Under these instruments its high-envelope one-channel records score fine structure **−0.021**, which is the negative control, and its one-channel envelope median re-measures at **0.114** over 603 scored records. Both figures are recorded; the 0.59 is not reproduced by the measurement here and is not relied on either way. @@ -427,11 +524,11 @@ The inverse error was looked for. **Not one of the 12 017 records that do not de `(pointer − start) / 2` is a per-channel frame index either way: in the double-length mono file this format produced before D18 it landed in the left block, and in the interleaved file it is the frame number. All seven per-disc loop counts are unchanged across the fix, which is the check that the two decodes of one pointer block agree. -The two records anywhere whose declared loop end lies past its own channel — `Mbira A3` and `Mbira F3` on `eiiix-1` — are both among the 65 the third condition rejects, so they stay mono and keep their loops. +The two records anywhere whose declared loop end lay past its own channel — `Mbira A3` and `Mbira F3` on `eiiix-1` — were both among the 65 the third condition rejected, so they stayed mono and kept their loops. Both are ordinary one-channel records under D21's extent, and both still keep them. ### Name pairing is the other mechanism, and it is the rare one -E-IV discs *also* pair separate mono records into stereo by name, the way the rest of the collection does ([ADR-0017](../adr/0017-the-stereo-side-marker-is-a-character-class.md)). Both mechanisms are real and they do not overlap: **12** samples across all seven discs are name-paired — six pairs, all on `eiv-analogia` — against **2 656** whose own record declares two channels, and no sample is both. An earlier revision of this doc gave name pairing as the answer to how these discs do stereo, which is the rare case given as the rule. +E-IV discs *also* pair separate mono records into stereo by name, the way the rest of the collection does ([ADR-0017](../adr/0017-the-stereo-side-marker-is-a-character-class.md)). Both mechanisms are real and they do not overlap: **14** samples across all ten discs are name-paired — six pairs on `eiv-analogia` and one on `ditto-drums` — against **2 843** whose own record declares two channels, and no sample is both. An earlier revision of this doc gave name pairing as the answer to how these discs do stereo, which is the rare case given as the rule. ## The payload is little-endian @@ -451,45 +548,55 @@ Beware the near-miss: comparing "LE from *n*" against "BE from *n+1*" cannot dis |---|---| | Bank header at | 842 752 | | First sample record at | bank + 138 317 | -| Sample records | 452 | -| Distinct names | 452 | -| Total record bytes | 7 345 200 (7.00 MiB) | +| Sample records | 531 | +| Distinct names | 531 | +| Total record bytes | 8 248 316 | | Bank declares (`0x30`, `0x34`) | sample area at 138 243, run of 8 248 316 | | First record | `Piano E0`, rate 12 000, 146 852 bytes of PCM | +**The records fill the declared run exactly**, which they did not before D21: this table gave 452 records and 7 345 200 bytes, 903 116 short of a run nothing accounted for. 76 of the 531 declare their one channel on the right and were invisible to a left-hand signature; the rest were 92 bytes short each ([ADR-0029](../adr/0029-a-record-is-closed-by-the-channel-it-declares.md)). + Whole-disc listings, with the samples the record declares stereo: | Disc | Volumes | Samples | Stereo | |---|---|---|---| -| `esi32-gm` | 10 | 2 265 | 28 | -| `eiiix-1` | 46 | 1 189 | 601 | -| `eiiix-2` | 46 | 1 333 | 592 | -| `protozoa` | 16 | 5 852 | 8 | +| `esi32-gm` | 10 | 2 635 | 28 | +| `protozoa` | 16 | 6 595 | 8 | +| `eiiix-1` | 46 | 1 248 | 601 | +| `eiiix-2` | 46 | 1 337 | 592 | +| `emu-classics` | 22 | 1 516 | 185 | +| `vintage` | 16 | 993 | 2 | +| `ditto-drums` | 48 | 948 | 0 | | `eiv-analogia` | 12 | 449 | 279 | | `eiv-studio` | 230 | 2 822 | 320 | | `eiv-vitous` | 44 | 828 | 828 | -**2 656 of 14 738**, and a stereo sample is still one sample: the counts above do not move when the channel count is read, only the file's shape does. +**2 843 of 19 371**, and a stereo sample is still one sample: the counts above do not move when the channel count is read, only the file's shape does. Samples carrying a loop, of those totals: | Disc | Samples | With a loop | | |---|---|---|---| -| `esi32-gm` | 2 265 | 107 | 5% | -| `eiiix-1` | 1 189 | 1 157 | 97% | -| `eiiix-2` | 1 333 | 1 260 | 95% | -| `protozoa` | 5 852 | 1 689 | 29% | +| `esi32-gm` | 2 635 | 1 778 | 67% | +| `protozoa` | 6 595 | 5 244 | 80% | +| `eiiix-1` | 1 248 | 1 215 | 97% | +| `eiiix-2` | 1 337 | 1 264 | 95% | +| `emu-classics` | 1 516 | 1 435 | 95% | +| `vintage` | 993 | 953 | 96% | +| `ditto-drums` | 948 | 948 | 100% | | `eiv-analogia` | 449 | 449 | 100% | | `eiv-studio` | 2 822 | 2 551 | 90% | | `eiv-vitous` | 828 | 826 | 100% | -**8 039 of 14 738.** The two low ones are the discs that declare a loop end past the audio they carry, which is refused rather than clamped. +**16 663 of 19 371.** `esi32-gm`'s 5% and `protozoa`'s 29% were what a 92-byte-short extent looked like from the loop side: those discs declare a loop end that runs to the very end of the audio, so losing 46 frames off the end lost the loop with them. `ditto-drums`'s 100% is not as good as it looks — 934 of its 948 span the whole sample, which is the format's "no loop" written with a small inset, and the guard here does not yet catch that form. These are the regression baseline: any change to the shared record parser is a bug if they move. **They are now asserted by `tests/test_discs.py`, pinned by disc size** — a table in a document is a note, not a test, and two of these numbers were wrong for a release with a green suite. The suite also pins the **SHA-256 of every sample payload per disc**, because a count table cannot see a payload that shifted by a byte while staying the same length, and the **stereo counts** beside them, because the third condition of the gate is exactly the kind of thing a later simplification removes. On a stereo sample the suite additionally de-interleaves what was written and requires it to reproduce the disc's two blocks byte for byte: the audio moved, and it must be the same audio. -`esi32-gm`'s 2 424 and `protozoa`'s 6 788 are what the previous revision of this table gave, and both counted another bank's records; [ADR-0021](../adr/0021-a-bank-owns-the-run-its-header-declares.md) has the accounting. `esi32-gm` is the instructive one: it was believed clean, and its last bank ran to the end of the image and was credited with 193 records belonging to the two banks in front of it. +**These are D21's numbers and the previous revision's are all wrong**, on every EIII/ESI disc: `esi32-gm` 2 265, `protozoa` 5 852, `eiiix-1` 1 189 and `eiiix-2` 1 333, each of them a record extent taken from the wrong channel's end pointer ([ADR-0029](../adr/0029-a-record-is-closed-by-the-channel-it-declares.md)). The three E-IV rows are unchanged and are the control: those discs size a record from their own big-endian directory and never read `+34`. + +`esi32-gm`'s 2 424 and `protozoa`'s 6 788 are what the revision before *that* gave, and both counted another bank's records; [ADR-0021](../adr/0021-a-bank-owns-the-run-its-header-declares.md) has the accounting. `esi32-gm` is the instructive one: it was believed clean, and its last bank ran to the end of the image and was credited with 193 records belonging to the two banks in front of it. -Each of the four EIII/ESI discs lists one index bank with a note and no samples, which is why their volume counts run one ahead of the banks that extract; `esi32-gm`, `eiiix-1` and `eiiix-2` also list the sampler's own code banks — `E3 Main Code`, `E3X Main Code` — which carry no bank header and are noted as such. On `eiv-studio`, 100 of the 230 banks have no confirmed sample directory and are listed with a note rather than guessed at. That disc carries 901 `E4P1` presets, and preset-only banks are the likely explanation — it is not established, so it is not claimed. +Each of the seven EIII/ESI discs lists one index bank with a note and no samples, which is why their volume counts run one ahead of the banks that extract; `esi32-gm`, `eiiix-1` and `eiiix-2` also list the sampler's own code banks — `E3 Main Code`, `E3X Main Code` — which carry no bank header and are noted as such. On `eiv-studio`, 100 of the 230 banks have no confirmed sample directory and are listed with a note rather than guessed at. That disc carries 901 `E4P1` presets, and preset-only banks are the likely explanation — it is not established, so it is not claimed. ## Traps @@ -507,12 +614,15 @@ Each of the four EIII/ESI discs lists one index bank with a note and no samples, - Each folder's bank directory must be bounded by the next folder's start block; they sit two blocks apart on `eiv-studio`. - On E-IV the record's own length field is unusable and the directory's is authoritative. The EIII rule matches 0 of 5 349 consecutive pairs. - An E-IV sample directory can appear twice. Deduplicate by address or every one of its records is listed twice. -- The paired length fields **are** a channel count, and the measurement that said otherwise tested interleaved stereo when the format splits into blocks. 2 656 samples are stereo. -- A channel count is not enough on its own. 65 records declare `start_R` half a payload on and then close `end_L` somewhere else; their halves are unrelated audio, and on `protozoa` the second half is another bank's record. Require `end_L + 2 == start_R`. +- The paired length fields **are** a channel count, and the measurement that said otherwise tested interleaved stereo when the format splits into blocks. 2 843 samples are stereo. +- A channel count is not enough on its own. Require `end_L + 2 == start_R`. It rejected 65 records when the extent came from `+34`; it rejects none now, and it is what makes the extent's own two-channel test exact. - The two halves of a decaying note correlate at 0.94 by RMS envelope, so that measure cannot tell a stereo pair from a single note. Divide the envelope by its own trend, or correlate the waveform at a lag. -- Name pairing is the *rare* mechanism here, not the rule: 12 samples across seven discs against 2 656 declaring two channels in the record. +- Name pairing is the *rare* mechanism here, not the rule: 14 samples across ten discs against 2 843 declaring two channels in the record. - The sample record's fields are at `+22`, `+26`, `+30` … `+50`, not at `+24`, `+28`, `+32` … A four-byte stride begun at `+18` straddles two real fields at every step and reads as nine-digit noise. -- `+22` is a start pointer, not a header length. It reads 0 where a record declares no left channel, and requiring 92 drops a fifth of `eiv-studio`. +- `+22` is a start pointer, not a header length. It reads 0 where a record declares no left channel, and requiring 92 drops a fifth of `eiv-studio` and every right-declared record on an EIII disc — 1 371 of them. +- **`+34` is not the record length.** It is the right channel's end, and it closes the record only where the right-hand set describes it: it is 92 bytes short on a mirror-92 record, zero on a record with the side unused, and a memory-frame address on some banks. Read the end of the set whose start is 92 ([ADR-0029](../adr/0029-a-record-is-closed-by-the-channel-it-declares.md)). +- A record may declare its one channel on the **right** on an EIII disc too, not only on E-IV. Scan for 92 at `+26` as well as at `+22`, or `vintage`'s `Juno Synths` reads as an empty bank. +- A bank's declared run is an independent check on the record extent, and it was already in this doc reading as a loose fit. If the last record of a bank stops one 92-byte header short of `0x30 + 74 + 0x34`, the extent is wrong, not the run. - A loop end past the payload must be **refused**, not clamped back the way AKAI and Roland clamp theirs. Clamping turns a splice correlation of +0.86 into −0.10 on `protozoa`'s own records. - There is no root key in the sample record. No byte tracks the note in the sample's name above chance, and `+58` — the field that looks most like one — is the sample rate. @@ -522,4 +632,4 @@ That disc was the awkward one throughout, and every awkwardness turned out to be Its two `4k` banks carry the third bank signature; nothing located them, so `Orbit Presets X` and `Phatt Presets X` were handed their regions and reported their records a second time. Its `Phatt Presets X` is written twice, so the copy at the end of the image was discarded as a duplicate name and `Protozoa X` ran to EOF. And every one of its banks carries the tail of a previous occupant inside its own region, which no bound between banks can reach. -The bank's own `0x30`/`0x34` answers all three, and the check is that every record the bound drops can be shown to be another bank's, at a constant shift — 264 of 264, 70 of 70, 59 of 59, 42 of 42, and so on for all fifteen located banks. `protozoa` now yields 16 volumes and 5 852 samples, with `Orbit Presets 4k` and `Phatt Presets 4K` extracting 535 and 239 under their own names where they previously listed empty. ([issue #15](https://github.com/bmxcode/samplerdisc/issues/15), [ADR-0021](../adr/0021-a-bank-owns-the-run-its-header-declares.md)) +The bank's own `0x30`/`0x34` answers all three, and the check is that every record the bound drops can be shown to be another bank's, at a constant shift — 264 of 264, 70 of 70, 59 of 59, 42 of 42, and so on for all fifteen located banks. `protozoa` now yields 16 volumes and 6 595 samples, with `Orbit Presets 4k` and `Phatt Presets 4K` extracting 558 and 255 under their own names where they previously listed empty. (Those were 5 852, 535 and 239 between ADR-0021 and ADR-0029; the difference is the record extent, not the bank bound.) ([issue #15](https://github.com/bmxcode/samplerdisc/issues/15), [ADR-0021](../adr/0021-a-bank-owns-the-run-its-header-declares.md)) diff --git a/src/samplerdisc/extract.py b/src/samplerdisc/extract.py index bbb8627..c8ab689 100644 --- a/src/samplerdisc/extract.py +++ b/src/samplerdisc/extract.py @@ -252,8 +252,8 @@ def extract_volume( if channels == 1: # Only mono files can be halves of an -L/-R pair: ``interleave`` # takes two mono buffers, and a sample that is already stereo is - # not one side of anything. No sample on the seven E-mu discs is - # both -- 2 656 declare two channels, 12 are name-paired, and the + # not one side of anything. No sample on the ten E-mu discs is + # both -- 2 843 declare two channels, 14 are name-paired, and the # two sets do not intersect -- so this changes nothing today and # is here so it cannot start to (ADR-0026). parsed[entry.name] = sample diff --git a/src/samplerdisc/fs/emu3.py b/src/samplerdisc/fs/emu3.py index 6b925a4..c86819b 100644 --- a/src/samplerdisc/fs/emu3.py +++ b/src/samplerdisc/fs/emu3.py @@ -105,9 +105,9 @@ OFF_BANK_SAMPLE_BYTES = 0x34 #: Sample record, relative to its own start. A record begins two bytes before -#: its name; those two bytes are zero on every record after the first. +#: its name; those two bytes are zero on every record after the first on the +#: EIII discs, and are not on `ditto-drums`, so nothing is tested on them. SAMPLE_NAME_OFFSET = 2 -OFF_SAMPLE_RECORD_LEN = 34 OFF_SAMPLE_RATE = 54 #: The eight-pointer block, and the reason the fields either side of it read as @@ -124,9 +124,12 @@ #: ``OFF_SAMPLE_START_R`` instead, which is a different value of a working #: field rather than a broken one. #: -#: ``OFF_SAMPLE_END_R`` and ``OFF_SAMPLE_RECORD_LEN`` are one field under two -#: names, and that also explains ``RECORD_LEN_BIAS``: the pointer names the -#: first byte of the *last* word, so the record ends two bytes further on. +#: An earlier revision of this module gave ``+34`` a second name, +#: ``OFF_SAMPLE_RECORD_LEN``, and took the record's extent from it +#: unconditionally. It is the **right channel's** end, and it closes the record +#: only where the right-hand set is what describes this record's audio. Four +#: shapes on seven discs say otherwise, and ``record_extent`` below is the rule +#: that replaced it (ADR-0029). OFF_SAMPLE_START_L = 22 OFF_SAMPLE_START_R = 26 OFF_SAMPLE_END_L = 30 @@ -154,10 +157,10 @@ ("start_r", "end_r", "loop_start_r", "loop_end_r"), ) -#: Records sit back to back and each declares its own length two short of the -#: distance to the next, verified across a 12-record chain. See the pointer -#: block above for why two: ``+34`` addresses the last word, not past it. -RECORD_LEN_BIAS = 2 +#: An end pointer names the first byte of the *last* word rather than one past +#: it, so the extent it closes runs two bytes further on. ``END_POINTER_BIAS`` +#: in sample/emu3.py is the same fact seen from the loop side. +END_POINTER_BIAS = 2 #: Every EIII/ESI sample header measured is this long. Treated as a validity #: check rather than a constant, since it is read from the record. @@ -247,6 +250,66 @@ def sample_pointers(head: bytes) -> tuple[tuple[str, int], ...]: ) +def record_extent(head: bytes) -> int | None: + """How far the record runs from its own start, or ``None`` if it says nothing. + + **The set that opens the audio is the set that closes it.** A record's + audio begins immediately after its 92-byte header, so the pointer set + describing it starts at ``SAMPLE_HEADER_LEN`` -- which is the rule + ``sample/emu3.py`` already used to pick a loop, and never applied to the + extent. Taking ``+34`` as the record length instead is right only where the + right-hand set happens to describe this record, and four shapes across the + seven EIII/ESI reference discs say it does not (ADR-0029): + + * ``start_r == start_l - 92`` with ``end_r == end_l - 92`` -- the same + channel written from the payload's start rather than the record's. The + extent taken from ``+34`` is **92 bytes short**, on 2 127 records of + `esi32-gm` and 3 965 of `protozoa`. + * ``start_r == end_r == 0`` -- the unused side zeroed. ``+34`` gives an + extent of 2, which is shorter than the header, so the record is rejected + outright: 872 records on `ditto-drums`, nine of its ten silent banks. + * ``start_r == 92 + F`` and ``end_r == 92 + 2F - 2`` for a constant memory + frame ``F`` -- 1 MiB on `emu-classics`, 2 MiB on `eiiix-1`. ``+34`` then + names an address past the whole bank region and the record is dropped as + unreadable. + * the one channel declared on the **right**: ``start_r == 92`` and the + left set not opening the audio, which on 1 371 of the 1 429 is + ``start_l == 0`` exactly. docs/formats/emu3.md records 542 of + `eiv-studio`'s that way; the EIII walk never looked, so 353 on + `esi32-gm`, 607 on `protozoa` and all of `vintage`'s ``Juno Synths`` + were never listed at all. + + The exception is a genuine two-channel record, where the payload is both + blocks and the far end is the right channel's: the right block opens + exactly where the left one closes and the two are the same length. That is + ADR-0026's gate stated from the pointer side instead of the payload side -- + the same three conditions, without needing the size the caller is asking + for, so there is no circularity here. + + ``None`` for a header too short to hold the block, or one where neither set + opens the audio. Both are refusals rather than guesses: a record that was + not read must not present as one that declared nothing. + """ + if len(head) < OFF_SAMPLE_END_R + 4: + return None + start_l, start_r, end_l, end_r = struct.unpack_from("<4I", head, OFF_SAMPLE_START_L) + if ( + start_l == SAMPLE_HEADER_LEN + and start_r == end_l + END_POINTER_BIAS + and end_r - start_r == end_l - start_l + ): + return end_r + END_POINTER_BIAS + # Both sets open the audio on many EIII records, and then they disagree + # about where it ends by a few bytes in either direction. The stride to the + # next record follows the larger on every disc measured. + declared = [ + end for start, end in ((start_l, end_l), (start_r, end_r)) if start == SAMPLE_HEADER_LEN + ] + if not declared: + return None + return max(declared) + END_POINTER_BIAS + + class _Bank(NamedTuple): name: str folder: str @@ -788,10 +851,19 @@ def _samples( records are found rather than followed. The signature is specific enough to survive a scan through megabytes of - audio: the header-length field must equal exactly ``SAMPLE_HEADER_LEN``, - the rate must be plausible, and sixteen bytes must be printable. On the - reference bank that yields 452 records with 452 distinct, sensible - names totalling 7.00 MiB inside a bank declaring 8 MiB. + audio: one of the two start pointers must open the audio at exactly + ``SAMPLE_HEADER_LEN``, the rate must be plausible, and sixteen bytes + must be printable. On the reference bank that yields 531 records with + 531 distinct, sensible names totalling 8 248 316 bytes, which is that + bank's declared run to the byte. + + **Either** start, not only the left one. A record may declare its one + channel on the right, with the left zeroed, and scanning for the left + pointer alone makes those invisible rather than wrong -- all of + `vintage`'s ``Juno Synths``, and 980 records across the four EIII/ESI + reference discs that were never listed (ADR-0029). The two anchors are + deduplicated by address, and the result is yielded in address order so + that which anchor found a record cannot change what is written. Specific is not the same as exclusive, which is why ``span`` matters as much as the signature does. A bank's region holds whatever the @@ -804,41 +876,46 @@ def _samples( first, last = span window = image.read(offset + bank_at, max(limit - bank_at, 0)) needle = struct.pack(" len(window): - continue - yield File( - name=name, - kind="sample", - size=record_len - header_len, - start_block=bank_at + at + header_len, - raw_type=rate, - meta=sample_pointers(window[at : at + SAMPLE_HEADER_LEN]), - ) - - def _parse_record(self, head: bytes) -> tuple[str, int, int, int] | None: + for anchor in (OFF_SAMPLE_START_L, OFF_SAMPLE_START_R): + at = match.start() - anchor + if at < 0 or at in found or not first <= at < last: + continue + record = self._parse_record(window[at : at + SAMPLE_HEADER_LEN]) + if record is None: + continue + name, extent, rate = record + if at + extent > len(window): + continue + found[at] = File( + name=name, + kind="sample", + size=extent - SAMPLE_HEADER_LEN, + start_block=bank_at + at + SAMPLE_HEADER_LEN, + raw_type=rate, + meta=sample_pointers(window[at : at + SAMPLE_HEADER_LEN]), + ) + for at in sorted(found): + yield found[at] + + def _parse_record(self, head: bytes) -> tuple[str, int, int] | None: raw = head[SAMPLE_NAME_OFFSET : SAMPLE_NAME_OFFSET + ENTRY_NAME_LEN] if not is_plausible_name(raw): return None name = decode_name(raw) if not name: return None - (header_len,) = struct.unpack_from(" bytes: return image.read(offset + entry.start_block, entry.size) diff --git a/src/samplerdisc/sample/emu3.py b/src/samplerdisc/sample/emu3.py index 245be2c..3d86e32 100644 --- a/src/samplerdisc/sample/emu3.py +++ b/src/samplerdisc/sample/emu3.py @@ -13,9 +13,10 @@ The same block declares a **channel count**, and where it declares two the payload is a *block* split -- all of the left channel, then all of the right, not interleaved. Read as one mono stream that is a file twice as long as the -sound, which is what this project shipped for 2 656 of the 14 738 E-mu samples -until D18. The two halves are interleaved here, in the sample layer, so what -``pcm`` holds is what a WAV data chunk holds either way (ADR-0026). +sound, which is what this project shipped for 2 656 of the E-mu samples until +D18 (2 843 of 19 371 under D21's record extent). The two halves are +interleaved here, in the sample layer, so what ``pcm`` holds is what a WAV data +chunk holds either way (ADR-0026). The endianness is worth a note, because it was got wrong first. Sampling this data at 2048-byte sector boundaries makes it read as big-endian, convincingly @@ -42,12 +43,12 @@ class NotASample(_NotASample): DATA_START = 92 #: An end pointer names the first byte of the *last* word rather than one past -#: it, so the extent it closes runs two bytes further. ``RECORD_LEN_BIAS`` in -#: fs/emu3.py is the same fact, seen from the record-length side. +#: it, so the extent it closes runs two bytes further. ``END_POINTER_BIAS`` in +#: fs/emu3.py is the same constant, used there to close the record itself. END_POINTER_BIAS = 2 #: A stereo payload splits into two equal blocks, so it holds a whole number of -#: frames on both channels only when it divides by four. Every one of the 2 656 +#: frames on both channels only when it divides by four. Every one of the 2 843 #: records the gate below selects does; the check is here because a payload #: that did not would be split half a sample out and sound like tape hiss. STEREO_ALIGNMENT = 4 @@ -153,7 +154,7 @@ def _is_block_split(pointers: dict[str, int], size: int) -> bool: `eiiix-1`, 6 on `eiiix-2` -- declaring a left channel that overlaps the right block or stops short of it. They are not stereo: their halves score 0.01 on fine structure and 0.01 on best-lag correlation, which is the - negative control of two unrelated records, while the 2 656 that pass score + negative control of two unrelated records, while the 2 843 that pass score with the known-true stereo pairs ADR-0017 joins by name. Six of `protozoa`'s are identified exactly: the first half of each is, byte for byte, the whole of a one-channel record of the same name in another bank, @@ -162,6 +163,14 @@ def _is_block_split(pointers: dict[str, int], size: int) -> bool: those come out with an unaccounted-for second sound in the right channel, and two `eiiix-1` records declare a loop that then ends past their own left channel. + + This test and ``fs/emu3.py``'s ``record_extent`` are one statement seen + from two sides. That one, from the pointer block alone, decides how long + the record is; this one, given the payload that produced, decides how to + read it. They have to agree, or a record sized as two channels comes out + written as one. Nothing in the code forces that -- the per-disc **stereo + counts** in tests/test_discs.py are what hold them together, which is why + they are pinned separately from the sample counts (ADR-0029). """ if size % STEREO_ALIGNMENT: return False diff --git a/tests/fixtures.py b/tests/fixtures.py index 5430d8f..fb69c1a 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -914,7 +914,7 @@ def _emu3_pointers(head: bytearray, payload_bytes: int, loop, stereo: bool = Fal ``stereo`` writes the two-channel form: the payload is a block split, all of the left channel then all of the right, and both pointer sets are written -- the right one a mirror of the left, half a payload on. That is - what 2 656 records across the seven reference discs hold (ADR-0026). + what 2 843 records across the ten reference discs hold (ADR-0026). """ from samplerdisc.fs.emu3 import ( OFF_SAMPLE_END_L, @@ -1017,6 +1017,7 @@ def emu3_disc( EIV_CHAIN_STRIDE, EIV_MAGIC, EIV_RECORD_OFFSET, + END_POINTER_BIAS, ENTRY_LEN, MAGIC, OFF_BANK_SAMPLE_BYTES, @@ -1025,10 +1026,9 @@ def emu3_disc( OFF_EIV_LENGTH, OFF_EIV_NAME, OFF_EIV_POSITION, + OFF_SAMPLE_END_R, OFF_SAMPLE_RATE, - OFF_SAMPLE_RECORD_LEN, OFF_SAMPLE_START_L, - RECORD_LEN_BIAS, SAMPLE_AREA_PREAMBLE, SAMPLE_HEADER_LEN, ) @@ -1124,7 +1124,7 @@ def name16(text: str) -> bytes: head = bytearray(SAMPLE_HEADER_LEN) head[2:18] = name16(sample_name) struct.pack_into(" bytes: head = bytearray(SAMPLE_HEADER_LEN) head[2:18] = name16(stale_name) struct.pack_into(" bytes: head = bytearray(SAMPLE_HEADER_LEN) head[2:18] = name16("Older Revision") struct.pack_into(" set[int]: } +#: How much of an image is hashed to tell two of one size apart, and the digest +#: for each pinned disc that needs it. +#: +#: Size alone named a disc across the 79 images this suite was built on, and it +#: stopped doing so the moment a whole publisher's series arrived at once: `Vol. +#: 03 - Orchestral` and `Vol. 07 - E-mu Classics` are both exactly 526 723 072 +#: bytes, and both `Studio Essentials` discs are 399 077 376. Same mastering +#: run, same disc geometry, different libraries. +#: +#: The tiebreak stays a property of the disc rather than of its filename +#: (ADR-0004): the first megabyte covers the ``EMU3`` header, the folder table +#: and the first bank directory, which is what actually differs. It is +#: consulted **only** where two images share a size, so every other pin is +#: unchanged and no disc needs one until its size collides with something. +_HEAD_BYTES = 1 << 20 +_HEAD_DIGEST = { + "emu-classics": "3882c2319cc27871", + "eiv-studio": "f1f1c805136d4881", +} + + +@lru_cache(maxsize=256) +def _head_digest(path: Path) -> str: + with path.open("rb") as handle: + return hashlib.sha256(handle.read(_HEAD_BYTES)).hexdigest()[:16] + + @lru_cache(maxsize=1) def _by_size() -> dict[int, tuple[Path, ...]]: """The collection indexed by file size, built once per run.""" @@ -144,9 +171,13 @@ def _pinned_disc(label: str, size: int) -> Path: """ matches = _by_size().get(size, ()) if len(matches) > 1: - # Sizes are distinct across all 79 images measured. Two files of - # exactly one size are far more likely a disc filed twice than a - # coincidence, and picking either would make the run order-dependent. + # Two files of one size are usually a disc filed twice, and picking + # either would make the run order-dependent. Where they are genuinely + # different discs off one mastering run, _HEAD_DIGEST says which is + # which -- and a label with no digest still fails rather than guessing. + wanted = _HEAD_DIGEST.get(label) + matches = tuple(p for p in matches if _head_digest(p) == wanted) if wanted else matches + if len(matches) > 1: listed = ", ".join(str(p.name) for p in matches) pytest.fail(f"{label}: {len(matches)} images are exactly {size} bytes: {listed}") if matches: @@ -544,24 +575,37 @@ def _smpl(payload: bytes) -> tuple[int, list[tuple[int, int]]]: #: The loop counts are pinned as tightly as the sample counts, for the reason #: the noted-volume counts are on the AKAI table: a loop appearing where none #: was measured is a gate that has come loose, and one disappearing is the -#: decode silently failing on a disc nobody looked at. `esi32-gm` yields the -#: fewest by far -- 107 of 2 265 -- because that disc declares a loop end past -#: the audio it carries on almost every record, and those are refused rather -#: than clamped (ADR-0025). +#: decode silently failing on a disc nobody looked at. +#: +#: The stereo counts are D18's pin and they are the records whose own pointer +#: block declares two channels *and* closes the left one where the right one +#: opens (ADR-0026). Under D21's extent they are the same four numbers on the +#: four discs that had them -- 28, 8, 601, 592 -- which is the check that +#: changing how long a record is did not change what shape it is. The gate's +#: third condition now rejects **nothing** on any of the ten discs: the 65 +#: records it used to catch were records sized from the wrong pointer, and the +#: right size stops them looking like a split at all (ADR-0029). #: -#: The stereo counts are D18's pin and they are **not** the count of records -#: satisfying ``start_R == start_L + P/2``, which is 2 721. It is that set -#: minus the 65 whose own ``end_L`` contradicts the split -- 19 on `protozoa`, -#: 40 on `eiiix-1`, 6 on `eiiix-2` -- which measure as unrelated audio and are -#: written mono. The two numbers differing on exactly three discs is what the -#: gate's third condition is for, so a change that loses it fails here rather -#: than shipping `protozoa`'s trombones with another bank's record in the -#: right channel (ADR-0026). +#: **The four EIII/ESI rows moved in D21 and every digest with them.** The +#: record's extent came from ``+34``, which is the right channel's end pointer +#: and closes the record only where the right-hand set describes it. It does +#: not on 6 092 records of `esi32-gm` and `protozoa`, which came out 92 bytes +#: short, and it silently loses every record that declares its one channel on +#: the right. The three E-IV rows are byte-identical across the change, which +#: is what says the shared parts were not disturbed: those discs size a record +#: from their own big-endian directory and never touch ``+34`` (ADR-0029). +#: +#: `emu-classics`, `vintage` and `ditto-drums` are the discs of issue #39 and +#: the ones D21 was measured against. `ditto-drums` is the whole shape of the +#: bug on one disc: 74 samples before, 948 after. _EMU3 = { - "esi32-gm": (93_077_504, 10, 2265, 107, 28, "b7964228d84cfc50"), - "protozoa": (131_690_496, 16, 5852, 1689, 8, "ac4b74a601955ca1"), - "eiiix-1": (304_128_000, 46, 1189, 1157, 601, "c26ee8fb959b3f91"), - "eiiix-2": (304_435_200, 46, 1333, 1260, 592, "2d5d002be060cc52"), + "esi32-gm": (93_077_504, 10, 2635, 1778, 28, "35a6c6edfc6f292a"), + "protozoa": (131_690_496, 16, 6595, 5244, 8, "3530e1e972f100ab"), + "eiiix-1": (304_128_000, 46, 1248, 1215, 601, "c73db948bf4f61cb"), + "eiiix-2": (304_435_200, 46, 1337, 1264, 592, "8bfcddf6d8dbd806"), + "emu-classics": (526_723_072, 22, 1516, 1435, 185, "e1398c11a9e7cb02"), + "vintage": (527_030_272, 16, 993, 953, 2, "fbcc378dd173f96c"), + "ditto-drums": (308_121_600, 48, 948, 948, 0, "3c756586711810de"), "eiv-analogia": (293_912_576, 12, 449, 449, 279, "5d8faa38572914cb"), "eiv-studio": (399_077_376, 230, 2822, 2551, 320, "8802808655deea30"), "eiv-vitous": (532_443_136, 44, 828, 826, 828, "66c179be5b78cbd2"), @@ -670,10 +714,10 @@ def test_protozoa_gives_each_bank_its_own_records() -> None: origin = find_origin(image) assert origin is not None volumes = {v.name: v for v in origin.backend.volumes(image, origin.offset)} - assert len(volumes["Orbit Presets X"].files) == 535 - assert len(volumes["Orbit Presets 4k"].files) == 535 - assert len(volumes["Phatt Presets X"].files) == 470 - assert len(volumes["Phatt Presets 4K"].files) == 239 + assert len(volumes["Orbit Presets X"].files) == 558 + assert len(volumes["Orbit Presets 4k"].files) == 558 + assert len(volumes["Phatt Presets X"].files) == 493 + assert len(volumes["Phatt Presets 4K"].files) == 255 # The index bank: empty because the disc made it empty, and saying so # is what tells that apart from a bound that failed. assert volumes["Protozoa X"].files == [] @@ -689,6 +733,152 @@ def test_protozoa_gives_each_bank_its_own_records() -> None: ) +#: The banks of [issue #39](https://github.com/bmxcode/samplerdisc/issues/39), +#: with the samples each yields and the right-hand pointer set that hid them. +#: Twelve banks across three discs claimed a volume, returned nothing and gave +#: no reason, which is the ADR-0012 signature -- and none of them was an index +#: bank, so ``OFF_BANK_SAMPLE_BYTES`` had nothing to say about any of them. +#: +#: They are pinned by the **shape** and not only by the count, because the +#: count alone would pass again the moment a record was found for the wrong +#: reason. Each shape is a different way for ``+34`` -- the right channel's end +#: pointer -- not to describe the record it sits in (ADR-0029): +#: +#: * ``zeroed`` -- the unused side is all zeros, so ``+34`` gives an extent of +#: 2 and the record is rejected as shorter than its own header. +#: * ``frame`` -- the right-hand set names a fixed memory frame rather than +#: this record, so ``+34`` points past the whole bank region. +#: * ``right`` -- the record's one channel is declared on the right, with the +#: left zeroed, so the walk's left-hand signature never sees it. +_EMU3_SILENT_BANKS = { + "emu-classics": (("Vox Haunt X", 14, "frame"),), + "vintage": (("Juno Synths", 44, "right"),), + "ditto-drums": ( + ("TAMJAZ KIT10 X", 28, "zeroed"), + ("PERCUSSION#2 X", 42, "zeroed"), + ("TIMPANI HDML X", 3, "zeroed"), + ("TIMPANI SFML X", 3, "zeroed"), + ("VIBRAPHONE X", 4, "zeroed"), + ("MARIMBA X", 26, "zeroed"), + ("XYLOPHONE X", 12, "zeroed"), + ("CONCERT BELL X", 5, "zeroed"), + ("TUBULAR BELL X", 3, "zeroed"), + ("OCTABONS X", 8, "zeroed"), + ), +} + + +@pytest.mark.parametrize("label", sorted(_EMU3_SILENT_BANKS)) +def test_emu3_banks_that_declared_a_sample_area_and_yielded_nothing(label: str) -> None: + """Issue #39, pinned by the mechanism rather than by the totals. + + Every one of these banks declares a non-zero sample area, so none of them + could be explained the way an index bank is (ADR-0021). What they have in + common is a right-hand pointer set that says nothing about the record it + sits in, and a walk that took the record's extent from it anyway. + + The shape is asserted on the records themselves, so the day this stops + holding it says which of the three ways it stopped. + """ + from samplerdisc.fs.emu3 import SAMPLE_HEADER_LEN + + size = _EMU3[label][0] + with open_image(_pinned_disc(label, size)) as image: + origin = find_origin(image) + assert origin is not None and origin.backend.name == "emu3" + volumes = {v.name: v for v in origin.backend.volumes(image, origin.offset)} + for name, expected, shape in _EMU3_SILENT_BANKS[label]: + files = volumes[name].files + assert len(files) == expected, f"{label}/{name}: {len(files)} samples" + for entry in files: + start_l, start_r = entry.get("start_l"), entry.get("start_r") + end_r = entry.get("end_r") + if shape == "zeroed": + assert (start_l, start_r, end_r) == (SAMPLE_HEADER_LEN, 0, 0) + elif shape == "right": + assert (start_l, start_r) == (0, SAMPLE_HEADER_LEN) + else: + # A fixed allocation frame: the right-hand set opens one + # frame past the audio and closes at the end of a second, + # the same two numbers on every record of the bank. + assert start_l == SAMPLE_HEADER_LEN + frame = start_r - SAMPLE_HEADER_LEN + assert frame > 0 and frame & (frame - 1) == 0 + assert end_r == SAMPLE_HEADER_LEN + 2 * frame - 2 + + +#: Banks whose last record must end exactly at ``0x30 + 74 + 0x34``, out of the +#: banks that yield records at all. This is the independent half of D21's +#: evidence: the bank header's declared run is a different field, written by a +#: different part of the mastering, from the record's own pointer block, and +#: the two now agree to the byte on 173 of 186 banks. Before D21 they agreed on +#: 86, and 33 of the rest missed by exactly one 92-byte sample header -- which +#: is what ADR-0021 recorded as a loose fit and is really this bug. +_EMU3_RUN_ENDS = { + "esi32-gm": (7, 7), + "protozoa": (15, 15), + "eiiix-1": (44, 39), + "eiiix-2": (44, 39), + "emu-classics": (19, 16), + "vintage": (13, 13), + "ditto-drums": (44, 44), +} + + +@pytest.mark.parametrize("label", sorted(_EMU3_RUN_ENDS)) +def test_an_emu3_banks_records_fill_the_run_its_header_declares(label: str) -> None: + """The bank header and the record pointers must agree about where the + records stop. + + The bank is located from its own first record rather than by scanning the + disc: the header that owns a record is the one whose declared sample area, + plus the 74-byte preamble, lands exactly on it. That is self-checking -- + a wrong header disagrees rather than being believed -- and it means this + test says nothing about how the walk found the bank, only about whether + the two independent statements of the bank's extent match. + """ + from samplerdisc.fs.emu3 import ( + BANK_MAGICS, + OFF_BANK_SAMPLE_BYTES, + OFF_BANK_SAMPLE_START, + SAMPLE_AREA_PREAMBLE, + SAMPLE_HEADER_LEN, + ) + + back = 1 << 20 + size, expected_banks, expected_exact = _EMU3[label][0], *_EMU3_RUN_ENDS[label] + with open_image(_pinned_disc(label, size)) as image: + origin = find_origin(image) + assert origin is not None and origin.backend.name == "emu3" + banks = exact = 0 + for volume in origin.backend.volumes(image, origin.offset): + if not volume.files: + continue + first = min(f.start_block for f in volume.files) - SAMPLE_HEADER_LEN + window = image.read(origin.offset + max(first - back, 0), min(first, back)) + base = max(first - back, 0) + located = None + for at in range(len(window)): + if not any(window.startswith(magic, at) for magic in BANK_MAGICS): + continue + head = window[at : at + OFF_BANK_SAMPLE_BYTES + 4] + if len(head) < OFF_BANK_SAMPLE_BYTES + 4: + continue + area, run = struct.unpack_from(" bytes: + """A 92-byte record header carrying just the four extent pointers.""" + head = bytearray(SAMPLE_HEADER_LEN) + for offset, value in ( + (OFF_SAMPLE_START_L, start_l), + (OFF_SAMPLE_END_L, end_l), + (OFF_SAMPLE_START_R, start_r), + (OFF_SAMPLE_END_R, end_r), + ): + struct.pack_into("