ISO 9660: convert the AIFF, and write each sound once

CHANGELOG.md

@@ -6,6 +6,20 @@ Notable changes to `samplerdisc`. Format-level findings live in [docs/formats/](
 
 ### Added
 
+- **AIFF payloads are converted, not copied out as `.aiff`.** An AIFF's samples are big-endian and a WAV's are little-endian, so the bytes within each value are reversed and the values are left alone — a re-ordering, exactly reversible, with no resampling or change of depth. Root key, tuning and the sustain loop come across from the AIFF's `INST` and `MARK` chunks into the WAV's `smpl`. ([docs/formats/aiff.md](docs/formats/aiff.md), [ADR-0024](docs/adr/0024-the-aiff-twin-is-converted-and-deduplicated.md))
+
+  AIFF-C is refused rather than guessed at — its payload may be compressed, and compressed data written out as PCM opens, plays as noise and reports nothing wrong. 8-bit is refused too: AIFF stores it signed and WAV unsigned, so carrying it would mean adding 128 to every sample, which is a change to the values and not to their order. Nothing in the collection is either.
+
+  **The loop convention was not guessed.** An AIFF marks its loop with two `MARK` positions and the spec does not say whether the frame at the second is played. The Best Service ProSamples discs carry a WAV of every sound beside the AIFF, and a WAV states its loop in a `smpl` chunk where the end *is* inclusive — so the two files answer the question about each other. On **195 of 195** pairs carrying a loop on both sides, the end marker is exclusive; on all **198** carrying an `INST`, the root key matches exactly.
+
+- **Duplicate audio on a disc is written once.** The thirteen ISO 9660 ProSamples discs ship each sound twice, as a full AIFF tree beside a full WAV tree — 7 926 WAV and 7 498 AIFF, of which **6 033 hold audio already coming out as a WAV**. Those are now reported as duplicates naming the file that holds the audio, instead of being written a second time. ([ADR-0024](docs/adr/0024-the-aiff-twin-is-converted-and-deduplicated.md))
+
+  Matched on the audio, never on the filename, and `vol.43` is why: its 1 386 AIFF all share a name with a WAV and **not one shares its audio**, being mastered a few frames longer — 17 638 bytes against 17 616 on `43e-01chh01`. A name-based rule would have discarded that whole disc and said nothing.
+
+  A twin is also kept where it carries something the written file lacks. On **314** pairs the AIFF has a root key and a loop and the WAV has no `smpl` chunk at all; the audio is the same and the files are not.
+
+- **EXS24 and HALion instruments are kept by `--keep-originals`.** `.exs`, `.fxp` and `.fxb` classify as programs, joining the vocabulary AKAI programs already used — they hold the key ranges and envelopes a WAV cannot carry, which is the argument [ADR-0011](docs/adr/0011-the-deliverable-is-daw-ready-wav.md) already made, and they are the shape [ConvertWithMoss](https://github.com/git-moss/ConvertWithMoss) reads. Roughly 2 200 across the collection, previously dropped.
+
 - **Every partition of an AKAI disc is read, not just the first.** An AKAI disc is a disk image — several partitions laid end to end — and the walk stopped at the one the origin resolved to. Across the 44 AKAI discs on the shelf that is the difference between **448 volumes and 14 670 files** and **2 154 volumes and 68 997 files**; the collection goes from 872 volumes and 56 662 files to **2 578 and 110 989**. `Loop Soup` alone goes from 7 volumes to 60. ([docs/formats/akai-fs.md](docs/formats/akai-fs.md), [ADR-0023](docs/adr/0023-partitions-come-from-the-table-the-disc-declares.md), [#22](https://github.com/bmxcode/samplerdisc/issues/22))
 
   The partitions are not guessed at. The disk declares them, in a table at `0x4500` of the first partition: a count, that many sizes in blocks, then the disk's total. All 44 discs carry one and on all 44 the sizes sum to the total. A partition is read only where a header sits at the position the table gives and restates the size the table gave it.
@@ -47,8 +61,17 @@ Notable changes to `samplerdisc`. Format-level findings live in [docs/formats/](
 - **The bank-count baselines are asserted rather than written down.** The whole-disc figures in the format doc are now a table in `tests/test_discs.py`, pinned by disc size. They were the stated regression guard for the shared record parser and nothing checked them, which is how two of them shipped wrong.
 - **A bank with no header on an EIII/ESI disc says so in its own terms.** `E3 Main Code` and `E3X Main Code` — the sampler's operating system, occupying a bank slot — were told they had "no sample directory", naming an E-IV structure those discs do not use.
 
+- **Every audio file copied off an ISO 9660 disc was reported as 0 Hz and 0 frames.** Nothing read the payload's header, so 15 424 files across thirteen discs came back with a manifest entry that said nothing about the audio. `wav.read_header` now reads the real rate and length. It walks to the end of the file rather than stopping at `data`, because a `smpl` chunk is written after the audio as often as before it.
+- **An original kept from an ISO 9660 disc was named twice over.** `original_suffix` appends the suffix a backend supplies, which is right for a sampler filesystem — names there carry no extension — and gave `BONGOS M.exs.exs` for a filesystem whose names already do. The ISO 9660 backend now supplies the file's own extension, and the suffix is not appended when the name already ends in it. Without both halves every kept `.exs` landed in `original/` renamed `.bin`: the bytes survived and nothing would open them.
+- **A clean disc reported itself as a damaged one.** `Skipped` covered both an entry lost to damage and an entry deliberately not written, and the summary called every one of them damage — `vol.42` printed *"skipped 423 damaged or unreadable entries"* when all 423 were sounds already written under another name. Duplicates are now counted and reported apart, in the summary and in the batch manifest.
+- **`--keep-originals` no longer calls its output "AKAI files"** in the summary. An ISO 9660 disc keeps EXS24 and HALion instruments through the same path.
+
 ### Known limits
 
+- **The ISO 9660 directory hierarchy is not preserved.** A disc's audio is written flat into one directory per volume, so `PS-34 AIFF …/056_Ballad de Boo/34a-bas-56Dmin.aif` becomes `34a-bas-56Dmin.wav` and the folder that grouped it by tempo is gone. Nothing is lost on these discs — Best Service named every file uniquely, and all thirteen list zero collisions after flattening — but a disc that reused a basename would rely on `unique_path` suffixing it.
+
+- **AIFF-C is refused, and no disc exercises a reader for it.** Nothing in the collection is AIFF-C, so there is nothing to check one against; a compressed payload written out as PCM would open, play as noise, and report nothing wrong. Same for 8-bit AIFF, where the sign convention differs from WAV's and carrying it would change the sample values.
+
 - **`AMG - Kickin' Lunatic Beats 2 AKAI CD1.mdx` is an incomplete image, and nine of the 669 files it yields are wrong.** It is short of the disc it was made from by four 32 KB blocks, so everything past the first gap has slid: the last nine files of `13-TRACK 06` extract audio belonging to other samples, with payload headers that no longer match the names their directory gives them. The container decodes every block the file does contain correctly — the file is not a complete copy of the disc, which two independent structures agree on. Nothing detects this yet; that is [#23](https://github.com/bmxcode/samplerdisc/issues/23). If you have extracted that disc, treat its last volume with suspicion, and prefer a fresh rip.
 - **Only the first partition of an AKAI disc is read.** Partitions are laid end to end at multiples of the size declared at `0x00`, and they hold real content — `Advance Orchestra` declares 7680 blocks of an image of 66 616, with a second partition's volume directory sitting right where the first ends. Most AKAI discs therefore list a fraction of what they hold. The format doc has said to walk the partition table since it was written and nothing does; the measurement is now in it, and the work is [#22](https://github.com/bmxcode/samplerdisc/issues/22).
 - **100 of `Studio Essentials`'s 230 banks list without extracting.** They have no confirmed sample directory, and carry a note saying so rather than being guessed at. That disc holds 901 `E4P1` presets and preset-only banks are the likely explanation — likely is not established, so it is not claimed.

README.md

@@ -65,6 +65,8 @@ The audio is a byte-for-byte copy: AKAI stores signed 16-bit little-endian PCM a
 
 **Filesystems** — AKAI S1000/S3000 family, E-mu `EMU3` (EIIIX, ESI-32/4000, Emulator IV), Roland `S770 MR25A` (S-770, S-750, S-760), and plain ISO 9660 for discs whose payload is already WAV or AIFF.
 
+**Audio payloads** — a WAV inside an ISO 9660 disc is copied out untouched. An AIFF is carried to WAV: its samples are big-endian and a WAV's are little-endian, so the bytes within each value are reversed and the values are left alone. Root key, tuning and loop points come across from the AIFF's `INST` and `MARK` chunks into the WAV's `smpl`.
+
 Compressed `.mdx` is the piece no other open-source tool reads today. The format is documented byte by byte in [docs/formats/mdx.md](docs/formats/mdx.md), along with [`.nrg`](docs/formats/nrg.md), [raw CD sectors](docs/formats/rawcd.md), the [AKAI](docs/formats/akai-fs.md), [E-mu](docs/formats/emu3.md) and [Roland S-7xx](docs/formats/roland-s7xx.md) filesystems, and [audio CDs](docs/formats/audio-cd.md).
 
 ## Tested against
@@ -90,6 +92,25 @@ Every WAV was checked against the disc it came from — **67 of 67 discs match e
 
 The five Roland S-7xx discs contribute **6 392 samples and 1 341 stereo pairs, with nothing skipped**.
 
+### Best Service ProSamples
+
+A third collection, [29 discs](https://archive.org/details/best-service-pro-samples-vol.-12-dance-vocals-akai-1-cd), added after the two above and measured separately. Sixteen are AKAI and thirteen are ISO 9660 discs holding plain audio.
+
+| | |
+|---|---|
+| Discs converted | 29 of 29 |
+| Samples | 21 435 |
+| Stereo pairs rejoined | 4 930 |
+| WAV files written | 26 365 |
+| Duplicate AIFF suppressed | 5 719 |
+| Entries skipped (damage) | 0 |
+
+Every one of the 26 365 files written back reads as a valid WAV with audio in it. The single entry not written that was not a duplicate is a stereo pair on `vol.20` whose two halves declare different rates — 44 033 against 44 100 — so the joiner refuses to fuse them and both mono halves are written instead.
+
+The thirteen ISO 9660 discs ship each sound twice, once as AIFF and once as WAV. 6 033 of the 7 498 AIFF hold audio that is already coming out as a WAV, and those are written once; the other 1 779 are converted, along with 314 whose AIFF carries a root key or a loop that its WAV twin does not. `vol.43` is the reason the check is on the audio and not the filename: its 1 386 AIFF all share a name with a WAV and **none of them share its audio**, being mastered a few frames longer ([ADR-0024](docs/adr/0024-the-aiff-twin-is-converted-and-deduplicated.md)).
+
+These discs are also the only place in this project where the correct output is known independently. Where a sound exists as both, the publisher's own WAV says what the AIFF conversion should produce — including the loop convention the AIFF spec leaves open, settled on 195 pairs out of 195.
+
 ## What doesn't work yet
 
 - **Roland S-550, Ensoniq and Kurzweil filesystems.** `Roland LCD1.iso` opens `* ROLAND S-550 *`, which shares nothing with the S-7xx format ([ADR-0014](docs/adr/0014-one-backend-per-on-disc-format.md)); neither archive holds a second specimen to check a backend against. The container layer opens all of these, so `export-iso` gets you the sectors meanwhile. Each backend is a self-contained module ([ADR-0003](docs/adr/0003-brand-neutral-pluggable-backends.md)), so adding one touches nothing else.
@@ -99,7 +120,8 @@ The five Roland S-7xx discs contribute **6 392 samples and 1 341 stereo pairs, w
 - **A `.mds`/`.mdf` pair is read, but its descriptor is not parsed.** One pair has now been through it end to end, and it reads the `.mdf` and sniffs its geometry rather than parsing the `.mds` — correct for a single-track data disc, which is what these are. A multi-track or offset image would be read from byte 0 and come out wrong; the track table is unread. Please open an issue if you have such a disc.
 - **`CUES` chunks in NRG** are not parsed; only `CUEX`. No disc using the older form was to hand to check the layout against.
 - **An audio CD with no cue sheet cannot be split into tracks.** `samplerdisc info` tells you when a disc's content looks like Red Book audio, and `extract --assume-audio-cd` writes the whole stream as one WAV, but the track boundaries live in a cue, not in the bytes.
-- **AIFF payloads on ISO 9660 discs are copied, not converted** — they come out as `.aiff`.
+- **AIFF-C, and 8-bit AIFF, are refused rather than read.** AIFF-C may be compressed, and compressed data written out as PCM plays as noise while reporting nothing wrong. 8-bit AIFF is signed where 8-bit WAV is unsigned, so carrying it would change every sample value rather than reorder its bytes — which is the one thing this tool does not do ([ADR-0024](docs/adr/0024-the-aiff-twin-is-converted-and-deduplicated.md)).
+- **EXS24 and HALion instruments are kept, not read.** `--keep-originals` writes the `.exs` and `.fxp` files out byte for byte; turning them into a playable instrument is ConvertWithMoss's job.
 - **AKAI S900, floppy images and the DD partition.** Use [akaiutil](https://sourceforge.net/projects/akaiutil/).
 
 ## If a disc doesn't work

docs/README.md

@@ -55,6 +55,7 @@ All of that vanishes into a working parser. Six months on, the code says *what*
 | D13 | Roland `S770 MR25A` backend ([ADR-0016](adr/0016-the-s7xx-hierarchy-is-located-not-walked.md), [ADR-0017](adr/0017-the-stereo-side-marker-is-a-character-class.md), [ADR-0018](adr/0018-the-s7xx-sample-rate-is-measured.md)) | done |
 | D14 | E-mu Emulator IV bank extraction ([ADR-0020](adr/0020-read-e-iv-through-its-sample-directory.md)) | done |
 | D15 | Every partition of an AKAI disc, from the table it declares ([ADR-0023](adr/0023-partitions-come-from-the-table-the-disc-declares.md)) | done |
+| D16 | AIFF payloads converted, deduplicated against their WAV twin, and EXS24/HALion instruments kept ([ADR-0024](adr/0024-the-aiff-twin-is-converted-and-deduplicated.md)) | done |
 
 Across the local collection, by listing: 71 of 79 images claimed, 2 578 volumes, 110 989 files, 77 620 of them samples. The AKAI discs are 44 of those images and 68 997 of those files, read across 276 partitions — before D15 they were 14 670 files, because only the partition at the origin was read.
 
@@ -66,4 +67,6 @@ Across the local collection, by listing: 71 of 79 images claimed, 2 578 volumes,
 - **`E4P1` presets are not read.** The three E-IV discs carry 916, 901 and 284 of them. On `eiv-studio` 100 of 230 banks have no sample directory and are listed with a note; preset-only banks are the likely explanation, and it is not established.
 - **The `.mds` track table is unread.** One real pair now reads end to end, but geometry is sniffed from the `.mdf` rather than taken from the descriptor, so a multi-track or offset image would be read from byte 0. What the one specimen's descriptor holds is written down in [formats/mdx.md](formats/mdx.md) without being relied on.
 - **CUES chunks in NRG.** Only `CUEX` is parsed; `CUES` encodes position as MSF and no disc using it was available to check the layout against.
-- **AIFF payloads on ISO 9660 discs are copied, not converted.** They come out as `.aiff`.
+- **EXS24 and HALion instruments are kept, not read.** `--keep-originals` writes the `.exs` and `.fxp` files out byte for byte, because they hold the key ranges and envelopes a WAV cannot. Nothing parses them, and nothing should: turning them into a playable instrument is [ConvertWithMoss](https://github.com/git-moss/ConvertWithMoss)'s job ([ADR-0011](adr/0011-the-deliverable-is-daw-ready-wav.md)).
+- **AIFF-C is refused rather than read.** Its payload may be compressed, and compressed data written out as PCM opens, plays as noise and reports nothing wrong. No disc in the collection has one, so there is nothing to check a reader against ([ADR-0024](adr/0024-the-aiff-twin-is-converted-and-deduplicated.md)).
+- **8-bit AIFF is refused.** AIFF stores 8-bit signed and WAV unsigned, so carrying one to the other changes every sample value rather than reordering its bytes. That is the line ADR-0024 draws; nothing in the collection is other than 16-bit.