Generic RT/NRT shared resource system (with MultiChannelBuffer as first class)

[kaoskorobase]

Motivation

The engine currently has no general-purpose mechanism for sharing data between the RT audio thread and NRT worker context that can be allocated/freed at runtime via OSC. AudioBus is preallocated at engine init; synth instances are RT-only. There's no story for, e.g., a sample buffer that:

• is allocated/freed via OSC,
• is written by a record synth in RT,
• is read by play synths in RT,
• can be analyzed (onset detection) or processed (heavy DSP) in NRT.

This issue specifies a generic resource protocol — plugins can register new resource types — with AudioBuffer as the first concrete class. Resources are identified by integer id; consumers obtain a type-checked pointer via URI match (similar to a dynamic cast).

Scope

In scope:

• Generic resource definition registration via plugin API.
• OSC verbs for resource lifecycle.
• Synth integration: new kMethcla_ResourcePort port kind.
• NRT-side use via a dispatch primitive that brackets the use with refcount.
• One concrete class: AudioBuffer (multi-channel sample buffer; replaces and renames the existing MultiChannelBuffer).

Prerequisite (separate issue): dead-code cleanup in src/Methcla/Audio/Resource.hpp plus ResourceIdAllocator → IdAllocator rename — tracked in #148. The names freed by that cleanup are reused in this design.

Design decisions

#	Decision
1	Payload-mutability is per-class; descriptor is always immutable. ResourceDef declares Mutable or Immutable.
2	RT-side acquire is per-synth-lifetime, not per-pass. Refcount mutated only at synth construct / destroy / re-map. See ADR draft 0003.
3	Resource refs are a new port kind kMethcla_ResourcePort; runtime re-map via /node/set i:port-index i:resource-id. Options may also carry resource ids for shape-hint cases consumed by port_descriptor.
4	Engine owns a fixed slot pool (Options::maxNumResources). Each slot stores {state, uri, body: Methcla_Resource*, destroy_fn, refcount, free_pending}. Plugin owns the body heap allocation.
5	Interface contract: a resource URI names a C struct layout declared in a plugin-supplied header. The engine treats body as opaque (Methcla_Resource*, mirroring the existing Methcla_Synth* opacity for synth instance bodies). Consumers cast the pointer after URI match. The struct may be POD, function-pointer-heavy, or both — the engine doesn't care.
6	All slot-state mutation, refcount mutation, and lifecycle dispatch run on RT. Worker is a pure callback path for construct / destroy / perform_with_resources. No atomics needed in the slot. See ADR draft 0002.
7	NRT-side use is bracketed by methcla_world_perform_with_resources (RT-side refcount++ on dispatch, auto-release after worker returns). The Methcla_Resource* body is valid only for the duration of one perform call.
8	Synth-side bookkeeping is the plugin's responsibility (explicit acquire/release primitives). C++ wrapper provides RAII ResourceRef<T> plus a try_acquire flavor returning std::optional.
9	Id encoding: plain int32 slot index; client picks ids (NodeId / AudioBusId precedent). No generation tag.
10	Lifecycle notifications (/resource/ready, /resource/error, /resource/destroyed) originate from RT post-state-transition, dispatched via worker — mirrors /node/done. Required to avoid races between worker direct-notify and follow-up /synth/new.
11	NRT-write race protection: layered — recommend "process-into-fresh-slot" as canonical pattern (β); exclusive=true flag on the dispatch primitive as fail-fast check (γ); per-class Immutable mutability enforced by engine (δ). Strict RW enforcement on Mutable classes is rejected (would break record+play). See ADR draft 0004.
12	kMethcla_ResourcePort supports direction (Input / Output) as informational metadata; the engine does not enforce mutability against direction. Methcla_PortDescriptor stays unchanged — URI checking remains in the synth's connect().

Plugin C API

typedef void Methcla_Resource;   // opaque handle to a plugin-allocated resource body
                                 // (mirrors `typedef void Methcla_Synth;`)

typedef enum {
    kMethcla_ResourceMutable,
    kMethcla_ResourceImmutable
} Methcla_ResourceMutability;

typedef struct Methcla_ResourceDef {
    const char* uri;
    Methcla_ResourceMutability mutability;

    // Worker context. Parse args, allocate body, return it.
    // Return NULL on failure; engine notifies /resource/error.
    Methcla_Resource* (*construct)(Methcla_Host* host,
                                   const void* tag_buffer, size_t tag_size,
                                   const void* arg_buffer, size_t arg_size);

    // Worker context. Free body. Must be infallible.
    void (*destroy)(Methcla_Host* host, Methcla_Resource* resource);
} Methcla_ResourceDef;

// Registered via Methcla_Host at plugin load time (parallels register_synthdef).
void (*register_resource_def)(Methcla_Host*, const Methcla_ResourceDef*);

RT-side primitives

typedef int32_t Methcla_ResourceId;

// Returns typed body pointer after URI check; increments refcount.
// Returns NULL on URI mismatch, slot not Live, free_pending, or out of range.
Methcla_Resource* methcla_world_resource_acquire(
    Methcla_World*, Methcla_ResourceId, const char* expected_uri);

// Decrements refcount; if hits 0 and free_pending, posts destroy-cmd to worker.
void methcla_world_resource_release(Methcla_World*, Methcla_ResourceId);

Consumers cast the returned pointer to the typed interface struct, e.g. (AudioBuffer*)methcla_world_resource_acquire(world, id, AudioBuffer_URI).

NRT-side primitive

// RT acquires each resource (refcount++); on worker return, RT auto-releases all.
// If exclusive=true, dispatch fails unless every listed resource has refcount==1.
// Worker callback receives body pointers via `data` (engine packs them).
void methcla_world_perform_with_resources(
    Methcla_World*,
    const Methcla_ResourceId* resources, size_t num_resources,
    bool exclusive,
    Methcla_HostPerformFunction perform, void* data);

C++ wrapper

RAII handle with throwing and optional variants:

// Throws methcla::ResourceAcquireError on failure.
methcla::ResourceRef<AudioBuffer> buf(world, opts.bufferId);

// std::optional, empty on failure.
auto buf = methcla::try_acquire<AudioBuffer>(world, opts.bufferId);

// In connect(): replace contents, return false on failure (no-op on failure).
bool ok = m_buf.try_replace(world, new_id);

The ResourceRef<T> name reuses the symbol freed by the dead-code cleanup in #148 (the previous unused ResourceRef<T> = intrusive_ptr<T> alias).

OSC API

Verb	Args	Direction	Notes
/resource/new	s:definition-name i:resource-id [class-specific args]	client → engine	Args parsed by plugin in worker. Arg label matches /synth/new s:definition-name.
/resource/free	i:resource-id	client → engine	Destroy now if refcount==0; else mark free_pending.
/resource/ready	i:resource-id	engine → client	Slot is Live.
/resource/error	i:resource-id s:reason	engine → client	Construct failed; slot returned to Free.
/resource/destroyed	i:resource-id	engine → client	Slot returned to Free; id reusable.

Synth integration via /node/set:

Port kind	OSC arg	Notes
kMethcla_ControlPort	f:value (canonical); i:value accepted (converted to float)	unchanged behavior
kMethcla_ResourcePort	i:value only	f: against a resource port = replyError

Slot state machine

                 /resource/new (validated, slot reserved)
        Free ──────────────────────────────────────────→ Constructing
         ▲                                                    │
         │                                                    │  worker construct returns
         │                                          ┌─────────┴─────────┐
         │                                          │                   │
         │                                       NULL                 ptr
         │                                          │                   │
         │ worker destroy returns                   ▼                   ▼
         │ (state → Free)                       (notify              Live
         │                                       /resource/error)     │
         │                                          │                  │ /resource/free
         │                                          │                  │   refcount==0 → Destroying
         │                                          ▼                  │   refcount>0  → free_pending=true,
         └──────────────────────────────────────  Free                 │                  drain via release path
                                                                       ▼
                                                                  Destroying
                                                                       │
                                                                       │ worker destroy returns
                                                                       └────────→ Free

free_pending is set on a Live slot when /resource/free arrives but refcount > 0; the last release to drain refcount then initiates Destroying. The same flag is set on a Constructing slot if free arrives mid-construct; transition to Destroying happens immediately after Constructing → Live, so the slot never sits Live-and-doomed beyond one transition.

Threading

Event	Thread	Touches
/resource/new arrives	RT	slot Free → Constructing; post worker construct
Worker construct returns	NRT → posts back	(worker) sends publish-cmd to RT
RT drains publish-cmd	RT	slot → Live; post ResourceReadyNotification to worker
Worker dispatches /resource/ready	NRT	calls packet handler
Synth bind in /synth/new or /node/set	RT	refcount++
Synth destruction (Node::free)	RT	refcount-- for each held resource
/resource/free arrives	RT	if refcount==0 → Destroying + post worker destroy; else free_pending=true
Worker destroy returns	NRT → posts back	RT publishes Free; notifies via worker
perform_with_resources dispatch	RT	per-resource refcount++; post worker perform
Worker perform returns	NRT → posts back	RT auto-releases all (refcount--)

Critical ordering: EnvironmentImpl::process drains the request queue before m_worker->perform(). Lifecycle notifications must therefore originate from RT after the state transition, dispatched via worker. Sending notifications directly from the worker would let a client's follow-up /synth/new race the publish-cmd.

Failure modes

Where	Surfaces as
/resource/new validate (id range / slot occupied / definition unknown)	replyError, RT-synchronous
Plugin construct returns NULL	/resource/error i:id s:reason notification; slot returns to Free
/resource/free validate (id invalid / state wrong)	replyError
Plugin destroy	Contractually infallible
Resource port acquire fails in synth construct	C++ wrapper throws → processMessage replyError on the /synth/new
Resource port acquire fails in synth connect (re-map)	Retain prior binding, log warning, silent to client
perform_with_resources dispatch with exclusive=true and refcount>1	Synchronous fail; no acquires performed; client-visible error

Mutation contract published to plugin authors

A ResourceDef is Mutable (default; suits buffers) or Immutable (engine rejects write-acquires; modifications = new resource id; suits wavetables, lookup tables).

For Mutable classes, NRT writes are safe only when nothing else holds the resource. Recommended pattern: process-into-fresh-slot — allocate destination Y, dispatch perform_with_resources({X, Y}, ...) reading X and writing Y, optionally re-bind synths from X to Y via /node/set, free X.

For in-place writes, use exclusive=true. The engine rejects dispatch if any other holder exists, catching lifecycle bugs at dispatch time.

RT-side mutation (record synths) on Mutable classes is the user's responsibility, controlled by node ordering. The engine does not detect this.

First resource class: AudioBuffer

URI: http://methc.la/resources/AudioBuffer/v1

Mutability: kMethcla_ResourceMutable.

Renamed from the existing MultiChannelBuffer (src/Methcla/Audio/MultiChannelBuffer.hpp) — the rename happens as part of this issue's implementation.

Interface header (include/methcla/plugins/audio_buffer.h) publishes the C struct layout (channels, frames, sample rate, float** data — see src/Methcla/Audio/MultiChannelBuffer.hpp:25-133 for the current C++ form).

construct(host, tags, args): parses i:numChannels i:numFrames f:sampleRate, allocates via methcla_host_alloc_aligned. Returns Methcla_Resource* pointing to the constructed AudioBuffer body.

destroy(host, resource): frees the channel arrays and the descriptor.

Engine implementation notes

• Slot pool: std::array<ResourceSlot, maxNumResources> (or fixed-size vector at engine init).
• All slot fields are plain types (no atomics); the state, refcount, free_pending, and body are RT-only.
• Worker→RT command FIFO is the existing Worker infrastructure (m_worker->sendFromWorker).
• Notifications follow the existing Notification / NodeNotification pattern in EngineImpl.hpp — define ResourceReadyNotification, ResourceErrorNotification, ResourceDestroyedNotification.
• Synth allocation layout is unchanged at the engine level (no engine-side per-synth resource binding table; the plugin owns its refs).

Engine shutdown ordering

1. Stop accepting new OSC.
2. Free all nodes (synth destructors release their resource refs).
3. For each remaining Live slot, force-destroy via worker.
4. Wait for in-flight Constructing slots' workers to finish; immediately destroy.
5. Drain worker FIFOs.
6. Tear down slot pool.

Plugin classes remain valid for the lifetime of the engine that registered them. Per-plugin unload is not supported in this iteration.

Implementation plan

Suggested PR slicing:

1. (Prerequisite, separate PR) Dead-code cleanup in src/Methcla/Audio/Resource.hpp and ResourceIdAllocator → IdAllocator rename (Delete unused refcount / Resource scaffolding in src/Methcla/Audio/Resource.hpp #148).
2. Slot pool + state machine + lifecycle protocol. No plugin classes registered yet; engine compiles with the new API surface. Internal test that registers a dummy class and exercises all state transitions.
3. OSC verbs. /resource/new, /resource/free, notifications.
4. Synth integration. kMethcla_ResourcePort, /node/set i:port-idx i:resource-id dispatch, C API for acquire/release.
5. NRT dispatch primitive. methcla_world_perform_with_resources (single + array, exclusive flag).
6. C++ wrapper. ResourceRef<T> + try_acquire.
7. AudioBuffer resource class. Rename MultiChannelBuffer → AudioBuffer; built-in registration; interface header in include/methcla/plugins/audio_buffer.h.
8. Disksampler / sampler / future record synth migration. Use resource ports for buffer references instead of plugin-internal sample storage. (Separate issue.)

Steps 2–6 are mechanical and orthogonal to plugin work. Step 7 produces the first observable feature.

CONTEXT.md updates (apply when implementing)

• Add Resource entry: "A plugin-registered shared data object managed by the Engine, identified by an integer id and a type URI." _Avoid_: instance (a Synth is an instance; a Resource has a body).
• Add ResourceDef entry mirroring SynthDef: "A Resource type registered by a Plugin. Describes the constructor, destructor, and interface URI for one kind of shared Resource." _Avoid_: resource class, resource type.
• Add AudioBuffer entry: "A multi-channel sample-data Resource. Used for recording and playback by Synths." _Acceptable shorthand: buffer, in code.
• Narrow AudioBus _Avoid_ from channel, buffer, bus (acceptable shorthand in code) to channel, bus (acceptable shorthand in code) — buffer drops out (it now names a distinct concept).

ADR drafts

To file as docs/adr/0002-...md, 0003-...md, 0004-...md when the implementation PR lands.

0002 — RT-only mutation of resource slot state

Methcla's RT thread owns OSC dispatch, runs every state transition through EnvironmentImpl::process, and is responsible for synth destruction in Node::free (src/Methcla/Audio/Node.cpp:57-64). Every event that mutates a resource slot — creation, refcount changes from synth bind/unbind, free request, completion of worker construct/destroy — originates from RT. The slot's state, refcount, and free_pending fields are therefore single-writer (RT) and need no atomic synchronization. The worker thread is a pure async-callback for the plugin's construct/destroy and never reads or writes the slot directly; it returns results to RT via the existing worker→RT command FIFO.

This deliberately departs from the protocol sketched in docs/rt-resource-lifecycle.md, which assumes RT-acquires + NRT-frees and prescribes an atomic refcount plus a dying flag with a seq_cst Dekker handshake. Methcla's threading topology makes that machinery unnecessary. A future maintainer reading the lifecycle doc may expect to see atomics in the slot and try to "fix" the apparent omission — they should not. NRT use of a resource is always bracketed by methcla_world_perform_with_resources, whose RT-side acquire/release brackets the worker call, so the only mutator of refcount remains RT.

0003 — Per-synth-lifetime resource acquire

A Synth binds to a Resource once — at construct time (from a synth option) or at runtime via /node/set re-map (in connect()) — and holds the binding for its entire lifetime, releasing in the synth destructor. The RT-side process() callback dereferences the cached Methcla_Resource* body pointer directly, doing no refcount work per pass.

This rejects the per-pass acquire/release pattern from docs/rt-resource-lifecycle.md, which had every RT call-site fetch_add/fetch_sub the refcount per pass. Per-pass gives a tight reclamation bound (one pass to drain in-flight references); per-synth-lifetime gives a looser bound — the resource is freed when the last synth using it is destroyed — in exchange for zero RT process()-time overhead. Methcla's typical use case is buffers held by long-lived record/play synths over seconds-to-minutes spans, where the per-pass cost would be all overhead and no benefit. NRT-side "use during processing" is handled separately by methcla_world_perform_with_resources, which brackets each NRT operation with its own RT-mediated acquire/release.

0004 — Mutable resources have discipline-only concurrency

A Methcla_ResourceDef declares its kind as kMethcla_ResourceMutable or kMethcla_ResourceImmutable. Mutable resources (like AudioBuffer) allow concurrent RT and NRT writes; the engine does not enforce reader/writer exclusion. The contract published to clients: (i) process-into-fresh-slot is the canonical safe pattern for NRT writes (allocate a destination, process into it, optionally re-bind synths via /node/set); (ii) methcla_world_perform_with_resources(..., exclusive=true) is a fail-fast check that rejects dispatch unless every listed resource has refcount == 1; (iii) RT-side mutation (a record synth writing while play synths read) is the user's responsibility, controlled by node ordering.

Strict reader/writer enforcement was considered and rejected because it would break the use case that motivated the whole design: a record synth (writer) and one or more play synths (readers) sharing the same buffer within a single audio pass, with node ordering ensuring the record synth runs before its readers. Enforcing exclusion would force clients to either kill and recreate synths around every write (high-friction) or pass an override flag that makes enforcement advisory anyway. Immutable resources exist as a stricter alternative for classes where mutation is meaningless — wavetables, lookup tables, FFT plans; the engine rejects writes on those.

Related issues

• Delete unused refcount / Resource scaffolding in src/Methcla/Audio/Resource.hpp #148 — dead-code cleanup + IdAllocator rename (prerequisite).