From e564365f80b0b9d70d8ce05532341af8c713d9aa Mon Sep 17 00:00:00 2001 From: alowrydi Date: Fri, 26 Jun 2026 16:26:33 +0100 Subject: [PATCH 01/14] Initial import of di.asyncdispatch --- di/asyncdispatch/asyncdispatch.md | 191 ++++++++++++++++++++++++++++++ di/asyncdispatch/asyncdispatch.q | 159 +++++++++++++++++++++++++ di/asyncdispatch/init.q | 10 ++ di/asyncdispatch/test.csv | 75 ++++++++++++ 4 files changed, 435 insertions(+) create mode 100644 di/asyncdispatch/asyncdispatch.md create mode 100644 di/asyncdispatch/asyncdispatch.q create mode 100644 di/asyncdispatch/init.q create mode 100644 di/asyncdispatch/test.csv diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md new file mode 100644 index 00000000..dbe5ee3c --- /dev/null +++ b/di/asyncdispatch/asyncdispatch.md @@ -0,0 +1,191 @@ +# asyncdispatch + +`asyncdispatch.q` is the async multi-process query coordinator extracted from the TorQ gateway's core engine (`.gw.*`). It queues client queries, dispatches them to available backend ("server") processes, collects the per-server results, applies a join function, and replies to the client — with timeout management and error propagation if a backend disconnects mid-query. + +**Standalone value:** any process that needs scatter-gather across multiple backend processes (not just a "gateway") can load this module to get queueing, dispatch, join and timeout handling for free. + +**Out of scope** (left to other modules in the gateway decomposition): +- **Routing** — deciding *which* servers satisfy a query is `di.serverselect`'s job. This module is handed a resolved list of servertypes and dispatches to whatever of those is currently idle. +- **Permissions** — `.pm.*`/`execas`/`valp`-style checks belong in `di.gateway`/`di.permissions`. +- **EOD / discovery / dashboard & REST handlers** — `di.gateway` concerns. + +--- + +## Loading + +```q +ad:use`di.asyncdispatch +``` + +Returns a handle (`ad`) exposing the exported API, e.g. `ad.execquery[...]`. + +--- + +## Configuration & pluggable hooks + +| Variable | Default | Setter | Purpose | +|---|---|---|---| +| `errorprefix` | `"error: "` | — | Prefix added to error strings sent back to clients | +| `querykeeptime` | `0D00:30` | — | How long `removequeries` keeps finished queries in `queryqueue` | +| `clearinactivetime` | `0D01:00` | — | How long `removeinactive` keeps records for disconnected servers | +| `synccallsallowed` | `0b` | — | Whether `execquery[...;1b]` (sync mode) is permitted | +| `cp` | `{.z.p}` | `setcp` | Current-time function. Override for simulation/backtesting | +| `formatresponse` | see below | `setformatresponse` | Final transform applied to a result/error before it is sent to the client | +| `availableservers` | built-in | `setavailableservers` | How "which servers are free" is computed | +| `getnextqueryid` | `fifo` | `setgetnextqueryid` | Scheduler — picks the next query to dispatch | +| `resultcallback` / `errorcallback` | `` `addserverresult `` / `` `addservererror `` | `setcallbacks` | Symbols backend servers call back into to report results/errors | + +### `formatresponse[status;sync;result]` +- `status` — `1b` success, `0b` error. +- `sync` — `1b` if the client used a deferred-sync call (`execquery[...;1b]`), `0b` for async. +- Default: on a synchronous error (`not status` and `sync`), **signals** `result` as an error back up the `-30!` deferred response so the client's sync call raises. Otherwise passes `result` through unchanged. + +### `availableservers[excludeinuse]` +- `excludeinuse=1b` → only **idle, active** servers. +- `excludeinuse=0b` → all **active** servers. +- Override for custom load-balancing. + +### `getnextqueryid[]` (scheduler) +- Default: oldest runnable query first (FIFO) — a runnable query is one where all required servertypes have an idle server available. +- Override `getnextqueryid` for priority queues. The override receives no arguments and must return a 1-row (or empty) table with the `queryqueue` schema. + +### `resultcallback` / `errorcallback` +`serverexecute` (the function sent to backend servers) posts its outcome back via `(resultcallback;queryid;result)` / `(errorcallback;queryid;error)` over `neg .z.w`. Since this module has no fixed namespace path when used standalone, the consumer must point these symbols at wherever it mounted this module on the **backend** processes, e.g.: + +```q +ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] +``` + +--- + +## Core data structures + +### `servers` (keyed table, key: `handle`) + +| Column | Type | Description | +|---|---|---| +| `handle` | `int` | Connection handle to the backend process | +| `servertype` | `symbol` | e.g. `` `rdb ``, `` `hdb `` | +| `inuse` | `boolean` | Currently running a query | +| `active` | `boolean` | Connected (`0b` once disconnected) | +| `disconnecttime` | `timestamp` | When the server was marked inactive; null while active | + +### `queryqueue` (keyed table, key: `queryid`) + +| Column | Type | Description | +|---|---|---| +| `queryid` | `long` | Allocated by `addquery` | +| `time` | `timestamp` | When the query was queued | +| `clienth` | `int` | `.z.w` of the requesting client | +| `query` | `()` | The query payload — whatever `value` can execute | +| `servertype` | `()` | List of servertype symbols the query must run on | +| `join` | `()` | Function applied to the collected per-server results | +| `postback` | `()` | `()` for a plain reply, else `(function;extra args...)` to wrap the reply | +| `timeout` | `timespan` | `0Wn` for none | +| `returntime` | `timestamp` | Set by `finishquery` once complete | +| `error` | `boolean` | `1b` if the query ended in error | +| `sync` | `boolean` | `1b` if the client is waiting on a deferred (`-30!`) response | + +### `clients` (table) + +| Column | Type | Description | +|---|---|---| +| `time` | `timestamp` | Connection time | +| `clienth` | `int` | `.z.w` of the client | +| `user` | `symbol` | `.z.u` | +| `ip` | `int` | `.z.a` | +| `host` | `symbol` | `.z.h` | + +### `results` (dict) + +`queryid -> (clienth; servertype!(handle;result;received))` + +- Keys are the servertype symbols passed to `addquery`. +- Each value is a 3-list: `(handle assigned; result once it arrives; received flag)`. +- A query is complete once every `received` flag is `1b`. +- Entries are removed by `finishquery` once the query completes. + +--- + +## Functions + +### Server registry +- **`addserver[handle;servertype]`** — register a backend connection. +- **`availableservers[excludeinuse]`** — query which servers can take work now (see Pluggable hooks). + +### Client tracking +- **`addclientdetails[h]`** — call from `.z.po` to record a new client connection in `clients`. +- **`removeclienthandle[h]`** — call from `.z.pc`. Stamps any of that client's unfinished queries as returned/errored and drops their `results` entries. + +### Queueing +- **`addquery[query;servertype;join;postback;timeout;sync]`** — low-level: insert a row into `queryqueue`. Does **not** dispatch — call `runnextquery[]` afterwards (or use `execquery` which does both). +- **`removequeries[age]`** — purge `queryqueue` rows with `returntime` older than `age`. + +### Scheduling +- **`getnextqueryid[]`** — returns the next runnable query (FIFO by default: oldest query for which all required servertypes have an idle server). Override via `setgetnextqueryid` for priority queues. + +### Result collection & joining +- **`addserverresult[qid;data]`** — called when a backend posts back success. Fills slot, frees server, tries to run the next query; once all slots are received applies `join`, sends the reply, and finishes the query. +- **`addservererror[qid;err]`** — called when a backend posts back an error. Sends the error to the client and finishes the query. + +### Dispatch +- **`serverexecute[qid;query]`** — **runs on the backend**. Executes `value query`, trapping errors, and posts the outcome back to the dispatcher via `resultcallback`/`errorcallback`. +- **`runnextquery[]`** — picks the next runnable query via `getnextqueryid`, resolves idle servers, and dispatches. + +### Timeouts & disconnects +- **`checktimeout[]`** — finds queries past their `timeout` with no `returntime`, sends a timeout error, and finishes them. +- **`removeserverhandle[serverh]`** — call from `.z.pc` for **backend** handles. Errors any in-flight or queued queries that depended on this server, marks the server `active:0b`, and calls `runnextquery[]`. +- **`removeinactive[age]`** — purge `servers` rows that have been inactive for longer than `age`. + +### Public API +- **`execquery[query;servertype;join;postback;timeout;sync]`** — queue + dispatch. `sync=0b` for async (uses `postback`); `sync=1b` for deferred sync via `-30!` (`postback` ignored, errors if `synccallsallowed` is `0b`). + +### Housekeeping +- **`init[timerrepeat]`** — optionally wires `removequeries`, `checktimeout` and `removeinactive` into a recurring timer. `timerrepeat` should have the signature `.timer.repeat[starttime;endtime;period;(func;params);description]`, or pass `(::)` to skip. + +--- + +## Example usage + +```q +/ -- dispatcher process -- +ad:use`di.asyncdispatch + +/ point backends' callbacks at this module's mount point on this process +ad.setcallbacks[`ad.addserverresult;`ad.addservererror] + +/ register backend connections as they connect +h:hopen`:backend1:5001 +ad.addserver[h;`rdb] + +/ track gateway clients +.z.po:{ad.addclientdetails[.z.w]} +.z.pc:{ad.removeclienthandle[.z.w]; ad.removeserverhandle[.z.w]} + +/ a client calls this asynchronously: +ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] +/ -> queues the query, dispatches it to the rdb, and (once the rdb replies) +/ razes the single result and sends it back to the calling client + +/ housekeeping - run periodically (e.g. via di.timer) +ad.checktimeout[] +ad.removequeries[ad.querykeeptime] +ad.removeinactive[ad.clearinactivetime] +``` + +```q +/ -- backend process -- +ad:use`di.asyncdispatch +/ nothing else required: when the dispatcher sends (serverexecute;qid;query), +/ this process runs value query and posts the result/error back via +/ ad.addserverresult / ad.addservererror on the dispatcher +``` + +--- + +## Notes + +- `.z.M.` is used for in-place mutation of tables (`upsert`/`insert`/`update from`/`delete from`), and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache`. +- Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use `.z.m.varname` form, since q-sql evaluates globals in the calling context rather than the module namespace. +- `servertype` on `queryqueue`/`addquery` is deliberately generic: it is a list of servertype symbols used to look up idle servers of each type. Pass the same servertype list to `addserver` and `addquery` to wire them together. +- This module does not open or accept any connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, by design (keeps this module dependency-free and testable in-process). diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q new file mode 100644 index 00000000..a7e0d40a --- /dev/null +++ b/di/asyncdispatch/asyncdispatch.q @@ -0,0 +1,159 @@ +/ di.asyncdispatch - async scatter-gather query coordinator. +/ Queues queries, dispatches to available backends, collects results per server, +/ applies a join function, and replies to the client. +/ Routing (which servers satisfy a query) is di.serverselect's responsibility. + +errorprefix:"error: "; +querykeeptime:0D00:30; +clearinactivetime:0D01:00; +synccallsallowed:0b; + +cp:{.z.p} / injectable clock; swap out in tests to control time +setcp:{.z.m.cp:x} / allows runtime replacement of the clock without editing the module + +formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]} / sync errors must be signalled with ' so the client receives a trapped error; async errors pass through unchanged +setformatresponse:{.z.m.formatresponse:x} / override reply formatting without editing the module + +/ symbols backend servers call back via; set to wherever this module is mounted +resultcallback:`addserverresult; / stored as symbol so the name survives IPC serialization to backend processes +errorcallback:`addservererror; +setcallbacks:{[resfn;errfn].z.m.resultcallback:resfn;.z.m.errorcallback:errfn} / update callback symbols when module is mounted under a non-default namespace + +/ registered backend servers +servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()) + +/ pending and in-flight client queries +queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$()) + +/ connected client tracking +clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()) + +/ per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) +results:()!() + +queryid:0; + +addserver:{[h;st].z.M.servers upsert (h;st;0b;1b;0Np)} / register a backend handle and servertype so it becomes eligible for dispatch + +availableservers:{[excludeinuse] / centralise the active+idle filter so dispatch and routing share one definition + $[excludeinuse; + select from servers where active, not inuse; + select from servers where active]}; +setavailableservers:{.z.m.availableservers:x} / swap in a custom routing strategy without forking core dispatch + +addclientdetails:{[h].z.M.clients insert (cp[];h;.z.u;.z.a;.z.h)} / record client identity on connect for audit and orphan-query cleanup on disconnect + +removeclienthandle:{[h] / on client disconnect, mark their pending queries errored so result slots are not leaked + update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; + .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results}; + +addquery:{[query;servertype;join;postback;timeout;sync] / enqueue a query without dispatching; caller must call runnextquery[] to trigger dispatch + .z.M.queryqueue upsert (queryid;cp[];.z.w;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync); + .z.m.queryid:queryid+1}; + +removequeries:{[age] / prevent queryqueue growing unboundedly; purge completed queries older than age + .z.m.queryqueue:0!delete from .z.m.queryqueue where not null returntime, .z.m.cp[]>returntime+age}; + +getnextqueryid:{ / pick the oldest FIFO-eligible query whose required servertypes are all currently idle + avail:exec distinct servertype from availableservers 1b; + runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; + 1 sublist select from runnable where time=min time}; +setgetnextqueryid:{.z.m.getnextqueryid:x} / inject a priority or custom scheduling strategy without forking the module + +addserverresult:{[qid;data] / fill one result slot; once all slots for a query are filled, run the join function and reply to the client + if[not qid in key results;:()]; + st:first exec servertype from .z.m.servers where handle=.z.w; + slots:results[qid;1]; + slots[st]:(.z.w;data;1b); + results[qid]:(results[qid;0];slots); + update inuse:0b from .z.M.servers where handle in .z.w; + runnextquery[]; + if[not qid in key results;:()]; + vals:value results[qid;1]; + if[not all vals[;2];:()]; + qd:queryqueue[qid]; + res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;errorprefix,"join failed: ",x)}]; + sendclientreply[qid;last res;not res 0]; + finishquery[qid;res 0]}; + +addservererror:{[qid;err] / short-circuit a query on backend failure; free the server and notify the client before moving on + sendclientreply[qid;errorprefix,err;0b]; + update inuse:0b from .z.M.servers where handle in .z.w; + runnextquery[]; + finishquery[qid;1b]}; + +sendclientreply:{[qid;result;status] / deliver result or error to the client, handling sync vs async send and postback wrapping in one place + qd:queryqueue[qid]; + if[qd`error;:()]; + tosend:$[()~qd`postback;result;qd[`postback],enlist[qd`query],enlist result]; + $[qd`sync; + @[-30!;(qd`clienth;not status;$[status;formatresponse[1b;1b;result];result]);{}]; + @[neg qd`clienth;formatresponse[status;0b;tosend];()]]}; + +finishquery:{[qid;err] / remove query from the live results accumulator and stamp its completion time; keeps queryqueue and results consistent + .z.m.results:(qid,())_results; + update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid}; + +serverexecute:{[qid;query] / runs on the backend; traps errors locally so a crash posts an error reply rather than silently dropping the result + res:@[{(0b;value x)};query;{(1b;"server ",(string .z.h),":",(string system"p"),": ",x)}]; + @[neg .z.w;$[res 0;(errorcallback;qid;res 1);(resultcallback;qid;res 1)]; + {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]}; + +sendquerytoserver:{[qid;query;handles] / fan the query out to all required handles and mark them in-use atomically to prevent double-dispatch + (neg handles,:())@\:(serverexecute;qid;query); + update inuse:1b from .z.M.servers where handle in handles}; + +runnextquery:{ / pick the next dispatchable query and fan out to one idle server per required servertype; called after any state change that may unblock work + if[0=count torun:getnextqueryid[];:()]; + torun:first torun; + avail:exec first handle by servertype from availableservers 1b; + types:torun`servertype; + handles:avail types; + qid:torun`queryid; + slots:types!(count[types],())#enlist(0Ni;(::);0b); + slots[types;0]:handles; + results[qid]:(torun`clienth;slots); + sendquerytoserver[qid;torun`query;handles]}; + +checktimeout:{ / periodic scan to error queries that have waited beyond their timeout, preventing them from hanging indefinitely + qids:exec queryid from .z.m.queryqueue where not timeout=0Wn, null returntime, .z.m.cp[]>time+timeout; + if[count qids; + sendclientreply[;errorprefix,"query timed out";0b] each qids; + finishquery[qids;1b]]}; + +removeserverhandle:{[serverh] / on backend disconnect, error in-flight queries using that handle and queued queries that can no longer be satisfied + if[null st:first exec servertype from .z.m.servers where handle=serverh;:()]; + err:errorprefix,"backend ",string[st]," server disconnected"; + + / in-flight: queries where this handle was assigned to a slot + qids:where {[h;qid]h in value[.z.m.results[qid;1]][;0]}[serverh] each key .z.m.results; + sendclientreply[;err," during query";0b] each qids; + finishquery[qids;1b]; + + / queued: queries that can no longer be satisfied by remaining active servers + activetypes:exec distinct servertype from .z.m.servers where active, handle<>serverh; + qids2:exec queryid from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, + not {all x in y}[;activetypes] each servertype; + sendclientreply[;err,", query cannot be satisfied";0b] each qids2; + finishquery[qids2;1b]; + + update active:0b,disconnecttime:.z.m.cp[] from .z.M.servers where handle=serverh; + runnextquery[]}; + +removeinactive:{[age]delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age} / prune stale disconnected-server rows to stop the servers table growing forever + +execquery:{[query;servertype;join;postback;timeout;sync] / public entry point; validate sync constraints then enqueue and kick dispatch + if[sync; + if[not synccallsallowed;'"syncexec: synchronous calls are not allowed"]; + if[not @[{-30!x;1b};(::);0b];'"syncexec: deferred response not supported on this connection"]; + .[{[q;s;j;t]addquery[q;s;j;();t;1b];runnextquery[]};(query;servertype;join;timeout);{-30!(.z.w;1b;x)}]; + :()]; + addquery[query;servertype;join;postback;timeout;0b]; + runnextquery[]}; + +/ wire housekeeping into a timer - pass (::) to skip +init:{[timerrepeat] / wire recurring housekeeping (timeout scan, query purge, server purge) into a provided timer; pass (::) to skip registration + if[not timerrepeat~(::); + timerrepeat[cp[];0Wp;0D00:05:00;(.z.m.removequeries;querykeeptime);"asyncdispatch: remove old queries"]; + timerrepeat[cp[];0Wp;0D00:00:05;(.z.m.checktimeout;`);"asyncdispatch: timeout expired queries"]; + timerrepeat[cp[];0Wp;0D00:05:00;(.z.m.removeinactive;clearinactivetime);"asyncdispatch: remove inactive servers"]]}; diff --git a/di/asyncdispatch/init.q b/di/asyncdispatch/init.q new file mode 100644 index 00000000..f7086488 --- /dev/null +++ b/di/asyncdispatch/init.q @@ -0,0 +1,10 @@ +\l ::asyncdispatch.q + +export:([ + errorprefix;querykeeptime;clearinactivetime;synccallsallowed; / user-tunable config: error text prefix and housekeeping intervals + setcp;setformatresponse;setcallbacks;setavailableservers;setgetnextqueryid; / user-injectable overrides: clock, reply format, callback namespace, routing and scheduling + addserver;removeserverhandle; / server lifecycle: register and deregister backends + addclientdetails;removeclienthandle; / client lifecycle: wire into .z.po / .z.pc + addserverresult;addservererror; / IPC return paths: backends resolve these by name to deliver results or errors + execquery; / public API: only entry point for submitting a query + init]) / startup: wires housekeeping into the provided timer diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv new file mode 100644 index 00000000..7366d787 --- /dev/null +++ b/di/asyncdispatch/test.csv @@ -0,0 +1,75 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,system"q -p 19501 -q &",1,,start backend process +before,0,0,q,system"sleep 1",1,,wait for backend to initialise +before,0,0,q,ad:use`di.asyncdispatch,1,,load module +before,0,0,q,srv:{.m.di.0asyncdispatch.servers},1,,live servers table +before,0,0,q,qq:{.m.di.0asyncdispatch.queryqueue},1,,live queryqueue table +before,0,0,q,cl:{.m.di.0asyncdispatch.clients},1,,live clients table +before,0,0,q,res:{.m.di.0asyncdispatch.results},1,,live results dict +before,0,0,q,bh:hopen`::19501,1,,open handle to backend +before,0,0,q,bh".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure backend result callback to dispatcher +before,0,0,q,bh".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure backend error callback to dispatcher +before,0,0,q,ad.addserver[bh;`mock],1,,register backend as mock server + +/ Test 0: server registry +true,0,0,q,(enlist bh)~exec handle from srv[] where active,1,,backend server is active +true,0,0,q,(enlist bh)~exec handle from ad.availableservers 1b,1,,idle backend server is available + +/ Test 1: queueing and FIFO scheduling +run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`mock;raze;();0Wn;0b],1,,queue a query for the mock servertype +true,0,0,q,1~count select from qq[] where query~\:"2+2",1,,query was added to the queue +true,0,0,q,1~count .m.di.0asyncdispatch.getnextqueryid[],1,,query is runnable - idle mock server available +true,0,0,q,(enlist`mock)~first exec servertype from .m.di.0asyncdispatch.getnextqueryid[],1,,getnextqueryid identifies the available servertype + +/ Test 2: full dispatch -> result -> join -> reply via real IPC round-trip +run,0,0,q,qid1:exec first queryid from qq[] where query~\:"2+2",1,,capture the queued query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch query to real backend process +run,0,0,q,bh"",1,,sync round-trip: flushes dispatch and waits for backend result callback before returning +true,0,0,q,0b~first exec error from qq[] where queryid=qid1,1,,query completed without error +true,0,0,q,not null first exec returntime from qq[] where queryid=qid1,1,,returntime is stamped +true,0,0,q,not qid1 in key res[],1,,result accumulator cleaned up after join + +/ Test 3: checktimeout +run,0,0,q,.m.di.0asyncdispatch.addquery["3+3";enlist`mock;raze;();0D00:00:00.000000001;0b],1,,queue with a near-zero timeout +run,0,0,q,qid2:exec first queryid from qq[] where query~\:"3+3",1,,capture the queued query id +run,0,0,q,.m.di.0asyncdispatch.checktimeout[],1,,timeout sweep should catch the expired query +true,0,0,q,1b~first exec error from qq[] where queryid=qid2,1,,timed-out query is flagged as error +true,0,0,q,not null first exec returntime from qq[] where queryid=qid2,1,,timed-out query has returntime stamped + +/ Test 4: removequeries purges completed queries +run,0,0,q,ad.setcp[{.z.p+2D}],1,,fast-forward the clock by 2 days +run,0,0,q,.m.di.0asyncdispatch.removequeries ad.querykeeptime,1,,purge queries older than querykeeptime +true,0,0,q,0~count select from qq[] where queryid in qid1,qid2,1,both completed queries purged +run,0,0,q,ad.setcp[{.z.p}],1,,restore the real clock + +/ Test 5: removeserverhandle errors queued queries when the only server disconnects +run,0,0,q,ad.addserver[1i;`mock2],1,,register a second mock server (fake handle - query is errored before dispatch) +run,0,0,q,.m.di.0asyncdispatch.addquery["4+4";enlist`mock2;raze;();0Wn;0b],1,,queue a query that can only run on mock2 +run,0,0,q,qid3:exec first queryid from qq[] where query~\:"4+4",1,,capture the queued query id +run,0,0,q,ad.removeserverhandle[1i],1,,disconnect the only server able to run the query +true,0,0,q,1b~first exec error from qq[] where queryid=qid3,1,,query errored as no server can satisfy it +true,0,0,q,0b~first exec active from srv[] where handle=1i,1,,disconnected server marked inactive + +/ Test 6: removeinactive purges inactive server records +run,0,0,q,ad.setcp[{.z.p+2D}],1,,fast-forward clock past clearinactivetime +run,0,0,q,.m.di.0asyncdispatch.removeinactive ad.clearinactivetime,1,,purge inactive servers +true,0,0,q,0~count select from srv[] where handle=1i,1,,inactive server record removed +run,0,0,q,ad.setcp[{.z.p}],1,,restore real clock + +/ Test 7: client tracking +run,0,0,q,ad.addclientdetails[2i],1,,record a connected client +true,0,0,q,1~count select from cl[] where clienth=2i,1,,client connection is tracked +run,0,0,q,ad.removeclienthandle[2i],1,,disconnect the client +true,0,0,q,0~count select from qq[] where clienth=2i,null returntime,1,client unfinished queries cleared + +/ Test 8: pluggable hooks +run,0,0,q,ad.setformatresponse[{[status;sync;result]result}],1,,override formatresponse to pass result through +true,0,0,q,5~.m.di.0asyncdispatch.formatresponse[1b;0b;5],1,,overridden formatresponse is used +run,0,0,q,ad.setgetnextqueryid[{()}],1,,override scheduler to never pick a query +true,0,0,q,0~count .m.di.0asyncdispatch.getnextqueryid[],1,,overridden scheduler is used +run,0,0,q,ad.setcallbacks[`myresult;`myerror],1,,override callback symbols +true,0,0,q,(`myresult;`myerror)~(.m.di.0asyncdispatch.resultcallback;.m.di.0asyncdispatch.errorcallback),1,,callback symbols updated + +/ Teardown +run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to backend process and flush +run,0,0,q,hclose bh,1,,close backend handle From c1c3557a3a11bca37c8fa836a96291df2ada01d1 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Mon, 29 Jun 2026 12:21:14 +0100 Subject: [PATCH 02/14] align with project standards, fix production bugs, real IPC integration tests, removing hardcoded ports/paths --- di/asyncdispatch/asyncdispatch.md | 293 +++++++++++++++++------------- di/asyncdispatch/asyncdispatch.q | 272 +++++++++++++++++++-------- di/asyncdispatch/init.q | 15 +- di/asyncdispatch/test.csv | 38 ++-- 4 files changed, 389 insertions(+), 229 deletions(-) diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md index dbe5ee3c..7511df84 100644 --- a/di/asyncdispatch/asyncdispatch.md +++ b/di/asyncdispatch/asyncdispatch.md @@ -1,191 +1,224 @@ -# asyncdispatch +# di.asyncdispatch -`asyncdispatch.q` is the async multi-process query coordinator extracted from the TorQ gateway's core engine (`.gw.*`). It queues client queries, dispatches them to available backend ("server") processes, collects the per-server results, applies a join function, and replies to the client — with timeout management and error propagation if a backend disconnects mid-query. +Async scatter-gather query coordinator for kdb-x gateway processes. Queues client queries, dispatches them to available backend processes by servertype, collects per-server results, applies a join function, and replies to the client — with timeout management and correct error propagation if a backend disconnects mid-query. -**Standalone value:** any process that needs scatter-gather across multiple backend processes (not just a "gateway") can load this module to get queueing, dispatch, join and timeout handling for free. +Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s responsibility. This module receives a resolved servertype list and dispatches to whatever idle backends of each type are registered. -**Out of scope** (left to other modules in the gateway decomposition): -- **Routing** — deciding *which* servers satisfy a query is `di.serverselect`'s job. This module is handed a resolved list of servertypes and dispatches to whatever of those is currently idle. -- **Permissions** — `.pm.*`/`execas`/`valp`-style checks belong in `di.gateway`/`di.permissions`. -- **EOD / discovery / dashboard & REST handlers** — `di.gateway` concerns. +--- + +## Features + +- Queue and dispatch async client queries to multiple backend process types simultaneously (scatter-gather) +- Collect per-server results and apply a caller-supplied join function once all slots are filled +- Timeout expired queries with configurable per-query timespan via `checktimeout` +- Handle backend disconnects mid-query — errors in-flight queries and queued queries that can no longer be satisfied +- Track connected clients and clean up orphaned queries on client disconnect +- Support synchronous deferred response mode (`-30!`) alongside the default async mode +- Accept fully pluggable scheduler (`setgetnextqueryid`), routing (`setavailableservers`), reply formatter (`setformatresponse`), and callback symbols (`setcallbacks`) — swap without touching core dispatch logic +- Detect and normalise `kx.log` instances automatically so callers can pass a logger directly without manual wrapping --- -## Loading +## Dependencies + +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | `info`, `warn`, `error` — each binary `{[c;m]}` where `c` is a symbol context and `m` is a string | + +The `log` dependency must be passed to `init` inside the `deps` dict. The module throws immediately if it is absent or missing any of the three required keys. All three are required since the module calls `info`, `warn`, and `error`. + +A `kx.log` instance can be passed directly — the module normalises monadic functions to the binary `{[c;m]}` contract automatically via `normlog`. Context is embedded in the output as `"context: message"`: ```q +kxlog:use`kx.log ad:use`di.asyncdispatch -``` -Returns a handle (`ad`) exposing the exported API, e.g. `ad.execquery[...]`. +/ minimal +ad.init[enlist[`log]!enlist kxlog.createLog[]] + +/ with config overrides +ad.init[`log`querykeeptime`synccallsallowed!(kxlog.createLog[];0D01:00;1b)] +``` --- -## Configuration & pluggable hooks +## Initialisation -| Variable | Default | Setter | Purpose | +`init[deps]` takes a single dictionary combining the `log` dependency with any configuration overrides. + +| Key | Required | Default | Description | |---|---|---|---| -| `errorprefix` | `"error: "` | — | Prefix added to error strings sent back to clients | -| `querykeeptime` | `0D00:30` | — | How long `removequeries` keeps finished queries in `queryqueue` | -| `clearinactivetime` | `0D01:00` | — | How long `removeinactive` keeps records for disconnected servers | -| `synccallsallowed` | `0b` | — | Whether `execquery[...;1b]` (sync mode) is permitted | -| `cp` | `{.z.p}` | `setcp` | Current-time function. Override for simulation/backtesting | -| `formatresponse` | see below | `setformatresponse` | Final transform applied to a result/error before it is sent to the client | -| `availableservers` | built-in | `setavailableservers` | How "which servers are free" is computed | -| `getnextqueryid` | `fifo` | `setgetnextqueryid` | Scheduler — picks the next query to dispatch | -| `resultcallback` / `errorcallback` | `` `addserverresult `` / `` `addservererror `` | `setcallbacks` | Symbols backend servers call back into to report results/errors | - -### `formatresponse[status;sync;result]` -- `status` — `1b` success, `0b` error. -- `sync` — `1b` if the client used a deferred-sync call (`execquery[...;1b]`), `0b` for async. -- Default: on a synchronous error (`not status` and `sync`), **signals** `result` as an error back up the `-30!` deferred response so the client's sync call raises. Otherwise passes `result` through unchanged. - -### `availableservers[excludeinuse]` -- `excludeinuse=1b` → only **idle, active** servers. -- `excludeinuse=0b` → all **active** servers. -- Override for custom load-balancing. - -### `getnextqueryid[]` (scheduler) -- Default: oldest runnable query first (FIFO) — a runnable query is one where all required servertypes have an idle server available. -- Override `getnextqueryid` for priority queues. The override receives no arguments and must return a 1-row (or empty) table with the `queryqueue` schema. - -### `resultcallback` / `errorcallback` -`serverexecute` (the function sent to backend servers) posts its outcome back via `(resultcallback;queryid;result)` / `(errorcallback;queryid;error)` over `neg .z.w`. Since this module has no fixed namespace path when used standalone, the consumer must point these symbols at wherever it mounted this module on the **backend** processes, e.g.: +| `` `log `` | yes | — | Binary log dep — `info`, `warn`, `error` functions each `{[c;m]}` | +| `` `errorprefix `` | no | `"error: "` | String prepended to all error messages sent back to clients | +| `` `querykeeptime `` | no | `0D00:30` | How long `removequeries` retains finished query rows | +| `` `clearinactivetime `` | no | `0D01:00` | How long `removeinactive` retains disconnected server rows | +| `` `synccallsallowed `` | no | `0b` | Whether `execquery[...;1b]` (deferred sync mode) is permitted | -```q -ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] -``` +Housekeeping — `checktimeout`, `removequeries`, and `removeinactive` — is the caller's responsibility. Wire them into your gateway's timer after `init`. The configured default age parameters are accessible as `querykeeptime` and `clearinactivetime` via module state. --- -## Core data structures +## Exported Functions -### `servers` (keyed table, key: `handle`) - -| Column | Type | Description | -|---|---|---| -| `handle` | `int` | Connection handle to the backend process | -| `servertype` | `symbol` | e.g. `` `rdb ``, `` `hdb `` | -| `inuse` | `boolean` | Currently running a query | -| `active` | `boolean` | Connected (`0b` once disconnected) | -| `disconnecttime` | `timestamp` | When the server was marked inactive; null while active | +### `init[deps]` +Initialise the module. Validates the log dependency and applies config overrides. +```q +ad.init[enlist[`log]!enlist logdep] +``` -### `queryqueue` (keyed table, key: `queryid`) +### `addserver[handle;servertype]` +Register a backend connection. `handle`: open int handle. `servertype`: symbol identifying the process type (e.g. `` `rdb ``, `` `hdb ``). +```q +ad.addserver[hopen`:backend1:5001;`rdb] +``` -| Column | Type | Description | -|---|---|---| -| `queryid` | `long` | Allocated by `addquery` | -| `time` | `timestamp` | When the query was queued | -| `clienth` | `int` | `.z.w` of the requesting client | -| `query` | `()` | The query payload — whatever `value` can execute | -| `servertype` | `()` | List of servertype symbols the query must run on | -| `join` | `()` | Function applied to the collected per-server results | -| `postback` | `()` | `()` for a plain reply, else `(function;extra args...)` to wrap the reply | -| `timeout` | `timespan` | `0Wn` for none | -| `returntime` | `timestamp` | Set by `finishquery` once complete | -| `error` | `boolean` | `1b` if the query ended in error | -| `sync` | `boolean` | `1b` if the client is waiting on a deferred (`-30!`) response | - -### `clients` (table) - -| Column | Type | Description | -|---|---|---| -| `time` | `timestamp` | Connection time | -| `clienth` | `int` | `.z.w` of the client | -| `user` | `symbol` | `.z.u` | -| `ip` | `int` | `.z.a` | -| `host` | `symbol` | `.z.h` | +### `removeserverhandle[handle]` +Call from `.z.pc` for **backend** handles. Errors any in-flight or queued queries that depended on this server, marks the server `active:0b`, and triggers `runnextquery`. +```q +.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} +``` -### `results` (dict) +### `addclientdetails[handle]` +Record client identity on connect. Call from `.z.po`. +```q +.z.po:{ad.addclientdetails[.z.w]} +``` -`queryid -> (clienth; servertype!(handle;result;received))` +### `removeclienthandle[handle]` +On client disconnect, mark their pending queries errored so result slots are not leaked. Call from `.z.pc`. +```q +.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} +``` -- Keys are the servertype symbols passed to `addquery`. -- Each value is a 3-list: `(handle assigned; result once it arrives; received flag)`. -- A query is complete once every `received` flag is `1b`. -- Entries are removed by `finishquery` once the query completes. +### `addserverresult[qid;data]` +Called when a backend posts back a successful result. Fills the result slot, frees the server, triggers `runnextquery`, and — once all slots for the query are received — applies the join function and replies to the client. +```q +/ called by serverexecute on the backend; not typically called directly +``` ---- +### `addservererror[qid;err]` +Called when a backend posts back an error. Sends the error to the client and finishes the query. +```q +/ called by serverexecute on the backend; not typically called directly +``` -## Functions +### `execquery[query;servertype;join;postback;timeout;sync]` +Public entry point. Validates sync constraints, enqueues the query, and triggers dispatch. -### Server registry -- **`addserver[handle;servertype]`** — register a backend connection. -- **`availableservers[excludeinuse]`** — query which servers can take work now (see Pluggable hooks). +| Argument | Type | Description | +|---|---|---| +| `query` | any | Payload passed to `value` on the backend | +| `servertype` | symbol list | One symbol per required backend type, e.g. `` enlist`rdb `` | +| `join` | function | Applied to the list of per-server results once all are received | +| `postback` | list or `()` | `()` for a plain reply; `(function;extra_args...)` to wrap the reply | +| `timeout` | timespan | `0Wn` for no timeout | +| `sync` | boolean | `1b` for deferred sync via `-30!`; `0b` for async | -### Client tracking -- **`addclientdetails[h]`** — call from `.z.po` to record a new client connection in `clients`. -- **`removeclienthandle[h]`** — call from `.z.pc`. Stamps any of that client's unfinished queries as returned/errored and drops their `results` entries. +```q +ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] +``` -### Queueing -- **`addquery[query;servertype;join;postback;timeout;sync]`** — low-level: insert a row into `queryqueue`. Does **not** dispatch — call `runnextquery[]` afterwards (or use `execquery` which does both). -- **`removequeries[age]`** — purge `queryqueue` rows with `returntime` older than `age`. +### `checktimeout[]` +Scan the queue for queries past their timeout, send a timeout error to each client, and mark them complete. Wire into your gateway's timer — every few seconds is typical. +```q +/ in gateway timer +timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] +``` -### Scheduling -- **`getnextqueryid[]`** — returns the next runnable query (FIFO by default: oldest query for which all required servertypes have an idle server). Override via `setgetnextqueryid` for priority queues. +### `removequeries[age]` +Purge completed `queryqueue` rows older than `age`. Prevents unbounded growth. +```q +/ default age is querykeeptime (0D00:30) +timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] +``` -### Result collection & joining -- **`addserverresult[qid;data]`** — called when a backend posts back success. Fills slot, frees server, tries to run the next query; once all slots are received applies `join`, sends the reply, and finishes the query. -- **`addservererror[qid;err]`** — called when a backend posts back an error. Sends the error to the client and finishes the query. +### `removeinactive[age]` +Purge `servers` rows for backends that have been disconnected longer than `age`. Prevents unbounded growth. +```q +/ default age is clearinactivetime (0D01:00) +timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] +``` -### Dispatch -- **`serverexecute[qid;query]`** — **runs on the backend**. Executes `value query`, trapping errors, and posts the outcome back to the dispatcher via `resultcallback`/`errorcallback`. -- **`runnextquery[]`** — picks the next runnable query via `getnextqueryid`, resolves idle servers, and dispatches. +### `setformatresponse[f]` +Override the reply formatter applied before a result or error is sent to the client. `f` must be `{[status;sync;result]}`. +```q +ad.setformatresponse[{[status;sync;result]result}] +``` -### Timeouts & disconnects -- **`checktimeout[]`** — finds queries past their `timeout` with no `returntime`, sends a timeout error, and finishes them. -- **`removeserverhandle[serverh]`** — call from `.z.pc` for **backend** handles. Errors any in-flight or queued queries that depended on this server, marks the server `active:0b`, and calls `runnextquery[]`. -- **`removeinactive[age]`** — purge `servers` rows that have been inactive for longer than `age`. +### `setcallbacks[resfn;errfn]` +Update the callback symbols used by `serverexecute`. Required when the module is mounted under a non-default namespace — point these at wherever `addserverresult` and `addservererror` are visible on the backend processes. +```q +ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] +``` -### Public API -- **`execquery[query;servertype;join;postback;timeout;sync]`** — queue + dispatch. `sync=0b` for async (uses `postback`); `sync=1b` for deferred sync via `-30!` (`postback` ignored, errors if `synccallsallowed` is `0b`). +### `setavailableservers[f]` +Swap in a custom routing strategy. `f` must be `{[excludeinuse]}` returning a table with a `servertype` column. +```q +ad.setavailableservers[{[excl]select from servers where active}] +``` -### Housekeeping -- **`init[timerrepeat]`** — optionally wires `removequeries`, `checktimeout` and `removeinactive` into a recurring timer. `timerrepeat` should have the signature `.timer.repeat[starttime;endtime;period;(func;params);description]`, or pass `(::)` to skip. +### `setgetnextqueryid[f]` +Inject a custom scheduling strategy. `f` must be niladic and return a 0- or 1-row table with the `queryqueue` schema. +```q +/ priority queue example - highest-priority query first +ad.setgetnextqueryid[{1 sublist `priority xdesc 0!select from .z.m.queryqueue where null returntime}] +``` --- -## Example usage +## Usage Example ```q -/ -- dispatcher process -- +kxlog:use`kx.log +timer:use`di.timer +timer.init[()!()] + ad:use`di.asyncdispatch +ad.init[enlist[`log]!enlist kxlog.createLog[]] / point backends' callbacks at this module's mount point on this process ad.setcallbacks[`ad.addserverresult;`ad.addservererror] / register backend connections as they connect -h:hopen`:backend1:5001 -ad.addserver[h;`rdb] +ad.addserver[hopen`:backend1:5001;`rdb] +ad.addserver[hopen`:backend2:5002;`hdb] -/ track gateway clients +/ wire client and server connection/disconnection handlers .z.po:{ad.addclientdetails[.z.w]} -.z.pc:{ad.removeclienthandle[.z.w]; ad.removeserverhandle[.z.w]} +.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} + +/ wire housekeeping into the gateway timer +timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] +timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] +timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] / a client calls this asynchronously: -ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] -/ -> queues the query, dispatches it to the rdb, and (once the rdb replies) -/ razes the single result and sends it back to the calling client +/ execquery dispatches to rdb and hdb in parallel, razes results, replies to client +ad.execquery[("select count i by sym from trade";"select count i by sym from trade");`rdb`hdb;raze;();0Wn;0b] -/ housekeeping - run periodically (e.g. via di.timer) -ad.checktimeout[] -ad.removequeries[ad.querykeeptime] -ad.removeinactive[ad.clearinactivetime] +/ synchronous deferred mode (requires synccallsallowed:1b in deps) +ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;1b] ``` +--- + +## Running Tests + ```q -/ -- backend process -- -ad:use`di.asyncdispatch -/ nothing else required: when the dispatcher sends (serverexecute;qid;query), -/ this process runs value query and posts the result/error back via -/ ad.addserverresult / ad.addservererror on the dispatcher +k4unit:use`di.k4unit +k4unit.moduletest`di.asyncdispatch ``` +52 tests. Requires a `q` binary in `PATH` — the test suite starts a real backend process on a dynamically selected free port and exercises the full dispatch lifecycle over a live IPC connection. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, checktimeout, removequeries, removeserverhandle with orphaned query cleanup, client tracking, removeinactive, in-flight server release on client disconnect, and all pluggable hook setters. + --- ## Notes -- `.z.M.` is used for in-place mutation of tables (`upsert`/`insert`/`update from`/`delete from`), and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache`. -- Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use `.z.m.varname` form, since q-sql evaluates globals in the calling context rather than the module namespace. -- `servertype` on `queryqueue`/`addquery` is deliberately generic: it is a list of servertype symbols used to look up idle servers of each type. Pass the same servertype list to `addserver` and `addquery` to wire them together. -- This module does not open or accept any connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, by design (keeps this module dependency-free and testable in-process). +- Housekeeping (`checktimeout`, `removequeries`, `removeinactive`) is the caller's responsibility — the gateway process already has a timer running and is better placed to decide intervals. Wire all three after `init`; see the usage example above +- `.z.M.` is used for in-place mutation of tables (`upsert`, `insert`, `update from`, `delete from`) and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache` +- Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use the `.z.m.varname` form since q-sql evaluates column expressions in the calling context rather than the module namespace +- `servertype` in `queryqueue` and `addquery` is a list of servertype symbols — one per required backend type. Pass `` enlist`rdb `` for single-server queries, `` `rdb`hdb `` for scatter-gather across two types +- `setcallbacks` must be called before any queries are dispatched if the module is mounted under a non-default path — `serverexecute` reads `resultcallback` and `errorcallback` by bare name on the backend process and posts back to whatever symbols they resolve to +- This module opens and accepts no connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, keeping the module dependency-free and testable in-process +- All three log keys (`info`, `warn`, `error`) are required — the module calls `info` on server/client connect and init, `warn` on disconnect and timeout, and `error` on backend error and join failure diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q index a7e0d40a..86bcc8ce 100644 --- a/di/asyncdispatch/asyncdispatch.q +++ b/di/asyncdispatch/asyncdispatch.q @@ -1,88 +1,85 @@ -/ di.asyncdispatch - async scatter-gather query coordinator. -/ Queues queries, dispatches to available backends, collects results per server, -/ applies a join function, and replies to the client. -/ Routing (which servers satisfy a query) is di.serverselect's responsibility. +/ di.asyncdispatch - async scatter-gather query coordinator +/ queues queries, dispatches to available backends, collects results per server, +/ applies a join function, and replies to the client +/ routing (which servers satisfy a query) is di.serverselect's responsibility +/ the log dependency is required - init errors immediately if absent or malformed +/ log functions are binary {[c;m]} where c is a symbol context and m is a string +/ ============================================================ +/ module state and defaults +/ ============================================================ + +/ error prefix prepended to all error strings returned to clients errorprefix:"error: "; + +/ how long completed queries are kept in queryqueue before being purged querykeeptime:0D00:30; + +/ how long disconnected servers are kept in the servers table before being removed clearinactivetime:0D01:00; -synccallsallowed:0b; -cp:{.z.p} / injectable clock; swap out in tests to control time -setcp:{.z.m.cp:x} / allows runtime replacement of the clock without editing the module +/ whether synchronous calls via -30! are permitted +synccallsallowed:0b; -formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]} / sync errors must be signalled with ' so the client receives a trapped error; async errors pass through unchanged -setformatresponse:{.z.m.formatresponse:x} / override reply formatting without editing the module +/ injectable clock - replaced in tests to control time without sleeping +cp:{.z.p}; -/ symbols backend servers call back via; set to wherever this module is mounted -resultcallback:`addserverresult; / stored as symbol so the name survives IPC serialization to backend processes +/ symbols backend servers call back via - stored as symbols so names survive IPC serialisation +resultcallback:`addserverresult; errorcallback:`addservererror; -setcallbacks:{[resfn;errfn].z.m.resultcallback:resfn;.z.m.errorcallback:errfn} / update callback symbols when module is mounted under a non-default namespace -/ registered backend servers -servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()) +/ reply formatting - sync errors must be signalled with ' so the client receives a trapped error +formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]}; -/ pending and in-flight client queries -queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$()) - -/ connected client tracking -clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()) - -/ per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) -results:()!() - -queryid:0; - -addserver:{[h;st].z.M.servers upsert (h;st;0b;1b;0Np)} / register a backend handle and servertype so it becomes eligible for dispatch +/ scheduling strategy - pick oldest FIFO-eligible query by default +getnextqueryid:{ + avail:exec distinct servertype from availableservers 1b; + runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; + 1 sublist select from runnable where time=min time}; -availableservers:{[excludeinuse] / centralise the active+idle filter so dispatch and routing share one definition +/ server routing strategy - active and idle by default +availableservers:{[excludeinuse] $[excludeinuse; select from servers where active, not inuse; select from servers where active]}; -setavailableservers:{.z.m.availableservers:x} / swap in a custom routing strategy without forking core dispatch -addclientdetails:{[h].z.M.clients insert (cp[];h;.z.u;.z.a;.z.h)} / record client identity on connect for audit and orphan-query cleanup on disconnect +/ ============================================================ +/ module tables +/ ============================================================ -removeclienthandle:{[h] / on client disconnect, mark their pending queries errored so result slots are not leaked - update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; - .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results}; +/ registered backend servers +servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()); -addquery:{[query;servertype;join;postback;timeout;sync] / enqueue a query without dispatching; caller must call runnextquery[] to trigger dispatch - .z.M.queryqueue upsert (queryid;cp[];.z.w;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync); - .z.m.queryid:queryid+1}; +/ pending and in-flight client queries +queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$()); -removequeries:{[age] / prevent queryqueue growing unboundedly; purge completed queries older than age - .z.m.queryqueue:0!delete from .z.m.queryqueue where not null returntime, .z.m.cp[]>returntime+age}; +/ connected client tracking +clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()); -getnextqueryid:{ / pick the oldest FIFO-eligible query whose required servertypes are all currently idle - avail:exec distinct servertype from availableservers 1b; - runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; - 1 sublist select from runnable where time=min time}; -setgetnextqueryid:{.z.m.getnextqueryid:x} / inject a priority or custom scheduling strategy without forking the module +/ per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) +results:()!(); -addserverresult:{[qid;data] / fill one result slot; once all slots for a query are filled, run the join function and reply to the client - if[not qid in key results;:()]; - st:first exec servertype from .z.m.servers where handle=.z.w; - slots:results[qid;1]; - slots[st]:(.z.w;data;1b); - results[qid]:(results[qid;0];slots); - update inuse:0b from .z.M.servers where handle in .z.w; - runnextquery[]; - if[not qid in key results;:()]; - vals:value results[qid;1]; - if[not all vals[;2];:()]; - qd:queryqueue[qid]; - res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;errorprefix,"join failed: ",x)}]; - sendclientreply[qid;last res;not res 0]; - finishquery[qid;res 0]}; +/ auto-incrementing query id counter +queryid:0; -addservererror:{[qid;err] / short-circuit a query on backend failure; free the server and notify the client before moving on - sendclientreply[qid;errorprefix,err;0b]; - update inuse:0b from .z.M.servers where handle in .z.w; - runnextquery[]; - finishquery[qid;1b]}; +/ ============================================================ +/ internal helpers +/ ============================================================ -sendclientreply:{[qid;result;status] / deliver result or error to the client, handling sync vs async send and postback wrapping in one place +normlog:{[logdict] + / detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) + / kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message + / plain {[c;m]} log dicts (info`warn`error only) pass through unchanged + $[any `getlvl`sinks`fmts in key logdict; + `info`warn`error!( + {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); + logdict] + }; + +sendclientreply:{[qid;result;status] + / deliver result or error to the client, handling sync vs async send and postback wrapping qd:queryqueue[qid]; if[qd`error;:()]; tosend:$[()~qd`postback;result;qd[`postback],enlist[qd`query],enlist result]; @@ -90,20 +87,25 @@ sendclientreply:{[qid;result;status] / deliver result or error to the client, ha @[-30!;(qd`clienth;not status;$[status;formatresponse[1b;1b;result];result]);{}]; @[neg qd`clienth;formatresponse[status;0b;tosend];()]]}; -finishquery:{[qid;err] / remove query from the live results accumulator and stamp its completion time; keeps queryqueue and results consistent +finishquery:{[qid;err] + / remove query from the live results accumulator and stamp its completion time .z.m.results:(qid,())_results; update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid}; -serverexecute:{[qid;query] / runs on the backend; traps errors locally so a crash posts an error reply rather than silently dropping the result +serverexecute:{[qid;query] + / runs on the backend - traps errors so a crash posts an error reply rather than dropping the result res:@[{(0b;value x)};query;{(1b;"server ",(string .z.h),":",(string system"p"),": ",x)}]; @[neg .z.w;$[res 0;(errorcallback;qid;res 1);(resultcallback;qid;res 1)]; {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]}; -sendquerytoserver:{[qid;query;handles] / fan the query out to all required handles and mark them in-use atomically to prevent double-dispatch +sendquerytoserver:{[qid;query;handles] + / fan the query out to all required handles and mark them in-use atomically (neg handles,:())@\:(serverexecute;qid;query); update inuse:1b from .z.M.servers where handle in handles}; -runnextquery:{ / pick the next dispatchable query and fan out to one idle server per required servertype; called after any state change that may unblock work +runnextquery:{[] + / pick the next dispatchable query and fan out to one idle server per required servertype + / called after any state change that may unblock work if[0=count torun:getnextqueryid[];:()]; torun:first torun; avail:exec first handle by servertype from availableservers 1b; @@ -115,34 +117,121 @@ runnextquery:{ / pick the next dispatchable query and fan out to one idle server results[qid]:(torun`clienth;slots); sendquerytoserver[qid;torun`query;handles]}; -checktimeout:{ / periodic scan to error queries that have waited beyond their timeout, preventing them from hanging indefinitely +addquery:{[query;servertype;join;postback;timeout;sync] + / enqueue a query without dispatching - caller must call runnextquery[] to trigger dispatch + .z.M.queryqueue upsert (queryid;.z.m.cp[];.z.w;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync); + .z.m.queryid:queryid+1}; + +removequeries:{[age] + / prevent queryqueue growing unboundedly - purge completed queries older than age + delete from .z.M.queryqueue where not null returntime, .z.m.cp[]>returntime+age}; + +removeinactive:{[age] + / prune stale disconnected-server rows to stop the servers table growing forever + delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age}; + +checktimeout:{[] + / periodic scan to error queries that have waited beyond their timeout qids:exec queryid from .z.m.queryqueue where not timeout=0Wn, null returntime, .z.m.cp[]>time+timeout; if[count qids; + .z.m.log[`warn][`asyncdispatch;"queries timed out: ",", " sv string qids]; sendclientreply[;errorprefix,"query timed out";0b] each qids; finishquery[qids;1b]]}; -removeserverhandle:{[serverh] / on backend disconnect, error in-flight queries using that handle and queued queries that can no longer be satisfied +/ ============================================================ +/ public api +/ ============================================================ + +setcp:{[f] + / replace the clock function - used in tests to control time without sleeping + .z.m.cp:f}; + +setformatresponse:{[f] + / override reply formatting - e.g. to wrap results in a standard envelope + .z.m.formatresponse:f}; + +setcallbacks:{[resfn;errfn] + / update callback symbols when module is mounted under a non-default namespace + .z.m.resultcallback:resfn; + .z.m.errorcallback:errfn}; + +setavailableservers:{[f] + / swap in a custom routing strategy without forking core dispatch + .z.m.availableservers:f}; + +setgetnextqueryid:{[f] + / inject a priority or custom scheduling strategy + .z.m.getnextqueryid:f}; + +addserver:{[h;st] + / register a backend handle and servertype so it becomes eligible for dispatch + .z.m.log[`info][`asyncdispatch;"server registered: ",string[st]," handle ",string h]; + .z.M.servers upsert (h;st;0b;1b;0Np)}; + +removeserverhandle:{[serverh] + / on backend disconnect, error in-flight queries using that handle and queued queries + / that can no longer be satisfied if[null st:first exec servertype from .z.m.servers where handle=serverh;:()]; err:errorprefix,"backend ",string[st]," server disconnected"; - + .z.m.log[`warn][`asyncdispatch;"backend disconnected: ",string st]; / in-flight: queries where this handle was assigned to a slot qids:where {[h;qid]h in value[.z.m.results[qid;1]][;0]}[serverh] each key .z.m.results; sendclientreply[;err," during query";0b] each qids; finishquery[qids;1b]; - / queued: queries that can no longer be satisfied by remaining active servers activetypes:exec distinct servertype from .z.m.servers where active, handle<>serverh; qids2:exec queryid from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, not {all x in y}[;activetypes] each servertype; sendclientreply[;err,", query cannot be satisfied";0b] each qids2; finishquery[qids2;1b]; - update active:0b,disconnecttime:.z.m.cp[] from .z.M.servers where handle=serverh; runnextquery[]}; -removeinactive:{[age]delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age} / prune stale disconnected-server rows to stop the servers table growing forever +addclientdetails:{[h] + / record client identity on connect for audit and orphan-query cleanup on disconnect + .z.m.log[`info][`asyncdispatch;"client connected: handle ",string h]; + .z.M.clients insert (.z.m.cp[];h;.z.u;.z.a;.z.h)}; + +removeclienthandle:{[h] + / on client disconnect, mark their pending queries errored so result slots are not leaked + / free any servers in-flight for this client before removing result slots, then re-dispatch + .z.m.log[`info][`asyncdispatch;"client disconnected: handle ",string h]; + inflightqids:(exec queryid from .z.m.queryqueue where clienth=h, null returntime) inter key .z.m.results; + if[count inflightqids; + inflighthandles:distinct raze {value[.z.m.results[x;1]][;0]} each inflightqids; + update inuse:0b from .z.M.servers where handle in inflighthandles]; + update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; + .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results; + runnextquery[]}; + +addserverresult:{[qid;data] + / fill one result slot - once all slots for a query are filled, run the join and reply + if[not qid in key results;:()]; + st:first exec servertype from .z.m.servers where handle=.z.w; + slots:results[qid;1]; + slots[st]:(.z.w;data;1b); + results[qid]:(results[qid;0];slots); + update inuse:0b from .z.M.servers where handle in .z.w; + runnextquery[]; + if[not qid in key results;:()]; + vals:value results[qid;1]; + if[not all vals[;2];:()]; + qd:queryqueue[qid]; + res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;errorprefix,"join failed: ",x)}]; + if[res 0;.z.m.log[`error][`asyncdispatch;"join failed for query ",string qid,": ",last res]]; + sendclientreply[qid;last res;not res 0]; + finishquery[qid;res 0]}; + +addservererror:{[qid;err] + / short-circuit a query on backend failure - free the server and notify the client + .z.m.log[`error][`asyncdispatch;"backend error for query ",string[qid],": ",err]; + sendclientreply[qid;errorprefix,err;0b]; + update inuse:0b from .z.M.servers where handle in .z.w; + runnextquery[]; + finishquery[qid;1b]}; -execquery:{[query;servertype;join;postback;timeout;sync] / public entry point; validate sync constraints then enqueue and kick dispatch +execquery:{[query;servertype;join;postback;timeout;sync] + / public entry point - validate sync constraints then enqueue and kick dispatch if[sync; if[not synccallsallowed;'"syncexec: synchronous calls are not allowed"]; if[not @[{-30!x;1b};(::);0b];'"syncexec: deferred response not supported on this connection"]; @@ -151,9 +240,32 @@ execquery:{[query;servertype;join;postback;timeout;sync] / public entry point; v addquery[query;servertype;join;postback;timeout;0b]; runnextquery[]}; -/ wire housekeeping into a timer - pass (::) to skip -init:{[timerrepeat] / wire recurring housekeeping (timeout scan, query purge, server purge) into a provided timer; pass (::) to skip registration - if[not timerrepeat~(::); - timerrepeat[cp[];0Wp;0D00:05:00;(.z.m.removequeries;querykeeptime);"asyncdispatch: remove old queries"]; - timerrepeat[cp[];0Wp;0D00:00:05;(.z.m.checktimeout;`);"asyncdispatch: timeout expired queries"]; - timerrepeat[cp[];0Wp;0D00:05:00;(.z.m.removeinactive;clearinactivetime);"asyncdispatch: remove inactive servers"]]}; +init:{[deps] + / initialise the asyncdispatch module - validate deps and apply config overrides + / deps: dict containing `log (required) plus optional config keys: + / `log - required: `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) binary loggers + / `errorprefix - optional: string prefix for client error messages. default: "error: " + / `querykeeptime - optional: timespan to keep completed queries. default: 0D00:30 + / `clearinactivetime - optional: timespan to keep disconnected servers. default: 0D01:00 + / `synccallsallowed - optional: boolean, whether sync calls are permitted. default: 0b + / housekeeping (checktimeout; removequeries; removeinactive) is the caller's responsibility - + / wire them into your timer after init; the exported defaults for age params are querykeeptime + / and clearinactivetime + / examples: + / ad.init[enlist[`log]!enlist logdep] + / ad.init[`log`querykeeptime!(logdep;0D01:00)] + if[99h<>type deps; + '"di.asyncdispatch: deps must be a dict with `log key"]; + if[not `log in key deps; + '"di.asyncdispatch: log dependency is required; pass `info`warn`error!(infofn;warnfn;errfn) keyed on `log"]; + if[99h<>type deps`log; + '"di.asyncdispatch: log value must be a dict; pass `info`warn`error functions"]; + if[not all `info`warn`error in key deps`log; + '"di.asyncdispatch: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.log:normlog deps`log; + if[`errorprefix in key deps; .z.m.errorprefix:deps`errorprefix]; + if[`querykeeptime in key deps; .z.m.querykeeptime:deps`querykeeptime]; + if[`clearinactivetime in key deps; .z.m.clearinactivetime:deps`clearinactivetime]; + if[`synccallsallowed in key deps; .z.m.synccallsallowed:deps`synccallsallowed]; + .z.m.log[`info][`asyncdispatch;"di.asyncdispatch initialised"]; + }; diff --git a/di/asyncdispatch/init.q b/di/asyncdispatch/init.q index f7086488..372cf91f 100644 --- a/di/asyncdispatch/init.q +++ b/di/asyncdispatch/init.q @@ -1,10 +1,11 @@ +/ asyncdispatch module - async scatter-gather query coordinator for multi-backend kdb+ gateway processes \l ::asyncdispatch.q export:([ - errorprefix;querykeeptime;clearinactivetime;synccallsallowed; / user-tunable config: error text prefix and housekeeping intervals - setcp;setformatresponse;setcallbacks;setavailableservers;setgetnextqueryid; / user-injectable overrides: clock, reply format, callback namespace, routing and scheduling - addserver;removeserverhandle; / server lifecycle: register and deregister backends - addclientdetails;removeclienthandle; / client lifecycle: wire into .z.po / .z.pc - addserverresult;addservererror; / IPC return paths: backends resolve these by name to deliver results or errors - execquery; / public API: only entry point for submitting a query - init]) / startup: wires housekeeping into the provided timer + setformatresponse;setcallbacks;setavailableservers;setgetnextqueryid; + addserver;removeserverhandle; + addclientdetails;removeclienthandle; + addserverresult;addservererror; + execquery; + checktimeout;removequeries;removeinactive; + init]) diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv index 7366d787..4a83ca96 100644 --- a/di/asyncdispatch/test.csv +++ b/di/asyncdispatch/test.csv @@ -1,19 +1,22 @@ action,ms,bytes,lang,code,repeat,minver,comment -before,0,0,q,system"q -p 19501 -q &",1,,start backend process +before,0,0,q,p:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19500+til 100],1,,find first free port in range 19500-19599 +before,0,0,q,system raze("q -p ";string p;" -q &"),1,,start backend process on free port before,0,0,q,system"sleep 1",1,,wait for backend to initialise before,0,0,q,ad:use`di.asyncdispatch,1,,load module before,0,0,q,srv:{.m.di.0asyncdispatch.servers},1,,live servers table before,0,0,q,qq:{.m.di.0asyncdispatch.queryqueue},1,,live queryqueue table before,0,0,q,cl:{.m.di.0asyncdispatch.clients},1,,live clients table before,0,0,q,res:{.m.di.0asyncdispatch.results},1,,live results dict -before,0,0,q,bh:hopen`::19501,1,,open handle to backend +before,0,0,q,mocklog:`info`warn`error!3#enlist{[c;m]},1,,mock log dep - binary {[c;m]} no-op +before,0,0,q,ad.init[enlist[`log]!enlist mocklog],1,,initialise with mock logger +before,0,0,q,bh:hopen`$raze("::";string p),1,,open handle to backend before,0,0,q,bh".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure backend result callback to dispatcher before,0,0,q,bh".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure backend error callback to dispatcher before,0,0,q,ad.addserver[bh;`mock],1,,register backend as mock server / Test 0: server registry true,0,0,q,(enlist bh)~exec handle from srv[] where active,1,,backend server is active -true,0,0,q,(enlist bh)~exec handle from ad.availableservers 1b,1,,idle backend server is available +true,0,0,q,(enlist bh)~exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,idle backend server is available / Test 1: queueing and FIFO scheduling run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`mock;raze;();0Wn;0b],1,,queue a query for the mock servertype @@ -32,15 +35,15 @@ true,0,0,q,not qid1 in key res[],1,,result accumulator cleaned up after join / Test 3: checktimeout run,0,0,q,.m.di.0asyncdispatch.addquery["3+3";enlist`mock;raze;();0D00:00:00.000000001;0b],1,,queue with a near-zero timeout run,0,0,q,qid2:exec first queryid from qq[] where query~\:"3+3",1,,capture the queued query id -run,0,0,q,.m.di.0asyncdispatch.checktimeout[],1,,timeout sweep should catch the expired query +run,0,0,q,ad.checktimeout[],1,,timeout sweep should catch the expired query true,0,0,q,1b~first exec error from qq[] where queryid=qid2,1,,timed-out query is flagged as error true,0,0,q,not null first exec returntime from qq[] where queryid=qid2,1,,timed-out query has returntime stamped / Test 4: removequeries purges completed queries -run,0,0,q,ad.setcp[{.z.p+2D}],1,,fast-forward the clock by 2 days -run,0,0,q,.m.di.0asyncdispatch.removequeries ad.querykeeptime,1,,purge queries older than querykeeptime -true,0,0,q,0~count select from qq[] where queryid in qid1,qid2,1,both completed queries purged -run,0,0,q,ad.setcp[{.z.p}],1,,restore the real clock +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward the clock by 2 days +run,0,0,q,ad.removequeries .m.di.0asyncdispatch.querykeeptime,1,,purge queries older than querykeeptime +true,0,0,q,0~count select from qq[] where queryid in (qid1;qid2),1,,both completed queries purged +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore the real clock / Test 5: removeserverhandle errors queued queries when the only server disconnects run,0,0,q,ad.addserver[1i;`mock2],1,,register a second mock server (fake handle - query is errored before dispatch) @@ -51,16 +54,16 @@ true,0,0,q,1b~first exec error from qq[] where queryid=qid3,1,,query errored as true,0,0,q,0b~first exec active from srv[] where handle=1i,1,,disconnected server marked inactive / Test 6: removeinactive purges inactive server records -run,0,0,q,ad.setcp[{.z.p+2D}],1,,fast-forward clock past clearinactivetime -run,0,0,q,.m.di.0asyncdispatch.removeinactive ad.clearinactivetime,1,,purge inactive servers +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward clock past clearinactivetime +run,0,0,q,ad.removeinactive .m.di.0asyncdispatch.clearinactivetime,1,,purge inactive servers true,0,0,q,0~count select from srv[] where handle=1i,1,,inactive server record removed -run,0,0,q,ad.setcp[{.z.p}],1,,restore real clock +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore real clock / Test 7: client tracking run,0,0,q,ad.addclientdetails[2i],1,,record a connected client true,0,0,q,1~count select from cl[] where clienth=2i,1,,client connection is tracked run,0,0,q,ad.removeclienthandle[2i],1,,disconnect the client -true,0,0,q,0~count select from qq[] where clienth=2i,null returntime,1,client unfinished queries cleared +true,0,0,q,0~count select from qq[] where (clienth=2i)&null returntime,1,,client unfinished queries cleared / Test 8: pluggable hooks run,0,0,q,ad.setformatresponse[{[status;sync;result]result}],1,,override formatresponse to pass result through @@ -70,6 +73,17 @@ true,0,0,q,0~count .m.di.0asyncdispatch.getnextqueryid[],1,,overridden scheduler run,0,0,q,ad.setcallbacks[`myresult;`myerror],1,,override callback symbols true,0,0,q,(`myresult;`myerror)~(.m.di.0asyncdispatch.resultcallback;.m.di.0asyncdispatch.errorcallback),1,,callback symbols updated +/ Test 9: removeclienthandle frees in-flight servers on client disconnect +run,0,0,q,ad.addserver[99i;`inflight],1,,register fake server for in-flight test +run,0,0,q,update inuse:1b from .m.di.0asyncdispatch.servers where handle=99i,1,,manually mark server in-use to simulate dispatched query +run,0,0,q,ad.addclientdetails[5i],1,,add fake client +run,0,0,q,.m.di.0asyncdispatch.addquery["5+5";enlist`inflight;raze;();0Wn;0b],1,,queue query for fake server +run,0,0,q,qid4:exec first queryid from qq[] where query~\:"5+5",1,,capture query id +run,0,0,q,.m.di.0asyncdispatch.results[qid4]:(5i;enlist[`inflight]!enlist(99i;(::);0b)),1,,inject in-flight slot to simulate dispatched-but-pending query +run,0,0,q,ad.removeclienthandle[5i],1,,disconnect client mid-flight +true,0,0,q,0b~first exec inuse from srv[] where handle=99i,1,,in-flight server freed on client disconnect +run,0,0,q,ad.removeserverhandle[99i],1,,clean up fake server + / Teardown run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to backend process and flush run,0,0,q,hclose bh,1,,close backend handle From fc58e48e774c50b4832a9224df8b1e7e05eb3b08 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Mon, 29 Jun 2026 17:42:06 +0100 Subject: [PATCH 03/14] Add tests for addservererror, postback wrapping and multi-servertype scatter-gather --- di/asyncdispatch/asyncdispatch.md | 2 +- di/asyncdispatch/test.csv | 56 +++++++++++++++++++++++++++---- 2 files changed, 50 insertions(+), 8 deletions(-) diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md index 7511df84..d720417b 100644 --- a/di/asyncdispatch/asyncdispatch.md +++ b/di/asyncdispatch/asyncdispatch.md @@ -209,7 +209,7 @@ k4unit:use`di.k4unit k4unit.moduletest`di.asyncdispatch ``` -52 tests. Requires a `q` binary in `PATH` — the test suite starts a real backend process on a dynamically selected free port and exercises the full dispatch lifecycle over a live IPC connection. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, checktimeout, removequeries, removeserverhandle with orphaned query cleanup, client tracking, removeinactive, in-flight server release on client disconnect, and all pluggable hook setters. +82 tests. Requires a `q` binary in `PATH` — the test suite starts two real backend processes on dynamically selected free ports and exercises the full dispatch lifecycle over live IPC connections. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, backend error path via real IPC callback (`addservererror`), postback wrapping (`tosend` tuple construction), multi-servertype scatter-gather with two backends dispatched in parallel and results joined, checktimeout, removequeries, removeserverhandle with orphaned query cleanup, client tracking, removeinactive, in-flight server release on client disconnect, and all pluggable hook setters. --- diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv index 4a83ca96..4dd1cba3 100644 --- a/di/asyncdispatch/test.csv +++ b/di/asyncdispatch/test.csv @@ -1,7 +1,9 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,p:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19500+til 100],1,,find first free port in range 19500-19599 -before,0,0,q,system raze("q -p ";string p;" -q &"),1,,start backend process on free port -before,0,0,q,system"sleep 1",1,,wait for backend to initialise +before,0,0,q,p2:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19600+til 100],1,,find second free port in range 19600-19699 +before,0,0,q,system raze("q -p ";string p;" -q &"),1,,start first backend process on free port +before,0,0,q,system raze("q -p ";string p2;" -q &"),1,,start second backend process on free port +before,0,0,q,system"sleep 1",1,,wait for both backends to initialise before,0,0,q,ad:use`di.asyncdispatch,1,,load module before,0,0,q,srv:{.m.di.0asyncdispatch.servers},1,,live servers table before,0,0,q,qq:{.m.di.0asyncdispatch.queryqueue},1,,live queryqueue table @@ -12,11 +14,17 @@ before,0,0,q,ad.init[enlist[`log]!enlist mocklog],1,,initialise with mock logger before,0,0,q,bh:hopen`$raze("::";string p),1,,open handle to backend before,0,0,q,bh".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure backend result callback to dispatcher before,0,0,q,bh".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure backend error callback to dispatcher -before,0,0,q,ad.addserver[bh;`mock],1,,register backend as mock server +before,0,0,q,ad.addserver[bh;`mock],1,,register first backend as mock server +before,0,0,q,bh2:hopen`$raze("::";string p2),1,,open handle to second backend +before,0,0,q,bh2".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure second backend result callback +before,0,0,q,bh2".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure second backend error callback +before,0,0,q,ad.addserver[bh2;`scatter],1,,register second backend as scatter servertype / Test 0: server registry -true,0,0,q,(enlist bh)~exec handle from srv[] where active,1,,backend server is active -true,0,0,q,(enlist bh)~exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,idle backend server is available +true,0,0,q,bh in exec handle from srv[] where active,1,,first backend server is active +true,0,0,q,bh2 in exec handle from srv[] where active,1,,second backend server is active +true,0,0,q,bh in exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,first backend is available for dispatch +true,0,0,q,bh2 in exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,second backend is available for dispatch / Test 1: queueing and FIFO scheduling run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`mock;raze;();0Wn;0b],1,,queue a query for the mock servertype @@ -84,6 +92,40 @@ run,0,0,q,ad.removeclienthandle[5i],1,,disconnect client mid-flight true,0,0,q,0b~first exec inuse from srv[] where handle=99i,1,,in-flight server freed on client disconnect run,0,0,q,ad.removeserverhandle[99i],1,,clean up fake server +/ Test 10: addservererror - real backend error path via IPC callback +run,0,0,q,ad.setcallbacks[`.m.di.0asyncdispatch.addserverresult;`.m.di.0asyncdispatch.addservererror],1,,restore callbacks overridden in test 8 +run,0,0,q,ad.setgetnextqueryid[{r:0!select from .m.di.0asyncdispatch.queryqueue where null returntime;1 sublist select from r where not queryid in key .m.di.0asyncdispatch.results}],1,,restore scheduler overridden in test 8 +run,0,0,q,.m.di.0asyncdispatch.addquery["'errortest";enlist`mock;raze;();0Wn;0b],1,,queue a query that will signal an error on the backend +run,0,0,q,qid5:exec first queryid from qq[] where query~\:"'errortest",1,,capture the queued query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch query to backend +run,0,0,q,bh"",1,,sync round-trip: ensures backend has processed and sent error callback +true,0,0,q,1b~first exec error from qq[] where queryid=qid5,1,,backend error callback flagged query as error +true,0,0,q,not null first exec returntime from qq[] where queryid=qid5,1,,returntime stamped on backend error + +/ Test 11: postback wrapping - non-empty postback produces (postback;query;result) tuple +run,0,0,q,captured::(),1,,global to capture what formatresponse receives +run,0,0,q,ad.setformatresponse[{[s;sync;r]captured::r;$[not[s]and sync;'r;r]}],1,,intercept formatresponse to verify tosend construction +run,0,0,q,.m.di.0asyncdispatch.addquery["7+7";enlist`mock;raze;enlist(::);0Wn;0b],1,,queue query with non-empty postback +run,0,0,q,qid6:exec first queryid from qq[] where query~\:"7+7",1,,capture query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch +run,0,0,q,bh"",1,,sync flush +true,0,0,q,0b~first exec error from qq[] where queryid=qid6,1,,postback query completed without error +true,0,0,q,(::;"7+7";enlist 14)~captured,1,,formatresponse received (postback;query;result) tuple +run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse + +/ Test 12: multi-servertype scatter-gather - two backends dispatched in parallel results joined +run,0,0,q,ad.setformatresponse[{[s;sync;r]captured::r;$[not[s]and sync;'r;r]}],1,,intercept formatresponse to capture joined result +run,0,0,q,.m.di.0asyncdispatch.addquery["1+1";`mock`scatter;{sum x};();0Wn;0b],1,,queue scatter-gather query requiring both servertypes +run,0,0,q,qid7:exec first queryid from qq[] where query~\:"1+1",1,,capture query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to both backends in parallel +run,0,0,q,bh"",1,,sync flush first backend +run,0,0,q,bh2"",1,,sync flush second backend +true,0,0,q,0b~first exec error from qq[] where queryid=qid7,1,,scatter-gather completed without error +true,0,0,q,4~captured,1,,joined result is sum of both backend results (1+1=2 from each; sum 2 2 = 4) +run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse + / Teardown -run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to backend process and flush -run,0,0,q,hclose bh,1,,close backend handle +run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to first backend process and flush +run,0,0,q,hclose bh,1,,close first backend handle +run,0,0,q,neg[bh2](exit;0);neg[bh2](::),1,,send exit to second backend process and flush +run,0,0,q,hclose bh2,1,,close second backend handle From d31f01aefa108639f892fe552d98fa730948160d Mon Sep 17 00:00:00 2001 From: alowrydi Date: Fri, 3 Jul 2026 17:19:39 +0100 Subject: [PATCH 04/14] Add execqueryto in-process routing, removeclients, brace style, normlog fix, 116 tests --- di/asyncdispatch/asyncdispatch.md | 68 +++++--- di/asyncdispatch/asyncdispatch.q | 264 ++++++++++++++++++------------ di/asyncdispatch/init.q | 6 +- di/asyncdispatch/test.csv | 47 ++++++ 4 files changed, 259 insertions(+), 126 deletions(-) diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md index d720417b..a4f1afc9 100644 --- a/di/asyncdispatch/asyncdispatch.md +++ b/di/asyncdispatch/asyncdispatch.md @@ -33,10 +33,10 @@ A `kx.log` instance can be passed directly — the module normalises monadic fun kxlog:use`kx.log ad:use`di.asyncdispatch -/ minimal +// minimal ad.init[enlist[`log]!enlist kxlog.createLog[]] -/ with config overrides +// with config overrides ad.init[`log`querykeeptime`synccallsallowed!(kxlog.createLog[];0D01:00;1b)] ``` @@ -54,7 +54,7 @@ ad.init[`log`querykeeptime`synccallsallowed!(kxlog.createLog[];0D01:00;1b)] | `` `clearinactivetime `` | no | `0D01:00` | How long `removeinactive` retains disconnected server rows | | `` `synccallsallowed `` | no | `0b` | Whether `execquery[...;1b]` (deferred sync mode) is permitted | -Housekeeping — `checktimeout`, `removequeries`, and `removeinactive` — is the caller's responsibility. Wire them into your gateway's timer after `init`. The configured default age parameters are accessible as `querykeeptime` and `clearinactivetime` via module state. +Housekeeping — `checktimeout`, `removequeries`, `removeinactive`, and `removeclients` — is the caller's responsibility. Wire them into your gateway's timer after `init`. The configured default age parameters are accessible as `querykeeptime` and `clearinactivetime` via module state. --- @@ -93,13 +93,13 @@ On client disconnect, mark their pending queries errored so result slots are not ### `addserverresult[qid;data]` Called when a backend posts back a successful result. Fills the result slot, frees the server, triggers `runnextquery`, and — once all slots for the query are received — applies the join function and replies to the client. ```q -/ called by serverexecute on the backend; not typically called directly +// called by serverexecute on the backend; not typically called directly ``` ### `addservererror[qid;err]` Called when a backend posts back an error. Sends the error to the client and finishes the query. ```q -/ called by serverexecute on the backend; not typically called directly +// called by serverexecute on the backend; not typically called directly ``` ### `execquery[query;servertype;join;postback;timeout;sync]` @@ -118,29 +118,54 @@ Public entry point. Validates sync constraints, enqueues the query, and triggers ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] ``` +### `execqueryto[replyto;query;servertype;join;postback;timeout;sync]` +Variant of `execquery` with an explicit reply target. Pass `replyto:0Ni` to invoke the postback locally via `value` instead of IPC-sending to a handle. Pass any valid int handle to route the reply to a specific process regardless of `.z.w`. Used when the caller is in the same process as the gateway (e.g. `di.dataaccess` dispatching shards). + +| Argument | Type | Description | +|---|---|---| +| `replyto` | int | `0Ni` for local in-process invocation; any int handle to IPC-send to a specific target | +| `query` | any | Payload passed to `value` on the backend | +| `servertype` | symbol list | One symbol per required backend type | +| `join` | function | Applied to the list of per-server results once all are received | +| `postback` | list | Required when `replyto` is `0Ni` — `(function;extra_args...)` invoked locally; `()` permitted for non-local targets | +| `timeout` | timespan | `0Wn` for no timeout | +| `sync` | boolean | Must be `0b` when `replyto` is `0Ni`; sync not supported for local invocation | + +```q +// di.dataaccess wiring: shard result delivered locally to da.shardresult +ad.execqueryto[0Ni;shardquery;servertypes;raze;(`da.shardresult;reqid);0Wn;0b] +``` + ### `checktimeout[]` Scan the queue for queries past their timeout, send a timeout error to each client, and mark them complete. Wire into your gateway's timer — every few seconds is typical. ```q -/ in gateway timer +// in gateway timer timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] ``` ### `removequeries[age]` Purge completed `queryqueue` rows older than `age`. Prevents unbounded growth. ```q -/ default age is querykeeptime (0D00:30) +// default age is querykeeptime (0D00:30) timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] ``` ### `removeinactive[age]` Purge `servers` rows for backends that have been disconnected longer than `age`. Prevents unbounded growth. ```q -/ default age is clearinactivetime (0D01:00) +// default age is clearinactivetime (0D01:00) timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] ``` +### `removeclients[age]` +Purge `clients` rows older than `age`. The `clients` table is appended to on every client connect (`addclientdetails`) for audit and is not otherwise pruned, so wire this into the timer to prevent unbounded growth. Choose `age` to match your audit retention requirements. +```q +// retain one day of client audit history +timer.addjob.default[`asyncdispatch.removeclients;{ad.removeclients[1D]};();300i;1] +``` + ### `setformatresponse[f]` -Override the reply formatter applied before a result or error is sent to the client. `f` must be `{[status;sync;result]}`. +Override the reply formatter applied before a result or error is sent to the client. `f` must be `{[status;sync;result]}`. Note: `formatresponse` is only applied on the remote IPC path — queries dispatched via `execqueryto` with `replyto:0Ni` invoke the postback directly and bypass this formatter. ```q ad.setformatresponse[{[status;sync;result]result}] ``` @@ -160,7 +185,7 @@ ad.setavailableservers[{[excl]select from servers where active}] ### `setgetnextqueryid[f]` Inject a custom scheduling strategy. `f` must be niladic and return a 0- or 1-row table with the `queryqueue` schema. ```q -/ priority queue example - highest-priority query first +// priority queue example - highest-priority query first ad.setgetnextqueryid[{1 sublist `priority xdesc 0!select from .z.m.queryqueue where null returntime}] ``` @@ -176,27 +201,28 @@ timer.init[()!()] ad:use`di.asyncdispatch ad.init[enlist[`log]!enlist kxlog.createLog[]] -/ point backends' callbacks at this module's mount point on this process +// point backends' callbacks at this module's mount point on this process ad.setcallbacks[`ad.addserverresult;`ad.addservererror] -/ register backend connections as they connect +// register backend connections as they connect ad.addserver[hopen`:backend1:5001;`rdb] ad.addserver[hopen`:backend2:5002;`hdb] -/ wire client and server connection/disconnection handlers +// wire client and server connection/disconnection handlers .z.po:{ad.addclientdetails[.z.w]} .z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} -/ wire housekeeping into the gateway timer +// wire housekeeping into the gateway timer timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] +timer.addjob.default[`asyncdispatch.removeclients;{ad.removeclients[1D]};();300i;1] -/ a client calls this asynchronously: -/ execquery dispatches to rdb and hdb in parallel, razes results, replies to client +// a client calls this asynchronously: +// execquery dispatches to rdb and hdb in parallel, razes results, replies to client ad.execquery[("select count i by sym from trade";"select count i by sym from trade");`rdb`hdb;raze;();0Wn;0b] -/ synchronous deferred mode (requires synccallsallowed:1b in deps) +// synchronous deferred mode (requires synccallsallowed:1b in deps) ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;1b] ``` @@ -209,16 +235,20 @@ k4unit:use`di.k4unit k4unit.moduletest`di.asyncdispatch ``` -82 tests. Requires a `q` binary in `PATH` — the test suite starts two real backend processes on dynamically selected free ports and exercises the full dispatch lifecycle over live IPC connections. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, backend error path via real IPC callback (`addservererror`), postback wrapping (`tosend` tuple construction), multi-servertype scatter-gather with two backends dispatched in parallel and results joined, checktimeout, removequeries, removeserverhandle with orphaned query cleanup, client tracking, removeinactive, in-flight server release on client disconnect, and all pluggable hook setters. +116 tests. Requires a `q` binary in `PATH` — the test suite starts two real backend processes on dynamically selected free ports and exercises the full dispatch lifecycle over live IPC connections. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, backend error path via real IPC callback (`addservererror`), join-failure path (throwing join function flagged as error), postback wrapping (`tosend` tuple construction), multi-servertype scatter-gather with two backends dispatched in parallel and results joined, checktimeout, removequeries, removeinactive, removeclients audit-row purge, removeserverhandle with orphaned query cleanup, client tracking, in-flight server release on client disconnect, all pluggable hook setters, and local in-process reply path via `execqueryto` (success, backend error, timeout, and non-null explicit replyto variants). --- ## Notes -- Housekeeping (`checktimeout`, `removequeries`, `removeinactive`) is the caller's responsibility — the gateway process already has a timer running and is better placed to decide intervals. Wire all three after `init`; see the usage example above +- Housekeeping (`checktimeout`, `removequeries`, `removeinactive`, `removeclients`) is the caller's responsibility — the gateway process already has a timer running and is better placed to decide intervals. Wire all four after `init`; see the usage example above +- When `checktimeout` fires for a query that is already in-flight (dispatched to a backend but not yet answered), the client receives the timeout error and the query is marked done, but the backend that received the dispatch remains `inuse:1b` until it eventually replies. A slow-but-alive backend holds its dispatch slot until `addserverresult` or `addservererror` fires; a permanently hung backend holds it until `removeserverhandle` fires on disconnect. Size backend pools and timeouts with this in mind — a run of timeouts against a stuck backend will not free its slot on timeout alone - `.z.M.` is used for in-place mutation of tables (`upsert`, `insert`, `update from`, `delete from`) and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache` - Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use the `.z.m.varname` form since q-sql evaluates column expressions in the calling context rather than the module namespace - `servertype` in `queryqueue` and `addquery` is a list of servertype symbols — one per required backend type. Pass `` enlist`rdb `` for single-server queries, `` `rdb`hdb `` for scatter-gather across two types - `setcallbacks` must be called before any queries are dispatched if the module is mounted under a non-default path — `serverexecute` reads `resultcallback` and `errorcallback` by bare name on the backend process and posts back to whatever symbols they resolve to - This module opens and accepts no connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, keeping the module dependency-free and testable in-process - All three log keys (`info`, `warn`, `error`) are required — the module calls `info` on server/client connect and init, `warn` on disconnect and timeout, and `error` on backend error and join failure +- `execqueryto` with `replyto:0Ni` invokes the postback locally via `value` — the postback head symbol must be mount-qualified (e.g. `` `da.shardresult ``) since bare exported names are not globals. This is the same resolution mechanism used by `setcallbacks` +- Local queries store `clienth:0Ni` and are not matched by `removeclienthandle` — the in-process caller owns disconnect cleanup for its own requests +- `setformatresponse` overrides only apply to the remote IPC path — local invocation via `execqueryto[0Ni;...]` calls the postback directly without applying `formatresponse`. The default `formatresponse` is a pass-through for async, so this is transparent by default; consumers that override it should not combine that with local invocation diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q index 86bcc8ce..85b7f5da 100644 --- a/di/asyncdispatch/asyncdispatch.q +++ b/di/asyncdispatch/asyncdispatch.q @@ -1,76 +1,79 @@ -/ di.asyncdispatch - async scatter-gather query coordinator -/ queues queries, dispatches to available backends, collects results per server, -/ applies a join function, and replies to the client -/ routing (which servers satisfy a query) is di.serverselect's responsibility -/ the log dependency is required - init errors immediately if absent or malformed -/ log functions are binary {[c;m]} where c is a symbol context and m is a string - -/ ============================================================ -/ module state and defaults -/ ============================================================ - -/ error prefix prepended to all error strings returned to clients +// di.asyncdispatch - async scatter-gather query coordinator +// queues queries, dispatches to available backends, collects results per server, +// applies a join function, and replies to the client +// routing (which servers satisfy a query) is di.serverselect's responsibility +// the log dependency is required - init errors immediately if absent or malformed +// log functions are binary {[c;m]} where c is a symbol context and m is a string + +// ============================================================ +// module state and defaults +// ============================================================ + +// error prefix prepended to all error strings returned to clients errorprefix:"error: "; -/ how long completed queries are kept in queryqueue before being purged +// how long completed queries are kept in queryqueue before being purged querykeeptime:0D00:30; -/ how long disconnected servers are kept in the servers table before being removed +// how long disconnected servers are kept in the servers table before being removed clearinactivetime:0D01:00; -/ whether synchronous calls via -30! are permitted +// whether synchronous calls via -30! are permitted synccallsallowed:0b; -/ injectable clock - replaced in tests to control time without sleeping +// injectable clock - replaced in tests to control time without sleeping cp:{.z.p}; -/ symbols backend servers call back via - stored as symbols so names survive IPC serialisation +// symbols backend servers call back via - stored as symbols so names survive IPC serialisation resultcallback:`addserverresult; errorcallback:`addservererror; -/ reply formatting - sync errors must be signalled with ' so the client receives a trapped error +// reply formatting - sync errors must be signalled with ' so the client receives a trapped error formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]}; -/ scheduling strategy - pick oldest FIFO-eligible query by default +// scheduling strategy - pick oldest FIFO-eligible query by default getnextqueryid:{ avail:exec distinct servertype from availableservers 1b; + // 0! is required - select from a keyed table stays keyed in kdb-x, and runnextquery needs queryid via first runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; - 1 sublist select from runnable where time=min time}; + 1 sublist select from runnable where time=min time + }; -/ server routing strategy - active and idle by default +// server routing strategy - active and idle by default availableservers:{[excludeinuse] $[excludeinuse; select from servers where active, not inuse; - select from servers where active]}; + select from servers where active] + }; -/ ============================================================ -/ module tables -/ ============================================================ +// ============================================================ +// module tables +// ============================================================ -/ registered backend servers +// registered backend servers servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()); -/ pending and in-flight client queries -queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$()); +// pending and in-flight client queries +queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$(); local:`boolean$()); -/ connected client tracking +// connected client tracking clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()); -/ per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) +// per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) results:()!(); -/ auto-incrementing query id counter +// auto-incrementing query id counter queryid:0; -/ ============================================================ -/ internal helpers -/ ============================================================ +// ============================================================ +// internal helpers +// ============================================================ normlog:{[logdict] - / detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) - / kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message - / plain {[c;m]} log dicts (info`warn`error only) pass through unchanged - $[any `getlvl`sinks`fmts in key logdict; + // detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) + // kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message + // plain {[c;m]} log dicts (info`warn`error only) pass through unchanged + $[all `getlvl`sinks`fmts in key logdict; `info`warn`error!( {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; @@ -79,122 +82,156 @@ normlog:{[logdict] }; sendclientreply:{[qid;result;status] - / deliver result or error to the client, handling sync vs async send and postback wrapping + // deliver result or error to the client, handling sync vs async send and postback wrapping + // local path invokes value tosend directly - formatresponse is not applied (it is a pass-through + // for async by default; consumers that override setformatresponse should not use local invocation) qd:queryqueue[qid]; if[qd`error;:()]; tosend:$[()~qd`postback;result;qd[`postback],enlist[qd`query],enlist result]; $[qd`sync; @[-30!;(qd`clienth;not status;$[status;formatresponse[1b;1b;result];result]);{}]; - @[neg qd`clienth;formatresponse[status;0b;tosend];()]]}; + $[qd`local; + @[value;tosend;{.z.m.log[`error][`asyncdispatch;"local postback failed: ",x]}]; + @[neg qd`clienth;formatresponse[status;0b;tosend];()]]]; + }; finishquery:{[qid;err] - / remove query from the live results accumulator and stamp its completion time + // remove query from the live results accumulator and stamp its completion time .z.m.results:(qid,())_results; - update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid}; + update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid; + }; serverexecute:{[qid;query] - / runs on the backend - traps errors so a crash posts an error reply rather than dropping the result + // runs on the backend - traps errors so a crash posts an error reply rather than dropping the result res:@[{(0b;value x)};query;{(1b;"server ",(string .z.h),":",(string system"p"),": ",x)}]; @[neg .z.w;$[res 0;(errorcallback;qid;res 1);(resultcallback;qid;res 1)]; - {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]}; + {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]; + }; sendquerytoserver:{[qid;query;handles] - / fan the query out to all required handles and mark them in-use atomically + // fan the query out to all required handles and mark them in-use atomically (neg handles,:())@\:(serverexecute;qid;query); - update inuse:1b from .z.M.servers where handle in handles}; + update inuse:1b from .z.M.servers where handle in handles; + }; runnextquery:{[] - / pick the next dispatchable query and fan out to one idle server per required servertype - / called after any state change that may unblock work + // pick the next dispatchable query and fan out to one idle server per required servertype + // called after any state change that may unblock work if[0=count torun:getnextqueryid[];:()]; torun:first torun; avail:exec first handle by servertype from availableservers 1b; types:torun`servertype; handles:avail types; qid:torun`queryid; - slots:types!(count[types],())#enlist(0Ni;(::);0b); + slots:types!count[types]#enlist(0Ni;(::);0b); slots[types;0]:handles; + // indexed assignment on bare results propagates through to .z.m in kdb-x (empirically verified) results[qid]:(torun`clienth;slots); - sendquerytoserver[qid;torun`query;handles]}; + sendquerytoserver[qid;torun`query;handles]; + }; + +addqueryto:{[query;servertype;join;postback;timeout;sync;replyto;local] + // enqueue a query with an explicit reply target - used by execqueryto for local in-process routing + .z.M.queryqueue upsert (queryid;.z.m.cp[];replyto;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync;local); + .z.m.queryid:queryid+1; + }; addquery:{[query;servertype;join;postback;timeout;sync] - / enqueue a query without dispatching - caller must call runnextquery[] to trigger dispatch - .z.M.queryqueue upsert (queryid;.z.m.cp[];.z.w;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync); - .z.m.queryid:queryid+1}; + // enqueue a query without dispatching - caller must call runnextquery[] to trigger dispatch + addqueryto[query;servertype;join;postback;timeout;sync;.z.w;0b]; + }; removequeries:{[age] - / prevent queryqueue growing unboundedly - purge completed queries older than age - delete from .z.M.queryqueue where not null returntime, .z.m.cp[]>returntime+age}; + // prevent queryqueue growing unboundedly - purge completed queries older than age + delete from .z.M.queryqueue where not null returntime, .z.m.cp[]>returntime+age; + }; removeinactive:{[age] - / prune stale disconnected-server rows to stop the servers table growing forever - delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age}; + // prune stale disconnected-server rows to stop the servers table growing forever + delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age; + }; + +removeclients:{[age] + // prune stale client audit rows to stop the clients table growing forever + // clients are recorded on every connect by addclientdetails and never removed otherwise + delete from .z.M.clients where .z.m.cp[]>time+age; + }; checktimeout:{[] - / periodic scan to error queries that have waited beyond their timeout + // periodic scan to error queries that have waited beyond their timeout qids:exec queryid from .z.m.queryqueue where not timeout=0Wn, null returntime, .z.m.cp[]>time+timeout; if[count qids; .z.m.log[`warn][`asyncdispatch;"queries timed out: ",", " sv string qids]; sendclientreply[;errorprefix,"query timed out";0b] each qids; - finishquery[qids;1b]]}; + finishquery[qids;1b]]; + }; -/ ============================================================ -/ public api -/ ============================================================ +// ============================================================ +// public api +// ============================================================ setcp:{[f] - / replace the clock function - used in tests to control time without sleeping - .z.m.cp:f}; + // replace the clock function - used in tests to control time without sleeping + .z.m.cp:f; + }; setformatresponse:{[f] - / override reply formatting - e.g. to wrap results in a standard envelope - .z.m.formatresponse:f}; + // override reply formatting - e.g. to wrap results in a standard envelope + .z.m.formatresponse:f; + }; setcallbacks:{[resfn;errfn] - / update callback symbols when module is mounted under a non-default namespace + // update callback symbols when module is mounted under a non-default namespace .z.m.resultcallback:resfn; - .z.m.errorcallback:errfn}; + .z.m.errorcallback:errfn; + }; setavailableservers:{[f] - / swap in a custom routing strategy without forking core dispatch - .z.m.availableservers:f}; + // swap in a custom routing strategy without forking core dispatch + .z.m.availableservers:f; + }; setgetnextqueryid:{[f] - / inject a priority or custom scheduling strategy - .z.m.getnextqueryid:f}; + // inject a priority or custom scheduling strategy + .z.m.getnextqueryid:f; + }; addserver:{[h;st] - / register a backend handle and servertype so it becomes eligible for dispatch + // register a backend handle and servertype so it becomes eligible for dispatch .z.m.log[`info][`asyncdispatch;"server registered: ",string[st]," handle ",string h]; - .z.M.servers upsert (h;st;0b;1b;0Np)}; + .z.M.servers upsert (h;st;0b;1b;0Np); + }; removeserverhandle:{[serverh] - / on backend disconnect, error in-flight queries using that handle and queued queries - / that can no longer be satisfied + // on backend disconnect, error in-flight queries using that handle and queued queries + // that can no longer be satisfied if[null st:first exec servertype from .z.m.servers where handle=serverh;:()]; err:errorprefix,"backend ",string[st]," server disconnected"; .z.m.log[`warn][`asyncdispatch;"backend disconnected: ",string st]; - / in-flight: queries where this handle was assigned to a slot + // in-flight: queries where this handle was assigned to a slot qids:where {[h;qid]h in value[.z.m.results[qid;1]][;0]}[serverh] each key .z.m.results; sendclientreply[;err," during query";0b] each qids; finishquery[qids;1b]; - / queued: queries that can no longer be satisfied by remaining active servers + // queued: queries that can no longer be satisfied by remaining active servers activetypes:exec distinct servertype from .z.m.servers where active, handle<>serverh; qids2:exec queryid from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, not {all x in y}[;activetypes] each servertype; sendclientreply[;err,", query cannot be satisfied";0b] each qids2; finishquery[qids2;1b]; update active:0b,disconnecttime:.z.m.cp[] from .z.M.servers where handle=serverh; - runnextquery[]}; + runnextquery[]; + }; addclientdetails:{[h] - / record client identity on connect for audit and orphan-query cleanup on disconnect + // record client identity on connect for audit and orphan-query cleanup on disconnect .z.m.log[`info][`asyncdispatch;"client connected: handle ",string h]; - .z.M.clients insert (.z.m.cp[];h;.z.u;.z.a;.z.h)}; + .z.M.clients insert (.z.m.cp[];h;.z.u;.z.a;.z.h); + }; removeclienthandle:{[h] - / on client disconnect, mark their pending queries errored so result slots are not leaked - / free any servers in-flight for this client before removing result slots, then re-dispatch + // on client disconnect, mark their pending queries errored so result slots are not leaked + // free any servers in-flight for this client before removing result slots, then re-dispatch + // local queries store clienth:0Ni and are not matched here - the in-process caller owns cleanup for its own requests .z.m.log[`info][`asyncdispatch;"client disconnected: handle ",string h]; inflightqids:(exec queryid from .z.m.queryqueue where clienth=h, null returntime) inter key .z.m.results; if[count inflightqids; @@ -202,10 +239,12 @@ removeclienthandle:{[h] update inuse:0b from .z.M.servers where handle in inflighthandles]; update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results; - runnextquery[]}; + runnextquery[]; + }; addserverresult:{[qid;data] - / fill one result slot - once all slots for a query are filled, run the join and reply + // fill one result slot - once all slots for a query are filled, run the join and reply + // bare indexed assignment on results propagates through to .z.m in kdb-x (empirically verified) if[not qid in key results;:()]; st:first exec servertype from .z.m.servers where handle=.z.w; slots:results[qid;1]; @@ -217,43 +256,60 @@ addserverresult:{[qid;data] vals:value results[qid;1]; if[not all vals[;2];:()]; qd:queryqueue[qid]; - res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;errorprefix,"join failed: ",x)}]; + res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;.z.m.errorprefix,"join failed: ",x)}]; if[res 0;.z.m.log[`error][`asyncdispatch;"join failed for query ",string qid,": ",last res]]; sendclientreply[qid;last res;not res 0]; - finishquery[qid;res 0]}; + finishquery[qid;res 0]; + }; addservererror:{[qid;err] - / short-circuit a query on backend failure - free the server and notify the client + // short-circuit a query on backend failure - free the server and notify the client .z.m.log[`error][`asyncdispatch;"backend error for query ",string[qid],": ",err]; sendclientreply[qid;errorprefix,err;0b]; update inuse:0b from .z.M.servers where handle in .z.w; runnextquery[]; - finishquery[qid;1b]}; + finishquery[qid;1b]; + }; execquery:{[query;servertype;join;postback;timeout;sync] - / public entry point - validate sync constraints then enqueue and kick dispatch + // public entry point - validate sync constraints then enqueue and kick dispatch if[sync; + if[not ()~postback;.z.m.log[`warn][`asyncdispatch;"execquery: postback ignored for sync call"]]; if[not synccallsallowed;'"syncexec: synchronous calls are not allowed"]; if[not @[{-30!x;1b};(::);0b];'"syncexec: deferred response not supported on this connection"]; .[{[q;s;j;t]addquery[q;s;j;();t;1b];runnextquery[]};(query;servertype;join;timeout);{-30!(.z.w;1b;x)}]; :()]; addquery[query;servertype;join;postback;timeout;0b]; - runnextquery[]}; + runnextquery[]; + }; + +execqueryto:{[replyto;query;servertype;join;postback;timeout;sync] + // in-process variant of execquery - replyto is 0Ni for local invocation via value + // postback must be non-empty when replyto is 0Ni - there is no handle to send a bare result to + // sync is not supported for local invocation + // mount-qualified postback symbols required for local invocation e.g. `da.shardresult not `shardresult + if[0Ni~replyto; + if[()~postback;'"di.asyncdispatch: local invocation requires a non-empty postback"]; + if[sync;'"di.asyncdispatch: local invocation does not support sync mode"]]; + local:0Ni~replyto; + addqueryto[query;servertype;join;postback;timeout;sync;replyto;local]; + runnextquery[]; + }; init:{[deps] - / initialise the asyncdispatch module - validate deps and apply config overrides - / deps: dict containing `log (required) plus optional config keys: - / `log - required: `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) binary loggers - / `errorprefix - optional: string prefix for client error messages. default: "error: " - / `querykeeptime - optional: timespan to keep completed queries. default: 0D00:30 - / `clearinactivetime - optional: timespan to keep disconnected servers. default: 0D01:00 - / `synccallsallowed - optional: boolean, whether sync calls are permitted. default: 0b - / housekeeping (checktimeout; removequeries; removeinactive) is the caller's responsibility - - / wire them into your timer after init; the exported defaults for age params are querykeeptime - / and clearinactivetime - / examples: - / ad.init[enlist[`log]!enlist logdep] - / ad.init[`log`querykeeptime!(logdep;0D01:00)] + // initialise the asyncdispatch module - validate deps and apply config overrides + // deps: dict containing `log (required) plus optional config keys: + // `log - required: `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) binary loggers + // `errorprefix - optional: string prefix for client error messages. default: "error: " + // `querykeeptime - optional: timespan to keep completed queries. default: 0D00:30 + // `clearinactivetime - optional: timespan to keep disconnected servers. default: 0D01:00 + // `synccallsallowed - optional: boolean, whether sync calls are permitted. default: 0b + // housekeeping (checktimeout; removequeries; removeinactive; removeclients) is the caller's + // responsibility - wire them into your timer after init; the exported defaults for age params are + // querykeeptime and clearinactivetime + // examples: + // ad.init[enlist[`log]!enlist logdep] + // ad.init[`log`querykeeptime!(logdep;0D01:00)] if[99h<>type deps; '"di.asyncdispatch: deps must be a dict with `log key"]; if[not `log in key deps; diff --git a/di/asyncdispatch/init.q b/di/asyncdispatch/init.q index 372cf91f..21200d8a 100644 --- a/di/asyncdispatch/init.q +++ b/di/asyncdispatch/init.q @@ -1,4 +1,4 @@ -/ asyncdispatch module - async scatter-gather query coordinator for multi-backend kdb+ gateway processes +// asyncdispatch module - async scatter-gather query coordinator for multi-backend kdb+ gateway processes \l ::asyncdispatch.q export:([ @@ -6,6 +6,6 @@ export:([ addserver;removeserverhandle; addclientdetails;removeclienthandle; addserverresult;addservererror; - execquery; - checktimeout;removequeries;removeinactive; + execquery;execqueryto; + checktimeout;removequeries;removeinactive;removeclients; init]) diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv index 4dd1cba3..8b202caf 100644 --- a/di/asyncdispatch/test.csv +++ b/di/asyncdispatch/test.csv @@ -124,6 +124,53 @@ true,0,0,q,0b~first exec error from qq[] where queryid=qid7,1,,scatter-gather co true,0,0,q,4~captured,1,,joined result is sum of both backend results (1+1=2 from each; sum 2 2 = 4) run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse +/ Test 13: execqueryto local path - successful result delivered in-process +run,0,0,q,ad.setcallbacks[`.m.di.0asyncdispatch.addserverresult;`.m.di.0asyncdispatch.addservererror],1,,restore callbacks +run,0,0,q,ad.setgetnextqueryid[{r:0!select from .m.di.0asyncdispatch.queryqueue where null returntime;1 sublist select from r where not queryid in key .m.di.0asyncdispatch.results}],1,,restore scheduler +run,0,0,q,.test.localcap:(::),1,,reset local capture +run,0,0,q,.test.localfn:{[reqid;qry;res] .test.localcap:(reqid;qry;res)},1,,define local capturing callback in root namespace +run,0,0,q,ad.execqueryto[0Ni;"2+2";enlist`mock;{sum x};(.test.localfn;`req1);0Wn;0b],1,,dispatch via local path with postback (sum collapses single-element result to scalar) +run,0,0,q,bh"",1,,sync flush - wait for backend result callback +true,0,0,q,(`req1;"2+2";4)~.test.localcap,1,,postback invoked in-process with (reqid;query;result) + +/ Test 14: execqueryto local path - backend error delivered in-process via postback +run,0,0,q,.test.localcap:(::),1,,reset local capture +run,0,0,q,ad.execqueryto[0Ni;"'errortest";enlist`mock;raze;(.test.localfn;`req2);0Wn;0b],1,,dispatch erroring query via local path +run,0,0,q,bh"",1,,sync flush - wait for backend error callback +true,0,0,q,`req2~.test.localcap[0],1,,reqid correct +true,0,0,q,.test.localcap[2] like "error:*",1,,error string delivered as result via postback + +/ Test 15: execqueryto local path - timeout delivered in-process via postback +run,0,0,q,.test.localcap:(::),1,,reset local capture +run,0,0,q,ad.execqueryto[0Ni;"3+3";enlist`mock;raze;(.test.localfn;`req3);0D00:00:00.000000001;0b],1,,dispatch with near-zero timeout +run,0,0,q,ad.checktimeout[],1,,trigger timeout sweep +true,0,0,q,`req3~.test.localcap[0],1,,reqid correct +true,0,0,q,.test.localcap[2] like "error:*",1,,timeout error delivered in-process via postback + +/ Test 16: removeclients purges stale client audit rows +run,0,0,q,ad.addclientdetails[7i],1,,record a client for the cleanup test +true,0,0,q,1~count select from cl[] where clienth=7i,1,,client audit row exists +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward clock two days past the audit age +run,0,0,q,ad.removeclients[0D00:01],1,,purge client rows older than one minute +true,0,0,q,0~count select from cl[] where clienth=7i,1,,client audit row purged +run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore the real clock + +/ Test 17: join-failure path - join function throws, client receives error and query is finished +/ dispatched to the scatter backend (bh2): the mock backend (bh) is left inuse by the in-flight timeout in test 15 +run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`scatter;{'"joinboom"};();0Wn;0b],1,,queue a query whose join function throws +run,0,0,q,qid8:exec first queryid from qq[] where join~\:{'"joinboom"},1,,capture the query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to the scatter backend +run,0,0,q,bh2"",1,,sync flush - wait for the result callback and join attempt +true,0,0,q,1b~first exec error from qq[] where queryid=qid8,1,,join failure correctly flagged as error +true,0,0,q,not null first exec returntime from qq[] where queryid=qid8,1,,returntime stamped after join failure + +/ Test 18: execqueryto with a non-null explicit replyto stores it as clienth with local=0b +run,0,0,q,ad.execqueryto[99i;"1+1";enlist`scatter;{sum x};();0Wn;0b],1,,dispatch with explicit replyto handle 99i +run,0,0,q,qid9:exec first queryid from qq[] where (clienth=99i)&query~\:"1+1",1,,capture the query id +true,0,0,q,99i~first exec clienth from qq[] where queryid=qid9,1,,replyto stored as clienth +true,0,0,q,0b~first exec local from qq[] where queryid=qid9,1,,local flag is 0b for a non-null replyto +run,0,0,q,bh2"",1,,sync flush - completes the backend round-trip (reply to fake 99i fails silently) + / Teardown run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to first backend process and flush run,0,0,q,hclose bh,1,,close first backend handle From 430166728a92e05d352ca2a1b8a87e3abbe93246 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Mon, 6 Jul 2026 15:10:35 +0100 Subject: [PATCH 05/14] added in pluggable server source without requiring a hard dependacy --- di/asyncdispatch/asyncdispatch.md | 17 +++++++++++++---- di/asyncdispatch/asyncdispatch.q | 8 ++++++-- di/asyncdispatch/test.csv | 12 ++++++++++++ 3 files changed, 31 insertions(+), 6 deletions(-) diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md index a4f1afc9..c11d5c80 100644 --- a/di/asyncdispatch/asyncdispatch.md +++ b/di/asyncdispatch/asyncdispatch.md @@ -2,7 +2,7 @@ Async scatter-gather query coordinator for kdb-x gateway processes. Queues client queries, dispatches them to available backend processes by servertype, collects per-server results, applies a join function, and replies to the client — with timeout management and correct error propagation if a backend disconnects mid-query. -Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s responsibility. This module receives a resolved servertype list and dispatches to whatever idle backends of each type are registered. +Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s responsibility, but **`di.serverselect` is not a dependency** — this module dispatches a resolved servertype list to whatever backends its *server source* reports. That source is pluggable with a default: by default it is asyncdispatch's own registry (populate it with `addserver`), or you point it at `di.serverselect`'s output via `setavailableservers`. Either works standalone; neither is required. --- @@ -14,6 +14,7 @@ Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s resp - Handle backend disconnects mid-query — errors in-flight queries and queued queries that can no longer be satisfied - Track connected clients and clean up orphaned queries on client disconnect - Support synchronous deferred response mode (`-30!`) alongside the default async mode +- Pluggable server source with a default — dispatch against the built-in registry (`addserver`) by default, or point it at `di.serverselect`'s output (or any source) via `setavailableservers`; `di.serverselect` is a composable option, never a dependency - Accept fully pluggable scheduler (`setgetnextqueryid`), routing (`setavailableservers`), reply formatter (`setformatresponse`), and callback symbols (`setcallbacks`) — swap without touching core dispatch logic - Detect and normalise `kx.log` instances automatically so callers can pass a logger directly without manual wrapping @@ -27,6 +28,8 @@ Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s resp The `log` dependency must be passed to `init` inside the `deps` dict. The module throws immediately if it is absent or missing any of the three required keys. All three are required since the module calls `info`, `warn`, and `error`. +> **`di.serverselect` is not a dependency.** asyncdispatch never imports or calls it. The server source is pluggable (`setavailableservers`) and defaults to the built-in registry (`addserver`), so you run standalone out of the box; to dispatch against serverselect's live view instead, inject its `getservers` output as the source (see `setavailableservers`). No registry copying, no dependency. + A `kx.log` instance can be passed directly — the module normalises monadic functions to the binary `{[c;m]}` contract automatically via `normlog`. Context is embedded in the output as `"context: message"`: ```q @@ -67,7 +70,7 @@ ad.init[enlist[`log]!enlist logdep] ``` ### `addserver[handle;servertype]` -Register a backend connection. `handle`: open int handle. `servertype`: symbol identifying the process type (e.g. `` `rdb ``, `` `hdb ``). +Register a backend connection into the **built-in (default) server source**. `handle`: open int handle. `servertype`: symbol identifying the process type (e.g. `` `rdb ``, `` `hdb ``). Use this when asyncdispatch owns the server list; to source servers from `di.serverselect` instead, leave `addserver` unused and inject via `setavailableservers`. ```q ad.addserver[hopen`:backend1:5001;`rdb] ``` @@ -177,10 +180,16 @@ ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] ``` ### `setavailableservers[f]` -Swap in a custom routing strategy. `f` must be `{[excludeinuse]}` returning a table with a `servertype` column. +Replace the **server source** — the function dispatch uses to find backends. `f` is `{[excludeinuse]}` and must return a table with `handle` and `servertype` columns. This is the seam for running against `di.serverselect` (or any external source) **without a dependency and without copying its registry**: by default the source reads asyncdispatch's own registry (`addserver`), and you override it to read serverselect instead. ```q -ad.setavailableservers[{[excl]select from servers where active}] +// default (built-in registry) — equivalent to not calling this at all +ad.setavailableservers[{[eu] $[eu; select from servers where active, not inuse; select from servers where active]}] + +// compose with di.serverselect (not a dependency — just its output as the source) +srvsel:use`di.serverselect +ad.setavailableservers[{[eu] select handle, servertype from srvsel.getservers[`servertype;`;()!()]}] ``` +Result routing does **not** depend on the source — a returning handle is matched to its servertype from the query's own dispatch record — so an injected source needs no registration in asyncdispatch. Note `inuse` throttling applies only to the built-in registry; an external source is expected to do its own idle/selection (e.g. serverselect's `roundrobin`), and the `excludeinuse` flag is a hint such a source may ignore. ### `setgetnextqueryid[f]` Inject a custom scheduling strategy. `f` must be niladic and return a 0- or 1-row table with the `queryqueue` schema. diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q index 85b7f5da..c2f812b5 100644 --- a/di/asyncdispatch/asyncdispatch.q +++ b/di/asyncdispatch/asyncdispatch.q @@ -197,7 +197,9 @@ setgetnextqueryid:{[f] }; addserver:{[h;st] - // register a backend handle and servertype so it becomes eligible for dispatch + // register a backend handle and servertype so it becomes eligible for dispatch. + // this is the default (built-in) server source; to dispatch against di.serverselect's view instead, + // inject it via setavailableservers - no registration and no di.serverselect dependency required .z.m.log[`info][`asyncdispatch;"server registered: ",string[st]," handle ",string h]; .z.M.servers upsert (h;st;0b;1b;0Np); }; @@ -246,8 +248,10 @@ addserverresult:{[qid;data] // fill one result slot - once all slots for a query are filled, run the join and reply // bare indexed assignment on results propagates through to .z.m in kdb-x (empirically verified) if[not qid in key results;:()]; - st:first exec servertype from .z.m.servers where handle=.z.w; slots:results[qid;1]; + // map the responding handle to its servertype from THIS query's own dispatch record, not a server + // registry - so any server source (built-in or an injected di.serverselect view) works unchanged + st:first where .z.w=slots[;0]; slots[st]:(.z.w;data;1b); results[qid]:(results[qid;0];slots); update inuse:0b from .z.M.servers where handle in .z.w; diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv index 8b202caf..41ba99a8 100644 --- a/di/asyncdispatch/test.csv +++ b/di/asyncdispatch/test.csv @@ -171,6 +171,18 @@ true,0,0,q,99i~first exec clienth from qq[] where queryid=qid9,1,,replyto stored true,0,0,q,0b~first exec local from qq[] where queryid=qid9,1,,local flag is 0b for a non-null replyto run,0,0,q,bh2"",1,,sync flush - completes the backend round-trip (reply to fake 99i fails silently) +/ Test 19: pluggable server source - dispatch against an injected source whose servertype is not in the +/ internal registry; result routing uses the query's dispatch record so no internal registration is needed +run,0,0,q,savedavail:.m.di.0asyncdispatch.availableservers,1,,save the default (internal-registry) server source +run,0,0,q,ad.setavailableservers[{[eu]([]handle:enlist bh;servertype:enlist`ext)}],1,,inject a source presenting bh under servertype ext (bh is registered as mock) +run,0,0,q,.m.di.0asyncdispatch.addquery["6*7";enlist`ext;raze;();0Wn;0b],1,,queue a query for the external-only servertype +run,0,0,q,qidx:exec first queryid from qq[] where query~\:"6*7",1,,capture the query id +run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to bh via the injected source +run,0,0,q,bh"",1,,sync flush - wait for the backend result callback +true,0,0,q,0b~first exec error from qq[] where queryid=qidx,1,,query completed without error via the injected source +true,0,0,q,not null first exec returntime from qq[] where queryid=qidx,1,,result routed via the dispatch record (servertype ext) not the servers table (mock) +run,0,0,q,ad.setavailableservers[savedavail],1,,restore the default internal-registry source + / Teardown run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to first backend process and flush run,0,0,q,hclose bh,1,,close first backend handle From 68b0ba68fd9dac63c4b1262348d7315d307fbb5e Mon Sep 17 00:00:00 2001 From: ascottDI Date: Thu, 9 Jul 2026 11:44:54 +0100 Subject: [PATCH 06/14] closing in on a completed v1 of this module --- di/config/config.md | 145 +++++++++++++++++++++++++++++ di/config/config.q | 219 ++++++++++++++++++++++++++++++++++++++++++++ di/config/init.q | 5 + di/config/test.csv | 115 +++++++++++++++++++++++ 4 files changed, 484 insertions(+) create mode 100644 di/config/config.md create mode 100644 di/config/config.q create mode 100644 di/config/init.q create mode 100644 di/config/test.csv diff --git a/di/config/config.md b/di/config/config.md new file mode 100644 index 00000000..d7419d37 --- /dev/null +++ b/di/config/config.md @@ -0,0 +1,145 @@ +# di.config + +Configuration loading and cascade resolution for the modular TorQ world. This +replaces TorQ's in-process config handling (`torq.q`: `loadf`, `loaddir`, +`loadconfig`, `loadaddconfig`, `overrideconfig`), where `di.torq` will later +distribute the resolved config to each module via that module's `init`. + +## Import and init + +`init` requires a `log` dependency — there is no silent fallback. Pass an +already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` loggers (context +symbol, message string). The module performs **no** adaptation, so a raw monadic +[`kx.log`](https://github.com/KxSystems/logging) instance must be wrapped by the +caller first (this is `di.log`'s job once it ships): + +```q +config:use`di.config + +/ option 1: di.log (standard, once it ships) - build the binary dict from its exports +logger:use`di.log +logdep:`info`warn`error!(logger.info;logger.warn;logger.error) +config.init[enlist[`log]!enlist logdep] + +/ option 2: hand-rolled binary {[c;m]} logger (works today) +mylog:`info`warn`error!( + {[c;m] -1 string[c],": INFO ",m;}; + {[c;m] -1 string[c],": WARN ",m;}; + {[c;m] -2 string[c],": ERROR ",m;}); +config.init[enlist[`log]!enlist mylog] +``` + +To wrap a raw `kx.log` instance yourself in the interim, fold the context into the +message per call: `` `info`warn`error!({[c;m]inst[`info][string[c],": ",m]};…) ``. + +`init` must be called before any other function — there is no default logger. + +## Exported functions + +| Function | Signature | Description | +|---|---|---| +| `init` | `init[deps]` | Wire the injected logger. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | +| `loadfile` | `loadfile[file]` | Load a single q config file (a string path) if it exists, tracking it so it is not re-loaded. Returns the path. A missing file is a logged warning, not an error; a failed load is signalled. | +| `loadconfig` | `loadconfig[dir;name]` | Load one cascade config file `dir/{name}.q` (`dir` a string path, `name` a symbol) if present. Returns the constructed path. A missing file is **normal** in the cascade, so its absence is logged at *info* and skipped (not warned). Present files load via `loadfile`. | +| `loaddir` | `loaddir[dir]` | Load every `.q`/`.k` file in a directory (a string path), honouring an optional `order.txt`. Files listed in `order.txt` load first, in that order; the rest follow. Returns the ordered list of file paths processed. A missing directory is a logged warning returning `()`. Delegates each load to `loadfile`, so already-loaded files are skipped. | +| `loadcascade` | `loadcascade[dirs;names]` | Resolve a config cascade: for each settings directory (in order) load each named file `dir/{name}.q` (in name order). `dirs` is a string path or list of paths; `names` is a symbol or symbol list. Later names and later directories override earlier ones (load order). Missing files are skipped. Returns the flat list of constructed paths in cascade order. | +| `overrideconfig` | `overrideconfig[params]` | Apply command-line-style overrides after the cascade. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | +| `get` | `get[namespace;key;default]` | Query the resolved config store: return the value of `.{namespace}.{key}`, or `default` if unset. `namespace` and `key` are **bare** symbols (no leading dot), e.g. `get[\`rdb;\`subscribeto;\`]`. (`get` is a reserved word, so the implementation function is named `getcfg` and exported under the `get` key.) | +| `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. | + +`order.txt` is a plain-text file, one filename per line, naming the files that must +load ahead of the rest (e.g. to satisfy load-order dependencies). + +Internal helpers (`raiseerror`, `applyoverride`) and the load-tracking state are deliberately +not exported. + +`loadcascade` is the top-level entry point. A caller (e.g. `di.torq`, which resolves the +process identity and directory paths) drives it as: + +```q +config.loadcascade[(kdbconfig;appconfig);`default,proctype,procname] +config.overrideconfig[`.myproc.enabled`.myproc.rows!(enlist"1";enlist"5000")] +/ then partition per module and hand each slice to that module's init: +rdbcfg:config.getmodule[`rdb] / -> `subscribeto`hdbtypes!(...) etc. +rdb.init[rdbcfg;`log`timer!(logdep;timerdep)] +/ or read a single value with a fallback: +config.get[`rdb;`subscribeto;`trade`quote] +``` + +`overrideconfig` runs after `loadcascade` so command-line values win over file config. +di.config is deliberately generic — it does not read environment variables or process +identity itself; the caller (di.torq) supplies the settings directories, the ordered +name sequence, and the override params, then uses `getmodule` to partition the resolved +store per module. + +### The config store + +`loadcascade` loads settings `.q` files that assign into root namespaces +(`.rdb.subscribeto:...`); those resolved namespaces **are** the config store, and +`get`/`getmodule` query them (so anything `overrideconfig` changes is reflected too). +Additional config sources (env vars, k8s config maps) are out of scope for v1 but would +plug in by populating the same namespaces before the store is queried — the `get`/ +`getmodule` contract stays unchanged. See the EXTENSION POINT comment in `config.q`. + +> **Note:** avoid `-` in config file paths — a source-level backtick symbol literal +> containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one +> symbol. Runtime string paths passed to `loadfile` are unaffected. + +## Injectable dependencies + +| Injectable | Required keys | Function signature | +|---|---|---| +| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string); caller passes an already-conforming binary dict (from `di.log` once it ships, or hand-rolled). No adaptation is done in the module — a raw monadic `kx.log` instance must be wrapped first | + +## Hard dependencies + +None — `di.config` is a standalone module. + +## Design notes & open gaps + +- **Return values are informational.** `loadfile`/`loadconfig`/`loaddir`/`loadcascade` return the + path(s) they *considered* — including files that were missing or skipped. The real effect is the + side-effect load; use the return to see what was attempted, not what definitely loaded. +- **Missing-file log level is deliberate.** `loadfile` logs a missing explicit path at *warn* (you + asked for it); `loadconfig` logs a missing cascade file at *info* (absence is normal — most + `{proctype}`/`{procname}` files won't exist); `loaddir` logs a missing directory at *warn* and + returns `()`. +- **Dedup / idempotency.** `loadfile` tracks loaded paths in `.z.m.loaded` and skips re-loads, so the + cascade is safe to re-run and `init` is safe to call again. +- **`overrideconfig` deviates from TorQ deliberately.** Per-variable problems (undefined name, + non-basic type, value that won't parse to the target type) are logged and skipped — a single bad + override never aborts the batch and a null is never written into config. +- **Entry point is `loadcascade[dirs;names]`, NOT a zero-arg `load[]`.** The Plan sketches + `di.config.load[]`, but that predates the integration design. Config loading happens once at + startup, so a blank call buys nothing and would force the cascade inputs into hidden `.z.m` state + (temporal coupling with `init`). Explicit args keep di.config a pure, env-free resolver: di.torq + resolves `KDBCONFIG`/`KDBAPPCONFIG` → dirs and `proctype`/`procname` → names and passes them. (Also + `load` is a reserved word, so the Plan's literal name is unusable anyway.) If di.torq ever wants a + blank trigger it's a one-line wrapper on its side — not di.config's concern. +- **Config store is live namespace globals, not an explicit internal store (1a, deliberate).** Settings + files assign globals when executed, so those namespaces *are* the store; `get`/`getmodule` read them. + An explicit store with provenance / cross-source precedence (1b) is deferred until a second config + source (env, k8s) exists — the `get`/`getmodule` contract is storage-independent, so that swap won't + touch consumers. See the `EXTENSION POINT` comment in `config.q`. + +Open gaps / not implemented: + +- **No runtime reload.** TorQ's `reloadf` (force-reload an already-loaded file) is not ported — + `loadfile` always dedups. Adequate for startup; revisit if runtime config reload is needed. +- **`order.txt` is matched against the full directory listing** (as in TorQ), so a non-`.q`/`.k` file + named in `order.txt` would still be prepended and load-attempted. Keep `order.txt` to q/k files. +- **`overrideconfig`'s null-parse guard targets scalar/vector basic types**; a string-typed (`10h`) + config var is an uncommon edge where the per-element null check is loose. +- **Standard logger (`di.log`) does not exist yet.** Callers must supply a hand-rolled binary + `` `info`warn`error `` dict in the interim; the kx.log wiring test wraps kx.log caller-side to prove + real emission. +- **Logger storage follows the skill's single-dict pattern** (`.z.m.log[\`info][…]`), which diverges + from `consistency.md`'s three-flat-var example (`.z.m.loginfo`/`…`). Per the skill this is the + directed pattern, pending `consistency.md` being reconciled once `di.log` ships. + +## Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.config +``` diff --git a/di/config/config.q b/di/config/config.q new file mode 100644 index 00000000..8167cd4f --- /dev/null +++ b/di/config/config.q @@ -0,0 +1,219 @@ +/ configuration loading and cascade resolution for the modular torq world. +/ replaces torq.q's in-process config handling (loadf, loaddir, loadconfig, +/ loadaddconfig, overrideconfig). the logger is an injected dependency wired via +/ init; the module reads no environment variables or process identity itself - +/ the caller (di.torq) supplies the settings directories, name sequence and any +/ command-line overrides. layered lowest to highest priority: +/ loadfile - load one file by path (deduplicated) +/ loadconfig - load one cascade file dir/{name}.q (missing is normal) +/ loaddir - load every .q/.k file in a directory (honours order.txt) +/ loadcascade - resolve the full cascade over dirs x names (name-major) +/ overrideconfig - apply command-line-style overrides after the cascade +/ get / getmodule - query the resolved config store (di.torq partitions per +/ module with getmodule and injects each slice via that module's init) + +raiseerror:{[ctx;msg] + / internal - log an error under ctx then signal it, so failures are observable + / in the log as well as thrown to the caller. + .z.m.log[`error][ctx;msg]; + '"di.config: ",string[ctx],": ",msg; + }; + +init:{[deps] + / wire injectable dependencies - log is required, there is no silent fallback. + / deps: a dict with a `log key; log must already be a binary `info`warn`error + / dict of {[c;m]} loggers (from di.log once it ships, or hand-rolled). no + / adaptation is performed here - a raw monadic kx.log instance must be wrapped + / by the caller before being passed. examples: + / config.init[enlist[`log]!enlist logdep] + / config.init[`log`timer!(logdep;timerdep)] + if[99h<>type deps; + '"di.config: deps must be a dict with `log key"]; + if[not `log in key deps; + '"di.config: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type deps`log; + '"di.config: log value must be a dict; pass `info`warn`error functions"]; + if[not all (`info`warn`error) in key deps`log; + '"di.config: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.log:deps`log; + .z.m.loaded:enlist""; + .z.m.log[`info][`init;"di.config initialised"]; + }; + +loadfile:{[file] + / load a single q config file if it exists, tracking it so it is not re-loaded. + / returns the file path; a missing file is a logged warning, not an error. + if[not 10h=abs type file; + raiseerror[`loadfile;"file must be a string path"]; + ]; + if[file in .z.m.loaded; + .z.m.log[`info][`loadfile;"already loaded ",file]; + :file; + ]; + if[()~key hsym `$file; + .z.m.log[`warn][`loadfile;"config file not found: ",file]; + :file; + ]; + .z.m.log[`info][`loadfile;"loading ",file]; + .[system;enlist"l ",file;{[f;e] raiseerror[`loadfile;"failed to load ",f,": ",e]}[file;]]; + .z.m.loaded,:enlist file; + :file; + }; + +loadconfig:{[dir;name] + / load a single cascade config file dir/{name}.q if it is present. a missing + / file is normal in the cascade (not every proctype/procname has one), so its + / absence is logged at info and skipped, NOT warned. present files load via + / loadfile, so they are tracked and de-duplicated. returns the file path. + if[not 10h=abs type dir; + raiseerror[`loadconfig;"dir must be a string path"]; + ]; + if[not -11h=type name; + raiseerror[`loadconfig;"name must be a symbol"]; + ]; + file:dir,"/",string[name],".q"; + if[()~key hsym `$file; + .z.m.log[`info][`loadconfig;"no config file (skipping): ",file]; + :file; + ]; + :loadfile file; + }; + +loaddir:{[dir] + / load every .q and .k file in a directory, honouring an optional order.txt. + / files listed in order.txt load first, in that order; the rest follow in the + / order key returns them. returns the ordered list of file paths processed; + / a missing directory is a logged warning returning (). each load is delegated + / to loadfile, so files already loaded are skipped. + if[not 10h=abs type dir; + raiseerror[`loaddir;"dir must be a string path"]; + ]; + if[()~files:key hsym `$dir; + .z.m.log[`warn][`loaddir;"directory not found: ",dir]; + :(); + ]; + haveorder:`order.txt in files; + if[haveorder; + .z.m.log[`info][`loaddir;"found order.txt in ",dir]; + ]; + order:$[haveorder;(`$read0 hsym `$dir,"/order.txt") inter files;`symbol$()]; + files:files where any files like/:("*.q";"*.k"); + files:order,files except order; + paths:(dir,"/"),/:string files; + .z.m.log[`info][`loaddir;"loading ",(string count paths)," file(s) from ",dir]; + loadfile each paths; + :paths; + }; + +loadcascade:{[dirs;names] + / resolve a configuration cascade name-major: for each config name (least to + / most specific) load it from every directory in turn. files load in sequence, + / so a more specific name always wins over a less specific one regardless of + / directory, and within a single name the later (higher-priority) directory + / overrides the earlier. dirs is a directory path or list of paths (strings), + / ordered lowest->highest priority; names is a config name or list of names + / (symbols), ordered least->most specific. missing files are skipped (by + / loadconfig). returns the flat list of constructed paths, in load order. + / typical caller (di.torq resolves the dirs and process identity): + / config.loadcascade[(kdbconfig;appconfig);`default,proctype,procname] + dirs:$[10h=type dirs;enlist dirs;dirs]; + names:(),names; + if[not 0h=type dirs; + raiseerror[`loadcascade;"dirs must be a string path or a list of string paths"]; + ]; + if[not all 10h=type each dirs; + raiseerror[`loadcascade;"dirs must be a string path or a list of string paths"]; + ]; + if[not 11h=abs type names; + raiseerror[`loadcascade;"names must be a symbol or a symbol list"]; + ]; + .z.m.log[`info][`loadcascade;"resolving cascade: ",(string count dirs)," dir(s) x ",(string count names)," name(s)"]; + :raze {[ds;nm] loadconfig[;nm] each ds}[dirs;] each names; + }; + +applyoverride:{[name;raw] + / internal - parse raw command-line value(s) into name's current type and set + / it. raw is a string or a list of strings. returns 1b if applied, 0b if the + / variable is not a basic type or a value failed to parse (null). the variable + / must already exist - its current type drives the parse. + t:type value name; + if[not (abs t) within (1;-1+count .Q.t); + .z.m.log[`error][`overrideconfig;"cannot override ",(string name),": not a basic type"]; + :0b; + ]; + raw:$[10h=type raw;enlist raw;raw]; + vals:(upper .Q.t abs t)$'raw; + if[t<0;vals:first vals]; + if[any null vals; + .z.m.log[`error][`overrideconfig;"cannot override ",(string name),": value did not parse"]; + :0b; + ]; + .z.m.log[`info][`overrideconfig;"setting ",(string name)," to ",-3!vals]; + set[name;vals]; + :1b; + }; + +overrideconfig:{[params] + / apply command-line-style overrides to already-defined variables, parsing each + / value into the variable's existing type. params is a dict keyed by variable + / name (symbol) with string (or list-of-string) values. only defined variables + / can be overridden (their type drives the parse); undefined names are logged + / and skipped. returns the list of variables actually overridden. call after + / loadcascade to let the command line win over file config. + if[99h<>type params; + raiseerror[`overrideconfig;"params must be a dict keyed by variable name"]; + ]; + vars:key params; + if[0 Date: Fri, 10 Jul 2026 12:26:26 +0100 Subject: [PATCH 07/14] Complete v1, trimmed to only essential API, new logging added, finalised di.logging and di.torq required for full deployment --- di/config/config.md | 117 ++++++++++++++++++++++++++++++-------------- di/config/config.q | 86 +++++++++----------------------- di/config/init.q | 2 +- di/config/test.csv | 72 ++++++--------------------- 4 files changed, 118 insertions(+), 159 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index d7419d37..d619583f 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -1,27 +1,28 @@ # di.config Configuration loading and cascade resolution for the modular TorQ world. This -replaces TorQ's in-process config handling (`torq.q`: `loadf`, `loaddir`, -`loadconfig`, `loadaddconfig`, `overrideconfig`), where `di.torq` will later -distribute the resolved config to each module via that module's `init`. +replaces TorQ's in-process config handling (`torq.q`: `loadf`, `loadconfig`, +`loadaddconfig`, `overrideconfig`), where `di.torq` will later distribute the +resolved config to each module via that module's `init`. ## Import and init `init` requires a `log` dependency — there is no silent fallback. Pass an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` loggers (context -symbol, message string). The module performs **no** adaptation, so a raw monadic -[`kx.log`](https://github.com/KxSystems/logging) instance must be wrapped by the -caller first (this is `di.log`'s job once it ships): +symbol, message string). The module performs **no** adaptation of the logger. + +The standard logger is **`di.log`**, which exports binary `info`/`warn`/`error` +functions directly — build the dict from its exports: ```q config:use`di.config -/ option 1: di.log (standard, once it ships) - build the binary dict from its exports +/ option 1: di.log — the standard logger (exports binary info/warn/error) logger:use`di.log logdep:`info`warn`error!(logger.info;logger.warn;logger.error) config.init[enlist[`log]!enlist logdep] -/ option 2: hand-rolled binary {[c;m]} logger (works today) +/ option 2: any hand-rolled binary {[c;m]} logger mylog:`info`warn`error!( {[c;m] -1 string[c],": INFO ",m;}; {[c;m] -1 string[c],": WARN ",m;}; @@ -29,8 +30,9 @@ mylog:`info`warn`error!( config.init[enlist[`log]!enlist mylog] ``` -To wrap a raw `kx.log` instance yourself in the interim, fold the context into the -message per call: `` `info`warn`error!({[c;m]inst[`info][string[c],": ",m]};…) ``. +A logger that is *not* already binary `{[c;m]}` (e.g. a raw monadic +[`kx.log`](https://github.com/KxSystems/logging) instance) must be wrapped by the +caller first — di.config does no wrapping: `` `info`warn`error!({[c;m]inst[`info][string[c],": ",m]};…) ``. `init` must be called before any other function — there is no default logger. @@ -39,19 +41,13 @@ message per call: `` `info`warn`error!({[c;m]inst[`info][string[c],": ",m]};…) | Function | Signature | Description | |---|---|---| | `init` | `init[deps]` | Wire the injected logger. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | -| `loadfile` | `loadfile[file]` | Load a single q config file (a string path) if it exists, tracking it so it is not re-loaded. Returns the path. A missing file is a logged warning, not an error; a failed load is signalled. | -| `loadconfig` | `loadconfig[dir;name]` | Load one cascade config file `dir/{name}.q` (`dir` a string path, `name` a symbol) if present. Returns the constructed path. A missing file is **normal** in the cascade, so its absence is logged at *info* and skipped (not warned). Present files load via `loadfile`. | -| `loaddir` | `loaddir[dir]` | Load every `.q`/`.k` file in a directory (a string path), honouring an optional `order.txt`. Files listed in `order.txt` load first, in that order; the rest follow. Returns the ordered list of file paths processed. A missing directory is a logged warning returning `()`. Delegates each load to `loadfile`, so already-loaded files are skipped. | -| `loadcascade` | `loadcascade[dirs;names]` | Resolve a config cascade: for each settings directory (in order) load each named file `dir/{name}.q` (in name order). `dirs` is a string path or list of paths; `names` is a symbol or symbol list. Later names and later directories override earlier ones (load order). Missing files are skipped. Returns the flat list of constructed paths in cascade order. | +| `loadcascade` | `loadcascade[dirs;names]` | Resolve the config cascade **name-major**: for each name (least→most specific) load `dir/{name}.q` from each directory in turn. `dirs` is a string path or list of paths (low→high priority); `names` is a symbol or symbol list (least→most specific). A more specific name wins over the directory layer; within a name the later directory overrides. Missing files are skipped (logged at *info*); already-loaded files are de-duplicated. Returns the flat list of constructed paths, in load order. | | `overrideconfig` | `overrideconfig[params]` | Apply command-line-style overrides after the cascade. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | | `get` | `get[namespace;key;default]` | Query the resolved config store: return the value of `.{namespace}.{key}`, or `default` if unset. `namespace` and `key` are **bare** symbols (no leading dot), e.g. `get[\`rdb;\`subscribeto;\`]`. (`get` is a reserved word, so the implementation function is named `getcfg` and exported under the `get` key.) | | `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. | -`order.txt` is a plain-text file, one filename per line, naming the files that must -load ahead of the rest (e.g. to satisfy load-order dependencies). - -Internal helpers (`raiseerror`, `applyoverride`) and the load-tracking state are deliberately -not exported. +Internal helpers — `loadconfig` (the per-file loader behind `loadcascade`), `raiseerror`, +`applyoverride`, and the load-tracking state — are deliberately not exported. `loadcascade` is the top-level entry point. A caller (e.g. `di.torq`, which resolves the process identity and directory paths) drives it as: @@ -83,13 +79,13 @@ plug in by populating the same namespaces before the store is queried — the `g > **Note:** avoid `-` in config file paths — a source-level backtick symbol literal > containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one -> symbol. Runtime string paths passed to `loadfile` are unaffected. +> symbol. Runtime string paths (built internally by `loadcascade`) are unaffected. ## Injectable dependencies | Injectable | Required keys | Function signature | |---|---|---| -| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string); caller passes an already-conforming binary dict (from `di.log` once it ships, or hand-rolled). No adaptation is done in the module — a raw monadic `kx.log` instance must be wrapped first | +| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string); caller passes an already-conforming binary dict — built from `di.log` (the standard logger, which exports binary `info`/`warn`/`error`) or hand-rolled. No adaptation is done in the module — a non-binary logger (e.g. a raw monadic `kx.log` instance) must be wrapped first | ## Hard dependencies @@ -97,15 +93,14 @@ None — `di.config` is a standalone module. ## Design notes & open gaps -- **Return values are informational.** `loadfile`/`loadconfig`/`loaddir`/`loadcascade` return the - path(s) they *considered* — including files that were missing or skipped. The real effect is the - side-effect load; use the return to see what was attempted, not what definitely loaded. -- **Missing-file log level is deliberate.** `loadfile` logs a missing explicit path at *warn* (you - asked for it); `loadconfig` logs a missing cascade file at *info* (absence is normal — most - `{proctype}`/`{procname}` files won't exist); `loaddir` logs a missing directory at *warn* and - returns `()`. -- **Dedup / idempotency.** `loadfile` tracks loaded paths in `.z.m.loaded` and skips re-loads, so the - cascade is safe to re-run and `init` is safe to call again. +- **Return values are informational.** `loadcascade` returns the path(s) it *considered* — including + files that were missing or skipped. The real effect is the side-effect load; use the return to see + what was attempted, not what definitely loaded. +- **Missing cascade files are skipped, not errored.** The internal `loadconfig` logs a missing + `dir/{name}.q` at *info* and skips it — absence is normal in the cascade (most `{proctype}`/ + `{procname}` files won't exist). +- **Dedup / idempotency.** `loadconfig` tracks loaded paths in `.z.m.loaded` and skips re-loads, so + the cascade is safe to re-run and `init` is safe to call again. - **`overrideconfig` deviates from TorQ deliberately.** Per-variable problems (undefined name, non-basic type, value that won't parse to the target type) are logged and skipped — a single bad override never aborts the batch and a null is never written into config. @@ -125,17 +120,18 @@ None — `di.config` is a standalone module. Open gaps / not implemented: - **No runtime reload.** TorQ's `reloadf` (force-reload an already-loaded file) is not ported — - `loadfile` always dedups. Adequate for startup; revisit if runtime config reload is needed. -- **`order.txt` is matched against the full directory listing** (as in TorQ), so a non-`.q`/`.k` file - named in `order.txt` would still be prepended and load-attempted. Keep `order.txt` to q/k files. + `loadconfig` always dedups. Adequate for startup; revisit if runtime config reload is needed. - **`overrideconfig`'s null-parse guard targets scalar/vector basic types**; a string-typed (`10h`) config var is an uncommon edge where the per-element null check is loose. -- **Standard logger (`di.log`) does not exist yet.** Callers must supply a hand-rolled binary - `` `info`warn`error `` dict in the interim; the kx.log wiring test wraps kx.log caller-side to prove - real emission. +- **No committed `di.log` integration test yet.** `di.log` (the standard logger) currently lives on + the `feature-logging` branch and isn't merged, so a committed `use\`di.log` test would fail on this + branch. di.config's contract match with `di.log` is **verified manually** — the real `di.log` was + wired end-to-end with no code change (`init`/`loadcascade`/`loadconfig` all emit through it). The + committed kx.log-wrapper test proves the same binary-`{[c;m]}`-dict contract `di.log` satisfies. Add + a `di.log` integration test once `di.log` merges. - **Logger storage follows the skill's single-dict pattern** (`.z.m.log[\`info][…]`), which diverges - from `consistency.md`'s three-flat-var example (`.z.m.loginfo`/`…`). Per the skill this is the - directed pattern, pending `consistency.md` being reconciled once `di.log` ships. + from `consistency.md`'s three-flat-var example (`.z.m.loginfo`/`…`). `di.log` is now built and the + single-dict pattern is the directed one; `consistency.md` still needs reconciling to match. ## Tests @@ -143,3 +139,48 @@ Open gaps / not implemented: k4unit:use`di.k4unit k4unit.moduletest`di.config ``` + +## Follow-ups (deferred until other modules land) + +di.config is complete for v1 in isolation. The items below are intentionally deferred; +each is keyed to the module or milestone that unblocks it. Do these **when the trigger +lands**, not before. + +### When `di.log` merges to main + +- **Add a committed `di.log` integration test.** Today `di.log` lives on the + `feature-logging` branch (unmerged), so a committed `` use`di.log `` test would fail + here — the contract match is verified manually and stood in for by the kx.log-wrapper + emission test. Once `di.log` resolves on `QPATH`, add a test that builds + `` `info`warn`error!(logger.info;logger.warn;logger.error) `` from `di.log` and drives + `loadcascade` through it. +- **Re-verify only if `di.log`'s contract changes.** It isn't final/approved yet. No + di.config code change is expected — di.config needs only binary `info`/`warn`/`error`, + which `di.log` exports — but if those signatures change, re-run the integration drive. + +### When `di.torq` is built + +- **Prove the end-to-end flow.** `getmodule` is unit-tested here, but "partition per + module and inject via `init`" only runs for real once `di.torq` exists. Add a + multi-module integration test (under `di.torq`/`di.inttest`, not here) exercising: + di.torq resolves `KDBCONFIG`/`KDBAPPCONFIG` → `dirs` and `proctype`/`procname` → + `names`, calls `loadcascade`, then `overrideconfig` with the command-line params, then + `getmodule` per module → each module's `init`. +- **Hold the env-free boundary.** di.config reads no env vars or process identity — + di.torq owns that resolution and passes `dirs`/`names` explicitly. Do **not** add a + zero-arg `load[]` to di.config; if di.torq wants one it wraps `loadcascade` in a line + on its own side. + +### When `di.depcheck` is built (versioning rollout) + +- **Add a `version` export and `deps.q`.** No module ships these yet and `di.depcheck` + (which consumes them) doesn't exist. When it lands, add `version:"…"` to the export and + a `deps.q` (empty/minimal — di.config has no hard deps) as part of the coordinated + repo-wide rollout, not a di.config-only change. + +### When a second config source (env vars, k8s config maps) is needed + +- **Implement the 1b explicit store.** Replace the live-namespace-globals store with an + internal store carrying provenance / cross-source precedence, behind the unchanged + `get`/`getmodule` contract (so consumers don't change). See the `EXTENSION POINT` + comment in `config.q`. diff --git a/di/config/config.q b/di/config/config.q index 8167cd4f..ec93c443 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -1,16 +1,14 @@ / configuration loading and cascade resolution for the modular torq world. -/ replaces torq.q's in-process config handling (loadf, loaddir, loadconfig, -/ loadaddconfig, overrideconfig). the logger is an injected dependency wired via -/ init; the module reads no environment variables or process identity itself - -/ the caller (di.torq) supplies the settings directories, name sequence and any -/ command-line overrides. layered lowest to highest priority: -/ loadfile - load one file by path (deduplicated) -/ loadconfig - load one cascade file dir/{name}.q (missing is normal) -/ loaddir - load every .q/.k file in a directory (honours order.txt) -/ loadcascade - resolve the full cascade over dirs x names (name-major) +/ replaces torq.q's in-process config handling (loadf, loadconfig, loadaddconfig, +/ overrideconfig). the logger is an injected dependency wired via init; the module +/ reads no environment variables or process identity itself - the caller (di.torq) +/ supplies the settings directories, name sequence and any command-line overrides. +/ public api: +/ loadcascade - resolve the full cascade over dirs x names (name-major) / overrideconfig - apply command-line-style overrides after the cascade / get / getmodule - query the resolved config store (di.torq partitions per / module with getmodule and injects each slice via that module's init) +/ internal helper loadconfig handles per-file loading behind loadcascade. raiseerror:{[ctx;msg] / internal - log an error under ctx then signal it, so failures are observable @@ -22,9 +20,10 @@ raiseerror:{[ctx;msg] init:{[deps] / wire injectable dependencies - log is required, there is no silent fallback. / deps: a dict with a `log key; log must already be a binary `info`warn`error - / dict of {[c;m]} loggers (from di.log once it ships, or hand-rolled). no - / adaptation is performed here - a raw monadic kx.log instance must be wrapped - / by the caller before being passed. examples: + / dict of {[c;m]} loggers - build it from di.log (the standard logger, which + / exports binary info/warn/error) or hand-roll one. no adaptation is performed + / here, so a non-conforming logger (e.g. a raw monadic kx.log instance) must be + / wrapped by the caller first. examples: / config.init[enlist[`log]!enlist logdep] / config.init[`log`timer!(logdep;timerdep)] if[99h<>type deps; @@ -40,31 +39,11 @@ init:{[deps] .z.m.log[`info][`init;"di.config initialised"]; }; -loadfile:{[file] - / load a single q config file if it exists, tracking it so it is not re-loaded. - / returns the file path; a missing file is a logged warning, not an error. - if[not 10h=abs type file; - raiseerror[`loadfile;"file must be a string path"]; - ]; - if[file in .z.m.loaded; - .z.m.log[`info][`loadfile;"already loaded ",file]; - :file; - ]; - if[()~key hsym `$file; - .z.m.log[`warn][`loadfile;"config file not found: ",file]; - :file; - ]; - .z.m.log[`info][`loadfile;"loading ",file]; - .[system;enlist"l ",file;{[f;e] raiseerror[`loadfile;"failed to load ",f,": ",e]}[file;]]; - .z.m.loaded,:enlist file; - :file; - }; - loadconfig:{[dir;name] - / load a single cascade config file dir/{name}.q if it is present. a missing - / file is normal in the cascade (not every proctype/procname has one), so its - / absence is logged at info and skipped, NOT warned. present files load via - / loadfile, so they are tracked and de-duplicated. returns the file path. + / internal - load one cascade config file dir/{name}.q if present, tracking it + / so it is not re-loaded. a missing file is normal in the cascade (not every + / proctype/procname has one), so its absence is logged at info and skipped, not + / warned. returns the file path. if[not 10h=abs type dir; raiseerror[`loadconfig;"dir must be a string path"]; ]; @@ -72,37 +51,18 @@ loadconfig:{[dir;name] raiseerror[`loadconfig;"name must be a symbol"]; ]; file:dir,"/",string[name],".q"; + if[file in .z.m.loaded; + .z.m.log[`info][`loadconfig;"already loaded ",file]; + :file; + ]; if[()~key hsym `$file; .z.m.log[`info][`loadconfig;"no config file (skipping): ",file]; :file; ]; - :loadfile file; - }; - -loaddir:{[dir] - / load every .q and .k file in a directory, honouring an optional order.txt. - / files listed in order.txt load first, in that order; the rest follow in the - / order key returns them. returns the ordered list of file paths processed; - / a missing directory is a logged warning returning (). each load is delegated - / to loadfile, so files already loaded are skipped. - if[not 10h=abs type dir; - raiseerror[`loaddir;"dir must be a string path"]; - ]; - if[()~files:key hsym `$dir; - .z.m.log[`warn][`loaddir;"directory not found: ",dir]; - :(); - ]; - haveorder:`order.txt in files; - if[haveorder; - .z.m.log[`info][`loaddir;"found order.txt in ",dir]; - ]; - order:$[haveorder;(`$read0 hsym `$dir,"/order.txt") inter files;`symbol$()]; - files:files where any files like/:("*.q";"*.k"); - files:order,files except order; - paths:(dir,"/"),/:string files; - .z.m.log[`info][`loaddir;"loading ",(string count paths)," file(s) from ",dir]; - loadfile each paths; - :paths; + .z.m.log[`info][`loadconfig;"loading ",file]; + .[system;enlist"l ",file;{[f;e] raiseerror[`loadconfig;"failed to load ",f,": ",e]}[file;]]; + .z.m.loaded,:enlist file; + :file; }; loadcascade:{[dirs;names] diff --git a/di/config/init.q b/di/config/init.q index b0e5237c..ae1d533f 100644 --- a/di/config/init.q +++ b/di/config/init.q @@ -2,4 +2,4 @@ / note: `get` is a q reserved word, so it cannot be a top-level name; the query / function is defined as `getcfg` in config.q and exported under the `get` key. \l ::config.q -export:([init;loadfile;loadconfig;loaddir;loadcascade;overrideconfig;getmodule]),(enlist`get)!enlist getcfg +export:([init;loadcascade;overrideconfig;getmodule]),(enlist`get)!enlist getcfg diff --git a/di/config/test.csv b/di/config/test.csv index 01daaa51..6433060a 100644 --- a/di/config/test.csv +++ b/di/config/test.csv @@ -3,77 +3,35 @@ before,0,0,q,cfg:use`di.config,1,1,load the module before,0,0,q,"logtab:([]lvl:`symbol$();ctx:`symbol$();msg:())",1,1,capturing log table before,0,0,q,"mocklog:`info`warn`error!({[c;m]`logtab upsert(`info;c;m)};{[c;m]`logtab upsert(`warn;c;m)};{[c;m]`logtab upsert(`error;c;m)})",1,1,define a capturing binary logger before,0,0,q,cfg.init[enlist[`log]!enlist mocklog],1,1,init with the required log dependency -before,0,0,q,"(`:/tmp/diconfigtest.q) 0: enlist ""diconfigtestvar:42;diconfigcount:1+@[value;`diconfigcount;0];""",1,1,write a temp config file that sets a var and counts loads -before,0,0,q,diconfigorder:`$(),1,1,tracker for the order files load in -before,0,0,q,"system""mkdir -p /tmp/diconfigloaddir""",1,1,make a directory with an order.txt -before,0,0,q,"(`:/tmp/diconfigloaddir/a.q) 0: enlist ""diconfiga:1;diconfigorder,:`a;""",1,1,dir file a.q -before,0,0,q,"(`:/tmp/diconfigloaddir/b.q) 0: enlist ""diconfigb:1;diconfigorder,:`b;""",1,1,dir file b.q -before,0,0,q,"(`:/tmp/diconfigloaddir/d.k) 0: enlist """"",1,1,dir file d.k (proves .k pickup) -before,0,0,q,"(`:/tmp/diconfigloaddir/notes.txt) 0: enlist ""not a q file""",1,1,non-q file that must be ignored -before,0,0,q,"(`:/tmp/diconfigloaddir/order.txt) 0: (""b.q"";""a.q"")",1,1,order.txt loads b before a -before,0,0,q,"system""mkdir -p /tmp/diconfignoorder""",1,1,make a directory without an order.txt -before,0,0,q,"(`:/tmp/diconfignoorder/p.q) 0: enlist ""diconfigp:1;""",1,1,noorder file p.q -before,0,0,q,"(`:/tmp/diconfignoorder/q.q) 0: enlist ""diconfigq:1;""",1,1,noorder file q.q -before,0,0,q,"system""mkdir -p /tmp/diconfigcascade""",1,1,make a cascade settings directory -before,0,0,q,"(`:/tmp/diconfigcascade/default.q) 0: enlist ""diconfigdefault:99;""",1,1,cascade default.q before,0,0,q,diconfigtrace:`symbol$(),1,1,tracer for cascade load order before,0,0,q,"system""mkdir -p /tmp/diconfigc1""",1,1,cascade directory one before,0,0,q,"system""mkdir -p /tmp/diconfigc2""",1,1,cascade directory two before,0,0,q,"(`:/tmp/diconfigc1/default.q) 0: enlist ""diconfigcx:1;diconfigtrace,:`c1default;""",1,1,c1 default.q before,0,0,q,"(`:/tmp/diconfigc1/rdb.q) 0: enlist ""diconfigcx:10;diconfigtrace,:`c1rdb;""",1,1,c1 rdb.q -before,0,0,q,"(`:/tmp/diconfigc2/default.q) 0: enlist ""diconfigcx:2;diconfigtrace,:`c2default;""",1,1,c2 default.q (no c2 rdb.q) +before,0,0,q,"(`:/tmp/diconfigc2/default.q) 0: enlist ""diconfigcx:2;diconfigtrace,:`c2default;""",1,1,c2 default.q (no c2 rdb.q - a missing cascade file) before,0,0,q,"system""mkdir -p /tmp/diconfigstore""",1,1,cascade dir for the config-store tests before,0,0,q,"(`:/tmp/diconfigstore/default.q) 0: enlist "".diconfigq.subscribeto:`trade`quote;.diconfigq.rows:100;""",1,1,settings file populating the .diconfigq namespace before,0,0,q,.diconfigtest.enabled:0b,1,1,override target - boolean before,0,0,q,.diconfigtest.rows:100,1,1,override target - long before,0,0,q,.diconfigtest.syms:`a`b,1,1,override target - symbol list before,0,0,q,.diconfigtest.tab:([]a:`long$()),1,1,override target - non-basic (table) -before,0,0,q,"(`:/tmp/diconfigkxreal.q) 0: enlist ""diconfigkxvar:1;""",1,1,file to load through a real kx.log +before,0,0,q,"system""mkdir -p /tmp/diconfigkxdir""",1,1,dir for the kx.log emission test +before,0,0,q,"(`:/tmp/diconfigkxdir/default.q) 0: enlist ""diconfigkxvar:1;""",1,1,file loaded through the real logger comment,,,,,,,init - dependency validation fail,0,0,q,cfg.init[(::)],1,1,init rejects a non-dictionary deps fail,0,0,q,cfg.init[enlist[`foo]!enlist 1],1,1,init rejects deps without a log key fail,0,0,q,cfg.init[enlist[`log]!enlist 42],1,1,init rejects a non-dict log value fail,0,0,q,cfg.init[enlist[`log]!enlist ((enlist`info)!enlist {[c;m]})],1,1,init rejects a log dict missing warn/error -comment,,,,,,,loadfile - missing file is a warning -true,0,0,q,"""/tmp/diconfignope.q""~cfg.loadfile[""/tmp/diconfignope.q""]",1,1,missing file returns its path -true,0,0,q,`warn in exec lvl from logtab,1,1,missing file logged a warning -comment,,,,,,,loadfile - real load and dedup -true,0,0,q,"""/tmp/diconfigtest.q""~cfg.loadfile[""/tmp/diconfigtest.q""]",1,1,real file returns its path -true,0,0,q,42=diconfigtestvar,1,1,loaded file executed and set its global -true,0,0,q,1=diconfigcount,1,1,file was loaded exactly once -run,0,0,q,"cfg.loadfile[""/tmp/diconfigtest.q""]",1,1,re-load the same file -true,0,0,q,1=diconfigcount,1,1,re-load is skipped (counter unchanged) -comment,,,,,,,loadfile - input validation -fail,0,0,q,cfg.loadfile[42],1,1,loadfile rejects a non-string path -comment,,,,,,,loaddir - ordered load honouring order.txt -run,0,0,q,"cfg.loaddir[""/tmp/diconfigloaddir""]",1,1,load the directory (first time) -true,0,0,q,"3=count cfg.loaddir[""/tmp/diconfigloaddir""]",1,1,three q/k files loaded (notes.txt ignored) -true,0,0,q,`b`a~diconfigorder,1,1,order.txt loaded b before a -true,0,0,q,"any (cfg.loaddir[""/tmp/diconfigloaddir""]) like ""*d.k""",1,1,.k files are picked up -true,0,0,q,"not any (cfg.loaddir[""/tmp/diconfigloaddir""]) like ""*.txt""",1,1,non-q/k files are excluded -true,0,0,q,2=diconfiga+diconfigb,1,1,directory files actually executed -comment,,,,,,,loaddir - directory without order.txt -true,0,0,q,"2=count cfg.loaddir[""/tmp/diconfignoorder""]",1,1,loads all q files when no order.txt present -comment,,,,,,,loaddir - missing directory and validation -true,0,0,q,"()~cfg.loaddir[""/tmp/diconfignodir""]",1,1,missing directory returns an empty list -true,0,0,q,"any (exec msg from logtab) like ""*directory not found*""",1,1,missing directory logged a warning -fail,0,0,q,cfg.loaddir[42],1,1,loaddir rejects a non-string path -comment,,,,,,,loadconfig - named cascade file -true,0,0,q,"""/tmp/diconfigcascade/default.q""~cfg.loadconfig[""/tmp/diconfigcascade"";`default]",1,1,named file returns its constructed path -true,0,0,q,99=diconfigdefault,1,1,named file was executed -true,0,0,q,"""/tmp/diconfigcascade/nosuch.q""~cfg.loadconfig[""/tmp/diconfigcascade"";`nosuch]",1,1,missing named file returns its path without erroring -true,0,0,q,"any (exec msg from logtab where lvl=`info) like ""*no config file*""",1,1,missing cascade file logged at info not warn -fail,0,0,q,cfg.loadconfig[42;`default],1,1,loadconfig rejects a non-string dir -fail,0,0,q,"cfg.loadconfig[""/tmp/diconfigcascade"";""default""]",1,1,loadconfig rejects a non-symbol name -comment,,,,,,,load - full cascade over directories and names +comment,,,,,,,loadcascade - full cascade over directories and names run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,run the cascade true,0,0,q,"4=count cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,one path per dir-name pair (skipped files included) true,0,0,q,10=diconfigcx,1,1,most-specific name wins over dir layer (c1/rdb loads last) -true,0,0,q,`c1default`c2default`c1rdb~diconfigtrace,1,1,cascade order is name-major (each name across all dirs) -true,0,0,q,"any (cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]) like ""*diconfigc2/rdb.q""",1,1,returned paths include skipped files +true,0,0,q,`c1default`c2default`c1rdb~diconfigtrace,1,1,name-major order; and re-runs don't re-execute (dedup) +true,0,0,q,"any (cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]) like ""*diconfigc2/rdb.q""",1,1,returned paths include skipped (missing) files +true,0,0,q,"any (exec msg from logtab where lvl=`info) like ""*no config file*""",1,1,a missing cascade file is skipped at info (internal loadconfig) true,0,0,q,"1=count cfg.loadcascade[""/tmp/diconfigc1"";`default]",1,1,accepts a single string dir and single symbol name -fail,0,0,q,cfg.loadcascade[42;`default],1,1,load rejects non-string dirs -fail,0,0,q,"cfg.loadcascade[""/tmp/diconfigc1"";""default""]",1,1,load rejects non-symbol names +fail,0,0,q,cfg.loadcascade[42;`default],1,1,loadcascade rejects non-string dirs +fail,0,0,q,"cfg.loadcascade[""/tmp/diconfigc1"";""default""]",1,1,loadcascade rejects non-symbol names comment,,,,,,,overrideconfig - parse and apply command-line overrides run,0,0,q,"cfg.overrideconfig[`.diconfigtest.enabled`.diconfigtest.rows!(enlist""1"";enlist""500"")]",1,1,override a bool and a long true,0,0,q,1b~.diconfigtest.enabled,1,1,bool parsed and set from string @@ -101,15 +59,15 @@ run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigq.rows]!enlist enlist""500""]",1, true,0,0,q,500~cfg.get[`diconfigq;`rows;0],1,1,get reflects overrideconfig changes fail,0,0,q,cfg.get[42;`k;0],1,1,get rejects a non-symbol namespace fail,0,0,q,cfg.getmodule[42],1,1,getmodule rejects a non-symbol namespace -comment,,,,,,,kx.log - real emission via a caller-side binary wrapper (must run last; re-inits cfg) +comment,,,,,,,logger emission - real logger via a caller-side binary wrapper (must run last; re-inits cfg) run,0,0,q,kxlogger:use`kx.log,1,1,load the kx.log module run,0,0,q,kxinst:kxlogger.createLog[],1,1,create a real kx.log instance run,0,0,q,kxh:hopen `:/tmp/diconfigkxsink.txt,1,1,open a file sink to capture output run,0,0,q,kxinst.add[kxh;`info],1,1,route info-level output to the file sink -run,0,0,q,"kxbinary:`info`warn`error!({[c;m]kxinst[`info][string[c],"": "",m]};{[c;m]kxinst[`warn][string[c],"": "",m]};{[c;m]kxinst[`error][string[c],"": "",m]})",1,1,wrap monadic kx.log into a binary dict (what di.log will do) +run,0,0,q,"kxbinary:`info`warn`error!({[c;m]kxinst[`info][string[c],"": "",m]};{[c;m]kxinst[`warn][string[c],"": "",m]};{[c;m]kxinst[`error][string[c],"": "",m]})",1,1,wrap the monadic logger into the binary {[c;m]} contract run,0,0,q,cfg.init[enlist[`log]!enlist kxbinary],1,1,re-init di.config with the conforming binary logger -run,0,0,q,"cfg.loadfile[""/tmp/diconfigkxreal.q""]",1,1,drive a real info-level log through kx.log +run,0,0,q,"cfg.loadcascade[""/tmp/diconfigkxdir"";`default]",1,1,drive a real info-level log through the wrapped logger run,0,0,q,hclose kxh,1,1,flush and close the file sink -true,0,0,q,1=diconfigkxvar,1,1,the real load executed the file -true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*loadfile: loading*""",1,1,kx.log emitted the wrapped context and message -true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*diconfigkxreal.q*""",1,1,emitted message includes the file path +true,0,0,q,1=diconfigkxvar,1,1,the cascade executed the file +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*loadconfig: loading*""",1,1,the internal loader emitted through the real logger +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*diconfigkxdir/default.q*""",1,1,emitted message includes the file path From b5f2e030d212ae30b0c91639bc086da3cc1a2fc3 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Mon, 13 Jul 2026 18:25:29 +0100 Subject: [PATCH 08/14] updating tests and removing not needed function after review call --- di/config/config.md | 61 +++++++++++++++++++++++++++++++-------------- di/config/config.q | 52 ++++++++++++++------------------------ di/config/init.q | 4 +-- di/config/test.csv | 8 ++---- 4 files changed, 64 insertions(+), 61 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index d619583f..bca13ab4 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -42,9 +42,8 @@ caller first — di.config does no wrapping: `` `info`warn`error!({[c;m]inst[`in |---|---|---| | `init` | `init[deps]` | Wire the injected logger. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | | `loadcascade` | `loadcascade[dirs;names]` | Resolve the config cascade **name-major**: for each name (least→most specific) load `dir/{name}.q` from each directory in turn. `dirs` is a string path or list of paths (low→high priority); `names` is a symbol or symbol list (least→most specific). A more specific name wins over the directory layer; within a name the later directory overrides. Missing files are skipped (logged at *info*); already-loaded files are de-duplicated. Returns the flat list of constructed paths, in load order. | -| `overrideconfig` | `overrideconfig[params]` | Apply command-line-style overrides after the cascade. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | -| `get` | `get[namespace;key;default]` | Query the resolved config store: return the value of `.{namespace}.{key}`, or `default` if unset. `namespace` and `key` are **bare** symbols (no leading dot), e.g. `get[\`rdb;\`subscribeto;\`]`. (`get` is a reserved word, so the implementation function is named `getcfg` and exported under the `get` key.) | -| `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. | +| `overrideconfig` | `overrideconfig[params]` | Apply the **command-line-argument layer** on top of the file cascade — di.torq parses the process command line (`.Q.opt .z.x`) and calls this at startup, so launch-time flags (e.g. `-loglevel info`, `-tablelist trade quote`) win over the settings files. This is the automatic top layer of the cascade, **not** an interactive/by-hand step. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | +| `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. Callers read a single setting by indexing the returned dict inline (e.g. `getmodule[\`rdb]\`subscribeto`). | Internal helpers — `loadconfig` (the per-file loader behind `loadcascade`), `raiseerror`, `applyoverride`, and the load-tracking state — are deliberately not exported. @@ -58,12 +57,14 @@ config.overrideconfig[`.myproc.enabled`.myproc.rows!(enlist"1";enlist"5000")] / then partition per module and hand each slice to that module's init: rdbcfg:config.getmodule[`rdb] / -> `subscribeto`hdbtypes!(...) etc. rdb.init[rdbcfg;`log`timer!(logdep;timerdep)] -/ or read a single value with a fallback: -config.get[`rdb;`subscribeto;`trade`quote] +/ read a single setting by indexing the slice inline (configs are booted with defaults, +/ so the key is always present — no fallback needed): +rdbcfg`subscribeto ``` -`overrideconfig` runs after `loadcascade` so command-line values win over file config. -di.config is deliberately generic — it does not read environment variables or process +`overrideconfig` runs after `loadcascade` so command-line values win over file config — it is the +top (command-line) layer of the cascade, applied automatically by di.torq at startup, not a manual +step. di.config is deliberately generic — it does not read environment variables or process identity itself; the caller (di.torq) supplies the settings directories, the ordered name sequence, and the override params, then uses `getmodule` to partition the resolved store per module. @@ -72,10 +73,10 @@ store per module. `loadcascade` loads settings `.q` files that assign into root namespaces (`.rdb.subscribeto:...`); those resolved namespaces **are** the config store, and -`get`/`getmodule` query them (so anything `overrideconfig` changes is reflected too). +`getmodule` queries them (so anything `overrideconfig` changes is reflected too). Additional config sources (env vars, k8s config maps) are out of scope for v1 but would -plug in by populating the same namespaces before the store is queried — the `get`/ -`getmodule` contract stays unchanged. See the EXTENSION POINT comment in `config.q`. +plug in by populating the same namespaces before the store is queried — the `getmodule` +contract stays unchanged. See the EXTENSION POINT comment in `config.q`. > **Note:** avoid `-` in config file paths — a source-level backtick symbol literal > containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one @@ -101,9 +102,14 @@ None — `di.config` is a standalone module. `{procname}` files won't exist). - **Dedup / idempotency.** `loadconfig` tracks loaded paths in `.z.m.loaded` and skips re-loads, so the cascade is safe to re-run and `init` is safe to call again. -- **`overrideconfig` deviates from TorQ deliberately.** Per-variable problems (undefined name, - non-basic type, value that won't parse to the target type) are logged and skipped — a single bad - override never aborts the batch and a null is never written into config. +- **`overrideconfig` is the command-line layer, applied by di.torq — not a manual step.** It mirrors + TorQ's `overrideconfig`/`override[]` (`torq.q`), which TorQ runs at startup off `.Q.opt .z.x` so + launch-time flags beat the settings files. In the modular world di.torq owns that command-line parse + and calls this after `loadcascade`; di.config just does the type-aware apply (which is why it stays + env-free and never reads `.z.x` itself). It *deviates from TorQ deliberately* on error handling: + per-variable problems (undefined name, non-basic type, value that won't parse to the target type) + are logged and skipped — a single bad override never aborts the batch and a null is never written + into config. - **Entry point is `loadcascade[dirs;names]`, NOT a zero-arg `load[]`.** The Plan sketches `di.config.load[]`, but that predates the integration design. Config loading happens once at startup, so a blank call buys nothing and would force the cascade inputs into hidden `.z.m` state @@ -112,10 +118,17 @@ None — `di.config` is a standalone module. `load` is a reserved word, so the Plan's literal name is unusable anyway.) If di.torq ever wants a blank trigger it's a one-line wrapper on its side — not di.config's concern. - **Config store is live namespace globals, not an explicit internal store (1a, deliberate).** Settings - files assign globals when executed, so those namespaces *are* the store; `get`/`getmodule` read them. + files assign globals when executed, so those namespaces *are* the store; `getmodule` reads them. An explicit store with provenance / cross-source precedence (1b) is deferred until a second config - source (env, k8s) exists — the `get`/`getmodule` contract is storage-independent, so that swap won't + source (env, k8s) exists — the `getmodule` contract is storage-independent, so that swap won't touch consumers. See the `EXTENSION POINT` comment in `config.q`. +- **No single-key getter — `getmodule` is the only query.** An earlier draft exported a + `get[namespace;key;default]` convenience. It was removed: `getmodule` already returns the whole + namespace as a dict, and the intended flow fetches that slice once (di.torq → each module's `init`) + and indexes it inline (`rdbcfg\`subscribeto`), so a per-key getter added a second query path for no + gain. The `default` fallback was dropped with it — every config is booted with a default, so a + requested key is always present and a fallback arg is dead weight. A missing key is now a caller + bug (surfaced as a q null on the inline index), not something the config layer silently papers over. Open gaps / not implemented: @@ -129,9 +142,10 @@ Open gaps / not implemented: wired end-to-end with no code change (`init`/`loadcascade`/`loadconfig` all emit through it). The committed kx.log-wrapper test proves the same binary-`{[c;m]}`-dict contract `di.log` satisfies. Add a `di.log` integration test once `di.log` merges. -- **Logger storage follows the skill's single-dict pattern** (`.z.m.log[\`info][…]`), which diverges - from `consistency.md`'s three-flat-var example (`.z.m.loginfo`/`…`). `di.log` is now built and the - single-dict pattern is the directed one; `consistency.md` still needs reconciling to match. +- **Logging uses the three-flat-var convention** — `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, + called as `.z.m.loginfo[\`ctx;"msg"]` — matching `consistency.md` and `di.compression`. The injected + `log` value is still the same `` `info`warn`error `` dict; `init` just fans it out into the three + module-local vars. ## Tests @@ -146,6 +160,15 @@ di.config is complete for v1 in isolation. The items below are intentionally def each is keyed to the module or milestone that unblocks it. Do these **when the trigger lands**, not before. +### ⚠️ Logging call convention — NOT set in stone (ask before changing) + +di.config uses the **three-flat-var** convention (`.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`), +matching `consistency.md` and `di.compression`. But the project hasn't finalised this versus the +**single-dict** form (`.z.m.log[\`info][…]`) that an earlier skill revision directed. **Before +changing di.config's logging — or wiring logging in another module — stop and ask the user which +convention is authoritative.** If the project settles on single-dict, switching di.config back is +mechanical (storage + call sites only; the injected input contract is identical either way). + ### When `di.log` merges to main - **Add a committed `di.log` integration test.** Today `di.log` lives on the @@ -182,5 +205,5 @@ lands**, not before. - **Implement the 1b explicit store.** Replace the live-namespace-globals store with an internal store carrying provenance / cross-source precedence, behind the unchanged - `get`/`getmodule` contract (so consumers don't change). See the `EXTENSION POINT` + `getmodule` contract (so consumers don't change). See the `EXTENSION POINT` comment in `config.q`. diff --git a/di/config/config.q b/di/config/config.q index ec93c443..872a9495 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -6,14 +6,14 @@ / public api: / loadcascade - resolve the full cascade over dirs x names (name-major) / overrideconfig - apply command-line-style overrides after the cascade -/ get / getmodule - query the resolved config store (di.torq partitions per -/ module with getmodule and injects each slice via that module's init) +/ getmodule - query the resolved config store as a per-namespace dict; di.torq +/ partitions config per module with it and injects each slice via that module's init / internal helper loadconfig handles per-file loading behind loadcascade. raiseerror:{[ctx;msg] / internal - log an error under ctx then signal it, so failures are observable / in the log as well as thrown to the caller. - .z.m.log[`error][ctx;msg]; + .z.m.logerr[ctx;msg]; '"di.config: ",string[ctx],": ",msg; }; @@ -34,9 +34,11 @@ init:{[deps] '"di.config: log value must be a dict; pass `info`warn`error functions"]; if[not all (`info`warn`error) in key deps`log; '"di.config: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; - .z.m.log:deps`log; + .z.m.loginfo:deps[`log]`info; + .z.m.logwarn:deps[`log]`warn; + .z.m.logerr:deps[`log]`error; .z.m.loaded:enlist""; - .z.m.log[`info][`init;"di.config initialised"]; + .z.m.loginfo[`init;"di.config initialised"]; }; loadconfig:{[dir;name] @@ -52,14 +54,14 @@ loadconfig:{[dir;name] ]; file:dir,"/",string[name],".q"; if[file in .z.m.loaded; - .z.m.log[`info][`loadconfig;"already loaded ",file]; + .z.m.loginfo[`loadconfig;"already loaded ",file]; :file; ]; if[()~key hsym `$file; - .z.m.log[`info][`loadconfig;"no config file (skipping): ",file]; + .z.m.loginfo[`loadconfig;"no config file (skipping): ",file]; :file; ]; - .z.m.log[`info][`loadconfig;"loading ",file]; + .z.m.loginfo[`loadconfig;"loading ",file]; .[system;enlist"l ",file;{[f;e] raiseerror[`loadconfig;"failed to load ",f,": ",e]}[file;]]; .z.m.loaded,:enlist file; :file; @@ -87,7 +89,7 @@ loadcascade:{[dirs;names] if[not 11h=abs type names; raiseerror[`loadcascade;"names must be a symbol or a symbol list"]; ]; - .z.m.log[`info][`loadcascade;"resolving cascade: ",(string count dirs)," dir(s) x ",(string count names)," name(s)"]; + .z.m.loginfo[`loadcascade;"resolving cascade: ",(string count dirs)," dir(s) x ",(string count names)," name(s)"]; :raze {[ds;nm] loadconfig[;nm] each ds}[dirs;] each names; }; @@ -98,17 +100,17 @@ applyoverride:{[name;raw] / must already exist - its current type drives the parse. t:type value name; if[not (abs t) within (1;-1+count .Q.t); - .z.m.log[`error][`overrideconfig;"cannot override ",(string name),": not a basic type"]; + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": not a basic type"]; :0b; ]; raw:$[10h=type raw;enlist raw;raw]; vals:(upper .Q.t abs t)$'raw; if[t<0;vals:first vals]; if[any null vals; - .z.m.log[`error][`overrideconfig;"cannot override ",(string name),": value did not parse"]; + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; :0b; ]; - .z.m.log[`info][`overrideconfig;"setting ",(string name)," to ",-3!vals]; + .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; set[name;vals]; :1b; }; @@ -132,7 +134,7 @@ overrideconfig:{[params] defined:vars where {@[{value x;1b};x;0b]} each vars; undefined:vars except defined; if[count undefined; - .z.m.log[`warn][`overrideconfig;"skipping undefined variable(s): ",", " sv string undefined]; + .z.m.logwarn[`overrideconfig;"skipping undefined variable(s): ",", " sv string undefined]; ]; applied:{[params;v] applyoverride[v;params v]}[params;] each defined; :defined where applied; @@ -140,31 +142,15 @@ overrideconfig:{[params] / --- queryable config store --- / the store is the set of root namespaces populated by the settings files that -/ loadcascade loads (plus anything overrideconfig changes). getcfg/getmodule -/ query it; di.torq uses getmodule to partition config per module and pass each -/ slice to that module's init. +/ loadcascade loads (plus anything overrideconfig changes). getmodule queries it; +/ di.torq uses getmodule to partition config per module and pass each slice to that +/ module's init. / EXTENSION POINT (out of scope for v1, flagged per the modularisation plan): / additional config sources - environment variables, k8s config maps, external / key-value stores - would plug in by populating the same root namespaces before / the store is queried. keep source reading (loadcascade et al.) separate from / querying (below) so a new source is an additive step, not a change to the -/ get/getmodule contract. - -getcfg:{[ns;k;dflt] - / query the resolved config store: return the value of the loaded config - / variable .{ns}.{k}, or dflt if it is not set. ns and k are bare symbols with - / no leading dot (e.g. getcfg[`rdb;`subscribeto;`]). exported under the `get` - / key. NB: params are ns/k/dflt because key and default are reserved words - - / used as parameter names they throw 'match when the function is called (and - / neither is listed in .Q.res). - if[not -11h=type ns; - raiseerror[`get;"namespace must be a symbol (no leading dot)"]; - ]; - if[not -11h=type k; - raiseerror[`get;"key must be a symbol"]; - ]; - :@[value;`$".",(string ns),".",string k;dflt]; - }; +/ getmodule contract. getmodule:{[namespace] / return all resolved config for a namespace as a bare-keyed value dict, for diff --git a/di/config/init.q b/di/config/init.q index ae1d533f..d5534dc9 100644 --- a/di/config/init.q +++ b/di/config/init.q @@ -1,5 +1,3 @@ / configuration loading and cascade resolution for the modular torq world. -/ note: `get` is a q reserved word, so it cannot be a top-level name; the query -/ function is defined as `getcfg` in config.q and exported under the `get` key. \l ::config.q -export:([init;loadcascade;overrideconfig;getmodule]),(enlist`get)!enlist getcfg +export:([init;loadcascade;overrideconfig;getmodule]) diff --git a/di/config/test.csv b/di/config/test.csv index 6433060a..9655d6e1 100644 --- a/di/config/test.csv +++ b/di/config/test.csv @@ -46,18 +46,14 @@ true,0,0,q,"any (exec msg from logtab) like ""*not a basic type*""",1,1,non-basi true,0,0,q,0=count cfg.overrideconfig[()!()],1,1,empty params is a no-op fail,0,0,q,cfg.overrideconfig[42],1,1,overrideconfig rejects a non-dict fail,0,0,q,"cfg.overrideconfig[enlist[1]!enlist enlist""x""]",1,1,overrideconfig rejects non-symbol keys -comment,,,,,,,get / getmodule - queryable config store +comment,,,,,,,getmodule - queryable config store run,0,0,q,"cfg.loadcascade[""/tmp/diconfigstore"";`default]",1,1,load a settings file into the store -true,0,0,q,`trade`quote~cfg.get[`diconfigq;`subscribeto;`],1,1,get returns a loaded config value -true,0,0,q,42~cfg.get[`diconfigq;`missing;42],1,1,get returns the default for an unset key -true,0,0,q,999~cfg.get[`nosuchns;`x;999],1,1,get returns the default for an unknown namespace true,0,0,q,2=count cfg.getmodule[`diconfigq],1,1,getmodule returns the whole namespace slice true,0,0,q,`trade`quote~cfg.getmodule[`diconfigq]`subscribeto,1,1,getmodule slice carries the values true,0,0,q,100~cfg.getmodule[`diconfigq]`rows,1,1,getmodule slice carries the second value true,0,0,q,0=count cfg.getmodule[`nosuchns],1,1,getmodule returns an empty dict for an unknown namespace run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigq.rows]!enlist enlist""500""]",1,1,override a stored value -true,0,0,q,500~cfg.get[`diconfigq;`rows;0],1,1,get reflects overrideconfig changes -fail,0,0,q,cfg.get[42;`k;0],1,1,get rejects a non-symbol namespace +true,0,0,q,500~cfg.getmodule[`diconfigq]`rows,1,1,getmodule reflects overrideconfig changes fail,0,0,q,cfg.getmodule[42],1,1,getmodule rejects a non-symbol namespace comment,,,,,,,logger emission - real logger via a caller-side binary wrapper (must run last; re-inits cfg) run,0,0,q,kxlogger:use`kx.log,1,1,load the kx.log module From bc5fa3c82548f01055fa37c2acfa0f70fa7f8426 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Mon, 13 Jul 2026 18:32:00 +0100 Subject: [PATCH 09/14] removing accidentally added module --- di/asyncdispatch/asyncdispatch.md | 263 ------------------------ di/asyncdispatch/asyncdispatch.q | 331 ------------------------------ di/asyncdispatch/init.q | 11 - di/asyncdispatch/test.csv | 190 ----------------- 4 files changed, 795 deletions(-) delete mode 100644 di/asyncdispatch/asyncdispatch.md delete mode 100644 di/asyncdispatch/asyncdispatch.q delete mode 100644 di/asyncdispatch/init.q delete mode 100644 di/asyncdispatch/test.csv diff --git a/di/asyncdispatch/asyncdispatch.md b/di/asyncdispatch/asyncdispatch.md deleted file mode 100644 index c11d5c80..00000000 --- a/di/asyncdispatch/asyncdispatch.md +++ /dev/null @@ -1,263 +0,0 @@ -# di.asyncdispatch - -Async scatter-gather query coordinator for kdb-x gateway processes. Queues client queries, dispatches them to available backend processes by servertype, collects per-server results, applies a join function, and replies to the client — with timeout management and correct error propagation if a backend disconnects mid-query. - -Routing (deciding which servertypes satisfy a query) is `di.serverselect`'s responsibility, but **`di.serverselect` is not a dependency** — this module dispatches a resolved servertype list to whatever backends its *server source* reports. That source is pluggable with a default: by default it is asyncdispatch's own registry (populate it with `addserver`), or you point it at `di.serverselect`'s output via `setavailableservers`. Either works standalone; neither is required. - ---- - -## Features - -- Queue and dispatch async client queries to multiple backend process types simultaneously (scatter-gather) -- Collect per-server results and apply a caller-supplied join function once all slots are filled -- Timeout expired queries with configurable per-query timespan via `checktimeout` -- Handle backend disconnects mid-query — errors in-flight queries and queued queries that can no longer be satisfied -- Track connected clients and clean up orphaned queries on client disconnect -- Support synchronous deferred response mode (`-30!`) alongside the default async mode -- Pluggable server source with a default — dispatch against the built-in registry (`addserver`) by default, or point it at `di.serverselect`'s output (or any source) via `setavailableservers`; `di.serverselect` is a composable option, never a dependency -- Accept fully pluggable scheduler (`setgetnextqueryid`), routing (`setavailableservers`), reply formatter (`setformatresponse`), and callback symbols (`setcallbacks`) — swap without touching core dispatch logic -- Detect and normalise `kx.log` instances automatically so callers can pass a logger directly without manual wrapping - ---- - -## Dependencies - -| Dependency | Key | Required | Description | -|---|---|---|---| -| logger | `` `log `` | yes | `info`, `warn`, `error` — each binary `{[c;m]}` where `c` is a symbol context and `m` is a string | - -The `log` dependency must be passed to `init` inside the `deps` dict. The module throws immediately if it is absent or missing any of the three required keys. All three are required since the module calls `info`, `warn`, and `error`. - -> **`di.serverselect` is not a dependency.** asyncdispatch never imports or calls it. The server source is pluggable (`setavailableservers`) and defaults to the built-in registry (`addserver`), so you run standalone out of the box; to dispatch against serverselect's live view instead, inject its `getservers` output as the source (see `setavailableservers`). No registry copying, no dependency. - -A `kx.log` instance can be passed directly — the module normalises monadic functions to the binary `{[c;m]}` contract automatically via `normlog`. Context is embedded in the output as `"context: message"`: - -```q -kxlog:use`kx.log -ad:use`di.asyncdispatch - -// minimal -ad.init[enlist[`log]!enlist kxlog.createLog[]] - -// with config overrides -ad.init[`log`querykeeptime`synccallsallowed!(kxlog.createLog[];0D01:00;1b)] -``` - ---- - -## Initialisation - -`init[deps]` takes a single dictionary combining the `log` dependency with any configuration overrides. - -| Key | Required | Default | Description | -|---|---|---|---| -| `` `log `` | yes | — | Binary log dep — `info`, `warn`, `error` functions each `{[c;m]}` | -| `` `errorprefix `` | no | `"error: "` | String prepended to all error messages sent back to clients | -| `` `querykeeptime `` | no | `0D00:30` | How long `removequeries` retains finished query rows | -| `` `clearinactivetime `` | no | `0D01:00` | How long `removeinactive` retains disconnected server rows | -| `` `synccallsallowed `` | no | `0b` | Whether `execquery[...;1b]` (deferred sync mode) is permitted | - -Housekeeping — `checktimeout`, `removequeries`, `removeinactive`, and `removeclients` — is the caller's responsibility. Wire them into your gateway's timer after `init`. The configured default age parameters are accessible as `querykeeptime` and `clearinactivetime` via module state. - ---- - -## Exported Functions - -### `init[deps]` -Initialise the module. Validates the log dependency and applies config overrides. -```q -ad.init[enlist[`log]!enlist logdep] -``` - -### `addserver[handle;servertype]` -Register a backend connection into the **built-in (default) server source**. `handle`: open int handle. `servertype`: symbol identifying the process type (e.g. `` `rdb ``, `` `hdb ``). Use this when asyncdispatch owns the server list; to source servers from `di.serverselect` instead, leave `addserver` unused and inject via `setavailableservers`. -```q -ad.addserver[hopen`:backend1:5001;`rdb] -``` - -### `removeserverhandle[handle]` -Call from `.z.pc` for **backend** handles. Errors any in-flight or queued queries that depended on this server, marks the server `active:0b`, and triggers `runnextquery`. -```q -.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} -``` - -### `addclientdetails[handle]` -Record client identity on connect. Call from `.z.po`. -```q -.z.po:{ad.addclientdetails[.z.w]} -``` - -### `removeclienthandle[handle]` -On client disconnect, mark their pending queries errored so result slots are not leaked. Call from `.z.pc`. -```q -.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} -``` - -### `addserverresult[qid;data]` -Called when a backend posts back a successful result. Fills the result slot, frees the server, triggers `runnextquery`, and — once all slots for the query are received — applies the join function and replies to the client. -```q -// called by serverexecute on the backend; not typically called directly -``` - -### `addservererror[qid;err]` -Called when a backend posts back an error. Sends the error to the client and finishes the query. -```q -// called by serverexecute on the backend; not typically called directly -``` - -### `execquery[query;servertype;join;postback;timeout;sync]` -Public entry point. Validates sync constraints, enqueues the query, and triggers dispatch. - -| Argument | Type | Description | -|---|---|---| -| `query` | any | Payload passed to `value` on the backend | -| `servertype` | symbol list | One symbol per required backend type, e.g. `` enlist`rdb `` | -| `join` | function | Applied to the list of per-server results once all are received | -| `postback` | list or `()` | `()` for a plain reply; `(function;extra_args...)` to wrap the reply | -| `timeout` | timespan | `0Wn` for no timeout | -| `sync` | boolean | `1b` for deferred sync via `-30!`; `0b` for async | - -```q -ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;0b] -``` - -### `execqueryto[replyto;query;servertype;join;postback;timeout;sync]` -Variant of `execquery` with an explicit reply target. Pass `replyto:0Ni` to invoke the postback locally via `value` instead of IPC-sending to a handle. Pass any valid int handle to route the reply to a specific process regardless of `.z.w`. Used when the caller is in the same process as the gateway (e.g. `di.dataaccess` dispatching shards). - -| Argument | Type | Description | -|---|---|---| -| `replyto` | int | `0Ni` for local in-process invocation; any int handle to IPC-send to a specific target | -| `query` | any | Payload passed to `value` on the backend | -| `servertype` | symbol list | One symbol per required backend type | -| `join` | function | Applied to the list of per-server results once all are received | -| `postback` | list | Required when `replyto` is `0Ni` — `(function;extra_args...)` invoked locally; `()` permitted for non-local targets | -| `timeout` | timespan | `0Wn` for no timeout | -| `sync` | boolean | Must be `0b` when `replyto` is `0Ni`; sync not supported for local invocation | - -```q -// di.dataaccess wiring: shard result delivered locally to da.shardresult -ad.execqueryto[0Ni;shardquery;servertypes;raze;(`da.shardresult;reqid);0Wn;0b] -``` - -### `checktimeout[]` -Scan the queue for queries past their timeout, send a timeout error to each client, and mark them complete. Wire into your gateway's timer — every few seconds is typical. -```q -// in gateway timer -timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] -``` - -### `removequeries[age]` -Purge completed `queryqueue` rows older than `age`. Prevents unbounded growth. -```q -// default age is querykeeptime (0D00:30) -timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] -``` - -### `removeinactive[age]` -Purge `servers` rows for backends that have been disconnected longer than `age`. Prevents unbounded growth. -```q -// default age is clearinactivetime (0D01:00) -timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] -``` - -### `removeclients[age]` -Purge `clients` rows older than `age`. The `clients` table is appended to on every client connect (`addclientdetails`) for audit and is not otherwise pruned, so wire this into the timer to prevent unbounded growth. Choose `age` to match your audit retention requirements. -```q -// retain one day of client audit history -timer.addjob.default[`asyncdispatch.removeclients;{ad.removeclients[1D]};();300i;1] -``` - -### `setformatresponse[f]` -Override the reply formatter applied before a result or error is sent to the client. `f` must be `{[status;sync;result]}`. Note: `formatresponse` is only applied on the remote IPC path — queries dispatched via `execqueryto` with `replyto:0Ni` invoke the postback directly and bypass this formatter. -```q -ad.setformatresponse[{[status;sync;result]result}] -``` - -### `setcallbacks[resfn;errfn]` -Update the callback symbols used by `serverexecute`. Required when the module is mounted under a non-default namespace — point these at wherever `addserverresult` and `addservererror` are visible on the backend processes. -```q -ad.setcallbacks[`.gw.dispatch.addserverresult;`.gw.dispatch.addservererror] -``` - -### `setavailableservers[f]` -Replace the **server source** — the function dispatch uses to find backends. `f` is `{[excludeinuse]}` and must return a table with `handle` and `servertype` columns. This is the seam for running against `di.serverselect` (or any external source) **without a dependency and without copying its registry**: by default the source reads asyncdispatch's own registry (`addserver`), and you override it to read serverselect instead. -```q -// default (built-in registry) — equivalent to not calling this at all -ad.setavailableservers[{[eu] $[eu; select from servers where active, not inuse; select from servers where active]}] - -// compose with di.serverselect (not a dependency — just its output as the source) -srvsel:use`di.serverselect -ad.setavailableservers[{[eu] select handle, servertype from srvsel.getservers[`servertype;`;()!()]}] -``` -Result routing does **not** depend on the source — a returning handle is matched to its servertype from the query's own dispatch record — so an injected source needs no registration in asyncdispatch. Note `inuse` throttling applies only to the built-in registry; an external source is expected to do its own idle/selection (e.g. serverselect's `roundrobin`), and the `excludeinuse` flag is a hint such a source may ignore. - -### `setgetnextqueryid[f]` -Inject a custom scheduling strategy. `f` must be niladic and return a 0- or 1-row table with the `queryqueue` schema. -```q -// priority queue example - highest-priority query first -ad.setgetnextqueryid[{1 sublist `priority xdesc 0!select from .z.m.queryqueue where null returntime}] -``` - ---- - -## Usage Example - -```q -kxlog:use`kx.log -timer:use`di.timer -timer.init[()!()] - -ad:use`di.asyncdispatch -ad.init[enlist[`log]!enlist kxlog.createLog[]] - -// point backends' callbacks at this module's mount point on this process -ad.setcallbacks[`ad.addserverresult;`ad.addservererror] - -// register backend connections as they connect -ad.addserver[hopen`:backend1:5001;`rdb] -ad.addserver[hopen`:backend2:5002;`hdb] - -// wire client and server connection/disconnection handlers -.z.po:{ad.addclientdetails[.z.w]} -.z.pc:{ad.removeserverhandle[.z.w];ad.removeclienthandle[.z.w]} - -// wire housekeeping into the gateway timer -timer.addjob.default[`asyncdispatch.checktimeout;{ad.checktimeout[]};();5i;1] -timer.addjob.default[`asyncdispatch.removequeries;{ad.removequeries[0D00:30]};();300i;1] -timer.addjob.default[`asyncdispatch.removeinactive;{ad.removeinactive[0D01:00]};();300i;1] -timer.addjob.default[`asyncdispatch.removeclients;{ad.removeclients[1D]};();300i;1] - -// a client calls this asynchronously: -// execquery dispatches to rdb and hdb in parallel, razes results, replies to client -ad.execquery[("select count i by sym from trade";"select count i by sym from trade");`rdb`hdb;raze;();0Wn;0b] - -// synchronous deferred mode (requires synccallsallowed:1b in deps) -ad.execquery["select count i by sym from trade";enlist`rdb;raze;();0Wn;1b] -``` - ---- - -## Running Tests - -```q -k4unit:use`di.k4unit -k4unit.moduletest`di.asyncdispatch -``` - -116 tests. Requires a `q` binary in `PATH` — the test suite starts two real backend processes on dynamically selected free ports and exercises the full dispatch lifecycle over live IPC connections. No TorQ installation or special libraries required. Covers: server registry, FIFO scheduling, full IPC round-trip with join and reply, backend error path via real IPC callback (`addservererror`), join-failure path (throwing join function flagged as error), postback wrapping (`tosend` tuple construction), multi-servertype scatter-gather with two backends dispatched in parallel and results joined, checktimeout, removequeries, removeinactive, removeclients audit-row purge, removeserverhandle with orphaned query cleanup, client tracking, in-flight server release on client disconnect, all pluggable hook setters, and local in-process reply path via `execqueryto` (success, backend error, timeout, and non-null explicit replyto variants). - ---- - -## Notes - -- Housekeeping (`checktimeout`, `removequeries`, `removeinactive`, `removeclients`) is the caller's responsibility — the gateway process already has a timer running and is better placed to decide intervals. Wire all four after `init`; see the usage example above -- When `checktimeout` fires for a query that is already in-flight (dispatched to a backend but not yet answered), the client receives the timeout error and the query is marked done, but the backend that received the dispatch remains `inuse:1b` until it eventually replies. A slow-but-alive backend holds its dispatch slot until `addserverresult` or `addservererror` fires; a permanently hung backend holds it until `removeserverhandle` fires on disconnect. Size backend pools and timeouts with this in mind — a run of timeouts against a stuck backend will not free its slot on timeout alone -- `.z.M.` is used for in-place mutation of tables (`upsert`, `insert`, `update from`, `delete from`) and `.z.m.:value` for whole-variable reassignment — the same convention used by `di.cache` -- Module globals referenced inside q-sql expressions (WHERE conditions, UPDATE SET values) must use the `.z.m.varname` form since q-sql evaluates column expressions in the calling context rather than the module namespace -- `servertype` in `queryqueue` and `addquery` is a list of servertype symbols — one per required backend type. Pass `` enlist`rdb `` for single-server queries, `` `rdb`hdb `` for scatter-gather across two types -- `setcallbacks` must be called before any queries are dispatched if the module is mounted under a non-default path — `serverexecute` reads `resultcallback` and `errorcallback` by bare name on the backend process and posts back to whatever symbols they resolve to -- This module opens and accepts no connections itself — `addserver` and the `.z.po`/`.z.pc` wiring are the consumer's responsibility, keeping the module dependency-free and testable in-process -- All three log keys (`info`, `warn`, `error`) are required — the module calls `info` on server/client connect and init, `warn` on disconnect and timeout, and `error` on backend error and join failure -- `execqueryto` with `replyto:0Ni` invokes the postback locally via `value` — the postback head symbol must be mount-qualified (e.g. `` `da.shardresult ``) since bare exported names are not globals. This is the same resolution mechanism used by `setcallbacks` -- Local queries store `clienth:0Ni` and are not matched by `removeclienthandle` — the in-process caller owns disconnect cleanup for its own requests -- `setformatresponse` overrides only apply to the remote IPC path — local invocation via `execqueryto[0Ni;...]` calls the postback directly without applying `formatresponse`. The default `formatresponse` is a pass-through for async, so this is transparent by default; consumers that override it should not combine that with local invocation diff --git a/di/asyncdispatch/asyncdispatch.q b/di/asyncdispatch/asyncdispatch.q deleted file mode 100644 index c2f812b5..00000000 --- a/di/asyncdispatch/asyncdispatch.q +++ /dev/null @@ -1,331 +0,0 @@ -// di.asyncdispatch - async scatter-gather query coordinator -// queues queries, dispatches to available backends, collects results per server, -// applies a join function, and replies to the client -// routing (which servers satisfy a query) is di.serverselect's responsibility -// the log dependency is required - init errors immediately if absent or malformed -// log functions are binary {[c;m]} where c is a symbol context and m is a string - -// ============================================================ -// module state and defaults -// ============================================================ - -// error prefix prepended to all error strings returned to clients -errorprefix:"error: "; - -// how long completed queries are kept in queryqueue before being purged -querykeeptime:0D00:30; - -// how long disconnected servers are kept in the servers table before being removed -clearinactivetime:0D01:00; - -// whether synchronous calls via -30! are permitted -synccallsallowed:0b; - -// injectable clock - replaced in tests to control time without sleeping -cp:{.z.p}; - -// symbols backend servers call back via - stored as symbols so names survive IPC serialisation -resultcallback:`addserverresult; -errorcallback:`addservererror; - -// reply formatting - sync errors must be signalled with ' so the client receives a trapped error -formatresponse:{[status;sync;result]$[not[status]and sync;'result;result]}; - -// scheduling strategy - pick oldest FIFO-eligible query by default -getnextqueryid:{ - avail:exec distinct servertype from availableservers 1b; - // 0! is required - select from a keyed table stays keyed in kdb-x, and runnextquery needs queryid via first - runnable:0!select from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, {all x in y}[;avail] each servertype; - 1 sublist select from runnable where time=min time - }; - -// server routing strategy - active and idle by default -availableservers:{[excludeinuse] - $[excludeinuse; - select from servers where active, not inuse; - select from servers where active] - }; - -// ============================================================ -// module tables -// ============================================================ - -// registered backend servers -servers:([handle:`u#`int$()] servertype:`symbol$(); inuse:`boolean$(); active:`boolean$(); disconnecttime:`timestamp$()); - -// pending and in-flight client queries -queryqueue:([queryid:`u#`long$()] time:`timestamp$(); clienth:`int$(); query:(); servertype:(); join:(); postback:(); timeout:`timespan$(); returntime:`timestamp$(); error:`boolean$(); sync:`boolean$(); local:`boolean$()); - -// connected client tracking -clients:([] time:`timestamp$(); clienth:`int$(); user:`symbol$(); ip:`int$(); host:`symbol$()); - -// per-query result accumulator: queryid -> (clienth; servertype!(handle;result;done)) -results:()!(); - -// auto-incrementing query id counter -queryid:0; - -// ============================================================ -// internal helpers -// ============================================================ - -normlog:{[logdict] - // detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) - // kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message - // plain {[c;m]} log dicts (info`warn`error only) pass through unchanged - $[all `getlvl`sinks`fmts in key logdict; - `info`warn`error!( - {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; - {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; - {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); - logdict] - }; - -sendclientreply:{[qid;result;status] - // deliver result or error to the client, handling sync vs async send and postback wrapping - // local path invokes value tosend directly - formatresponse is not applied (it is a pass-through - // for async by default; consumers that override setformatresponse should not use local invocation) - qd:queryqueue[qid]; - if[qd`error;:()]; - tosend:$[()~qd`postback;result;qd[`postback],enlist[qd`query],enlist result]; - $[qd`sync; - @[-30!;(qd`clienth;not status;$[status;formatresponse[1b;1b;result];result]);{}]; - $[qd`local; - @[value;tosend;{.z.m.log[`error][`asyncdispatch;"local postback failed: ",x]}]; - @[neg qd`clienth;formatresponse[status;0b;tosend];()]]]; - }; - -finishquery:{[qid;err] - // remove query from the live results accumulator and stamp its completion time - .z.m.results:(qid,())_results; - update error:err,returntime:.z.m.cp[] from .z.M.queryqueue where queryid in qid; - }; - -serverexecute:{[qid;query] - // runs on the backend - traps errors so a crash posts an error reply rather than dropping the result - res:@[{(0b;value x)};query;{(1b;"server ",(string .z.h),":",(string system"p"),": ",x)}]; - @[neg .z.w;$[res 0;(errorcallback;qid;res 1);(resultcallback;qid;res 1)]; - {@[neg .z.w;(errorcallback;x;"failed to return result: ",y);()]}[qid]]; - }; - -sendquerytoserver:{[qid;query;handles] - // fan the query out to all required handles and mark them in-use atomically - (neg handles,:())@\:(serverexecute;qid;query); - update inuse:1b from .z.M.servers where handle in handles; - }; - -runnextquery:{[] - // pick the next dispatchable query and fan out to one idle server per required servertype - // called after any state change that may unblock work - if[0=count torun:getnextqueryid[];:()]; - torun:first torun; - avail:exec first handle by servertype from availableservers 1b; - types:torun`servertype; - handles:avail types; - qid:torun`queryid; - slots:types!count[types]#enlist(0Ni;(::);0b); - slots[types;0]:handles; - // indexed assignment on bare results propagates through to .z.m in kdb-x (empirically verified) - results[qid]:(torun`clienth;slots); - sendquerytoserver[qid;torun`query;handles]; - }; - -addqueryto:{[query;servertype;join;postback;timeout;sync;replyto;local] - // enqueue a query with an explicit reply target - used by execqueryto for local in-process routing - .z.M.queryqueue upsert (queryid;.z.m.cp[];replyto;query;servertype;join;{$[11h=type x;enlist x;x]}postback;timeout;0Np;0b;sync;local); - .z.m.queryid:queryid+1; - }; - -addquery:{[query;servertype;join;postback;timeout;sync] - // enqueue a query without dispatching - caller must call runnextquery[] to trigger dispatch - addqueryto[query;servertype;join;postback;timeout;sync;.z.w;0b]; - }; - -removequeries:{[age] - // prevent queryqueue growing unboundedly - purge completed queries older than age - delete from .z.M.queryqueue where not null returntime, .z.m.cp[]>returntime+age; - }; - -removeinactive:{[age] - // prune stale disconnected-server rows to stop the servers table growing forever - delete from .z.M.servers where not active, .z.m.cp[]>disconnecttime+age; - }; - -removeclients:{[age] - // prune stale client audit rows to stop the clients table growing forever - // clients are recorded on every connect by addclientdetails and never removed otherwise - delete from .z.M.clients where .z.m.cp[]>time+age; - }; - -checktimeout:{[] - // periodic scan to error queries that have waited beyond their timeout - qids:exec queryid from .z.m.queryqueue where not timeout=0Wn, null returntime, .z.m.cp[]>time+timeout; - if[count qids; - .z.m.log[`warn][`asyncdispatch;"queries timed out: ",", " sv string qids]; - sendclientreply[;errorprefix,"query timed out";0b] each qids; - finishquery[qids;1b]]; - }; - -// ============================================================ -// public api -// ============================================================ - -setcp:{[f] - // replace the clock function - used in tests to control time without sleeping - .z.m.cp:f; - }; - -setformatresponse:{[f] - // override reply formatting - e.g. to wrap results in a standard envelope - .z.m.formatresponse:f; - }; - -setcallbacks:{[resfn;errfn] - // update callback symbols when module is mounted under a non-default namespace - .z.m.resultcallback:resfn; - .z.m.errorcallback:errfn; - }; - -setavailableservers:{[f] - // swap in a custom routing strategy without forking core dispatch - .z.m.availableservers:f; - }; - -setgetnextqueryid:{[f] - // inject a priority or custom scheduling strategy - .z.m.getnextqueryid:f; - }; - -addserver:{[h;st] - // register a backend handle and servertype so it becomes eligible for dispatch. - // this is the default (built-in) server source; to dispatch against di.serverselect's view instead, - // inject it via setavailableservers - no registration and no di.serverselect dependency required - .z.m.log[`info][`asyncdispatch;"server registered: ",string[st]," handle ",string h]; - .z.M.servers upsert (h;st;0b;1b;0Np); - }; - -removeserverhandle:{[serverh] - // on backend disconnect, error in-flight queries using that handle and queued queries - // that can no longer be satisfied - if[null st:first exec servertype from .z.m.servers where handle=serverh;:()]; - err:errorprefix,"backend ",string[st]," server disconnected"; - .z.m.log[`warn][`asyncdispatch;"backend disconnected: ",string st]; - // in-flight: queries where this handle was assigned to a slot - qids:where {[h;qid]h in value[.z.m.results[qid;1]][;0]}[serverh] each key .z.m.results; - sendclientreply[;err," during query";0b] each qids; - finishquery[qids;1b]; - // queued: queries that can no longer be satisfied by remaining active servers - activetypes:exec distinct servertype from .z.m.servers where active, handle<>serverh; - qids2:exec queryid from .z.m.queryqueue where null returntime, not queryid in key .z.m.results, - not {all x in y}[;activetypes] each servertype; - sendclientreply[;err,", query cannot be satisfied";0b] each qids2; - finishquery[qids2;1b]; - update active:0b,disconnecttime:.z.m.cp[] from .z.M.servers where handle=serverh; - runnextquery[]; - }; - -addclientdetails:{[h] - // record client identity on connect for audit and orphan-query cleanup on disconnect - .z.m.log[`info][`asyncdispatch;"client connected: handle ",string h]; - .z.M.clients insert (.z.m.cp[];h;.z.u;.z.a;.z.h); - }; - -removeclienthandle:{[h] - // on client disconnect, mark their pending queries errored so result slots are not leaked - // free any servers in-flight for this client before removing result slots, then re-dispatch - // local queries store clienth:0Ni and are not matched here - the in-process caller owns cleanup for its own requests - .z.m.log[`info][`asyncdispatch;"client disconnected: handle ",string h]; - inflightqids:(exec queryid from .z.m.queryqueue where clienth=h, null returntime) inter key .z.m.results; - if[count inflightqids; - inflighthandles:distinct raze {value[.z.m.results[x;1]][;0]} each inflightqids; - update inuse:0b from .z.M.servers where handle in inflighthandles]; - update error:1b,returntime:.z.m.cp[] from .z.M.queryqueue where clienth=h, null returntime; - .z.m.results:(exec queryid from .z.m.queryqueue where clienth=h)_results; - runnextquery[]; - }; - -addserverresult:{[qid;data] - // fill one result slot - once all slots for a query are filled, run the join and reply - // bare indexed assignment on results propagates through to .z.m in kdb-x (empirically verified) - if[not qid in key results;:()]; - slots:results[qid;1]; - // map the responding handle to its servertype from THIS query's own dispatch record, not a server - // registry - so any server source (built-in or an injected di.serverselect view) works unchanged - st:first where .z.w=slots[;0]; - slots[st]:(.z.w;data;1b); - results[qid]:(results[qid;0];slots); - update inuse:0b from .z.M.servers where handle in .z.w; - runnextquery[]; - if[not qid in key results;:()]; - vals:value results[qid;1]; - if[not all vals[;2];:()]; - qd:queryqueue[qid]; - res:@[{(0b;x y)}[qd`join];vals[;1];{(1b;.z.m.errorprefix,"join failed: ",x)}]; - if[res 0;.z.m.log[`error][`asyncdispatch;"join failed for query ",string qid,": ",last res]]; - sendclientreply[qid;last res;not res 0]; - finishquery[qid;res 0]; - }; - -addservererror:{[qid;err] - // short-circuit a query on backend failure - free the server and notify the client - .z.m.log[`error][`asyncdispatch;"backend error for query ",string[qid],": ",err]; - sendclientreply[qid;errorprefix,err;0b]; - update inuse:0b from .z.M.servers where handle in .z.w; - runnextquery[]; - finishquery[qid;1b]; - }; - -execquery:{[query;servertype;join;postback;timeout;sync] - // public entry point - validate sync constraints then enqueue and kick dispatch - if[sync; - if[not ()~postback;.z.m.log[`warn][`asyncdispatch;"execquery: postback ignored for sync call"]]; - if[not synccallsallowed;'"syncexec: synchronous calls are not allowed"]; - if[not @[{-30!x;1b};(::);0b];'"syncexec: deferred response not supported on this connection"]; - .[{[q;s;j;t]addquery[q;s;j;();t;1b];runnextquery[]};(query;servertype;join;timeout);{-30!(.z.w;1b;x)}]; - :()]; - addquery[query;servertype;join;postback;timeout;0b]; - runnextquery[]; - }; - -execqueryto:{[replyto;query;servertype;join;postback;timeout;sync] - // in-process variant of execquery - replyto is 0Ni for local invocation via value - // postback must be non-empty when replyto is 0Ni - there is no handle to send a bare result to - // sync is not supported for local invocation - // mount-qualified postback symbols required for local invocation e.g. `da.shardresult not `shardresult - if[0Ni~replyto; - if[()~postback;'"di.asyncdispatch: local invocation requires a non-empty postback"]; - if[sync;'"di.asyncdispatch: local invocation does not support sync mode"]]; - local:0Ni~replyto; - addqueryto[query;servertype;join;postback;timeout;sync;replyto;local]; - runnextquery[]; - }; - -init:{[deps] - // initialise the asyncdispatch module - validate deps and apply config overrides - // deps: dict containing `log (required) plus optional config keys: - // `log - required: `info`warn`error!({[c;m]};{[c;m]};{[c;m]}) binary loggers - // `errorprefix - optional: string prefix for client error messages. default: "error: " - // `querykeeptime - optional: timespan to keep completed queries. default: 0D00:30 - // `clearinactivetime - optional: timespan to keep disconnected servers. default: 0D01:00 - // `synccallsallowed - optional: boolean, whether sync calls are permitted. default: 0b - // housekeeping (checktimeout; removequeries; removeinactive; removeclients) is the caller's - // responsibility - wire them into your timer after init; the exported defaults for age params are - // querykeeptime and clearinactivetime - // examples: - // ad.init[enlist[`log]!enlist logdep] - // ad.init[`log`querykeeptime!(logdep;0D01:00)] - if[99h<>type deps; - '"di.asyncdispatch: deps must be a dict with `log key"]; - if[not `log in key deps; - '"di.asyncdispatch: log dependency is required; pass `info`warn`error!(infofn;warnfn;errfn) keyed on `log"]; - if[99h<>type deps`log; - '"di.asyncdispatch: log value must be a dict; pass `info`warn`error functions"]; - if[not all `info`warn`error in key deps`log; - '"di.asyncdispatch: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; - .z.m.log:normlog deps`log; - if[`errorprefix in key deps; .z.m.errorprefix:deps`errorprefix]; - if[`querykeeptime in key deps; .z.m.querykeeptime:deps`querykeeptime]; - if[`clearinactivetime in key deps; .z.m.clearinactivetime:deps`clearinactivetime]; - if[`synccallsallowed in key deps; .z.m.synccallsallowed:deps`synccallsallowed]; - .z.m.log[`info][`asyncdispatch;"di.asyncdispatch initialised"]; - }; diff --git a/di/asyncdispatch/init.q b/di/asyncdispatch/init.q deleted file mode 100644 index 21200d8a..00000000 --- a/di/asyncdispatch/init.q +++ /dev/null @@ -1,11 +0,0 @@ -// asyncdispatch module - async scatter-gather query coordinator for multi-backend kdb+ gateway processes -\l ::asyncdispatch.q - -export:([ - setformatresponse;setcallbacks;setavailableservers;setgetnextqueryid; - addserver;removeserverhandle; - addclientdetails;removeclienthandle; - addserverresult;addservererror; - execquery;execqueryto; - checktimeout;removequeries;removeinactive;removeclients; - init]) diff --git a/di/asyncdispatch/test.csv b/di/asyncdispatch/test.csv deleted file mode 100644 index 41ba99a8..00000000 --- a/di/asyncdispatch/test.csv +++ /dev/null @@ -1,190 +0,0 @@ -action,ms,bytes,lang,code,repeat,minver,comment -before,0,0,q,p:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19500+til 100],1,,find first free port in range 19500-19599 -before,0,0,q,p2:first {x where {10h=type @[hopen;`$raze("::";string x);{x}]} each x}[19600+til 100],1,,find second free port in range 19600-19699 -before,0,0,q,system raze("q -p ";string p;" -q &"),1,,start first backend process on free port -before,0,0,q,system raze("q -p ";string p2;" -q &"),1,,start second backend process on free port -before,0,0,q,system"sleep 1",1,,wait for both backends to initialise -before,0,0,q,ad:use`di.asyncdispatch,1,,load module -before,0,0,q,srv:{.m.di.0asyncdispatch.servers},1,,live servers table -before,0,0,q,qq:{.m.di.0asyncdispatch.queryqueue},1,,live queryqueue table -before,0,0,q,cl:{.m.di.0asyncdispatch.clients},1,,live clients table -before,0,0,q,res:{.m.di.0asyncdispatch.results},1,,live results dict -before,0,0,q,mocklog:`info`warn`error!3#enlist{[c;m]},1,,mock log dep - binary {[c;m]} no-op -before,0,0,q,ad.init[enlist[`log]!enlist mocklog],1,,initialise with mock logger -before,0,0,q,bh:hopen`$raze("::";string p),1,,open handle to backend -before,0,0,q,bh".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure backend result callback to dispatcher -before,0,0,q,bh".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure backend error callback to dispatcher -before,0,0,q,ad.addserver[bh;`mock],1,,register first backend as mock server -before,0,0,q,bh2:hopen`$raze("::";string p2),1,,open handle to second backend -before,0,0,q,bh2".m.di.0asyncdispatch.resultcallback:`.m.di.0asyncdispatch.addserverresult",1,,configure second backend result callback -before,0,0,q,bh2".m.di.0asyncdispatch.errorcallback:`.m.di.0asyncdispatch.addservererror",1,,configure second backend error callback -before,0,0,q,ad.addserver[bh2;`scatter],1,,register second backend as scatter servertype - -/ Test 0: server registry -true,0,0,q,bh in exec handle from srv[] where active,1,,first backend server is active -true,0,0,q,bh2 in exec handle from srv[] where active,1,,second backend server is active -true,0,0,q,bh in exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,first backend is available for dispatch -true,0,0,q,bh2 in exec handle from .m.di.0asyncdispatch.availableservers 1b,1,,second backend is available for dispatch - -/ Test 1: queueing and FIFO scheduling -run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`mock;raze;();0Wn;0b],1,,queue a query for the mock servertype -true,0,0,q,1~count select from qq[] where query~\:"2+2",1,,query was added to the queue -true,0,0,q,1~count .m.di.0asyncdispatch.getnextqueryid[],1,,query is runnable - idle mock server available -true,0,0,q,(enlist`mock)~first exec servertype from .m.di.0asyncdispatch.getnextqueryid[],1,,getnextqueryid identifies the available servertype - -/ Test 2: full dispatch -> result -> join -> reply via real IPC round-trip -run,0,0,q,qid1:exec first queryid from qq[] where query~\:"2+2",1,,capture the queued query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch query to real backend process -run,0,0,q,bh"",1,,sync round-trip: flushes dispatch and waits for backend result callback before returning -true,0,0,q,0b~first exec error from qq[] where queryid=qid1,1,,query completed without error -true,0,0,q,not null first exec returntime from qq[] where queryid=qid1,1,,returntime is stamped -true,0,0,q,not qid1 in key res[],1,,result accumulator cleaned up after join - -/ Test 3: checktimeout -run,0,0,q,.m.di.0asyncdispatch.addquery["3+3";enlist`mock;raze;();0D00:00:00.000000001;0b],1,,queue with a near-zero timeout -run,0,0,q,qid2:exec first queryid from qq[] where query~\:"3+3",1,,capture the queued query id -run,0,0,q,ad.checktimeout[],1,,timeout sweep should catch the expired query -true,0,0,q,1b~first exec error from qq[] where queryid=qid2,1,,timed-out query is flagged as error -true,0,0,q,not null first exec returntime from qq[] where queryid=qid2,1,,timed-out query has returntime stamped - -/ Test 4: removequeries purges completed queries -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward the clock by 2 days -run,0,0,q,ad.removequeries .m.di.0asyncdispatch.querykeeptime,1,,purge queries older than querykeeptime -true,0,0,q,0~count select from qq[] where queryid in (qid1;qid2),1,,both completed queries purged -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore the real clock - -/ Test 5: removeserverhandle errors queued queries when the only server disconnects -run,0,0,q,ad.addserver[1i;`mock2],1,,register a second mock server (fake handle - query is errored before dispatch) -run,0,0,q,.m.di.0asyncdispatch.addquery["4+4";enlist`mock2;raze;();0Wn;0b],1,,queue a query that can only run on mock2 -run,0,0,q,qid3:exec first queryid from qq[] where query~\:"4+4",1,,capture the queued query id -run,0,0,q,ad.removeserverhandle[1i],1,,disconnect the only server able to run the query -true,0,0,q,1b~first exec error from qq[] where queryid=qid3,1,,query errored as no server can satisfy it -true,0,0,q,0b~first exec active from srv[] where handle=1i,1,,disconnected server marked inactive - -/ Test 6: removeinactive purges inactive server records -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward clock past clearinactivetime -run,0,0,q,ad.removeinactive .m.di.0asyncdispatch.clearinactivetime,1,,purge inactive servers -true,0,0,q,0~count select from srv[] where handle=1i,1,,inactive server record removed -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore real clock - -/ Test 7: client tracking -run,0,0,q,ad.addclientdetails[2i],1,,record a connected client -true,0,0,q,1~count select from cl[] where clienth=2i,1,,client connection is tracked -run,0,0,q,ad.removeclienthandle[2i],1,,disconnect the client -true,0,0,q,0~count select from qq[] where (clienth=2i)&null returntime,1,,client unfinished queries cleared - -/ Test 8: pluggable hooks -run,0,0,q,ad.setformatresponse[{[status;sync;result]result}],1,,override formatresponse to pass result through -true,0,0,q,5~.m.di.0asyncdispatch.formatresponse[1b;0b;5],1,,overridden formatresponse is used -run,0,0,q,ad.setgetnextqueryid[{()}],1,,override scheduler to never pick a query -true,0,0,q,0~count .m.di.0asyncdispatch.getnextqueryid[],1,,overridden scheduler is used -run,0,0,q,ad.setcallbacks[`myresult;`myerror],1,,override callback symbols -true,0,0,q,(`myresult;`myerror)~(.m.di.0asyncdispatch.resultcallback;.m.di.0asyncdispatch.errorcallback),1,,callback symbols updated - -/ Test 9: removeclienthandle frees in-flight servers on client disconnect -run,0,0,q,ad.addserver[99i;`inflight],1,,register fake server for in-flight test -run,0,0,q,update inuse:1b from .m.di.0asyncdispatch.servers where handle=99i,1,,manually mark server in-use to simulate dispatched query -run,0,0,q,ad.addclientdetails[5i],1,,add fake client -run,0,0,q,.m.di.0asyncdispatch.addquery["5+5";enlist`inflight;raze;();0Wn;0b],1,,queue query for fake server -run,0,0,q,qid4:exec first queryid from qq[] where query~\:"5+5",1,,capture query id -run,0,0,q,.m.di.0asyncdispatch.results[qid4]:(5i;enlist[`inflight]!enlist(99i;(::);0b)),1,,inject in-flight slot to simulate dispatched-but-pending query -run,0,0,q,ad.removeclienthandle[5i],1,,disconnect client mid-flight -true,0,0,q,0b~first exec inuse from srv[] where handle=99i,1,,in-flight server freed on client disconnect -run,0,0,q,ad.removeserverhandle[99i],1,,clean up fake server - -/ Test 10: addservererror - real backend error path via IPC callback -run,0,0,q,ad.setcallbacks[`.m.di.0asyncdispatch.addserverresult;`.m.di.0asyncdispatch.addservererror],1,,restore callbacks overridden in test 8 -run,0,0,q,ad.setgetnextqueryid[{r:0!select from .m.di.0asyncdispatch.queryqueue where null returntime;1 sublist select from r where not queryid in key .m.di.0asyncdispatch.results}],1,,restore scheduler overridden in test 8 -run,0,0,q,.m.di.0asyncdispatch.addquery["'errortest";enlist`mock;raze;();0Wn;0b],1,,queue a query that will signal an error on the backend -run,0,0,q,qid5:exec first queryid from qq[] where query~\:"'errortest",1,,capture the queued query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch query to backend -run,0,0,q,bh"",1,,sync round-trip: ensures backend has processed and sent error callback -true,0,0,q,1b~first exec error from qq[] where queryid=qid5,1,,backend error callback flagged query as error -true,0,0,q,not null first exec returntime from qq[] where queryid=qid5,1,,returntime stamped on backend error - -/ Test 11: postback wrapping - non-empty postback produces (postback;query;result) tuple -run,0,0,q,captured::(),1,,global to capture what formatresponse receives -run,0,0,q,ad.setformatresponse[{[s;sync;r]captured::r;$[not[s]and sync;'r;r]}],1,,intercept formatresponse to verify tosend construction -run,0,0,q,.m.di.0asyncdispatch.addquery["7+7";enlist`mock;raze;enlist(::);0Wn;0b],1,,queue query with non-empty postback -run,0,0,q,qid6:exec first queryid from qq[] where query~\:"7+7",1,,capture query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch -run,0,0,q,bh"",1,,sync flush -true,0,0,q,0b~first exec error from qq[] where queryid=qid6,1,,postback query completed without error -true,0,0,q,(::;"7+7";enlist 14)~captured,1,,formatresponse received (postback;query;result) tuple -run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse - -/ Test 12: multi-servertype scatter-gather - two backends dispatched in parallel results joined -run,0,0,q,ad.setformatresponse[{[s;sync;r]captured::r;$[not[s]and sync;'r;r]}],1,,intercept formatresponse to capture joined result -run,0,0,q,.m.di.0asyncdispatch.addquery["1+1";`mock`scatter;{sum x};();0Wn;0b],1,,queue scatter-gather query requiring both servertypes -run,0,0,q,qid7:exec first queryid from qq[] where query~\:"1+1",1,,capture query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to both backends in parallel -run,0,0,q,bh"",1,,sync flush first backend -run,0,0,q,bh2"",1,,sync flush second backend -true,0,0,q,0b~first exec error from qq[] where queryid=qid7,1,,scatter-gather completed without error -true,0,0,q,4~captured,1,,joined result is sum of both backend results (1+1=2 from each; sum 2 2 = 4) -run,0,0,q,ad.setformatresponse[{[status;sync;result]$[not[status]and sync;'result;result]}],1,,restore default formatresponse - -/ Test 13: execqueryto local path - successful result delivered in-process -run,0,0,q,ad.setcallbacks[`.m.di.0asyncdispatch.addserverresult;`.m.di.0asyncdispatch.addservererror],1,,restore callbacks -run,0,0,q,ad.setgetnextqueryid[{r:0!select from .m.di.0asyncdispatch.queryqueue where null returntime;1 sublist select from r where not queryid in key .m.di.0asyncdispatch.results}],1,,restore scheduler -run,0,0,q,.test.localcap:(::),1,,reset local capture -run,0,0,q,.test.localfn:{[reqid;qry;res] .test.localcap:(reqid;qry;res)},1,,define local capturing callback in root namespace -run,0,0,q,ad.execqueryto[0Ni;"2+2";enlist`mock;{sum x};(.test.localfn;`req1);0Wn;0b],1,,dispatch via local path with postback (sum collapses single-element result to scalar) -run,0,0,q,bh"",1,,sync flush - wait for backend result callback -true,0,0,q,(`req1;"2+2";4)~.test.localcap,1,,postback invoked in-process with (reqid;query;result) - -/ Test 14: execqueryto local path - backend error delivered in-process via postback -run,0,0,q,.test.localcap:(::),1,,reset local capture -run,0,0,q,ad.execqueryto[0Ni;"'errortest";enlist`mock;raze;(.test.localfn;`req2);0Wn;0b],1,,dispatch erroring query via local path -run,0,0,q,bh"",1,,sync flush - wait for backend error callback -true,0,0,q,`req2~.test.localcap[0],1,,reqid correct -true,0,0,q,.test.localcap[2] like "error:*",1,,error string delivered as result via postback - -/ Test 15: execqueryto local path - timeout delivered in-process via postback -run,0,0,q,.test.localcap:(::),1,,reset local capture -run,0,0,q,ad.execqueryto[0Ni;"3+3";enlist`mock;raze;(.test.localfn;`req3);0D00:00:00.000000001;0b],1,,dispatch with near-zero timeout -run,0,0,q,ad.checktimeout[],1,,trigger timeout sweep -true,0,0,q,`req3~.test.localcap[0],1,,reqid correct -true,0,0,q,.test.localcap[2] like "error:*",1,,timeout error delivered in-process via postback - -/ Test 16: removeclients purges stale client audit rows -run,0,0,q,ad.addclientdetails[7i],1,,record a client for the cleanup test -true,0,0,q,1~count select from cl[] where clienth=7i,1,,client audit row exists -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p+2D}],1,,fast-forward clock two days past the audit age -run,0,0,q,ad.removeclients[0D00:01],1,,purge client rows older than one minute -true,0,0,q,0~count select from cl[] where clienth=7i,1,,client audit row purged -run,0,0,q,.m.di.0asyncdispatch.setcp[{.z.p}],1,,restore the real clock - -/ Test 17: join-failure path - join function throws, client receives error and query is finished -/ dispatched to the scatter backend (bh2): the mock backend (bh) is left inuse by the in-flight timeout in test 15 -run,0,0,q,.m.di.0asyncdispatch.addquery["2+2";enlist`scatter;{'"joinboom"};();0Wn;0b],1,,queue a query whose join function throws -run,0,0,q,qid8:exec first queryid from qq[] where join~\:{'"joinboom"},1,,capture the query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to the scatter backend -run,0,0,q,bh2"",1,,sync flush - wait for the result callback and join attempt -true,0,0,q,1b~first exec error from qq[] where queryid=qid8,1,,join failure correctly flagged as error -true,0,0,q,not null first exec returntime from qq[] where queryid=qid8,1,,returntime stamped after join failure - -/ Test 18: execqueryto with a non-null explicit replyto stores it as clienth with local=0b -run,0,0,q,ad.execqueryto[99i;"1+1";enlist`scatter;{sum x};();0Wn;0b],1,,dispatch with explicit replyto handle 99i -run,0,0,q,qid9:exec first queryid from qq[] where (clienth=99i)&query~\:"1+1",1,,capture the query id -true,0,0,q,99i~first exec clienth from qq[] where queryid=qid9,1,,replyto stored as clienth -true,0,0,q,0b~first exec local from qq[] where queryid=qid9,1,,local flag is 0b for a non-null replyto -run,0,0,q,bh2"",1,,sync flush - completes the backend round-trip (reply to fake 99i fails silently) - -/ Test 19: pluggable server source - dispatch against an injected source whose servertype is not in the -/ internal registry; result routing uses the query's dispatch record so no internal registration is needed -run,0,0,q,savedavail:.m.di.0asyncdispatch.availableservers,1,,save the default (internal-registry) server source -run,0,0,q,ad.setavailableservers[{[eu]([]handle:enlist bh;servertype:enlist`ext)}],1,,inject a source presenting bh under servertype ext (bh is registered as mock) -run,0,0,q,.m.di.0asyncdispatch.addquery["6*7";enlist`ext;raze;();0Wn;0b],1,,queue a query for the external-only servertype -run,0,0,q,qidx:exec first queryid from qq[] where query~\:"6*7",1,,capture the query id -run,0,0,q,.m.di.0asyncdispatch.runnextquery[],1,,dispatch to bh via the injected source -run,0,0,q,bh"",1,,sync flush - wait for the backend result callback -true,0,0,q,0b~first exec error from qq[] where queryid=qidx,1,,query completed without error via the injected source -true,0,0,q,not null first exec returntime from qq[] where queryid=qidx,1,,result routed via the dispatch record (servertype ext) not the servers table (mock) -run,0,0,q,ad.setavailableservers[savedavail],1,,restore the default internal-registry source - -/ Teardown -run,0,0,q,neg[bh](exit;0);neg[bh](::),1,,send exit to first backend process and flush -run,0,0,q,hclose bh,1,,close first backend handle -run,0,0,q,neg[bh2](exit;0);neg[bh2](::),1,,send exit to second backend process and flush -run,0,0,q,hclose bh2,1,,close second backend handle From 06cdf5a6a713a853f5229b1695be8aa60214804a Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 14 Jul 2026 11:04:40 +0100 Subject: [PATCH 10/14] responding to automated generated comments --- di/config/config.md | 14 ++++++++++++-- di/config/config.q | 39 +++++++++++++++++++++++++++++---------- di/config/test.csv | 20 +++++++++++++++++++- 3 files changed, 60 insertions(+), 13 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index bca13ab4..9bdc921a 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -129,13 +129,23 @@ None — `di.config` is a standalone module. gain. The `default` fallback was dropped with it — every config is booted with a default, so a requested key is always present and a fallback arg is dead weight. A missing key is now a caller bug (surfaced as a q null on the inline index), not something the config layer silently papers over. +- **`getmodule` returns leaf settings only — child namespaces are excluded.** `\v .rdb` lists any + child namespace (e.g. `.rdb.sub`) alongside the real settings, so `getmodule` filters them out — + a module's config slice should be flat setting→value pairs, not a nested namespace. A child + namespace is detected as a `99h` dict carrying a null-symbol self-reference key; a genuine + dict-valued setting has no such key and is kept, so it is not misidentified as a namespace. Open gaps / not implemented: - **No runtime reload.** TorQ's `reloadf` (force-reload an already-loaded file) is not ported — `loadconfig` always dedups. Adequate for startup; revisit if runtime config reload is needed. -- **`overrideconfig`'s null-parse guard targets scalar/vector basic types**; a string-typed (`10h`) - config var is an uncommon edge where the per-element null check is loose. +- **`overrideconfig` parse-failure detection is type-aware.** Most basic types parse to a null on a + bad value, which `applyoverride` rejects via a null check. Boolean (`1h`) and byte (`4h`) are the + only in-scope basic types with **no null** — `"B"$"bad"` yields `0b`, `"X"$"gg"` yields `0x00`, both + non-null — so a bad value would slip past a null check and silently corrupt config. `parsefailed` + handles these two explicitly (boolean must be `"0"`/`"1"`; byte must be even-length hex); anything + else is rejected and logged, never written. A string-typed (`10h`) config var still can't be + overridden (its per-element char cast is rejected as non-parsing) — this fails safe and is a rare edge. - **No committed `di.log` integration test yet.** `di.log` (the standard logger) currently lives on the `feature-logging` branch and isn't merged, so a committed `use\`di.log` test would fail on this branch. di.config's contract match with `di.log` is **verified manually** — the real `di.log` was diff --git a/di/config/config.q b/di/config/config.q index 872a9495..9efbc945 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -11,9 +11,10 @@ / internal helper loadconfig handles per-file loading behind loadcascade. raiseerror:{[ctx;msg] - / internal - log an error under ctx then signal it, so failures are observable - / in the log as well as thrown to the caller. - .z.m.logerr[ctx;msg]; + / internal - log an error under ctx then signal it, so failures are observable. + / guard the log call: the logger is unset until init runs, so a public fn misused + / before init still signals the real message rather than a name error on logerr. + .[{.z.m.logerr[x;y]};(ctx;msg);{}]; '"di.config: ",string[ctx],": ",msg; }; @@ -93,11 +94,23 @@ loadcascade:{[dirs;names] :raze {[ds;nm] loadconfig[;nm] each ds}[dirs;] each names; }; +hexchars:"0123456789abcdefABCDEF"; + +parsefailed:{[t;raw;vals] + / internal - true if any raw string failed to parse to type t. boolean (1h) and byte + / (4h) are the only in-scope basic types with no null, so a bad parse ("B"$"bad" -> 0b, + / "X"$"gg" -> 0x00) slips past a null check and would silently corrupt config - validate + / their raw string form explicitly. every other type yields a null on a bad parse. + :$[1h=abs t;not all raw in (enlist"0";enlist"1"); + 4h=abs t;not all {(0=count[x] mod 2) and all x in hexchars} each raw; + any null vals]; + }; + applyoverride:{[name;raw] - / internal - parse raw command-line value(s) into name's current type and set - / it. raw is a string or a list of strings. returns 1b if applied, 0b if the - / variable is not a basic type or a value failed to parse (null). the variable - / must already exist - its current type drives the parse. + / internal - parse raw command-line value(s) into name's current type and set it. raw + / is a string or a list of strings. returns 1b if applied, 0b if the variable is not a + / basic type or a value failed to parse. the variable must already exist - its current + / type drives the parse. t:type value name; if[not (abs t) within (1;-1+count .Q.t); .z.m.logerr[`overrideconfig;"cannot override ",(string name),": not a basic type"]; @@ -105,11 +118,11 @@ applyoverride:{[name;raw] ]; raw:$[10h=type raw;enlist raw;raw]; vals:(upper .Q.t abs t)$'raw; - if[t<0;vals:first vals]; - if[any null vals; + if[parsefailed[t;raw;vals]; .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; :0b; ]; + if[t<0;vals:first vals]; .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; set[name;vals]; :1b; @@ -161,5 +174,11 @@ getmodule:{[namespace] ]; ns:`$".",string namespace; vars:@[{system"v ",x};".",string namespace;`$()]; - :vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; + cfg:vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; + / \v lists child namespaces alongside settings, so drop them - only leaf settings + / belong in a module's config slice. a child namespace is a 99h dict carrying a + / null-symbol self-reference key; a genuine dict-valued setting has no such key and + / is kept. guard the key lookup with a cond (not `and`, which evaluates eagerly and + / would run `key` on non-dict values). + :(where {$[99h=type x;(`) in key x;0b]} each cfg) _ cfg; }; diff --git a/di/config/test.csv b/di/config/test.csv index 9655d6e1..601132cf 100644 --- a/di/config/test.csv +++ b/di/config/test.csv @@ -15,6 +15,10 @@ before,0,0,q,.diconfigtest.enabled:0b,1,1,override target - boolean before,0,0,q,.diconfigtest.rows:100,1,1,override target - long before,0,0,q,.diconfigtest.syms:`a`b,1,1,override target - symbol list before,0,0,q,.diconfigtest.tab:([]a:`long$()),1,1,override target - non-basic (table) +before,0,0,q,.diconfigtest.byte:0x01,1,1,override target - byte (a no-null basic type) +before,0,0,q,.diconfiggm.rows:100,1,1,getmodule target - flat scalar setting +before,0,0,q,.diconfiggm.limits:`lo`hi!1 2,1,1,getmodule target - dict-valued setting (99h - must be kept) +before,0,0,q,.diconfiggm.sub.x:1,1,1,getmodule target - child namespace (must be excluded) before,0,0,q,"system""mkdir -p /tmp/diconfigkxdir""",1,1,dir for the kx.log emission test before,0,0,q,"(`:/tmp/diconfigkxdir/default.q) 0: enlist ""diconfigkxvar:1;""",1,1,file loaded through the real logger comment,,,,,,,init - dependency validation @@ -26,7 +30,11 @@ comment,,,,,,,loadcascade - full cascade over directories and names run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,run the cascade true,0,0,q,"4=count cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,one path per dir-name pair (skipped files included) true,0,0,q,10=diconfigcx,1,1,most-specific name wins over dir layer (c1/rdb loads last) -true,0,0,q,`c1default`c2default`c1rdb~diconfigtrace,1,1,name-major order; and re-runs don't re-execute (dedup) +run,0,0,q,cfg.init[enlist[`log]!enlist mocklog],1,1,re-init to reset the dedup set so the trace check is independent of any prior load (order/re-run safe) +run,0,0,q,diconfigtrace:`symbol$(),1,1,reset the tracer here in the assert phase - a before-row reset is batched once and cannot isolate this +run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,load once - populates the tracer +run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,re-run - dedup must NOT re-execute the files +true,0,0,q,`c1default`c2default`c1rdb~diconfigtrace,1,1,name-major order AND dedup (tracer unchanged by the re-run) true,0,0,q,"any (cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]) like ""*diconfigc2/rdb.q""",1,1,returned paths include skipped (missing) files true,0,0,q,"any (exec msg from logtab where lvl=`info) like ""*no config file*""",1,1,a missing cascade file is skipped at info (internal loadconfig) true,0,0,q,"1=count cfg.loadcascade[""/tmp/diconfigc1"";`default]",1,1,accepts a single string dir and single symbol name @@ -36,6 +44,13 @@ comment,,,,,,,overrideconfig - parse and apply command-line overrides run,0,0,q,"cfg.overrideconfig[`.diconfigtest.enabled`.diconfigtest.rows!(enlist""1"";enlist""500"")]",1,1,override a bool and a long true,0,0,q,1b~.diconfigtest.enabled,1,1,bool parsed and set from string true,0,0,q,500=.diconfigtest.rows,1,1,long parsed and set from string +true,0,0,q,"0=count cfg.overrideconfig[enlist[`.diconfigtest.enabled]!enlist ""bad""]",1,1,unparseable boolean is rejected (no null - must not slip through) +true,0,0,q,1b~.diconfigtest.enabled,1,1,rejected boolean override leaves the variable unchanged (not silently set to 0b) +true,0,0,q,"any (exec msg from logtab) like ""*did not parse*""",1,1,rejected boolean override is logged +true,0,0,q,"(enlist`.diconfigtest.byte)~cfg.overrideconfig[enlist[`.diconfigtest.byte]!enlist ""ff""]",1,1,valid hex byte parsed and applied +true,0,0,q,0xff~.diconfigtest.byte,1,1,byte set from valid hex string +true,0,0,q,"0=count cfg.overrideconfig[enlist[`.diconfigtest.byte]!enlist ""gg""]",1,1,invalid hex byte is rejected (no null - must not slip through as 0x00) +true,0,0,q,0xff~.diconfigtest.byte,1,1,rejected byte override leaves the variable unchanged run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigtest.syms]!enlist (""xx"";""yy"")]",1,1,override a symbol list true,0,0,q,`xx`yy~.diconfigtest.syms,1,1,symbol list parsed and set from strings true,0,0,q,"(enlist`.diconfigtest.rows)~cfg.overrideconfig[enlist[`.diconfigtest.rows]!enlist enlist""7""]",1,1,returns the variables actually overridden @@ -52,6 +67,9 @@ true,0,0,q,2=count cfg.getmodule[`diconfigq],1,1,getmodule returns the whole nam true,0,0,q,`trade`quote~cfg.getmodule[`diconfigq]`subscribeto,1,1,getmodule slice carries the values true,0,0,q,100~cfg.getmodule[`diconfigq]`rows,1,1,getmodule slice carries the second value true,0,0,q,0=count cfg.getmodule[`nosuchns],1,1,getmodule returns an empty dict for an unknown namespace +true,0,0,q,`limits`rows~asc key cfg.getmodule[`diconfiggm],1,1,getmodule excludes child namespaces but keeps flat and dict-valued settings +true,0,0,q,not `sub in key cfg.getmodule[`diconfiggm],1,1,child namespace excluded from the config slice +true,0,0,q,(`lo`hi!1 2)~cfg.getmodule[`diconfiggm]`limits,1,1,dict-valued setting preserved (not misidentified as a namespace) run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigq.rows]!enlist enlist""500""]",1,1,override a stored value true,0,0,q,500~cfg.getmodule[`diconfigq]`rows,1,1,getmodule reflects overrideconfig changes fail,0,0,q,cfg.getmodule[42],1,1,getmodule rejects a non-symbol namespace From 84533b3810302659d1755758d113009353bb100c Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 14 Jul 2026 11:21:13 +0100 Subject: [PATCH 11/14] updated following automated reviewer comments --- di/config/config.md | 9 ++++++--- di/config/config.q | 28 +++++++++++++++++----------- di/config/test.csv | 4 +++- 3 files changed, 26 insertions(+), 15 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index 9bdc921a..81fa2c2b 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -131,9 +131,12 @@ None — `di.config` is a standalone module. bug (surfaced as a q null on the inline index), not something the config layer silently papers over. - **`getmodule` returns leaf settings only — child namespaces are excluded.** `\v .rdb` lists any child namespace (e.g. `.rdb.sub`) alongside the real settings, so `getmodule` filters them out — - a module's config slice should be flat setting→value pairs, not a nested namespace. A child - namespace is detected as a `99h` dict carrying a null-symbol self-reference key; a genuine - dict-valued setting has no such key and is kept, so it is not misidentified as a namespace. + a module's config slice should be flat setting→value pairs, not a nested namespace. Each name is + tested by asking kdb+ directly whether it is itself a namespace (`\v` on the qualified name succeeds + for a namespace and signals for a plain variable). This is deliberately used over the tempting + null-symbol-self-key heuristic (`` (`) in key value ``), which is fragile at the edges — a + dict-valued setting could coincidentally carry a null-symbol key (and be wrongly dropped), or a + namespace reassigned to a plain dict could lack one (and be wrongly kept). Open gaps / not implemented: diff --git a/di/config/config.q b/di/config/config.q index 9efbc945..ee057029 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -11,10 +11,12 @@ / internal helper loadconfig handles per-file loading behind loadcascade. raiseerror:{[ctx;msg] - / internal - log an error under ctx then signal it, so failures are observable. - / guard the log call: the logger is unset until init runs, so a public fn misused - / before init still signals the real message rather than a name error on logerr. - .[{.z.m.logerr[x;y]};(ctx;msg);{}]; + / internal - log an error under ctx then signal it, so the domain error is always + / observable. the logger is unset until init runs, so log only when it is actually + / wired (checked without calling it) - a public fn misused before init still signals + / the real message instead of a name error on logerr. a wired logger is called + / directly, as everywhere else in the module, so its own errors are not swallowed. + if[@[{.z.m.logerr;1b};::;0b];.z.m.logerr[ctx;msg]]; '"di.config: ",string[ctx],": ",msg; }; @@ -122,6 +124,9 @@ applyoverride:{[name;raw] .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; :0b; ]; + / reduce to a scalar only after parsefailed - it validates the full per-element list + / (its null check spans every parsed value); this runs before the log and set below, + / so both use the scalar form. if[t<0;vals:first vals]; .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; set[name;vals]; @@ -174,11 +179,12 @@ getmodule:{[namespace] ]; ns:`$".",string namespace; vars:@[{system"v ",x};".",string namespace;`$()]; - cfg:vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; - / \v lists child namespaces alongside settings, so drop them - only leaf settings - / belong in a module's config slice. a child namespace is a 99h dict carrying a - / null-symbol self-reference key; a genuine dict-valued setting has no such key and - / is kept. guard the key lookup with a cond (not `and`, which evaluates eagerly and - / would run `key` on non-dict values). - :(where {$[99h=type x;(`) in key x;0b]} each cfg) _ cfg; + / \v lists child namespaces alongside settings, so drop them - a module's config slice + / is flat setting->value pairs, not nested namespaces. ask kdb+ directly whether each + / name is itself a namespace (\v succeeds on a namespace, signals on a plain variable); + / this is more robust than inspecting the dict for a null-symbol self-reference key, + / which a plain dict-valued setting could coincidentally carry (or a namespace lack). + issub:{[ns;v] @[{system"v ",x;1b};string ` sv (ns;v);0b]}; + vars:vars where not issub[ns;] each vars; + :vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; }; diff --git a/di/config/test.csv b/di/config/test.csv index 601132cf..ef09ba2c 100644 --- a/di/config/test.csv +++ b/di/config/test.csv @@ -19,6 +19,7 @@ before,0,0,q,.diconfigtest.byte:0x01,1,1,override target - byte (a no-null basic before,0,0,q,.diconfiggm.rows:100,1,1,getmodule target - flat scalar setting before,0,0,q,.diconfiggm.limits:`lo`hi!1 2,1,1,getmodule target - dict-valued setting (99h - must be kept) before,0,0,q,.diconfiggm.sub.x:1,1,1,getmodule target - child namespace (must be excluded) +before,0,0,q,.diconfiggm.nullkey:(enlist `)!enlist 42,1,1,getmodule target - dict setting carrying a null-symbol key (must NOT be treated as a namespace) before,0,0,q,"system""mkdir -p /tmp/diconfigkxdir""",1,1,dir for the kx.log emission test before,0,0,q,"(`:/tmp/diconfigkxdir/default.q) 0: enlist ""diconfigkxvar:1;""",1,1,file loaded through the real logger comment,,,,,,,init - dependency validation @@ -67,9 +68,10 @@ true,0,0,q,2=count cfg.getmodule[`diconfigq],1,1,getmodule returns the whole nam true,0,0,q,`trade`quote~cfg.getmodule[`diconfigq]`subscribeto,1,1,getmodule slice carries the values true,0,0,q,100~cfg.getmodule[`diconfigq]`rows,1,1,getmodule slice carries the second value true,0,0,q,0=count cfg.getmodule[`nosuchns],1,1,getmodule returns an empty dict for an unknown namespace -true,0,0,q,`limits`rows~asc key cfg.getmodule[`diconfiggm],1,1,getmodule excludes child namespaces but keeps flat and dict-valued settings +true,0,0,q,`limits`nullkey`rows~asc key cfg.getmodule[`diconfiggm],1,1,getmodule excludes child namespaces but keeps flat and dict-valued settings true,0,0,q,not `sub in key cfg.getmodule[`diconfiggm],1,1,child namespace excluded from the config slice true,0,0,q,(`lo`hi!1 2)~cfg.getmodule[`diconfiggm]`limits,1,1,dict-valued setting preserved (not misidentified as a namespace) +true,0,0,q,42~first cfg.getmodule[`diconfiggm]`nullkey,1,1,dict setting with a null-symbol key kept (namespace test is \v-based not self-key-based) run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigq.rows]!enlist enlist""500""]",1,1,override a stored value true,0,0,q,500~cfg.getmodule[`diconfigq]`rows,1,1,getmodule reflects overrideconfig changes fail,0,0,q,cfg.getmodule[42],1,1,getmodule rejects a non-symbol namespace From a2a24f8fac47b7e2a9a1163bbbed87bc7c444e72 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 14 Jul 2026 12:10:10 +0100 Subject: [PATCH 12/14] responding to J.Masset comments --- di/config/config.md | 4 +- di/config/config.q | 147 ++++++++++++++++++-------------------------- 2 files changed, 62 insertions(+), 89 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index 81fa2c2b..fdd6eeeb 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -45,8 +45,8 @@ caller first — di.config does no wrapping: `` `info`warn`error!({[c;m]inst[`in | `overrideconfig` | `overrideconfig[params]` | Apply the **command-line-argument layer** on top of the file cascade — di.torq parses the process command line (`.Q.opt .z.x`) and calls this at startup, so launch-time flags (e.g. `-loglevel info`, `-tablelist trade quote`) win over the settings files. This is the automatic top layer of the cascade, **not** an interactive/by-hand step. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | | `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. Callers read a single setting by indexing the returned dict inline (e.g. `getmodule[\`rdb]\`subscribeto`). | -Internal helpers — `loadconfig` (the per-file loader behind `loadcascade`), `raiseerror`, -`applyoverride`, and the load-tracking state — are deliberately not exported. +Internal helpers — `loadconfig` (the per-file loader behind `loadcascade`), +`applyoverride`, `parsefailed`, and the load-tracking state — are deliberately not exported. `loadcascade` is the top-level entry point. A caller (e.g. `di.torq`, which resolves the process identity and directory paths) drives it as: diff --git a/di/config/config.q b/di/config/config.q index ee057029..d70ec27a 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -1,34 +1,16 @@ -/ configuration loading and cascade resolution for the modular torq world. -/ replaces torq.q's in-process config handling (loadf, loadconfig, loadaddconfig, -/ overrideconfig). the logger is an injected dependency wired via init; the module -/ reads no environment variables or process identity itself - the caller (di.torq) -/ supplies the settings directories, name sequence and any command-line overrides. -/ public api: -/ loadcascade - resolve the full cascade over dirs x names (name-major) -/ overrideconfig - apply command-line-style overrides after the cascade -/ getmodule - query the resolved config store as a per-namespace dict; di.torq -/ partitions config per module with it and injects each slice via that module's init -/ internal helper loadconfig handles per-file loading behind loadcascade. - -raiseerror:{[ctx;msg] - / internal - log an error under ctx then signal it, so the domain error is always - / observable. the logger is unset until init runs, so log only when it is actually - / wired (checked without calling it) - a public fn misused before init still signals - / the real message instead of a name error on logerr. a wired logger is called - / directly, as everywhere else in the module, so its own errors are not swallowed. - if[@[{.z.m.logerr;1b};::;0b];.z.m.logerr[ctx;msg]]; - '"di.config: ",string[ctx],": ",msg; - }; +/ configuration loading and cascade resolution for the modular torq world - replaces +/ torq.q's loadf/loadconfig/loadaddconfig/overrideconfig. the logger is injected via init; +/ the module reads no env/identity - the caller (di.torq) supplies dirs, names and overrides. +/ public api: loadcascade (resolve cascade over dirs x names, name-major), overrideconfig +/ (command-line overrides, applied after), getmodule (query the store as a per-namespace +/ dict; di.torq partitions config per module with it). loadconfig is the internal per-file +/ loader behind loadcascade. init:{[deps] - / wire injectable dependencies - log is required, there is no silent fallback. - / deps: a dict with a `log key; log must already be a binary `info`warn`error - / dict of {[c;m]} loggers - build it from di.log (the standard logger, which - / exports binary info/warn/error) or hand-roll one. no adaptation is performed - / here, so a non-conforming logger (e.g. a raw monadic kx.log instance) must be - / wrapped by the caller first. examples: - / config.init[enlist[`log]!enlist logdep] - / config.init[`log`timer!(logdep;timerdep)] + / wire the injected logger - required, no silent fallback. deps: a dict with a `log key + / holding a binary `info`warn`error dict of {[c;m]} loggers (from di.log or hand-rolled). + / no adaptation here, so a monadic kx.log instance must be wrapped first. + / e.g. config.init[enlist[`log]!enlist logdep] if[99h<>type deps; '"di.config: deps must be a dict with `log key"]; if[not `log in key deps; @@ -45,15 +27,16 @@ init:{[deps] }; loadconfig:{[dir;name] - / internal - load one cascade config file dir/{name}.q if present, tracking it - / so it is not re-loaded. a missing file is normal in the cascade (not every - / proctype/procname has one), so its absence is logged at info and skipped, not - / warned. returns the file path. + / internal - load cascade file dir/{name}.q if present, tracking it so it is not + / re-loaded. a missing file is normal in the cascade, so logged at info and skipped. + / returns the file path. if[not 10h=abs type dir; - raiseerror[`loadconfig;"dir must be a string path"]; + .z.m.logerr[`loadconfig;err:"di.config: dir must be a string path"]; + 'err; ]; if[not -11h=type name; - raiseerror[`loadconfig;"name must be a symbol"]; + .z.m.logerr[`loadconfig;err:"di.config: name must be a symbol"]; + 'err; ]; file:dir,"/",string[name],".q"; if[file in .z.m.loaded; @@ -65,32 +48,30 @@ loadconfig:{[dir;name] :file; ]; .z.m.loginfo[`loadconfig;"loading ",file]; - .[system;enlist"l ",file;{[f;e] raiseerror[`loadconfig;"failed to load ",f,": ",e]}[file;]]; + .[system;enlist"l ",file;{[f;e] .z.m.logerr[`loadconfig;err:"di.config: failed to load ",f,": ",e];'err}[file;]]; .z.m.loaded,:enlist file; :file; }; loadcascade:{[dirs;names] - / resolve a configuration cascade name-major: for each config name (least to - / most specific) load it from every directory in turn. files load in sequence, - / so a more specific name always wins over a less specific one regardless of - / directory, and within a single name the later (higher-priority) directory - / overrides the earlier. dirs is a directory path or list of paths (strings), - / ordered lowest->highest priority; names is a config name or list of names - / (symbols), ordered least->most specific. missing files are skipped (by - / loadconfig). returns the flat list of constructed paths, in load order. - / typical caller (di.torq resolves the dirs and process identity): - / config.loadcascade[(kdbconfig;appconfig);`default,proctype,procname] + / resolve a cascade name-major: for each name (least->most specific) load it from every + / dir in turn, so a more specific name always wins and, within a name, the later (higher- + / priority) dir overrides. dirs is a path or list of paths (low->high priority); names a + / symbol or symbol list (least->most specific). missing files are skipped. returns the + / constructed paths in load order. e.g. loadcascade[(kdbconfig;appconfig);`default,proctype,procname] dirs:$[10h=type dirs;enlist dirs;dirs]; names:(),names; if[not 0h=type dirs; - raiseerror[`loadcascade;"dirs must be a string path or a list of string paths"]; + .z.m.logerr[`loadcascade;err:"di.config: dirs must be a string path or a list of string paths"]; + 'err; ]; if[not all 10h=type each dirs; - raiseerror[`loadcascade;"dirs must be a string path or a list of string paths"]; + .z.m.logerr[`loadcascade;err:"di.config: dirs must be a string path or a list of string paths"]; + 'err; ]; if[not 11h=abs type names; - raiseerror[`loadcascade;"names must be a symbol or a symbol list"]; + .z.m.logerr[`loadcascade;err:"di.config: names must be a symbol or a symbol list"]; + 'err; ]; .z.m.loginfo[`loadcascade;"resolving cascade: ",(string count dirs)," dir(s) x ",(string count names)," name(s)"]; :raze {[ds;nm] loadconfig[;nm] each ds}[dirs;] each names; @@ -99,20 +80,18 @@ loadcascade:{[dirs;names] hexchars:"0123456789abcdefABCDEF"; parsefailed:{[t;raw;vals] - / internal - true if any raw string failed to parse to type t. boolean (1h) and byte - / (4h) are the only in-scope basic types with no null, so a bad parse ("B"$"bad" -> 0b, - / "X"$"gg" -> 0x00) slips past a null check and would silently corrupt config - validate - / their raw string form explicitly. every other type yields a null on a bad parse. + / internal - true if any raw string failed to parse to type t. boolean (1h) and byte (4h) + / have no null, so a bad parse ("B"$"bad"->0b, "X"$"gg"->0x00) slips past a null check and + / would corrupt config - validate their string form explicitly. other types null on failure. :$[1h=abs t;not all raw in (enlist"0";enlist"1"); 4h=abs t;not all {(0=count[x] mod 2) and all x in hexchars} each raw; any null vals]; }; applyoverride:{[name;raw] - / internal - parse raw command-line value(s) into name's current type and set it. raw - / is a string or a list of strings. returns 1b if applied, 0b if the variable is not a - / basic type or a value failed to parse. the variable must already exist - its current - / type drives the parse. + / internal - parse raw value(s) into name's current type and set it. raw is a string or + / list of strings. returns 1b if applied, 0b if name is not a basic type or a value failed + / to parse. name must already exist - its type drives the parse. t:type value name; if[not (abs t) within (1;-1+count .Q.t); .z.m.logerr[`overrideconfig;"cannot override ",(string name),": not a basic type"]; @@ -124,9 +103,8 @@ applyoverride:{[name;raw] .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; :0b; ]; - / reduce to a scalar only after parsefailed - it validates the full per-element list - / (its null check spans every parsed value); this runs before the log and set below, - / so both use the scalar form. + / reduce to scalar only after parsefailed (which checks the full per-element list); runs + / before the log and set, so both use the scalar form. if[t<0;vals:first vals]; .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; set[name;vals]; @@ -134,19 +112,19 @@ applyoverride:{[name;raw] }; overrideconfig:{[params] - / apply command-line-style overrides to already-defined variables, parsing each - / value into the variable's existing type. params is a dict keyed by variable - / name (symbol) with string (or list-of-string) values. only defined variables - / can be overridden (their type drives the parse); undefined names are logged - / and skipped. returns the list of variables actually overridden. call after - / loadcascade to let the command line win over file config. + / apply command-line-style overrides to defined variables, parsing each value into the + / variable's existing type. params is a dict of variable name (symbol) -> string (or list + / of strings). only defined names are overridden; undefined ones are logged and skipped. + / returns the variables actually overridden. call after loadcascade so the command line wins. if[99h<>type params; - raiseerror[`overrideconfig;"params must be a dict keyed by variable name"]; + .z.m.logerr[`overrideconfig;err:"di.config: params must be a dict keyed by variable name"]; + 'err; ]; vars:key params; if[0value pairs, not nested namespaces. ask kdb+ directly whether each - / name is itself a namespace (\v succeeds on a namespace, signals on a plain variable); - / this is more robust than inspecting the dict for a null-symbol self-reference key, - / which a plain dict-valued setting could coincidentally carry (or a namespace lack). + / \v lists child namespaces alongside settings; drop them so the slice is flat + / setting->value pairs. ask kdb+ whether each name is itself a namespace (\v succeeds on a + / namespace, signals on a plain variable) - more robust than checking for a null-symbol + / self-key, which a dict-valued setting could carry or a namespace lack. issub:{[ns;v] @[{system"v ",x;1b};string ` sv (ns;v);0b]}; vars:vars where not issub[ns;] each vars; :vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; From be101845f41b5f027f274e379d2092ef34aad231 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Fri, 24 Jul 2026 12:20:43 +0100 Subject: [PATCH 13/14] refactoring over to include the use of .toml files --- di/config/config.md | 310 ++++++++++++++++++++++---------------------- di/config/config.q | 208 ++++++++++++++--------------- di/config/init.q | 2 +- di/config/test.csv | 113 ++++++---------- 4 files changed, 303 insertions(+), 330 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index fdd6eeeb..e6ae6bd9 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -1,22 +1,85 @@ # di.config -Configuration loading and cascade resolution for the modular TorQ world. This -replaces TorQ's in-process config handling (`torq.q`: `loadf`, `loadconfig`, -`loadaddconfig`, `overrideconfig`), where `di.torq` will later distribute the -resolved config to each module via that module's `init`. +Configuration loading and cascade resolution for the modular TorQ world. It replaces +TorQ's in-process config handling (`torq.q`: `loadf`, `loadconfig`, `loadaddconfig`, +`overrideconfig`). `di.config` resolves a settings **cascade** over a builtin root and an +app root — reading **both** flat `name:value` `.q` files and `.toml` files — and returns +the merged configuration as **one flat dict**. `di.torq` calls it once at startup +(`config:cascade[...]`) and hands that dict to each module's `init`. -## Import and init +## The config model — one flat dict + +`di.config` resolves config into a **returned flat dict**; it does not execute settings +files into namespaces or hold any global store. This is the shape `di.torq` consumes: + +```q +config:use`di.config +cfg:config.cascade[builtinroot;approot;`rdb;`rdb1] / -> `subscribeto`rows`... !(...) +/ di.torq stamps identity and hands the whole dict to each module's init: +cfg:cfg,`proctype`procname!(`rdb;`rdb1) +rdb.init[cfg;`log`timer!(logdep;timerdep)] +``` + +`cascade` and `parsefile` are **pure** — no `init`, no logger, no globals — because +`di.torq` resolves config *before* the logger dependency has been built. Only +`overrideconfig` logs (it reports skipped/rejected overrides), so it — and only it — +requires `init`. + +## Precedence + +The cascade is resolved over **two roots** (`builtinroot`, `approot`) × a **three-tier +name sequence** (`default` → `{proctype}` → `{procname}`). Precedence is **name-major**: + +1. A more specific **name** always wins — `{procname}` beats `{proctype}` beats `default`. +2. **Within a name**, the later (app) root overrides the builtin root. +3. **Within a single tier** (one root + one name), the `.toml` file wins over the `.q` + file on a key clash — useful mid-migration when both formats coexist. + +So a key set in both `builtin/{proctype}` and `app/default` resolves to the **builtin +proctype** value, not the app default (name beats root). Effective load order, low → high +priority: + +``` +builtin/default(.q,.toml) → app/default → builtin/{proctype} → app/{proctype} + → builtin/{procname} → app/{procname} (each tier: .toml overrides .q) +``` + +`overrideconfig` sits on **top** of all of this — the command-line layer (see below). + +> **Note — name-major, not root-major.** The KDBX-POC's own vendored `di.config` used +> *root-major* precedence (all builtin tiers, then all app tiers), where `app/default` +> would beat `builtin/{proctype}`. `di.config` deliberately keeps the **name-major** rule +> TorQ has always used; `.toml` support was layered on without changing it. + +> **Note — avoid `-` in config file paths.** A source-level backtick symbol literal +> containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one +> symbol. The runtime string paths `cascade` builds internally are unaffected. + +## Settings file formats + +- **`.q`** — plain `name:value` lines. Split on the first `:`; the RHS is run through + `value`, so `` `:hdb ``, `` `trade`quote ``, `` `symbol$() `` all work. Blank lines and + `/`-comment lines are skipped. These are **flat** files (one process's config), **not** + the namespaced, executable settings files legacy TorQ `\l`-loaded. +- **`.toml`** — delegated to **`di.toml`**, loaded lazily (only when a `.toml` file is + actually present). TOML has no symbol type, so every TOML string comes back as a q + string, never a symbol — consumers that need a symbol normalise at the point of use + (`` `$ `` is a no-op on an already-a-symbol value). -`init` requires a `log` dependency — there is no silent fallback. Pass an -already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` loggers (context -symbol, message string). The module performs **no** adaptation of the logger. +## Import and init -The standard logger is **`di.log`**, which exports binary `info`/`warn`/`error` -functions directly — build the dict from its exports: +`cascade` and `parsefile` need no `init` and no logger — call them directly: ```q config:use`di.config +cfg:config.cascade["/opt/torqx/di/torq/settings";"/opt/app/settings";`rdb;`rdb1] +``` + +`overrideconfig` logs, so wire a `log` dependency via `init` first — there is no silent +fallback. Pass an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` +loggers (context symbol, message string); the module performs **no** adaptation. +```q / option 1: di.log — the standard logger (exports binary info/warn/error) logger:use`di.log logdep:`info`warn`error!(logger.info;logger.warn;logger.error) @@ -31,134 +94,78 @@ config.init[enlist[`log]!enlist mylog] ``` A logger that is *not* already binary `{[c;m]}` (e.g. a raw monadic -[`kx.log`](https://github.com/KxSystems/logging) instance) must be wrapped by the -caller first — di.config does no wrapping: `` `info`warn`error!({[c;m]inst[`info][string[c],": ",m]};…) ``. - -`init` must be called before any other function — there is no default logger. +[`kx.log`](https://github.com/KxSystems/logging) instance) must be wrapped by the caller +first — di.config does no wrapping. ## Exported functions | Function | Signature | Description | |---|---|---| -| `init` | `init[deps]` | Wire the injected logger. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | -| `loadcascade` | `loadcascade[dirs;names]` | Resolve the config cascade **name-major**: for each name (least→most specific) load `dir/{name}.q` from each directory in turn. `dirs` is a string path or list of paths (low→high priority); `names` is a symbol or symbol list (least→most specific). A more specific name wins over the directory layer; within a name the later directory overrides. Missing files are skipped (logged at *info*); already-loaded files are de-duplicated. Returns the flat list of constructed paths, in load order. | -| `overrideconfig` | `overrideconfig[params]` | Apply the **command-line-argument layer** on top of the file cascade — di.torq parses the process command line (`.Q.opt .z.x`) and calls this at startup, so launch-time flags (e.g. `-loglevel info`, `-tablelist trade quote`) win over the settings files. This is the automatic top layer of the cascade, **not** an interactive/by-hand step. `params` is a dict keyed by variable name (symbol) with string (or list-of-string) values, each parsed into that variable's **existing** type. Only already-defined, basic-typed variables can be overridden; undefined names, non-basic types, and unparseable values are logged and skipped. Returns the list of variables actually overridden. | -| `getmodule` | `getmodule[namespace]` | Return a namespace's whole resolved config as a bare-keyed value dict — the slice di.torq passes to a module's `init`. `namespace` is a bare symbol; an unconfigured namespace yields an empty dict. Callers read a single setting by indexing the returned dict inline (e.g. `getmodule[\`rdb]\`subscribeto`). | - -Internal helpers — `loadconfig` (the per-file loader behind `loadcascade`), -`applyoverride`, `parsefailed`, and the load-tracking state — are deliberately not exported. - -`loadcascade` is the top-level entry point. A caller (e.g. `di.torq`, which resolves the -process identity and directory paths) drives it as: - -```q -config.loadcascade[(kdbconfig;appconfig);`default,proctype,procname] -config.overrideconfig[`.myproc.enabled`.myproc.rows!(enlist"1";enlist"5000")] -/ then partition per module and hand each slice to that module's init: -rdbcfg:config.getmodule[`rdb] / -> `subscribeto`hdbtypes!(...) etc. -rdb.init[rdbcfg;`log`timer!(logdep;timerdep)] -/ read a single setting by indexing the slice inline (configs are booted with defaults, -/ so the key is always present — no fallback needed): -rdbcfg`subscribeto -``` +| `cascade` | `cascade[builtinroot;approot;proctype;procname]` | Resolve the settings cascade over the two roots × the `default`/`{proctype}`/`{procname}` name sequence and **return one flat dict**. Pure — no `init`, no logger, no globals. Precedence is name-major with `.toml > .q` within a tier (see above). A key set nowhere is simply absent; an all-missing cascade returns an empty dict. | +| `parsefile` | `parsefile[path]` | Parse a single settings file into a flat dict. A `.toml` path is delegated to `di.toml` (loaded lazily); a `.q` path is flat `name:value` lines. A missing file (either extension) → empty dict. Pure. | +| `overrideconfig` | `overrideconfig[config;params]` | Apply the **command-line override layer** on top of a resolved config dict — the top tier of the cascade. `di.torq` parses the process command line (`.Q.opt .z.x`) and calls this after `cascade`, so launch-time flags (e.g. `-loglevel info`, `-rows 5000`) win over the settings files. `config` is the dict from `cascade`; `params` is a dict keyed by setting name (symbol) with string (or list-of-string) values, each parsed into that setting's **existing** type. Only keys already present with a basic type are overridden; unknown keys, non-basic types, and unparseable values are logged and skipped. Returns the **updated config dict**. Requires `init`. | +| `init` | `init[deps]` | Wire the injected logger for `overrideconfig`. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | +| `getapimeta` | `getapimeta[]` | This module's API metadata — one `(name;public;descrip;params;return)` row per exported function, for `di.torq` to register with `di.api`. | -`overrideconfig` runs after `loadcascade` so command-line values win over file config — it is the -top (command-line) layer of the cascade, applied automatically by di.torq at startup, not a manual -step. di.config is deliberately generic — it does not read environment variables or process -identity itself; the caller (di.torq) supplies the settings directories, the ordered -name sequence, and the override params, then uses `getmodule` to partition the resolved -store per module. - -### The config store - -`loadcascade` loads settings `.q` files that assign into root namespaces -(`.rdb.subscribeto:...`); those resolved namespaces **are** the config store, and -`getmodule` queries them (so anything `overrideconfig` changes is reflected too). -Additional config sources (env vars, k8s config maps) are out of scope for v1 but would -plug in by populating the same namespaces before the store is queried — the `getmodule` -contract stays unchanged. See the EXTENSION POINT comment in `config.q`. - -> **Note:** avoid `-` in config file paths — a source-level backtick symbol literal -> containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one -> symbol. Runtime string paths (built internally by `loadcascade`) are unaffected. +Internal helpers — `parsetier` (per-tier `.q`+`.toml` merge), `applyoverride`, +`parsefailed`, and `hexchars` — are deliberately not exported. ## Injectable dependencies | Injectable | Required keys | Function signature | |---|---|---| -| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string); caller passes an already-conforming binary dict — built from `di.log` (the standard logger, which exports binary `info`/`warn`/`error`) or hand-rolled. No adaptation is done in the module — a non-binary logger (e.g. a raw monadic `kx.log` instance) must be wrapped first | +| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string). Caller passes an already-conforming binary dict — built from `di.log` or hand-rolled. No adaptation is done in the module. Needed only by `overrideconfig`. | ## Hard dependencies -None — `di.config` is a standalone module. +None mandatory. `cascade`/`parsefile` gain a **lazy, soft dependency on `di.toml`**: it is +`` use``d only when a `.toml` file is actually encountered, so the `.q`-only path needs +nothing. + +> **⚠️ `di.toml` is not yet in kdbx-modules** — it currently lives only in the KDBX-POC and +> lands here with the PoC merge. Until then the `.toml` half of a tier is **dormant** in +> this repo: `parsefile` on a `.toml` path would `` use`di.toml `` and fail unless `di.toml` +> is on `QPATH`. The `.q` path works standalone. The `.toml` resolution is therefore +> verified in the PoC (where `di.toml` is vendored), not in this repo's test suite. ## Design notes & open gaps -- **Return values are informational.** `loadcascade` returns the path(s) it *considered* — including - files that were missing or skipped. The real effect is the side-effect load; use the return to see - what was attempted, not what definitely loaded. -- **Missing cascade files are skipped, not errored.** The internal `loadconfig` logs a missing - `dir/{name}.q` at *info* and skips it — absence is normal in the cascade (most `{proctype}`/ - `{procname}` files won't exist). -- **Dedup / idempotency.** `loadconfig` tracks loaded paths in `.z.m.loaded` and skips re-loads, so - the cascade is safe to re-run and `init` is safe to call again. -- **`overrideconfig` is the command-line layer, applied by di.torq — not a manual step.** It mirrors - TorQ's `overrideconfig`/`override[]` (`torq.q`), which TorQ runs at startup off `.Q.opt .z.x` so - launch-time flags beat the settings files. In the modular world di.torq owns that command-line parse - and calls this after `loadcascade`; di.config just does the type-aware apply (which is why it stays - env-free and never reads `.z.x` itself). It *deviates from TorQ deliberately* on error handling: - per-variable problems (undefined name, non-basic type, value that won't parse to the target type) - are logged and skipped — a single bad override never aborts the batch and a null is never written - into config. -- **Entry point is `loadcascade[dirs;names]`, NOT a zero-arg `load[]`.** The Plan sketches - `di.config.load[]`, but that predates the integration design. Config loading happens once at - startup, so a blank call buys nothing and would force the cascade inputs into hidden `.z.m` state - (temporal coupling with `init`). Explicit args keep di.config a pure, env-free resolver: di.torq - resolves `KDBCONFIG`/`KDBAPPCONFIG` → dirs and `proctype`/`procname` → names and passes them. (Also - `load` is a reserved word, so the Plan's literal name is unusable anyway.) If di.torq ever wants a - blank trigger it's a one-line wrapper on its side — not di.config's concern. -- **Config store is live namespace globals, not an explicit internal store (1a, deliberate).** Settings - files assign globals when executed, so those namespaces *are* the store; `getmodule` reads them. - An explicit store with provenance / cross-source precedence (1b) is deferred until a second config - source (env, k8s) exists — the `getmodule` contract is storage-independent, so that swap won't - touch consumers. See the `EXTENSION POINT` comment in `config.q`. -- **No single-key getter — `getmodule` is the only query.** An earlier draft exported a - `get[namespace;key;default]` convenience. It was removed: `getmodule` already returns the whole - namespace as a dict, and the intended flow fetches that slice once (di.torq → each module's `init`) - and indexes it inline (`rdbcfg\`subscribeto`), so a per-key getter added a second query path for no - gain. The `default` fallback was dropped with it — every config is booted with a default, so a - requested key is always present and a fallback arg is dead weight. A missing key is now a caller - bug (surfaced as a q null on the inline index), not something the config layer silently papers over. -- **`getmodule` returns leaf settings only — child namespaces are excluded.** `\v .rdb` lists any - child namespace (e.g. `.rdb.sub`) alongside the real settings, so `getmodule` filters them out — - a module's config slice should be flat setting→value pairs, not a nested namespace. Each name is - tested by asking kdb+ directly whether it is itself a namespace (`\v` on the qualified name succeeds - for a namespace and signals for a plain variable). This is deliberately used over the tempting - null-symbol-self-key heuristic (`` (`) in key value ``), which is fragile at the edges — a - dict-valued setting could coincidentally carry a null-symbol key (and be wrongly dropped), or a - namespace reassigned to a plain dict could lack one (and be wrongly kept). - -Open gaps / not implemented: - -- **No runtime reload.** TorQ's `reloadf` (force-reload an already-loaded file) is not ported — - `loadconfig` always dedups. Adequate for startup; revisit if runtime config reload is needed. -- **`overrideconfig` parse-failure detection is type-aware.** Most basic types parse to a null on a - bad value, which `applyoverride` rejects via a null check. Boolean (`1h`) and byte (`4h`) are the - only in-scope basic types with **no null** — `"B"$"bad"` yields `0b`, `"X"$"gg"` yields `0x00`, both - non-null — so a bad value would slip past a null check and silently corrupt config. `parsefailed` - handles these two explicitly (boolean must be `"0"`/`"1"`; byte must be even-length hex); anything - else is rejected and logged, never written. A string-typed (`10h`) config var still can't be - overridden (its per-element char cast is rejected as non-parsing) — this fails safe and is a rare edge. -- **No committed `di.log` integration test yet.** `di.log` (the standard logger) currently lives on - the `feature-logging` branch and isn't merged, so a committed `use\`di.log` test would fail on this - branch. di.config's contract match with `di.log` is **verified manually** — the real `di.log` was - wired end-to-end with no code change (`init`/`loadcascade`/`loadconfig` all emit through it). The - committed kx.log-wrapper test proves the same binary-`{[c;m]}`-dict contract `di.log` satisfies. Add - a `di.log` integration test once `di.log` merges. +- **Returned dict, not a global store.** Resolution is a pure function returning a dict. + There is no queryable config store and no per-namespace `getmodule` — `di.torq` passes the + whole resolved dict to every module and each reads the keys it cares about. A missing key + is a caller bug (surfaced as a q null on an inline index), not something the config layer + silently papers over — every setting is expected to be booted with a default in the + `default` tier. +- **Retired: the executable-namespaced `.q` path.** An earlier revision also shipped a + `loadcascade`/`loadconfig`/`getmodule` path that `\l`-loaded namespaced, code-bearing `.q` + settings files (`.rdb.subscribeto:...`) into root namespaces and sliced them per module. + That parallel model was **removed** in favour of the single flat-dict cascade `di.torq` + actually consumes. If runtime config reload or an in-process store is ever needed, that is + a deliberate, separately-designed addition — not a silent revival of the globals path. +- **`overrideconfig` is the command-line layer, applied by `di.torq` — not a manual step.** + It mirrors TorQ's `overrideconfig`/`override[]`, which TorQ runs at startup off + `.Q.opt .z.x` so launch-time flags beat the settings files. di.config just does the + type-aware apply onto the resolved dict (which is why it stays env-free and never reads + `.z.x` itself). It deviates from TorQ deliberately on error handling: per-setting problems + (unknown name, non-basic type, value that won't parse to the target type) are logged and + skipped — a single bad override never aborts the batch and a null is never written. +- **`overrideconfig` parse-failure detection is type-aware.** Most basic types parse to a + null on a bad value, which `applyoverride` rejects via a null check. Boolean (`1h`) and + byte (`4h`) are the only in-scope basic types with **no null** — `"B"$"bad"` yields `0b`, + `"X"$"gg"` yields `0x00`, both non-null — so a bad value would slip past a null check and + silently corrupt config. `parsefailed` handles these two explicitly (boolean must be + `"0"`/`"1"`; byte must be even-length hex); anything else is rejected and logged, never + written. A string-typed (`10h`) setting still can't be overridden (its per-element char + cast is rejected as non-parsing) — this fails safe and is a rare edge. +- **The sentinel merge key.** `cascade`'s accumulator is seeded with a sentinel + `` (enlist`)!enlist(::) `` key so its value list stays general from the first merge — a + same-typed value list coalesces to a typed vector that `,:` then refuses to widen (e.g. + adding a `` `symbol$() `` key to a dict whose values so far were all plain symbols). The + sentinel is dropped before the dict is returned. - **Logging uses the three-flat-var convention** — `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, - called as `.z.m.loginfo[\`ctx;"msg"]` — matching `consistency.md` and `di.compression`. The injected - `log` value is still the same `` `info`warn`error `` dict; `init` just fans it out into the three - module-local vars. + called as `.z.m.loginfo[\`ctx;"msg"]` — matching `consistency.md` and `di.compression`. The + injected `log` value is still the same `` `info`warn`error `` dict; `init` fans it out into + the three module-local vars. ## Tests @@ -167,56 +174,47 @@ k4unit:use`di.k4unit k4unit.moduletest`di.config ``` +The `.toml` path is exercised in the KDBX-POC (where `di.toml` is vendored), not here — +see the hard-dependencies note above. + ## Follow-ups (deferred until other modules land) di.config is complete for v1 in isolation. The items below are intentionally deferred; -each is keyed to the module or milestone that unblocks it. Do these **when the trigger -lands**, not before. +each is keyed to the module or milestone that unblocks it. ### ⚠️ Logging call convention — NOT set in stone (ask before changing) di.config uses the **three-flat-var** convention (`.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`), -matching `consistency.md` and `di.compression`. But the project hasn't finalised this versus the -**single-dict** form (`.z.m.log[\`info][…]`) that an earlier skill revision directed. **Before -changing di.config's logging — or wiring logging in another module — stop and ask the user which -convention is authoritative.** If the project settles on single-dict, switching di.config back is -mechanical (storage + call sites only; the injected input contract is identical either way). +matching `consistency.md` and `di.compression`, but the project hasn't finalised this versus +the **single-dict** form (`.z.m.log[\`info][…]`). **Before changing di.config's logging — or +wiring logging in another module — stop and ask the user which convention is authoritative.** +Switching is mechanical (storage + call sites only; the injected input contract is identical). + +### When `di.toml` lands in kdbx-modules (with the PoC merge) + +- **Add committed `.toml` tests.** Today `di.toml` is not on this repo's `QPATH`, so a + committed `.toml`/mixed-tier test would fail here (verified in the PoC instead). Once + `di.toml` resolves, add tests covering: a `.toml` tier parsed into the dict, `.toml` + winning over `.q` within a tier, and a mixed `.q`+`.toml` cascade. ### When `di.log` merges to main -- **Add a committed `di.log` integration test.** Today `di.log` lives on the - `feature-logging` branch (unmerged), so a committed `` use`di.log `` test would fail - here — the contract match is verified manually and stood in for by the kx.log-wrapper - emission test. Once `di.log` resolves on `QPATH`, add a test that builds - `` `info`warn`error!(logger.info;logger.warn;logger.error) `` from `di.log` and drives - `loadcascade` through it. -- **Re-verify only if `di.log`'s contract changes.** It isn't final/approved yet. No - di.config code change is expected — di.config needs only binary `info`/`warn`/`error`, - which `di.log` exports — but if those signatures change, re-run the integration drive. +- **Add a committed `di.log` integration test.** The contract match (binary + `` `info`warn`error `` dict) is verified manually and stood in for by the kx.log-wrapper + emission test. Once `di.log` resolves on `QPATH`, add a test that builds the dict from + `di.log` and drives `overrideconfig` through it. ### When `di.torq` is built -- **Prove the end-to-end flow.** `getmodule` is unit-tested here, but "partition per - module and inject via `init`" only runs for real once `di.torq` exists. Add a - multi-module integration test (under `di.torq`/`di.inttest`, not here) exercising: - di.torq resolves `KDBCONFIG`/`KDBAPPCONFIG` → `dirs` and `proctype`/`procname` → - `names`, calls `loadcascade`, then `overrideconfig` with the command-line params, then - `getmodule` per module → each module's `init`. -- **Hold the env-free boundary.** di.config reads no env vars or process identity — - di.torq owns that resolution and passes `dirs`/`names` explicitly. Do **not** add a - zero-arg `load[]` to di.config; if di.torq wants one it wraps `loadcascade` in a line - on its own side. +- **Prove the end-to-end flow.** Add a multi-module integration test (under + `di.torq`/`di.inttest`, not here) exercising: `di.torq` resolves `KDBCONFIG`/`KDBAPPCONFIG` + → roots and `proctype`/`procname` → identity, calls `cascade`, then `overrideconfig` with + the command-line params, then hands the resolved dict to each module's `init`. +- **Hold the env-free boundary.** di.config reads no env vars or process identity — di.torq + owns that resolution and passes the roots and identity explicitly. ### When `di.depcheck` is built (versioning rollout) -- **Add a `version` export and `deps.q`.** No module ships these yet and `di.depcheck` - (which consumes them) doesn't exist. When it lands, add `version:"…"` to the export and - a `deps.q` (empty/minimal — di.config has no hard deps) as part of the coordinated +- **Add a `version` export and `deps.q`.** When `di.depcheck` lands, add `version:"…"` to the + export and a minimal `deps.q` (di.config has no hard deps) as part of the coordinated repo-wide rollout, not a di.config-only change. - -### When a second config source (env vars, k8s config maps) is needed - -- **Implement the 1b explicit store.** Replace the live-namespace-globals store with an - internal store carrying provenance / cross-source precedence, behind the unchanged - `getmodule` contract (so consumers don't change). See the `EXTENSION POINT` - comment in `config.q`. diff --git a/di/config/config.q b/di/config/config.q index d70ec27a..0a5df8df 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -1,16 +1,27 @@ / configuration loading and cascade resolution for the modular torq world - replaces -/ torq.q's loadf/loadconfig/loadaddconfig/overrideconfig. the logger is injected via init; -/ the module reads no env/identity - the caller (di.torq) supplies dirs, names and overrides. -/ public api: loadcascade (resolve cascade over dirs x names, name-major), overrideconfig -/ (command-line overrides, applied after), getmodule (query the store as a per-namespace -/ dict; di.torq partitions config per module with it). loadconfig is the internal per-file -/ loader behind loadcascade. +/ torq.q's loadf/loadconfig/loadaddconfig/overrideconfig. the module resolves a settings +/ cascade over a builtin root and an app root, reading BOTH flat name:value .q files and +/ .toml files, and RETURNS the merged config as a single flat dict - the shape di.torq +/ consumes (config:cascade[...] straight after use`di.config, then passed to each module's +/ init). the module reads no env/identity - the caller (di.torq) supplies the roots and the +/ process identity (proctype/procname). +/ public api: cascade (resolve the cascade over two roots x default/proctype/procname -> +/ merged flat dict), parsefile (parse one .q or .toml settings file -> flat dict), and +/ overrideconfig (apply the command-line override layer on top of the resolved dict). +/ precedence is name-major (a more specific NAME wins; within a name the later/app root wins) +/ and, within a single tier, .toml wins over .q on a key clash (useful mid-migration). +/ NOTE: di.toml is not yet in kdbx-modules (it lands with the PoC merge) - the .toml half of +/ a tier stays dormant here (parsefile delegates .toml to di.toml lazily, only when a .toml +/ file actually exists); the .q path needs it never. +/ cascade/parsefile are PURE - no init, no logger, no globals - because di.torq resolves +/ config before the logger dependency is even built. overrideconfig logs (it is the one place +/ that reports skipped/rejected overrides), so it - and only it - requires init. init:{[deps] - / wire the injected logger - required, no silent fallback. deps: a dict with a `log key - / holding a binary `info`warn`error dict of {[c;m]} loggers (from di.log or hand-rolled). - / no adaptation here, so a monadic kx.log instance must be wrapped first. - / e.g. config.init[enlist[`log]!enlist logdep] + / wire the injected logger - required for overrideconfig, no silent fallback. deps: a dict + / with a `log key holding a binary `info`warn`error dict of {[c;m]} loggers (from di.log or + / hand-rolled). no adaptation here, so a monadic kx.log instance must be wrapped first. + / cascade/parsefile do not need init; overrideconfig does. e.g. config.init[enlist[`log]!enlist logdep] if[99h<>type deps; '"di.config: deps must be a dict with `log key"]; if[not `log in key deps; @@ -22,61 +33,55 @@ init:{[deps] .z.m.loginfo:deps[`log]`info; .z.m.logwarn:deps[`log]`warn; .z.m.logerr:deps[`log]`error; - .z.m.loaded:enlist""; .z.m.loginfo[`init;"di.config initialised"]; }; -loadconfig:{[dir;name] - / internal - load cascade file dir/{name}.q if present, tracking it so it is not - / re-loaded. a missing file is normal in the cascade, so logged at info and skipped. - / returns the file path. - if[not 10h=abs type dir; - .z.m.logerr[`loadconfig;err:"di.config: dir must be a string path"]; - 'err; - ]; - if[not -11h=type name; - .z.m.logerr[`loadconfig;err:"di.config: name must be a symbol"]; - 'err; - ]; - file:dir,"/",string[name],".q"; - if[file in .z.m.loaded; - .z.m.loginfo[`loadconfig;"already loaded ",file]; - :file; - ]; - if[()~key hsym `$file; - .z.m.loginfo[`loadconfig;"no config file (skipping): ",file]; - :file; - ]; - .z.m.loginfo[`loadconfig;"loading ",file]; - .[system;enlist"l ",file;{[f;e] .z.m.logerr[`loadconfig;err:"di.config: failed to load ",f,": ",e];'err}[file;]]; - .z.m.loaded,:enlist file; - :file; +/ --- settings cascade resolution (.q + .toml -> flat dict) --- + +parsefile:{[path] + / parse ONE settings file into a flat dict. a .toml path is delegated to di.toml, loaded + / lazily (only when a .toml file actually appears, so the .q-only path never needs it). + / a .q file is plain `name:value` lines: split on the first ":", the RHS run through value + / (so `:hdb, `trade`quote, `symbol$() all work); blank and "/"-comment lines are skipped. + / a missing file of either extension contributes nothing (empty dict), so the cascade needs + / no presence checks. existence is checked BEFORE the .toml delegation so a missing .toml + / tier never forces di.toml to load - di.toml is pulled in only when a .toml file actually + / exists (di.toml lives only in the PoC today; the .q-only path needs it never). + fsym:`$":",path; + if[0=count key fsym; :()!()]; + if[path like "*.toml"; :(use`di.toml)[`parsefile] path]; + lines:read0 fsym; + lines:lines where 0most specific) load it from every - / dir in turn, so a more specific name always wins and, within a name, the later (higher- - / priority) dir overrides. dirs is a path or list of paths (low->high priority); names a - / symbol or symbol list (least->most specific). missing files are skipped. returns the - / constructed paths in load order. e.g. loadcascade[(kdbconfig;appconfig);`default,proctype,procname] - dirs:$[10h=type dirs;enlist dirs;dirs]; - names:(),names; - if[not 0h=type dirs; - .z.m.logerr[`loadcascade;err:"di.config: dirs must be a string path or a list of string paths"]; - 'err; - ]; - if[not all 10h=type each dirs; - .z.m.logerr[`loadcascade;err:"di.config: dirs must be a string path or a list of string paths"]; - 'err; - ]; - if[not 11h=abs type names; - .z.m.logerr[`loadcascade;err:"di.config: names must be a symbol or a symbol list"]; - 'err; - ]; - .z.m.loginfo[`loadcascade;"resolving cascade: ",(string count dirs)," dir(s) x ",(string count names)," name(s)"]; - :raze {[ds;nm] loadconfig[;nm] each ds}[dirs;] each names; +parsetier:{[base] + / internal - resolve one cascade tier: try base.q then base.toml, .toml winning on a key + / clash if both exist (useful mid-migration). each missing file yields an empty dict. + (parsefile base,".q"),parsefile base,".toml" }; +cascade:{[builtinroot;approot;proctype;procname] + / resolve the settings cascade over the two roots x the (default;proctype;procname) name + / sequence and RETURN the merged flat dict. pure - no init, no logger, no globals - so + / di.torq can call it directly after use`di.config, before the logger exists. precedence is + / name-major: a more specific NAME wins over the root layer (procname beats proctype beats + / default), and within a name the later (app) root overrides the builtin root - so a key set + / in both builtin/{proctype} and app/default resolves to the builtin proctype value. within a + / single tier, .toml wins over .q (see parsetier). the accumulator is seeded with a sentinel + / `!(::) key so its value list stays general from the first merge (a same-typed value list + / coalesces to a typed vector that ,: then refuses to widen); the sentinel is dropped before return. + dirs:(builtinroot;approot); + names:`default,proctype,procname; + bases:raze {[ds;nm] ds,\:"/",string nm}[dirs;] each names; + acc:{[a;b] a,parsetier b}/[(enlist `)!enlist(::);bases]; + acc _ ` + }; + +/ --- command-line override layer (top of the cascade) --- + hexchars:"0123456789abcdefABCDEF"; parsefailed:{[t;raw;vals] @@ -88,76 +93,73 @@ parsefailed:{[t;raw;vals] any null vals]; }; -applyoverride:{[name;raw] - / internal - parse raw value(s) into name's current type and set it. raw is a string or - / list of strings. returns 1b if applied, 0b if name is not a basic type or a value failed - / to parse. name must already exist - its type drives the parse. - t:type value name; +applyoverride:{[name;cur;raw] + / internal - parse raw value(s) into cur's type and return (applied;newvalue). cur is the + / setting's current value from the config dict; its type drives the parse. raw is a string or + / list of strings. applied is 0b (and newvalue is cur, unchanged) if cur is not a basic type + / or a value failed to parse. + t:type cur; if[not (abs t) within (1;-1+count .Q.t); .z.m.logerr[`overrideconfig;"cannot override ",(string name),": not a basic type"]; - :0b; + :(0b;cur); ]; raw:$[10h=type raw;enlist raw;raw]; vals:(upper .Q.t abs t)$'raw; if[parsefailed[t;raw;vals]; .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; - :0b; + :(0b;cur); ]; / reduce to scalar only after parsefailed (which checks the full per-element list); runs - / before the log and set, so both use the scalar form. + / before the log and return so both use the scalar form. if[t<0;vals:first vals]; .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; - set[name;vals]; - :1b; + :(1b;vals); }; -overrideconfig:{[params] - / apply command-line-style overrides to defined variables, parsing each value into the - / variable's existing type. params is a dict of variable name (symbol) -> string (or list - / of strings). only defined names are overridden; undefined ones are logged and skipped. - / returns the variables actually overridden. call after loadcascade so the command line wins. +overrideconfig:{[config;params] + / apply the command-line override layer on top of a resolved config dict, parsing each value + / into the matching setting's existing type. this is the TOP tier of the cascade - di.torq + / parses the process command line (.Q.opt .z.x) and calls this after cascade, so launch-time + / flags win over the settings files. config is the flat dict from cascade; params a dict keyed + / by setting name (symbol) with string (or list-of-string) values. only keys already present in + / config with a basic type are overridden; unknown keys, non-basic types, and unparseable + / values are logged and skipped (a single bad override never aborts the batch and a null is + / never written into config). returns the UPDATED config dict. + if[99h<>type config; + .z.m.logerr[`overrideconfig;err:"di.config: config must be a dict"]; + 'err; + ]; if[99h<>type params; - .z.m.logerr[`overrideconfig;err:"di.config: params must be a dict keyed by variable name"]; + .z.m.logerr[`overrideconfig;err:"di.config: params must be a dict keyed by setting name"]; 'err; ]; vars:key params; if[0value pairs. ask kdb+ whether each name is itself a namespace (\v succeeds on a - / namespace, signals on a plain variable) - more robust than checking for a null-symbol - / self-key, which a dict-valued setting could carry or a namespace lack. - issub:{[ns;v] @[{system"v ",x;1b};string ` sv (ns;v);0b]}; - vars:vars where not issub[ns;] each vars; - :vars!{[ns;v] value ` sv (ns;v)}[ns;] each vars; +getapimeta:{[] + / this module's api metadata, one row per exported function, for di.torq to collect and register + / with di.api. names are bare (the module's own); di.torq applies the process-wide qualification. + / one self-contained (name;public;descrip;params;return) row per line - flip cols!flip rows. + :flip `name`public`descrip`params`return!flip( + (`init; 0b; "wire the injected logger (required by overrideconfig)"; "[dict: deps with a `log key]"; "null"); + (`cascade; 1b; "resolve the settings cascade over two roots -> merged flat dict (name-major, .toml>.q)"; "[string: builtinroot; string: approot; symbol: proctype; symbol: procname]"; "dict: merged setting -> value"); + (`overrideconfig; 1b; "apply the command-line override layer onto a resolved config dict"; "[dict: resolved config; dict: setting name (symbol) -> string or list of strings]"; "dict: updated config"); + (`parsefile; 1b; "parse one .q or .toml settings file into a flat dict"; "[string: file path (.q flat name:value, or .toml via di.toml)]"; "dict: setting -> value (empty if the file is missing)"); + (`getapimeta; 0b; "this module's api metadata rows"; "[]"; "table: metadata rows")); }; diff --git a/di/config/init.q b/di/config/init.q index d5534dc9..2e06d014 100644 --- a/di/config/init.q +++ b/di/config/init.q @@ -1,3 +1,3 @@ / configuration loading and cascade resolution for the modular torq world. \l ::config.q -export:([init;loadcascade;overrideconfig;getmodule]) +export:([init;cascade;overrideconfig;parsefile;getapimeta]) diff --git a/di/config/test.csv b/di/config/test.csv index ef09ba2c..4c955d6a 100644 --- a/di/config/test.csv +++ b/di/config/test.csv @@ -3,78 +3,51 @@ before,0,0,q,cfg:use`di.config,1,1,load the module before,0,0,q,"logtab:([]lvl:`symbol$();ctx:`symbol$();msg:())",1,1,capturing log table before,0,0,q,"mocklog:`info`warn`error!({[c;m]`logtab upsert(`info;c;m)};{[c;m]`logtab upsert(`warn;c;m)};{[c;m]`logtab upsert(`error;c;m)})",1,1,define a capturing binary logger before,0,0,q,cfg.init[enlist[`log]!enlist mocklog],1,1,init with the required log dependency -before,0,0,q,diconfigtrace:`symbol$(),1,1,tracer for cascade load order -before,0,0,q,"system""mkdir -p /tmp/diconfigc1""",1,1,cascade directory one -before,0,0,q,"system""mkdir -p /tmp/diconfigc2""",1,1,cascade directory two -before,0,0,q,"(`:/tmp/diconfigc1/default.q) 0: enlist ""diconfigcx:1;diconfigtrace,:`c1default;""",1,1,c1 default.q -before,0,0,q,"(`:/tmp/diconfigc1/rdb.q) 0: enlist ""diconfigcx:10;diconfigtrace,:`c1rdb;""",1,1,c1 rdb.q -before,0,0,q,"(`:/tmp/diconfigc2/default.q) 0: enlist ""diconfigcx:2;diconfigtrace,:`c2default;""",1,1,c2 default.q (no c2 rdb.q - a missing cascade file) -before,0,0,q,"system""mkdir -p /tmp/diconfigstore""",1,1,cascade dir for the config-store tests -before,0,0,q,"(`:/tmp/diconfigstore/default.q) 0: enlist "".diconfigq.subscribeto:`trade`quote;.diconfigq.rows:100;""",1,1,settings file populating the .diconfigq namespace -before,0,0,q,.diconfigtest.enabled:0b,1,1,override target - boolean -before,0,0,q,.diconfigtest.rows:100,1,1,override target - long -before,0,0,q,.diconfigtest.syms:`a`b,1,1,override target - symbol list -before,0,0,q,.diconfigtest.tab:([]a:`long$()),1,1,override target - non-basic (table) -before,0,0,q,.diconfigtest.byte:0x01,1,1,override target - byte (a no-null basic type) -before,0,0,q,.diconfiggm.rows:100,1,1,getmodule target - flat scalar setting -before,0,0,q,.diconfiggm.limits:`lo`hi!1 2,1,1,getmodule target - dict-valued setting (99h - must be kept) -before,0,0,q,.diconfiggm.sub.x:1,1,1,getmodule target - child namespace (must be excluded) -before,0,0,q,.diconfiggm.nullkey:(enlist `)!enlist 42,1,1,getmodule target - dict setting carrying a null-symbol key (must NOT be treated as a namespace) -before,0,0,q,"system""mkdir -p /tmp/diconfigkxdir""",1,1,dir for the kx.log emission test -before,0,0,q,"(`:/tmp/diconfigkxdir/default.q) 0: enlist ""diconfigkxvar:1;""",1,1,file loaded through the real logger +before,0,0,q,"system""mkdir -p /tmp/diconfigbuiltin""",1,1,builtin root for the cascade +before,0,0,q,"system""mkdir -p /tmp/diconfigapp""",1,1,app root for the cascade +before,0,0,q,"(`:/tmp/diconfigbuiltin/default.q) 0: (""diconfiga:1"";""diconfigb:1"")",1,1,builtin/default flat name:value (two settings) +before,0,0,q,"(`:/tmp/diconfigbuiltin/rdb.q) 0: enlist ""diconfigb:10""",1,1,builtin/rdb proctype tier sets diconfigb +before,0,0,q,"(`:/tmp/diconfigapp/default.q) 0: (""diconfiga:100"";""diconfigb:5"")",1,1,app/default flat name:value (two settings) +before,0,0,q,"(`:/tmp/diconfigapp/rdb1.q) 0: enlist ""diconfiga:999""",1,1,app/rdb1 procname tier sets diconfiga +before,0,0,q,"basecfg:`enabled`rows`syms`tab`byte!(0b;100;`a`b;([]a:`long$());0x01)",1,1,base config dict for the overrideconfig tests comment,,,,,,,init - dependency validation fail,0,0,q,cfg.init[(::)],1,1,init rejects a non-dictionary deps fail,0,0,q,cfg.init[enlist[`foo]!enlist 1],1,1,init rejects deps without a log key fail,0,0,q,cfg.init[enlist[`log]!enlist 42],1,1,init rejects a non-dict log value fail,0,0,q,cfg.init[enlist[`log]!enlist ((enlist`info)!enlist {[c;m]})],1,1,init rejects a log dict missing warn/error -comment,,,,,,,loadcascade - full cascade over directories and names -run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,run the cascade -true,0,0,q,"4=count cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,one path per dir-name pair (skipped files included) -true,0,0,q,10=diconfigcx,1,1,most-specific name wins over dir layer (c1/rdb loads last) -run,0,0,q,cfg.init[enlist[`log]!enlist mocklog],1,1,re-init to reset the dedup set so the trace check is independent of any prior load (order/re-run safe) -run,0,0,q,diconfigtrace:`symbol$(),1,1,reset the tracer here in the assert phase - a before-row reset is batched once and cannot isolate this -run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,load once - populates the tracer -run,0,0,q,"cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]",1,1,re-run - dedup must NOT re-execute the files -true,0,0,q,`c1default`c2default`c1rdb~diconfigtrace,1,1,name-major order AND dedup (tracer unchanged by the re-run) -true,0,0,q,"any (cfg.loadcascade[(""/tmp/diconfigc1"";""/tmp/diconfigc2"");`default`rdb]) like ""*diconfigc2/rdb.q""",1,1,returned paths include skipped (missing) files -true,0,0,q,"any (exec msg from logtab where lvl=`info) like ""*no config file*""",1,1,a missing cascade file is skipped at info (internal loadconfig) -true,0,0,q,"1=count cfg.loadcascade[""/tmp/diconfigc1"";`default]",1,1,accepts a single string dir and single symbol name -fail,0,0,q,cfg.loadcascade[42;`default],1,1,loadcascade rejects non-string dirs -fail,0,0,q,"cfg.loadcascade[""/tmp/diconfigc1"";""default""]",1,1,loadcascade rejects non-symbol names -comment,,,,,,,overrideconfig - parse and apply command-line overrides -run,0,0,q,"cfg.overrideconfig[`.diconfigtest.enabled`.diconfigtest.rows!(enlist""1"";enlist""500"")]",1,1,override a bool and a long -true,0,0,q,1b~.diconfigtest.enabled,1,1,bool parsed and set from string -true,0,0,q,500=.diconfigtest.rows,1,1,long parsed and set from string -true,0,0,q,"0=count cfg.overrideconfig[enlist[`.diconfigtest.enabled]!enlist ""bad""]",1,1,unparseable boolean is rejected (no null - must not slip through) -true,0,0,q,1b~.diconfigtest.enabled,1,1,rejected boolean override leaves the variable unchanged (not silently set to 0b) +comment,,,,,,,parsefile - parse one flat .q settings file into a dict (.toml delegated to di.toml, exercised in the PoC not here) +true,0,0,q,"1~cfg.parsefile[""/tmp/diconfigbuiltin/default.q""]`diconfiga",1,1,parsefile reads a flat name:value .q file into a dict +true,0,0,q,"2=count cfg.parsefile[""/tmp/diconfigbuiltin/default.q""]",1,1,parsefile returns every setting in the file +true,0,0,q,"0=count cfg.parsefile[""/tmp/diconfignope/default.q""]",1,1,parsefile returns an empty dict for a missing file +comment,,,,,,,cascade - merged flat dict over two roots x default/proctype/procname (name-major; .toml>.q within a tier) +true,0,0,q,"100~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdbnofile])`diconfiga",1,1,within a name the later (app) root overrides the builtin root +true,0,0,q,"10~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdbnofile])`diconfigb",1,1,name-major: builtin proctype beats app default (would be 5 under root-major) +true,0,0,q,"999~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb1])`diconfiga",1,1,name-major: the procname tier (app/rdb1) wins over every less-specific name +true,0,0,q,"10~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb1])`diconfigb",1,1,a key absent from the procname tier keeps its most-specific set value (builtin proctype) +true,0,0,q,"not (`) in key cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb1]",1,1,the sentinel merge key is dropped from the returned dict +true,0,0,q,"0=count cfg.cascade[""/tmp/diconfignope1"";""/tmp/diconfignope2"";`rdb;`rdb1]",1,1,an all-missing cascade resolves to an empty dict +comment,,,,,,,overrideconfig - command-line override layer applied onto a resolved config dict +true,0,0,q,"1b~(cfg.overrideconfig[basecfg;`enabled`rows!(enlist""1"";enlist""500"")])`enabled",1,1,bool parsed and set from string +true,0,0,q,"500~(cfg.overrideconfig[basecfg;`enabled`rows!(enlist""1"";enlist""500"")])`rows",1,1,long parsed and set from string +true,0,0,q,"basecfg~cfg.overrideconfig[basecfg;()!()]",1,1,empty params returns the config unchanged +true,0,0,q,"0b~(cfg.overrideconfig[basecfg;enlist[`enabled]!enlist ""bad""])`enabled",1,1,unparseable boolean is rejected - the value is left unchanged (not silently set to 0b via a null) true,0,0,q,"any (exec msg from logtab) like ""*did not parse*""",1,1,rejected boolean override is logged -true,0,0,q,"(enlist`.diconfigtest.byte)~cfg.overrideconfig[enlist[`.diconfigtest.byte]!enlist ""ff""]",1,1,valid hex byte parsed and applied -true,0,0,q,0xff~.diconfigtest.byte,1,1,byte set from valid hex string -true,0,0,q,"0=count cfg.overrideconfig[enlist[`.diconfigtest.byte]!enlist ""gg""]",1,1,invalid hex byte is rejected (no null - must not slip through as 0x00) -true,0,0,q,0xff~.diconfigtest.byte,1,1,rejected byte override leaves the variable unchanged -run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigtest.syms]!enlist (""xx"";""yy"")]",1,1,override a symbol list -true,0,0,q,`xx`yy~.diconfigtest.syms,1,1,symbol list parsed and set from strings -true,0,0,q,"(enlist`.diconfigtest.rows)~cfg.overrideconfig[enlist[`.diconfigtest.rows]!enlist enlist""7""]",1,1,returns the variables actually overridden -true,0,0,q,"0=count cfg.overrideconfig[enlist[`.diconfigtest.nope]!enlist enlist""1""]",1,1,undefined variable overrides nothing -true,0,0,q,"any (exec msg from logtab) like ""*undefined variable*""",1,1,undefined variable logged -true,0,0,q,"0=count cfg.overrideconfig[enlist[`.diconfigtest.tab]!enlist enlist""1""]",1,1,non-basic-type variable overrides nothing -true,0,0,q,"any (exec msg from logtab) like ""*not a basic type*""",1,1,non-basic-type variable logged -true,0,0,q,0=count cfg.overrideconfig[()!()],1,1,empty params is a no-op -fail,0,0,q,cfg.overrideconfig[42],1,1,overrideconfig rejects a non-dict -fail,0,0,q,"cfg.overrideconfig[enlist[1]!enlist enlist""x""]",1,1,overrideconfig rejects non-symbol keys -comment,,,,,,,getmodule - queryable config store -run,0,0,q,"cfg.loadcascade[""/tmp/diconfigstore"";`default]",1,1,load a settings file into the store -true,0,0,q,2=count cfg.getmodule[`diconfigq],1,1,getmodule returns the whole namespace slice -true,0,0,q,`trade`quote~cfg.getmodule[`diconfigq]`subscribeto,1,1,getmodule slice carries the values -true,0,0,q,100~cfg.getmodule[`diconfigq]`rows,1,1,getmodule slice carries the second value -true,0,0,q,0=count cfg.getmodule[`nosuchns],1,1,getmodule returns an empty dict for an unknown namespace -true,0,0,q,`limits`nullkey`rows~asc key cfg.getmodule[`diconfiggm],1,1,getmodule excludes child namespaces but keeps flat and dict-valued settings -true,0,0,q,not `sub in key cfg.getmodule[`diconfiggm],1,1,child namespace excluded from the config slice -true,0,0,q,(`lo`hi!1 2)~cfg.getmodule[`diconfiggm]`limits,1,1,dict-valued setting preserved (not misidentified as a namespace) -true,0,0,q,42~first cfg.getmodule[`diconfiggm]`nullkey,1,1,dict setting with a null-symbol key kept (namespace test is \v-based not self-key-based) -run,0,0,q,"cfg.overrideconfig[enlist[`.diconfigq.rows]!enlist enlist""500""]",1,1,override a stored value -true,0,0,q,500~cfg.getmodule[`diconfigq]`rows,1,1,getmodule reflects overrideconfig changes -fail,0,0,q,cfg.getmodule[42],1,1,getmodule rejects a non-symbol namespace +true,0,0,q,"0xff~(cfg.overrideconfig[basecfg;enlist[`byte]!enlist ""ff""])`byte",1,1,valid hex byte parsed and applied +true,0,0,q,"0x01~(cfg.overrideconfig[basecfg;enlist[`byte]!enlist ""gg""])`byte",1,1,invalid hex byte is rejected (no null - must not slip through as 0x00) +true,0,0,q,"`xx`yy~(cfg.overrideconfig[basecfg;enlist[`syms]!enlist (""xx"";""yy"")])`syms",1,1,symbol list parsed and set from strings +true,0,0,q,"not `nope in key cfg.overrideconfig[basecfg;enlist[`nope]!enlist enlist""1""]",1,1,an unknown setting is not added to the config dict +true,0,0,q,"(key basecfg)~key cfg.overrideconfig[basecfg;enlist[`nope]!enlist enlist""1""]",1,1,an unknown setting leaves the config keys unchanged +true,0,0,q,"any (exec msg from logtab) like ""*unknown setting*""",1,1,unknown setting logged +true,0,0,q,"(cfg.overrideconfig[basecfg;enlist[`tab]!enlist enlist""1""])[`tab]~basecfg`tab",1,1,a non-basic-type setting (table) is left unchanged +true,0,0,q,"any (exec msg from logtab) like ""*not a basic type*""",1,1,non-basic-type setting logged +fail,0,0,q,cfg.overrideconfig[42;()!()],1,1,overrideconfig rejects a non-dict config +fail,0,0,q,cfg.overrideconfig[basecfg;42],1,1,overrideconfig rejects non-dict params +fail,0,0,q,"cfg.overrideconfig[basecfg;enlist[1]!enlist enlist""x""]",1,1,overrideconfig rejects non-symbol params keys +comment,,,,,,,getapimeta - module api metadata +true,0,0,q,(asc key cfg)~asc exec name from cfg.getapimeta[],1,1,getapimeta documents exactly the module's exports +true,0,0,q,`name`public`descrip`params`return~cols cfg.getapimeta[],1,1,getapimeta rows carry the registry columns +true,0,0,q,1b~(1!cfg.getapimeta[])[`cascade]`public,1,1,a public API function (cascade) is marked public +true,0,0,q,0b~(1!cfg.getapimeta[])[`init]`public,1,1,framework plumbing (init) is not public comment,,,,,,,logger emission - real logger via a caller-side binary wrapper (must run last; re-inits cfg) run,0,0,q,kxlogger:use`kx.log,1,1,load the kx.log module run,0,0,q,kxinst:kxlogger.createLog[],1,1,create a real kx.log instance @@ -82,8 +55,8 @@ run,0,0,q,kxh:hopen `:/tmp/diconfigkxsink.txt,1,1,open a file sink to capture ou run,0,0,q,kxinst.add[kxh;`info],1,1,route info-level output to the file sink run,0,0,q,"kxbinary:`info`warn`error!({[c;m]kxinst[`info][string[c],"": "",m]};{[c;m]kxinst[`warn][string[c],"": "",m]};{[c;m]kxinst[`error][string[c],"": "",m]})",1,1,wrap the monadic logger into the binary {[c;m]} contract run,0,0,q,cfg.init[enlist[`log]!enlist kxbinary],1,1,re-init di.config with the conforming binary logger -run,0,0,q,"cfg.loadcascade[""/tmp/diconfigkxdir"";`default]",1,1,drive a real info-level log through the wrapped logger +run,0,0,q,"kxres:cfg.overrideconfig[enlist[`kxrows]!enlist 5;enlist[`kxrows]!enlist enlist""7""]",1,1,drive a real info-level log through the wrapped logger run,0,0,q,hclose kxh,1,1,flush and close the file sink -true,0,0,q,1=diconfigkxvar,1,1,the cascade executed the file -true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*loadconfig: loading*""",1,1,the internal loader emitted through the real logger -true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*diconfigkxdir/default.q*""",1,1,emitted message includes the file path +true,0,0,q,7~kxres`kxrows,1,1,the override was applied (returned dict carries the new value) +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*overrideconfig: setting*""",1,1,the override emitted an info line through the real logger +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*kxrows*""",1,1,emitted message names the overridden setting From 1a1d4fe62d4e6afe9b0e383fed0ea41708c072c0 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Mon, 27 Jul 2026 11:29:38 +0100 Subject: [PATCH 14/14] adding guards to new .toml functions, causing errors to throw instead of failing when di.toml not found and allowing toml loaded configs to be overridden. NOTE: di.toml not being present when loading a toml file will not error through logger due to cascade order from di.torq --- di/config/config.md | 61 +++++++++++++++++++++++++++++++++++++++------ di/config/config.q | 41 +++++++++++++++++++++++------- di/config/test.csv | 14 ++++++++--- 3 files changed, 96 insertions(+), 20 deletions(-) diff --git a/di/config/config.md b/di/config/config.md index e6ae6bd9..97ee0cc7 100644 --- a/di/config/config.md +++ b/di/config/config.md @@ -102,7 +102,7 @@ first — di.config does no wrapping. | Function | Signature | Description | |---|---|---| | `cascade` | `cascade[builtinroot;approot;proctype;procname]` | Resolve the settings cascade over the two roots × the `default`/`{proctype}`/`{procname}` name sequence and **return one flat dict**. Pure — no `init`, no logger, no globals. Precedence is name-major with `.toml > .q` within a tier (see above). A key set nowhere is simply absent; an all-missing cascade returns an empty dict. | -| `parsefile` | `parsefile[path]` | Parse a single settings file into a flat dict. A `.toml` path is delegated to `di.toml` (loaded lazily); a `.q` path is flat `name:value` lines. A missing file (either extension) → empty dict. Pure. | +| `parsefile` | `parsefile[path]` | Parse a single settings file into a flat dict. A `.toml` path is delegated to `di.toml` (loaded lazily, behind the guard rail below); a `.q` path is flat `name:value` lines. A missing file (either extension) → empty dict. An existing `.toml` file with no `di.toml` on `QPATH` signals a clear error naming the file. Pure. | | `overrideconfig` | `overrideconfig[config;params]` | Apply the **command-line override layer** on top of a resolved config dict — the top tier of the cascade. `di.torq` parses the process command line (`.Q.opt .z.x`) and calls this after `cascade`, so launch-time flags (e.g. `-loglevel info`, `-rows 5000`) win over the settings files. `config` is the dict from `cascade`; `params` is a dict keyed by setting name (symbol) with string (or list-of-string) values, each parsed into that setting's **existing** type. Only keys already present with a basic type are overridden; unknown keys, non-basic types, and unparseable values are logged and skipped. Returns the **updated config dict**. Requires `init`. | | `init` | `init[deps]` | Wire the injected logger for `overrideconfig`. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | | `getapimeta` | `getapimeta[]` | This module's API metadata — one `(name;public;descrip;params;return)` row per exported function, for `di.torq` to register with `di.api`. | @@ -120,13 +120,50 @@ Internal helpers — `parsetier` (per-tier `.q`+`.toml` merge), `applyoverride`, None mandatory. `cascade`/`parsefile` gain a **lazy, soft dependency on `di.toml`**: it is `` use``d only when a `.toml` file is actually encountered, so the `.q`-only path needs -nothing. +nothing. **di.toml does not need to be in the repo (or on `QPATH`) for di.config to load or +run** — it is required only at the moment a `.toml` settings file is actually parsed. + +### The `.toml` guard rail + +`parsefile` guards the di.toml load so a missing module produces a clear, actionable error +rather than a cryptic `notfound`. The order is deliberate: + +1. **Check the file exists first.** A missing file (either extension) → empty dict. So a + missing `.toml` tier *never* triggers the di.toml requirement — the internal + `parsetier`/`cascade` probe `base,".toml"` on every tier, and absent tiers cost nothing. +2. **Then, only for a `.toml` file that actually exists, require di.toml.** The internal + `requiretoml` helper attempts `` use`di.toml `` under protected evaluation; if it is not + found, it signals a clear error **naming the offending file**: + ``` + 'di.config: cannot parse TOML file '/path/to/x.toml' - the di.toml module was not found + on QPATH; a di.toml module is required to parse .toml settings (underlying: notfound: di.toml) + ``` +3. **Parse.** With di.toml resolved, the file is delegated to it. + +Because this signal happens at config-resolution time — *before* the logger dependency is +wired — it surfaces via the raised error (which aborts startup with the reason), not via the +injected logger. + +### Contract di.config expects from di.toml + +A di.toml module must export a **`parsefile`** function taking a settings-file path string +and returning a **flat dict** of `setting → value`: + +```q +(use`di.toml)[`parsefile] "/path/to/settings.toml" / -> `dir`rows!(":appdb";100) +``` + +TOML has no symbol type, so di.toml returns every TOML string as a **q char string** (never +a symbol); di.config applies no coercion, so consumers that need a symbol normalise at the +point of use (`` `$ `` is a no-op on an already-a-symbol value). > **⚠️ `di.toml` is not yet in kdbx-modules** — it currently lives only in the KDBX-POC and -> lands here with the PoC merge. Until then the `.toml` half of a tier is **dormant** in -> this repo: `parsefile` on a `.toml` path would `` use`di.toml `` and fail unless `di.toml` -> is on `QPATH`. The `.q` path works standalone. The `.toml` resolution is therefore -> verified in the PoC (where `di.toml` is vendored), not in this repo's test suite. +> lands here with the PoC merge (a di.toml module is in development against the contract +> above). Until it is on `QPATH`, the `.toml` half of a tier is **dormant** in this repo: a +> `.toml` file that exists but cannot be parsed raises the guard error above; the `.q` path +> works standalone. Full `.toml` resolution is verified in the PoC (where `di.toml` is +> vendored); this repo's suite covers the guard's error path (di.toml absent) and the +> file-exists-before-delegation ordering. ## Design notes & open gaps @@ -155,8 +192,16 @@ nothing. `"X"$"gg"` yields `0x00`, both non-null — so a bad value would slip past a null check and silently corrupt config. `parsefailed` handles these two explicitly (boolean must be `"0"`/`"1"`; byte must be even-length hex); anything else is rejected and logged, never - written. A string-typed (`10h`) setting still can't be overridden (its per-element char - cast is rejected as non-parsing) — this fails safe and is a rare edge. + written. +- **String-typed (`10h`) settings are overridable — as text, no parse.** A char-string + setting's override value *is* already a string, so `applyoverride` takes it as-is rather + than casting it (any string is valid; there is no parse-failure case). This matters because + TOML has no symbol type, so a `.toml`-origin setting like `dir = ":appdb"` or + `loglevel = "info"` comes back as a string — without this, such settings could not be + command-line-overridden at all, whereas their `.q`-origin symbol equivalents (`` dir:`:appdb ``) + could. di.config stays policy-free: it preserves whatever type the setting already had + (string in, string out; symbol in, symbol out), leaving symbol-vs-string normalisation to + the consumer as everywhere else. - **The sentinel merge key.** `cascade`'s accumulator is seeded with a sentinel `` (enlist`)!enlist(::) `` key so its value list stays general from the first merge — a same-typed value list coalesces to a typed vector that `,:` then refuses to widen (e.g. diff --git a/di/config/config.q b/di/config/config.q index 0a5df8df..9a1b8b07 100644 --- a/di/config/config.q +++ b/di/config/config.q @@ -38,18 +38,32 @@ init:{[deps] / --- settings cascade resolution (.q + .toml -> flat dict) --- +requiretoml:{[path] + / internal - a .toml settings file (path) exists and must be parsed, which needs the di.toml + / module. di.toml is a SOFT dependency: it is not required to be in the repo/on QPATH and the + / .q-only path never touches it - it is needed ONLY once a .toml file actually appears. guard + / the load so a missing di.toml gives a clear, actionable error naming the file, instead of the + / cryptic `notfound: di.toml` that a bare use would raise. returns the resolved di.toml module. + :@[use;`di.toml;{[p;e] + msg:"di.config: cannot parse TOML file '",p,"' - the di.toml module was not found on QPATH; "; + msg,:"a di.toml module is required to parse .toml settings (underlying: ",e,")"; + 'msg + }[path;]]; + }; + parsefile:{[path] - / parse ONE settings file into a flat dict. a .toml path is delegated to di.toml, loaded - / lazily (only when a .toml file actually appears, so the .q-only path never needs it). - / a .q file is plain `name:value` lines: split on the first ":", the RHS run through value - / (so `:hdb, `trade`quote, `symbol$() all work); blank and "/"-comment lines are skipped. - / a missing file of either extension contributes nothing (empty dict), so the cascade needs - / no presence checks. existence is checked BEFORE the .toml delegation so a missing .toml - / tier never forces di.toml to load - di.toml is pulled in only when a .toml file actually - / exists (di.toml lives only in the PoC today; the .q-only path needs it never). + / parse ONE settings file into a flat dict. a .toml path is delegated to di.toml (via the + / requiretoml guard), loaded lazily - only when a .toml file actually appears, so the .q-only + / path never needs it. a .q file is plain `name:value` lines: split on the first ":", the RHS + / run through value (so `:hdb, `trade`quote, `symbol$() all work); blank and "/"-comment lines + / are skipped. a missing file of either extension contributes nothing (empty dict), so the + / cascade needs no presence checks. order matters: existence is checked FIRST, so a missing + / .toml tier never triggers the di.toml requirement; only a .toml file that actually EXISTS + / requires di.toml - if it is absent, requiretoml signals a clear error (di.toml lives only in + / the PoC today; the .q-only path needs it never). fsym:`$":",path; if[0=count key fsym; :()!()]; - if[path like "*.toml"; :(use`di.toml)[`parsefile] path]; + if[path like "*.toml"; :(requiretoml[path])[`parsefile] path]; lines:read0 fsym; lines:lines where 0.q within a tier) true,0,0,q,"100~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdbnofile])`diconfiga",1,1,within a name the later (app) root overrides the builtin root true,0,0,q,"10~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdbnofile])`diconfigb",1,1,name-major: builtin proctype beats app default (would be 5 under root-major) @@ -35,6 +41,8 @@ true,0,0,q,"any (exec msg from logtab) like ""*did not parse*""",1,1,rejected bo true,0,0,q,"0xff~(cfg.overrideconfig[basecfg;enlist[`byte]!enlist ""ff""])`byte",1,1,valid hex byte parsed and applied true,0,0,q,"0x01~(cfg.overrideconfig[basecfg;enlist[`byte]!enlist ""gg""])`byte",1,1,invalid hex byte is rejected (no null - must not slip through as 0x00) true,0,0,q,"`xx`yy~(cfg.overrideconfig[basecfg;enlist[`syms]!enlist (""xx"";""yy"")])`syms",1,1,symbol list parsed and set from strings +true,0,0,q,""":newdb""~(cfg.overrideconfig[basecfg;enlist[`str]!enlist "":newdb""])`str",1,1,a string-typed setting is overridden as-is (this is what makes .toml string settings overridable) +true,0,0,q,"10h~type (cfg.overrideconfig[basecfg;enlist[`str]!enlist "":newdb""])`str",1,1,string override preserves the string (10h) type true,0,0,q,"not `nope in key cfg.overrideconfig[basecfg;enlist[`nope]!enlist enlist""1""]",1,1,an unknown setting is not added to the config dict true,0,0,q,"(key basecfg)~key cfg.overrideconfig[basecfg;enlist[`nope]!enlist enlist""1""]",1,1,an unknown setting leaves the config keys unchanged true,0,0,q,"any (exec msg from logtab) like ""*unknown setting*""",1,1,unknown setting logged