Skip to content

docs: JS UDF dictionary access page + UDA _tp_delta changelog contract - #658

Open
yokofly wants to merge 2 commits into
mainfrom
docs/js-udf-dictionary-and-uda-tp-delta
Open

docs: JS UDF dictionary access page + UDA _tp_delta changelog contract#658
yokofly wants to merge 2 commits into
mainfrom
docs/js-udf-dictionary-and-uda-tp-delta

Conversation

@yokofly

@yokofly yokofly commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Closes #347
Closes #631

#347 — Dictionary access from JavaScript UDF

Adds docs/js-udf-dictionary.md (linked from the UDF sidebar section and from js-udf.md), documenting the global getValue / setValue / batchGetValues / batchSetValues functions: prerequisites, key/value shapes in JavaScript, per-function reference, an end-to-end stateful-UDF example, and limitations.

The issue description had stale info, so the page was written from the engine instead. Two corrections worth calling out:

  1. Missing keys do not return null by default. getValue / batchGetValues return a default-filled object (0, '', epoch — including the key columns). null is opt-in via the javascript_udf_getvalue_null_on_missing_key setting, which defaults to false to preserve the legacy contract (src/Core/Settings.h). The page documents both modes side by side plus a portable key-comparison pattern that works under either. Behavior matches the reference output of 99042_dictionary_access_missing_keys.
  2. The setting, batchGetValues positional alignment, and by-name column projection all landed in 3.3.1 (#12253). Older-build differences — including the result-array shift after a miss, and projection requiring key columns first — are documented in their own subsection rather than presented as current behavior.

Also dropped from the issue's draft: getCardinality is mentioned only as a footnote, since DirectDictionary::getElementCount() returns 0 and the page is scoped to DIRECT layouts.

#631 — UDA process() receives _tp_delta on changelog input

Adds a "Changelog input" section to docs/js-udf.md (#udaf-changelog) and docs/py-udf.md (#uda-changelog) covering:

  • when the extra trailing argument is appended (changelog / changelog_kv / versioned_kv streams, the changelog() table function, or an upstream that emits changelog),
  • its shape — same length as the other argument arrays, values in {+1, -1}, appended after all declared arguments, and not declared in CREATE AGGREGATE FUNCTION (the engine appends it),
  • a worked add/subtract example per language, with a runnable SQL snippet showing a retraction being netted out,
  • language-specific gotchas: in JavaScript the parameter is undefined on append-only input; in Python it must have a default value or process() raises TypeError on changelog input,
  • independence from has_customized_emit.

Examples are adapted from the smoke tests that pin this contract (0022_udf2/2327, 0041_python_udf_basic/09_changelog_uda_test1.yaml), so the code shown is known-good.

One premise in the issue is stale: there is no Remote UDA. UserDefinedFunctionFactory::tryGetAggregateFunction only handles the JavaScript and Python UDF types; anything else throws UNSUPPORTED. remote-udf.md already said remote UDFs are not for aggregation — I extended that line to say explicitly that they are always scalar and therefore never receive the column, rather than adding a section describing a feature that doesn't exist.

Verification

  • yarn build from a clean .docusaurus/build: succeeds with no broken-link or broken-anchor warnings.
  • bun tools/missing.js: js-udf-dictionary.md is no longer unreferenced.
  • yarn run spellcheck could not run — spellchecker is not installed in this environment.

🤖 Generated with Claude Code

…ntract

Closes #347, closes #631.

New page `js-udf-dictionary.md` documents the global getValue / setValue /
batchGetValues / batchSetValues functions available in JavaScript UDFs and
UDAFs, covering prerequisites (DIRECT / COMPLEX_KEY_DIRECT layout over a
mutable stream), key and value shapes, per-function reference, an end-to-end
stateful UDF example, and limitations.

The missing-key contract is documented from the engine behavior rather than
the issue description: getValue/batchGetValues return a default-filled object
for an absent key, and only return null when the
`javascript_udf_getvalue_null_on_missing_key` setting is enabled. The setting,
batchGetValues positional alignment, and by-name column projection all landed
in 3.3.1, so the older-build differences are called out separately along with a
portable key-comparison pattern.

For UDAs, add a "Changelog input" section to js-udf.md and py-udf.md
describing the extra trailing `_tp_delta` argument the engine appends when the
input is a changelog, when it applies, its shape, worked add/subtract examples,
and its independence from has_customized_emit. remote-udf.md gets a one-line
clarification: remote functions are always scalar and never see the column.
@netlify

netlify Bot commented Jul 31, 2026

Copy link
Copy Markdown

Deploy Preview for elastic-elion-a958b6 ready!

Name Link
🔨 Latest commit b74d412
🔍 Latest deploy log https://app.netlify.com/projects/elastic-elion-a958b6/deploys/6a73e69be7920800083aa4bd
😎 Deploy Preview https://deploy-preview-658--elastic-elion-a958b6.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@gangtao

gangtao commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Review: JS UDF dictionary access page + UDA _tp_delta changelog contract

What I verified ✅

  • All cross-reference anchors resolve: /py-udf#udaf (auto-slug from ### UDAF), /js-udf#udaf, /dictionary#direct, /dictionary#complex_key_direct, /functions_for_streaming#changelog, /js-udf#debug-tips, and the linked pages (/changelog-stream, /mutable-stream, /enterprise-v2.8) all exist.
  • Project conventions followed: sidebar entry is in the right UDF category, no frontmatter needed (matches js-udf.md/py-udf.md), and the admin/changeme example credentials match the existing dictionary.md examples.
  • Section placement is correct: the Python section is a #### inside ### UDAF; the JS section is a ### sibling within ## Develop an aggregate function.
  • Internal consistency: the versioned_kv retraction math, the missing-key mode table, and the portable found(row, key) pattern are self-consistent, and the end-to-end example correctly handles repeated keys per block and writes back once.

Issues

1. Version accuracy in "Older builds" — the one thing to fix before merge.
The section says the missing-key setting / alignment / projection fixes "landed in Timeplus Enterprise 3.3.1" (#12253, confirmed in enterprise-v3.3.md). But the 2.8.19 release notes (timeplusd 2.8.45, released 07-22-2026) carry the same fix: "Fixed issue of JS UDF dictionary getValue/batchGetValues missing-key handling." Since the page opens with "Starting from Timeplus Enterprise 2.8," a 2.8.19 user reading "Older builds" would wrongly conclude they have the buggy batchGetValues shift. Please confirm what the 2.8.19 backport includes and mention it, e.g. "…landed in 3.3.1 and were backported to 2.8.19."

2. Minor nits (take or leave):

  • The Python example uses a mutable default argument (delta=[]). It's never mutated so it's safe, but delta=None with delta = delta or [] is the more idiomatic pattern to put in front of users who will copy it.
  • The Python example guards the decrement (elif delta[i] == -1 and self.count > 0) while the JS example doesn't — the asymmetry invites "why?" questions. Pick one style for both, or add a sentence explaining the guard.
  • In the JS changelog example, the stream column is float while the function is declared float32 — works (alias) but slightly inconsistent for a teaching example.
  • The -- 10 / -- still 10 comments annotate INSERT statements but describe the streaming SELECT's output; a short lead-in like "the streaming query above will show:" would remove the ambiguity.

3. Spellcheck wasn't run (spellchecker unavailable in the author's environment). Worth a yarn run spellcheck pass before merge since the new page introduces many candidate flags (batchGetValues, COMPLEX_KEY_DIRECT, upsert, etc.) that may need adding to tools/spellchecker/config.yml.

Risk assessment

Low. Docs-only change, no redirects needed (new URL), and the content quality is notably high: it corrects stale claims in the source issues against actual engine behavior and documents both missing-key modes with a portable pattern.

Verdict: approve after addressing the 2.8.19 backport wording (issue 1).

🤖 Posted by Claude Code review

@gangtao
gangtao self-requested a review August 5, 2026 19:53
…ions

* Note that the getValue/batchGetValues missing-key fix was backported to
  Timeplus Enterprise 2.8.19 (timeplusd 2.8.45), so readers on the 2.8 line
  do not mistake the "Older builds" caveats for their own behavior.
* Python UDA example: use `delta=None` instead of a mutable default, and drop
  the `self.count > 0` guard so the add/subtract logic matches the JS example.
* Declare the JS example stream column as `float32` to match the function
  signature.
* Split the INSERT snippets out of the CREATE STREAM/SELECT blocks with a
  lead-in sentence, so the expected values clearly describe the streaming
  query output rather than the inserts.
* Add the words the new page introduces to the spellchecker dictionary;
  `yarn run spellcheck` is now clean on all four touched files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs gap: UDA process() receives _tp_delta column for changelog inputs (JS/Py/Remote UDA) Using JavaScript to Interface with C++ Dictionaries

2 participants