feat(benchmark): support structured prompt tags (value+category) and origin#716
Draft
RapidPoseidon wants to merge 2 commits into
Draft
feat(benchmark): support structured prompt tags (value+category) and origin#716RapidPoseidon wants to merge 2 commits into
RapidPoseidon wants to merge 2 commits into
Conversation
Prompt tags are now structured: each tag carries a value and an optional category, and each prompt can record an origin. New user-facing Tag and Origin types are added and exported at the top level. RapidataBenchmark.add_prompts and RapidataBenchmarkManager.create_new_benchmark gain taggings and origins parameters; the read-back surface gains taggings and origins accessors alongside the existing flat, values-only tags. Plain-string tags remain accepted (mapped to Tag(value, category=None)) and are still sent to the backend for compatibility during rollout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: manuel <49093103+manueltollis@users.noreply.github.com>
Fold in the final contract from the parent task: there is no separate taggings field. A prompt's tags are a single list whose items are each either a plain string or a Tag(value, category?). Bare strings are wrapped into Tag(value, category=None) internally and sent to the backend as clean Tag objects (no union type on the wire, no deprecated string[]). - add_prompts / create_new_benchmark: tags param now accepts list[list[str | Tag]]; the separate taggings param is removed. origins unchanged. - Generated create-input and read models: tags is List[Tag] (PromptTagging); reads defensively wrap a bare string as an uncategorized tag during rollout. - Read-back: .tags stays list[list[str]] (values, back-compat); structured tags are exposed via the new .structured_tags accessor; origins via .origins. - Reverted the out-of-scope update-input and orphan read-model edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Co-Authored-By: manuel <49093103+manueltollis@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Exposes structured prompt tags and a prompt
originin the Python SDK. A prompt's tags are now a list whose items are each either a plainstror aTag(value, category=None)— you can mix both. Plain strings keep working.SDK leg of a 3-part full-stack feature:
Contract (as finalized by the parent task)
str | Tagunion:add_prompts(tags=...)/create_new_benchmark(tags=...)accept a list ofstr, a list ofTag, or a mix. Bare strings are wrapped toTag(value, category=None)internally.Tag[]objects — no union type in the generated model, notaggingsfield, no deprecatedstring[].originis a separate additive{ value }field..tagsstayslist[list[str]](values, back-compat); new.structured_tags -> list[list[Tag]]for categories; new.origins. Reads defensively wrap a bare string as an uncategorized tag so an older backend still parses during rollout.Changes
Generated client (
src/rapidata/api_client/models/)PromptTagging(value, optionalcategory— the wireTag) andPromptOrigin(value), in the generator'sLazyValidatedModelstyle.tags→List[PromptTagging]and additiveoriginon the create input (CreatePromptForBenchmarkEndpointInput) and the read model actually returned by the prompts endpoint (GetPromptsByBenchmarkEndpointOutput).PromptByBenchmarkResult, an orphan not returned by any endpoint — the live read model isGetPromptsByBenchmarkEndpointOutput. The orphan is left untouched.High-level surface
Tag(value, category=None)andOrigin(value), exported fromrapidata,rapidata.rapidata_client, andrapidata.types.add_prompts/create_new_benchmarkgain thestr | Tagtagsunion +origins; bare strings/strorigins are coerced.BenchmarkPromptcarries structuredtags/origin; the uploader sends cleanTagobjects..tags(values), new.structured_tags, new.origins.Version & docs
3.17.2→3.18.0.docs/mri_advanced.mdupdated.Changelog
Prompt tags now accept
str | Tag— aTagcarries avalueand optionalcategory; plain strings still work and are auto-wrapped. Prompts can also carry an additiveorigin. Read categories viabenchmark.structured_tagsand origins viabenchmark.origins;benchmark.tagsremains the values-only view.Deferred to the post-deploy regen pass
Two agreed-contract items — update-editable prompts and server-side filter/sort — are now finalized in backend #4879 but are not yet in the deployed spec, so they're intentionally left for the regen pass (triggered once #4879 merges & deploys) rather than hand-edited into the generated client and thrown away by that regen. Documented final shapes, captured here so nothing is lost:
tags: Tag[]+origin { value }(same shape as create). Will add a high-levelupdate_prompt(...)surface on top of the regenerated client.tag_values[in],tag_categories[in],origin_value.id,created_at,origin_valueonly — tag value/category are filter-only.Regen-pass checklist: regenerate from the live spec → generate the backend's real
Tagmodel → delete theprompt_tagging.pyplaceholder →grep -ri tagging src/returns zero → wire the update + filter/sort high-level surface.Validation
python -c "from rapidata import RapidataClient; print('OK')"→ OKpyright src/rapidata/rapidata_client→ 0 errorsblackclean on changedrapidata_clientfilesadd_promptsmixed-str/Tagcoercion, cache, and validation exercised locally (incl. old-backend bare-string reads)🔗 Session: https://poseidon.rapidata.internal/chat/node-ae73dd05