Skip to content

feat(benchmark): support structured prompt tags (value+category) and origin#716

Draft
RapidPoseidon wants to merge 2 commits into
mainfrom
feat(benchmark)/prompt-taggings-and-origin
Draft

feat(benchmark): support structured prompt tags (value+category) and origin#716
RapidPoseidon wants to merge 2 commits into
mainfrom
feat(benchmark)/prompt-taggings-and-origin

Conversation

@RapidPoseidon

@RapidPoseidon RapidPoseidon commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

What

Exposes structured prompt tags and a prompt origin in the Python SDK. A prompt's tags are now a list whose items are each either a plain str or a Tag(value, category=None) — you can mix both. Plain strings keep working.

SDK leg of a 3-part full-stack feature:

  • Backend: RapidataAI/rapidata-backend#4879
  • Frontend: RapidataAI/app-frontend#2834

Contract (as finalized by the parent task)

  • High-level API is a str | Tag union: add_prompts(tags=...) / create_new_benchmark(tags=...) accept a list of str, a list of Tag, or a mix. Bare strings are wrapped to Tag(value, category=None) internally.
  • Wire is clean Tag[] objects — no union type in the generated model, no taggings field, no deprecated string[]. origin is a separate additive { value } field.
  • Read-back: .tags stays list[list[str]] (values, back-compat); new .structured_tags -> list[list[Tag]] for categories; new .origins. Reads defensively wrap a bare string as an uncategorized tag so an older backend still parses during rollout.

Changes

Generated client (src/rapidata/api_client/models/)

  • New sub-models PromptTagging (value, optional category — the wire Tag) and PromptOrigin (value), in the generator's LazyValidatedModel style.
  • tagsList[PromptTagging] and additive origin on the create input (CreatePromptForBenchmarkEndpointInput) and the read model actually returned by the prompts endpoint (GetPromptsByBenchmarkEndpointOutput).
  • Hand-edited rather than regenerated: the regen script pulls specs from prod, where these fields aren't deployed yet, so a regen today would drop them. A normal regen reproduces this shape once the backend spec lands.
  • ⚠️ The seed referenced PromptByBenchmarkResult, an orphan not returned by any endpoint — the live read model is GetPromptsByBenchmarkEndpointOutput. The orphan is left untouched.

High-level surface

  • New user-facing dataclasses Tag(value, category=None) and Origin(value), exported from rapidata, rapidata.rapidata_client, and rapidata.types.
  • add_prompts / create_new_benchmark gain the str | Tag tags union + origins; bare strings/str origins are coerced. BenchmarkPrompt carries structured tags/origin; the uploader sends clean Tag objects.
  • Read accessors: .tags (values), new .structured_tags, new .origins.

Version & docs

  • 3.17.23.18.0. docs/mri_advanced.md updated.

Changelog

Prompt tags now accept str | Tag — a Tag carries a value and optional category; plain strings still work and are auto-wrapped. Prompts can also carry an additive origin. Read categories via benchmark.structured_tags and origins via benchmark.origins; benchmark.tags remains the values-only view.

Deferred to the post-deploy regen pass

Two agreed-contract items — update-editable prompts and server-side filter/sort — are now finalized in backend #4879 but are not yet in the deployed spec, so they're intentionally left for the regen pass (triggered once #4879 merges & deploys) rather than hand-edited into the generated client and thrown away by that regen. Documented final shapes, captured here so nothing is lost:

  • Update endpoint: prompt update accepts tags: Tag[] + origin { value } (same shape as create). Will add a high-level update_prompt(...) surface on top of the regenerated client.
  • Filter (prompts listing): tag_values[in], tag_categories[in], origin_value.
  • Sort: id, created_at, origin_value only — tag value/category are filter-only.

Regen-pass checklist: regenerate from the live spec → generate the backend's real Tag model → delete the prompt_tagging.py placeholdergrep -ri tagging src/ returns zero → wire the update + filter/sort high-level surface.

Validation

  • python -c "from rapidata import RapidataClient; print('OK')" → OK
  • pyright src/rapidata/rapidata_client → 0 errors
  • black clean on changed rapidata_client files
  • Round-trip + add_prompts mixed-str/Tag coercion, cache, and validation exercised locally (incl. old-backend bare-string reads)

🔗 Session: https://poseidon.rapidata.internal/chat/node-ae73dd05

RapidPoseidon and others added 2 commits July 24, 2026 12:36
Prompt tags are now structured: each tag carries a value and an optional
category, and each prompt can record an origin. New user-facing Tag and
Origin types are added and exported at the top level.

RapidataBenchmark.add_prompts and RapidataBenchmarkManager.create_new_benchmark
gain taggings and origins parameters; the read-back surface gains taggings and
origins accessors alongside the existing flat, values-only tags.

Plain-string tags remain accepted (mapped to Tag(value, category=None)) and are
still sent to the backend for compatibility during rollout.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: manuel <49093103+manueltollis@users.noreply.github.com>
Fold in the final contract from the parent task: there is no separate
taggings field. A prompt's tags are a single list whose items are each either
a plain string or a Tag(value, category?). Bare strings are wrapped into
Tag(value, category=None) internally and sent to the backend as clean Tag
objects (no union type on the wire, no deprecated string[]).

- add_prompts / create_new_benchmark: tags param now accepts list[list[str | Tag]];
  the separate taggings param is removed. origins unchanged.
- Generated create-input and read models: tags is List[Tag] (PromptTagging);
  reads defensively wrap a bare string as an uncategorized tag during rollout.
- Read-back: .tags stays list[list[str]] (values, back-compat); structured tags
  are exposed via the new .structured_tags accessor; origins via .origins.
- Reverted the out-of-scope update-input and orphan read-model edits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: manuel <49093103+manueltollis@users.noreply.github.com>
@RapidPoseidon RapidPoseidon changed the title feat(benchmark): expose structured prompt taggings and origin feat(benchmark): support structured prompt tags (value+category) and origin Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant