docs: add proposed server-side policy evaluation tutorial - #338
Draft
pbeckham wants to merge 41 commits into
Draft
docs: add proposed server-side policy evaluation tutorial#338pbeckham wants to merge 41 commits into
pbeckham wants to merge 41 commits into
Conversation
Contributor
|
Claude finished @pbeckham's task in 3m 50s —— View job Reviewing PR #338 — Server-side policy evaluation tutorial
Review summaryThorough, well-structured design-preview tutorial that has clearly benefited from 41 commits of iterative review. The page is honest about its preview status (banner, callouts, and consistent "proposed" framing), and the writing is direct and precise. Most findings from earlier review rounds have been addressed. Three items remain worth noting. FindingsImprovement1. Non-existent schema URLs in Three schema URLs are referenced as downloadable resources and used in
These don't exist in this repo or on the live site. A reader following the type-checking or capabilities-checking sections will get 404s. Since the page is a design preview, this is expected — but the (Previously flagged by a reviewer; acknowledged as deferred.) 2. Relative link in The mockups README contains Suggestion3. Page length and Diátaxis classification — At ~1,156 lines this is very dense for a tutorial. The "Start here" section and audience-based This is understandable for a design preview seeking feedback on the whole surface — consolidating everything makes review easier. When the feature ships and the page is no longer a preview, consider splitting the Rego authoring contract and the manage/audit sections into separate reference or how-to pages, leaving the tutorial focused on the core loop. What looks good
Overall: 0 critical, 2 improvement, 1 suggestion. The page is in good shape for a design preview. The outstanding items are minor and can be addressed when the feature ships. |
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
pbeckham
commented
Jul 29, 2026
pbeckham
commented
Jul 29, 2026
pbeckham
commented
Jul 29, 2026
Thoughts
AI generation tells
|
Review flagged several phrases as reading like generated marketing copy rather than documentation. Each is replaced with the underlying fact. - "records the decision itself — one it can vouch for" states the claim instead of the mechanism; say what is recorded alongside the decision. Same for the comparison table's "a decision Kosli computed and can vouch for". - "The whole loop in two commands" / "That is the feature:" sells rather than informs. - "worth reading before you send feedback, so you don't spend it on something already deferred" — the Feedback section already covers this. - "Note that --context is doing real work here:" editorialises about the sentence that follows it. - "core loop" was a drafting label that meant nothing to readers; three uses replaced with what the section actually covers. - "changes what a decision means" asserted significance without saying what changed. It is who computed the decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--context defaulted to the trail named by --flow / --trail. Review pointed out that the note defending the default was describing a footgun rather than removing one: with the default in place, a command that reads and writes different trails looks almost identical to one that does not. Makes --context required, the same decision already taken for --trail and for the same reason — this command does not infer trails. Every example on the page already passed it explicitly, so no example changes. This also removes the warning that existed only to manage the hazard the default created; three lines of plain description replace it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The --context cell was a ~700 character paragraph inside a table, which does not read on narrow viewports. The cell now states what the flag is and that it is required; the reference-type syntax, repeat semantics, and room for future context types move to prose under the table. Also drops a "by default the decision is recorded on the same trail whose facts you evaluate" left over from when --context had a default. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
"Result" names what an evaluation produces better than "output", and pairs with the input schema: input schema in, result schema out. Renames the seven uses that mean the policy's return value, including the "Output" section heading and "output schema" in the validation list. The four that mean something else are unchanged: the --output flag on kosli list decisions and kosli get evaluation-context, OPA's print/trace output, and the draft banner's "commands, flags, and output shown". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A policy can be published as a bundle of several Rego files, but two places still assumed a single file. "Re-run a decision offline" exported the policy as one policy.rego and replayed it with --data, which cannot round-trip a multi-file bundle. The export is now a policy/ directory laid out as published, replayed with --bundle so the same command works for one file or many. Says that the layout is preserved so the digest on the decision can be verified, not just the outcome re-run. The "composable policies" scope bullet read as a description of a bundle, which is what prompted the question. It now names the distinction: composing separately published policies is out of scope, one policy spanning several files is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The publish step offered a directory of Rego files, but every other mention of --policy said "file" only — so a reader who took up that offer had no documented way to validate or ad-hoc-evaluate what they wrote. --policy now consistently takes a single .rego file or a directory holding one policy's bundle, on create, validate and evaluate. States that a directory is one policy's bundle, not a collection of separate policies, which "a directory of them" left ambiguous. Renames "Evaluate a local policy file" to "Evaluate a local policy" since it is no longer file-only, and updates the one link to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The validation list was written for a single file, leaving it unclear whether a multi-file bundle is checked as one unit. States that it is, without specifying the mechanism, and that publishing runs the same checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two terms a bundle author needs that the page left implicit. Tests: the page states twice that tests are stored but never executed, which depends on Kosli telling a test file from a policy file. Names the convention (*_test.rego) so a misnamed file does not silently become evaluated policy. Roots: previously appeared only inside the policy-tab screenshot alt text, never defined. Glossed where the tab's contents are described. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two {/* TODO */} comments did not render, so the open questions they held
were visible only in source. The surrounding prose reads complete without
them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follows the earlier pass on flagged phrasing with a sweep for the same patterns elsewhere. Each cut removes a verdict about a fact, not the fact. - "the strongest form of the audit story, because it doesn't ask anyone to trust Kosli to check Kosli" — the sentence after it states the point. - "nobody has to trust that version 3 is still what version 3 was" — the preceding clause already says a digest proves the bytes. - "a summary with no attachment behind it is just another self-reported number" — a closer, after the reason was already given. - "what makes a server-side decision worth more than a self-reported one" — keeps who can do what, drops the judgement. - "This is what makes publishing safe to run" — leads with the fact: publishing is idempotent. - The control owner's question was framed twice, ~80 lines apart, in the same construction. Keeps the one in the review workflow, where it is actionable. - Drops the trade-booking analogy from the two-timestamp explanation, and a filler "Note that". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Evaluation is asynchronous by default, but the page never showed what the default returns and gave no way to observe it — every example passed --sync or --assert. Review asked what the behaviour is without --sync and how a customer checks status. The evaluation context answers both, since it exists from the moment an evaluation is queued and already holds the requested and recorded times. - Shows what the asynchronous command returns: the context that will carry the outcome. - Adds "Check an asynchronous evaluation": kosli get evaluation-context reports STATUS (queued, completed, failed) and names the decision it produced. Also documents --show-input for returning the policy input that was evaluated, matching the flag of the same name on the client-side evaluate commands so jq '.input' and kosli evaluate input keep working. - States that a typed error lands on the context, so a broken policy is distinguishable from a non-compliant artifact without --sync. Corrects the fixture export from --output fixture to --fixture. In the CLI, --output selects a format ([table, json, markdown]) and never content, so overloading it here contradicted every other kosli get. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review read "Be deliberate about which flows may satisfy a production control." as alarmist for a design preview of a feature nobody can use yet. The sentence added a directive tone without adding information — the rest of the callout already states the gap and what to do about it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review asked for the request timestamp to be stated explicitly as the query time for trail data, and for a worked example to make the model concrete. States the mechanism: Kosli fixes the trail moment (which attestations are in scope) at the requested/enqueue time, then queries and serializes that data only when the evaluation executes — always as of the fixed timestamp, not as of whatever the trail contains by then. This is what makes a queued evaluation deterministic regardless of queue delay, and is why no drift-detection or invalidation mechanism is needed for a fact that lands mid-queue: the query never looks at "now," only at the timestamp fixed at enqueue. Replaces the abstract "facts arrive mid-flight" paragraph with a worked example using the same evaluation context (ec_7f3a91c2) and timestamps already introduced in "Check an asynchronous evaluation", so the same example carries through both sections instead of introducing new arbitrary numbers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review pointed out that recommending --user-data for a summarized attachment may depend on attestation type — it's probably part of the main attestation_data for a custom attestation type schema. Checked against the CLI reference: --attestation-data exists only on kosli attest custom, documented as the custom attestation data validated against the type's schema. --user-data is documented as "additional data" on every attestation command, never the schema-validated payload. So the earlier guidance to always use --user-data was incomplete: for a custom attestation type you define, --attestation-data is the schema- validated field and the better fit for a recurring summary shape (finding counts, severities, package IDs). --user-data remains correct for built-in types (generic, snyk, sonar, and others) that have no schema field for the summary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review asked what --dry-run's "evaluate without recording" actually adds once every other way to test a policy is accounted for: - Rego correctness and schema checks: kosli validate policy — no trail data touched, nothing recorded. - Behavioral correctness: Rego unit tests / opa test, including regression fixtures already exported from past decisions. - Behavior against real trail data: "Evaluate a local policy" already covers this. It records a decision, but the page already says to treat that as expected — the Warning right below it states these decisions cite an unpublished source, should be treated as iterating rather than audit evidence, and that the beta is soliciting feedback on marking or restricting them. Once that one real evaluation exists, its exported fixture can be re-run locally with opa eval against any number of different candidate policies, for free, with nothing further recorded. --dry-run also collided with the CLI-wide convention (--dry-run means "no data sent, exits 0 regardless of errors" everywhere else), which this turn's discussion had been trying to fix by renaming. Removing the overlapping capability is simpler than renaming it. Drops the --dry-run flag-table row, the "Run an ad-hoc evaluation" section, and the dangling --dry-run mention at the end of "Evaluate a local policy". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Separate evaluating a policy from claiming a control is satisfied. Omitting --control runs the evaluation and records the result without writing a decision, so a policy can be checked against real facts without adding to the set of decisions an auditor samples. The destination flags become required-together-with --control rather than unconditionally required. Rename evaluation-context to evaluation, since it is now the primary record of a run rather than a byproduct of a decision. Document that an evaluation reaches a terminal status once and is never revised, that clients branch on STATUS rather than on field presence, and add --wait so the async path does not require a hand-written poll loop. Abandoned evaluations terminate with a typed error instead of sitting in queued. Resolve three open questions the change makes answerable: local policies can no longer record decisions (publishing is the org-admin gate), built-in errors abort rather than evaluating to undefined, and print/trace output is discarded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
||
| ```shell | ||
| mkdir -p schemas | ||
| curl -so schemas/trail-v1.json https://docs.kosli.com/schemas/evaluation-input/trail/v1.json |
Contributor
There was a problem hiding this comment.
Improvement: These schema URLs (evaluation-input/trail/v1.json, trails/v1.json in the table above, and rego/capabilities/v1.json at line 677) don't exist yet — the curl commands will 404 on the preview site. Since the rest of the page is carefully framed as a design preview, consider adding "(not yet published)" after each URL in the table, or an <Info> note near the curl blocks, so readers who try to follow along hit an explanation rather than a silent failure.
| # Proposed Policies UI — wireframe mockups | ||
|
|
||
| Sources for the mockup images used by | ||
| [`tutorials/evaluate_policies_server_side.mdx`](../../tutorials/evaluate_policies_server_side.mdx). |
Contributor
There was a problem hiding this comment.
Improvement (minor): This is a relative link (../../tutorials/...). CLAUDE.md requires root-relative paths. Since this README isn't published by Mintlify it won't break the site, but it would break if the directory structure changes.
Suggested change
| [`tutorials/evaluate_policies_server_side.mdx`](../../tutorials/evaluate_policies_server_side.mdx). | |
| [`tutorials/evaluate_policies_server_side.mdx`](/tutorials/evaluate_policies_server_side). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Documentation-driven development for the server-side Kosli Evaluate project (kosli-dev/server#6267). Adds a tutorial describing the proposed V1
kosli evaluate policyflow, published so it can be shared internally and with design-partner customers for feedback before the feature is built.This is a design preview — nothing in it is runnable today. The page is marked as such throughout.
Scope (this draft)
Core happy path for a governance platform engineer:
allowmandatory).kosli evaluate policyruns server-side and records a decision against a control; synchronous.attestation_ids, policy version, params, timestamp; includes the T / T+1 / T+2 temporal-correctness case.An explicit out-of-scope-for-V1 section keeps reviewer feedback on target (managing policies, testing/validation, reporting/audit views, unhappy-path inspection, composable policies, non-trail targets, scheduled evals, risk measurement).
Design notes surfaced for feedback
kosli attest decisionandkosli evaluate trailflags.--compliantand evidence (--attachments/--user-data) are intentionally not flags — the server derives compliance and captures the policy/report/violations as output.kosli create policy, which collides with the existing environment-policy command — flagged inline as an open question.inputshape is borrowed from the client-side tutorial and may differ server-side.Draft because
Content describes proposed behavior and will change as the design firms up; opening as draft to gather review before it's merged/published.
🤖 Generated with Claude Code