feat(reconcile): add a MetaSchema kind for declarative metaschema management - #1882
Conversation
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe PR adds ChangesMetaSchema support
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: 🟡 Moderate · up to The new declarative metaschema flow can currently export configurations that reconciliation rejects and accept schemas that may fail during metadata validation, causing failed plans or applies and runtime validation errors. Merge should wait until export and runtime schema validation use consistent rules. Possibly related PRs
🚥 Pre-merge checks | ✅ 2✅ Passed checks (2 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Coverage Report for CI Build 31779017944Warning Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes. Coverage increased (+0.2%) to 48.458%Details
Uncovered Changes
Coverage Regressions1 previously-covered line in 1 file lost coverage.
Coverage Stats
💛 - Coveralls |
… match names case-insensitively
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 4e656a94-6d75-48b5-97c2-96e085fbc34c
📒 Files selected for processing (15)
cmd/reconcile.gocore/metaschema/defaults.gocore/metaschema/defaults_test.gocore/metaschema/metaschemas/group.jsoncore/metaschema/metaschemas/org.jsoncore/metaschema/metaschemas/prospect.jsoncore/metaschema/metaschemas/role.jsoncore/metaschema/metaschemas/user.jsondocs/content/docs/reconcile.mdxdocs/rfcs/0001-declarative-reconcile.mdinternal/reconcile/metaschema.gointernal/reconcile/metaschema_reconciler.gointernal/reconcile/metaschema_reconciler_test.gointernal/reconcile/metaschema_test.gointernal/store/postgres/metaschema_repository.go
| func validateSchemaDocument(schema string) error { | ||
| root, err := decodeJSON(schema) | ||
| if err != nil { | ||
| return fmt.Errorf("schema is not valid JSON: %w", err) | ||
| } | ||
| if _, ok := root.(map[string]any); !ok { | ||
| return fmt.Errorf("schema must be a JSON object") | ||
| } | ||
| return nil |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🌐 Web query:
For the repository version of github.com/xeipuuv/gojsonschema, does validating data against the schema {"type":123} return a schema-compilation error, and can schema validation be performed without resolving remote $ref targets?
💡 Result:
For the github.com/xeipuuv/gojsonschema repository, the behavior for validating against the schema {"type": 123} is as follows: 1. Schema-compilation error: No, this does not trigger a schema-compilation error by default [1]. The library treats the schema as a valid JSON object. During validation, the "type" keyword expects a string or an array of strings (per JSON Schema specifications). Because 123 is a number, the library will fail to correctly process the "type" constraint during the validation phase, typically resulting in a validation result indicating the data does not match, rather than a hard failure during the NewSchema or Compile compilation step [1]. If you enable meta-schema validation via the SchemaLoader (setting sl.Validate = true), the library may detect the schema itself is invalid against the JSON Schema meta-schema, which would then return an error during the compilation/loading phase [1]. 2. Resolving remote $ref targets: No, the library does not provide a native configuration option to disable the automatic resolution of remote $ref targets [2]. The implementation automatically attempts to fetch remote references (e.g., via http/https) when encountered [3]. Users have reported that this behavior is "baked in" and difficult to bypass without external workarounds, such as pre-loading all required schemas using AddSchema/AddSchemas to satisfy dependencies before they are fetched over the network [2][3].
Citations:
- 1: https://chromium.googlesource.com/external/github.com/xeipuuv/gojsonschema/+/065759f9c3d72653843069184bab4f5622926edf/README.md
- 2: Enable use of local schema in favor of fetch via http xeipuuv/gojsonschema#197
- 3: https://chromium.googlesource.com/external/github.com/xeipuuv/gojsonschema/+/refs/tags/v1.0/README.md
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- candidate files ---'
git ls-files | grep -E '(^|/)(metaschema\.go|service\.go|go\.mod|go\.sum)$' || true
printf '%s\n' '--- relevant symbols and calls ---'
rg -n -C 5 'validateSchemaDocument|gojsonschema|SchemaLoader|Validate|exportMetaSchemas|MetaSchemaReconciler' internal core go.mod go.sum 2>/dev/null || true
printf '%s\n' '--- targeted source excerpts ---'
for f in internal/reconcile/metaschema.go core/metaschema/service.go go.mod; do
if [ -f "$f" ]; then
echo "### $f"
nl -ba "$f" | sed -n '1,180p'
fi
doneRepository: raystack/frontier
Length of output: 50374
🏁 Script executed:
#!/bin/bash
set -eu
for f in internal/reconcile/metaschema.go internal/reconcile/metaschema_reconciler.go core/metaschema/service.go core/metaschema/metaschema.go internal/reconcile/metaschema_test.go internal/reconcile/metaschema_reconciler_test.go go.mod; do
if [ -f "$f" ]; then
echo "### $f"
wc -l "$f"
nl -ba "$f" | sed -n '1,220p'
fi
done
echo '### gojsonschema references'
rg -n -C 8 'gojsonschema|SchemaLoader|NewSchema|Validate\(' core internal --glob '*.go' | head -n 300
echo '### local module source availability'
go env GOMODCACHE GOPATH 2>/dev/null || true
find "${GOMODCACHE:-/nonexistent}" -maxdepth 5 -type d -path '*gojsonschema*' -print 2>/dev/null | head -n 20Repository: raystack/frontier
Length of output: 26225
🏁 Script executed:
#!/bin/bash
set -eu
for spec in \
'internal/reconcile/metaschema.go:1,210' \
'internal/reconcile/metaschema_reconciler.go:1,112' \
'core/metaschema/service.go:1,117' \
'core/metaschema/metaschema.go:1,24' \
'internal/reconcile/metaschema_test.go:1,251' \
'internal/reconcile/metaschema_reconciler_test.go:1,146'; do
file=${spec%:*}
range=${spec#*:}
echo "### $file"
sed -n "${range}p" "$file"
done
echo '### direct gojsonschema imports and calls'
rg -n -C 10 'github.com/xeipuuv/gojsonschema|gojsonschema\.' --glob '*.go' . || true
mod=/home/jailuser/go/pkg/mod/github.com/xeipuuv/gojsonschema@v1.2.0
echo "### gojsonschema v1.2.0 files"
find "$mod" -maxdepth 1 -type f -printf '%f\n' | sort
echo '### loader/compiler/validation implementation'
rg -n -C 12 'type SchemaLoader|Validate|NewSchema|Compile|validateSchema|typeSchema|remote|ReferenceLoader' "$mod" --glob '*.go' | head -n 500Repository: raystack/frontier
Length of output: 50373
🏁 Script executed:
#!/bin/bash
set -eu
mod=/home/jailuser/go/pkg/mod/github.com/xeipuuv/gojsonschema@v1.2.0
echo '### SchemaLoader and Validate defaults'
sed -n '1,240p' "$mod/schemaLoader.go"
sed -n '1,260p' "$mod/schema.go"
echo '### type keyword handling'
rg -n -C 15 'Type|type.*keyword|NewType|parseType|TypeError|expected.*type' "$mod" --glob '*.go' | head -n 300
echo '### remote reference loading'
sed -n '130,240p' "$mod/schemaPool.go"
rg -n -C 15 'func \(.*\) LoadJSON|http.Get|Get\(|NewStringLoader|NewSchemaLoader|Validate =' "$mod" --glob '*.go' | head -n 300
echo '### isolated runtime probe'
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cat > "$tmp/go.mod" <<EOF
module isolated-probe
go 1.20
require github.com/xeipuuv/gojsonschema v1.2.0
EOF
cat > "$tmp/main.go" <<'EOF'
package main
import (
"fmt"
"github.com/xeipuuv/gojsonschema"
)
func main() {
schemas := []string{
`{"type":123}`,
`{"type":"object"}`,
`{"$ref":"https://127.0.0.1:1/unreachable.json"}`,
}
for _, raw := range schemas {
result, err := gojsonschema.Validate(
gojsonschema.NewStringLoader(raw),
gojsonschema.NewGoLoader(map[string]any{}),
)
fmt.Printf("schema=%s\nerr=%v\n", raw, err)
if result != nil {
fmt.Printf("valid=%v\n", result.Valid())
}
}
}
EOF
(
cd "$tmp"
GOPROXY=off GOSUMDB=off go run .
)Repository: raystack/frontier
Length of output: 50373
🏁 Script executed:
#!/bin/bash
set -u
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cat > "$tmp/go.mod" <<'EOF'
module isolated-probe
go 1.20
require github.com/xeipuuv/gojsonschema v1.2.0
EOF
cat > "$tmp/main.go" <<'EOF'
package main
import (
"fmt"
"github.com/xeipuuv/gojsonschema"
)
func main() {
schemas := []string{
`{"type":123}`,
`{"type":["object",123]}`,
`{"required":123}`,
`{"properties":123}`,
`{"items":123}`,
`{"minimum":"x"}`,
`{"$ref":"https://127.0.0.1:1/unreachable.json"}`,
`{"type":"object"}`,
}
for _, raw := range schemas {
result, err := gojsonschema.Validate(
gojsonschema.NewStringLoader(raw),
gojsonschema.NewGoLoader(map[string]any{}),
)
fmt.Printf("schema=%s\nerr=%v\n", raw, err)
if result != nil {
fmt.Printf("valid=%v errors=%d\n", result.Valid(), len(result.Errors()))
}
}
}
EOF
cd "$tmp"
GOPROXY=off GOSUMDB=off GOFLAGS=-mod=mod go run . 2>&1Repository: raystack/frontier
Length of output: 392
🏁 Script executed:
#!/bin/bash
set -u
echo '### cached dependency versions'
find /home/jailuser/go/pkg/mod/github.com/xeipuuv -maxdepth 1 -mindepth 1 -type d -printf '%f\n' | sort
find /home/jailuser/go/pkg/mod/cache/download/github.com/xeipuuv -type f -name '*.mod' -o -name '*.ziphash' 2>/dev/null | sort | head -n 100
mod=/home/jailuser/go/pkg/mod/github.com/xeipuuv/gojsonschema@v1.2.0
echo '### NewStringLoader and loader factories'
rg -n -C 12 'func NewStringLoader|type stringJSONLoader|LoaderFactory|func \(.*\) LoaderFactory' "$mod" --glob '*.go'
echo '### exact malformed-keyword branches'
rg -n -C 20 'KEY_TYPE|KEY_REQUIRED|KEY_PROPERTIES|KEY_ITEMS|KEY_MINIMUM|KEY_MAXIMUM' "$mod/schema.go" | head -n 500
echo '### compile a local-replacement probe when cached modules permit it'
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
cat > "$tmp/go.mod" <<'EOF'
module isolated-probe
go 1.20
require (
github.com/xeipuuv/gojsonschema v1.2.0
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415
)
replace github.com/xeipuuv/gojsonschema => /home/jailuser/go/pkg/mod/github.com/xeipuuv/gojsonschema@v1.2.0
replace github.com/xeipuuv/gojsonpointer => /home/jailuser/go/pkg/mod/github.com/xeipuuv/gojsonpointer@v0.0.0-20190905194746-02993c407bfb
replace github.com/xeipuuv/gojsonreference => /home/jailuser/go/pkg/mod/github.com/xeipuuv/gojsonreference@v0.0.0-20180127040603-bd5ef7bd5415
EOF
cat > "$tmp/main.go" <<'EOF'
package main
import (
"fmt"
"github.com/xeipuuv/gojsonschema"
)
func main() {
for _, raw := range []string{
`{"type":123}`,
`{"type":["object",123]}`,
`{"required":123}`,
`{"properties":123}`,
`{"items":123}`,
`{"minimum":"x"}`,
`{"$ref":"https://127.0.0.1:1/unreachable.json"}`,
} {
result, err := gojsonschema.Validate(
gojsonschema.NewStringLoader(raw),
gojsonschema.NewGoLoader(map[string]any{}),
)
fmt.Printf("schema=%s\nerr=%v\n", raw, err)
if result != nil {
fmt.Printf("valid=%v errors=%d\n", result.Valid(), len(result.Errors()))
}
}
}
EOF
(cd "$tmp" && GOPROXY=off GOSUMDB=off GOFLAGS=-mod=mod go run .) 2>&1Repository: raystack/frontier
Length of output: 44934
🏁 Script executed:
#!/bin/bash
set -eu
mod=/home/jailuser/go/pkg/mod/github.com/xeipuuv/gojsonschema@v1.2.0
echo '### public Validate wrapper'
rg -n -C 20 '^func Validate|^func \(.*\) Validate' "$mod" --glob '*.go'
echo '### HTTP reference loader'
sed -n '132,225p' "$mod/jsonLoader.go"
echo '### reference parsing and remote resolution'
sed -n '48,125p' "$mod/schemaPool.go"
sed -n '180,225p' "$mod/schemaPool.go"Repository: raystack/frontier
Length of output: 11302
Reject schemas that gojsonschema cannot compile before planning updates.
validateSchemaDocument accepts {"type":123}, but gojsonschema.Validate returns a schema-compilation error for this input. Metadata validation can then fail for that entity type. Use the same gojsonschema compilation rules during reconcile. Define how remote $ref values are handled because NewStringLoader resolves uncached references over HTTP.
| curCanon, err := canonicalJSON(cur.Schema) | ||
| if err != nil { | ||
| // A stored schema that is not valid JSON still round-trips as its own | ||
| // literal, so emit it rather than dropping it. | ||
| specs = append(specs, MetaSchemaSpec{Name: name, Schema: cur.Schema}) | ||
| continue |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Do not export a spec that reconcile rejects.
When a stored schema is invalid JSON, this branch emits it verbatim. MetaSchemaReconciler.Validate rejects that exported value before reconcile starts. Therefore, frontier export metaschema can produce a document that its own reconcile flow cannot consume.
Return an export error with the metaschema name for invalid stored JSON, or define an exported representation that validation accepts.
Proposed fix
curCanon, err := canonicalJSON(cur.Schema)
if err != nil {
- // A stored schema that is not valid JSON still round-trips as its own
- // literal, so emit it rather than dropping it.
- specs = append(specs, MetaSchemaSpec{Name: name, Schema: cur.Schema})
- continue
+ return nil, fmt.Errorf("metaschema %q contains invalid JSON: %w", name, err)
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| curCanon, err := canonicalJSON(cur.Schema) | |
| if err != nil { | |
| // A stored schema that is not valid JSON still round-trips as its own | |
| // literal, so emit it rather than dropping it. | |
| specs = append(specs, MetaSchemaSpec{Name: name, Schema: cur.Schema}) | |
| continue | |
| curCanon, err := canonicalJSON(cur.Schema) | |
| if err != nil { | |
| return nil, fmt.Errorf("metaschema %q contains invalid JSON: %w", name, err) | |
| } |
2 participants
Summary
This adds a
MetaSchemakind to thefrontier reconcileflow. It manages the built-in entity metaschemas, the JSON schemas that validate themetadatafield on users, organizations, groups, roles, and prospects, as declarative files. Before this, the only way to change a metaschema was a hand-run API call, with no file, no plan, and no record of what should exist.Metaschemas are managed as values, mirroring predefined roles: the file sets the JSON schema for a built-in, and a built-in left out of the file resets to its shipped default. There is no delete.
Changes
core/metaschema.Defaults, one source for both the server seeding and the reconcile kind. Adding a metaschema for a new resource later is then a single edit.internal/reconcile/metaschema.go.MetaSchemaReconciler.MetaSchemakind incmd/reconcile.go, which enables bothfrontier reconcileandfrontier export metaschema.Technical Details
core/metaschema.Defaultskeeps it to one source.FrontierService, andUpdateMetaSchemakeys by UUID, so the reconciler resolves name to id fromListMetaSchemasfirst.MigrateDefaults, which now ranges over the shared map. Seeding behavior is unchanged.Merge ordering
This should merge after #1878 (the metaschema cache refresh). Without that refresh, a schema change made through the API reaches only the pod that handled it, so on a multi-pod server the reconcile plan and applies would not be consistent across pods. The RFC edit here removes the "blocked on per-pod cache" future-work bullet, and the guide notes that a change propagates within the cache refresh interval. Both assume #1878 is deployed.
Test Plan
go build ./...,go vet)go test -race(core/metaschema,internal/reconcile), covering the diff (set, reset, create), validation (unknown name, duplicate, invalid JSON), JSON normalization (whitespace and key order), and the export round-tripSQL Safety (touches
metaschema_repository.go)metaschema_repository.goisMigrateDefaultsranging over the sharedmetaschema.Defaultsmap instead of a local one. The placeholder,ToSQL(), andgoqu.Litems do not apply.