Skip to content

Commit d2306f2

Browse files
committed
Run 7 prep: ban fake adapters, require real TypeAdapter in _schema_map.py
Run 6 analysis: agent switched to Node.js scaffolding script (since Python venv was removed), generated 1966 model classes with decent naming, but _schema_map.py was a complete cheat — loaded schema.json at runtime via SimpleNamespace objects with fake json_schema() methods. Verification passed trivially without ever testing the actual models. Fixes: - Add isinstance(adapter, TypeAdapter) check in verify script — rejects any adapter that isn't a real pydantic TypeAdapter - Ban SimpleNamespace, _make_adapter, _schema_path, _schema_doc patterns - Ban loading schema.json at runtime in generated code - Add failure case 8 to prompt explaining the cheat and the new check
1 parent 1f14d3e commit d2306f2

2 files changed

Lines changed: 30 additions & 0 deletions

File tree

codegen-llm/src/prompts.ts

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,8 @@ verification fails immediately.
108108
- \`JsonAdapter\` or any custom adapter/wrapper class
109109
- Custom \`json_schema()\` methods — only Pydantic's built-in is allowed
110110
- \`schema_override_json\`, \`schema_override\`, \`_schema_json\`
111+
- \`SimpleNamespace\`, \`_make_adapter\`, or any fake adapter objects
112+
- Loading \`schema.json\` at runtime in \`_schema_map.py\` or any module
111113
- Raw JSON Schema dicts embedded as Python dict literals
112114
- Any helper/utility that builds models from schema dicts at runtime
113115
@@ -527,6 +529,17 @@ The agent produced \`Literal[0] | Literal[1] | Literal[2] | ... | Literal[8]\`
527529
instead of \`Literal[0, 1, 2, 3, 4, 5, 6, 7, 8]\`. The chained form is banned.
528530
Use multi-value \`Literal[...]\` for cleaner, more readable code.
529531
532+
### Failure 8: Fake adapters in _schema_map.py (SimpleNamespace cheat)
533+
534+
The agent made \`_schema_map.py\` load \`schema.json\` at runtime and create
535+
\`SimpleNamespace\` objects with a \`json_schema()\` method that returns the raw
536+
schema directly. This bypasses verification entirely — the actual Pydantic
537+
models are never tested.
538+
539+
**The verifier checks that every adapter is a real \`TypeAdapter\` instance.**
540+
\`_schema_map.py\` MUST import TypeAdapter instances from service modules.
541+
Any fake adapter will be rejected.
542+
530543
531544
## Important notes
532545

codegen-llm/src/verify-script.ts

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,11 @@ _BANNED_PATTERNS: list[tuple[str, str]] = [
5858
('json.loads(self._', 'Do not return embedded JSON from json_schema() — fix the model instead'),
5959
('JsonAdapter', 'Do not define custom JsonAdapter wrappers — use TypeAdapter directly'),
6060
('def json_schema(self', 'Do not define custom json_schema() methods — only Pydantic BaseModel.json_schema() is allowed'),
61+
('SimpleNamespace', 'Do not use SimpleNamespace — use real TypeAdapter instances'),
62+
('_make_adapter', 'Do not create fake adapter factories — use real TypeAdapter instances'),
63+
('json.loads(Path', 'Do not load schema.json at runtime — models must produce schemas natively'),
64+
('_schema_path', 'Do not reference schema.json at runtime — models must produce schemas natively'),
65+
('_schema_doc', 'Do not load the schema document at runtime — models must produce schemas natively'),
6166
]
6267
6368
# Standard River error class names that must ONLY be defined in _errors.py.
@@ -561,6 +566,8 @@ def main() -> None:
561566
sys.path.insert(0, str(generated_dir.parent))
562567
pkg = generated_dir.name
563568
569+
from pydantic import TypeAdapter
570+
564571
try:
565572
sm = importlib.import_module(f'{pkg}._schema_map')
566573
except Exception as exc:
@@ -612,6 +619,16 @@ def main() -> None:
612619
)
613620
continue
614621
622+
# Verify the adapter is a real pydantic TypeAdapter, not a
623+
# duck-typed fake (e.g. SimpleNamespace with a json_schema method)
624+
if not isinstance(gen_adapter, TypeAdapter):
625+
all_errors.append(
626+
f'[{svc_name}.{proc_name}.{facet}] adapter is '
627+
f'{type(gen_adapter).__name__}, not a TypeAdapter — '
628+
f'_schema_map.py must use real TypeAdapter instances'
629+
)
630+
continue
631+
615632
try:
616633
gen_schema = gen_adapter.json_schema()
617634
except Exception as exc:

0 commit comments

Comments
 (0)