Test/graphql validation fuzz#6
Open
cliedeman wants to merge 4 commits into
Open
Conversation
…Chocolate) Adds fuzzing/graphql-validation: a harness that validates the same documents against graphql-js v17 (spec reference) and HotChocolate's default rule set and diffs the results, hunting for documents graphql-js rejects but HotChocolate accepts. - node-runner.mjs (graphql-js) + HcValidationRunner (HotChocolate) share one SDL and a JSONL corpus/result contract; diff.mjs reports hc_missed / hc_extra. - fuzz.mjs: AST-mutation fuzzer (targets specific rules; edge-value injection). - seeds.json (36) + hard.json (35) curated corpora. Findings (see FINDINGS.md): HotChocolate validation accepts (graphql-js rejects) (1) Int literals outside the 32-bit range in any position, and (2) a list literal where an input object is expected. Otherwise HC matched graphql-js across ~15k fuzzed documents.
… gaps - schema: add NestedInput to exercise value validation at depth - fuzz: add a nested-input base query (edge-value injection already targets Int/list edges) - hard2.json: deep value + variable-default probes Findings (FINDINGS.md), neither matching any existing ChilliCream/graphql-platform issue: 1. Int literals are not 32-bit range-checked - a localized miss at every position and depth (direct/non-null args, nested input fields, nested list items, variable defaults). HC validates Int *kind* but not range. 2. A list value where an input object is expected makes HotChocolate report the ENTIRE document valid, suppressing all other validation errors (error-masking bypass). Root cause: ValueVisitor.Enter(ListValueNode) sets UnexpectedErrorsDetected and returns Break without ReportError (src/HotChocolate/Core/src/Validation/Rules/ValueVisitor.cs:323-324), aborting the traversal. Triggers only on (list value) x (input-object type).
…o new classes Added OverlappingFieldsCanBeMerged, directive-wrong-location, and variable-type mutators and ran 50k more documents. No new divergences surfaced: HotChocolate's field-merging, directive-location, fragment, and variable rules match graphql-js. The two value-coercion findings stand. Noted a second root-cause site for the list-for-input bypass: VariableVisitor.Enter(ListValueNode) also returns Break without ReportError (VariableVisitor.cs:283-293).
Enriched the schema (a second union; same-named fields with conflicting types
across union members) and added interface/union probes (hard3.json), an
inline-fragment-injector mutator, and abstract-type base queries (~30k more docs).
Interface/union STRUCTURAL validation is solid in HotChocolate: fragment-spread
possibility, fields-on-abstract-types, and OverlappingFieldsCanBeMerged's
sameResponseShape across mutually-exclusive union members all match graphql-js in
both directions.
Two new findings, both on the __typename meta-field:
3. ScalarLeafs not enforced on __typename selected on an abstract (union/interface)
parent: { catOrDog { __typename { x } } } is accepted, though it is caught on
object/root parents.
4. Directive ARGUMENTS not validated on __typename (@Skip without if, @Skip(if: 5),
@Skip(wrong: true) all accepted), while the directive's existence/location is
checked and ordinary fields are fully validated. Root cause:
ArgumentVisitor.cs:48-64 returns Skip past the field's directives.
10 tasks
cliedeman
force-pushed
the
test/graphql-validation-fuzz
branch
from
June 27, 2026 14:34
02d71c7 to
a165627
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Research branch (not for merge): a differential fuzzer that validates the same documents against the same schema with the spec reference validator (graphql-js v17.0.1) and HotChocolate's full default rule set, then diffs. Prime signal: graphql-js reports invalid, HotChocolate reports valid.
Corpus: 36 curated seeds + ~95 hand-written hard cases + ~125k seeded AST-mutation fuzz documents (value coercion, field merging, directive locations, fragment spreads/cycles, variable usage/types, operation structure, inline-fragment type conditions on abstract types).
Findings (all unreported upstream)
Intliterals not 32-bit range-checked. HC validates the literal kind but accepts out-of-range integers (2147483648, etc.) that graphql-js rejects. Boundary is exactly Int32; holds at every literal position (arg, nested input field, list element, variable default). Localized (does not suppress siblings).ValueVisitor.cs:323andVariableVisitor.cs:292returnBreakwithoutReportError.ScalarLeafsnot enforced on__typenameunder abstract types.{ catOrDog { __typename { x } } }/{ pet { __typename { x } } }accepted by HC (caught on object/root parents).__typename.@skipwithoutif,@skip(if: 5),@skip(wrong: true)all accepted. Root cause:ArgumentVisitor.cs:48-64returnsSkippast the field's directives.Interface/union structural validation is solid (fragment-spread possibility, fields-on-abstract-types, and
OverlappingFieldsCanBeMerged'ssameResponseShapeacross mutually-exclusive union members all match graphql-js). No case where HC was stricter than the spec.Full write-up, repros, and line refs:
fuzzing/graphql-validation/FINDINGS.md.— Claude
🤖 Generated with Claude Code