[WIP] Add C++ regular-expression parser and RegexTreeViewSig implementation

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

Overview

C++ (codeql/cpp-all) currently has no regular-expression parse tree and no implementation of the shared RegexTreeViewSig signature. As a result, none of the shared ReDoS analyses (shared/regex) can be instantiated for C++, and there is no cpp/polynomial-redos or cpp/redos query.

This task is Phase 1 of adding ReDoS support to C++: build the self-contained foundation — a C++ regular-expression parser plus a RegexTreeViewSig implementation — with no queries and no dataflow. Later phases (regex flow modeling, and the actual ReDoS queries) are explicitly out of scope here.

Background / references

The shared regex library lives in shared/regex and is parameterized over the signature module RegexTreeViewSig defined in shared/regex/codeql/regex/RegexTreeView.qll. Every language that supports the shared ReDoS analyses provides:

• an internal/ParseRegExp.qll that tokenizes a string literal into start/end offset spans, and
• a RegexTreeView.qll defining a newtype TRegExpParent and a module Impl implements RegexTreeViewSig.

There are two families of existing implementations:

• Wrap an existing parser (Java, JavaScript).
• Parse from scratch in QL (Python, Ruby, Swift).

C++ has no parser, so it belongs to the parse-from-scratch family. Use Python's python/ql/lib/semmle/python/regexp/internal/ParseRegExp.qll and Ruby's ruby/ql/lib/codeql/ruby/regexp/internal/ParseRegExp.qll as the structural template, and the corresponding RegexTreeView.qll files (Python: python/ql/lib/semmle/python/regexp/RegexTreeView.qll, Ruby: ruby/ql/lib/codeql/ruby/regexp/RegExpTreeView.qll) as the template for the tree view.

Important dialect note: C++ std::regex defaults to ECMAScript syntax. Therefore the syntax/semantics decisions must follow ECMAScript, cross-checked against the JavaScript implementation (javascript/ql/lib/semmle/javascript/security/regexp/RegExpTreeView.qll), NOT Ruby/Python semantics where they differ. Concrete consequences:

• isPossessive(term) should hold for nothing (none()), like JavaScript — ECMAScript has no possessive quantifiers.
• Named-group syntax should be ECMAScript-style (?<name>...) and \k<name> backreferences, NOT Python's (?P<name>...)/(?P=name).
• Flags relevant to the signature are i (ignore case) and s (dotAll). Since C++ regex flags are passed via std::regex::flag_type at construction (not inline), for Phase 1 it is acceptable to implement isIgnoreCase/isDotAll conservatively (e.g. hold for nothing, with a clear TODO), rather than modeling construction-site flags — that requires dataflow which is Phase 2.

Scope — do this

1. Parser: Add cpp/ql/lib/semmle/code/cpp/regex/internal/ParseRegExp.qll.

   • Define class RegExp extends StringLiteral (using semmle.code.cpp / the C++ StringLiteral class from semmle.code.cpp.exprs.Literal).
   • Provide the offset-span predicates the tree view needs, mirroring the Python/Ruby parser: getChar, escapingChar/escaping, escapedCharacter, charSet/char_set_start/charRange/char_set_child, group/groupContents/getGroupNumber/getGroupName, the various lookahead/lookbehind assertion starts, backreference (numbered \1 and ECMAScript-named \k<name>), qualifier/short_qualifier/multiples, qualifiedItem/qualifiedPart/item, sequence, alternation/alternationOption, specialCharacter, normalCharacter/normalCharacterSequence, and char/anchor handling for ^, $, \b, \B, ..
   • Use ECMAScript group syntax ((?:...), (?=...), (?!...), (?<=...), (?<!...), (?<name>...)).
   • It is acceptable to restrict initial support to the common ECMAScript constructs; clearly mark anything unsupported with TODO comments. Correctness of the core constructs (quantifiers *, +, ?, {n}, {n,}, {n,m}, alternation, grouping, character classes/ranges, escapes) is the priority since ReDoS depends on it.
   • You do NOT need to model which string literals are used as regexes via dataflow in this phase. Provide a simple entry point (e.g. treat all C++ StringLiterals as candidate RegExps, or gate on a simple heuristic) and leave a TODO that Phase 2 will restrict this using std::regex construction/using-site flow. Keep this cheap so it doesn't blow up performance in tests.

2. Tree view: Add cpp/ql/lib/semmle/code/cpp/regex/RegexTreeView.qll.

   • Follow the Ruby/Python structure: private newtype TRegExpParent = TRegExpLiteral(...) | TRegExpQuantifier(...) | TRegExpSequence(...) | TRegExpAlt(...) | TRegExpCharacterClass(...) | TRegExpCharacterRange(...) | TRegExpGroup(...) | TRegExpSpecialChar(...) | TRegExpNormalChar(...) | TRegExpBackRef(...) (add named-property/other variants only if you implement them).
   • import Impl as RegexTreeView and import Impl, and define module Impl implements RegexTreeViewSig (import the signature from codeql.regex.RegexTreeView).
   • Implement every member the signature requires, including the helper predicates isEscapeClass, isPossessive (→ none()), matchesAnyPrefix, matchesAnySuffix, isExcluded, isIgnoreCase, isDotAll. Provide Top, RegExpParent, RegExpTerm and all the term subclasses (RegExpQuantifier, RegExpStar, RegExpPlus, RegExpOpt, RegExpRange, InfiniteRepetitionQuantifier, RegExpAlt, RegExpSequence, RegExpGroup, RegExpCharacterClass, RegExpCharacterRange, RegExpNormalChar/constant, RegExpDot, RegExpAnchor/RegExpDollar/RegExpCaret, RegExpWordBoundary, RegExpNonWordBoundary, RegExpBackRef, RegExpEscape/RegExpCharEscape/RegExpCharacterClassEscape, RegExpSubPattern/lookahead/lookbehind classes, RegExpConstant). Ensure hasLocationInfo/location maps back to the underlying C++ StringLiteral offsets so alerts point at the right source range.

3. Packaging: Update cpp/ql/lib/qlpack.yml to add the dependency codeql/regex: ${workspace} under dependencies: (keeping alphabetical ordering consistent with the existing entries).

4. Sanity instantiation (compile check only): To prove the tree view satisfies the signature and works with the shared engine, verify that codeql.regex.nfa.SuperlinearBackTracking::Make<RegexTreeView> (and NfaUtils::Make<RegexTreeView>) compile against the new implementation. You may do this via a small internal/test .ql that imports and instantiates them; do NOT add a production query.

5. Tests: Add library tests under cpp/ql/test/library-tests/regex/ following the pattern used by ruby/ql/test/library-tests/regexp / the Python equivalents:

   • A C++ test source file containing a variety of regex string literals (quantifiers, nested quantifiers like (a+)+, alternations, character classes/ranges, groups, named groups, lookaheads/lookbehinds, backreferences, anchors, escapes).
   • A .ql test that prints the parsed RegExpTerm tree (term class + location + raw value) with an accompanying .expected file.
   • A consistency query that flags failedToParse/unparsed offsets so parser gaps are visible.

6. Change note: Add an appropriate change note for codeql/cpp-all announcing the new (internal) regex library, following the repo's change-note conventions (cpp/ql/lib/change-notes/).

Explicitly out of scope (do NOT do)

• No RegexFlowConfigs.qll / dataflow modeling of std::regex/regex_match/regex_search/regex_replace/boost/PCRE/RE2.
• No PolynomialReDoSQuery.qll, no .ql production queries, no .qhelp, no query-suite changes.
• No exponential ReDoS query.

Acceptance criteria

• cpp/ql/lib/semmle/code/cpp/regex/RegexTreeView.qll compiles and module Impl implements RegexTreeViewSig type-checks.
• SuperlinearBackTracking::Make<RegexTreeView> and NfaUtils::Make<RegexTreeView> compile when instantiated with the new tree view.
• Library tests parse the sample regexes into the expected tree, with .expected files committed and passing.
• cpp/ql/lib/qlpack.yml declares the codeql/regex dependency.
• No production queries or dataflow are added.
• A change note is included.

Hints

• The signature to satisfy is in shared/regex/codeql/regex/RegexTreeView.qll (RegexTreeViewSig). Read its doc comments carefully — each class/predicate contract is documented there.
• The cleanest end-to-end reference for "parse-from-scratch + tree view + satisfy signature" is Ruby (ruby/ql/lib/codeql/ruby/regexp/), and Python (python/ql/lib/semmle/python/regexp/). Copy structure from these.
• Copy ECMAScript-specific decisions (possessive = none, prefix/suffix predicates) from javascript/.../RegExpTreeView.qll.
• Keep the "used as a regex" detection trivial/cheap in this phase (a TODO for Phase 2), but make sure it doesn't cause the tests to time out.

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: In github/codeql there is a shared library for regexes. Based on that library and the fact that there is no redos query for C++, could you make a plan for writing a C++ redos query?

Assistant: I'll research this. Let me start by understanding the shared regex library and how existing ReDoS queries are structured in the CodeQL repo.

Assistant: I now have a good understanding of the shared regex library structure. Let me examine what regex support already exists for C++ and how a language plugs into the shared library.

Assistant: Confirmed: there's no regex/ReDoS infrastructure in the C++ pack. Let me look at how another language (Rust, which was the most recent addition) wired up the shared regex pack, since that's the best template for a new language, and check the shared library's TreeView interface requirements.

Assistant: Now let me check whether C++ has any regex parse tree at all, and see how a recently-added language (e.g., Rust) wired up its ReDoS query as a template.

Assistant: C++ has no regex tree view library at all, which is the crux. Let me confirm which languages already implement the shared regex library.

Assistant: I now have a clear picture. Let me confirm whether C++ has any regex parsing infrastructure at all.

Assistant: I now have enough understanding of the shared library architecture and the state of C++ regex support. Let me confirm one thing about how the Rust language (a recent addition) wired up its regex tree view, since it's the most recent example and likely closest to what C++ would need.

I have enough to lay out a solid plan.

Note: My code searches only return the top 10 results and may be incomplete. You can browse the full results in the GitHub UI, e.g. ReDoS-related files in the shared regex pack and regex usage in the C++ library.

Plan: A C++ ReDoS query built on the shared regex library

What already exists (the reusable pieces)

The shared/regex pack (codeql/regex) contains language-agnostic ReDoS analysis, all parameterized over a single signature module RegexTreeViewSig (RegexTreeView.qll):

• SuperlinearBackTracking.qll → PolynomialBackTrackingTerm (this is the polynomial ReDoS engine — the analog of js/polynomial-redos).
• ExponentialBackTracking.qll → exponential ReDoS engine (js/redos).
• NfaUtils.qll, RegexpMatching.qll → NFA construction and matching support.

Each engine is a Make<RegexTreeViewSig TreeImpl> functor. Java is the cleanest reference implementation to copy from:

• RegexTreeView.qll implements the signature.
• RegexFlowConfigs.qll figures out which string literals are used as regexes and what they're matched against.
• PolynomialReDoSQuery.qll is the taint config gluing sources → superlinear-backtracking sinks.

What C++ is missing

C++ has no regex parse tree at all. Everything under cpp/ql/lib/.../security/ uses raw regexpMatch/regexpCapture on strings — there is nothing implementing RegexTreeViewSig. This is the bulk of the work and the key difference from Java/Ruby/Python (which all had a pre-existing RegExp parser to wrap).

Proposed work breakdown

Phase 1 — Regex parser + tree view (the hard part)

1. Write a C++ regex parser in QL, cpp/ql/lib/semmle/code/cpp/regex/internal/ParseRegExp.qll, that tokenizes a StringLiteral into start/end offset spans (quantifiers, sequences, alternations, char classes, ranges, groups, backreferences, escapes). Model it on Ruby's/Python's ParseRegExp.qll and their newtype TRegExpParent structure.
2. Write cpp/ql/lib/semmle/code/cpp/regex/RegexTreeView.qll — a module Impl implements RegexTreeViewSig, exporting RegExpTerm, RegExpQuantifier, InfiniteRepetitionQuantifier, etc., plus the required helper predicates (isEscapeClass, isPossessive, isIgnoreCase, isDotAll, matchesAnyPrefix/Suffix, isExcluded). Follow the Ruby/Python file structure closely.
   • Dialect decision: target ECMAScript-style std::regex (the default), and note in the code that other syntaxes (grep, awk, boost::regex, std::regex::extended, RE2, PCRE) differ. This affects things like possessive quantifiers.

Phase 2 — "Used as a regex" flow modeling
3. Add cpp/ql/lib/semmle/code/cpp/regex/RegexFlowConfigs.qll modeling the C++ regex APIs so you know which string literals become regexes and against what they're matched:

• Sinks: construction of std::basic_regex/std::regex, and std::regex_match/std::regex_search/std::regex_replace; optionally boost::regex, PCRE (pcre_compile), RE2.
• Reuse the existing dataflow to connect a literal StringLiteral → regex object → match call (analogous to Java's regexMatchedAgainst). Restrict, as Java does, to regexes containing +/*/{n,} for performance.

Phase 3 — The query itself
4. Add cpp/ql/lib/semmle/code/cpp/security/regexp/PolynomialReDoSQuery.qll instantiating SuperlinearBackTracking::Make<TreeView>, with a DataFlow::ConfigSig:

• Sources: ActiveThreatModelSource / remote flow sources (FlowSources.qll).
• Sinks: the string matched against a PolynomialBackTrackingTerm.
• Barriers: length-restricted values, simple type sanitizers (mirror Java's LengthRestrictedMethod).

5. Add the query cpp/ql/src/Security/CWE/CWE-1333/PolynomialReDoS.ql + .qhelp, with @id cpp/polynomial-redos, appropriate @tags, and CWE-1333 metadata.
6. (Optional, follow-up) Add the exponential variant using ExponentialBackTracking.qll → cpp/redos.

Phase 4 — Packaging & tests
7. Update cpp/ql/lib/qlpack.yml to depend on the codeql/regex shared pack.
8. Add library tests for the parser (consistency queries + expected parse trees, like ruby/ql/test/.../regexp) and query tests with vulnerable/safe std::regex samples.
9. Add change notes and, if it ships enabled, query-suite membership + experimental→stable promotion considerations.

Key risks / decisions to flag

• Parser correctness is the whole ballgame. A weak parser yields both FPs and FNs; budget most effort here and lean heavily on the Ruby parser as a template.
• Regex dialect ambiguity in C++ (std::regex supports ECMAScript/basic/extended/awk/grep/egrep). Start with ECMAScript-only and gate on the syntax_option_type flag if detectable.
• Precedent: Ruby was added this exact way (PR wiring ParseRegExp + RegExpTreeView + shared engine), so this is a proven path.

Suggested first PR

Land Phase 1 alone (parser + tree view + parser tests) since it's self-contained, reviewable, and unblocks both the polynomial and exponential queries.

Would you like me to kick off a coding-agent task to scaffold Phase 1 (the C++ regex parser and RegexTreeView implementation modeled on Ruby's), or open a tracking issue with this plan broken into the phases above?

User: Why model phase 1 based on Ruby?

Assistant: Good question — let me be precise, because "model on Ruby" was a shortcut in my phrasing and it's worth unpacking whether it's actually the right choice.

Why I reached for Ruby

The shared engine consumes a RegexTreeViewSig implementation, and among the existing implementations there are effectively two styles:

1. Wrap a pre-existing regex parser — Java does this. Its RegexTreeView.qll imports semmle.code.java.regex.regex and just adapts an already-written Regex class to the signature.
2. Parse from scratch in QL, then adapt — Ruby, Python, and Swift do this. They each have an internal/ParseRegExp.qll that tokenizes a string literal into start/end offset spans, a newtype TRegExpParent built from those spans, and an Impl implements RegexTreeViewSig wrapper.

C++ has neither a regex parser nor a tree view today, so it falls into category 2 — it needs the parser written from scratch. Ruby, Python, and Swift are the three worked examples of category 2, and their ParseRegExp.qll + TRegExpParent + Impl structure is nearly identical across all three. So the real recommendation is: model Phase 1 on the "parse-from-scratch" family, not specifically on Ruby.

Why Ruby specifically over Python/Swift

The tie-breakers are minor, but:

• Ruby and Python are near-identical in their TRegExpParent newtypes (I saw both in the code search — they differ mainly in Ruby including TRegExpNamedCharacterProperty and Python's extra guards on empty sequences/alternations).
• Swift wraps a Swift-specific RegExp and pulls in codeql/util/Numbers, so it's slightly more entangled with language specifics.
• Ruby's parser is one of the more complete and actively-maintained ones and handles a broad character/escape set.

But honestly, none of these is decisive.

The more important point: match the dialect, not the language

The thing that should actually drive the choice is which regex syntax the target resembles, not which QL implementation looks tidiest:

• C++ std::regex defaults to ECMAScript syntax.
• The most faithful ECMAScript reference is JavaScript's implementation — but JS is category 1 (it wraps the JS engine's own RegExpTerm), so it's not directly copyable as a parser skeleton.

So the pragmatic recommendation is:

• Copy the structure (ParseRegExp.qll → TRegExpParent → Impl) from Python or Ruby (whichever you find cleaner — they're interchangeable), and
• Copy the syntax/semantics decisions (escape classes, possessive quantifiers = none for ECMAScript, ./\d/\w handling, flags like i/s) from the JavaScript RegExpTreeView and the signature doc comments.

Concretely, several signature predicates are dialect-sensitive and should follow JS, not Ruby: isPossessive (JS returns none(); Ruby/PCRE do support them), and matchesAnyPrefix/matchesAnySuffix.

Bottom line

I shouldn't have said "model on Ruby" as if Ruby were special. The accurate guidance is:

• Structure of the parser: copy from a from-scratch implementation (Ruby or Python — pick the cleaner one).
• Regex dialect/semantics: follow ECMAScript, cross-checking against JavaScript's RegExpTreeView, because that's what std::regex defaults to.

Would you like me to look at the Ruby vs. Python ParseRegExp.qll side by side and recommend which is the better structural starting point for C++?

User: Fair. In that case create a plan for phase 1, and start an agent that creates a PR for it.

Assistant: I'll look at the Ruby and Python ParseRegExp.qll implementations side by side to ground the plan, then kick off the agent.

Assistant: I now have a clear picture of the parser structure. Let me verify the C++ qlpack dependencies so the agent knows how to wire in the shared pack.

Assistant: I have everything I need. Here's the Phase 1 plan and I'll kick off an agent to implement it.

Phase 1 plan: C++ regex parser + RegexTreeView

Goal: Add a from-scratch C++ regular-expression parser and a RegexTreeViewSig implementation to codeql/cpp-all, plus tests — with no queries yet. This is the self-contained foundation that later unblocks the polynomial/exponential ReDoS queries.

Deliverables:

1. cpp/ql/lib/semmle/code/cpp/regex/internal/ParseRegExp.qll — offset-span parser (RegExp extends StringLiteral), structurally modeled on Python/Ruby's ParseRegExp.qll but with ECMAScript (std::regex default) semantics.
2. cpp/ql/lib/semmle/code/cpp/regex/RegexTreeView.qll — Impl implements RegexTreeViewSig with the TRegExpParent newtype and all required classes/predicates.
3. cpp/ql/lib/qlpack.yml — add codeql/regex: ${workspace} dependency.
4. Tests under cpp/ql/test/library-tests/regex/ — parse-tree printing + consistency queries.
5. A change note.

Scope guardrails: no RegexFlowConfigs, no dataflow, no queries — those are Phase 2/3.

Let me start the agent.

Assistant: I've drafted the Phase 1 plan and queued an agent task to open the PR — it's now waiting for your confirmation before it runs.

Here's a summary of what I'm asking the agent to build (and deliberately not build):

Phase 1 plan: C++ regex parser + RegexTreeView (foundation only)

Goal: Give C++ the missing regex parse tree and a RegexTreeViewSig implementation, so the shared ReDoS engines can later be instantiated — but ship no queries and no dataflow in this phase.

Deliverables the agent will produce:

1. cpp/ql/lib/semmle/code/cpp/regex/internal/ParseRegExp.qll — a from-scratch parser over C++ StringLiteral, structurally modeled on Python's and Ruby's ParseRegExp.qll (offset-span predicates for quantifiers, groups, char classes/ranges, alternation, escapes, anchors, backreferences).
2. cpp/ql/lib/semmle/code/cpp/regex/RegexTreeView.qll — newtype TRegExpParent + module Impl implements RegexTreeViewSig, following the Ruby/Python tree-view structure.
3. qlpack.yml — add the codeql/regex dependency.
4. A compile-only instantiation of SuperlinearBackTracking::Make<RegexTreeView> / NfaUtils::Make<RegexTreeView> to prove the signature is satisfied.
5. Library tests under cpp/ql/test/library-tests/regex/ (sample regexes → expected parse tree + a failedToParse consistency check).
6. A change note.

Key dialect guardrails I baked in (this is the crucial correction from our earlier discussion): structure comes from Python/Ruby, but semantics follow ECMAScript / JavaScript because std::regex defaults to ECMAScript — so isPossessive → none(), ECMAScript-style (?<name>...)/\k<name> (not Python's (?P<...>)), and flags (i/s) handled conservatively since C++ passes them at construction (deferred to Phase 2).

Explicitly out of scope: RegexFlowConfigs.qll, any std::regex/boost/PCRE dataflow, PolynomialReDoSQuery.qll, .ql/.qhelp queries, and the exponential variant.

If that scope looks right, confirm the task and the agent will open the PR. If you'd like to tweak anything first — e.g. widen it to also include Phase 2 flow modeling, or narrow the initial parser to a smaller construct set — tell me and I'll adjust before it runs.

User: @copilot Accepted Confirmation: Confirm cloud agent session

Assistant: The task is confirmed. Let me start the agent.

Assistant: I've started the coding agent task to implement Phase 1. It will work in the background and open a pull request against github/codeql when it's done.

What the agent is building (Phase 1 — foundation only):

• cpp/ql/lib/semmle/code/cpp/regex/internal/ParseRegExp.qll — a from-scratch C++ regex parser over StringLiteral, structured like Python's/Ruby's parser but with ECMAScript syntax (std::regex's default).
• cpp/ql/lib/semmle/code/cpp/regex/RegexTreeView.qll — newtype TRegExpParent + module Impl implements RegexTreeViewSig, with ECMAScript semantics (e.g. isPossessive → none(), (?<name>...)/\k<name>).
• qlpack.yml — adds the codeql/regex dependency.
• A compile-only instantiation of SuperlinearBackTracking::Make<RegexTreeView> / NfaUtils::Make<RegexTreeView> to prove the signature is satisfied.
• Library tests under cpp/ql/test/library-tests/regex/ (parse-tree .expected + a failedToParse consistency check).
• A change note.

Explicitly excluded (deferred to later phases): any std::regex/boost/PCRE dataflow modeling, PolynomialReDoSQuery.qll, production .ql/.qhelp queries, and the exponential ReDoS variant.

You can follow progress on the PR once it appears. When it's up, I can help review the parser, check the CI/test results, or draft the Phase 2 (regex flow modeling) task. Would you like me to prepare the Phase 2 plan now so it's ready to go once Phase 1 lands?