Fix directoryName NameConstraints bypass

[holtrop-wolfssl]

Description

Excluded and permitted directoryName subtrees were compared with a byte prefix memcmp, so a leaf DN escaped an exclusion by varying case, string type or spacing.

Compare the RDN sequence per RFC 5280 Sec. 7.1 instead. UTF-8 decoding provided by wc_Utf8_DecodeChar() in coding.c.

Fixes ZD#22282

Testing

Added CI tests.

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[Copilot]

Pull request overview

This PR hardens X.509 directoryName nameConstraints enforcement by replacing a byte-prefix memcmp-style comparison with RFC 5280 §7.1-style comparisons over RDN sequences (case/spacing normalization and cross-string-type equivalence), and adds a UTF-8 code point decoder used by the comparison logic.

Changes:

• Implement directoryName subtree matching by parsing and comparing RDN sequences (RFC 5280 §4.2.1.10 + §7.1 folding), instead of comparing raw DER prefixes.
• Add wc_Utf8_DecodeChar() plus supporting Unicode constants to decode/validate UTF-8 code points for name comparisons.
• Add API tests covering UTF-8 decoding and directoryName constraint matching behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
wolfssl/wolfcrypt/coding.h	Adds Unicode/UTF-8 constants, enables and declares wc_Utf8_DecodeChar() under build conditions.
wolfcrypt/src/coding.c	Implements wc_Utf8_DecodeChar() and adjusts include placement.
wolfcrypt/src/asn.c	Reworks directoryName matching to compare parsed RDN sequences with RFC 5280 §7.1 folding, and updates nameConstraints list checks to rely on wolfssl_local_MatchBaseName() for length safety.
tests/api/test_coding.h	Registers the new UTF-8 decode unit test.
tests/api/test_coding.c	Adds unit tests for wc_Utf8_DecodeChar() (valid forms and invalid sequences).
tests/api/test_asn.c	Adds directoryName nameConstraints tests (case, spacing, string-type normalization, subsequence behavior).
doc/dox_comments/header_files/coding.h	Adds Doxygen documentation for wc_Utf8_DecodeChar().

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m0plus

• FLASH: .text +1,276 B (+2.0%, 66,111 B / 262,144 B, total: 25% used)

gcc-arm-cortex-m3

• FLASH: .text +1,184 B (+1.0%, 124,707 B / 262,144 B, total: 48% used)

gcc-arm-cortex-m4

• FLASH: .text +1,216 B (+0.6%, 203,055 B / 262,144 B, total: 77% used)

gcc-arm-cortex-m4-baremetal

• FLASH: .text +1,152 B (+1.7%, 68,579 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-crypto-only

• FLASH: .text +1,216 B (+0.7%, 177,130 B / 262,144 B, total: 68% used)

gcc-arm-cortex-m4-dtls13

• FLASH: .text +1,152 B (+0.6%, 184,060 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-min-ecc

• FLASH: .text +1,216 B (+2.0%, 63,557 B / 262,144 B, total: 24% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +1,152 B (+0.1%, 776,636 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pkcs7

• FLASH: .text +1,216 B (+0.6%, 215,982 B / 262,144 B, total: 82% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +1,152 B (+0.4%, 298,716 B / 1,048,576 B, total: 28% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +1,152 B (+0.4%, 329,264 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m4-sp-math

• FLASH: .text +1,216 B (+2.0%, 63,557 B / 262,144 B, total: 24% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text +1,216 B (+1.0%, 125,491 B / 262,144 B, total: 48% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +1,216 B (+0.5%, 239,521 B / 262,144 B, total: 91% used)

gcc-arm-cortex-m7

• FLASH: .text +1,152 B (+0.6%, 202,991 B / 262,144 B, total: 77% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +1,216 B (+0.4%, 299,676 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +1,152 B (+0.5%, 239,521 B / 262,144 B, total: 91% used)

linuxkm-pie

• Data: __patchable_function_entries +72 B (+0.3%, 26,632 B)

linuxkm-standard

• Data: __patchable_function_entries +16 B (+0.0%, 49,392 B)

stm32-sim-stm32h753

• FLASH: .text +1,152 B (+0.6%, 187,312 B / 2,097,152 B, total: 9% used)

[holtrop-wolfssl]

The wolfBoot Integration / keytools CI failure needs this fix on the wolfBoot side: wolfSSL/wolfBoot#857