feat: pure-Lua LDAP decoder (additive)

[janiussyafiq]

2/3 of the #32 split; follows #34 (merged).

Adds asn1.get_object/asn1.decode and protocol.decode_message, plus the decoder unit and hardening test suite. Additive — nothing in lib/ calls the new decoder yet and rasn still decodes production traffic (client.lua untouched), so the security-sensitive parser can be reviewed in isolation before the cutover in 3/3.

~4,300 LoC: ~540 parser code (asn1.lua +298, protocol.lua +244) / ~3,800 tests. The review surface is the ~540 code lines.

Two behaviours the tests pin — old-vs-new differences surfaced by #34's differential oracle:

• Non-UTF-8 attribute values decode byte-for-byte. AD-shaped entries carry binary objectSid/objectGUID and invalid-UTF-8 values; the pure-Lua decoder returns them untouched. rasn calls String::from_utf8().unwrap() and panics on these, crashing the worker — a shipping availability bug against Active Directory.
• Multi-valued attribute wire order is preserved. rasn reorders (sorts) the values; the pure-Lua decoder keeps wire order. SET OF is unordered so this is likely benign — flagging for awareness at cutover.

Summary by CodeRabbit

• New Features
  • Added LDAP response message decoding with strict ASN.1 validation, covering results, search entries, references, and extended responses.
  • Added ASN.1 inspection/decoding support and stricter encoding behavior (including BOOLEAN requiring a real boolean value).
• Bug Fixes
  • Rejects malformed/hostile BER/ASN.1, non-canonical encodings, trailing/extra bytes, invalid types, truncation, oversize values, and excessive nesting depth.
  • Prevents integer truncation/overflow issues, strengthens bind/search request validation, ensures message-id wraparound, and fixes encoding memory leaks.
• Tests
  • Added extensive integration tests for correctness, boundaries, depth handling, and leak detection.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 26620dc7-2926-405d-89dd-8e76d70abaf3

📥 Commits

Reviewing files that changed from the base of the PR and between d650078 and 8c61f88.

📒 Files selected for processing (2)

• lib/resty/ldap/protocol.lua
• t/asn1_bounds.t

🚧 Files skipped from review as they are similar to previous changes (2)

• lib/resty/ldap/protocol.lua
• t/asn1_bounds.t

---

📝 Walkthrough

Walkthrough

The PR hardens ASN.1 DER/BER encoding and decoding, adds strict LDAP response parsing, validates LDAP request inputs, bounds message identifiers, and introduces broad tests for malformed, non-canonical, oversized, binary, and deeply nested inputs.

Changes

ASN.1 and LDAP protocol handling

Layer / File(s)	Summary
ASN.1 encoding and memory ownership lib/resty/ldap/asn1.lua, t/asn1_encode*.t, t/asn1_ffi.t	Encoding validates payloads and exact integer/boolean types, preserves binary data, frees OpenSSL buffers, and reports unsupported tags.
Bounded BER decoding lib/resty/ldap/asn1.lua, t/asn1*.t, t/decode_hostile.t	New get_object and decode APIs validate tags, classes, lengths, offsets, constructedness, integer ranges, child boundaries, malformed members, and recursion depth.
LDAP request validation lib/resty/ldap/protocol.lua, t/asn1_encode.t	Request builders normalize inputs, propagate encoding errors, encode types_only correctly, and wrap LDAP message IDs at the maximum value.
LDAP response decoding lib/resty/ldap/protocol.lua, t/decode*.t, t/lib/*	decode_message parses supported LDAP responses, search entries, and references with strict structure, type, trailing-data, and malformed-input checks.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LDAPPacket
  participant decode_message
  participant ASN1Decoder
  participant ResponseParser
  LDAPPacket->>decode_message: decode_message(packet)
  decode_message->>ASN1Decoder: get_object and decode message envelope
  ASN1Decoder-->>decode_message: messageID and protocolOp
  decode_message->>ResponseParser: dispatch supported protocolOp
  ResponseParser->>ASN1Decoder: decode typed response fields
  ASN1Decoder-->>ResponseParser: validated values
  ResponseParser-->>decode_message: decoded LDAP response

Loading

Possibly related PRs

• api7/lua-resty-ldap#32: Updates the same ASN.1 parsing/encoding paths and LDAP protocol decoding integration.

🚥 Pre-merge checks | ✅ 5 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
E2e Test Quality Review	⚠️ Warning	Tests are strong at packet level, but not E2E: they never exercise the real client/server path, and some pcall checks accept thrown exceptions instead of enforcing (nil, err).	Add at least one full request→response integration test through client.lua against a real LDAP server fixture, and tighten the pcall cases so crashes fail the test.

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main additive change: a pure-Lua LDAP decoder.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Security Check	✅ Passed	PASS: Changed decoder/builder code adds stricter parsing and no code in the PR logs or stores secrets, bypasses auth, or weakens TLS/ownership checks.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/resty/ldap/protocol.lua`:
- Around line 276-323: In parse_search_entry, capture the position returned
after decoding the attribute vals SET and verify it equals pastop before
accepting the PartialAttribute. Return an appropriate trailing-bytes error when
additional data remains, then continue advancing apos only for fully consumed
PartialAttribute sequences.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e6bbbd68-bc38-4ca4-acb5-cc5f1cfb3b72

📥 Commits

Reviewing files that changed from the base of the PR and between ad089e9 and d650078.

📒 Files selected for processing (20)

• lib/resty/ldap/asn1.lua
• lib/resty/ldap/protocol.lua
• t/asn1.t
• t/asn1_bounds.t
• t/asn1_class.t
• t/asn1_constructed.t
• t/asn1_depth.t
• t/asn1_encode.t
• t/asn1_encode_leak.t
• t/asn1_ffi.t
• t/asn1_integer.t
• t/asn1_nil_member.t
• t/asn1_oracle.t
• t/asn1_vectors.t
• t/decode_hostile.t
• t/decode_op_dispatch.t
• t/decode_protocol.t
• t/decode_rfc4511.t
• t/lib/ldap_msgid.lua
• t/lib/upvalue.lua

[bzp2010]

Please check #35 (comment)

[janiussyafiq]

@membphis need help review

[membphis]

Non-blocking follow-ups for future consideration:

• Propagate simple_bind_request validation errors before starting socket I/O.
• Require every PartialAttribute value to be an OCTET STRING.
• Fully validate optional LDAPResult fields and Controls.
• Enforce the RFC 4511 messageID range (0..2147483647).

I do not consider these blockers for this additive PR: APISIX supplies string credentials, production still uses rasn, and the decoder-side cases require a non-conforming or malicious LDAP peer. Please use your judgment on whether they are worth addressing in a future cleanup.

[membphis]

LGTM