fix(files): stop rejecting ordinary HTML documents at the parser limits

[waleedlatif1]

Summary

Follow-up to #6423, which set the HTML parser's markup-token limit too tight.

• Raise the markup-token cap to a value measured safe. The previous one rejected a 30,000-row table export at 3.6 MB, which is an ordinary document.
• Leave the byte cap unchanged. That one is load-bearing, so its reasoning and the measurements behind it are now recorded next to the constant instead of being implied.

Rebased on #6442, which fixed the sanitizer independently — the sanitizer and dead-helper changes that were in this PR are dropped in favour of that one.

Type of Change

• Bug fix

Testing

Added a regression test for the table export the old cap rejected, and re-verified the documented memory thresholds against the rebased tree. bun run type-check, bun run lint, and check:api-validation clean; file-parsers and parse-route suites pass (118 tests).

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Aug 8, 2026 9:05pm

[cursor]

PR Summary

Medium Risk
Raises HTML markup limits and changes sanitization implementation on the file-parse path; behavior is intended to be equivalent but affects memory use and PostgreSQL-safe text cleaning for all parsed HTML.

Overview
Follow-up to overly tight HTML parser limits: markup-token cap rises from 500k to 1M so typical large table exports (e.g. ~30k rows) parse instead of hitting HtmlComplexityError. The 32 MB input byte cap is unchanged; comments now document cheerio memory measurements and why it stays below the shared upload limit.

sanitizeTextForUTF8 uses one UNSTORABLE_CHARACTERS regex instead of four chained .replace passes (same stripped set, fewer full-string copies). Unused sanitizeTextArray and hasInvalidUTF8Characters are removed.

Tests pin the byte-cap rejection, the new token limit, a regression for large tables, and that parseFile still surfaces HtmlComplexityError for the parse route’s fail-closed path.

^{Reviewed by Cursor Bugbot for commit c3e1373. Configure here.}

[greptile-apps]

Greptile Summary

The PR raises the HTML parser’s markup-token limit while retaining the existing byte limit, and reduces sanitizer allocation overhead without changing its established output behavior.

• Raises the markup-token cap from 500,000 to 1,000,000 and documents the measured memory boundaries.
• Adds regression coverage for large table exports and parser-limit error handling.
• Consolidates UTF-8 sanitization into one replacement pass and removes two unused helpers.
• Adds focused sanitizer behavior tests.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete blocking or independently actionable non-blocking issues identified.

The parser retains its byte-level memory backstop, the expanded token boundary is covered by acceptance and rejection tests, and the sanitizer refactor preserves the previous character-removal set without leaving repository consumers of the deleted helpers.

Important Files Changed

Filename	Overview
apps/sim/lib/file-parsers/html-parser.ts	Raises the token cap with documented memory measurements while preserving the 32 MiB input backstop.
apps/sim/lib/file-parsers/html-parser.test.ts	Updates boundary fixtures and adds coverage proving that a realistic large table is accepted.
apps/sim/lib/file-parsers/utils.ts	Replaces equivalent chained sanitizer passes with one combined character class and removes unused exports.
apps/sim/lib/file-parsers/utils.test.ts	Adds coverage for invalid-character removal, preserved whitespace, ordinary Unicode, and established surrogate handling.

_{Reviews (1): Last reviewed commit: "fix(files): stop rejecting ordinary HTML..." | Re-trigger Greptile}