[P1][Correctness] Reject or surface malformed and lossy CSV structures

[baixiangcpp]

Summary

The CSV-to-JSON converter now preserves quoted multiline fields, but several remaining parser behaviors still silently reinterpret or discard data:

• auto-delimiter detection counts delimiter characters inside quoted fields;
• unterminated/malformed quotes are accepted without an error;
• duplicate or blank headers overwrite object properties;
• extra row columns are silently dropped;
• row-width mismatches are not surfaced.

The common failure mode is silent data loss rather than an actionable parse error.

Confirmed cases

1. Auto-detection is not quote-aware

detectDelimiter() counts characters in the first physical line using split():

const count = firstLine.split(d).length - 1

For example:

"name,alias";age
"John,Doe";42

The comma inside the quoted field competes with the actual semicolon delimiter. Ties are resolved by candidate order, so the input can be parsed as a one-column comma CSV instead of a two-column semicolon CSV.

2. Duplicate/blank headers overwrite earlier values

Header-based conversion assigns directly into an object:

obj[h.trim()] = value

Input such as:

a,a
1,2

produces one a property with the value 2; the first column is lost. Multiple blank headers have the same problem.

3. Extra columns are dropped

The converter iterates headers, not row values. For:

a,b
1,2,3

column 3 has no header and is ignored without a warning.

4. Unterminated quotes are accepted

The tokenizer reaches end-of-input while still in quote mode and returns a record instead of reporting malformed CSV. Missing fields may then be synthesized as empty strings, hiding the structural error.

Why this matters

CSV conversion is used for migration and inspection workflows where users expect a lossless transformation. Silent overwrites or dropped columns can propagate incorrect data into subsequent tools and exports while the output still looks valid JSON.

Recommended implementation

Introduce a structured CSV parse result with records plus warnings/errors, or adopt a well-tested RFC 4180-capable parser that remains compatible with the worker/performance constraints.

Key behaviors should be explicit:

• delimiter detection must tokenize quoted fields for each candidate and score structural consistency across multiple records;
• unclosed quotes and invalid characters after a closing quote should be reported;
• duplicate/blank headers should either fail, be deterministically disambiguated with visible warnings, or offer an explicit policy;
• row-width mismatches must not silently discard cells;
• users should be able to override auto-detection after seeing the detected delimiter and warnings.

Acceptance criteria

• Delimiters inside quoted fields do not influence auto-detection counts.
• Auto-detection uses more than the first physical line and prefers a structurally consistent candidate.
• Unterminated quotes produce a parse error with row/column context.
• Duplicate and blank headers cannot silently overwrite prior columns.
• Rows with extra columns do not lose data silently; they produce an error/warning or a documented generated-column policy.
• Missing-column behavior is documented and surfaced when it changes row shape.
• The UI displays delimiter/structure warnings before export or handoff.
• Tests cover quoted delimiters, duplicate headers, blank headers, extra/missing columns, malformed quotes, CRLF, BOM, and trailing empty fields.
• Existing multiline and lossless-number regression tests continue to pass.

Relevant files

• src/features/tools/csv-json-converter/logic.ts
• src/features/tools/csv-json-converter/csv-json-task-logic.ts
• src/features/tools/csv-json-converter/page.tsx
• tests/unit/csv-json-task-logic.test.ts
• tests/unit/csv-json-task.test.ts

[baixiangcpp]

Resolved in merged PR #335 by commit 243d8ca: lossless CSV diagnostics, prototype-safe keys, and path-collision handling. The final PR CI build (Node 20) and Cloudflare Pages checks passed.