From 91404264d36249f65c2dbdb67fa7d93bdf709ace Mon Sep 17 00:00:00 2001 From: Andreas Stefl Date: Thu, 16 Jul 2026 19:33:01 +0200 Subject: [PATCH 1/2] feat(doc): per-run character formatting from CHPX Implement Direct Character Formatting ([MS-DOC] 2.4.6.2): read the PlcBteChpx and its ChpxFkp pages, split the decoded piece text at every formatting-run boundary, and emit styled span elements carrying bold, italic, underline, strike, font size, font name (SttbfFfn), font color (sprmCCv/sprmCIco) and highlight (sprmCHighlight). Replaces the flat 11pt placeholder; unformatted text now defaults to 10pt (the sprmCHps default). Paragraphs store the style of their first run so empty paragraphs keep their height. Equal Chpx byte sequences share one resolved style and adjacent equal-style runs merge. Field-code hiding is now stateful across runs and paragraphs. Pcd.Prm modifications and STSH style layering remain open (documented in AGENTS.md). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01R9Z3kyUaHNkJacAEggevHH --- src/odr/internal/oldms/text/AGENTS.md | 152 ++++----- src/odr/internal/oldms/text/doc_document.cpp | 26 +- .../oldms/text/doc_element_registry.cpp | 23 ++ .../oldms/text/doc_element_registry.hpp | 14 + src/odr/internal/oldms/text/doc_helper.cpp | 310 ++++++++++++++++++ src/odr/internal/oldms/text/doc_helper.hpp | 56 ++++ src/odr/internal/oldms/text/doc_parser.cpp | 279 ++++++++++------ src/odr/internal/oldms/text/doc_structs.hpp | 49 +++ test/src/internal/oldms/oldms_test.cpp | 264 +++++++++++++++ 9 files changed, 979 insertions(+), 194 deletions(-) diff --git a/src/odr/internal/oldms/text/AGENTS.md b/src/odr/internal/oldms/text/AGENTS.md index ab513d25..7cf20fcc 100644 --- a/src/odr/internal/oldms/text/AGENTS.md +++ b/src/odr/internal/oldms/text/AGENTS.md @@ -4,24 +4,26 @@ The **why** and the roadmap; what is concretely done lives in the code. Shared `oldms/` conventions are in [`../AGENTS.md`](../AGENTS.md). **Scope.** Extract the **visible text of the main document body**, split into -paragraphs and manual line breaks, through the abstract model so the generic HTML -renderer lays it out as a flat run of paragraphs. No character/paragraph styles, -headers/footers/notes/annotations, tables, frames, images, or fields beyond their -result text. +paragraphs and manual line breaks, plus **direct character formatting** (font, +size, bold, italic, underline, strike, color, highlight) as styled spans, +through the abstract model. No paragraph styles or style-sheet (STSH) +inheritance, headers/footers/notes/annotations, tables, frames, images, or +fields beyond their result text. -**Specs.** `[MS-DOC]` (FIB, Clx / piece table, text decoding) + `[MS-CFB]` -container. The implemented read path matches *Retrieving Text* (§2.4.1 steps 1–6); -section numbers are cited inline in code and in the read-path map below. +**Specs.** `[MS-DOC]` (FIB, Clx / piece table, text decoding, CHPX) + +`[MS-CFB]` container. The implemented read path matches *Retrieving Text* +(§2.4.1 steps 1–6) and *Direct Character Formatting* (§2.4.6.2); section +numbers are cited inline in code and in the read-path maps below. ## Module layout (sibling of `../presentation`) | File (`oldms/text/`) | Role | |---|---| -| `doc_structs.hpp` | `#pragma pack(1)` PODs (`FibBase`, the `FibRgFcLcb97/2000/…/2007` chain, `Sprm`, `FcCompressed`, `Pcd`) + `static_assert` sizes + `PlcPcdMap` + `ParsedFib` | +| `doc_structs.hpp` | `#pragma pack(1)` PODs (`FibBase`, the `FibRgFcLcb97/2000/…/2007` chain, `Sprm`, `FcCompressed`, `Pcd`, `PnFkpChpx`, `FfnFixed`) + `static_assert` sizes + `PlcPcdMap`/`PlcBteChpxMap` + `ParsedFib` + the character SPRM opcodes | | `doc_io.{hpp,cpp}` | `read(...)` over `std::istream`: variable-length FIB, Clx walk, string decoding | -| `doc_helper.{hpp,cpp}` | `CharacterIndex` (decoded piece table) + `read_character_index` | -| `doc_parser.{hpp,cpp}` | `parse_tree` → body text + tree; `clean_text` (field & control-char handling) | -| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + text side-payload | +| `doc_helper.{hpp,cpp}` | `CharacterIndex` (decoded piece table) + `read_character_index`; `CharacterStyles` (fc-keyed style runs) + `read_character_styles` (PlcBteChpx/ChpxFkp walk), `apply_character_sprms`, `read_font_names` (SttbfFfn) | +| `doc_parser.{hpp,cpp}` | `parse_tree` → styled runs + tree; `TextCleaner` (field & control-char handling) | +| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + text side-payload + per-element `TextStyle` map + font-name intern store | | `doc_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | ## Design decisions @@ -59,23 +61,47 @@ main body, `Prc` formatting runs, and every control/field char `clean_text` drop **Endianness.** Host byte order / LSB-first bit-fields assumed; shared `oldms/` assumption + fix plan in [`../AGENTS.md`](../AGENTS.md). -**The `font_size = 11pt` in the adapters is a placeholder hack** so empty -paragraphs still have height (same as the `.ppt`/`.xls` modules); removed when -character formatting lands (open work §1). `Document::is_editable()` → `true` and -`is_savable(!encrypted)`, but `save`/`text_set_content` throw — read-only in -practice. - -**`clean_text` control-character handling** (the non-obvious cases): `0x0D`/`0x0C`/ -`0x0B` are consumed by the caller's paragraph/page/line splits and never reach -`clean_text`; `0x13`/`0x14`/`0x15` delimit a **field** — instruction hidden, -result shown, the `0x14` separator optional (§2.8.25), nesting tracked with a -per-field stack; `0x09` tab kept; `0x1E` non-breaking hyphen → `-`; `0x1F` -optional hyphen dropped; every other control char < `0x20` dropped. +**Direct character formatting only, resolved to styled spans.** The tree is +`root → paragraph → span → text`; each span (and each paragraph, for +empty-paragraph height) stores a resolved `TextStyle` in a registry side-map. +The §2.4.6.2 walk: `PlcBteChpx` (table stream) → 512-byte `ChpxFkp` pages +(WordDocument stream) → per-run `Chpx.grpprl`, applied over a default of 10pt +(the `sprmCHps` default of 20 half-points; the old flat `11pt` placeholder is +gone). The non-obvious bits: +- **Runs are keyed by stream offset (fc), not CP**, so the piece decode walks + each piece in style-uniform chunks (`CharacterStyles::chunk_end`); a + boundary inside a 2-byte CP is pushed past it. +- **Equal `Chpx` bytes share one resolved style**, and adjacent equal-style + runs merge, so span count stays low. `rgb[j] == 0` means default properties. +- **`ToggleOperand`** (§2.9.327) values `0x80`/`0x81` refer to the (unmodelled) + style value, which defaults to off → `0x80` = off, `0x81` = on. +- **Colors**: `sprmCCv` is a `COLORREF` (`fAuto` → unset); the legacy + `sprmCIco`/`sprmCHighlight` use the `Ico` palette (§2.9.119) — the spec's + extracted table repeats `0x0C` for `0x0D`, which is dark red (`0x800000`). +- **`Pcd.Prm` modifications and STSH styles are not applied** (open work §1). +- Font names from `SttbfFfn` are interned in a registry `std::deque` so + `TextStyle::font_name` (`const char *`) stays valid. + +`Document::is_editable()` → `false`; `save`/`text_set_content` throw. + +**`TextCleaner` control-character handling** (the non-obvious cases): `0x0D`/ +`0x0C`/`0x0B` are consumed by the caller's paragraph/page/line splits and never +reach the cleaner; `0x13`/`0x14`/`0x15` delimit a **field** — instruction +hidden, result shown, the `0x14` separator optional (§2.8.25), nesting tracked +with a per-field stack that now **persists across style runs and paragraphs** +(a field can span both); `0x09` tab kept; `0x1E` non-breaking hyphen → `-`; +`0x1F` optional hyphen dropped; every other control char < `0x20` dropped. ## Tests - `OldMs.doc_read_string_compressed` — the compressed decoder against the §2.9.73 byte map (ASCII passthrough, `0x82–0x9F` remap, `0xA0–0xFF` round-trip). +- `OldMs.doc_apply_character_sprms` — every modelled SPRM, toggle semantics, + cvAuto reset, unknown fixed/variable SPRM skipping, malformed-grpprl throws. +- `OldMs.doc_read_font_names` — synthetic SttbfFfn. +- `OldMs.doc_character_formatting` — end-to-end over a synthetic `.doc` + (inline bytes: FIB + ChpxFkp + piece table): span splitting at run + boundaries, default 10pt, bold + font-name run, paragraph style. **Not yet tested:** FIB robustness (negative `ccpText`, newer-than-2007 fallback), `0x0C` page-break emission, and there is **no assertion-based render test** over a @@ -122,67 +148,21 @@ Retrieving Text: §2.4.1 (steps 1–6) Field chars 0x13/0x14/0x15: §2.8.25 # Open work -## 1. Character (font) formatting → the IR (the next feature) - -**Goal.** Extract per-run character properties (font name, size, bold, italic, -underline, strike, colour, highlight) and surface them through `TextStyle`, so the -renderer styles text instead of emitting one flat 11pt run. Replaces the -`font_size = 11pt` placeholder. - -`TextStyle` (`src/odr/style.hpp`) maps almost 1:1 onto the `.doc` character SPRMs: - -| `TextStyle` field | SPRM (opcode) | operand → value | -|---|---|---| -| `font_size` | `sprmCHps` (0x4A43) | u16 **half-points** → `Measure(hps/2, pt)` (default 20 = 10pt) | -| `font_weight` | `sprmCFBold` (0x0835) | `ToggleOperand` → `bold` | -| `font_style` | `sprmCFItalic` (0x0836) | `ToggleOperand` → `italic` | -| `font_underline` | `sprmCKul` (0x2A3E) | `Kul`, `0x00` = none → `bool` | -| `font_line_through` | `sprmCFStrike` (0x0837) | `ToggleOperand` → `bool` | -| `font_color` | `sprmCCv` (0x6870) | `COLORREF` → `Color` (legacy `sprmCIco` 0x2A42 is a palette index) | -| `background_color` | `sprmCHighlight` (0x2A0C) | `Ico` highlight index → `Color` | -| `font_name` | `sprmCRgFtc0` (0x4A4F) | s16 index into `SttbfFfn` → font name | - -`font_name` is a `const char *`, so the resolved name needs stable storage — intern -it in the `ElementRegistry` (e.g. a `std::deque` whose elements never -move) and hand out the pointer. - -**How `[MS-DOC]` retrieves character properties** — Direct Character Formatting -(§2.4.6.2) reuses the *Retrieving Text* walk we already have: -1. For `cp`, run §2.4.1 to get its byte offset `fc` and owning `Pcd` (already - computed). -2. Read **`PlcBteChpx`** (§2.8.5) at `fcPlcfBteChpx` in the table stream — a PLC - keyed by stream offset: `aFC[n+1]` + `aPnBteChpx[n]` (`PnFkpChpx`, 4 B). -3. Largest `i` with `aFC[i] ≤ fc` → read a **`ChpxFkp`** (§2.9.33) at - `aPnBteChpx[i].pn * 512` (fixed 512-byte page: `rgfc` boundaries, parallel - `rgb`, `crun` in the last byte). -4. Largest `j` with `rgfc[j] ≤ fc` → the `Chpx` (§2.9.32) at `rgb[j] * 2` within - the page. `Chpx.grpprl` is an array of `Prl` = `Sprm` (2 B) + operand. -5. Append `Pcd.Prm` mods (§2.9.214–216): `Prm0` (inline) or `Prm1` (index). - -`Prl`/`Sprm` is already modelled (`Sprm` with `ispmd/fSpec/sgc/spra` + -`operand_size()`); a **character** property is a SPRM with `sgc == 2` (note -`spra == 6` is length-prefixed/variable). - -**First cut — direct formatting only.** Implement §2.4.6.2 (`Chpx.grpprl` + -`Pcd.Prm`) and map the SPRMs above (bold/italic/size/font/colour applied directly -to runs). Resolve `sprmCRgFtc0` by reading **`SttbfFfn`** (§2.9.286) once and -indexing it. Drop the 11pt; use 10pt (the `sprmCHps` default). - -**Full fidelity — styles (later).** *Determining Formatting Properties* (§2.4.6.6) -layers: document defaults → `STSH` (§2.4.6.5) para/char-style `grpprl`s via the -paragraph's `istd` → table-style → direct paragraph → direct character. The first -cut skips STSH (style-dependent props fall back to defaults). - -**Wiring to the abstract model.** Per-run styling needs run boundaries in -`/WordDocument` byte offsets, so: -1. **Keep the FC↔text mapping** while concatenating (thread the `CharacterIndex` - through instead of discarding it after `body_text`). -2. **Split paragraphs into runs** at every `ChpxFkp` boundary, resolve each run's - `TextStyle`, emit a **`span`** (already wired via `SpanAdapter`) per run. -3. **Store the style** in a `TextStyle` side-map keyed by span id (mirror the text - side-payload + the `presentation` frame-payload) plus the font-name intern - store. `SpanAdapter::span_style` returns it; `text_style`/ - `paragraph_text_style` then return `{}` instead of the 11pt hack. +## 1. Character formatting fidelity + +Direct formatting (§2.4.6.2) is implemented — see the design notes above and +the read-path map below. Remaining layers of *Determining Formatting +Properties* (§2.4.6.6): + +- **`Pcd.Prm` modifications** (§2.9.214–216): per-piece property diffs — + `Prm0` (one inline SPRM) or `Prm1` (index into the Clx's `Prc` array, which + `read_character_index` currently skips). Rare for character properties, but + incremental saves can carry them. +- **STSH style layering** (§2.4.6.5): document defaults → paragraph/character + style `grpprl`s via the paragraph's `istd` → direct formatting. Without it, + style-derived properties (e.g. heading fonts defined only in the style + sheet, the usual Times New Roman default font) fall back to defaults; + `ToggleOperand` `0x80`/`0x81` resolve against "off". Character-formatting read path, keyed by `/WordDocument` byte offset `fc`: @@ -190,12 +170,12 @@ Character-formatting read path, keyed by `/WordDocument` byte offset `fc`: Table stream ├─ PlcBteChpx @ fcPlcfBteChpx §2.8.5 aFC[n+1] + aPnBteChpx[n] (PnFkpChpx, 4 B) ├─ SttbfFfn @ fcSttbfFfn (font names, FFN.xszFfn) §2.9.286 -└─ STSH @ fcStshf (styles — full fidelity only) §2.4.6.5 +└─ STSH @ fcStshf (styles — not read) §2.4.6.5 WordDocument stream └─ ChpxFkp @ aPnBteChpx[i].pn * 512 (512-byte page) §2.9.33 ├─ rgfc[crun+1] boundaries (stream offsets), rgb[crun] → Chpx @ rgb[j]*2, crun (last byte) - └─ Chpx = cb + grpprl(Prl[]) §2.9.32 Prl = Sprm(2 B) + operand; char SPRMs sgc==2; + Pcd.Prm §2.9.214–216 + └─ Chpx = cb + grpprl(Prl[]) §2.9.32 Prl = Sprm(2 B) + operand; char SPRMs sgc==2; + Pcd.Prm §2.9.214–216 (not read) ``` ## 2. Coverage gaps diff --git a/src/odr/internal/oldms/text/doc_document.cpp b/src/odr/internal/oldms/text/doc_document.cpp index 5aec89d2..b9719ddd 100644 --- a/src/odr/internal/oldms/text/doc_document.cpp +++ b/src/odr/internal/oldms/text/doc_document.cpp @@ -157,19 +157,12 @@ class ElementAdapter final : public abstract::ElementAdapter, } [[nodiscard]] TextStyle paragraph_text_style(const ElementIdentifier element_id) const override { - (void)element_id; - // TODO setting font size otherwise the text will be invisible. upstream - // this is used to make empty paragraphs works correctly - TextStyle style{}; - // TODO - style.font_size = Measure("11pt"); - return style; + return stored_style(element_id); } [[nodiscard]] TextStyle span_style(const ElementIdentifier element_id) const override { - (void)element_id; - return {}; // TODO + return stored_style(element_id); } [[nodiscard]] std::string @@ -184,13 +177,9 @@ class ElementAdapter final : public abstract::ElementAdapter, } [[nodiscard]] TextStyle text_style(const ElementIdentifier element_id) const override { + // The enclosing span carries the character style. (void)element_id; - // TODO setting font size otherwise the text will be invisible. upstream - // this is used to make empty paragraphs works correctly - TextStyle style{}; - // TODO - style.font_size = Measure("11pt"); - return style; + return {}; } private: @@ -198,6 +187,13 @@ class ElementAdapter final : public abstract::ElementAdapter, [[maybe_unused]] const Document *m_document{nullptr}; ElementRegistry *m_registry{nullptr}; + + /// The character style stored for a paragraph or span element. + [[nodiscard]] TextStyle + stored_style(const ElementIdentifier element_id) const { + const TextStyle *style = m_registry->element_style(element_id); + return style != nullptr ? *style : TextStyle{}; + } }; std::unique_ptr diff --git a/src/odr/internal/oldms/text/doc_element_registry.cpp b/src/odr/internal/oldms/text/doc_element_registry.cpp index 772fd677..f8ce1336 100644 --- a/src/odr/internal/oldms/text/doc_element_registry.cpp +++ b/src/odr/internal/oldms/text/doc_element_registry.cpp @@ -1,5 +1,6 @@ #include +#include #include namespace odr::internal::oldms::text { @@ -7,6 +8,8 @@ namespace odr::internal::oldms::text { void ElementRegistry::clear() noexcept { m_elements.clear(); m_texts.clear(); + m_styles.clear(); + m_font_names.clear(); } [[nodiscard]] std::size_t ElementRegistry::size() const noexcept { @@ -76,6 +79,26 @@ void ElementRegistry::append_child(const ElementIdentifier parent_id, element_at(parent_id).last_child_id = child_id; } +const char *ElementRegistry::intern_font_name(const std::string &name) { + if (const auto it = std::ranges::find(m_font_names, name); + it != m_font_names.end()) { + return it->c_str(); + } + return m_font_names.emplace_back(name).c_str(); +} + +void ElementRegistry::set_element_style(const ElementIdentifier id, + TextStyle style) { + check_element_id(id); + m_styles[id] = std::move(style); +} + +const TextStyle * +ElementRegistry::element_style(const ElementIdentifier id) const { + const auto it = m_styles.find(id); + return it != m_styles.end() ? &it->second : nullptr; +} + void ElementRegistry::check_element_id(const ElementIdentifier id) const { if (id == null_element_id) { throw std::out_of_range( diff --git a/src/odr/internal/oldms/text/doc_element_registry.hpp b/src/odr/internal/oldms/text/doc_element_registry.hpp index 41bcaea4..48fbe8d1 100644 --- a/src/odr/internal/oldms/text/doc_element_registry.hpp +++ b/src/odr/internal/oldms/text/doc_element_registry.hpp @@ -2,8 +2,11 @@ #include #include +#include +#include #include +#include #include #include @@ -39,9 +42,20 @@ class ElementRegistry final { void append_child(ElementIdentifier parent_id, ElementIdentifier child_id); + /// Stores a font name and returns a pointer that stays valid for the + /// registry's lifetime (`TextStyle::font_name` is a `const char *`). + const char *intern_font_name(const std::string &name); + + /// Character style of a span or paragraph element. + void set_element_style(ElementIdentifier id, TextStyle style); + [[nodiscard]] const TextStyle *element_style(ElementIdentifier id) const; + private: std::vector m_elements; std::unordered_map m_texts; + std::unordered_map m_styles; + /// Deque: elements never move, so interned `c_str()`s stay valid. + std::deque m_font_names; void check_element_id(ElementIdentifier id) const; void check_text_id(ElementIdentifier id) const; diff --git a/src/odr/internal/oldms/text/doc_helper.cpp b/src/odr/internal/oldms/text/doc_helper.cpp index 0cd437b1..468d56ad 100644 --- a/src/odr/internal/oldms/text/doc_helper.cpp +++ b/src/odr/internal/oldms/text/doc_helper.cpp @@ -3,8 +3,59 @@ #include #include #include +#include +#include +#include +#include #include +#include +#include + +namespace odr::internal::oldms::text { + +namespace { + +/// The Ico palette ([MS-DOC] 2.9.119); index 0 is cvAuto (no explicit color). +constexpr std::array ico_colors = { + 0x000000, 0x000000, 0x0000FF, 0x00FFFF, 0x00FF00, 0xFF00FF, + 0xFF0000, 0xFFFF00, 0xFFFFFF, 0x000080, 0x008080, 0x008000, + 0x800080, 0x800000, 0x808000, 0x808080, 0xC0C0C0}; + +std::optional ico_color(const std::uint8_t ico) { + if (ico >= ico_colors.size()) { + throw std::runtime_error("doc: Ico value out of range"); + } + if (ico == 0) { + return std::nullopt; // cvAuto + } + return Color(ico_colors[ico]); +} + +/// ToggleOperand ([MS-DOC] 2.9.327) against the (unmodelled) style value: +/// 0x80 matches the style, 0x81 inverts it; styles default to off. +bool toggle_on(const std::uint8_t value) { + switch (value) { + case 0x00: + case 0x80: + return false; + case 0x01: + case 0x81: + return true; + default: + throw std::runtime_error("doc: unexpected ToggleOperand value"); + } +} + +std::uint16_t read_u16(const std::string_view bytes, const std::size_t at) { + std::uint16_t value; + std::memcpy(&value, bytes.data() + at, sizeof(value)); + return value; +} + +} // namespace + +} // namespace odr::internal::oldms::text namespace odr::internal::oldms { @@ -32,4 +83,263 @@ text::CharacterIndex text::read_character_index(std::istream &in) { return result; } +text::CharacterStyles::CharacterStyles(TextStyle default_style) { + m_styles.push_back(std::move(default_style)); +} + +std::uint32_t text::CharacterStyles::add_style(TextStyle style) { + m_styles.push_back(std::move(style)); + return static_cast(m_styles.size() - 1); +} + +void text::CharacterStyles::append_run(const std::uint32_t begin_fc, + const std::uint32_t end_fc, + const std::uint32_t style_index) { + if (begin_fc >= end_fc || + (!m_runs.empty() && begin_fc < m_runs.back().end_fc)) { + throw std::runtime_error("doc: character runs must be ascending"); + } + if (style_index >= m_styles.size()) { + throw std::out_of_range("doc: style index out of range"); + } + if (!m_runs.empty() && m_runs.back().end_fc == begin_fc && + m_runs.back().style_index == style_index) { + m_runs.back().end_fc = end_fc; + return; + } + m_runs.push_back({begin_fc, end_fc, style_index}); +} + +std::uint32_t text::CharacterStyles::index_at(const std::uint32_t fc) const { + const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); + if (it == m_runs.begin()) { + return 0; + } + const Run &run = *std::prev(it); + return fc < run.end_fc ? run.style_index : 0; +} + +std::uint32_t text::CharacterStyles::chunk_end(const std::uint32_t fc) const { + const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); + if (it != m_runs.begin() && fc < std::prev(it)->end_fc) { + return std::prev(it)->end_fc; + } + if (it != m_runs.end()) { + return it->begin_fc; + } + return std::numeric_limits::max(); +} + +const TextStyle &text::CharacterStyles::style(const std::uint32_t index) const { + return m_styles.at(index); +} + +TextStyle +text::apply_character_sprms(TextStyle style, const std::string_view grpprl, + const std::vector &font_names) { + std::size_t at = 0; + while (at + sizeof(Sprm) <= grpprl.size()) { + Sprm sprm; + std::memcpy(&sprm, grpprl.data() + at, sizeof(sprm)); + const std::uint16_t opcode = std::bit_cast(sprm); + at += sizeof(sprm); + + std::size_t operand_size; + if (const int fixed_size = sprm.operand_size(); fixed_size >= 0) { + operand_size = static_cast(fixed_size); + } else { + // spra == 6: the first operand byte is the size of the remainder. The + // two SPRMs encoded differently (sprmTDefTable, sprmPChgTabs with + // cb == 0xFF) are not character properties and must not appear here. + if (at >= grpprl.size()) { + throw std::runtime_error("doc: Prl operand overruns grpprl"); + } + operand_size = static_cast(grpprl[at]); + ++at; + if (opcode == 0xD608 || (opcode == 0xC615 && operand_size == 0xFF)) { + throw std::runtime_error("doc: unexpected table SPRM in a Chpx"); + } + } + if (at + operand_size > grpprl.size()) { + throw std::runtime_error("doc: Prl operand overruns grpprl"); + } + const std::string_view operand = grpprl.substr(at, operand_size); + at += operand_size; + + switch (opcode) { + case sprmCFBold: + style.font_weight = toggle_on(static_cast(operand[0])) + ? FontWeight::bold + : FontWeight::normal; + break; + case sprmCFItalic: + style.font_style = toggle_on(static_cast(operand[0])) + ? FontStyle::italic + : FontStyle::normal; + break; + case sprmCFStrike: + style.font_line_through = + toggle_on(static_cast(operand[0])); + break; + case sprmCHighlight: + style.background_color = ico_color(static_cast(operand[0])); + break; + case sprmCKul: + style.font_underline = operand[0] != 0; + break; + case sprmCIco: + style.font_color = ico_color(static_cast(operand[0])); + break; + case sprmCHps: { + const std::uint16_t half_points = read_u16(operand, 0); + if (half_points < 2 || half_points > 3276) { + throw std::runtime_error("doc: sprmCHps value out of range"); + } + style.font_size = Measure(half_points / 2.0, DynamicUnit("pt")); + } break; + case sprmCRgFtc0: { + const auto ftc = static_cast(read_u16(operand, 0)); + if (ftc < 0 || static_cast(ftc) >= font_names.size()) { + throw std::runtime_error("doc: sprmCRgFtc0 font index out of range"); + } + style.font_name = font_names[static_cast(ftc)]; + } break; + case sprmCCv: { + // COLORREF ([MS-DOC] 2.9.43): red, green, blue, fAuto. + if (static_cast(operand[3]) == 0xFF) { + style.font_color = std::nullopt; // cvAuto + } else { + style.font_color = Color(static_cast(operand[0]), + static_cast(operand[1]), + static_cast(operand[2])); + } + } break; + default: + break; // not modelled + } + } + if (at != grpprl.size()) { + throw std::runtime_error("doc: grpprl does not hold whole Prls"); + } + + return style; +} + +std::vector text::read_font_names(std::istream &table_stream, + const FcLcb sttbf_ffn) { + std::vector result; + if (sttbf_ffn.lcb == 0) { + return result; + } + table_stream.seekg(sttbf_ffn.fc); + + // SttbfFfn is a non-extended STTB ([MS-DOC] 2.2.4, 2.9.286): u16 cData, + // u16 cbExtra (0), then per entry a u8 byte count and an FFN. + const auto c_data = util::byte_stream::read(table_stream); + if (c_data == 0xFFFF) { + throw std::runtime_error("doc: unexpected extended SttbfFfn"); + } + const auto cb_extra = util::byte_stream::read(table_stream); + + result.reserve(c_data); + for (std::uint16_t i = 0; i < c_data; ++i) { + const auto cch_data = util::byte_stream::read(table_stream); + if (cch_data < sizeof(FfnFixed)) { + throw std::runtime_error("doc: FFN too short"); + } + const auto ffn = util::byte_stream::read(table_stream); + (void)ffn; + + // xszFfn: null-terminated UTF-16 within the remaining bytes; an + // alternative name (xszAlt) may follow the terminator. + const std::size_t name_units = (cch_data - sizeof(FfnFixed)) / 2; + std::u16string name = read_string_uncompressed(table_stream, name_units); + if ((cch_data - sizeof(FfnFixed)) % 2 != 0) { + table_stream.ignore(1); + } + if (const std::size_t nul = name.find(u'\0'); nul != std::u16string::npos) { + name.resize(nul); + } + result.push_back(util::string::u16string_to_string(name)); + + table_stream.ignore(cb_extra); + } + + return result; +} + +text::CharacterStyles text::read_character_styles( + std::istream &document_stream, std::istream &table_stream, + const FcLcb plcf_bte_chpx, const TextStyle &default_style, + const std::vector &font_names) { + CharacterStyles result(default_style); + if (plcf_bte_chpx.lcb == 0) { + return result; + } + + table_stream.seekg(plcf_bte_chpx.fc); + std::string plc_bytes = util::stream::read(table_stream, plcf_bte_chpx.lcb); + const PlcBteChpxMap plc(plc_bytes.data(), plc_bytes.size()); + + // One resolved style per distinct Chpx byte sequence. + std::unordered_map style_cache; + const auto resolve = [&](const std::string_view grpprl) -> std::uint32_t { + if (grpprl.empty()) { + return 0; + } + const auto it = style_cache.find(std::string(grpprl)); + if (it != style_cache.end()) { + return it->second; + } + const std::uint32_t index = result.add_style( + apply_character_sprms(default_style, grpprl, font_names)); + style_cache.emplace(std::string(grpprl), index); + return index; + }; + + for (std::uint32_t i = 0; i < plc.n(); ++i) { + // A 512-byte ChpxFkp page ([MS-DOC] 2.9.33) at pn * 512: crun in the last + // byte, crun+1 rgfc offsets, crun rgb bytes; rgb[j] * 2 locates the Chpx + // within the page, 0 means default properties. + std::array page; + document_stream.seekg(static_cast(plc.aData(i).pn) * 512); + document_stream.read(page.data(), page.size()); + if (!document_stream) { + throw std::runtime_error("doc: truncated ChpxFkp"); + } + + const auto crun = static_cast(page[511]); + if (crun < 0x01 || crun > 0x65) { + throw std::runtime_error("doc: ChpxFkp crun out of range"); + } + + for (std::uint8_t j = 0; j < crun; ++j) { + std::uint32_t begin_fc; + std::uint32_t end_fc; + std::memcpy(&begin_fc, page.data() + static_cast(j) * 4, 4); + std::memcpy(&end_fc, page.data() + static_cast(j + 1) * 4, + 4); + + const auto rgb = + static_cast(page[(crun + 1) * 4 + std::size_t{j}]); + std::string_view grpprl; + if (rgb != 0) { + const std::size_t chpx_at = static_cast(rgb) * 2; + if (chpx_at >= page.size() - 1) { + throw std::runtime_error("doc: Chpx offset out of range"); + } + const auto cb = static_cast(page[chpx_at]); + if (chpx_at + 1 + cb > page.size() - 1) { + throw std::runtime_error("doc: Chpx overruns its ChpxFkp"); + } + grpprl = std::string_view(page.data() + chpx_at + 1, cb); + } + + result.append_run(begin_fc, end_fc, resolve(grpprl)); + } + } + + return result; +} + } // namespace odr::internal::oldms diff --git a/src/odr/internal/oldms/text/doc_helper.hpp b/src/odr/internal/oldms/text/doc_helper.hpp index 62e346cd..537d4941 100644 --- a/src/odr/internal/oldms/text/doc_helper.hpp +++ b/src/odr/internal/oldms/text/doc_helper.hpp @@ -1,9 +1,15 @@ #pragma once +#include + +#include + #include #include #include #include +#include +#include #include namespace odr::internal::oldms::text { @@ -106,4 +112,54 @@ class CharacterIndex { CharacterIndex read_character_index(std::istream &in); +/// The document's character-formatting runs, keyed by WordDocument-stream +/// offset (fc). Style index 0 is the default style; offsets outside any run +/// resolve to it. +class CharacterStyles final { +public: + explicit CharacterStyles(TextStyle default_style); + + std::uint32_t add_style(TextStyle style); + /// Appends a run; runs must be ascending and non-overlapping. Adjacent runs + /// with the same style index are merged. + void append_run(std::uint32_t begin_fc, std::uint32_t end_fc, + std::uint32_t style_index); + + /// The style index of the run containing `fc` (0 if none). + [[nodiscard]] std::uint32_t index_at(std::uint32_t fc) const; + /// End offset of the run containing `fc`, or the next run's begin, so text + /// can be walked in style-uniform chunks; UINT32_MAX past the last run. + [[nodiscard]] std::uint32_t chunk_end(std::uint32_t fc) const; + [[nodiscard]] const TextStyle &style(std::uint32_t index) const; + +private: + struct Run { + std::uint32_t begin_fc; + std::uint32_t end_fc; + std::uint32_t style_index; + }; + + std::vector m_styles; + std::vector m_runs; +}; + +/// Applies the character SPRMs of a Chpx grpprl ([MS-DOC] 2.6.1) on top of +/// `style`; non-character SPRMs are skipped via their operand size. +/// `font_names` resolves sprmCRgFtc0 (pointers must outlive the style). +TextStyle apply_character_sprms(TextStyle style, std::string_view grpprl, + const std::vector &font_names); + +/// Reads the font names (FFN.xszFfn) of the SttbfFfn ([MS-DOC] 2.9.286) at +/// `fc` in the table stream; empty for lcb == 0. +std::vector read_font_names(std::istream &table_stream, + FcLcb sttbf_ffn); + +/// Reads the PlcBteChpx at `fc` in the table stream and every referenced +/// ChpxFkp page ([MS-DOC] 2.9.33) from the WordDocument stream, resolving +/// each run's Chpx against `default_style`. Equal Chpx bytes share one style. +CharacterStyles +read_character_styles(std::istream &document_stream, std::istream &table_stream, + FcLcb plcf_bte_chpx, const TextStyle &default_style, + const std::vector &font_names); + } // namespace odr::internal::oldms::text diff --git a/src/odr/internal/oldms/text/doc_parser.cpp b/src/odr/internal/oldms/text/doc_parser.cpp index c2f106cd..25f150f2 100644 --- a/src/odr/internal/oldms/text/doc_parser.cpp +++ b/src/odr/internal/oldms/text/doc_parser.cpp @@ -8,10 +8,13 @@ #include #include #include -#include #include +#include +#include +#include #include +#include #include namespace odr::internal::oldms { @@ -29,63 +32,125 @@ constexpr char page_break_mark = '\x0C'; // visible text. A field is 0x13 begin ... [0x14 separator] ... 0x15 end // ([MS-DOC] 2.8.25): the instruction (begin..separator) is hidden, the result // (separator..end) shown. A separator-less field is hidden entirely. -std::string clean_text(const std::string &in) { - std::string out; - out.reserve(in.size()); +// Stateful: a field can span style runs and paragraphs, so one cleaner +// processes the whole body in order. +class TextCleaner { +public: + std::string clean(const std::string_view in) { + std::string out; + out.reserve(in.size()); - // Open fields; entry is true while still in that field's instruction part. - // instruction_depth counts those; text is hidden whenever it is > 0. - std::vector field_in_instruction; - int instruction_depth = 0; - - for (const char c : in) { - switch (c) { - case '\x13': // field begin: a new field opens in its instruction part - field_in_instruction.push_back(true); - ++instruction_depth; - continue; - case '\x14': // field separator: the innermost field's instruction ends - if (!field_in_instruction.empty() && field_in_instruction.back()) { - field_in_instruction.back() = false; - --instruction_depth; + for (const char c : in) { + switch (c) { + case '\x13': // field begin: a new field opens in its instruction part + m_field_in_instruction.push_back(true); + ++m_instruction_depth; + continue; + case '\x14': // field separator: the innermost field's instruction ends + if (!m_field_in_instruction.empty() && m_field_in_instruction.back()) { + m_field_in_instruction.back() = false; + --m_instruction_depth; + } + continue; + case '\x15': // field end: close the innermost field + if (!m_field_in_instruction.empty()) { + if (m_field_in_instruction.back()) { + // separator-less field: its instruction part ends here, not + // earlier + --m_instruction_depth; + } + m_field_in_instruction.pop_back(); + } + continue; + default: + break; } - continue; - case '\x15': // field end: close the innermost field - if (!field_in_instruction.empty()) { - if (field_in_instruction.back()) { - // separator-less field: its instruction part ends here, not earlier - --instruction_depth; + + if (m_instruction_depth > 0) { + continue; // drop the hidden field instruction + } + + switch (c) { + case '\x09': // tab: keep as real text + out.push_back(c); + break; + case '\x1E': // non-breaking hyphen + out.push_back('-'); + break; + case '\x1F': // optional hyphen: drop + break; + default: + // Drop remaining control/anchor characters (< 0x20); keep the rest. + if (static_cast(c) >= 0x20) { + out.push_back(c); } - field_in_instruction.pop_back(); + break; } - continue; - default: - break; } - if (instruction_depth > 0) { - continue; // drop the hidden field instruction - } + return out; + } - switch (c) { - case '\x09': // tab: keep as real text - out.push_back(c); - break; - case '\x1E': // non-breaking hyphen - out.push_back('-'); - break; - case '\x1F': // optional hyphen: drop +private: + // Open fields; entry is true while still in that field's instruction part. + // m_instruction_depth counts those; text is hidden whenever it is > 0. + std::vector m_field_in_instruction; + int m_instruction_depth{0}; +}; + +/// A maximal piece of body text with uniform character formatting. +struct StyledRun { + std::string text; + std::uint32_t style_index{0}; +}; + +/// Decodes the main-body pieces (the first `ccp_text` CPs), split further at +/// every character-formatting boundary. +std::vector decode_styled_runs( + std::istream &document_stream, const text::CharacterIndex &character_index, + const text::CharacterStyles &styles, const std::size_t ccp_text) { + std::vector runs; + std::size_t consumed_cp = 0; + + for (const auto &entry : character_index) { + if (consumed_cp >= ccp_text) { break; - default: - // Drop remaining control/anchor characters (< 0x20); keep the rest. - if (static_cast(c) >= 0x20) { - out.push_back(c); + } + const std::size_t take = std::min(entry.length_cp, ccp_text - consumed_cp); + const std::size_t bytes_per_cp = entry.is_compressed ? 1 : 2; + + std::size_t cp = 0; + while (cp < take) { + const auto fc = + static_cast(entry.data_offset + cp * bytes_per_cp); + const std::uint32_t style_index = styles.index_at(fc); + + // CPs of this piece still in the same style run; a boundary that falls + // inside a 2-byte CP is pushed past it. + const std::uint32_t chunk_end_fc = styles.chunk_end(fc); + std::size_t chunk_cp = take - cp; + if (chunk_end_fc != std::numeric_limits::max()) { + chunk_cp = std::min( + chunk_cp, + std::max(1, (chunk_end_fc - fc + bytes_per_cp - 1) / + bytes_per_cp)); } - break; + + document_stream.seekg(fc); + std::string chunk_text = + text::read_string(document_stream, chunk_cp, entry.is_compressed); + if (!runs.empty() && runs.back().style_index == style_index) { + runs.back().text += chunk_text; + } else { + runs.push_back({std::move(chunk_text), style_index}); + } + cp += chunk_cp; } + + consumed_cp += take; } - return out; + return runs; } } // namespace @@ -100,63 +165,91 @@ ElementIdentifier text::parse_tree(ElementRegistry ®istry, const std::string tableStreamPath = fib.base.fWhichTblStm == 1 ? "/1Table" : "/0Table"; - const auto table_stream = files.open(AbsPath(tableStreamPath))->stream(); - table_stream->ignore(fib.fibRgFcLcb->clx.fc); + // Font table, interned so TextStyle::font_name pointers stay valid. + std::vector font_names; + for (const std::string &name : + read_font_names(*table_stream, fib.fibRgFcLcb->sttbfFfn)) { + font_names.push_back(registry.intern_font_name(name)); + } + + // Direct character formatting ([MS-DOC] 2.4.6.2); Pcd.Prm modifications are + // not modelled. Without sprmCHps the size defaults to 20 half-points. + TextStyle default_style; + default_style.font_size = Measure(10.0, DynamicUnit("pt")); + const CharacterStyles styles = read_character_styles( + *document_stream, *table_stream, fib.fibRgFcLcb->plcfBteChpx, + default_style, font_names); + + table_stream->seekg(fib.fibRgFcLcb->clx.fc); const CharacterIndex character_index = read_character_index(*table_stream); - // Concatenate the decoded pieces of the main document body only (the first - // ccpText CPs). Pieces are in ascending CP order, so clamp each piece to the - // remaining budget and stop once it is exhausted. - const std::size_t ccp_text = fib.ccpText(); - std::size_t consumed_cp = 0; - std::string body_text; - for (const auto &entry : character_index) { - if (consumed_cp >= ccp_text) { - break; - } - const std::size_t take = std::min(entry.length_cp, ccp_text - consumed_cp); - document_stream->seekg(entry.data_offset); - body_text += read_string(*document_stream, take, entry.is_compressed); - consumed_cp += take; - } + const auto ccp_text = static_cast(fib.ccpText()); + const std::vector runs = + decode_styled_runs(*document_stream, character_index, styles, ccp_text); - // Split into paragraphs. The main body always ends with a guard paragraph - // mark, so drop the resulting trailing empty paragraph. - auto paragraphs = - util::string::split(body_text, std::string(1, paragraph_mark)); - if (!paragraphs.empty() && paragraphs.back().empty()) { - paragraphs.pop_back(); - } + // Build the tree: paragraphs are opened lazily so the trailing guard + // paragraph mark does not produce an extra empty paragraph. Each paragraph + // and span stores its style; empty paragraphs keep their height through the + // paragraph style. + TextCleaner cleaner; + ElementIdentifier paragraph_id = null_element_id; - for (const auto ¶graph : paragraphs) { - // A paragraph may contain end-of-section / manual page-break characters - // (0x0C): split on them and mark each boundary with a page_break element. - const auto pages = - util::string::split(paragraph, std::string(1, page_break_mark)); - for (std::size_t page_i = 0; page_i < pages.size(); ++page_i) { - if (page_i > 0) { - auto [page_break_id, _] = - registry.create_element(ElementType::page_break); - registry.append_child(root_id, page_break_id); - } + const auto ensure_paragraph = [&](const std::uint32_t style_index) { + if (paragraph_id == null_element_id) { + auto [id, paragraph] = registry.create_element(ElementType::paragraph); + registry.append_child(root_id, id); + registry.set_element_style(id, styles.style(style_index)); + paragraph_id = id; + } + }; - auto [paragraph_id, _] = registry.create_element(ElementType::paragraph); - registry.append_child(root_id, paragraph_id); + for (const StyledRun &run : runs) { + std::size_t at = 0; + while (at <= run.text.size()) { + const std::size_t control = run.text.find_first_of("\r\x0B\x0C", at); + const std::size_t segment_end = + control == std::string::npos ? run.text.size() : control; - const auto lines = - util::string::split(pages[page_i], std::string(1, line_break_mark)); - for (std::uint32_t line_i = 0; line_i < lines.size(); ++line_i) { - if (line_i > 0) { - auto [line_id, _] = registry.create_element(ElementType::line_break); - registry.append_child(paragraph_id, line_id); - } + if (std::string cleaned = cleaner.clean( + std::string_view(run.text).substr(at, segment_end - at)); + !cleaned.empty()) { + ensure_paragraph(run.style_index); + auto [span_id, span] = registry.create_element(ElementType::span); + registry.set_element_style(span_id, styles.style(run.style_index)); + registry.append_child(paragraph_id, span_id); + + auto [text_id, text_element, text_entry] = + registry.create_text_element(); + text_entry.text = std::move(cleaned); + registry.append_child(span_id, text_id); + } - auto [text_id, _, text_element] = registry.create_text_element(); - text_element.text = clean_text(lines[line_i]); - registry.append_child(paragraph_id, text_id); + if (control == std::string::npos) { + break; + } + switch (run.text[control]) { + case paragraph_mark: + ensure_paragraph(run.style_index); + paragraph_id = null_element_id; + break; + case page_break_mark: { + ensure_paragraph(run.style_index); + paragraph_id = null_element_id; + auto [page_break_id, page_break] = + registry.create_element(ElementType::page_break); + registry.append_child(root_id, page_break_id); + } break; + case line_break_mark: { + ensure_paragraph(run.style_index); + auto [line_id, line] = registry.create_element(ElementType::line_break); + registry.append_child(paragraph_id, line_id); + } break; + default: + break; } + at = control + 1; } } diff --git a/src/odr/internal/oldms/text/doc_structs.hpp b/src/odr/internal/oldms/text/doc_structs.hpp index 1cb3b4a3..8731f822 100644 --- a/src/odr/internal/oldms/text/doc_structs.hpp +++ b/src/odr/internal/oldms/text/doc_structs.hpp @@ -288,6 +288,20 @@ struct FibRgCswNewData2007 : FibRgCswNewData2000 { static_assert(sizeof(FibRgCswNewData2007) == 8, "FibRgCswNewData2007 should be 8 bytes"); +/// The character SPRMs mapped to TextStyle ([MS-DOC] 2.6.1); everything else +/// in a Chpx is skipped via Sprm::operand_size. +enum CharacterSprms : std::uint16_t { + sprmCFBold = 0x0835, //< ToggleOperand + sprmCFItalic = 0x0836, //< ToggleOperand + sprmCFStrike = 0x0837, //< ToggleOperand + sprmCHighlight = 0x2A0C, //< Ico + sprmCKul = 0x2A3E, //< Kul; 0x00 = none + sprmCIco = 0x2A42, //< Ico (legacy palette color) + sprmCHps = 0x4A43, //< u16 half-points (default 20) + sprmCRgFtc0 = 0x4A4F, //< s16 index into SttbfFfn + sprmCCv = 0x6870, //< COLORREF +}; + struct Sprm { std::uint16_t ispmd : 9; std::uint16_t fSpec : 1; @@ -333,6 +347,26 @@ struct Pcd { }; static_assert(sizeof(Pcd) == 8, "Pcd should be 8 bytes"); +/// Location of a ChpxFkp in the WordDocument stream, at pn * 512 +/// ([MS-DOC] 2.9.206). +struct PnFkpChpx { + std::uint32_t pn : 22; + std::uint32_t unused : 10; +}; +static_assert(sizeof(PnFkpChpx) == 4, "PnFkpChpx should be 4 bytes"); + +/// Fixed head of an FFN ([MS-DOC] 2.9.82); the null-terminated UTF-16 font +/// name (xszFfn) follows. +struct FfnFixed { + std::uint8_t ffid; + std::int16_t wWeight; + std::uint8_t chs; + std::uint8_t ixchSzAlt; + std::array panose; + std::array fs; +}; +static_assert(sizeof(FfnFixed) == 39, "FfnFixed should be 39 bytes"); + #pragma pack(pop) struct ParsedFibRgCswNew { @@ -409,4 +443,19 @@ class PlcPcdMap : public PlcPcdBase { std::size_t m_cbPlc{0}; }; +/// PlcBteChpx ([MS-DOC] 2.8.5): a PLC keyed by WordDocument-stream offsets +/// (fc, not CP) whose data elements locate the ChpxFkp pages. +class PlcBteChpxMap : public PlcBase { +public: + PlcBteChpxMap(char *data, const std::size_t cbPlc) + : m_data(data), m_cbPlc(cbPlc) {} + + [[nodiscard]] char *data() const { return m_data; } + [[nodiscard]] std::size_t cbPlc() const { return m_cbPlc; } + +private: + char *m_data{nullptr}; + std::size_t m_cbPlc{0}; +}; + } // namespace odr::internal::oldms::text diff --git a/test/src/internal/oldms/oldms_test.cpp b/test/src/internal/oldms/oldms_test.cpp index f414dcbe..b282d5ac 100644 --- a/test/src/internal/oldms/oldms_test.cpp +++ b/test/src/internal/oldms/oldms_test.cpp @@ -8,11 +8,19 @@ #include #include +#include + +#include +#include +#include #include +#include +#include #include #include #include +#include #include #include #include @@ -57,6 +65,89 @@ TEST(OldMs, doc_read_string_compressed) { namespace { +void append_doc_u16(std::string &out, const std::uint16_t value) { + out.push_back(static_cast(value & 0xFF)); + out.push_back(static_cast(value >> 8)); +} + +void append_doc_u32(std::string &out, const std::uint32_t value) { + append_doc_u16(out, static_cast(value & 0xFFFF)); + append_doc_u16(out, static_cast(value >> 16)); +} + +std::string doc_prl(const std::uint16_t opcode, const std::string &operand) { + std::string prl; + append_doc_u16(prl, opcode); + prl += operand; + return prl; +} + +/// A minimal FIB ([MS-DOC] 2.5.1): nFib97, table stream /1Table, with the +/// three FcLcbs the parser reads (clx, plcfBteChpx, sttbfFfn). +std::string make_fib(const std::uint32_t ccp_text, + const internal::oldms::text::FcLcb clx, + const internal::oldms::text::FcLcb plcf_bte_chpx, + const internal::oldms::text::FcLcb sttbf_ffn) { + std::string fib; + // FibBase (32 bytes) + append_doc_u16(fib, 0xA5EC); // wIdent + append_doc_u16(fib, 0x00C1); // nFib97 + append_doc_u16(fib, 0); // unused + append_doc_u16(fib, 0x0409); // lid + append_doc_u16(fib, 0); // pnNext + append_doc_u16(fib, 0x0200); // flags: fWhichTblStm = 1 + append_doc_u16(fib, 0x00C1); // nFibBack + append_doc_u32(fib, 0); // lKey + fib += std::string(2, '\0'); // envr, flags + fib += std::string(12, '\0'); // reserved3-6 + // csw + fibRgW + append_doc_u16(fib, 14); + fib += std::string(28, '\0'); + // cslw + fibRgLw; ccpText is the 4th 32-bit field + append_doc_u16(fib, 22); + std::string rg_lw(88, '\0'); + std::string ccp; + append_doc_u32(ccp, ccp_text); + rg_lw.replace(12, 4, ccp); + fib += rg_lw; + // cbRgFcLcb + FibRgFcLcb97: plcfBteChpx is FcLcb index 12, sttbfFfn 15, + // clx 33 + append_doc_u16(fib, 93); + std::string rg(744, '\0'); + const auto put_fclcb = [&rg](const std::size_t index, + const internal::oldms::text::FcLcb value) { + std::string bytes; + append_doc_u32(bytes, value.fc); + append_doc_u32(bytes, value.lcb); + rg.replace(index * 8, 8, bytes); + }; + put_fclcb(12, plcf_bte_chpx); + put_fclcb(15, sttbf_ffn); + put_fclcb(33, clx); + fib += rg; + // cswNew + append_doc_u16(fib, 0); + return fib; +} + +/// An SttbfFfn ([MS-DOC] 2.9.286) with the given font names. +std::string make_sttbf_ffn(const std::vector &names) { + std::string sttb; + append_doc_u16(sttb, static_cast(names.size())); // cData + append_doc_u16(sttb, 0); // cbExtra + for (const std::string &name : names) { + std::string ffn(39, '\0'); // FfnFixed + for (const char c : name) { + ffn.push_back(c); + ffn.push_back('\0'); + } + ffn += std::string(2, '\0'); // terminator + sttb.push_back(static_cast(ffn.size())); + sttb += ffn; + } + return sttb; +} + // Collects the text content of an element subtree, joining a slide's paragraphs // with newlines so the paragraph structure is observable. std::string collect_text(const Element element) { @@ -77,6 +168,179 @@ std::string collect_text(const Element element) { } // namespace +// Character SPRMs ([MS-DOC] 2.6.1) map onto TextStyle; unknown SPRMs are +// skipped via their operand size, malformed grpprls throw. +TEST(OldMs, doc_apply_character_sprms) { + using internal::oldms::text::apply_character_sprms; + + TextStyle base; + base.font_size = Measure("10pt"); + const std::vector fonts = {"Arial", "Courier New"}; + + { + std::string grpprl; + grpprl += doc_prl(0x0835, "\x01"); // bold: on + grpprl += doc_prl(0x0836, "\x81"); // italic: invert style default -> on + grpprl += doc_prl(0x0837, "\x80"); // strike: match style default -> off + grpprl += doc_prl(0x2A3E, "\x01"); // single underline + grpprl += doc_prl(0x4A43, std::string("\x20\x00", 2)); // 32 half-points + grpprl += doc_prl(0x4A4F, std::string("\x01\x00", 2)); // font index 1 + grpprl += doc_prl(0x6870, std::string("\x11\x22\x33\x00", 4)); // COLORREF + grpprl += doc_prl(0x2A0C, "\x07"); // highlight: yellow + const TextStyle style = apply_character_sprms(base, grpprl, fonts); + EXPECT_EQ(style.font_weight, FontWeight::bold); + EXPECT_EQ(style.font_style, FontStyle::italic); + EXPECT_EQ(style.font_line_through, false); + EXPECT_EQ(style.font_underline, true); + EXPECT_EQ(style.font_size, Measure("16pt")); + EXPECT_STREQ(style.font_name, "Courier New"); + ASSERT_TRUE(style.font_color.has_value()); + EXPECT_EQ(style.font_color->rgb(), 0x112233u); + ASSERT_TRUE(style.background_color.has_value()); + EXPECT_EQ(style.background_color->rgb(), 0xFFFF00u); + } + + { + // Legacy palette color; unknown fixed-size and variable-size SPRMs are + // skipped without disturbing later Prls. + std::string grpprl; + grpprl += doc_prl(0x2A42, "\x06"); // ico: red + grpprl += doc_prl(0x0838, "\x01"); // unknown, 1 B + grpprl += doc_prl(0xC875, std::string("\x02\xAA\xBB", 3)); // unknown, var + grpprl += doc_prl(0x0835, "\x01"); // bold: on + const TextStyle style = apply_character_sprms(base, grpprl, fonts); + ASSERT_TRUE(style.font_color.has_value()); + EXPECT_EQ(style.font_color->rgb(), 0xFF0000u); + EXPECT_EQ(style.font_weight, FontWeight::bold); + } + + { + // cvAuto ([MS-DOC] 2.9.43) resets an earlier explicit color. + TextStyle colored = base; + colored.font_color = Color(std::uint32_t{0x123456}); + const TextStyle style = apply_character_sprms( + colored, doc_prl(0x6870, std::string("\x00\x00\x00\xFF", 4)), fonts); + EXPECT_FALSE(style.font_color.has_value()); + } + + // Truncated operand, half a Sprm, and an out-of-range sprmCHps all throw. + EXPECT_THROW(apply_character_sprms(base, doc_prl(0x4A43, "\x01"), fonts), + std::runtime_error); + EXPECT_THROW(apply_character_sprms(base, std::string("\x35", 1), fonts), + std::runtime_error); + EXPECT_THROW(apply_character_sprms( + base, doc_prl(0x4A43, std::string("\x01\x00", 2)), fonts), + std::runtime_error); +} + +TEST(OldMs, doc_read_font_names) { + const std::string sttb = make_sttbf_ffn({"Arial", "Times New Roman"}); + std::istringstream in(sttb); + const std::vector names = internal::oldms::text::read_font_names( + in, {0, static_cast(sttb.size())}); + ASSERT_EQ(names.size(), 2); + EXPECT_EQ(names[0], "Arial"); + EXPECT_EQ(names[1], "Times New Roman"); +} + +// End-to-end over a synthetic .doc: the PlcBteChpx/ChpxFkp run boundaries +// split the text into styled spans ([MS-DOC] 2.4.6.2), the default run keeps +// the 10pt default, and the paragraph carries a style for empty-height. +TEST(OldMs, doc_character_formatting) { + const std::string body = "plain bold\r"; + constexpr std::uint32_t text_fc = 1536; + + // ChpxFkp page ([MS-DOC] 2.9.33) at pn 2: run "plain " -> no Chpx (rgb 0), + // run "bold\r" -> bold + font 0. + std::string fkp(512, '\0'); + { + std::string head; + append_doc_u32(head, text_fc); + append_doc_u32(head, text_fc + 6); + append_doc_u32(head, text_fc + 11); + head.push_back('\0'); // rgb[0]: default properties + head.push_back(static_cast(240)); // rgb[1]: Chpx at byte 480 + fkp.replace(0, head.size(), head); + + std::string chpx; + chpx += doc_prl(0x0835, "\x01"); // bold + chpx += doc_prl(0x4A4F, std::string("\x00\x00", 2)); // font index 0 + fkp[480] = static_cast(chpx.size()); + fkp.replace(481, chpx.size(), chpx); + fkp[511] = 2; // crun + } + + // Table stream: Clx at 0 (one compressed piece), SttbfFfn at 64, + // PlcBteChpx at 128. + std::string plc_pcd; + append_doc_u32(plc_pcd, 0); + append_doc_u32(plc_pcd, 11); + append_doc_u16(plc_pcd, 0); // Pcd flags + append_doc_u32(plc_pcd, (1u << 30) | (text_fc * 2u)); // fCompressed | fc + append_doc_u16(plc_pcd, 0); // prm + std::string clx; + clx.push_back('\x02'); + append_doc_u32(clx, static_cast(plc_pcd.size())); + clx += plc_pcd; + + const std::string sttbf_ffn = make_sttbf_ffn({"Arial"}); + + std::string plc_bte; + append_doc_u32(plc_bte, text_fc); + append_doc_u32(plc_bte, text_fc + 11); + append_doc_u32(plc_bte, 2); // PnFkpChpx: page at 2 * 512 + + std::string table(256, '\0'); + table.replace(0, clx.size(), clx); + table.replace(64, sttbf_ffn.size(), sttbf_ffn); + table.replace(128, plc_bte.size(), plc_bte); + + std::string word_document = + make_fib(11, {0, static_cast(clx.size())}, + {128, static_cast(plc_bte.size())}, + {64, static_cast(sttbf_ffn.size())}); + word_document.resize(1024, '\0'); + word_document += fkp; + word_document += body; + + auto files = std::make_shared(); + files->copy(std::make_shared(word_document), + internal::AbsPath("/WordDocument")); + files->copy(std::make_shared(table), + internal::AbsPath("/1Table")); + const Document document( + std::make_shared(files)); + + std::vector paragraphs; + for (const Element child : document.root_element().children()) { + paragraphs.push_back(child); + } + ASSERT_EQ(paragraphs.size(), 1); + ASSERT_EQ(paragraphs[0].type(), ElementType::paragraph); + + std::vector spans; + for (const Element child : paragraphs[0].children()) { + ASSERT_EQ(child.type(), ElementType::span); + spans.push_back(child); + } + ASSERT_EQ(spans.size(), 2); + + EXPECT_EQ(collect_text(spans[0]), "plain "); + const TextStyle plain = spans[0].as_span().style(); + EXPECT_EQ(plain.font_size, Measure("10pt")); + EXPECT_FALSE(plain.font_weight.has_value()); + EXPECT_EQ(plain.font_name, nullptr); + + EXPECT_EQ(collect_text(spans[1]), "bold"); + const TextStyle bold = spans[1].as_span().style(); + EXPECT_EQ(bold.font_weight, FontWeight::bold); + EXPECT_STREQ(bold.font_name, "Arial"); + EXPECT_EQ(bold.font_size, Measure("10pt")); + + EXPECT_EQ(paragraphs[0].as_paragraph().text_style().font_size, + Measure("10pt")); +} + namespace { void append_u16(std::string &out, const std::uint16_t value) { From 7277797bcbdb26fdc0b4c7142fba927e9c0ec6d1 Mon Sep 17 00:00:00 2001 From: Andreas Stefl Date: Thu, 16 Jul 2026 21:26:21 +0200 Subject: [PATCH 2/2] fix(doc): allow empty font table for sprmCRgFtc0 Per [MS-DOC] 2.6.1 an sprmCRgFtc0 operand of 0 is valid when SttbfFfn has no entries; the style-sheet default font (unmodelled) applies, so leave the font unset instead of throwing. Genuinely out-of-range indexes still throw. Addresses Codex review feedback on PR #608. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01AEvWWwe1EyeWH7ANHoAvPH --- src/odr/internal/oldms/text/doc_helper.cpp | 10 ++++++++-- test/src/internal/oldms/oldms_test.cpp | 21 +++++++++++++++++++++ 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/src/odr/internal/oldms/text/doc_helper.cpp b/src/odr/internal/oldms/text/doc_helper.cpp index 468d56ad..5bd278b5 100644 --- a/src/odr/internal/oldms/text/doc_helper.cpp +++ b/src/odr/internal/oldms/text/doc_helper.cpp @@ -198,11 +198,17 @@ text::apply_character_sprms(TextStyle style, const std::string_view grpprl, style.font_size = Measure(half_points / 2.0, DynamicUnit("pt")); } break; case sprmCRgFtc0: { + // SttbfFfn index ([MS-DOC] 2.6.1); with 0 entries the value MUST be 0 + // and the (unmodelled) style-sheet default font applies — leave unset. const auto ftc = static_cast(read_u16(operand, 0)); - if (ftc < 0 || static_cast(ftc) >= font_names.size()) { + if (ftc < 0 || (font_names.empty() ? ftc != 0 + : static_cast(ftc) >= + font_names.size())) { throw std::runtime_error("doc: sprmCRgFtc0 font index out of range"); } - style.font_name = font_names[static_cast(ftc)]; + if (!font_names.empty()) { + style.font_name = font_names[static_cast(ftc)]; + } } break; case sprmCCv: { // COLORREF ([MS-DOC] 2.9.43): red, green, blue, fAuto. diff --git a/test/src/internal/oldms/oldms_test.cpp b/test/src/internal/oldms/oldms_test.cpp index b282d5ac..d5df9b67 100644 --- a/test/src/internal/oldms/oldms_test.cpp +++ b/test/src/internal/oldms/oldms_test.cpp @@ -223,6 +223,27 @@ TEST(OldMs, doc_apply_character_sprms) { EXPECT_FALSE(style.font_color.has_value()); } + { + // sprmCRgFtc0 with an empty SttbfFfn: 0 is valid ([MS-DOC] 2.6.1) and + // leaves the font unset; anything else is out of range. + const std::vector no_fonts; + const TextStyle style = apply_character_sprms( + base, doc_prl(0x4A4F, std::string("\x00\x00", 2)), no_fonts); + EXPECT_EQ(style.font_name, nullptr); + EXPECT_THROW( + apply_character_sprms(base, doc_prl(0x4A4F, std::string("\x01\x00", 2)), + no_fonts), + std::runtime_error); + } + + // Out-of-range and negative sprmCRgFtc0 indexes throw. + EXPECT_THROW(apply_character_sprms( + base, doc_prl(0x4A4F, std::string("\x02\x00", 2)), fonts), + std::runtime_error); + EXPECT_THROW(apply_character_sprms( + base, doc_prl(0x4A4F, std::string("\xFF\xFF", 2)), fonts), + std::runtime_error); + // Truncated operand, half a Sprm, and an out-of-range sprmCHps all throw. EXPECT_THROW(apply_character_sprms(base, doc_prl(0x4A43, "\x01"), fonts), std::runtime_error);