diff --git a/src/odr/internal/oldms/text/AGENTS.md b/src/odr/internal/oldms/text/AGENTS.md index ab513d25..7cf20fcc 100644 --- a/src/odr/internal/oldms/text/AGENTS.md +++ b/src/odr/internal/oldms/text/AGENTS.md @@ -4,24 +4,26 @@ The **why** and the roadmap; what is concretely done lives in the code. Shared `oldms/` conventions are in [`../AGENTS.md`](../AGENTS.md). **Scope.** Extract the **visible text of the main document body**, split into -paragraphs and manual line breaks, through the abstract model so the generic HTML -renderer lays it out as a flat run of paragraphs. No character/paragraph styles, -headers/footers/notes/annotations, tables, frames, images, or fields beyond their -result text. +paragraphs and manual line breaks, plus **direct character formatting** (font, +size, bold, italic, underline, strike, color, highlight) as styled spans, +through the abstract model. No paragraph styles or style-sheet (STSH) +inheritance, headers/footers/notes/annotations, tables, frames, images, or +fields beyond their result text. -**Specs.** `[MS-DOC]` (FIB, Clx / piece table, text decoding) + `[MS-CFB]` -container. The implemented read path matches *Retrieving Text* (§2.4.1 steps 1–6); -section numbers are cited inline in code and in the read-path map below. +**Specs.** `[MS-DOC]` (FIB, Clx / piece table, text decoding, CHPX) + +`[MS-CFB]` container. The implemented read path matches *Retrieving Text* +(§2.4.1 steps 1–6) and *Direct Character Formatting* (§2.4.6.2); section +numbers are cited inline in code and in the read-path maps below. ## Module layout (sibling of `../presentation`) | File (`oldms/text/`) | Role | |---|---| -| `doc_structs.hpp` | `#pragma pack(1)` PODs (`FibBase`, the `FibRgFcLcb97/2000/…/2007` chain, `Sprm`, `FcCompressed`, `Pcd`) + `static_assert` sizes + `PlcPcdMap` + `ParsedFib` | +| `doc_structs.hpp` | `#pragma pack(1)` PODs (`FibBase`, the `FibRgFcLcb97/2000/…/2007` chain, `Sprm`, `FcCompressed`, `Pcd`, `PnFkpChpx`, `FfnFixed`) + `static_assert` sizes + `PlcPcdMap`/`PlcBteChpxMap` + `ParsedFib` + the character SPRM opcodes | | `doc_io.{hpp,cpp}` | `read(...)` over `std::istream`: variable-length FIB, Clx walk, string decoding | -| `doc_helper.{hpp,cpp}` | `CharacterIndex` (decoded piece table) + `read_character_index` | -| `doc_parser.{hpp,cpp}` | `parse_tree` → body text + tree; `clean_text` (field & control-char handling) | -| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + text side-payload | +| `doc_helper.{hpp,cpp}` | `CharacterIndex` (decoded piece table) + `read_character_index`; `CharacterStyles` (fc-keyed style runs) + `read_character_styles` (PlcBteChpx/ChpxFkp walk), `apply_character_sprms`, `read_font_names` (SttbfFfn) | +| `doc_parser.{hpp,cpp}` | `parse_tree` → styled runs + tree; `TextCleaner` (field & control-char handling) | +| `doc_element_registry.{hpp,cpp}` | Flat element store (id = vector index) + text side-payload + per-element `TextStyle` map + font-name intern store | | `doc_document.{hpp,cpp}` | `internal::Document` subclass + the `ElementAdapter` | ## Design decisions @@ -59,23 +61,47 @@ main body, `Prc` formatting runs, and every control/field char `clean_text` drop **Endianness.** Host byte order / LSB-first bit-fields assumed; shared `oldms/` assumption + fix plan in [`../AGENTS.md`](../AGENTS.md). -**The `font_size = 11pt` in the adapters is a placeholder hack** so empty -paragraphs still have height (same as the `.ppt`/`.xls` modules); removed when -character formatting lands (open work §1). `Document::is_editable()` → `true` and -`is_savable(!encrypted)`, but `save`/`text_set_content` throw — read-only in -practice. - -**`clean_text` control-character handling** (the non-obvious cases): `0x0D`/`0x0C`/ -`0x0B` are consumed by the caller's paragraph/page/line splits and never reach -`clean_text`; `0x13`/`0x14`/`0x15` delimit a **field** — instruction hidden, -result shown, the `0x14` separator optional (§2.8.25), nesting tracked with a -per-field stack; `0x09` tab kept; `0x1E` non-breaking hyphen → `-`; `0x1F` -optional hyphen dropped; every other control char < `0x20` dropped. +**Direct character formatting only, resolved to styled spans.** The tree is +`root → paragraph → span → text`; each span (and each paragraph, for +empty-paragraph height) stores a resolved `TextStyle` in a registry side-map. +The §2.4.6.2 walk: `PlcBteChpx` (table stream) → 512-byte `ChpxFkp` pages +(WordDocument stream) → per-run `Chpx.grpprl`, applied over a default of 10pt +(the `sprmCHps` default of 20 half-points; the old flat `11pt` placeholder is +gone). The non-obvious bits: +- **Runs are keyed by stream offset (fc), not CP**, so the piece decode walks + each piece in style-uniform chunks (`CharacterStyles::chunk_end`); a + boundary inside a 2-byte CP is pushed past it. +- **Equal `Chpx` bytes share one resolved style**, and adjacent equal-style + runs merge, so span count stays low. `rgb[j] == 0` means default properties. +- **`ToggleOperand`** (§2.9.327) values `0x80`/`0x81` refer to the (unmodelled) + style value, which defaults to off → `0x80` = off, `0x81` = on. +- **Colors**: `sprmCCv` is a `COLORREF` (`fAuto` → unset); the legacy + `sprmCIco`/`sprmCHighlight` use the `Ico` palette (§2.9.119) — the spec's + extracted table repeats `0x0C` for `0x0D`, which is dark red (`0x800000`). +- **`Pcd.Prm` modifications and STSH styles are not applied** (open work §1). +- Font names from `SttbfFfn` are interned in a registry `std::deque` so + `TextStyle::font_name` (`const char *`) stays valid. + +`Document::is_editable()` → `false`; `save`/`text_set_content` throw. + +**`TextCleaner` control-character handling** (the non-obvious cases): `0x0D`/ +`0x0C`/`0x0B` are consumed by the caller's paragraph/page/line splits and never +reach the cleaner; `0x13`/`0x14`/`0x15` delimit a **field** — instruction +hidden, result shown, the `0x14` separator optional (§2.8.25), nesting tracked +with a per-field stack that now **persists across style runs and paragraphs** +(a field can span both); `0x09` tab kept; `0x1E` non-breaking hyphen → `-`; +`0x1F` optional hyphen dropped; every other control char < `0x20` dropped. ## Tests - `OldMs.doc_read_string_compressed` — the compressed decoder against the §2.9.73 byte map (ASCII passthrough, `0x82–0x9F` remap, `0xA0–0xFF` round-trip). +- `OldMs.doc_apply_character_sprms` — every modelled SPRM, toggle semantics, + cvAuto reset, unknown fixed/variable SPRM skipping, malformed-grpprl throws. +- `OldMs.doc_read_font_names` — synthetic SttbfFfn. +- `OldMs.doc_character_formatting` — end-to-end over a synthetic `.doc` + (inline bytes: FIB + ChpxFkp + piece table): span splitting at run + boundaries, default 10pt, bold + font-name run, paragraph style. **Not yet tested:** FIB robustness (negative `ccpText`, newer-than-2007 fallback), `0x0C` page-break emission, and there is **no assertion-based render test** over a @@ -122,67 +148,21 @@ Retrieving Text: §2.4.1 (steps 1–6) Field chars 0x13/0x14/0x15: §2.8.25 # Open work -## 1. Character (font) formatting → the IR (the next feature) - -**Goal.** Extract per-run character properties (font name, size, bold, italic, -underline, strike, colour, highlight) and surface them through `TextStyle`, so the -renderer styles text instead of emitting one flat 11pt run. Replaces the -`font_size = 11pt` placeholder. - -`TextStyle` (`src/odr/style.hpp`) maps almost 1:1 onto the `.doc` character SPRMs: - -| `TextStyle` field | SPRM (opcode) | operand → value | -|---|---|---| -| `font_size` | `sprmCHps` (0x4A43) | u16 **half-points** → `Measure(hps/2, pt)` (default 20 = 10pt) | -| `font_weight` | `sprmCFBold` (0x0835) | `ToggleOperand` → `bold` | -| `font_style` | `sprmCFItalic` (0x0836) | `ToggleOperand` → `italic` | -| `font_underline` | `sprmCKul` (0x2A3E) | `Kul`, `0x00` = none → `bool` | -| `font_line_through` | `sprmCFStrike` (0x0837) | `ToggleOperand` → `bool` | -| `font_color` | `sprmCCv` (0x6870) | `COLORREF` → `Color` (legacy `sprmCIco` 0x2A42 is a palette index) | -| `background_color` | `sprmCHighlight` (0x2A0C) | `Ico` highlight index → `Color` | -| `font_name` | `sprmCRgFtc0` (0x4A4F) | s16 index into `SttbfFfn` → font name | - -`font_name` is a `const char *`, so the resolved name needs stable storage — intern -it in the `ElementRegistry` (e.g. a `std::deque` whose elements never -move) and hand out the pointer. - -**How `[MS-DOC]` retrieves character properties** — Direct Character Formatting -(§2.4.6.2) reuses the *Retrieving Text* walk we already have: -1. For `cp`, run §2.4.1 to get its byte offset `fc` and owning `Pcd` (already - computed). -2. Read **`PlcBteChpx`** (§2.8.5) at `fcPlcfBteChpx` in the table stream — a PLC - keyed by stream offset: `aFC[n+1]` + `aPnBteChpx[n]` (`PnFkpChpx`, 4 B). -3. Largest `i` with `aFC[i] ≤ fc` → read a **`ChpxFkp`** (§2.9.33) at - `aPnBteChpx[i].pn * 512` (fixed 512-byte page: `rgfc` boundaries, parallel - `rgb`, `crun` in the last byte). -4. Largest `j` with `rgfc[j] ≤ fc` → the `Chpx` (§2.9.32) at `rgb[j] * 2` within - the page. `Chpx.grpprl` is an array of `Prl` = `Sprm` (2 B) + operand. -5. Append `Pcd.Prm` mods (§2.9.214–216): `Prm0` (inline) or `Prm1` (index). - -`Prl`/`Sprm` is already modelled (`Sprm` with `ispmd/fSpec/sgc/spra` + -`operand_size()`); a **character** property is a SPRM with `sgc == 2` (note -`spra == 6` is length-prefixed/variable). - -**First cut — direct formatting only.** Implement §2.4.6.2 (`Chpx.grpprl` + -`Pcd.Prm`) and map the SPRMs above (bold/italic/size/font/colour applied directly -to runs). Resolve `sprmCRgFtc0` by reading **`SttbfFfn`** (§2.9.286) once and -indexing it. Drop the 11pt; use 10pt (the `sprmCHps` default). - -**Full fidelity — styles (later).** *Determining Formatting Properties* (§2.4.6.6) -layers: document defaults → `STSH` (§2.4.6.5) para/char-style `grpprl`s via the -paragraph's `istd` → table-style → direct paragraph → direct character. The first -cut skips STSH (style-dependent props fall back to defaults). - -**Wiring to the abstract model.** Per-run styling needs run boundaries in -`/WordDocument` byte offsets, so: -1. **Keep the FC↔text mapping** while concatenating (thread the `CharacterIndex` - through instead of discarding it after `body_text`). -2. **Split paragraphs into runs** at every `ChpxFkp` boundary, resolve each run's - `TextStyle`, emit a **`span`** (already wired via `SpanAdapter`) per run. -3. **Store the style** in a `TextStyle` side-map keyed by span id (mirror the text - side-payload + the `presentation` frame-payload) plus the font-name intern - store. `SpanAdapter::span_style` returns it; `text_style`/ - `paragraph_text_style` then return `{}` instead of the 11pt hack. +## 1. Character formatting fidelity + +Direct formatting (§2.4.6.2) is implemented — see the design notes above and +the read-path map below. Remaining layers of *Determining Formatting +Properties* (§2.4.6.6): + +- **`Pcd.Prm` modifications** (§2.9.214–216): per-piece property diffs — + `Prm0` (one inline SPRM) or `Prm1` (index into the Clx's `Prc` array, which + `read_character_index` currently skips). Rare for character properties, but + incremental saves can carry them. +- **STSH style layering** (§2.4.6.5): document defaults → paragraph/character + style `grpprl`s via the paragraph's `istd` → direct formatting. Without it, + style-derived properties (e.g. heading fonts defined only in the style + sheet, the usual Times New Roman default font) fall back to defaults; + `ToggleOperand` `0x80`/`0x81` resolve against "off". Character-formatting read path, keyed by `/WordDocument` byte offset `fc`: @@ -190,12 +170,12 @@ Character-formatting read path, keyed by `/WordDocument` byte offset `fc`: Table stream ├─ PlcBteChpx @ fcPlcfBteChpx §2.8.5 aFC[n+1] + aPnBteChpx[n] (PnFkpChpx, 4 B) ├─ SttbfFfn @ fcSttbfFfn (font names, FFN.xszFfn) §2.9.286 -└─ STSH @ fcStshf (styles — full fidelity only) §2.4.6.5 +└─ STSH @ fcStshf (styles — not read) §2.4.6.5 WordDocument stream └─ ChpxFkp @ aPnBteChpx[i].pn * 512 (512-byte page) §2.9.33 ├─ rgfc[crun+1] boundaries (stream offsets), rgb[crun] → Chpx @ rgb[j]*2, crun (last byte) - └─ Chpx = cb + grpprl(Prl[]) §2.9.32 Prl = Sprm(2 B) + operand; char SPRMs sgc==2; + Pcd.Prm §2.9.214–216 + └─ Chpx = cb + grpprl(Prl[]) §2.9.32 Prl = Sprm(2 B) + operand; char SPRMs sgc==2; + Pcd.Prm §2.9.214–216 (not read) ``` ## 2. Coverage gaps diff --git a/src/odr/internal/oldms/text/doc_document.cpp b/src/odr/internal/oldms/text/doc_document.cpp index 5aec89d2..b9719ddd 100644 --- a/src/odr/internal/oldms/text/doc_document.cpp +++ b/src/odr/internal/oldms/text/doc_document.cpp @@ -157,19 +157,12 @@ class ElementAdapter final : public abstract::ElementAdapter, } [[nodiscard]] TextStyle paragraph_text_style(const ElementIdentifier element_id) const override { - (void)element_id; - // TODO setting font size otherwise the text will be invisible. upstream - // this is used to make empty paragraphs works correctly - TextStyle style{}; - // TODO - style.font_size = Measure("11pt"); - return style; + return stored_style(element_id); } [[nodiscard]] TextStyle span_style(const ElementIdentifier element_id) const override { - (void)element_id; - return {}; // TODO + return stored_style(element_id); } [[nodiscard]] std::string @@ -184,13 +177,9 @@ class ElementAdapter final : public abstract::ElementAdapter, } [[nodiscard]] TextStyle text_style(const ElementIdentifier element_id) const override { + // The enclosing span carries the character style. (void)element_id; - // TODO setting font size otherwise the text will be invisible. upstream - // this is used to make empty paragraphs works correctly - TextStyle style{}; - // TODO - style.font_size = Measure("11pt"); - return style; + return {}; } private: @@ -198,6 +187,13 @@ class ElementAdapter final : public abstract::ElementAdapter, [[maybe_unused]] const Document *m_document{nullptr}; ElementRegistry *m_registry{nullptr}; + + /// The character style stored for a paragraph or span element. + [[nodiscard]] TextStyle + stored_style(const ElementIdentifier element_id) const { + const TextStyle *style = m_registry->element_style(element_id); + return style != nullptr ? *style : TextStyle{}; + } }; std::unique_ptr diff --git a/src/odr/internal/oldms/text/doc_element_registry.cpp b/src/odr/internal/oldms/text/doc_element_registry.cpp index 772fd677..f8ce1336 100644 --- a/src/odr/internal/oldms/text/doc_element_registry.cpp +++ b/src/odr/internal/oldms/text/doc_element_registry.cpp @@ -1,5 +1,6 @@ #include +#include #include namespace odr::internal::oldms::text { @@ -7,6 +8,8 @@ namespace odr::internal::oldms::text { void ElementRegistry::clear() noexcept { m_elements.clear(); m_texts.clear(); + m_styles.clear(); + m_font_names.clear(); } [[nodiscard]] std::size_t ElementRegistry::size() const noexcept { @@ -76,6 +79,26 @@ void ElementRegistry::append_child(const ElementIdentifier parent_id, element_at(parent_id).last_child_id = child_id; } +const char *ElementRegistry::intern_font_name(const std::string &name) { + if (const auto it = std::ranges::find(m_font_names, name); + it != m_font_names.end()) { + return it->c_str(); + } + return m_font_names.emplace_back(name).c_str(); +} + +void ElementRegistry::set_element_style(const ElementIdentifier id, + TextStyle style) { + check_element_id(id); + m_styles[id] = std::move(style); +} + +const TextStyle * +ElementRegistry::element_style(const ElementIdentifier id) const { + const auto it = m_styles.find(id); + return it != m_styles.end() ? &it->second : nullptr; +} + void ElementRegistry::check_element_id(const ElementIdentifier id) const { if (id == null_element_id) { throw std::out_of_range( diff --git a/src/odr/internal/oldms/text/doc_element_registry.hpp b/src/odr/internal/oldms/text/doc_element_registry.hpp index 41bcaea4..48fbe8d1 100644 --- a/src/odr/internal/oldms/text/doc_element_registry.hpp +++ b/src/odr/internal/oldms/text/doc_element_registry.hpp @@ -2,8 +2,11 @@ #include #include +#include +#include #include +#include #include #include @@ -39,9 +42,20 @@ class ElementRegistry final { void append_child(ElementIdentifier parent_id, ElementIdentifier child_id); + /// Stores a font name and returns a pointer that stays valid for the + /// registry's lifetime (`TextStyle::font_name` is a `const char *`). + const char *intern_font_name(const std::string &name); + + /// Character style of a span or paragraph element. + void set_element_style(ElementIdentifier id, TextStyle style); + [[nodiscard]] const TextStyle *element_style(ElementIdentifier id) const; + private: std::vector m_elements; std::unordered_map m_texts; + std::unordered_map m_styles; + /// Deque: elements never move, so interned `c_str()`s stay valid. + std::deque m_font_names; void check_element_id(ElementIdentifier id) const; void check_text_id(ElementIdentifier id) const; diff --git a/src/odr/internal/oldms/text/doc_helper.cpp b/src/odr/internal/oldms/text/doc_helper.cpp index 0cd437b1..5bd278b5 100644 --- a/src/odr/internal/oldms/text/doc_helper.cpp +++ b/src/odr/internal/oldms/text/doc_helper.cpp @@ -3,8 +3,59 @@ #include #include #include +#include +#include +#include +#include #include +#include +#include + +namespace odr::internal::oldms::text { + +namespace { + +/// The Ico palette ([MS-DOC] 2.9.119); index 0 is cvAuto (no explicit color). +constexpr std::array ico_colors = { + 0x000000, 0x000000, 0x0000FF, 0x00FFFF, 0x00FF00, 0xFF00FF, + 0xFF0000, 0xFFFF00, 0xFFFFFF, 0x000080, 0x008080, 0x008000, + 0x800080, 0x800000, 0x808000, 0x808080, 0xC0C0C0}; + +std::optional ico_color(const std::uint8_t ico) { + if (ico >= ico_colors.size()) { + throw std::runtime_error("doc: Ico value out of range"); + } + if (ico == 0) { + return std::nullopt; // cvAuto + } + return Color(ico_colors[ico]); +} + +/// ToggleOperand ([MS-DOC] 2.9.327) against the (unmodelled) style value: +/// 0x80 matches the style, 0x81 inverts it; styles default to off. +bool toggle_on(const std::uint8_t value) { + switch (value) { + case 0x00: + case 0x80: + return false; + case 0x01: + case 0x81: + return true; + default: + throw std::runtime_error("doc: unexpected ToggleOperand value"); + } +} + +std::uint16_t read_u16(const std::string_view bytes, const std::size_t at) { + std::uint16_t value; + std::memcpy(&value, bytes.data() + at, sizeof(value)); + return value; +} + +} // namespace + +} // namespace odr::internal::oldms::text namespace odr::internal::oldms { @@ -32,4 +83,269 @@ text::CharacterIndex text::read_character_index(std::istream &in) { return result; } +text::CharacterStyles::CharacterStyles(TextStyle default_style) { + m_styles.push_back(std::move(default_style)); +} + +std::uint32_t text::CharacterStyles::add_style(TextStyle style) { + m_styles.push_back(std::move(style)); + return static_cast(m_styles.size() - 1); +} + +void text::CharacterStyles::append_run(const std::uint32_t begin_fc, + const std::uint32_t end_fc, + const std::uint32_t style_index) { + if (begin_fc >= end_fc || + (!m_runs.empty() && begin_fc < m_runs.back().end_fc)) { + throw std::runtime_error("doc: character runs must be ascending"); + } + if (style_index >= m_styles.size()) { + throw std::out_of_range("doc: style index out of range"); + } + if (!m_runs.empty() && m_runs.back().end_fc == begin_fc && + m_runs.back().style_index == style_index) { + m_runs.back().end_fc = end_fc; + return; + } + m_runs.push_back({begin_fc, end_fc, style_index}); +} + +std::uint32_t text::CharacterStyles::index_at(const std::uint32_t fc) const { + const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); + if (it == m_runs.begin()) { + return 0; + } + const Run &run = *std::prev(it); + return fc < run.end_fc ? run.style_index : 0; +} + +std::uint32_t text::CharacterStyles::chunk_end(const std::uint32_t fc) const { + const auto it = std::ranges::upper_bound(m_runs, fc, {}, &Run::begin_fc); + if (it != m_runs.begin() && fc < std::prev(it)->end_fc) { + return std::prev(it)->end_fc; + } + if (it != m_runs.end()) { + return it->begin_fc; + } + return std::numeric_limits::max(); +} + +const TextStyle &text::CharacterStyles::style(const std::uint32_t index) const { + return m_styles.at(index); +} + +TextStyle +text::apply_character_sprms(TextStyle style, const std::string_view grpprl, + const std::vector &font_names) { + std::size_t at = 0; + while (at + sizeof(Sprm) <= grpprl.size()) { + Sprm sprm; + std::memcpy(&sprm, grpprl.data() + at, sizeof(sprm)); + const std::uint16_t opcode = std::bit_cast(sprm); + at += sizeof(sprm); + + std::size_t operand_size; + if (const int fixed_size = sprm.operand_size(); fixed_size >= 0) { + operand_size = static_cast(fixed_size); + } else { + // spra == 6: the first operand byte is the size of the remainder. The + // two SPRMs encoded differently (sprmTDefTable, sprmPChgTabs with + // cb == 0xFF) are not character properties and must not appear here. + if (at >= grpprl.size()) { + throw std::runtime_error("doc: Prl operand overruns grpprl"); + } + operand_size = static_cast(grpprl[at]); + ++at; + if (opcode == 0xD608 || (opcode == 0xC615 && operand_size == 0xFF)) { + throw std::runtime_error("doc: unexpected table SPRM in a Chpx"); + } + } + if (at + operand_size > grpprl.size()) { + throw std::runtime_error("doc: Prl operand overruns grpprl"); + } + const std::string_view operand = grpprl.substr(at, operand_size); + at += operand_size; + + switch (opcode) { + case sprmCFBold: + style.font_weight = toggle_on(static_cast(operand[0])) + ? FontWeight::bold + : FontWeight::normal; + break; + case sprmCFItalic: + style.font_style = toggle_on(static_cast(operand[0])) + ? FontStyle::italic + : FontStyle::normal; + break; + case sprmCFStrike: + style.font_line_through = + toggle_on(static_cast(operand[0])); + break; + case sprmCHighlight: + style.background_color = ico_color(static_cast(operand[0])); + break; + case sprmCKul: + style.font_underline = operand[0] != 0; + break; + case sprmCIco: + style.font_color = ico_color(static_cast(operand[0])); + break; + case sprmCHps: { + const std::uint16_t half_points = read_u16(operand, 0); + if (half_points < 2 || half_points > 3276) { + throw std::runtime_error("doc: sprmCHps value out of range"); + } + style.font_size = Measure(half_points / 2.0, DynamicUnit("pt")); + } break; + case sprmCRgFtc0: { + // SttbfFfn index ([MS-DOC] 2.6.1); with 0 entries the value MUST be 0 + // and the (unmodelled) style-sheet default font applies — leave unset. + const auto ftc = static_cast(read_u16(operand, 0)); + if (ftc < 0 || (font_names.empty() ? ftc != 0 + : static_cast(ftc) >= + font_names.size())) { + throw std::runtime_error("doc: sprmCRgFtc0 font index out of range"); + } + if (!font_names.empty()) { + style.font_name = font_names[static_cast(ftc)]; + } + } break; + case sprmCCv: { + // COLORREF ([MS-DOC] 2.9.43): red, green, blue, fAuto. + if (static_cast(operand[3]) == 0xFF) { + style.font_color = std::nullopt; // cvAuto + } else { + style.font_color = Color(static_cast(operand[0]), + static_cast(operand[1]), + static_cast(operand[2])); + } + } break; + default: + break; // not modelled + } + } + if (at != grpprl.size()) { + throw std::runtime_error("doc: grpprl does not hold whole Prls"); + } + + return style; +} + +std::vector text::read_font_names(std::istream &table_stream, + const FcLcb sttbf_ffn) { + std::vector result; + if (sttbf_ffn.lcb == 0) { + return result; + } + table_stream.seekg(sttbf_ffn.fc); + + // SttbfFfn is a non-extended STTB ([MS-DOC] 2.2.4, 2.9.286): u16 cData, + // u16 cbExtra (0), then per entry a u8 byte count and an FFN. + const auto c_data = util::byte_stream::read(table_stream); + if (c_data == 0xFFFF) { + throw std::runtime_error("doc: unexpected extended SttbfFfn"); + } + const auto cb_extra = util::byte_stream::read(table_stream); + + result.reserve(c_data); + for (std::uint16_t i = 0; i < c_data; ++i) { + const auto cch_data = util::byte_stream::read(table_stream); + if (cch_data < sizeof(FfnFixed)) { + throw std::runtime_error("doc: FFN too short"); + } + const auto ffn = util::byte_stream::read(table_stream); + (void)ffn; + + // xszFfn: null-terminated UTF-16 within the remaining bytes; an + // alternative name (xszAlt) may follow the terminator. + const std::size_t name_units = (cch_data - sizeof(FfnFixed)) / 2; + std::u16string name = read_string_uncompressed(table_stream, name_units); + if ((cch_data - sizeof(FfnFixed)) % 2 != 0) { + table_stream.ignore(1); + } + if (const std::size_t nul = name.find(u'\0'); nul != std::u16string::npos) { + name.resize(nul); + } + result.push_back(util::string::u16string_to_string(name)); + + table_stream.ignore(cb_extra); + } + + return result; +} + +text::CharacterStyles text::read_character_styles( + std::istream &document_stream, std::istream &table_stream, + const FcLcb plcf_bte_chpx, const TextStyle &default_style, + const std::vector &font_names) { + CharacterStyles result(default_style); + if (plcf_bte_chpx.lcb == 0) { + return result; + } + + table_stream.seekg(plcf_bte_chpx.fc); + std::string plc_bytes = util::stream::read(table_stream, plcf_bte_chpx.lcb); + const PlcBteChpxMap plc(plc_bytes.data(), plc_bytes.size()); + + // One resolved style per distinct Chpx byte sequence. + std::unordered_map style_cache; + const auto resolve = [&](const std::string_view grpprl) -> std::uint32_t { + if (grpprl.empty()) { + return 0; + } + const auto it = style_cache.find(std::string(grpprl)); + if (it != style_cache.end()) { + return it->second; + } + const std::uint32_t index = result.add_style( + apply_character_sprms(default_style, grpprl, font_names)); + style_cache.emplace(std::string(grpprl), index); + return index; + }; + + for (std::uint32_t i = 0; i < plc.n(); ++i) { + // A 512-byte ChpxFkp page ([MS-DOC] 2.9.33) at pn * 512: crun in the last + // byte, crun+1 rgfc offsets, crun rgb bytes; rgb[j] * 2 locates the Chpx + // within the page, 0 means default properties. + std::array page; + document_stream.seekg(static_cast(plc.aData(i).pn) * 512); + document_stream.read(page.data(), page.size()); + if (!document_stream) { + throw std::runtime_error("doc: truncated ChpxFkp"); + } + + const auto crun = static_cast(page[511]); + if (crun < 0x01 || crun > 0x65) { + throw std::runtime_error("doc: ChpxFkp crun out of range"); + } + + for (std::uint8_t j = 0; j < crun; ++j) { + std::uint32_t begin_fc; + std::uint32_t end_fc; + std::memcpy(&begin_fc, page.data() + static_cast(j) * 4, 4); + std::memcpy(&end_fc, page.data() + static_cast(j + 1) * 4, + 4); + + const auto rgb = + static_cast(page[(crun + 1) * 4 + std::size_t{j}]); + std::string_view grpprl; + if (rgb != 0) { + const std::size_t chpx_at = static_cast(rgb) * 2; + if (chpx_at >= page.size() - 1) { + throw std::runtime_error("doc: Chpx offset out of range"); + } + const auto cb = static_cast(page[chpx_at]); + if (chpx_at + 1 + cb > page.size() - 1) { + throw std::runtime_error("doc: Chpx overruns its ChpxFkp"); + } + grpprl = std::string_view(page.data() + chpx_at + 1, cb); + } + + result.append_run(begin_fc, end_fc, resolve(grpprl)); + } + } + + return result; +} + } // namespace odr::internal::oldms diff --git a/src/odr/internal/oldms/text/doc_helper.hpp b/src/odr/internal/oldms/text/doc_helper.hpp index 62e346cd..537d4941 100644 --- a/src/odr/internal/oldms/text/doc_helper.hpp +++ b/src/odr/internal/oldms/text/doc_helper.hpp @@ -1,9 +1,15 @@ #pragma once +#include + +#include + #include #include #include #include +#include +#include #include namespace odr::internal::oldms::text { @@ -106,4 +112,54 @@ class CharacterIndex { CharacterIndex read_character_index(std::istream &in); +/// The document's character-formatting runs, keyed by WordDocument-stream +/// offset (fc). Style index 0 is the default style; offsets outside any run +/// resolve to it. +class CharacterStyles final { +public: + explicit CharacterStyles(TextStyle default_style); + + std::uint32_t add_style(TextStyle style); + /// Appends a run; runs must be ascending and non-overlapping. Adjacent runs + /// with the same style index are merged. + void append_run(std::uint32_t begin_fc, std::uint32_t end_fc, + std::uint32_t style_index); + + /// The style index of the run containing `fc` (0 if none). + [[nodiscard]] std::uint32_t index_at(std::uint32_t fc) const; + /// End offset of the run containing `fc`, or the next run's begin, so text + /// can be walked in style-uniform chunks; UINT32_MAX past the last run. + [[nodiscard]] std::uint32_t chunk_end(std::uint32_t fc) const; + [[nodiscard]] const TextStyle &style(std::uint32_t index) const; + +private: + struct Run { + std::uint32_t begin_fc; + std::uint32_t end_fc; + std::uint32_t style_index; + }; + + std::vector m_styles; + std::vector m_runs; +}; + +/// Applies the character SPRMs of a Chpx grpprl ([MS-DOC] 2.6.1) on top of +/// `style`; non-character SPRMs are skipped via their operand size. +/// `font_names` resolves sprmCRgFtc0 (pointers must outlive the style). +TextStyle apply_character_sprms(TextStyle style, std::string_view grpprl, + const std::vector &font_names); + +/// Reads the font names (FFN.xszFfn) of the SttbfFfn ([MS-DOC] 2.9.286) at +/// `fc` in the table stream; empty for lcb == 0. +std::vector read_font_names(std::istream &table_stream, + FcLcb sttbf_ffn); + +/// Reads the PlcBteChpx at `fc` in the table stream and every referenced +/// ChpxFkp page ([MS-DOC] 2.9.33) from the WordDocument stream, resolving +/// each run's Chpx against `default_style`. Equal Chpx bytes share one style. +CharacterStyles +read_character_styles(std::istream &document_stream, std::istream &table_stream, + FcLcb plcf_bte_chpx, const TextStyle &default_style, + const std::vector &font_names); + } // namespace odr::internal::oldms::text diff --git a/src/odr/internal/oldms/text/doc_parser.cpp b/src/odr/internal/oldms/text/doc_parser.cpp index c2f106cd..25f150f2 100644 --- a/src/odr/internal/oldms/text/doc_parser.cpp +++ b/src/odr/internal/oldms/text/doc_parser.cpp @@ -8,10 +8,13 @@ #include #include #include -#include #include +#include +#include +#include #include +#include #include namespace odr::internal::oldms { @@ -29,63 +32,125 @@ constexpr char page_break_mark = '\x0C'; // visible text. A field is 0x13 begin ... [0x14 separator] ... 0x15 end // ([MS-DOC] 2.8.25): the instruction (begin..separator) is hidden, the result // (separator..end) shown. A separator-less field is hidden entirely. -std::string clean_text(const std::string &in) { - std::string out; - out.reserve(in.size()); +// Stateful: a field can span style runs and paragraphs, so one cleaner +// processes the whole body in order. +class TextCleaner { +public: + std::string clean(const std::string_view in) { + std::string out; + out.reserve(in.size()); - // Open fields; entry is true while still in that field's instruction part. - // instruction_depth counts those; text is hidden whenever it is > 0. - std::vector field_in_instruction; - int instruction_depth = 0; - - for (const char c : in) { - switch (c) { - case '\x13': // field begin: a new field opens in its instruction part - field_in_instruction.push_back(true); - ++instruction_depth; - continue; - case '\x14': // field separator: the innermost field's instruction ends - if (!field_in_instruction.empty() && field_in_instruction.back()) { - field_in_instruction.back() = false; - --instruction_depth; + for (const char c : in) { + switch (c) { + case '\x13': // field begin: a new field opens in its instruction part + m_field_in_instruction.push_back(true); + ++m_instruction_depth; + continue; + case '\x14': // field separator: the innermost field's instruction ends + if (!m_field_in_instruction.empty() && m_field_in_instruction.back()) { + m_field_in_instruction.back() = false; + --m_instruction_depth; + } + continue; + case '\x15': // field end: close the innermost field + if (!m_field_in_instruction.empty()) { + if (m_field_in_instruction.back()) { + // separator-less field: its instruction part ends here, not + // earlier + --m_instruction_depth; + } + m_field_in_instruction.pop_back(); + } + continue; + default: + break; } - continue; - case '\x15': // field end: close the innermost field - if (!field_in_instruction.empty()) { - if (field_in_instruction.back()) { - // separator-less field: its instruction part ends here, not earlier - --instruction_depth; + + if (m_instruction_depth > 0) { + continue; // drop the hidden field instruction + } + + switch (c) { + case '\x09': // tab: keep as real text + out.push_back(c); + break; + case '\x1E': // non-breaking hyphen + out.push_back('-'); + break; + case '\x1F': // optional hyphen: drop + break; + default: + // Drop remaining control/anchor characters (< 0x20); keep the rest. + if (static_cast(c) >= 0x20) { + out.push_back(c); } - field_in_instruction.pop_back(); + break; } - continue; - default: - break; } - if (instruction_depth > 0) { - continue; // drop the hidden field instruction - } + return out; + } - switch (c) { - case '\x09': // tab: keep as real text - out.push_back(c); - break; - case '\x1E': // non-breaking hyphen - out.push_back('-'); - break; - case '\x1F': // optional hyphen: drop +private: + // Open fields; entry is true while still in that field's instruction part. + // m_instruction_depth counts those; text is hidden whenever it is > 0. + std::vector m_field_in_instruction; + int m_instruction_depth{0}; +}; + +/// A maximal piece of body text with uniform character formatting. +struct StyledRun { + std::string text; + std::uint32_t style_index{0}; +}; + +/// Decodes the main-body pieces (the first `ccp_text` CPs), split further at +/// every character-formatting boundary. +std::vector decode_styled_runs( + std::istream &document_stream, const text::CharacterIndex &character_index, + const text::CharacterStyles &styles, const std::size_t ccp_text) { + std::vector runs; + std::size_t consumed_cp = 0; + + for (const auto &entry : character_index) { + if (consumed_cp >= ccp_text) { break; - default: - // Drop remaining control/anchor characters (< 0x20); keep the rest. - if (static_cast(c) >= 0x20) { - out.push_back(c); + } + const std::size_t take = std::min(entry.length_cp, ccp_text - consumed_cp); + const std::size_t bytes_per_cp = entry.is_compressed ? 1 : 2; + + std::size_t cp = 0; + while (cp < take) { + const auto fc = + static_cast(entry.data_offset + cp * bytes_per_cp); + const std::uint32_t style_index = styles.index_at(fc); + + // CPs of this piece still in the same style run; a boundary that falls + // inside a 2-byte CP is pushed past it. + const std::uint32_t chunk_end_fc = styles.chunk_end(fc); + std::size_t chunk_cp = take - cp; + if (chunk_end_fc != std::numeric_limits::max()) { + chunk_cp = std::min( + chunk_cp, + std::max(1, (chunk_end_fc - fc + bytes_per_cp - 1) / + bytes_per_cp)); } - break; + + document_stream.seekg(fc); + std::string chunk_text = + text::read_string(document_stream, chunk_cp, entry.is_compressed); + if (!runs.empty() && runs.back().style_index == style_index) { + runs.back().text += chunk_text; + } else { + runs.push_back({std::move(chunk_text), style_index}); + } + cp += chunk_cp; } + + consumed_cp += take; } - return out; + return runs; } } // namespace @@ -100,63 +165,91 @@ ElementIdentifier text::parse_tree(ElementRegistry ®istry, const std::string tableStreamPath = fib.base.fWhichTblStm == 1 ? "/1Table" : "/0Table"; - const auto table_stream = files.open(AbsPath(tableStreamPath))->stream(); - table_stream->ignore(fib.fibRgFcLcb->clx.fc); + // Font table, interned so TextStyle::font_name pointers stay valid. + std::vector font_names; + for (const std::string &name : + read_font_names(*table_stream, fib.fibRgFcLcb->sttbfFfn)) { + font_names.push_back(registry.intern_font_name(name)); + } + + // Direct character formatting ([MS-DOC] 2.4.6.2); Pcd.Prm modifications are + // not modelled. Without sprmCHps the size defaults to 20 half-points. + TextStyle default_style; + default_style.font_size = Measure(10.0, DynamicUnit("pt")); + const CharacterStyles styles = read_character_styles( + *document_stream, *table_stream, fib.fibRgFcLcb->plcfBteChpx, + default_style, font_names); + + table_stream->seekg(fib.fibRgFcLcb->clx.fc); const CharacterIndex character_index = read_character_index(*table_stream); - // Concatenate the decoded pieces of the main document body only (the first - // ccpText CPs). Pieces are in ascending CP order, so clamp each piece to the - // remaining budget and stop once it is exhausted. - const std::size_t ccp_text = fib.ccpText(); - std::size_t consumed_cp = 0; - std::string body_text; - for (const auto &entry : character_index) { - if (consumed_cp >= ccp_text) { - break; - } - const std::size_t take = std::min(entry.length_cp, ccp_text - consumed_cp); - document_stream->seekg(entry.data_offset); - body_text += read_string(*document_stream, take, entry.is_compressed); - consumed_cp += take; - } + const auto ccp_text = static_cast(fib.ccpText()); + const std::vector runs = + decode_styled_runs(*document_stream, character_index, styles, ccp_text); - // Split into paragraphs. The main body always ends with a guard paragraph - // mark, so drop the resulting trailing empty paragraph. - auto paragraphs = - util::string::split(body_text, std::string(1, paragraph_mark)); - if (!paragraphs.empty() && paragraphs.back().empty()) { - paragraphs.pop_back(); - } + // Build the tree: paragraphs are opened lazily so the trailing guard + // paragraph mark does not produce an extra empty paragraph. Each paragraph + // and span stores its style; empty paragraphs keep their height through the + // paragraph style. + TextCleaner cleaner; + ElementIdentifier paragraph_id = null_element_id; - for (const auto ¶graph : paragraphs) { - // A paragraph may contain end-of-section / manual page-break characters - // (0x0C): split on them and mark each boundary with a page_break element. - const auto pages = - util::string::split(paragraph, std::string(1, page_break_mark)); - for (std::size_t page_i = 0; page_i < pages.size(); ++page_i) { - if (page_i > 0) { - auto [page_break_id, _] = - registry.create_element(ElementType::page_break); - registry.append_child(root_id, page_break_id); - } + const auto ensure_paragraph = [&](const std::uint32_t style_index) { + if (paragraph_id == null_element_id) { + auto [id, paragraph] = registry.create_element(ElementType::paragraph); + registry.append_child(root_id, id); + registry.set_element_style(id, styles.style(style_index)); + paragraph_id = id; + } + }; - auto [paragraph_id, _] = registry.create_element(ElementType::paragraph); - registry.append_child(root_id, paragraph_id); + for (const StyledRun &run : runs) { + std::size_t at = 0; + while (at <= run.text.size()) { + const std::size_t control = run.text.find_first_of("\r\x0B\x0C", at); + const std::size_t segment_end = + control == std::string::npos ? run.text.size() : control; - const auto lines = - util::string::split(pages[page_i], std::string(1, line_break_mark)); - for (std::uint32_t line_i = 0; line_i < lines.size(); ++line_i) { - if (line_i > 0) { - auto [line_id, _] = registry.create_element(ElementType::line_break); - registry.append_child(paragraph_id, line_id); - } + if (std::string cleaned = cleaner.clean( + std::string_view(run.text).substr(at, segment_end - at)); + !cleaned.empty()) { + ensure_paragraph(run.style_index); + auto [span_id, span] = registry.create_element(ElementType::span); + registry.set_element_style(span_id, styles.style(run.style_index)); + registry.append_child(paragraph_id, span_id); + + auto [text_id, text_element, text_entry] = + registry.create_text_element(); + text_entry.text = std::move(cleaned); + registry.append_child(span_id, text_id); + } - auto [text_id, _, text_element] = registry.create_text_element(); - text_element.text = clean_text(lines[line_i]); - registry.append_child(paragraph_id, text_id); + if (control == std::string::npos) { + break; + } + switch (run.text[control]) { + case paragraph_mark: + ensure_paragraph(run.style_index); + paragraph_id = null_element_id; + break; + case page_break_mark: { + ensure_paragraph(run.style_index); + paragraph_id = null_element_id; + auto [page_break_id, page_break] = + registry.create_element(ElementType::page_break); + registry.append_child(root_id, page_break_id); + } break; + case line_break_mark: { + ensure_paragraph(run.style_index); + auto [line_id, line] = registry.create_element(ElementType::line_break); + registry.append_child(paragraph_id, line_id); + } break; + default: + break; } + at = control + 1; } } diff --git a/src/odr/internal/oldms/text/doc_structs.hpp b/src/odr/internal/oldms/text/doc_structs.hpp index 1cb3b4a3..8731f822 100644 --- a/src/odr/internal/oldms/text/doc_structs.hpp +++ b/src/odr/internal/oldms/text/doc_structs.hpp @@ -288,6 +288,20 @@ struct FibRgCswNewData2007 : FibRgCswNewData2000 { static_assert(sizeof(FibRgCswNewData2007) == 8, "FibRgCswNewData2007 should be 8 bytes"); +/// The character SPRMs mapped to TextStyle ([MS-DOC] 2.6.1); everything else +/// in a Chpx is skipped via Sprm::operand_size. +enum CharacterSprms : std::uint16_t { + sprmCFBold = 0x0835, //< ToggleOperand + sprmCFItalic = 0x0836, //< ToggleOperand + sprmCFStrike = 0x0837, //< ToggleOperand + sprmCHighlight = 0x2A0C, //< Ico + sprmCKul = 0x2A3E, //< Kul; 0x00 = none + sprmCIco = 0x2A42, //< Ico (legacy palette color) + sprmCHps = 0x4A43, //< u16 half-points (default 20) + sprmCRgFtc0 = 0x4A4F, //< s16 index into SttbfFfn + sprmCCv = 0x6870, //< COLORREF +}; + struct Sprm { std::uint16_t ispmd : 9; std::uint16_t fSpec : 1; @@ -333,6 +347,26 @@ struct Pcd { }; static_assert(sizeof(Pcd) == 8, "Pcd should be 8 bytes"); +/// Location of a ChpxFkp in the WordDocument stream, at pn * 512 +/// ([MS-DOC] 2.9.206). +struct PnFkpChpx { + std::uint32_t pn : 22; + std::uint32_t unused : 10; +}; +static_assert(sizeof(PnFkpChpx) == 4, "PnFkpChpx should be 4 bytes"); + +/// Fixed head of an FFN ([MS-DOC] 2.9.82); the null-terminated UTF-16 font +/// name (xszFfn) follows. +struct FfnFixed { + std::uint8_t ffid; + std::int16_t wWeight; + std::uint8_t chs; + std::uint8_t ixchSzAlt; + std::array panose; + std::array fs; +}; +static_assert(sizeof(FfnFixed) == 39, "FfnFixed should be 39 bytes"); + #pragma pack(pop) struct ParsedFibRgCswNew { @@ -409,4 +443,19 @@ class PlcPcdMap : public PlcPcdBase { std::size_t m_cbPlc{0}; }; +/// PlcBteChpx ([MS-DOC] 2.8.5): a PLC keyed by WordDocument-stream offsets +/// (fc, not CP) whose data elements locate the ChpxFkp pages. +class PlcBteChpxMap : public PlcBase { +public: + PlcBteChpxMap(char *data, const std::size_t cbPlc) + : m_data(data), m_cbPlc(cbPlc) {} + + [[nodiscard]] char *data() const { return m_data; } + [[nodiscard]] std::size_t cbPlc() const { return m_cbPlc; } + +private: + char *m_data{nullptr}; + std::size_t m_cbPlc{0}; +}; + } // namespace odr::internal::oldms::text diff --git a/test/src/internal/oldms/oldms_test.cpp b/test/src/internal/oldms/oldms_test.cpp index f414dcbe..d5df9b67 100644 --- a/test/src/internal/oldms/oldms_test.cpp +++ b/test/src/internal/oldms/oldms_test.cpp @@ -8,11 +8,19 @@ #include #include +#include + +#include +#include +#include #include +#include +#include #include #include #include +#include #include #include #include @@ -57,6 +65,89 @@ TEST(OldMs, doc_read_string_compressed) { namespace { +void append_doc_u16(std::string &out, const std::uint16_t value) { + out.push_back(static_cast(value & 0xFF)); + out.push_back(static_cast(value >> 8)); +} + +void append_doc_u32(std::string &out, const std::uint32_t value) { + append_doc_u16(out, static_cast(value & 0xFFFF)); + append_doc_u16(out, static_cast(value >> 16)); +} + +std::string doc_prl(const std::uint16_t opcode, const std::string &operand) { + std::string prl; + append_doc_u16(prl, opcode); + prl += operand; + return prl; +} + +/// A minimal FIB ([MS-DOC] 2.5.1): nFib97, table stream /1Table, with the +/// three FcLcbs the parser reads (clx, plcfBteChpx, sttbfFfn). +std::string make_fib(const std::uint32_t ccp_text, + const internal::oldms::text::FcLcb clx, + const internal::oldms::text::FcLcb plcf_bte_chpx, + const internal::oldms::text::FcLcb sttbf_ffn) { + std::string fib; + // FibBase (32 bytes) + append_doc_u16(fib, 0xA5EC); // wIdent + append_doc_u16(fib, 0x00C1); // nFib97 + append_doc_u16(fib, 0); // unused + append_doc_u16(fib, 0x0409); // lid + append_doc_u16(fib, 0); // pnNext + append_doc_u16(fib, 0x0200); // flags: fWhichTblStm = 1 + append_doc_u16(fib, 0x00C1); // nFibBack + append_doc_u32(fib, 0); // lKey + fib += std::string(2, '\0'); // envr, flags + fib += std::string(12, '\0'); // reserved3-6 + // csw + fibRgW + append_doc_u16(fib, 14); + fib += std::string(28, '\0'); + // cslw + fibRgLw; ccpText is the 4th 32-bit field + append_doc_u16(fib, 22); + std::string rg_lw(88, '\0'); + std::string ccp; + append_doc_u32(ccp, ccp_text); + rg_lw.replace(12, 4, ccp); + fib += rg_lw; + // cbRgFcLcb + FibRgFcLcb97: plcfBteChpx is FcLcb index 12, sttbfFfn 15, + // clx 33 + append_doc_u16(fib, 93); + std::string rg(744, '\0'); + const auto put_fclcb = [&rg](const std::size_t index, + const internal::oldms::text::FcLcb value) { + std::string bytes; + append_doc_u32(bytes, value.fc); + append_doc_u32(bytes, value.lcb); + rg.replace(index * 8, 8, bytes); + }; + put_fclcb(12, plcf_bte_chpx); + put_fclcb(15, sttbf_ffn); + put_fclcb(33, clx); + fib += rg; + // cswNew + append_doc_u16(fib, 0); + return fib; +} + +/// An SttbfFfn ([MS-DOC] 2.9.286) with the given font names. +std::string make_sttbf_ffn(const std::vector &names) { + std::string sttb; + append_doc_u16(sttb, static_cast(names.size())); // cData + append_doc_u16(sttb, 0); // cbExtra + for (const std::string &name : names) { + std::string ffn(39, '\0'); // FfnFixed + for (const char c : name) { + ffn.push_back(c); + ffn.push_back('\0'); + } + ffn += std::string(2, '\0'); // terminator + sttb.push_back(static_cast(ffn.size())); + sttb += ffn; + } + return sttb; +} + // Collects the text content of an element subtree, joining a slide's paragraphs // with newlines so the paragraph structure is observable. std::string collect_text(const Element element) { @@ -77,6 +168,200 @@ std::string collect_text(const Element element) { } // namespace +// Character SPRMs ([MS-DOC] 2.6.1) map onto TextStyle; unknown SPRMs are +// skipped via their operand size, malformed grpprls throw. +TEST(OldMs, doc_apply_character_sprms) { + using internal::oldms::text::apply_character_sprms; + + TextStyle base; + base.font_size = Measure("10pt"); + const std::vector fonts = {"Arial", "Courier New"}; + + { + std::string grpprl; + grpprl += doc_prl(0x0835, "\x01"); // bold: on + grpprl += doc_prl(0x0836, "\x81"); // italic: invert style default -> on + grpprl += doc_prl(0x0837, "\x80"); // strike: match style default -> off + grpprl += doc_prl(0x2A3E, "\x01"); // single underline + grpprl += doc_prl(0x4A43, std::string("\x20\x00", 2)); // 32 half-points + grpprl += doc_prl(0x4A4F, std::string("\x01\x00", 2)); // font index 1 + grpprl += doc_prl(0x6870, std::string("\x11\x22\x33\x00", 4)); // COLORREF + grpprl += doc_prl(0x2A0C, "\x07"); // highlight: yellow + const TextStyle style = apply_character_sprms(base, grpprl, fonts); + EXPECT_EQ(style.font_weight, FontWeight::bold); + EXPECT_EQ(style.font_style, FontStyle::italic); + EXPECT_EQ(style.font_line_through, false); + EXPECT_EQ(style.font_underline, true); + EXPECT_EQ(style.font_size, Measure("16pt")); + EXPECT_STREQ(style.font_name, "Courier New"); + ASSERT_TRUE(style.font_color.has_value()); + EXPECT_EQ(style.font_color->rgb(), 0x112233u); + ASSERT_TRUE(style.background_color.has_value()); + EXPECT_EQ(style.background_color->rgb(), 0xFFFF00u); + } + + { + // Legacy palette color; unknown fixed-size and variable-size SPRMs are + // skipped without disturbing later Prls. + std::string grpprl; + grpprl += doc_prl(0x2A42, "\x06"); // ico: red + grpprl += doc_prl(0x0838, "\x01"); // unknown, 1 B + grpprl += doc_prl(0xC875, std::string("\x02\xAA\xBB", 3)); // unknown, var + grpprl += doc_prl(0x0835, "\x01"); // bold: on + const TextStyle style = apply_character_sprms(base, grpprl, fonts); + ASSERT_TRUE(style.font_color.has_value()); + EXPECT_EQ(style.font_color->rgb(), 0xFF0000u); + EXPECT_EQ(style.font_weight, FontWeight::bold); + } + + { + // cvAuto ([MS-DOC] 2.9.43) resets an earlier explicit color. + TextStyle colored = base; + colored.font_color = Color(std::uint32_t{0x123456}); + const TextStyle style = apply_character_sprms( + colored, doc_prl(0x6870, std::string("\x00\x00\x00\xFF", 4)), fonts); + EXPECT_FALSE(style.font_color.has_value()); + } + + { + // sprmCRgFtc0 with an empty SttbfFfn: 0 is valid ([MS-DOC] 2.6.1) and + // leaves the font unset; anything else is out of range. + const std::vector no_fonts; + const TextStyle style = apply_character_sprms( + base, doc_prl(0x4A4F, std::string("\x00\x00", 2)), no_fonts); + EXPECT_EQ(style.font_name, nullptr); + EXPECT_THROW( + apply_character_sprms(base, doc_prl(0x4A4F, std::string("\x01\x00", 2)), + no_fonts), + std::runtime_error); + } + + // Out-of-range and negative sprmCRgFtc0 indexes throw. + EXPECT_THROW(apply_character_sprms( + base, doc_prl(0x4A4F, std::string("\x02\x00", 2)), fonts), + std::runtime_error); + EXPECT_THROW(apply_character_sprms( + base, doc_prl(0x4A4F, std::string("\xFF\xFF", 2)), fonts), + std::runtime_error); + + // Truncated operand, half a Sprm, and an out-of-range sprmCHps all throw. + EXPECT_THROW(apply_character_sprms(base, doc_prl(0x4A43, "\x01"), fonts), + std::runtime_error); + EXPECT_THROW(apply_character_sprms(base, std::string("\x35", 1), fonts), + std::runtime_error); + EXPECT_THROW(apply_character_sprms( + base, doc_prl(0x4A43, std::string("\x01\x00", 2)), fonts), + std::runtime_error); +} + +TEST(OldMs, doc_read_font_names) { + const std::string sttb = make_sttbf_ffn({"Arial", "Times New Roman"}); + std::istringstream in(sttb); + const std::vector names = internal::oldms::text::read_font_names( + in, {0, static_cast(sttb.size())}); + ASSERT_EQ(names.size(), 2); + EXPECT_EQ(names[0], "Arial"); + EXPECT_EQ(names[1], "Times New Roman"); +} + +// End-to-end over a synthetic .doc: the PlcBteChpx/ChpxFkp run boundaries +// split the text into styled spans ([MS-DOC] 2.4.6.2), the default run keeps +// the 10pt default, and the paragraph carries a style for empty-height. +TEST(OldMs, doc_character_formatting) { + const std::string body = "plain bold\r"; + constexpr std::uint32_t text_fc = 1536; + + // ChpxFkp page ([MS-DOC] 2.9.33) at pn 2: run "plain " -> no Chpx (rgb 0), + // run "bold\r" -> bold + font 0. + std::string fkp(512, '\0'); + { + std::string head; + append_doc_u32(head, text_fc); + append_doc_u32(head, text_fc + 6); + append_doc_u32(head, text_fc + 11); + head.push_back('\0'); // rgb[0]: default properties + head.push_back(static_cast(240)); // rgb[1]: Chpx at byte 480 + fkp.replace(0, head.size(), head); + + std::string chpx; + chpx += doc_prl(0x0835, "\x01"); // bold + chpx += doc_prl(0x4A4F, std::string("\x00\x00", 2)); // font index 0 + fkp[480] = static_cast(chpx.size()); + fkp.replace(481, chpx.size(), chpx); + fkp[511] = 2; // crun + } + + // Table stream: Clx at 0 (one compressed piece), SttbfFfn at 64, + // PlcBteChpx at 128. + std::string plc_pcd; + append_doc_u32(plc_pcd, 0); + append_doc_u32(plc_pcd, 11); + append_doc_u16(plc_pcd, 0); // Pcd flags + append_doc_u32(plc_pcd, (1u << 30) | (text_fc * 2u)); // fCompressed | fc + append_doc_u16(plc_pcd, 0); // prm + std::string clx; + clx.push_back('\x02'); + append_doc_u32(clx, static_cast(plc_pcd.size())); + clx += plc_pcd; + + const std::string sttbf_ffn = make_sttbf_ffn({"Arial"}); + + std::string plc_bte; + append_doc_u32(plc_bte, text_fc); + append_doc_u32(plc_bte, text_fc + 11); + append_doc_u32(plc_bte, 2); // PnFkpChpx: page at 2 * 512 + + std::string table(256, '\0'); + table.replace(0, clx.size(), clx); + table.replace(64, sttbf_ffn.size(), sttbf_ffn); + table.replace(128, plc_bte.size(), plc_bte); + + std::string word_document = + make_fib(11, {0, static_cast(clx.size())}, + {128, static_cast(plc_bte.size())}, + {64, static_cast(sttbf_ffn.size())}); + word_document.resize(1024, '\0'); + word_document += fkp; + word_document += body; + + auto files = std::make_shared(); + files->copy(std::make_shared(word_document), + internal::AbsPath("/WordDocument")); + files->copy(std::make_shared(table), + internal::AbsPath("/1Table")); + const Document document( + std::make_shared(files)); + + std::vector paragraphs; + for (const Element child : document.root_element().children()) { + paragraphs.push_back(child); + } + ASSERT_EQ(paragraphs.size(), 1); + ASSERT_EQ(paragraphs[0].type(), ElementType::paragraph); + + std::vector spans; + for (const Element child : paragraphs[0].children()) { + ASSERT_EQ(child.type(), ElementType::span); + spans.push_back(child); + } + ASSERT_EQ(spans.size(), 2); + + EXPECT_EQ(collect_text(spans[0]), "plain "); + const TextStyle plain = spans[0].as_span().style(); + EXPECT_EQ(plain.font_size, Measure("10pt")); + EXPECT_FALSE(plain.font_weight.has_value()); + EXPECT_EQ(plain.font_name, nullptr); + + EXPECT_EQ(collect_text(spans[1]), "bold"); + const TextStyle bold = spans[1].as_span().style(); + EXPECT_EQ(bold.font_weight, FontWeight::bold); + EXPECT_STREQ(bold.font_name, "Arial"); + EXPECT_EQ(bold.font_size, Measure("10pt")); + + EXPECT_EQ(paragraphs[0].as_paragraph().text_style().font_size, + Measure("10pt")); +} + namespace { void append_u16(std::string &out, const std::uint16_t value) {