Add sectional reading strategy for large documents

Author: apardawala

partner-built/magellan/commands/magellan.md

@@ -131,14 +131,18 @@ Display: "Found N files to process."
 
 For each file in the processing list:
 
-1. **Read** the file with the Read tool.
+1. **Check file size** via Bash (`wc -l` for text, `wc -c` for binary).
+2. **Read** the file following the ingestion skill's reading strategy:
+   - Small files (under ~5,000 lines): read entire file in one pass.
+   - Large files (over ~5,000 lines): read in sections using `offset` and `limit`.
+     See the "Reading Large Documents" section in the ingestion skill.
    - If it's a code file, check `.magellan/language_guides/` for a matching guide.
      Read the guide once per language (cache in context for subsequent files).
-2. **Extract facts** by applying the ingestion skill.
-3. **Write facts** to `.magellan/domains/<domain>/facts/<source_slug>.json`
+3. **Extract facts** by applying the ingestion skill.
+4. **Write facts** to `.magellan/domains/<domain>/facts/<source_slug>.json`
    following the fact schema in file-conventions.
-4. **Record disposition** in `.magellan/processed_files.json` (read, update, write back).
-5. Display: "Ingested [N/total]: filename (M facts → domain)"
+5. **Record disposition** in `.magellan/processed_files.json` (read, update, write back).
+6. Display: "Ingested [N/total]: filename (M facts → domain)"
 
 **Track affected domains** as you process files.
 

partner-built/magellan/skills/ingestion/SKILL.md

@@ -273,6 +273,48 @@ These are guidelines, not hard minimums. A boilerplate README genuinely has 0
 extractable facts. But a 200-page QA manual with 5 facts means you are skimming
 and need to go deeper.
 
+## Reading Large Documents
+
+Not all documents can be processed in a single read. Long documents suffer from
+attention degradation — facts from the middle and end of a long file get thinner
+coverage than facts from the beginning.
+
+**Determine the reading strategy before you start extracting:**
+
+**Small documents (under ~5,000 lines or ~50 pages):**
+Read the entire file in one pass. Extract facts normally.
+
+**Large documents (over ~5,000 lines or ~50 pages):**
+Read and extract in sections. Do NOT read the entire file at once.
+
+1. **First pass — structure scan.** Read the first 200 lines with the Read tool
+   (use `offset: 0, limit: 200`) to understand the document structure: table of
+   contents, section headers, chapter boundaries, or natural break points.
+
+2. **Plan sections.** Divide the document into sections of ~2,000-3,000 lines
+   (~30-40 pages) based on the structure you found. Use natural boundaries
+   (chapters, sections, modules) when possible. If none exist (e.g., a flat
+   CSV or continuous log), use fixed-size chunks with 100-line overlap.
+
+3. **Process each section independently.** For each section:
+   a. Read ONLY that section using `offset` and `limit` on the Read tool.
+   b. Extract facts from that section.
+   c. Write facts immediately (batch write to the fact file).
+   d. Display: "Section N/M: extracted K facts (lines X-Y)"
+
+4. **Track sections.** After all sections, verify total lines processed matches
+   the document's total line count. If any range was skipped, go back and read it.
+   Display: "Document complete: N facts from M sections (lines 1-total)"
+
+**Why this matters:** A 200-page manual read in one pass might yield 30 facts,
+heavily weighted toward the first 50 pages. The same manual read in 5 sections
+of 40 pages each will yield 60-80 facts with even coverage. The section boundary
+forces Claude to give full attention to every part of the document.
+
+**Code files:** Most code files are under 5,000 lines and can be read in one pass.
+For very large programs (e.g., 10,000+ line COBOL), split at paragraph/section
+boundaries (COBOL) or subroutine boundaries (RPG) rather than arbitrary line counts.
+
 ## After Extraction
 
 ### Critical: Write in Batches to Avoid Output Limits