From 3bfd77be228aa3954b86ff0a95e74fdcc5731d3f Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:15:17 +0800 Subject: [PATCH 01/12] fix(build): #247 driver-style link/archive/shared go through rspfile on Windows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Driver-style toolchains (g++/clang++ as link driver — the gnu dialect, which is what Windows clang-MSVC resolves to via dialect_for) inlined $in into cxx_link/cxx_archive/cxx_shared. On Windows ninja spawns via CreateProcess (32 KiB command-line ceiling), so ffmpeg/opencv-class packages (thousands of objects) overflow at the final link. Route $in through @$out.rsp on Windows only (gcc/clang drivers and GNU/llvm ar all accept @rspfile); POSIX rule text stays byte-identical (ARG_MAX is ample). The three rules share one driver_rule emitter. Also lands the batch design doc (P0-P2). --- ...m-fixes-and-buildmcpp-generation-design.md | 211 ++++++++++++++++++ src/build/ninja_backend.cppm | 38 +++- tests/unit/test_ninja_backend.cpp | 28 +++ 3 files changed, 266 insertions(+), 11 deletions(-) create mode 100644 .agents/docs/2026-07-19-large-source-pkg-platform-fixes-and-buildmcpp-generation-design.md diff --git a/.agents/docs/2026-07-19-large-source-pkg-platform-fixes-and-buildmcpp-generation-design.md b/.agents/docs/2026-07-19-large-source-pkg-platform-fixes-and-buildmcpp-generation-design.md new file mode 100644 index 00000000..e1a6cb64 --- /dev/null +++ b/.agents/docs/2026-07-19-large-source-pkg-platform-fixes-and-buildmcpp-generation-design.md @@ -0,0 +1,211 @@ +# 大型源码直编包全平台化:平台三修(#247/#248/#249)+ build.mcpp 构建期生成能力 + 描述符复杂度治理 + +日期:2026-07-19 +状态:设计评审中(未实施) +关联:mcpp#247 mcpp#248 mcpp#249;mcpp-index compat.ffmpeg(17,543 行)/ compat.opencv(2,739 行); +前置设计:`2026-06-30-l3-build-mcpp-implementation-design.md`(L3 协议)、`2026-07-17-asm-sources-and-general-build-capabilities-design.md`(G1–G9,G2/G3 已落地 0.0.95) + +--- + +## 0. 结论先行 + +三个 issue 不是三件孤立平台 bug,而是同一件事的三个断面:**ffmpeg/opencv 级"全源码直编"包(数千 TU、冻结 configure 快照、consumer-side 合成)把 mcpp 的平台机械层第一次推到了极限**。linux 腿全绿恰恰说明包形态(描述符文法 + build.mcpp 协议)已经够用;macOS/Windows 挂掉的三处全部是 mcpp 核心的平台实现债,不是包的问题: + +- **#247**(Windows):driver-style 链接规则不走 rspfile → 2283 个对象内联溢出 CreateProcess 32KiB。 +- **#248**(macOS):`capture_exec` 非 Linux 路径用 shell 字符串拼 env,`ENV… cd && bin` 里 env 只绑给 `cd` → dep build.mcpp 收不到 G3 契约环境。 +- **#249**(macOS):dep `include_dirs` 一律发 `-I`(最高优先级),大小写不敏感文件系统上 ffmpeg 源根的 `VERSION` 顶替 libc++ ``。 + +三修全部小而清晰(合计 ~150 行改动),建议一个版本(0.0.100)一起出,并**顺手修掉本次调研发现的第四个隐性大坑**:`generated_files` 每次构建无条件重写 → mtime 抖动 → 冻结快照包(config.h 被全部 TU include)**增量构建完全失效,每次 build 都是全量重编**。这个对 ffmpeg 级包的伤害不亚于三个 issue 本身。 + +描述符复杂度问题(compat.ffmpeg.lua 17.5k 行)的答案不是"把 configure 搬进 build.mcpp 构建期跑"——那会把维护期的确定性换成消费期的风险,违背"依赖图声明式、命令式困在叶子"的 à la carte 纪律。正确方向是**数据与逻辑分家**: + +> **17.5k 行不是逻辑复杂度,是冻结数据放错了容器的观感。** +> 描述符保持声明式骨架;冻结快照(config 头 + 源清单)是维护期流水线生成的**数据**,用生成器压缩(glob 压缩 + common/delta 拆分,预计 17.5k → ~6k 行); +> **不可约的动态部分**(stub 合成、二进制嵌入、per-OS 选择)交给 build.mcpp——为此把 build.mcpp 指令面补全(`source=` / `include-dir=` / `include-dir-after=`),让 compat.opencv 已经验证的混合形态成为**标准模板**,可复制到任意 cmake/autoconf 上游。 + +--- + +## 1. 三个 issue 的代码级核实(HEAD = 0.0.99) + +三个 issue 的根因描述**全部核实成立**,另有若干 issue 未提到的 nuance 影响修法。 + +### 1.1 #247 Windows driver-style 链接不走 rspfile —— 确认,含两个修法要点 + +`src/build/ninja_backend.cppm:567-598`:链接/归档/动态库规则按 `separateLinker` 分两支。 +`separateLinker` 不是 dialect 字段,是本地推导(`ninja_backend.cppm:364-365`): + +```cpp +const bool separateLinker = + dial.linkStyle == mcpp::toolchain::CommandDialect::LinkStyle::SeparateLinker; +``` + +- `SeparateLinker` 支(L567-585,MSVC dialect):三条规则都已走 `@$out.rsp`。✅ +- driver-style else 支(L586-598):`cxx_link` / `cxx_archive`(gnu dialect `.archiveCmd = "$ar rcs $out $in"`,`dialect.cppm:84-86`)/ `cxx_shared` 全部内联 `$in`,无 rspfile。❌ + +**Nuance A(修法落点)**:Windows 托管工具链是 clang-MSVC-ABI,但 `dialect_for`(`dialect.cppm:111-114`)只对 `CompilerId::MSVC`(原生 cl.exe)选 msvc dialect;Clang 落 `kGnuDialect` → driver-style。也就是说**今天 Windows 上唯一可达的 dialect 恰是永远拿不到 rspfile 的那支**。修必须落在 driver-style 分支,不能指望"换 dialect"。 + +**Nuance B(现成工具)**:`mcpp::platform::is_windows` constexpr(`common.cppm:46-60`)已存在,且 ninja_backend.cppm:601 已在用 `if constexpr` 同款写法;无需新基建。 + +**Nuance C(边界)**:编译规则(`cxx_module`/`cxx_object`/`cc`/asm/nasm/`cxx_scan`)全部无 rspfile——单 TU 命令行长度有界,不受此 bug 影响,**不必扩改**。 + +**修法**(与 issue 建议一致,细化两点): + +1. driver-style 分支在 `if constexpr (mcpp::platform::is_windows)` 下发 rspfile 三规则(clang++/gcc/GNU 及 llvm-ar 均支持 `@file`);POSIX 分支保持内联(ARG_MAX 足够,零行为变化)。 +2. 三处 rspfile 样板已在 SeparateLinker 支重复过一轮,此次再加三条会有六份;顺手抽一个 `emit_rule(name, cmd, useRsp)` 小 helper 消重(纯重构,规则文本逐字节不变)。归档命令沿用 dialect `.archiveCmd` 的 `$ar`,rsp 变体在 helper 内统一改写 `$in → @$out.rsp`,不必给 dialect 加 `archiveRspCmd` 字段(YAGNI:两个 dialect 的 archive 文法都以 `$in` 结尾)。 + +**验证**:e2e 新增"千对象链接"合成用例(脚本生成 ~1100 个 1 行 .c,`kind=lib` + bin 消费),Windows CI 腿必须过;Linux/macOS 断言 ninja 文件不含 rspfile(防误扩)。 + +### 1.2 #248 macOS dep build.mcpp 丢 G3 契约环境 —— 确认,建议直接 launcher-unify + +`src/platform/process.cppm`: + +- Linux 支(L337 起):`posix_spawn` + `merged_environ(extraEnv)` + `addchdir_np(cwd)`。✅ +- macOS/else 支(L378-383):拼 shell 字符串,`capture_with_env`(L218-233)把 env 前缀到**最前**,得到 `MCPP_*=… cd && 2>&1` —— POSIX 语义下赋值只绑 `cd`,真正的 bin 一个变量都收不到。❌ + +**全仓调用点排查**(本次调研新增的关键事实):`extraEnv` 与 `cwd` **同时非空**的调用点全仓只有一处——`build/build_program.cppm:598`(build.mcpp 运行);其余 8 处(编译探针、execute.cppm 的 run_exec 系、ninja 调用)要么 env 空要么 cwd 空,前缀恰好绑对。所以此 bug 精确地只炸 macOS dep/root build.mcpp,与 issue 的现象面完全吻合。 + +**修法:采纳 issue 的首选方案(launcher-unify),不做 shell 字符串补丁**: + +1. `run_exec`(L308)与 `capture_exec`(L337)的 `#if defined(__linux__)` 扩为 `#if defined(__linux__) || defined(__APPLE__)`。 +2. `posix_spawn_file_actions_addchdir_np`:macOS 10.15+ 可用,mcpp 的 macOS floor 是 14.0,无需运行时探测。 +3. **可移植性唯一坑**:`extern char **environ` 在 macOS 上仅对可执行文件可直接链接,dylib 需 `_NSGetEnviron()`(``)。mcpp 是可执行文件,现状可用;但 `merged_environ` 里加 `#if defined(__APPLE__)` 走 `*_NSGetEnviron()` 是零成本的 portable-correct 写法,防未来 process 模块进库。 +4. 附带收益:macOS 从此消掉 shell 拼接的 quoting/注入面(0.0.97 的 C3 过度引号回归就是这条路径的血债),`TODO(launcher-unify)`(L301-303)兑现一半(Windows 仍走 `_putenv_s`+继承,行为正确,不动)。 + +**验证**:单测——`capture_exec` 带 `extraEnv`+`cwd` 双非空,断言子进程 `getenv` 可见且 cwd 正确(Linux/macOS 同一断言);e2e——带 build.mcpp 读 `MCPP_OUT_DIR` 的 dep fixture,macOS CI 腿必过(现有 e2e 恰好全部是"不读契约变量"的 build.mcpp,静默容忍了缺失——这本身是测试覆盖教训:**契约变量必须有断言型消费者**)。 + +### 1.3 #249 `-I` 源根顶替系统头 —— 确认,`include_dirs_after` 是正解,另发现一处不一致 + +- dep `include_dirs` → `local_include_flags`(`ninja_backend.cppm:90-97`)一律发 `-I`,置于 `$local_includes`,排在 `$cxxflags`(内含工具链 `-isystem`/`-idirafter` 系统头)之前;且 gcc/clang 语义下 `-I` 恒先于一切 system 链搜索。**没有任何降级通道**:`kKnownXpkgKeys`(`manifest/xpkg.cppm:119-124`)只有 `include_dirs`,无 after/system 变体。 +- 机制先例已在仓内:`linkmodel.cppm:70-73` 对 payload 头按模式选 `-isystem`/`-idirafter`(GCC 用 `-idirafter` 保 `#include_next` 链),`build_program.cppm:177/200` 同款。此次是把该能力**开放给描述符**。 + +**为什么选 `-idirafter` 不选 `-isystem`**:`-isystem` 目录仍排在默认系统目录**之前**——`VERSION` 照样顶替 libc++ ``,只是降了警告等级;`-idirafter` 是唯一保证"排在全部系统头之后"的选项,而 `` 这类非系统头名不受影响。issue 判断正确。 + +**修法**: + +1. 数据模型:`BuildConfig::includeDirsAfter`(`types.cppm`,紧邻 `includeDirs`); +2. 文法:xpkg `include_dirs_after`(`manifest/xpkg.cppm`,进 `kKnownXpkgKeys` + 别名表)+ mcpp.toml `[build] include_dirs_after`(`toml.cppm`),per-OS 段自动继承(additive overlay 免费获得); +3. 展开:与 `include_dirs` 同走 `expand_dir_glob`(`prepare.cppm:1916` 一带),支持 `*` 源根约定; +4. 传播:沿 Public/Interface 边与 `includeDirs` 同规则传播,**保持 after 属性不衰减**(消费者收到的仍是 `-idirafter`); +5. 发射:`ninja_backend.cppm` `local_include_flags` 旁增 after 变体,附加在 `$local_includes` 尾部(位置无所谓,语义由 flag 本身保证); +6. 生态:compat.ffmpeg 生成器把 `"*"`、`"*/libavcodec"` 等**源根条目**挪到 `include_dirs_after`,`mcpp_generated/*`(自有生成头,需要顶替源内同名头)留在 `include_dirs`。opencv 同步。 + +**顺手修(本次调研发现的不一致)**:主工程直连路径 `plan.cppm:242-251` 的 `local_include_dirs_for_manifest` 是裸 `root / inc` 拼接,**不走 glob 展开**——与 dep 路径(`prepare.cppm:1916` 走 `expand_dir_glob`)行为分叉。统一到 `expand_dir_glob`,消掉"同一个键两条求值路径"(0.0.98 #242 的教训在 include 轴上的翻版)。 + +**兼容性注意**:老版本 mcpp 遇到未知键 `include_dirs_after` 会**静默跳过**(`xpkgUnknownKeys` 仅在 `mcpp xpkg parse` 时告警)——即老 mcpp 装新描述符会退化为"缺了这些 include"而非报错。发描述符时用 xpm 版本条目区隔(新形态只发给 >= 0.0.100 的索引轨道),或接受"老版本 macOS 上本来就是坏的"这一现实(#249 场景下降级无损失:目录仍可经 `-I` 补发一份到老轨道)。 + +### 1.4 第四坑(issue 之外,本次调研发现):`generated_files` 每次构建无条件重写 + +`materialize_generated_files`(`prepare.cppm:311-361`):每次 prepare 对每个条目无条件 `create_directories` + `ofstream` 写入(L342-354),**无内容比对、无 mtime 保护**。root 每次 build 都走(L1119-1123),dep 在 `loadVersionDep` 每次都走(L1823-1826)。 + +ninja 是 mtime 驱动的。冻结快照形态下 `mcpp_generated/config.h` 被**全部 2283 个 TU** include(depfile 追踪)——每次 `mcpp build` 重写它 → mtime 变 → **全量重编,增量构建对 ffmpeg 级包完全失效**。讽刺的是 build.mcpp 的缓存设计文档明确写了防的就是这个("regeneration … would otherwise bump mtimes and force spurious rebuilds"),`generated_files` 这条老通道没享受同款纪律。 + +**修法**(10 行级):写前读旧文件比内容,逐字节相同则跳过写。正确性零风险(内容变化本来就折入指纹,`prepare.cppm:301-306`);收益是冻结快照形态的增量构建从"全废"变"正常"。**此修与 #247/#248/#249 同批出**,否则 ffmpeg 包即使全平台绿了,开发体验也是每次全量重编的假绿。 + +--- + +## 2. compat.ffmpeg.lua 为什么 17,543 行 —— 复杂度解剖 + +生成流水线(mcpp-index `tools/compat-ffmpeg/`,commit 6e844cf):`fetch_upstream.sh`(拉官方 tarball)→ `gen_config.sh`(**维护期跑一次** hermetic `./configure --disable-autodetect`,`make -n` 干跑得到精确源清单)→ `gen_descriptor.py`(吐 lua)。可字节级复现。行数构成: + +| 段 | 行数 | 本质 | +|---|---|---| +| 骨架(spec/xpm/顶层 build 配置) | ~350 | 真正的"描述符" | +| linux `sources`(2283 TU 逐条列出) | ~2,300 | **数据**:configure 的 CONFIG_* 门控决定哪些文件编,glob 表达不了 | +| linux `generated_files`(config.h/config.asm/*_list.c 等 15 个文件内联) | ~8,100 | **数据**:configure 输出的冻结快照 | +| macosx `sources`(与 linux 重叠 ~90%) | ~2,100 | **重复数据** | +| macosx `generated_files` | ~5,000 | 数据(per-OS 确实不同,但结构同源) | + +即:**~98% 是维护期流水线生成的冻结数据,~2% 是人写的声明**。Windows 腿加入后将再涨 ~7k 行。对照 compat.opencv(2,739 行)已用三招压缩: + +1. **glob 压缩**:385 个源文件 → 19 个 dir/brace glob(`{a,b}` 与 `!` 排除,scanner 均已支持,`scanner.cppm:344-398/763-789`); +2. **build.mcpp 合成**:字体嵌入、OpenCL kernel 转码、jpeg12/16 重编 stub——不可约的动态部分收进 ~270 行 C++; +3. **数据文件中转**:`generated_files` 只放小的 `tu_manifest.txt`(数据),build.mcpp 读它展开成 stub(逻辑)——**数据与逻辑分家的雏形**。 + +ffmpeg 没法把 sources 全 glob 化(configure 门控是文件粒度),但 common/delta 拆分与目录级 glob 收缩空间很大(见 §4 P2)。 + +### 2.1 方向裁决:复杂度应该住在哪里? + +| 方案 | 形态 | 判定 | +|---|---|---| +| A. 全冻结(ffmpeg 现状) | 快照+清单全部内联 | 确定性/可审计最优,但体积随 OS×包数线性爆炸,且掩盖了"哪些是人的决策" | +| B. build.mcpp 构建期跑 configure/等价物 | 描述符极小,构建期算一切 | **否决**。把维护期确定性换成消费期风险:等于每个消费者机器上重跑 autoconf 探测(哪怕用 C++ 重写),破坏 hermetic("--disable-autodetect 后快照可复现"正是这套模式的立身之本)、破坏零 shell、破坏"叶子命令式"纪律,且 per-包重写 configure 是不可维护的天量 | +| **C. 数据/逻辑分家(opencv 形态标准化,推荐)** | 描述符=骨架;冻结快照=压缩后的数据段;build.mcpp=只做合成(读数据、写产物、发指令) | 保留 A 的确定性,吸收 B 的表达力,体积可控(ffmpeg 预估 17.5k → ~6k) | + +**原则句**:冻结快照是**维护期决策的存档**,必须留在描述符(或其数据段)里可审计;build.mcpp 只承担**从存档到产物的机械展开**,不承担决策。configure 永远不进消费者机器。 + +--- + +## 3. build.mcpp 构建期生成:现有能力与缺口(HEAD 核实) + +指令协议现状(`build_program.cppm:104-123`,typed `import mcpp;` 1:1 镜像): + +| 能力 | 现状 | 对大型包的影响 | +|---|---|---| +| 追加 c/cxx/ld flags、`-D` | ✅ `cxxflag/cflag/cfg/link-lib/link-search` | 够用 | +| 声明生成**源文件**入编译集 | ✅ `generated=`(要求运行后文件存在,`:617-623`;进 `bc.sources`+`modules.sources`) | 够用(opencv stub 即此) | +| **选择既有源文件**(非生成) | ⚠️ 灰色可行:`generated=<绝对路径>` 恰好能过存在性检查,scanner 对绝对字面量直取(`scanner.cppm:776-783`)——**未文档化、语义错位** | 需要正名 | +| 追加 include 目录 | ❌ 无指令;只能裸发 `cxxflag=-I…`(不规范化、c/cxx 要各发一遍、无 after 变体) | opencv 已被迫用裸 flag(`:412-413`) | +| per-glob/per-file flags | ❌ 无指令 | 实际不缺:描述符 `flags` 的 `**/…` glob 对 OUT_DIR 产物也能命中(leading-`**` 匹配绝对路径),opencv `**/tu/jpeg12/**` 即此用法——**文档化即可,不加指令** | +| `.o` 直入链接 | ❌(设计文档显式非目标) | 维持:汇编已是一等源,无需求 | +| root 拿 `MCPP_DEP_*_DIR` | ❌(root build.mcpp 跑在依赖解析前,`prepare.cppm:2991-2993` 已记 follow-up) | 消费侧合成(如 ffmpeg-m 读 compat.ffmpeg 源树)需要 | +| 缓存 | ✅ 契约 env 整体折入 ctxHash,声明式 I/O 重跑门 | 设计良好,新指令按既有 `d ` 记录扩展即可 | + +### 3.1 P1:指令面补全(小步、正名、不扩权) + +新增三条指令 + 一处时序修正,全部落 `build_program.cppm` 的 `parse_line`/`apply` + typed 库同步: + +1. **`mcpp:source=`** —— 把"选择既有源入编译集"正名。相对路径:root 相对包根、dep 相对 `MCPP_MANIFEST_DIR`;绝对路径直取。与 `generated=` 的差异仅在语义申明(不承诺"本程序写出的");落点同款(`bc.sources`+`modules.sources`,存在性检查同款)。`generated=` 的绝对路径灰色用法保持兼容并在文档里指向 `source=`。 +2. **`mcpp:include-dir=` / `mcpp:include-dir-after=`** —— 路径按上述规则解析规范化,分别进 `buildConfig.includeDirs` / `includeDirsAfter`(与 §1.3 的新字段共用通道,一次实现两处受益)。**作用域维持 Cargo 纪律:私有**(只进本包 TU),不做 usage requirement 传播——需要传播的 include 属于描述符声明面,不属于构建期程序(防"构建期程序悄悄改公共接口"这一供应链面)。 +3. **`MCPP_TARGET_OS` / `MCPP_TARGET_ARCH` / `MCPP_TARGET_ENV`**(契约 env 便利拆分,Cargo `CARGO_CFG_TARGET_*` 对应)—— 三元组解析在 mcpp 侧做一次,免每个 build.mcpp 手撕字符串(ffmpeg/opencv 的 per-OS 选择逻辑直接受益)。 +4. **root `MCPP_DEP_*_DIR`**:把 root build.mcpp 的运行点从依赖解析前(`prepare.cppm:1180`)移到 dep 解析完成后、modgraph scan 前(dep build.mcpp 循环 `prepare.cppm:2953` 同一带)。风险点单测锁死:root build.mcpp 的 `generated=` 输出仍须先于 scan 注册(现有顺序保证不变量:materialize → build.mcpp → scan)。 + +不做的(显式非目标):per-glob flags 指令(现有描述符 glob 通道已覆盖)、`.o` 注入、任何"加依赖"能力(叶子纪律不破)、`install()` 替代(那是 xim 层)。 + +### 3.2 兼容与信任面 + +- 新指令老版本 mcpp 收到会打 "ignoring unknown directive" 警告并继续(`:121` 前向兼容已设计好)——包侧用 `min_mcpp`/xpm 版本轨道声明下限。 +- 信任模型不变(build.rs 等价:构建即执行包的 build.mcpp);新指令没有扩大能力面(include-dir 私有、source 仅指向已下载 payload 内文件)。 + +--- + +## 4. 描述符瘦身:P2(mcpp-index 侧,mcpp 零改动或近零改动) + +**目标:把 opencv 三招沉淀为可复制的生成器框架,ffmpeg 描述符 17.5k → ~6k 行,新包(SDL/curl/sqlite/…)按模板一周内成型。** + +1. **`tools/compat-gen/` 通用框架**(把 `tools/compat-ffmpeg/gen_descriptor.py` 泛化): + 输入 = per-OS 构建快照(CI matrix 各腿跑维护期 configure/cmake 一次,上传 artifact)+ 包插件(~100 行:声明 GEN_FILES、flags 规则、glob 压缩策略); + 输出 = 单一 lua 描述符 + 字节级复现校验(现有 "byte-identical regeneration" 纪律固化为 CI check)。 +2. **common/delta 拆分**(纯生成器逻辑,mcpp 免改):per-OS 段是 additive overlay(`xpkg.cppm:801-805`),生成器把 sources 交集提升到顶层、per-OS 只留差集——ffmpeg linux/macosx 源清单重叠 ~90%,仅此一项省 ~2,000 行,加 Windows 腿的边际成本从 ~7k 降到 ~1k。 + ⚠️ 边界:`generated_files` 是 map-insert(先到先得),**同 key 不能 per-OS 覆盖顶层**——config 快照必须整文件留在 per-OS 段(现状即正确,不要试图 delta 化配置文件,得不偿失)。 +3. **glob 收缩**:`make -n` 清单先按目录聚类,整目录全选的收成 `dir/*.c`,零散的保留逐条 + `!` 排除;opencv 已验证(385→19)。 +4. **数据/逻辑分家模板**:大数据表(如 stub 清单)走 `generated_files` 落成 `.txt` 数据文件,build.mcpp 读取展开——描述符里数据是数据、程序是程序,review 面清晰。 +5. **per-OS 段跨 OS 校验**(本次调研发现的盲区):非宿主 OS 段被 skip-table 跳过(`xpkg.cppm:1394-1399`),windows 段的 typo 在 linux CI 上不可见——`mcpp xpkg parse` 增加 `--all-os` 模式逐 OS 各 parse 一遍,mcpp-index CI 接入。此为 mcpp 侧小改(parse 命令层,不动构建)。 + +**显式否决的备选**:快照外置为独立下载资产(第二条 url/sha256 腿)——多一个供应链验证面、破坏索引单文件原子性、离线优先受损;17.5k 行文本在索引 tarball 里压缩后不足 100KB,体积不是真问题,**review 信噪比**才是,而那由 common/delta+glob 收缩解决。 + +--- + +## 5. 实施切分与版本 + +| 批次 | 内容 | 落点 | 规模 | +|---|---|---|---| +| **P0(0.0.100,单 PR)** | #247 rspfile(含 emit_rule 消重)· #248 launcher-unify(`\|\| __APPLE__` + `_NSGetEnviron`)· #249 `include_dirs_after` 全链(types/xpkg/toml/expand/propagate/ninja)+ plan.cppm include glob 统一 · §1.4 generated_files 内容比对跳写 | mcpp | ~150 行核心 + 测试 | +| **P1(0.0.101)** | `mcpp:source=` · `mcpp:include-dir[/-after]=` · `MCPP_TARGET_OS/ARCH/ENV` · root `MCPP_DEP_*`(时序后移) · feature 未知子键告警(`xpkg.cppm:1183-1187` 现静默吞) | mcpp | 中 | +| **P2(随 P0/P1 各自发布后)** | compat-gen 框架 · ffmpeg/opencv 描述符再生成(common/delta+glob 收缩+include_dirs_after 迁移)· `xpkg parse --all-os` + CI 接入 · windows 腿(ffmpeg PR#89 系列收口) | mcpp-index(+mcpp parse 层小改) | 生成器工程 | +| P3(挂账不排期) | feature 门控 include_dirs/generated_files · sources 外置清单键(`sources_manifest =`)· 描述符规模守卫(parse 内存/条目数上限告警) | mcpp | 待需求实证 | + +P0 内四项互不纠缠但共享同一验收场景(ffmpeg/opencv 全平台),按 0.0.97 的经验单 PR 逐簇 commit;**e2e 必须含全平台冒烟**(#247 只在 Windows 现形、#248/#249 只在 macOS 现形、§1.4 只在二次构建现形——单平台单次构建的 CI 全绿恰是这批 bug 能活到今天的原因)。 + +验收基线(全部来自现成复现件): +- Windows:mcpp-index spike 分支 multi-platform compat.ffmpeg `mcpp test` 链接过(#247); +- macOS:spike PR#90 compat.opencvmac build.mcpp 拿到 `MCPP_OUT_DIR`(#248);PR#91 `mcpp test -p ffmpeg-module`(`import ffmpeg.av`)过(#249); +- 全平台:ffmpeg 包二次 `mcpp build` 零重编(§1.4)。 + +--- + +## 6. 风险与开放问题 + +1. **#248 改的是进程启动原语**,macOS 全部 `capture_exec`/`run_exec` 调用点(含编译探针、ninja 调起)行为从 shell 换 posix_spawn——理论上更严格(不再有 shell 展开)。需全量 e2e 在 macOS 过一遍;若有调用点暗依赖 shell 语义(如命令里带重定向字符串),会在此暴露。现调研未发现此类调用点(`2>&1` 由 capture 侧 pipe 取代),列为 PR 内验证项。 +2. **include_dirs_after 传播语义**:传播链(Public/Interface)上 after 属性保持,但消费者自己的 `-I` 仍在 dep 的 `-idirafter` 之前——符合预期(消费者自有头最优先)。开放问题:是否给 `include_dirs` 整体加"默认 `-isystem` 化"的远期开关(压制 dep 头的告警噪声)——与 #249 正交,挂 P3。 +3. **root build.mcpp 时序后移**(P1-4)是 P1 里唯一动既有不变量的项:root 的 `generated=` 源此前在 dep 解析前就已注册,后移不影响(scan 在更后),但 root build.mcpp 发出的 `cxxflag` 此前理论上可影响 dep 的…… 不能——dep 编译用 dep 自己的 buildConfig,root flags 不下渗(Cargo 纪律),确认无耦合。单测锁:后移前后 root+dep 双 build.mcpp fixture 的 ninja 产物逐字节一致。 +4. **老 mcpp × 新描述符**的静默降级(§1.3 兼容性注意):接受现实 + xpm 轨道区隔,不做描述符侧 shim。 diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index e1a97dd5..d2362cdf 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -584,17 +584,33 @@ std::string emit_ninja_string(const BuildPlan& plan) { append(" rspfile_content = $in\n"); append(" description = SHARED $out\n\n"); } else { - append("rule cxx_link\n"); - append(" command = $cxx $in -o $out $ldflags $unit_ldflags\n"); - append(" description = LINK $out\n\n"); - - append("rule cxx_archive\n"); - append(std::format(" command = {}\n", dial.archiveCmd)); - append(" description = AR $out\n\n"); - - append("rule cxx_shared\n"); - append(" command = $cxx -shared $in -o $out $ldflags $soname_flag $unit_ldflags\n"); - append(" description = SHARED $out\n\n"); + // Driver-style toolchains (g++/clang++ as the link driver). On + // Windows these still spawn through CreateProcess (32 KiB command + // line ceiling), and large source packages (ffmpeg/opencv-class) + // link thousands of objects — an inlined $in overflows it (#247). + // gcc/clang drivers and GNU/llvm ar all accept @rspfile, so route + // $in through one there. POSIX keeps the inline form byte-identical: + // ARG_MAX is ample and the plain command is easier to reproduce. + auto driver_rule = [&](std::string_view name, std::string cmd, + std::string_view desc) { + append(std::format("rule {}\n", name)); + if constexpr (mcpp::platform::is_windows) { + if (auto pos = cmd.find("$in"); pos != std::string::npos) + cmd.replace(pos, 3, "@$out.rsp"); + append(std::format(" command = {}\n", cmd)); + append(" rspfile = $out.rsp\n"); + append(" rspfile_content = $in\n"); + } else { + append(std::format(" command = {}\n", cmd)); + } + append(std::format(" description = {} $out\n\n", desc)); + }; + driver_rule("cxx_link", + "$cxx $in -o $out $ldflags $unit_ldflags", "LINK"); + driver_rule("cxx_archive", std::string(dial.archiveCmd), "AR"); + driver_rule("cxx_shared", + "$cxx -shared $in -o $out $ldflags $soname_flag $unit_ldflags", + "SHARED"); } append("rule runtime_alias\n"); diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index 364cb452..7f8a527d 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -400,6 +400,34 @@ TEST(NinjaBackend, RawMultiTokenFlagIsNotQuoted) { std::string::npos) << ninja; } +// mcpp#247: driver-style (gnu dialect) link/archive/shared rules must route +// the object list through a response file on Windows — every command spawns +// via CreateProcess (32 KiB command-line ceiling), and ffmpeg/opencv-class +// source packages link thousands of objects, so an inlined $in overflows it. +// POSIX keeps the inline form byte-identical (ARG_MAX is ample). +TEST(NinjaBackend, DriverStyleLinkRulesUseRspfileOnWindowsOnly) { + auto plan = minimal_plan(); // GCC → gnu dialect → driver-style branch + + auto ninja = emit_ninja_string(plan); + + for (std::string_view rule : {"rule cxx_link\n", "rule cxx_archive\n", + "rule cxx_shared\n"}) { + auto start = ninja.find(rule); + ASSERT_NE(start, std::string::npos) << ninja; + auto end = ninja.find("\n\n", start); + ASSERT_NE(end, std::string::npos) << ninja; + auto body = ninja.substr(start, end - start); + if constexpr (mcpp::platform::is_windows) { + EXPECT_NE(body.find("@$out.rsp"), std::string::npos) << body; + EXPECT_NE(body.find("rspfile = $out.rsp"), std::string::npos) << body; + EXPECT_NE(body.find("rspfile_content = $in"), std::string::npos) << body; + } else { + EXPECT_EQ(body.find("rspfile"), std::string::npos) << body; + EXPECT_NE(body.find("$in"), std::string::npos) << body; + } + } +} + TEST(NinjaBackend, RootPackageCxxflagsAreEmittedOncePerUnit) { auto plan = minimal_plan(); plan.manifest.buildConfig.cxxflags = {"-DROOT_FLAG=1"}; From 1425a13480e81a7ae6340769f95ee28b3d894df3 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:23:57 +0800 Subject: [PATCH 02/12] feat(cli): mcpp xpkg parse --all-os validates every shipped per-OS section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The build path splices only the running host's per-OS block and skip- tables the rest, so a typo in the windows section is invisible on linux CI. --all-os re-parses the descriptor once per OS the xpm table declares versions for (a linux+macosx-only package legitimately has no windows section), via a new osOverride seam on synthesize_from_xpkg_lua — same function, same grammar as the resolver. mcpp-index CI can now lint all legs of a multi-platform descriptor from one runner (design doc 2026-07-19 §4). --- src/cli.cppm | 5 ++++- src/cli/cmd_xpkg.cppm | 43 +++++++++++++++++++++++++++++++++++- src/manifest/xpkg.cppm | 16 +++++++++++--- tests/unit/test_manifest.cpp | 12 ++++++++++ 4 files changed, 71 insertions(+), 5 deletions(-) diff --git a/src/cli.cppm b/src/cli.cppm index 85f63df8..c247a031 100644 --- a/src/cli.cppm +++ b/src/cli.cppm @@ -333,7 +333,10 @@ int run(int argc, char** argv) { .description("Parse a descriptor's mcpp segment exactly as the resolver would (strict: unknown keys are errors)") .option(cl::Option("json").help("Emit machine-readable JSON")) .option(cl::Option("allow-unknown") - .help("Downgrade unknown mcpp-segment keys from error to warning"))) + .help("Downgrade unknown mcpp-segment keys from error to warning")) + .option(cl::Option("all-os") + .help("Validate every per-OS section (linux/macosx/windows), " + "not just the running host's"))) .action(wrap_rc([&dispatch_sub](const cl::ParsedArgs& p) { return dispatch_sub("xpkg", p, {{"parse", cmd_xpkg_parse}}); }))) diff --git a/src/cli/cmd_xpkg.cppm b/src/cli/cmd_xpkg.cppm index ff1f24f5..aef896e9 100644 --- a/src/cli/cmd_xpkg.cppm +++ b/src/cli/cmd_xpkg.cppm @@ -58,7 +58,7 @@ std::string json_array(const std::vector& v) { export int cmd_xpkg_parse(const mcpplibs::cmdline::ParsedArgs& parsed) { auto file = parsed.positional(0); if (file.empty()) { - mcpp::ui::error("usage: mcpp xpkg parse [--json] [--allow-unknown]"); + mcpp::ui::error("usage: mcpp xpkg parse [--json] [--allow-unknown] [--all-os]"); return 2; } std::ifstream is{std::filesystem::path{file}}; @@ -70,6 +70,11 @@ export int cmd_xpkg_parse(const mcpplibs::cmdline::ParsedArgs& parsed) { const bool asJson = parsed.is_flag_set("json"); const bool allowUnknown = parsed.is_flag_set("allow-unknown"); + const bool allOs = parsed.is_flag_set("all-os"); + if (allOs && asJson) { + mcpp::ui::error("--all-os and --json are mutually exclusive"); + return 2; + } // Identity — same normalization the filename-lookup gate uses. auto id = mcpp::manifest::canonical_xpkg_identity_from_lua(lua); @@ -111,6 +116,42 @@ export int cmd_xpkg_parse(const mcpplibs::cmdline::ParsedArgs& parsed) { return 0; } + // --all-os: re-parse the descriptor once per OS section. The build path + // splices only the running host's per-OS block and skip-tables the rest, + // so a typo in e.g. the `windows` block is invisible on linux CI; this + // is the lint-time closure over all three (design doc 2026-07-19 §4). + if (allOs) { + int allRc = 0; + for (auto plat : kPlatforms) { + // Only OSes the xpm table ships for: a linux+macosx-only package + // legitimately has no windows section to validate. + if (versions[std::string(plat)].empty()) { + std::println("parse -- [{}] (no xpm versions declared)", plat); + continue; + } + auto pm = mcpp::manifest::synthesize_from_xpkg_lua( + lua, fqn, anyVersion, plat); + if (!pm) { + mcpp::ui::error(std::format("{} [{}]: {}", file, plat, + pm.error().format())); + allRc = 1; + continue; + } + for (auto& k : pm->xpkgUnknownKeys) { + auto msg = std::format( + "{} [{}]: unknown mcpp-segment key '{}' — silently " + "ignored at build time by this mcpp version", file, plat, k); + if (allowUnknown) std::println(stderr, "warning: {}", msg); + else { mcpp::ui::error(msg); allRc = 1; } + } + std::println("parse OK [{}] sources {} includes {} generated {}", + plat, pm->modules.sources.size(), + pm->buildConfig.includeDirs.size(), + pm->buildConfig.generatedFiles.size()); + } + return allRc; + } + // The parse users get at build time — same function, same grammar. auto m = mcpp::manifest::synthesize_from_xpkg_lua(lua, fqn, anyVersion); if (!m) { diff --git a/src/manifest/xpkg.cppm b/src/manifest/xpkg.cppm index 8e55ce19..9178d91d 100644 --- a/src/manifest/xpkg.cppm +++ b/src/manifest/xpkg.cppm @@ -91,10 +91,17 @@ bool xpkg_lua_identity_matches(std::string_view luaContent, // // The resulting Manifest is in-memory only; sourcePath is set to the // supplied package name + version so error messages can refer to it. +// +// `osOverride` selects which per-OS section (linux/macosx/windows) is +// spliced into the parse instead of the running host's — the seam behind +// `mcpp xpkg parse --all-os`, which validates the sections a host build +// never reads (they are skip-tabled, so a typo in the `windows` block is +// otherwise invisible on linux CI). Empty = host platform (build path). std::expected synthesize_from_xpkg_lua(std::string_view luaContent, std::string_view packageName, - std::string_view packageVersion); + std::string_view packageVersion, + std::string_view osOverride = {}); // mcpp#237: the mcpp-segment key vocabulary is a CLOSED whitelist (the parse // loop's else-if chain). An unrecognised key is collected into @@ -786,7 +793,8 @@ list_xpkg_versions(std::string_view luaContent, std::string_view platform) { std::expected synthesize_from_xpkg_lua(std::string_view luaContent, std::string_view packageName, - std::string_view packageVersion) + std::string_view packageVersion, + std::string_view osOverride) { auto body = extract_mcpp_segment_body(luaContent); if (body.empty()) { @@ -798,7 +806,9 @@ synthesize_from_xpkg_lua(std::string_view luaContent, std::format("xpkg-lua of {}@{}", packageName, packageVersion), 0, 0}); } - if (auto platformBody = top_level_table_body_for_key(body, mcpp::platform::xpkg_platform); + const std::string_view osKey = + osOverride.empty() ? mcpp::platform::xpkg_platform : osOverride; + if (auto platformBody = top_level_table_body_for_key(body, osKey); !platformBody.empty()) { body += "\n"; body += platformBody; diff --git a/tests/unit/test_manifest.cpp b/tests/unit/test_manifest.cpp index 3a3e6d14..05a52c74 100644 --- a/tests/unit/test_manifest.cpp +++ b/tests/unit/test_manifest.cpp @@ -849,6 +849,18 @@ package = { EXPECT_EQ(m->buildConfig.cflags[1], expectedCflag); EXPECT_EQ(m->dependencies.count("compat.base"), 1u); EXPECT_EQ(m->dependencies.count(expectedDep), 1u); + + // osOverride seam (`mcpp xpkg parse --all-os`): a foreign OS section is + // spliced on request, independent of the running host — the build path + // (empty override) above stays host-selected. Reuses this fixture. + auto win = mcpp::manifest::synthesize_from_xpkg_lua(src, "tinyc", "1.0.0", + "windows"); + ASSERT_TRUE(win.has_value()) << win.error().format(); + ASSERT_EQ(win->modules.sources.size(), 2u); + EXPECT_EQ(win->modules.sources[1], "*/src/win32.c"); + ASSERT_EQ(win->buildConfig.cflags.size(), 2u); + EXPECT_EQ(win->buildConfig.cflags[1], "-DWINDOWS=1"); + EXPECT_EQ(win->dependencies.count("compat.win32"), 1u); } TEST(SynthesizeFromXpkgLua, GeneratedFiles) { From 28f2f326bcfb60d66bf5f1313ada31558b2b7678 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:21:37 +0800 Subject: [PATCH 03/12] fix(platform): #248 macOS capture_exec/run_exec take the posix_spawn path (launcher-unify) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause: on macOS the shell fallback built `KEY='v' cd && prog 2>&1` when both extraEnv and cwd were non-empty. POSIX binds the env assignment prefix to the first simple command only — `cd` — so across `&&` the real program received NO extra env. The only env+cwd call site is the build.mcpp child (build_program.cppm), which therefore lost the entire MCPP_* G3 contract (MCPP_OUT_DIR etc.) on macOS and aborted. Fix: extend the run_exec / capture_exec guards from __linux__ to __linux__ || __APPLE__ so macOS shares the direct posix_spawn path — child-only env via merged_environ, cwd via posix_spawn_file_actions_addchdir_np (macOS 10.15+; mcpp's floor is macOS 14, no runtime probe needed). Portability: on Apple, `environ` is only linkable from executables, not dylibs, so merged_environ now reads the host block through a host_environ() helper using _NSGetEnviron() from ; Linux keeps the plain symbol. Windows _WIN32 branches and the residual non-POSIX shell fallback are untouched; the L286 rationale + TODO(launcher-unify) now document Windows as the sole remaining shell-path exception. As a side effect macOS also sheds the shell quoting/signal/injection surface for these two launchers. Regression test: capture_exec with BOTH non-empty extraEnv and cwd (previously the untested combination — all prior call sites passed at most one) asserting the child sees the env value AND runs in cwd; runs on Linux (same shared code path) and macOS CI. --- src/platform/process.cppm | 75 ++++++++++++++++++---------- tests/unit/test_process_run_exec.cpp | 38 ++++++++++++-- 2 files changed, 83 insertions(+), 30 deletions(-) diff --git a/src/platform/process.cppm b/src/platform/process.cppm index 8906c256..d0cd16cf 100644 --- a/src/platform/process.cppm +++ b/src/platform/process.cppm @@ -28,14 +28,19 @@ module; #include // _putenv_s #define popen _popen #define pclose _pclose -#elif defined(__linux__) -// Linux is the only platform where the launcher does a direct exec (see -// run_exec / capture_exec below); macOS keeps the std::system shell path. -#include // pipe, dup2, close, read, environ +#elif defined(__linux__) || defined(__APPLE__) +// Linux and macOS launchers do a direct exec (see run_exec / capture_exec +// below); only Windows keeps the std::system shell path (#248). +#include // pipe, dup2, close, read #include // waitpid -#include // posix_spawnp, posix_spawn_file_actions_* +#include // posix_spawnp, posix_spawn_file_actions_* (incl. addchdir_np) +#if defined(__APPLE__) +#include // _NSGetEnviron — direct `environ` is only linkable + // from executables on Apple, not from dylibs +#else extern "C" char **environ; #endif +#endif export module mcpp.platform.process; @@ -136,7 +141,18 @@ int normalize_exit_code(int rc) { #endif } -#if defined(__linux__) +#if defined(__linux__) || defined(__APPLE__) +// Portable accessor for the host environment block. On Apple, `environ` is +// only linkable from executables (not dylibs), so _NSGetEnviron() is the +// sanctioned spelling; Linux keeps the plain `environ` symbol. +char** host_environ() { +#if defined(__APPLE__) + return *::_NSGetEnviron(); +#else + return environ; +#endif +} + // Build a child environment block = the current environ with `extra` overrides // applied. Returned vector owns the strings; the caller derives a NUL-terminated // char* array from it. Built in the PARENT so the child env never requires a @@ -147,7 +163,7 @@ std::vector merged_environ( std::vector out; std::set overridden; for (auto& [k, v] : extra) { out.push_back(k + "=" + v); overridden.insert(k); } - for (char** e = environ; e && *e; ++e) { + for (char** e = host_environ(); e && *e; ++e) { std::string_view entry(*e); auto eq = entry.find('='); std::string key(eq == std::string_view::npos ? entry : entry.substr(0, eq)); @@ -156,10 +172,11 @@ std::vector merged_environ( return out; } #else -// Build a shell command line from an argv vector. The first token (program) +// Build a shell command line from an argv vector (Windows + residual non-POSIX +// fallback only; Linux/macOS exec directly, #248). The first token (program) // is kept RAW on Windows — quoting it would make cmd.exe's `/c "..."` strip the -// outer quotes and mangle the path (see platform.shell) — and shell-quoted on -// macOS. Remaining args are always shell-quoted. +// outer quotes and mangle the path (see platform.shell) — and shell-quoted +// otherwise. Remaining args are always shell-quoted. std::string command_from_argv(const std::vector& argv) { if (argv.empty()) return ""; #if defined(_WIN32) @@ -285,27 +302,31 @@ int run_passthrough(std::string_view command, std::string* output) { // run_exec / capture_exec are split by platform on purpose: // -// Linux — DIRECT exec via posix_spawn. The runtime env (a bundled-glibc -// LD_LIBRARY_PATH) goes into the child's envp ONLY; it never enters -// mcpp's own environment nor the host /bin/sh. That is the exact -// fix for the newer-glibc `sh:` crash, and a direct exec also drops -// the shell quoting / signal / injection surface entirely. -// macOS / -// Windows — KEEP the proven std::system shell path. The leak does not exist -// here (macOS injects no runtime library env; Windows has no glibc -// symbol versioning), so we deliberately do not swap the launch -// primitive on platforms we cannot iterate on locally. The env is -// applied as a `KEY='val' cmd` prefix (macOS) / _putenv_s (Windows) -// via the existing build_env_prefix / capture_with_env helpers. +// Linux / +// macOS — DIRECT exec via posix_spawn (unified in #248). The extra env goes +// into the child's envp ONLY (merged_environ); it never enters +// mcpp's own environment nor a host /bin/sh. On Linux that is the +// exact fix for the newer-glibc `sh:` crash; on macOS the old shell +// path built `KEY='v' cd && prog`, where POSIX binds the env +// assignments to `cd` alone — the real program (build.mcpp, the +// only env+cwd call site) received NO extra env and lost the whole +// MCPP_* contract. Direct exec also drops the shell quoting / +// signal / injection surface entirely. cwd is applied via +// posix_spawn_file_actions_addchdir_np, available on both glibc +// and macOS 10.15+ (mcpp's floor is macOS 14). +// Windows — KEEP the proven std::system shell path. The env-binding hazard +// does not exist here (env goes through _putenv_s, not a prefix), +// so we deliberately do not swap the launch primitive on a platform +// we cannot iterate on locally. // -// TODO(launcher-unify): if macOS/Windows ever need the same child-only env -// isolation (e.g. they start bundling a runtime), unify both onto posix_spawn -// and a Windows CreateProcess/_spawn equivalent, and delete the shell branch. +// TODO(launcher-unify): Windows is the remaining exception; if it ever needs +// child-only env isolation, move it onto a CreateProcess/_spawn equivalent and +// delete the residual shell branch below. int run_exec(const std::vector& argv, const std::vector>& extraEnv) { if (argv.empty()) return 127; -#if defined(__linux__) +#if defined(__linux__) || defined(__APPLE__) auto envStore = merged_environ(extraEnv); std::vector envp; for (auto& s : envStore) envp.push_back(s.data()); @@ -334,7 +355,7 @@ RunResult capture_exec( { RunResult result; if (argv.empty()) { result.exit_code = 127; return result; } -#if defined(__linux__) +#if defined(__linux__) || defined(__APPLE__) // posix_spawn + a pipe; stdout and stderr both go to the pipe so the // captured text is combined (replaces the old `2>&1`). int fds[2]; diff --git a/tests/unit/test_process_run_exec.cpp b/tests/unit/test_process_run_exec.cpp index bdc478a4..0fb646f4 100644 --- a/tests/unit/test_process_run_exec.cpp +++ b/tests/unit/test_process_run_exec.cpp @@ -7,9 +7,10 @@ import mcpp.platform.process; using namespace mcpp::platform; // These exercise the POSIX launch path with POSIX program paths (/bin/true, -// /bin/sh, /bin/echo) — the env is applied as a `KEY='val' cmd` shell prefix, -// so the parent environment is never mutated. The Windows path is covered by -// the integration build (ninja launched via capture_exec). +// /bin/sh, /bin/echo) — Linux and macOS share the direct posix_spawn path +// (#248 launcher-unify): extraEnv goes into the child's envp only, so the +// parent environment is never mutated. The Windows path is covered by the +// integration build (ninja launched via capture_exec). #if !defined(_WIN32) // The regression that matters: launching with an injected loader var must NOT @@ -57,6 +58,37 @@ TEST(CaptureExec, CapturesStderrCombined) { EXPECT_EQ(r.output, "oops\n"); } +// #248 regression: extraEnv AND cwd BOTH non-empty — the previously untested +// combination (every prior call site passed at most one of the two; the only +// env+cwd caller is the build.mcpp child). The old macOS shell fallback built +// `KEY='v' cd && prog 2>&1`, and POSIX binds the env assignments to the +// first command (`cd`) across `&&` — so the real program received NO extra env +// and build.mcpp lost the whole MCPP_* contract. Linux and macOS now share the +// posix_spawn path, so this locks the contract on both. +TEST(CaptureExec, EnvAndCwdCombinedBothReachChild) { + auto dir = std::filesystem::temp_directory_path() / "mcpp_248_env_cwd"; + std::filesystem::create_directories(dir); + auto r = process::capture_exec( + {"/bin/sh", "-c", "printf '%s|%s' \"$MCPP_TEST_CONTRACT\" \"$PWD\""}, + {{"MCPP_TEST_CONTRACT", "hello"}}, + dir.string()); + EXPECT_EQ(r.exit_code, 0); + // Env value must reach the child (the old prefix bound it to `cd` only). + EXPECT_NE(r.output.find("hello|"), std::string::npos) << r.output; + // Child must actually run in `cwd`. Compare the leaf name, not the full + // path: macOS resolves /var → /private/var in $PWD. + EXPECT_NE(r.output.find("mcpp_248_env_cwd"), std::string::npos) << r.output; +} + +// Companion check for run_exec (no cwd parameter there, but extraEnv must +// still land in the child via the same merged_environ posix_spawn path). +TEST(RunExec, InjectedEnvSurvivesShellChildChain) { + int rc = process::run_exec( + {"/bin/sh", "-c", "[ \"$MCPP_TEST_CONTRACT\" = hello ]"}, + {{"MCPP_TEST_CONTRACT", "hello"}}); + EXPECT_EQ(rc, 0); +} + #else // _WIN32 TEST(RunExec, WindowsCoveredByIntegration) { From 55abc53e015bcf64c4c0656ac1fbcfe8cdd6ba26 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:24:39 +0800 Subject: [PATCH 04/12] fix(build): generated_files materialization skips identical content (keeps incremental builds alive) materialize_generated_files rewrote every entry unconditionally whenever prepare_build ran -> mtime bump -> ninja (mtime-driven, header reached via depfiles) recompiled every TU including the materialized header, even when only an unrelated source changed. Frozen-snapshot packages (compat.ffmpeg: config.h included by 2283 TUs) degraded to full rebuilds on every edit. Now the intended content is byte-compared against the on-disk file and the write is skipped when identical, preserving the mtime. Correctness is unchanged: content changes are already folded into the fingerprint, which owns change detection; the skip only avoids the mtime churn (mirroring the build.mcpp cache design). Path-escape guards, create_directories, and the fingerprint interplay are untouched. e2e 142 locks it: touching an unrelated TU (abandoning the P0 fast-path so prepare re-runs) must keep the generated header's mtime and not recompile the including TU; changing the generated content in mcpp.toml must still rewrite the file and rebuild (no false-fresh). A truly no-op rebuild never reaches materialization (fast-path), so the unrelated-touch case is the load-bearing repro - verified failing on the unfixed binary. --- src/build/prepare.cppm | 19 +++++ tests/e2e/142_generated_files_incremental.sh | 90 ++++++++++++++++++++ 2 files changed, 109 insertions(+) create mode 100755 tests/e2e/142_generated_files_incremental.sh diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 7636a692..e7e949c5 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -346,6 +346,25 @@ materialize_generated_files(const std::filesystem::path& root, out.string(), ec.message())); } + // Skip the write when the on-disk content is already identical: ninja + // is mtime-driven, and an unconditional rewrite bumps the mtime every + // build, recompiling every TU that #includes the materialized file + // (via depfiles) — e.g. a frozen-snapshot config.h included by + // thousands of TUs. Change detection is already owned by the + // fingerprint (content is folded in above), so skipping only + // preserves the mtime — mirroring the build.mcpp cache design, + // which likewise avoids mtime churn on unchanged outputs. + { + std::ifstream is(out, std::ios::binary); + if (is) { + std::string existing((std::istreambuf_iterator(is)), + std::istreambuf_iterator()); + if (is && existing == content) { + continue; + } + } + } + std::ofstream os(out, std::ios::binary); if (!os) { return std::unexpected(std::format( diff --git a/tests/e2e/142_generated_files_incremental.sh b/tests/e2e/142_generated_files_incremental.sh new file mode 100755 index 00000000..3c00c2e5 --- /dev/null +++ b/tests/e2e/142_generated_files_incremental.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# requires: gcc +# Incremental x generated_files: an unchanged [generated_files] entry must NOT +# be rewritten when prepare_build runs — ninja is mtime-driven, so a gratuitous +# rewrite of a materialized header recompiles every TU that #includes it (via +# depfiles). A truly no-op rebuild is masked by the P0 fast-path (prepare is +# skipped), so the load-bearing case is: touch an UNRELATED source, which +# abandons the fast-path and re-runs prepare (and materialization). Assert the +# generated header keeps its mtime and the including TU is not recompiled; +# a content change must still rewrite and rebuild (no false-fresh). +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +mtime() { stat -c %Y "$1" 2>/dev/null || stat -f %m "$1"; } + +cd "$TMP" +"$MCPP" new genincr > /dev/null +cd genincr +rm -f src/main.cpp + +cat > mcpp.toml <<'EOF' +[package] +name = "genincr" +version = "0.1.0" + +[generated_files] +"src/gen/config.h" = """ +#define GEN_ANSWER 41 +""" +EOF + +cat > src/main.cpp <<'EOF' +#include "gen/config.h" +#include +extern int other(); +int main() { + std::printf("gen = %d\n", GEN_ANSWER + other()); + return 0; +} +EOF + +cat > src/other.cpp <<'EOF' +int other() { return 0; } +EOF + +"$MCPP" build > build.log 2>&1 || { cat build.log; echo "build failed"; exit 1; } +[[ -f src/gen/config.h ]] || { echo "generated file not materialized"; exit 1; } + +main_obj=$(find target -name 'main*.o' | head -1) +other_obj=$(find target -name 'other*.o' | head -1) +[[ -n "$main_obj" && -n "$other_obj" ]] || { echo "object files not found under target/"; exit 1; } + +gen_before=$(mtime src/gen/config.h) +main_before=$(mtime "$main_obj") + +sleep 1 # make a rewrite observable at 1s mtime granularity + +# Touch an unrelated TU: the fast-path is abandoned, prepare_build re-runs, +# and generated_files are re-materialized. +touch src/other.cpp +# (c) the incremental rebuild still succeeds ... +"$MCPP" build > rebuild.log 2>&1 || { cat rebuild.log; echo "incremental rebuild failed"; exit 1; } +# ... and other.o really was recompiled (prepare + compile did run). +other_after=$(mtime "$other_obj") +[[ "$other_after" -gt "$gen_before" ]] || { echo "other.o was not recompiled — test premise broken"; exit 1; } +# (a) the generated header was NOT rewritten (mtime preserved) ... +gen_after=$(mtime src/gen/config.h) +[[ "$gen_before" == "$gen_after" ]] || { echo "generated file rewritten on incremental rebuild ($gen_before -> $gen_after)"; exit 1; } +# (b) ... so main.o (which #includes it, untouched) was NOT recompiled. +main_after=$(mtime "$main_obj") +[[ "$main_before" == "$main_after" ]] || { echo "main.o recompiled on incremental rebuild ($main_before -> $main_after)"; exit 1; } + +# Content change: the file must be rewritten and the TU recompiled (no +# false-fresh — change detection lives in the fingerprint). +sed -i.bak 's/GEN_ANSWER 41/GEN_ANSWER 42/' mcpp.toml +sleep 1 +"$MCPP" build > change.log 2>&1 || { cat change.log; echo "rebuild after content change failed"; exit 1; } +gen_changed=$(mtime src/gen/config.h) +[[ "$gen_after" != "$gen_changed" ]] || { echo "generated file not rewritten after content change"; exit 1; } +grep -q "GEN_ANSWER 42" src/gen/config.h || { echo "generated file has stale content"; exit 1; } +# The content change alters the fingerprint, which may relocate the build +# dir — assert on the NEWEST main*.o, wherever it lives. +main_changed=$(find target -name 'main*.o' | while read -r f; do mtime "$f"; done | sort -n | tail -1) +[[ "$main_changed" -ge "$gen_changed" ]] || { echo "main.o not recompiled after content change"; exit 1; } +out="$("$MCPP" run 2>&1 | tail -1)" +[[ "$out" == "gen = 42" ]] || { echo "stale binary after content change: $out"; exit 1; } + +echo "OK" From 98c8f61f503fe7e24f18d6bd9f799c1b5631b763 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:30:13 +0800 Subject: [PATCH 05/12] feat(build): #249 include_dirs_after -> -idirafter (dep source roots stop shadowing system headers) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause: dependency include_dirs are always emitted as -I (highest priority). On case-insensitive macOS filesystems a dep's extracted-tarball root on -I that contains a file named like a standard header shadows it for every C++ TU of every consumer — ffmpeg's top-level VERSION file shadows libc++'s , breaking any consumer that touches the standard library. There was no lower-priority include variant in the descriptor vocabulary. Why -idirafter and not -isystem: -isystem dirs still precede the default system dirs, so the poison file still wins. -idirafter dirs are searched AFTER all system dirs (gcc+clang), so system headers win while non-system headers () are still found. Full chain: - manifest/types.cppm: BuildConfig.includeDirsAfter next to includeDirs - manifest/xpkg.cppm: include_dirs_after key (+ include_after / include_dir_after aliases), parse branch mirroring include_dirs; platform overlays get it for free via the shared key loop - manifest/toml.cppm: [build] include_dirs_after - modgraph: UsageRequirements.includeDirsAfter, SourceUnit.localIncludeDirsAfter, threaded through scan_package / scan_packages / scan_packages_p1689 with the same expand_dir_glob treatment (`*` extracted-tarball-root convention) - build/prepare.cppm: expansion into privateBuild/publicUsage, propagation along the SAME dependency edges as includeDirs — consumers receive them as after-dirs, never upgraded to -I; after-dirs enter the resolved-usage identity string; staged-secondary absolutization mirrored - build/plan.cppm: CompileUnit.localIncludeDirsAfter populated wherever localIncludeDirs is (scanned units + synthesized entry main) - build/ninja_backend.cppm: emitted as -idirafter into $local_includes, appended after the -I entries (flag semantics, not position, carry the priority). MSVC dialect: cl.exe has no -idirafter — documented degradation to trailing /I (clang-MSVC uses the gnu dialect and gets real -idirafter) - build/flags.cppm: main-manifest [build] include_dirs_after on the global typed-path channel (-idirafter / trailing /I; nasm degrades to -I) - build/compile_commands.cppm: -idirafter spelling preserved in the compile DB so clangd reproduces the search order Adjacent consistency fix: plan.cppm local_include_dirs_for_manifest did a bare root/inc join with no glob expansion, unlike the dep path in prepare.cppm. Now unified on expand_dir_glob, with a plain-join fallback for literal wildcard-free entries that don't exist yet (expand_dir_glob only returns existing dirs; the old unconditional join must keep working for dirs created later by build steps). Tests: unit (ninja -idirafter emission + ordering, msvc /I degradation, toml + xpkg parse), e2e 141 (poisoned stdlib.h + real mylib.h in a path dep's include root: include_dirs_after builds and runs, plain include_dirs control fails on the poison). --- docs/05-mcpp-toml.md | 13 ++++ docs/zh/05-mcpp-toml.md | 11 +++ src/build/compile_commands.cppm | 5 ++ src/build/flags.cppm | 18 +++++ src/build/ninja_backend.cppm | 26 ++++++-- src/build/plan.cppm | 42 +++++++++++- src/build/prepare.cppm | 44 ++++++++++++ src/manifest/toml.cppm | 4 ++ src/manifest/types.cppm | 2 + src/manifest/xpkg.cppm | 22 +++++- src/modgraph/graph.cppm | 3 + src/modgraph/scanner.cppm | 38 ++++++++++- tests/e2e/141_include_dirs_after.sh | 100 ++++++++++++++++++++++++++++ tests/unit/test_manifest.cpp | 52 +++++++++++++++ tests/unit/test_ninja_backend.cpp | 94 ++++++++++++++++++++++++++ 15 files changed, 466 insertions(+), 8 deletions(-) create mode 100755 tests/e2e/141_include_dirs_after.sh diff --git a/docs/05-mcpp-toml.md b/docs/05-mcpp-toml.md index f339f682..6ee702d2 100644 --- a/docs/05-mcpp-toml.md +++ b/docs/05-mcpp-toml.md @@ -136,6 +136,7 @@ the package/feature boundary, not on an individual target. [build] sources = ["src/**/*.cppm", "src/**/*.cpp"] # Source globs (default: src/**/*.{cppm,cpp,cc,c,S,s,asm}) include_dirs = ["include", "third_party/include"] # Header search paths +include_dirs_after = ["*"] # Header dirs searched AFTER system dirs (-idirafter) c_standard = "c11" # Standard for C source files (default c11) cflags = ["-DFOO=1"] # Extra C compile flags cxxflags = ["-DBAR=2"] # Extra C++ compile flags (do not put -std=... here) @@ -146,6 +147,18 @@ target = "x86_64-linux-musl" # Default build target when no --target is pa macos_deployment_target = "14.0" # Minimum supported OS version for macOS artifacts (macOS only) ``` +`include_dirs_after` (#249) lists header directories that are searched **after** +the toolchain's system directories (emitted as `-idirafter` on GCC/Clang, as +trailing `/I` under the MSVC dialect, which has no equivalent). Use it instead of +`include_dirs` when the directory is an extracted source-tarball root that +contains files whose names collide with standard headers — e.g. ffmpeg's +top-level `VERSION` file shadows libc++'s `` on case-insensitive macOS +filesystems when the root is put on `-I`. With `include_dirs_after` the system +header always wins while the package's real headers (``) +remain findable. Entries support the same `*` glob convention as +`include_dirs`, and they propagate to dependent packages along the same edges — +consumers receive them as after-dirs, never upgraded to `-I`. + `macos_deployment_target` sets the minimum system version in the artifact's Mach-O header (`LC_BUILD_VERSION minos`), i.e. the oldest macOS the binary can run on. The precedence follows ecosystem convention: the `MACOSX_DEPLOYMENT_TARGET` diff --git a/docs/zh/05-mcpp-toml.md b/docs/zh/05-mcpp-toml.md index 31fd131b..92384de8 100644 --- a/docs/zh/05-mcpp-toml.md +++ b/docs/zh/05-mcpp-toml.md @@ -131,6 +131,7 @@ mcpp 刻意不在一次构建里把同一个共享源编译成两份:一个源 [build] sources = ["src/**/*.cppm", "src/**/*.cpp"] # 源文件 glob(默认: src/**/*.{cppm,cpp,cc,c,S,s,asm}) include_dirs = ["include", "third_party/include"] # 头文件搜索路径 +include_dirs_after = ["*"] # 排在系统目录之后搜索的头文件目录(-idirafter) c_standard = "c11" # C 源文件的标准(默认 c11) cflags = ["-DFOO=1"] # 额外 C 编译参数 cxxflags = ["-DBAR=2"] # 额外 C++ 编译参数(不要放 -std=...) @@ -139,6 +140,16 @@ static_stdlib = true # 静态链接 libstdc++(默认 true) macos_deployment_target = "14.0" # macOS 产物的最低支持系统版本(仅 macOS 生效) ``` +`include_dirs_after`(#249)列出**排在工具链系统目录之后**搜索的头文件目录 +(GCC/Clang 发射为 `-idirafter`;MSVC 方言无对应能力,退化为排在末尾的 +`/I`)。当目录是解压后的源码 tarball 根目录、且其中的文件名会与标准头冲突时, +用它代替 `include_dirs` —— 例如 ffmpeg 根目录的 `VERSION` 文件在大小写不敏感 +的 macOS 文件系统上会把 libc++ 的 `` 遮蔽(若该根目录挂在 `-I` 上)。 +使用 `include_dirs_after` 时系统头永远优先,而包自己的真实头文件 +(``)仍能找到。条目支持与 `include_dirs` 相同的 `*` glob +约定,并沿相同的依赖边传播给消费者 —— 消费者收到的仍是 after 目录, +永远不会被升级为 `-I`。 + `macos_deployment_target` 设定产物 Mach-O 头里的最低系统版本 (`LC_BUILD_VERSION minos`),即二进制能运行的最老 macOS。优先级与各生态 惯例一致:环境变量 `MACOSX_DEPLOYMENT_TARGET`(单次调用的显式覆盖, diff --git a/src/build/compile_commands.cppm b/src/build/compile_commands.cppm index d2c47629..eeef67f8 100644 --- a/src/build/compile_commands.cppm +++ b/src/build/compile_commands.cppm @@ -104,6 +104,11 @@ std::vector local_include_args(const CompileUnit& cu) { for (auto const& inc : cu.localIncludeDirs) { args.push_back("-I" + inc.string()); } + // #249: after-dirs keep their -idirafter spelling in the compile DB so + // tooling (clangd) reproduces the compiler's search order. + for (auto const& inc : cu.localIncludeDirsAfter) { + args.push_back("-idirafter" + inc.string()); + } return args; } diff --git a/src/build/flags.cppm b/src/build/flags.cppm index 594b2704..f4625a2c 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -224,6 +224,17 @@ CompileFlags compute_flags(const BuildPlan& plan) { std::filesystem::path p = inc.has_root_path() ? inc : (plan.projectRoot / inc); includeTokens.push_back(std::string(d.includePrefix) + p.string()); } + // #249: `[build] include_dirs_after` — searched AFTER the toolchain's + // system dirs via -idirafter (gcc+clang), so entries can't shadow + // standard headers. cl.exe has no -idirafter; under the msvc dialect + // they degrade to regular /I appended at the END of the include list + // (documented degradation; clang-MSVC uses the gnu dialect). + const bool msvcInclude = d.includePrefix == std::string_view("/I"); + for (auto& inc : plan.manifest.buildConfig.includeDirsAfter) { + std::filesystem::path ip(inc); + std::filesystem::path p = ip.has_root_path() ? ip : (plan.projectRoot / ip); + includeTokens.push_back((msvcInclude ? "/I" : "-idirafter") + p.string()); + } std::string include_flags; for (auto& t : includeTokens) { include_flags += ' '; @@ -405,6 +416,13 @@ CompileFlags compute_flags(const BuildPlan& plan) { auto abs = inc.is_absolute() ? inc : (plan.projectRoot / inc); nasm_includes += " -I" + escape_path(abs); } + // #249: nasm has no system header dirs to defer to — after-dirs + // degrade to plain -I appended at the end. + for (auto& inc : plan.manifest.buildConfig.includeDirsAfter) { + std::filesystem::path ip(inc); + auto abs = ip.is_absolute() ? ip : (plan.projectRoot / ip); + nasm_includes += " -I" + escape_path(abs); + } std::string nasm_debug; if (prof.debug && plan.nasmFormat.starts_with("elf")) nasm_debug = " -g -F dwarf"; diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index d2362cdf..127b5b92 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -87,12 +87,30 @@ std::string escape_flag_path(const std::filesystem::path& p) { return out; } -std::string local_include_flags(const CompileUnit& cu) { +std::string local_include_flags(const CompileUnit& cu, bool msvcDialect, + bool nasmUnit) { std::string flags; for (auto const& inc : cu.localIncludeDirs) { flags += " -I"; flags += escape_flag_path(inc); } + // #249: after-dirs are searched AFTER the toolchain's system dirs + // (-idirafter, gcc+clang), so a dep source root that contains a file + // named like a standard header (ffmpeg's VERSION vs libc++'s + // on case-insensitive macOS) can't shadow it, while its real headers + // are still found. Appended after the -I entries — the flag's + // semantics, not its position, carry the priority; the ordering is + // just for readability. Two degradations, both safe because only the + // C/C++ frontends have a system-header chain to protect: + // • cl.exe has no -idirafter → plain /I at the END of the list + // (clang targeting MSVC uses the gnu dialect and gets the real flag); + // • nasm_object edges share $local_includes but NASM would parse + // `-idirafter

` as its `-i` option with value `dirafter

` — + // a silently wrong search dir — so nasm units get plain -I. + for (auto const& inc : cu.localIncludeDirsAfter) { + flags += nasmUnit ? " -I" : (msvcDialect ? " /I" : " -idirafter"); + flags += escape_flag_path(inc); + } return flags; } @@ -732,7 +750,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { append(std::format("build {} : cxx_scan {}\n", escape_ninja_path(ddi), escape_ninja_path(cu.source))); append(std::format(" compile_target = {}\n", escape_ninja_path(cu.object))); - if (auto includes = local_include_flags(cu); !includes.empty()) + if (auto includes = local_include_flags(cu, msvcDeps, is_nasm_source(cu.source)); !includes.empty()) append(std::format(" local_includes ={}\n", includes)); if (auto flags = join_flags(cu.packageCxxflags); !flags.empty()) append(std::format(" unit_cxxflags ={}\n", flags)); @@ -811,7 +829,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { } else { out_line += "\n"; } - if (auto includes = local_include_flags(cu); !includes.empty()) + if (auto includes = local_include_flags(cu, msvcDeps, is_nasm_source(cu.source)); !includes.empty()) out_line += " local_includes =" + includes + "\n"; if (is_gas_source(cu.source) || is_nasm_source(cu.source)) { if (auto flags = join_flags(asm_unit_flags(cu)); !flags.empty()) @@ -861,7 +879,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { if (!implicit.empty()) out_line += " |" + implicit; out_line += "\n"; - if (auto includes = local_include_flags(cu); !includes.empty()) + if (auto includes = local_include_flags(cu, msvcDeps, is_nasm_source(cu.source)); !includes.empty()) out_line += " local_includes =" + includes + "\n"; if (is_gas_source(cu.source) || is_nasm_source(cu.source)) { if (auto flags = join_flags(asm_unit_flags(cu)); !flags.empty()) diff --git a/src/build/plan.cppm b/src/build/plan.cppm index bd6c872a..68eaff8e 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -22,6 +22,9 @@ struct CompileUnit { std::filesystem::path object; // relative to plan.outputDir std::string packageName; std::vector localIncludeDirs; + // #249: emitted as -idirafter (searched after the toolchain's system + // dirs) — a dep source root on this list can't shadow standard headers. + std::vector localIncludeDirsAfter; std::vector packageCflags; std::vector packageCxxflags; std::vector packageAsmflags; // per-glob asmflags (G4) @@ -239,13 +242,46 @@ std::vector shared_library_link_flags(const mcpp::manifest::Target& return flags; } +// #249 consistency fix: expand include_dirs entries with the same +// `expand_dir_glob` the dep path (prepare.cppm) uses, so a main-manifest +// `include_dirs = ["*/include"]` glob works identically here. For a literal +// (wildcard-free) entry expand_dir_glob only returns EXISTING directories, +// whereas this helper historically joined unconditionally — keep the plain +// join as a fallback so an -I for a dir created later (e.g. by a build +// step) isn't silently dropped. +std::vector +expand_manifest_include_entry(const std::filesystem::path& root, + const std::filesystem::path& inc) +{ + if (inc.is_absolute()) return { inc }; + const auto glob = inc.generic_string(); + auto expanded = mcpp::modgraph::expand_dir_glob(root, glob); + if (expanded.empty() && glob.find('*') == std::string::npos) + expanded.push_back(root / inc); + return expanded; +} + std::vector local_include_dirs_for_manifest(const std::filesystem::path& root, const mcpp::manifest::Manifest& manifest) { std::vector dirs; for (auto const& inc : manifest.buildConfig.includeDirs) { - dirs.push_back(inc.is_absolute() ? inc : root / inc); + for (auto& d : expand_manifest_include_entry(root, inc)) + dirs.push_back(std::move(d)); + } + return dirs; +} + +// #249: same, for the -idirafter channel. +std::vector +local_include_dirs_after_for_manifest(const std::filesystem::path& root, + const mcpp::manifest::Manifest& manifest) +{ + std::vector dirs; + for (auto const& inc : manifest.buildConfig.includeDirsAfter) { + for (auto& d : expand_manifest_include_entry(root, std::filesystem::path(inc))) + dirs.push_back(std::move(d)); } return dirs; } @@ -527,6 +563,7 @@ make_plan(const mcpp::manifest::Manifest& manifest, cu.source = u.path; cu.packageName = u.packageName; cu.localIncludeDirs = u.localIncludeDirs; + cu.localIncludeDirsAfter = u.localIncludeDirsAfter; cu.packageCflags = u.packageCflags; cu.packageCxxflags = u.packageCxxflags; cu.packageAsmflags = u.packageAsmflags; @@ -797,10 +834,13 @@ make_plan(const mcpp::manifest::Manifest& manifest, main_cu.packageName = qualified_package_name(manifest); if (!packages.empty() && packages[0].usageResolved) { main_cu.localIncludeDirs = packages[0].privateBuild.includeDirs; + main_cu.localIncludeDirsAfter = packages[0].privateBuild.includeDirsAfter; main_cu.packageCflags = packages[0].privateBuild.cflags; main_cu.packageCxxflags = packages[0].privateBuild.cxxflags; } else { main_cu.localIncludeDirs = local_include_dirs_for_manifest(projectRoot, manifest); + main_cu.localIncludeDirsAfter = + local_include_dirs_after_for_manifest(projectRoot, manifest); main_cu.packageCflags = manifest.buildConfig.cflags; main_cu.packageCxxflags = manifest.buildConfig.cxxflags; } diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index e7e949c5..c11e4c7f 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -297,6 +297,14 @@ std::string canonical_package_build_metadata( s += " public_include:"; s += dir.generic_string(); } + for (auto const& dir : pkg.privateBuild.includeDirsAfter) { + s += " private_include_after:"; + s += dir.generic_string(); + } + for (auto const& dir : pkg.publicUsage.includeDirsAfter) { + s += " public_include_after:"; + s += dir.generic_string(); + } } for (auto const& [path, content] : pkg.manifest.buildConfig.generatedFiles) { s += " genfile:"; @@ -1945,6 +1953,27 @@ prepare_build(bool print_fingerprint, return dirs; }; + // #249: same glob expansion for `include_dirs_after` (the -idirafter + // channel — searched after the toolchain's system dirs). + auto expandIncludeDirsAfter = + [&](const std::filesystem::path& packageRoot, + const mcpp::manifest::Manifest& manifest) + { + std::vector dirs; + for (auto const& inc : manifest.buildConfig.includeDirsAfter) { + std::filesystem::path p(inc); + if (p.is_absolute()) { + appendUniquePath(dirs, p); + continue; + } + for (auto& dir : mcpp::modgraph::expand_dir_glob( + packageRoot, p.generic_string())) { + appendUniquePath(dirs, dir); + } + } + return dirs; + }; + auto makePackageRoot = [&](const std::filesystem::path& packageRoot, const mcpp::manifest::Manifest& manifest) @@ -1955,9 +1984,11 @@ prepare_build(bool print_fingerprint, pkg.usageResolved = true; pkg.privateBuild.includeDirs = expandIncludeDirs(packageRoot, manifest); + pkg.privateBuild.includeDirsAfter = expandIncludeDirsAfter(packageRoot, manifest); pkg.privateBuild.cflags = manifest.buildConfig.cflags; pkg.privateBuild.cxxflags = manifest.buildConfig.cxxflags; pkg.publicUsage.includeDirs = pkg.privateBuild.includeDirs; + pkg.publicUsage.includeDirsAfter = pkg.privateBuild.includeDirsAfter; pkg.linkUsage.ldflags = manifest.buildConfig.ldflags; return pkg; }; @@ -2010,6 +2041,12 @@ prepare_build(bool print_fingerprint, changed = appendUniquePaths(consumer.privateBuild.includeDirs, dependency.publicUsage.includeDirs) || changed; + // #249: after-dirs ride the same edges but keep their + // after-ness — consumers receive them as -idirafter, + // never upgraded to -I. + changed = appendUniquePaths(consumer.privateBuild.includeDirsAfter, + dependency.publicUsage.includeDirsAfter) + || changed; // Interface defines (a dependency's active-feature `defines`) // ride the same edges as include dirs: they must reach the // consumer's own TUs so header-only switches like @@ -2026,6 +2063,9 @@ prepare_build(bool print_fingerprint, changed = appendUniquePaths(consumer.publicUsage.includeDirs, dependency.publicUsage.includeDirs) || changed; + changed = appendUniquePaths(consumer.publicUsage.includeDirsAfter, + dependency.publicUsage.includeDirsAfter) + || changed; changed = appendUniqueFlags(consumer.publicUsage.cflags, dependency.publicUsage.cflags) || changed; @@ -2415,6 +2455,10 @@ prepare_build(bool print_fingerprint, for (auto& inc : stagedManifest.buildConfig.includeDirs) { if (inc.is_relative()) inc = secondaryRoot / inc; } + for (auto& inc : stagedManifest.buildConfig.includeDirsAfter) { + if (std::filesystem::path p(inc); p.is_relative()) + inc = (secondaryRoot / p).generic_string(); + } dep_manifests.push_back( std::make_unique(std::move(stagedManifest))); diff --git a/src/manifest/toml.cppm b/src/manifest/toml.cppm index 1d4848f3..2d35269c 100644 --- a/src/manifest/toml.cppm +++ b/src/manifest/toml.cppm @@ -165,6 +165,10 @@ std::expected parse_string(std::string_view content, if (auto v = doc->get_string_array("build.include_dirs")) { for (auto& s : *v) m.buildConfig.includeDirs.emplace_back(s); } + // [build].include_dirs_after (#249) — searched after system dirs (-idirafter). + if (auto v = doc->get_string_array("build.include_dirs_after")) { + for (auto& s : *v) m.buildConfig.includeDirsAfter.emplace_back(s); + } // [targets.*] — M5.0: now optional. If absent, defer to auto-inference (in load()). // [profile.] — bundled build settings. diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index 3c6e47b6..1cf949a4 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -152,6 +152,8 @@ struct BuildConfig { // .agents/docs/2026-06-29-feature-capability-model-design.md. std::map> featureDefines; std::vector includeDirs; // relative to package root + // #249: emitted as -idirafter (searched after system dirs) + std::vector includeDirsAfter; std::map generatedFiles; // Form B package-owned support files std::vector globFlags; // [build] flags = [...] (ordered) bool staticStdlib = true; diff --git a/src/manifest/xpkg.cppm b/src/manifest/xpkg.cppm index 9178d91d..af7cea00 100644 --- a/src/manifest/xpkg.cppm +++ b/src/manifest/xpkg.cppm @@ -125,7 +125,8 @@ namespace mcpp::manifest { // the parser accepts. inline constexpr std::string_view kKnownXpkgKeys[] = { "cflags", "c_standard", "cxxflags", "deps", "features", "flags", - "generated_files", "import_std", "include_dirs", "language", "ldflags", + "generated_files", "import_std", "include_dirs", "include_dirs_after", + "language", "ldflags", "linux", "macosx", "modules", "provides", "runtime", "scan_overrides", "schema", "sources", "target_cfg", "targets", "windows", }; @@ -143,6 +144,8 @@ inline constexpr std::pair kXpkgKeyAliases[] { "target", "targets" }, { "include", "include_dirs" }, { "include_dir", "include_dirs" }, + { "include_after", "include_dirs_after" }, + { "include_dir_after", "include_dirs_after" }, }; std::string closest_known_xpkg_key(std::string_view unknownKey) { @@ -900,6 +903,23 @@ synthesize_from_xpkg_lua(std::string_view luaContent, } cur.consume('}'); } + else if (key == "include_dirs_after") { + // #249: header dirs searched AFTER the toolchain's system dirs + // (-idirafter). Use for extracted-tarball roots that contain + // files shadowing standard headers on case-insensitive + // filesystems (ffmpeg's VERSION vs libc++'s ). + if (!cur.consume('{')) { + return std::unexpected(ManifestError{ + "expected '{' after `include_dirs_after =`", m.sourcePath, 0, 0}); + } + cur.skip_ws_and_comments(); + while (!cur.eof() && cur.peek() != '}') { + auto s = cur.read_string(); + if (!s.empty()) m.buildConfig.includeDirsAfter.emplace_back(s); + cur.skip_ws_and_comments(); + } + cur.consume('}'); + } else if (key == "provides") { // Package-level capabilities (Feature System v2 S3): this package // satisfies the listed abstract capability names for any dependent diff --git a/src/modgraph/graph.cppm b/src/modgraph/graph.cppm index 38878ca8..05312bf6 100644 --- a/src/modgraph/graph.cppm +++ b/src/modgraph/graph.cppm @@ -26,6 +26,9 @@ struct SourceUnit { std::filesystem::path relPath; std::string packageName; std::vector localIncludeDirs; + // #249: dirs emitted as -idirafter — searched after the toolchain's + // system dirs, so a dep's source root can't shadow standard headers. + std::vector localIncludeDirsAfter; std::vector packageCflags; std::vector packageCxxflags; std::vector packageAsmflags; // per-glob asmflags (G4) diff --git a/src/modgraph/scanner.cppm b/src/modgraph/scanner.cppm index 10e9126c..d493ef99 100644 --- a/src/modgraph/scanner.cppm +++ b/src/modgraph/scanner.cppm @@ -89,6 +89,9 @@ enum class DependencyVisibility { struct UsageRequirements { std::vector includeDirs; + // #249: emitted as -idirafter (searched after system dirs); propagated + // along the same consumer edges as includeDirs, never upgraded to -I. + std::vector includeDirsAfter; std::vector cflags; std::vector cxxflags; std::vector ldflags; @@ -750,6 +753,26 @@ local_include_dirs_for(const std::filesystem::path& root, return dirs; } +// #249: same expansion for `include_dirs_after` entries (the -idirafter +// channel). Shares the `*` extracted-tarball-root glob convention. +std::vector +local_include_dirs_after_for(const std::filesystem::path& root, + const mcpp::manifest::Manifest& manifest) +{ + std::vector dirs; + for (auto const& inc : manifest.buildConfig.includeDirsAfter) { + std::filesystem::path p(inc); + if (p.is_absolute()) { + dirs.push_back(std::move(p)); + continue; + } + for (auto& d : expand_dir_glob(root, p.generic_string())) { + dirs.push_back(std::move(d)); + } + } + return dirs; +} + // Phase 1: scan a single package, append units to result.graph.units; // errors go straight into result.errors. producerOf/edges are NOT built // here — the caller does that after all packages are scanned. @@ -757,6 +780,7 @@ void scan_one_into(ScanResult& result, const std::filesystem::path& root, const mcpp::manifest::Manifest& manifest, const std::vector& localIncludeDirs, + const std::vector& localIncludeDirsAfter, const std::vector& packageCflags, const std::vector& packageCxxflags) { @@ -871,6 +895,7 @@ void scan_one_into(ScanResult& result, for (auto const& name : ov->imports) u.requires_.push_back(ModuleId{name}); u.localIncludeDirs = localIncludeDirs; + u.localIncludeDirsAfter = localIncludeDirsAfter; u.packageCflags = packageCflags; u.packageCxxflags = packageCxxflags; apply_glob_flags(u); @@ -889,6 +914,7 @@ void scan_one_into(ScanResult& result, // scan_overrides branch above. r->relPath = std::filesystem::relative(f, root); r->localIncludeDirs = localIncludeDirs; + r->localIncludeDirsAfter = localIncludeDirsAfter; r->packageCflags = packageCflags; r->packageCxxflags = packageCxxflags; apply_glob_flags(*r); @@ -954,7 +980,8 @@ ScanResult scan_package(const std::filesystem::path& root, { ScanResult result; auto localIncludeDirs = local_include_dirs_for(root, manifest); - scan_one_into(result, root, manifest, localIncludeDirs, + auto localIncludeDirsAfter = local_include_dirs_after_for(root, manifest); + scan_one_into(result, root, manifest, localIncludeDirs, localIncludeDirsAfter, manifest.buildConfig.cflags, manifest.buildConfig.cxxflags); resolve_graph(result); return result; @@ -966,6 +993,9 @@ ScanResult scan_packages(const std::vector& packages) { auto localIncludeDirs = p.usageResolved ? p.privateBuild.includeDirs : local_include_dirs_for(p.root, p.manifest); + auto localIncludeDirsAfter = p.usageResolved + ? p.privateBuild.includeDirsAfter + : local_include_dirs_after_for(p.root, p.manifest); auto const& packageCflags = p.usageResolved ? p.privateBuild.cflags : p.manifest.buildConfig.cflags; @@ -973,7 +1003,7 @@ ScanResult scan_packages(const std::vector& packages) { ? p.privateBuild.cxxflags : p.manifest.buildConfig.cxxflags; scan_one_into(result, p.root, p.manifest, localIncludeDirs, - packageCflags, packageCxxflags); + localIncludeDirsAfter, packageCflags, packageCxxflags); } resolve_graph(result); return result; @@ -993,6 +1023,9 @@ ScanResult scan_packages_p1689(const std::vector& packages, const auto localIncludeDirs = p.usageResolved ? p.privateBuild.includeDirs : local_include_dirs_for(p.root, p.manifest); + const auto localIncludeDirsAfter = p.usageResolved + ? p.privateBuild.includeDirsAfter + : local_include_dirs_after_for(p.root, p.manifest); for (auto const& f : all_files) { auto r = mcpp::modgraph::p1689::scan_file( f, p.manifest.package.name, tc, tmpDir, @@ -1004,6 +1037,7 @@ ScanResult scan_packages_p1689(const std::vector& packages, // mcpp#233: same relPath contract as scan_one_into above. r->relPath = std::filesystem::relative(f, p.root); r->localIncludeDirs = localIncludeDirs; + r->localIncludeDirsAfter = localIncludeDirsAfter; r->packageCflags = p.usageResolved ? p.privateBuild.cflags : p.manifest.buildConfig.cflags; diff --git a/tests/e2e/141_include_dirs_after.sh b/tests/e2e/141_include_dirs_after.sh new file mode 100755 index 00000000..8c781283 --- /dev/null +++ b/tests/e2e/141_include_dirs_after.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash +# requires: elf gcc +# #249: `[build] include_dirs_after` → -idirafter. A path dependency whose +# include root contains BOTH a file named like a standard header (stdlib.h, +# poisoned) AND a real header (mylib.h) must not break consumers: -idirafter +# dirs are searched AFTER the toolchain's system dirs, so the system stdlib.h +# wins while mylib.h is still found. Control: the same dep exposed via plain +# include_dirs (-I, highest priority) picks the poisoned stdlib.h and the +# build fails. This is the Linux encoding of the macOS case-insensitive +# problem where a tarball root's VERSION file shadows libc++'s . +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +cd "$TMP" + +# Dependency — a header-only-ish C++ lib whose include root is poisoned. +"$MCPP" new poisonlib > /dev/null +cd poisonlib +rm -f src/main.cpp +cat > src/lib.cppm <<'EOF' +export module poisonlib.lib; +export int poisonlib_anchor() { return 1; } +EOF +mkdir -p inc +cat > inc/stdlib.h <<'EOF' +#error poisoned +EOF +cat > inc/mylib.h <<'EOF' +#pragma once +inline int mylib_answer() { return 42; } +EOF + +cat > mcpp.toml <<'EOF' +[package] +name = "poisonlib" +version = "0.1.0" +[language] +standard = "c++23" +modules = true +import_std = true +[modules] +sources = ["src/**/*.cppm"] +[build] +include_dirs_after = ["inc"] +[targets.poisonlib] +kind = "lib" +EOF +cd .. + +# Consumer — includes BOTH the standard header the dep poisons and the +# dep's real header. +"$MCPP" new consumer > /dev/null +cd consumer +cat > src/main.cpp <<'EOF' +#include +#include +import std; +int main() { + std::println("answer = {}", mylib_answer()); + return 0; +} +EOF + +cat > mcpp.toml <<'EOF' +[package] +name = "consumer" +version = "0.1.0" +[language] +standard = "c++23" +modules = true +import_std = true +[modules] +sources = ["src/**/*.cppm", "src/**/*.cpp"] +[targets.consumer] +kind = "bin" +main = "src/main.cpp" +[dependencies.poisonlib] +path = "../poisonlib" +EOF + +"$MCPP" build > build.log 2>&1 || { cat build.log; echo "include_dirs_after build failed"; exit 1; } +out=$("$MCPP" run 2>&1) +[[ "$out" == *"answer = 42"* ]] || { echo "dep header not found via -idirafter: $out"; exit 1; } + +# The -idirafter flag (not -I) must carry the dep's include root. +ninja_file="$(find target -name build.ninja)" +grep -q -- '-idirafter[^ ]*poisonlib[^ ]*inc' "$ninja_file" || { + echo "ninja missing -idirafter for dep include root"; exit 1; } + +# Control: the same dir via plain include_dirs (-I) precedes system dirs — +# the poisoned stdlib.h is picked and the build MUST fail. +sed -i.bak 's/include_dirs_after = \["inc"\]/include_dirs = ["inc"]/' ../poisonlib/mcpp.toml +rm -f ../poisonlib/mcpp.toml.bak +rm -rf target +"$MCPP" build > poisoned.log 2>&1 && { echo "expected poisoned build to fail"; exit 1; } +grep -q "poisoned" poisoned.log || { cat poisoned.log; echo "wrong failure (not the poison)"; exit 1; } + +echo "OK" diff --git a/tests/unit/test_manifest.cpp b/tests/unit/test_manifest.cpp index 05a52c74..64ec996b 100644 --- a/tests/unit/test_manifest.cpp +++ b/tests/unit/test_manifest.cpp @@ -340,6 +340,32 @@ kind = "lib" EXPECT_EQ(m->buildConfig.cStandard, "c11"); } +// #249: `[build] include_dirs_after` parses into buildConfig.includeDirsAfter +// — the -idirafter channel (searched AFTER the toolchain's system dirs), so +// an extracted-tarball root containing a file named like a standard header +// (ffmpeg's VERSION vs libc++'s on case-insensitive macOS) can't +// shadow it while its real headers stay findable. +TEST(Manifest, BuildIncludeDirsAfterToml) { + constexpr auto src = R"( +[package] +name = "x" +version = "0.1.0" +[build] +sources = ["src/**/*.cpp"] +include_dirs = ["include"] +include_dirs_after = ["*", "vendor/include"] +[targets.x] +kind = "lib" +)"; + auto m = mcpp::manifest::parse_string(src); + ASSERT_TRUE(m.has_value()) << m.error().format(); + ASSERT_EQ(m->buildConfig.includeDirs.size(), 1u); + EXPECT_EQ(m->buildConfig.includeDirs[0], "include"); + ASSERT_EQ(m->buildConfig.includeDirsAfter.size(), 2u); + EXPECT_EQ(m->buildConfig.includeDirsAfter[0], "*"); + EXPECT_EQ(m->buildConfig.includeDirsAfter[1], "vendor/include"); +} + // Feature System v2 Stage 1: a [features] entry may be a TABLE carrying // package-owned `defines` (and `implies`), while the array shorthand keeps // meaning "implied features". See @@ -887,6 +913,32 @@ package = { EXPECT_EQ(it->second, "#pragma once\n#define TINYC_OK 1\n"); } +// #249: the xpkg descriptor body accepts `include_dirs_after` alongside +// `include_dirs` and keeps the two channels separate (after-dirs are never +// upgraded to -I). +TEST(SynthesizeFromXpkgLua, IncludeDirsAfter) { + constexpr auto src = R"( +package = { + spec = "1", + name = "tinyc", + xpm = { linux = { ["1.0.0"] = { url = "u", sha256 = "h" } } }, + mcpp = { + sources = { "*/src/*.c" }, + include_dirs = { "*/include" }, + include_dirs_after = { "*" }, + targets = { ["tinyc"] = { kind = "lib" } }, + }, +} +)"; + auto m = mcpp::manifest::synthesize_from_xpkg_lua(src, "tinyc", "1.0.0"); + ASSERT_TRUE(m.has_value()) << m.error().format(); + ASSERT_EQ(m->buildConfig.includeDirs.size(), 1u); + EXPECT_EQ(m->buildConfig.includeDirs[0], "*/include"); + ASSERT_EQ(m->buildConfig.includeDirsAfter.size(), 1u); + EXPECT_EQ(m->buildConfig.includeDirsAfter[0], "*"); + EXPECT_TRUE(m->xpkgUnknownKeys.empty()); +} + TEST(Manifest, DependenciesFlatDefaultNamespace) { constexpr auto src = R"( [package] diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index 7f8a527d..8ef0ab3a 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -168,6 +168,100 @@ TEST(NinjaBackend, CxxFlagsIncludeBuildIncludeDirs) { << flags.cxx; } +// #249: a compile unit's localIncludeDirsAfter emit as -idirafter into the +// same $local_includes variable, APPENDED after the -I entries. -idirafter +// dirs are searched after the toolchain's system dirs (gcc+clang), so a dep +// source root containing a file named like a standard header (ffmpeg's +// VERSION vs libc++'s on case-insensitive macOS) can't shadow it +// while the dep's real headers stay findable. The flag's semantics — not +// its position — carry the priority; the -I-then-idirafter ordering is for +// readability. +TEST(NinjaBackend, LocalIncludeDirsAfterEmitIdirafterAppendedAfterDashI) { + auto plan = minimal_plan(); + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "after_test", + .localIncludeDirs = {"/dep/include"}, + .localIncludeDirsAfter = {"/dep/tarball-root"}, + }); + + auto ninja = emit_ninja_string(plan); + + auto line_start = ninja.find("local_includes ="); + ASSERT_NE(line_start, std::string::npos) << ninja; + auto line_end = ninja.find('\n', line_start); + auto line = ninja.substr(line_start, line_end - line_start); + + auto i_pos = line.find("-I/dep/include"); + auto after_pos = line.find("-idirafter/dep/tarball-root"); + ASSERT_NE(i_pos, std::string::npos) << line; + ASSERT_NE(after_pos, std::string::npos) << line; + // After-dirs come after the -I entries within $local_includes. + EXPECT_LT(i_pos, after_pos) << line; + // Never upgraded to -I. + EXPECT_EQ(line.find("-I/dep/tarball-root"), std::string::npos) << line; +} + +// #249 MSVC degradation: cl.exe has no -idirafter, so under the msvc +// dialect after-dirs are emitted as regular /I at the END of the include +// list (clang targeting MSVC uses the gnu dialect and gets -idirafter). +TEST(NinjaBackend, MsvcDialectEmitsIncludeDirsAfterAsTrailingSlashI) { + auto plan = minimal_plan(); + plan.toolchain.compiler = mcpp::toolchain::CompilerId::MSVC; + plan.toolchain.binaryPath = "cl.exe"; + plan.toolchain.targetTriple = "x86_64-pc-windows-msvc"; + plan.compileUnits.push_back({ + .source = "src/main.cpp", + .object = "obj/main.o", + .packageName = "after_test", + .localIncludeDirs = {"/dep/include"}, + .localIncludeDirsAfter = {"/dep/tarball-root"}, + }); + + auto ninja = emit_ninja_string(plan); + + auto line_start = ninja.find("local_includes ="); + ASSERT_NE(line_start, std::string::npos) << ninja; + auto line_end = ninja.find('\n', line_start); + auto line = ninja.substr(line_start, line_end - line_start); + + auto i_pos = line.find("-I/dep/include"); + auto after_pos = line.find("/I/dep/tarball-root"); + ASSERT_NE(i_pos, std::string::npos) << line; + ASSERT_NE(after_pos, std::string::npos) << line; + EXPECT_LT(i_pos, after_pos) << line; + EXPECT_EQ(line.find("-idirafter"), std::string::npos) << line; +} + +// #249 NASM degradation: nasm_object edges share $local_includes, but NASM +// would parse `-idirafter

` as its `-i` option with value `dirafter

` — +// a silently wrong search dir. Only the C/C++ frontends have a system-header +// chain to protect, so a nasm unit's after-dirs degrade to plain -I. +TEST(NinjaBackend, NasmUnitEmitsIncludeDirsAfterAsPlainDashI) { + auto plan = minimal_plan(); + plan.nasmPath = "/usr/bin/nasm"; + plan.compileUnits.push_back({ + .source = "src/scale.asm", + .object = "obj/scale.asm.o", + .packageName = "after_test", + .localIncludeDirs = {"/dep/x86"}, + .localIncludeDirsAfter = {"/dep/tarball-root"}, + }); + + auto ninja = emit_ninja_string(plan); + + auto edge_start = ninja.find("build obj/scale.asm.o : nasm_object"); + ASSERT_NE(edge_start, std::string::npos) << ninja; + auto line_start = ninja.find("local_includes =", edge_start); + ASSERT_NE(line_start, std::string::npos) << ninja; + auto line = ninja.substr(line_start, ninja.find('\n', line_start) - line_start); + + EXPECT_NE(line.find("-I/dep/x86"), std::string::npos) << line; + EXPECT_NE(line.find("-I/dep/tarball-root"), std::string::npos) << line; + EXPECT_EQ(line.find("-idirafter"), std::string::npos) << line; +} + // Cluster A review fix (#226/#234 follow-up): `[build] include_dirs` is a // TYPED PATH channel — bare paths from the manifest, dialect prefix applied // at emission (-I under GNU, /I under MSVC) — not the FLAG-STRING channel From ef28b3c6ba0b5623e13515bde763ed491c30025e Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:31:20 +0800 Subject: [PATCH 06/12] =?UTF-8?q?feat(build):=20mcpp:source=3D=20=E2=80=94?= =?UTF-8?q?=20select=20existing=20files=20into=20the=20compile=20set?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Names the capability that the grey `mcpp:generated=` usage provided by accident: a build.mcpp can now SELECT a pre-existing source (tarball payload / vendored tree) into the compile set, without claiming it wrote the file. Design: 2026-07-19-large-source-pkg-platform-fixes-and- buildmcpp-generation-design.md §3.1 item 1. - parse_line: new `source` directive key; value verbatim into Directives::sources. - Path rules: absolute taken as-is; relative resolves against the PACKAGE ROOT in both root and dependency mode (unlike generated=, whose dep-mode relatives resolve to genBase/OUT_DIR) — a payload file lives in the package tree, never in OUT_DIR. `generated=` behavior is byte-for-byte untouched, including the absolute-path grey usage. - apply(): same landing as generated= (bc.sources + m.modules.sources, both lists so the scanner sees files outside the base globs). - Existence: post-run hard error with a source=-specific message ("selected source ... no such file"); cache_fresh also invalidates when a selected source vanishes (mirrors generated=). - Cache: new `d source ` record kind (write_cache/read_cache round-trip). - Typed lib: mcpp::source(path) mirrors mcpp::generated 1:1. - execute.cppm sources_newer_than: the root build.mcpp is now a fast-path input — editing ONLY build.mcpp previously never fell through to prepare_build (fresh build.ninja short-circuited before the program-hash cache key could see the change), so the documented "re-runs when the build.mcpp source itself changes" was unreachable on the fast path. Found by the new e2e's missing-source case; covers try_fast_build AND try_fast_run. - Docs: directive + typed-API tables (en/zh). No unit seam exists for parse_line/apply (anonymous namespace in the module impl unit) — covered by e2e 143_build_mcpp_source_directive.sh: pre-existing file outside all sources globs is compiled+linked via the typed API, `d source` cache record replays on rebuild without a re-run, and a missing selection fails with the actionable error. Verified: unit 35/35; e2e 143/110/111/125 green. --- docs/07-build-mcpp.md | 2 + docs/zh/07-build-mcpp.md | 2 + src/build/build_program.cppm | 40 +++++++-- src/build/execute.cppm | 10 +++ tests/e2e/143_build_mcpp_source_directive.sh | 90 ++++++++++++++++++++ 5 files changed, 139 insertions(+), 5 deletions(-) create mode 100755 tests/e2e/143_build_mcpp_source_directive.sh diff --git a/docs/07-build-mcpp.md b/docs/07-build-mcpp.md index ab3ebd42..39529850 100644 --- a/docs/07-build-mcpp.md +++ b/docs/07-build-mcpp.md @@ -49,6 +49,7 @@ is ignored, so you can freely log diagnostics. | `mcpp:link-search=

` | add a library search dir (`-L`; relative dirs resolve against the project root) | | `mcpp:cfg=` | define `-D` for both C and C++ | | `mcpp:generated=` | add a generated source (relative to the project root) to the build | +| `mcpp:source=` | select a **pre-existing** source file into the build (absolute, or relative to the package root). Same downstream effect as `generated=`; use it for files the program *chose* (payload/vendored tree) rather than wrote — e.g. a per-target source selection over a large tarball | | `mcpp:rerun-if-changed=` | re-run `build.mcpp` when this file changes | | `mcpp:rerun-if-env-changed=` | re-run `build.mcpp` when this env var changes | @@ -85,6 +86,7 @@ int main() { | `mcpp::link_lib(s)` / `mcpp::link_search(s)` | `mcpp:link-lib=` / `mcpp:link-search=` | | `mcpp::define(s)` | `mcpp:cfg=` (i.e. `-D`) | | `mcpp::generated(p)` | `mcpp:generated=` | +| `mcpp::source(p)` | `mcpp:source=` | | `mcpp::rerun_if_changed(p)` / `mcpp::rerun_if_env_changed(v)` | the matching `rerun-*` directives | If your `build.mcpp` also needs to *write* a generated file, mix in a textual diff --git a/docs/zh/07-build-mcpp.md b/docs/zh/07-build-mcpp.md index 5b98ca99..fa61340c 100644 --- a/docs/zh/07-build-mcpp.md +++ b/docs/zh/07-build-mcpp.md @@ -46,6 +46,7 @@ mcpp build # 编译 + 运行 build.mcpp,然后构建工程 | `mcpp:link-search=` | 增加库搜索目录(`-L`;相对路径按工程根目录解析) | | `mcpp:cfg=` | 为 C 与 C++ 同时定义 `-D` | | `mcpp:generated=` | 把生成的源码(相对工程根目录)加入构建 | +| `mcpp:source=` | 把一份**既有**源文件选入构建(绝对路径,或相对包根)。下游效果与 `generated=` 相同;语义区别在于文件是程序*选中*的(tarball payload / vendored 源树)而非程序写出的——例如对大型源码包做 per-target 源选择 | | `mcpp:rerun-if-changed=` | 该文件变化时重跑 `build.mcpp` | | `mcpp:rerun-if-env-changed=` | 该环境变量变化时重跑 `build.mcpp` | @@ -80,6 +81,7 @@ int main() { | `mcpp::link_lib(s)` / `mcpp::link_search(s)` | `mcpp:link-lib=` / `mcpp:link-search=` | | `mcpp::define(s)` | `mcpp:cfg=`(即 `-D`) | | `mcpp::generated(p)` | `mcpp:generated=` | +| `mcpp::source(p)` | `mcpp:source=` | | `mcpp::rerun_if_changed(p)` / `mcpp::rerun_if_env_changed(v)` | 对应的 `rerun-*` 指令 | 如果 `build.mcpp` 还需要*写*生成文件,混入一个文本 `#include ` 即可——这没问题, diff --git a/src/build/build_program.cppm b/src/build/build_program.cppm index ae36c182..dc489fd6 100644 --- a/src/build/build_program.cppm +++ b/src/build/build_program.cppm @@ -79,6 +79,13 @@ struct Directives { std::vector ldflags; // -> buildConfig.ldflags (already -l/-L) std::vector defines; // cfg= -> -D, into BOTH c/cxx flags std::vector generated; // relative source paths + // source= — select a PRE-EXISTING file (tarball payload / vendored tree) + // into the compile set. Downstream identical to generated=; the semantic + // difference is intent only: the program did not write this file, it chose + // it. Relative paths resolve against the PACKAGE ROOT in both root and + // dependency mode (a payload file lives in the package tree, never in + // OUT_DIR) — unlike generated=, whose dep-mode relatives resolve to genBase. + std::vector sources; std::vector rerunFiles; // declared file inputs std::vector rerunEnv; // declared env-var inputs }; @@ -116,6 +123,7 @@ bool parse_line(const fs::path& root, std::string_view raw, Directives& d) { else if (key == "link-search") d.ldflags.push_back("-L" + abs_against_root(root, val)); else if (key == "cfg") d.defines.push_back("-D" + val); else if (key == "generated") d.generated.push_back(val); + else if (key == "source") d.sources.push_back(val); else if (key == "rerun-if-changed") d.rerunFiles.push_back(val); else if (key == "rerun-if-env-changed") d.rerunEnv.push_back(val); else mcpp::ui::warning(std::format("build.mcpp: ignoring unknown directive 'mcpp:{}'", key)); @@ -235,6 +243,7 @@ inline void link_lib(const char* name) { std::printf("mcpp:link-lib=% inline void link_search(const char* dir) { std::printf("mcpp:link-search=%s\n", dir); } inline void define(const char* name) { std::printf("mcpp:cfg=%s\n", name); } inline void generated(const char* path) { std::printf("mcpp:generated=%s\n", path); } +inline void source(const char* path) { std::printf("mcpp:source=%s\n", path); } inline void rerun_if_changed(const char* path) { std::printf("mcpp:rerun-if-changed=%s\n", path); } inline void rerun_if_env_changed(const char* var) { std::printf("mcpp:rerun-if-env-changed=%s\n", var); } // ── environment contract (read side; values injected by the engine) ───── @@ -326,7 +335,7 @@ build_mcpp_module(const fs::path& bdir, const fs::path& compiler, // compiler // in // env -// d cxxflag|cflag|ldflag|define|generated +// d cxxflag|cflag|ldflag|define|generated|source // The leading program/compiler/in/env lines are the re-run key; the `d` lines // are the directives to reapply on a hit. @@ -423,6 +432,7 @@ void write_cache(const fs::path& bdir, const fs::path& root, emit("ldflag", d.ldflags); emit("define", d.defines); emit("generated", d.generated); + emit("source", d.sources); } struct CacheRecord { @@ -463,6 +473,7 @@ CacheRecord read_cache(const fs::path& bdir) { else if (kind == "ldflag") r.directives.ldflags.push_back(val); else if (kind == "define") r.directives.defines.push_back(val); else if (kind == "generated") r.directives.generated.push_back(val); + else if (kind == "source") r.directives.sources.push_back(val); } } r.loaded = true; @@ -481,9 +492,12 @@ bool cache_fresh(const fs::path& root, const CacheRecord& c, if (mcpp::toolchain::hash_file(abs_against_root(root, path)) != h) return false; for (auto const& [h, name] : c.envs) if (mcpp::toolchain::hash_string(env_value(name)) != h) return false; - // A declared generated output that vanished invalidates the cache. + // A declared generated output / selected source that vanished invalidates + // the cache. for (auto const& g : c.directives.generated) if (!fs::exists(abs_against_root(root, g))) return false; + for (auto const& s : c.directives.sources) + if (!fs::exists(abs_against_root(root, s))) return false; return true; } @@ -495,13 +509,18 @@ void apply(mcpp::manifest::Manifest& m, const Directives& d) { // cfg defines apply to both C and C++ translation units. bc.cflags.insert(bc.cflags.end(), d.defines.begin(), d.defines.end()); bc.cxxflags.insert(bc.cxxflags.end(), d.defines.begin(), d.defines.end()); - // Generated sources join the source set. BOTH lists: the scanner walks the - // legacy modules.sources mirror — pushing only bc.sources left a generated - // file outside the base globs invisible to the scan (latent since L3). + // Generated + selected (source=) sources join the source set. BOTH lists: + // the scanner walks the legacy modules.sources mirror — pushing only + // bc.sources left a generated file outside the base globs invisible to the + // scan (latent since L3). for (auto const& g : d.generated) { bc.sources.push_back(g); m.modules.sources.push_back(g); } + for (auto const& s : d.sources) { + bc.sources.push_back(s); + m.modules.sources.push_back(s); + } } } // namespace @@ -607,6 +626,8 @@ std::expected run_build_program( // Dependency mode (genBase set): relative `generated=` paths resolve // against OUT_DIR-style genBase, not the (possibly read-only, shared) // package root — rewrite them to absolute before validation/apply/cache. + // `source=` paths are NOT rewritten: they name pre-existing files in the + // package tree (MCPP_MANIFEST_DIR-relative), never OUT_DIR products. if (!env.genBase.empty()) { for (auto& g : d.generated) { fs::path gp(g); @@ -621,6 +642,15 @@ std::expected run_build_program( "build.mcpp declared generated source '{}' but it does not exist after the run", g)); } } + // A `source=` selection must already exist — the program selects a file it + // did NOT write (payload / vendored tree); a missing one is a typo or a + // broken payload, surfaced now instead of as a later glob no-match. + for (auto const& s : d.sources) { + if (!fs::exists(abs_against_root(root, s))) { + return std::unexpected(std::format( + "build.mcpp selected source '{}' (mcpp:source=) but no such file exists", s)); + } + } apply(m, d); write_cache(bdir, root, programHash, compilerHash, ctxHash, d); diff --git a/src/build/execute.cppm b/src/build/execute.cppm index 6926e37d..2a6c4d78 100644 --- a/src/build/execute.cppm +++ b/src/build/execute.cppm @@ -359,6 +359,16 @@ export int run_build_plan(BuildContext& ctx, bool verbose, bool no_cache, bool sources_newer_than(const std::filesystem::path& projectRoot, std::filesystem::file_time_type ninjaTime) { std::error_code ec; + // The root build.mcpp is a build input too — its directives shape + // build.ninja (flags, generated/selected sources). A changed program must + // abandon the fast path and fall through to prepare_build, where the + // declared-input cache decides whether it actually re-runs. Without this + // the documented "re-runs when the build.mcpp source itself changes" was + // unreachable behind a fresh build.ninja. + if (auto bp = projectRoot / "build.mcpp"; std::filesystem::exists(bp, ec)) { + auto bt = std::filesystem::last_write_time(bp, ec); + if (ec || bt > ninjaTime) return true; + } for (auto& f : mcpp::modgraph::expand_glob(projectRoot, "src/**/*")) { auto ext = f.extension().string(); if (ext != ".cppm" && ext != ".cpp" && ext != ".cc" && diff --git a/tests/e2e/143_build_mcpp_source_directive.sh b/tests/e2e/143_build_mcpp_source_directive.sh new file mode 100755 index 00000000..55e1b974 --- /dev/null +++ b/tests/e2e/143_build_mcpp_source_directive.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# requires: gcc +# P1 (large-source-pkg design §3.1): `mcpp:source=` selects a PRE-EXISTING +# source file into the compile set — the proper name for the old `generated=` +# absolute-path grey usage. The file lives in the package tree, is NOT matched +# by any [build]/[modules] sources glob, and the program did not write it. +# Also covers: the typed `mcpp::source(...)` emitter, and the `d source` cache +# record round-trip (a rebuild that replays the cache must still compile it). +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +cd "$TMP" +mkdir -p srcsel/src srcsel/extra +cd srcsel + +# Pre-existing payload file OUTSIDE the sources globs (extra/, not src/). +cat > extra/impl.cpp <<'EOF' +extern "C" int selected_value() { return 42; } +EOF + +# build.mcpp selects it via the typed API (relative to the package root). +cat > build.mcpp <<'EOF' +import mcpp; +int main() { + mcpp::source("extra/impl.cpp"); + return 0; +} +EOF + +cat > src/main.cpp <<'EOF' +import std; +extern "C" int selected_value(); +int main() { + std::println("SELECTED={}", selected_value()); + return 0; +} +EOF + +cat > mcpp.toml <<'EOF' +[package] +name = "srcsel" +version = "0.1.0" + +[modules] +sources = ["src/**/*.cpp"] + +[targets.srcsel] +kind = "bin" +main = "src/main.cpp" +EOF + +"$MCPP" build > build1.log 2>&1 || { cat build1.log; echo "FAIL: build failed"; exit 1; } +grep -q "ignoring unknown directive" build1.log && { + cat build1.log; echo "FAIL: mcpp:source= not recognized"; exit 1; } || true + +out="$("$MCPP" run 2>&1 | grep '^SELECTED=' | tail -1)" +[[ "$out" == "SELECTED=42" ]] || { echo "FAIL: selected source not linked: $out"; exit 1; } + +# Cache round-trip: touch main.cpp (invalidates the ninja build, not the +# build.mcpp inputs) → the cached `d source` record must reapply so the +# selected file stays in the compile set without a re-run. +cat > src/main.cpp <<'EOF' +import std; +extern "C" int selected_value(); +int main() { + std::println("SELECTED2={}", selected_value()); + return 0; +} +EOF +"$MCPP" build > build2.log 2>&1 || { cat build2.log; echo "FAIL: rebuild failed"; exit 1; } +grep -q "build.mcpp running" build2.log && { + cat build2.log; echo "FAIL: unchanged build.mcpp re-ran"; exit 1; } || true +out="$("$MCPP" run 2>&1 | grep '^SELECTED2=' | tail -1)" +[[ "$out" == "SELECTED2=42" ]] || { echo "FAIL: cached source record lost: $out"; exit 1; } + +# A source= pointing at a missing file is a hard, actionable error. +cat > build.mcpp <<'EOF' +import mcpp; +int main() { + mcpp::source("extra/nope.cpp"); + return 0; +} +EOF +"$MCPP" build > build3.log 2>&1 && { cat build3.log; echo "FAIL: missing source did not fail"; exit 1; } +grep -q "selected source" build3.log || { + cat build3.log; echo "FAIL: missing-source error not surfaced"; exit 1; } + +echo "OK" From ed017cbb99176c965ebfc80746230bcb0eaec24d Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:36:04 +0800 Subject: [PATCH 07/12] feat(build): mcpp:include-dir= / mcpp:include-dir-after= directives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two new directives (design §3.1 item 2) replacing the unnormalized `cxxflag=-I` + `cflag=-I` double-emission hack (opencv descriptor:412-413): - Paths resolve like link-search: absolute as-is, relative against the package root, lexically normalized at parse time (abs_against_root). - include-dir → buildConfig.includeDirs; include-dir-after → buildConfig.includeDirsAfter. The includeDirsAfter FIELD is added to manifest/types.cppm here (line coordinated with the parallel #249 worktree); the -idirafter NINJA EMISSION chain is deliberately NOT implemented — that worktree owns ninja_backend/plan. This commit lands values into buildConfig only; the root-path end-to-end -idirafter arrives when the branches merge. - Cache: `d include-dir` / `d include-dir-after` record kinds round-trip. - Typed lib: mcpp::include_dir / mcpp::include_dir_after. Scope decision (dep path — the invariant is: directive include dirs NEVER leak to consumers): a dep's own TUs read privateBuild, consumers read publicUsage (re-flowed by the usage fixpoint right after the dep loop). The dep loop therefore mirrors the new includeDirs tail into pkg.privateBuild.includeDirs ONLY (publicUsage untouched → no leak; the post-snapshot bcDep entries are inert since descriptor include_dirs propagation snapshotted publicUsage at makePackageRoot). includeDirsAfter has no privateBuild slot (emission chain is #249's), so for deps it is spelled as private "-idirafter " flag pairs into privateBuild cflags+cxxflags — deps get the semantics today, same two-token spelling as host_base_flags. For the ROOT the dirs join buildConfig before the packages[0] snapshot, where expandIncludeDirs handles absolute entries; root has no consumers, so privacy holds trivially. Also folded here (nearest theme — unsupported per-feature include_dirs was the motivating case): xpkg feature UNKNOWN SUB-KEY is now recorded as `features..` in xpkgUnknownKeys instead of being silently swallowed; the existing adoption-site warn_unknown_xpkg_keys (prepare.cppm) surfaces it as a warning. Unit: XpkgUnknownKeys.FeatureSubKeyRecorded. e2e 144_build_mcpp_include_dir.sh: header written under MCPP_OUT_DIR + absolute include-dir → #include resolves with no manual -I; include-dir-after accepted (no unknown-directive warning); both replay from the directive cache. Verified: unit 35/35 green; e2e 144/143/110/111/125 green. --- docs/07-build-mcpp.md | 9 +++ docs/zh/07-build-mcpp.md | 7 +++ src/build/build_program.cppm | 28 ++++++++- src/build/prepare.cppm | 27 +++++++- src/manifest/xpkg.cppm | 10 ++- tests/e2e/144_build_mcpp_include_dir.sh | 84 +++++++++++++++++++++++++ tests/unit/test_manifest.cpp | 30 +++++++++ 7 files changed, 192 insertions(+), 3 deletions(-) create mode 100755 tests/e2e/144_build_mcpp_include_dir.sh diff --git a/docs/07-build-mcpp.md b/docs/07-build-mcpp.md index 39529850..7cb475a6 100644 --- a/docs/07-build-mcpp.md +++ b/docs/07-build-mcpp.md @@ -50,6 +50,8 @@ is ignored, so you can freely log diagnostics. | `mcpp:cfg=` | define `-D` for both C and C++ | | `mcpp:generated=` | add a generated source (relative to the project root) to the build | | `mcpp:source=` | select a **pre-existing** source file into the build (absolute, or relative to the package root). Same downstream effect as `generated=`; use it for files the program *chose* (payload/vendored tree) rather than wrote — e.g. a per-target source selection over a large tarball | +| `mcpp:include-dir=` | add a **private** include directory (`-I`) for this package's own TUs (absolute, or relative to the package root; normalized). Replaces the `cxxflag=-I` + `cflag=-I` double emission | +| `mcpp:include-dir-after=` | like `include-dir`, but searched **after** the system directories (`-idirafter`) — for payload trees that shadow system headers | | `mcpp:rerun-if-changed=` | re-run `build.mcpp` when this file changes | | `mcpp:rerun-if-env-changed=` | re-run `build.mcpp` when this env var changes | @@ -58,6 +60,12 @@ registry dependency — keep your dependency graph declarative in `mcpp.toml` (including platform-conditional `[target.windows.dependencies]`). `build.mcpp` is for *leaf* decisions: flags, codegen, link requirements. +`include-dir`/`include-dir-after` are deliberately **private** (Cargo +discipline): they color only this package's own TUs and are never propagated +to consumers. An include directory consumers must see is part of the public +interface and belongs in the declarative manifest/descriptor +(`[build] include_dirs`), not in a build-time program. + ## Typed API: `import mcpp;` (recommended) Instead of printing raw strings you can write `build.mcpp` **modules-first** — @@ -87,6 +95,7 @@ int main() { | `mcpp::define(s)` | `mcpp:cfg=` (i.e. `-D`) | | `mcpp::generated(p)` | `mcpp:generated=` | | `mcpp::source(p)` | `mcpp:source=` | +| `mcpp::include_dir(d)` / `mcpp::include_dir_after(d)` | `mcpp:include-dir=` / `mcpp:include-dir-after=` | | `mcpp::rerun_if_changed(p)` / `mcpp::rerun_if_env_changed(v)` | the matching `rerun-*` directives | If your `build.mcpp` also needs to *write* a generated file, mix in a textual diff --git a/docs/zh/07-build-mcpp.md b/docs/zh/07-build-mcpp.md index fa61340c..fffbadc5 100644 --- a/docs/zh/07-build-mcpp.md +++ b/docs/zh/07-build-mcpp.md @@ -47,6 +47,8 @@ mcpp build # 编译 + 运行 build.mcpp,然后构建工程 | `mcpp:cfg=` | 为 C 与 C++ 同时定义 `-D` | | `mcpp:generated=` | 把生成的源码(相对工程根目录)加入构建 | | `mcpp:source=` | 把一份**既有**源文件选入构建(绝对路径,或相对包根)。下游效果与 `generated=` 相同;语义区别在于文件是程序*选中*的(tarball payload / vendored 源树)而非程序写出的——例如对大型源码包做 per-target 源选择 | +| `mcpp:include-dir=` | 为本包自身 TU 增加一个**私有** include 目录(`-I`;绝对路径或相对包根,自动规范化)。取代过去 `cxxflag=-I` + `cflag=-I` 的双重裸发 | +| `mcpp:include-dir-after=` | 同 `include-dir`,但排在系统目录**之后**搜索(`-idirafter`)——用于会遮蔽系统头的 payload 源树 | | `mcpp:rerun-if-changed=` | 该文件变化时重跑 `build.mcpp` | | `mcpp:rerun-if-env-changed=` | 该环境变量变化时重跑 `build.mcpp` | @@ -54,6 +56,10 @@ mcpp build # 编译 + 运行 build.mcpp,然后构建工程 `mcpp.toml` 里声明式管理(包括平台条件依赖 `[target.windows.dependencies]`)。 `build.mcpp` 用于*叶子*决策:开关、代码生成、链接需求。 +`include-dir`/`include-dir-after` 刻意保持**私有**(Cargo 纪律):只染色本包自身的 +TU,绝不向消费者传播。需要消费者可见的 include 目录属于公共接口,应写在声明式 +manifest/描述符里(`[build] include_dirs`),而不是构建期程序里。 + ## 类型化 API:`import mcpp;`(推荐) 除了打印裸字符串,你还可以把 `build.mcpp` 写成**模块优先**——`import mcpp;`,无 @@ -82,6 +88,7 @@ int main() { | `mcpp::define(s)` | `mcpp:cfg=`(即 `-D`) | | `mcpp::generated(p)` | `mcpp:generated=` | | `mcpp::source(p)` | `mcpp:source=` | +| `mcpp::include_dir(d)` / `mcpp::include_dir_after(d)` | `mcpp:include-dir=` / `mcpp:include-dir-after=` | | `mcpp::rerun_if_changed(p)` / `mcpp::rerun_if_env_changed(v)` | 对应的 `rerun-*` 指令 | 如果 `build.mcpp` 还需要*写*生成文件,混入一个文本 `#include ` 即可——这没问题, diff --git a/src/build/build_program.cppm b/src/build/build_program.cppm index dc489fd6..e7cc81d4 100644 --- a/src/build/build_program.cppm +++ b/src/build/build_program.cppm @@ -86,6 +86,15 @@ struct Directives { // dependency mode (a payload file lives in the package tree, never in // OUT_DIR) — unlike generated=, whose dep-mode relatives resolve to genBase. std::vector sources; + // include-dir[/-after]= — private include directories for THIS package's + // own TUs (-I / -idirafter). Normalized to absolute at parse time (abs + // taken as-is, relative against the package root, same as link-search). + // Cargo discipline: NEVER propagated as a usage requirement — an include + // dir that consumers must see belongs in the declarative descriptor, not + // in a build-time program (supply-chain surface: a build program must not + // silently widen a package's public interface). + std::vector includeDirs; + std::vector includeDirsAfter; std::vector rerunFiles; // declared file inputs std::vector rerunEnv; // declared env-var inputs }; @@ -124,6 +133,9 @@ bool parse_line(const fs::path& root, std::string_view raw, Directives& d) { else if (key == "cfg") d.defines.push_back("-D" + val); else if (key == "generated") d.generated.push_back(val); else if (key == "source") d.sources.push_back(val); + else if (key == "include-dir") d.includeDirs.push_back(abs_against_root(root, val)); + else if (key == "include-dir-after") + d.includeDirsAfter.push_back(abs_against_root(root, val)); else if (key == "rerun-if-changed") d.rerunFiles.push_back(val); else if (key == "rerun-if-env-changed") d.rerunEnv.push_back(val); else mcpp::ui::warning(std::format("build.mcpp: ignoring unknown directive 'mcpp:{}'", key)); @@ -244,6 +256,8 @@ inline void link_search(const char* dir) { std::printf("mcpp:link-searc inline void define(const char* name) { std::printf("mcpp:cfg=%s\n", name); } inline void generated(const char* path) { std::printf("mcpp:generated=%s\n", path); } inline void source(const char* path) { std::printf("mcpp:source=%s\n", path); } +inline void include_dir(const char* dir) { std::printf("mcpp:include-dir=%s\n", dir); } +inline void include_dir_after(const char* dir) { std::printf("mcpp:include-dir-after=%s\n", dir); } inline void rerun_if_changed(const char* path) { std::printf("mcpp:rerun-if-changed=%s\n", path); } inline void rerun_if_env_changed(const char* var) { std::printf("mcpp:rerun-if-env-changed=%s\n", var); } // ── environment contract (read side; values injected by the engine) ───── @@ -335,7 +349,7 @@ build_mcpp_module(const fs::path& bdir, const fs::path& compiler, // compiler // in // env -// d cxxflag|cflag|ldflag|define|generated|source +// d cxxflag|cflag|ldflag|define|generated|source|include-dir|include-dir-after // The leading program/compiler/in/env lines are the re-run key; the `d` lines // are the directives to reapply on a hit. @@ -433,6 +447,8 @@ void write_cache(const fs::path& bdir, const fs::path& root, emit("define", d.defines); emit("generated", d.generated); emit("source", d.sources); + emit("include-dir", d.includeDirs); + emit("include-dir-after", d.includeDirsAfter); } struct CacheRecord { @@ -474,6 +490,8 @@ CacheRecord read_cache(const fs::path& bdir) { else if (kind == "define") r.directives.defines.push_back(val); else if (kind == "generated") r.directives.generated.push_back(val); else if (kind == "source") r.directives.sources.push_back(val); + else if (kind == "include-dir") r.directives.includeDirs.push_back(val); + else if (kind == "include-dir-after") r.directives.includeDirsAfter.push_back(val); } } r.loaded = true; @@ -521,6 +539,14 @@ void apply(mcpp::manifest::Manifest& m, const Directives& d) { bc.sources.push_back(s); m.modules.sources.push_back(s); } + // Include dirs (already absolute from parse_line). PRIVATE by design: for + // the root they join buildConfig before the package snapshot; for a + // dependency the caller (prepare.cppm dep loop) mirrors them into the + // dep's privateBuild only — never into publicUsage (see Directives note). + for (auto const& p : d.includeDirs) + bc.includeDirs.emplace_back(p); + for (auto const& p : d.includeDirsAfter) + bc.includeDirsAfter.push_back(p); } } // namespace diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index c11e4c7f..847a7c7a 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -3065,7 +3065,9 @@ prepare_build(bool print_fingerprint, } auto& bcDep = pkg.manifest.buildConfig; const auto cN = bcDep.cflags.size(), cxN = bcDep.cxxflags.size(), - ldN = bcDep.ldflags.size(); + ldN = bcDep.ldflags.size(), + incN = bcDep.includeDirs.size(), + incAfterN = bcDep.includeDirsAfter.size(); if (auto r = mcpp::build::run_build_program( pkg.manifest, pkg.root, host->first, host->second, pkg.manifest.cppStandard.canonical, bpEnv); @@ -3085,6 +3087,29 @@ prepare_build(bool print_fingerprint, bcDep.cxxflags.begin() + cxN, bcDep.cxxflags.end()); m->buildConfig.ldflags.insert(m->buildConfig.ldflags.end(), bcDep.ldflags.begin() + ldN, bcDep.ldflags.end()); + // include-dir[/-after] directives are PRIVATE (design §3.1: Cargo + // discipline — a build-time program must not widen the package's + // public interface). The dep's own TUs read privateBuild; its + // consumers read publicUsage (re-flowed by the usage fixpoint + // below) — so mirror the new tail into privateBuild ONLY, never + // publicUsage. The bcDep entries themselves are inert here: the + // descriptor include_dirs propagation snapshotted publicUsage at + // makePackageRoot, long before this pass. + for (auto it = bcDep.includeDirs.begin() + incN; + it != bcDep.includeDirs.end(); ++it) + appendUniquePath(pkg.privateBuild.includeDirs, *it); + // No privateBuild slot exists for after-dirs (the -idirafter + // emission chain is #249's); spell them as private compile flags + // (two argv tokens, matching host_base_flags) so a dep gets the + // semantics today. + for (auto it = bcDep.includeDirsAfter.begin() + incAfterN; + it != bcDep.includeDirsAfter.end(); ++it) { + for (auto* dst : { &pkg.privateBuild.cflags, + &pkg.privateBuild.cxxflags }) { + dst->push_back("-idirafter"); + dst->push_back(*it); + } + } } // apply() may have added interface defines to packages' publicUsage diff --git a/src/manifest/xpkg.cppm b/src/manifest/xpkg.cppm index af7cea00..4081878d 100644 --- a/src/manifest/xpkg.cppm +++ b/src/manifest/xpkg.cppm @@ -1211,7 +1211,15 @@ synthesize_from_xpkg_lua(std::string_view luaContent, } cur.consume('}'); } else { - // unknown subfield — skip its value + // Unknown subfield — skip its value, but RECORD it so + // the adoption-site diagnostic (warn_unknown_xpkg_keys, + // prepare.cppm) can name it: a descriptor author writing + // an unsupported per-feature key (e.g. + // `features.X.include_dirs`) used to be silently + // swallowed, leaving the feature half-configured with + // no signal. + m.xpkgUnknownKeys.push_back( + std::format("features.{}.{}", fname, sub)); if (cur.peek() == '{') cur.skip_table(); else (void)cur.read_bareword(); } diff --git a/tests/e2e/144_build_mcpp_include_dir.sh b/tests/e2e/144_build_mcpp_include_dir.sh new file mode 100755 index 00000000..c80753b0 --- /dev/null +++ b/tests/e2e/144_build_mcpp_include_dir.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# requires: gcc +# P1 (large-source-pkg design §3.1): `mcpp:include-dir=` adds a private include +# directory for the package's own TUs — replacing the old double-emission hack +# (`mcpp:cxxflag=-I…` + `mcpp:cflag=-I…`, unnormalized). Scenario: build.mcpp +# writes a header under MCPP_OUT_DIR and points include-dir at it (absolute); +# a source #includes it; the build succeeds with NO -I flag emitted manually. +# Also: `mcpp:include-dir-after=` is accepted (no unknown-directive warning) +# and both round-trip the directive cache. Its -idirafter ninja emission for +# the root lands with #249; this test pins the directive+cache surface. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT + +cd "$TMP" +mkdir -p incdir/src +cd incdir + +cat > build.mcpp <<'EOF' +#include +#include +#include +#include +#include +import mcpp; +int main() { + std::string inc = std::string(mcpp::out_dir()) + "/inc"; + std::filesystem::create_directories(inc); + std::ofstream(inc + "/bp_config.h") << "#define BP_ANSWER 42\n"; + mcpp::include_dir(inc.c_str()); // absolute + mcpp::include_dir_after("vendor/sysinc"); // relative -> package root; accepted + return 0; +} +EOF +mkdir -p vendor/sysinc + +cat > src/main.cpp <<'EOF' +import std; +#include +int main() { + std::println("ANSWER={}", BP_ANSWER); + return 0; +} +EOF + +cat > mcpp.toml <<'EOF' +[package] +name = "incdir" +version = "0.1.0" + +[modules] +sources = ["src/**/*.cpp"] + +[targets.incdir] +kind = "bin" +main = "src/main.cpp" +EOF + +"$MCPP" build > build1.log 2>&1 || { cat build1.log; echo "FAIL: build failed"; exit 1; } +grep -q "ignoring unknown directive" build1.log && { + cat build1.log; echo "FAIL: include-dir directive not recognized"; exit 1; } || true + +out="$("$MCPP" run 2>&1 | grep '^ANSWER=' | tail -1)" +[[ "$out" == "ANSWER=42" ]] || { echo "FAIL: include dir not on the -I chain: $out"; exit 1; } + +# Cache round-trip: touch a source (defeats the whole-build fast path, keeps +# build.mcpp inputs unchanged) → the cached include-dir record must reapply +# without a re-run, and the header must still resolve. +cat > src/main.cpp <<'EOF' +import std; +#include +int main() { + std::println("ANSWER2={}", BP_ANSWER); + return 0; +} +EOF +"$MCPP" build > build2.log 2>&1 || { cat build2.log; echo "FAIL: rebuild failed"; exit 1; } +grep -q "build.mcpp running" build2.log && { + cat build2.log; echo "FAIL: unchanged build.mcpp re-ran"; exit 1; } || true +out="$("$MCPP" run 2>&1 | grep '^ANSWER2=' | tail -1)" +[[ "$out" == "ANSWER2=42" ]] || { echo "FAIL: cached include-dir record lost: $out"; exit 1; } + +echo "OK" diff --git a/tests/unit/test_manifest.cpp b/tests/unit/test_manifest.cpp index 64ec996b..ce2a1136 100644 --- a/tests/unit/test_manifest.cpp +++ b/tests/unit/test_manifest.cpp @@ -654,6 +654,36 @@ package = { EXPECT_EQ(m->xpkgUnknownKeys[0], "dependencies"); } +// A feature's UNKNOWN sub-key (e.g. an unsupported `include_dirs`) is recorded +// as `features..` instead of being silently swallowed — the +// adoption-site warning (warn_unknown_xpkg_keys) then names it. Known sub-keys +// keep parsing around it. +TEST(XpkgUnknownKeys, FeatureSubKeyRecorded) { + constexpr auto lua = R"( +package = { + name = "x", + xpm = { linux = { ["1.0.0"] = { url = "u", sha256 = "h" } } }, + mcpp = { + sources = { "*/a.c" }, + targets = { ["x"] = { kind = "lib" } }, + features = { + ["opt"] = { + include_dirs = { "inc" }, + defines = { "OPT_ON" }, + }, + }, + }, +} +)"; + auto m = mcpp::manifest::synthesize_from_xpkg_lua(lua, "x", "1.0.0"); + ASSERT_TRUE(m.has_value()) << m.error().format(); + ASSERT_EQ(m->xpkgUnknownKeys.size(), 1u); + EXPECT_EQ(m->xpkgUnknownKeys[0], "features.opt.include_dirs"); + // the known sub-key after the unknown one still parsed. + ASSERT_TRUE(m->buildConfig.featureDefines.contains("opt")); + EXPECT_EQ(m->buildConfig.featureDefines.at("opt").size(), 1u); +} + TEST(Manifest, BuildMacosDeploymentTarget) { constexpr auto src = R"( [package] From f779150324e26a545d643ecd94b7033d9c976ed8 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:38:05 +0800 Subject: [PATCH 08/12] feat(build): MCPP_TARGET_OS/ARCH/ENV contract env splits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cargo CARGO_CFG_TARGET_* parity (design §3.1 item 3): contract_env now parses the resolved target ONCE through the canonical triple parser (mcpp.toolchain.triple — already imported for MCPP_HOST) and injects MCPP_TARGET_OS linux|macos|windows MCPP_TARGET_ARCH GNU spelling (x86_64, aarch64, ...) MCPP_TARGET_ENV gnu|musl|msvc, "" when the triple has no env segment so every build.mcpp (ffmpeg/opencv per-OS selection) stops hand-splitting MCPP_TARGET. All three are always SET; an escape-hatch triple outside the canonical vocabulary yields "" for all three (parse failure is not an error here — MCPP_TARGET still carries the verbatim spelling). Re-run key: verified — contract_hash hashes the WHOLE (name,value) env vector, and the new vars ride that same vector, so they fold into ctxHash with no extra plumbing (a target change already re-ran the program via MCPP_TARGET; the splits add no new invalidation axis). Typed lib: mcpp::target_os()/target_arch()/target_env() readers. e2e 110 extended: asserts the three vars are set, OS/ARCH agree with the full MCPP_TARGET triple's leading segments, and ENV is set (content may be empty on macOS). Docs env tables updated (en/zh). Verified: unit 35/35; e2e 110/143/144/111/125 green. --- docs/07-build-mcpp.md | 3 +++ docs/zh/07-build-mcpp.md | 3 +++ src/build/build_program.cppm | 17 +++++++++++++++++ tests/e2e/110_build_mcpp_env_contract.sh | 21 +++++++++++++++++++-- 4 files changed, 42 insertions(+), 2 deletions(-) diff --git a/docs/07-build-mcpp.md b/docs/07-build-mcpp.md index 7cb475a6..3d34f8ef 100644 --- a/docs/07-build-mcpp.md +++ b/docs/07-build-mcpp.md @@ -111,6 +111,9 @@ The running program receives the build context as `MCPP_*` variables | Variable | Typed reader | Value | |---|---|---| | `MCPP_TARGET` | `mcpp::target()` | resolved canonical triple (the `--target` triple under cross; the host triple natively) | +| `MCPP_TARGET_OS` | `mcpp::target_os()` | the target's OS segment (`linux`/`macos`/`windows`) — no need to hand-split `MCPP_TARGET` | +| `MCPP_TARGET_ARCH` | `mcpp::target_arch()` | the target's arch segment (GNU spelling: `x86_64`, `aarch64`, …) | +| `MCPP_TARGET_ENV` | `mcpp::target_env()` | the target's env segment (`gnu`/`musl`/`msvc`); empty string when the triple has none (macOS) | | `MCPP_HOST` | `mcpp::host()` | the host triple | | `MCPP_PROFILE` | `mcpp::profile()` | effective profile name (`dev`/`release`/…) | | `MCPP_OUT_DIR` | `mcpp::out_dir()` | a writable scratch/output dir owned by mcpp | diff --git a/docs/zh/07-build-mcpp.md b/docs/zh/07-build-mcpp.md index fffbadc5..4f29e5b1 100644 --- a/docs/zh/07-build-mcpp.md +++ b/docs/zh/07-build-mcpp.md @@ -103,6 +103,9 @@ int main() { | 变量 | 类型化读取 | 值 | |---|---|---| | `MCPP_TARGET` | `mcpp::target()` | 解析后的 canonical 三元组(交叉构建下是 `--target` 三元组,原生构建是宿主) | +| `MCPP_TARGET_OS` | `mcpp::target_os()` | 目标的 OS 段(`linux`/`macos`/`windows`)——不必再手撕 `MCPP_TARGET` | +| `MCPP_TARGET_ARCH` | `mcpp::target_arch()` | 目标的 arch 段(GNU 拼写:`x86_64`、`aarch64`…) | +| `MCPP_TARGET_ENV` | `mcpp::target_env()` | 目标的 env 段(`gnu`/`musl`/`msvc`);三元组无 env 段(macOS)时为空串 | | `MCPP_HOST` | `mcpp::host()` | 宿主三元组 | | `MCPP_PROFILE` | `mcpp::profile()` | 生效 profile 名(`dev`/`release`/…) | | `MCPP_OUT_DIR` | `mcpp::out_dir()` | mcpp 提供的可写输出/暂存目录 | diff --git a/src/build/build_program.cppm b/src/build/build_program.cppm index e7cc81d4..970cf3c4 100644 --- a/src/build/build_program.cppm +++ b/src/build/build_program.cppm @@ -263,6 +263,9 @@ inline void rerun_if_env_changed(const char* var) { std::printf("mcpp:rerun-if-e // ── environment contract (read side; values injected by the engine) ───── inline const char* env_or(const char* n) { const char* v = std::getenv(n); return v ? v : ""; } inline const char* target() { return env_or("MCPP_TARGET"); } +inline const char* target_os() { return env_or("MCPP_TARGET_OS"); } +inline const char* target_arch() { return env_or("MCPP_TARGET_ARCH"); } +inline const char* target_env() { return env_or("MCPP_TARGET_ENV"); } inline const char* host() { return env_or("MCPP_HOST"); } inline const char* profile() { return env_or("MCPP_PROFILE"); } inline const char* out_dir() { return env_or("MCPP_OUT_DIR"); } @@ -383,6 +386,20 @@ contract_env(const fs::path& root, const fs::path& outDir, const BuildProgramEnv std::vector> e; auto hostT = mcpp::toolchain::triple::host_triple().str(); e.emplace_back("MCPP_TARGET", env.targetTriple.empty() ? hostT : env.targetTriple); + // Convenience splits of the resolved target (Cargo CARGO_CFG_TARGET_* + // parity): parsed ONCE here through the canonical triple parser so every + // build.mcpp stops hand-splitting MCPP_TARGET. MCPP_TARGET_ENV is "" when + // the triple has no env segment (macOS); all three are "" for an + // escape-hatch triple outside the canonical vocabulary. They ride the + // same env vector, so contract_hash folds them into the re-run key. + { + mcpp::toolchain::triple::Triple t{}; + if (env.targetTriple.empty()) t = mcpp::toolchain::triple::host_triple(); + else if (auto p = mcpp::toolchain::triple::parse(env.targetTriple)) t = *p; + e.emplace_back("MCPP_TARGET_OS", t.os); + e.emplace_back("MCPP_TARGET_ARCH", t.arch); + e.emplace_back("MCPP_TARGET_ENV", t.env); + } e.emplace_back("MCPP_HOST", hostT); e.emplace_back("MCPP_PROFILE", env.profile); e.emplace_back("MCPP_OUT_DIR", outDir.string()); diff --git a/tests/e2e/110_build_mcpp_env_contract.sh b/tests/e2e/110_build_mcpp_env_contract.sh index b1faae9f..f735fc6f 100755 --- a/tests/e2e/110_build_mcpp_env_contract.sh +++ b/tests/e2e/110_build_mcpp_env_contract.sh @@ -24,6 +24,11 @@ int main() { f << "extern \"C\" const char* bp_target() { return \"" << env_or("MCPP_TARGET") << "\"; }\n"; f << "extern \"C\" const char* bp_host() { return \"" << env_or("MCPP_HOST") << "\"; }\n"; f << "extern \"C\" const char* bp_profile() { return \"" << env_or("MCPP_PROFILE") << "\"; }\n"; + f << "extern \"C\" const char* bp_tos() { return \"" << env_or("MCPP_TARGET_OS") << "\"; }\n"; + f << "extern \"C\" const char* bp_tarch() { return \"" << env_or("MCPP_TARGET_ARCH") << "\"; }\n"; + // ENV may legitimately be empty (macOS) — assert set-ness, not content. + f << "extern \"C\" int bp_tenv_set() { return " + << (std::getenv("MCPP_TARGET_ENV") ? 1 : 0) << "; }\n"; f << "extern \"C\" const char* bp_features() { return \"" << env_or("MCPP_FEATURES") << "\"; }\n"; f << "extern \"C\" int bp_has_extra() { return " << (std::getenv("MCPP_FEATURE_EXTRA") ? 1 : 0) << "; }\n"; @@ -45,9 +50,13 @@ extern "C" const char* bp_host(); extern "C" const char* bp_profile(); extern "C" const char* bp_features(); extern "C" int bp_has_extra(); +extern "C" const char* bp_tos(); +extern "C" const char* bp_tarch(); +extern "C" int bp_tenv_set(); int main() { - std::println("target={} host={} profile={} features=[{}] extra={}", - bp_target(), bp_host(), bp_profile(), bp_features(), bp_has_extra()); + std::println("target={} host={} profile={} features=[{}] extra={} tos={} tarch={} tenvset={}", + bp_target(), bp_host(), bp_profile(), bp_features(), bp_has_extra(), + bp_tos(), bp_tarch(), bp_tenv_set()); return 0; } EOF @@ -67,6 +76,14 @@ out="$("$MCPP" run 2>&1 | tail -1)" host_triple_re='[a-z0-9_]+-(linux|macos|windows)(-[a-z]+)?' [[ "$out" =~ target=$host_triple_re\ host=$host_triple_re\ profile=dev\ features=\[\]\ extra=0 ]] || { echo "unexpected contract values: $out"; exit 1; } +# MCPP_TARGET_OS/ARCH/ENV splits: OS is one of the canonical spellings, ARCH is +# non-empty, ENV is always SET (may be "" — no env segment on macOS) and must +# agree with the full triple's segments. +[[ "$out" =~ tos=(linux|macos|windows)\ tarch=([a-z0-9_]+)\ tenvset=1 ]] || { + echo "target split vars missing/odd: $out"; exit 1; } +tos="${BASH_REMATCH[1]}"; tarch="${BASH_REMATCH[2]}" +[[ "$out" == *"target=${tarch}-${tos}"* ]] || { + echo "target split disagrees with MCPP_TARGET: $out"; exit 1; } # Identical build → no re-run (either the whole-build fast path short-circuits # before build.mcpp, or the build.mcpp cache hits — both are fine; a "running" From 4bb71666801c59d471fe793960c4afc7599f4792 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:42:00 +0800 Subject: [PATCH 09/12] feat(build): root build.mcpp runs after dep resolution and receives MCPP_DEP_*_DIR MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design §3.1 item 4 (+ #230-#243 ledger follow-up): the ROOT project's build.mcpp used to run BEFORE dependency resolution (prepare.cppm ~1180), so unlike a dependency's it never saw MCPP_DEP__DIR — consumer-side synthesis (ffmpeg-m reading compat.ffmpeg's payload tree) was impossible. The call now sits right after the dep build.mcpp loop / usage fixpoint / capability binding, BEFORE the [targets.*] gate and the modgraph scan, and populates env.depDirs from the SAME authoritative dependencyEdges graph as the dep path (consumer index 0 = root; canonical + namespace-stripped spellings; same collision guard in contract_env). Ordering invariants, each verified at the new call point: (1) generated=/source= registration precedes the modgraph scan — the scan walks packages[0].manifest, so the new-source TAILS are mirrored into packages[0].manifest.{buildConfig.sources,modules.sources} (see wrinkle below); the scan call sits ~200 lines further down. (2) directive flags precede canonicalization/fingerprint — fpi.compileFlags = canonical_compile_flags(*m) + canonical_package_build_metadata(packages) is computed AFTER the scan, i.e. after this call; flag tails are mirrored into packages[0].privateBuild (per-TU assembly) AND packages[0].manifest.buildConfig (fingerprint metadata parity). (3) materialize_generated_files (root) still runs earlier — untouched at its pre-move position, so [generated_files] may still produce build.mcpp. (4) L1 cfg-conditional merge ordering unchanged — it still runs before makePackageRoot; ONLY the build.mcpp call moved later (e2e 108 green). (5) root features env — recomputed with the byte-identical expression (feature_closure(*m, parse_feature_request(overrides.features))) so the contract hash / build.mcpp cache is stable across the move. The wrinkle the old ordering hid: apply() used to mutate *m before `packages[0] = makePackageRoot(*root, *m)` snapshotted buildConfig into privateBuild/manifest — the copies the scan and per-TU flag assembly read. Post-move the snapshot (and root feature activation on it) already happened, so the directive tails are mirrored explicitly, dep-loop style: sources → manifest.buildConfig.sources + manifest.modules.sources; c/cxx flags → privateBuild + manifest.buildConfig; ldflags → linkUsage + manifest.buildConfig (final link reads *m, already applied); include-dir → privateBuild.includeDirs ONLY (private discipline — under the old order they also leaked into root publicUsage via the snapshot, harmless with no consumers but now deliberately private); include-dir-after → manifest.buildConfig.includeDirsAfter. Known benign delta: directive flags now append AFTER feature/usage-propagated flags in privateBuild instead of before them (order within one flags vector). e2e 145_root_build_mcpp_dep_dirs.sh (modeled on 125): root with a path dep; the ROOT's build.mcpp hard-fails unless dep_dir("datad") is set, generates a source returning it; asserts the dir exists and is datad's root. Docs: MCPP_DEP__DIR documented in the env table (en/zh) — previously undocumented — with the root now receiving it. Verified: unit 35/35; e2e 145/110/111/112/125/143/144 green + regression batch 106/107/108/109/114/100/04/09/15/126/128 green. --- docs/07-build-mcpp.md | 1 + docs/zh/07-build-mcpp.md | 1 + src/build/prepare.cppm | 140 +++++++++++++++++----- tests/e2e/145_root_build_mcpp_dep_dirs.sh | 80 +++++++++++++ 4 files changed, 191 insertions(+), 31 deletions(-) create mode 100755 tests/e2e/145_root_build_mcpp_dep_dirs.sh diff --git a/docs/07-build-mcpp.md b/docs/07-build-mcpp.md index 3d34f8ef..f714eebc 100644 --- a/docs/07-build-mcpp.md +++ b/docs/07-build-mcpp.md @@ -120,6 +120,7 @@ The running program receives the build context as `MCPP_*` variables | `MCPP_MANIFEST_DIR` | `mcpp::manifest_dir()` | the package root (= CWD) | | `MCPP_FEATURE_` | `mcpp::has_feature("name")` | set to `1` per active feature (same `` sanitization as the `MCPP_FEATURE_` compile macro) | | `MCPP_FEATURES` | — | comma-separated active feature list | +| `MCPP_DEP__DIR` | `mcpp::dep_dir("name")` | the resolved install dir of each declared dependency (canonical **and** namespace-stripped name spellings; same `` sanitization as `MCPP_FEATURE_`). Received by dependencies' build.mcpp **and** the root project's (the root runs after dependency resolution) | These values are folded into the re-run key **unconditionally** — changing the target, profile, or feature set re-runs the program without any diff --git a/docs/zh/07-build-mcpp.md b/docs/zh/07-build-mcpp.md index 4f29e5b1..f8c650d6 100644 --- a/docs/zh/07-build-mcpp.md +++ b/docs/zh/07-build-mcpp.md @@ -112,6 +112,7 @@ int main() { | `MCPP_MANIFEST_DIR` | `mcpp::manifest_dir()` | 包根(= CWD) | | `MCPP_FEATURE_` | `mcpp::has_feature("name")` | 每个活跃 feature 置 `1`(`` 消毒规则与 `MCPP_FEATURE_` 编译宏一致) | | `MCPP_FEATURES` | — | 活跃 feature 逗号列表 | +| `MCPP_DEP__DIR` | `mcpp::dep_dir("name")` | 每个已声明依赖解析后的安装目录(canonical 名与去命名空间短名两种拼写都可用;`` 消毒规则同 `MCPP_FEATURE_`)。依赖包的 build.mcpp **和**根工程的 build.mcpp 都能拿到(根工程的 build.mcpp 在依赖解析之后运行) | 这些契约值**无条件**折入重跑键——换 target、换 profile、开关 feature 都会触发重跑, 不需要任何 `rerun-if-env-changed` 声明。 diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 847a7c7a..88955b7b 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -1129,20 +1129,24 @@ prepare_build(bool print_fingerprint, // self-describing. See docs: 2026-05-21-linux-sysroot-missing-kernel-headers.md // ── L3: project-local `build.mcpp` imperative build program ───────────── - // Compiled with the HOST toolchain and run now — after target resolution - // + the L1 cfg-flag merge (buildConfig flags are final) and BEFORE the - // modgraph scan (so its `generated=` sources are picked up). Its stdout - // directives augment buildConfig; a declared-input cache re-runs it only - // when its source/inputs/env/contract change. It cannot gate the top-level - // dependency graph (leaf-only rule). Under a cross --target it runs with a - // host-resolved toolchain and sees MCPP_TARGET = the cross triple (G3). - // Dependencies' build.mcpp run in a later pass (G2), after their features - // are known. See .agents/docs/2026-06-30-l3-build-mcpp-implementation-design.md - // and 2026-07-17-asm-sources-and-general-build-capabilities-design.md §2.4. + // The ROOT program is compiled with the HOST toolchain and run AFTER + // dependency resolution + feature activation (so it receives + // MCPP_DEP__DIR like a dependency's does — design §3.1 item 4) and + // BEFORE the modgraph scan (so its `generated=`/`source=` sources are + // picked up) — see the call site further below, after the dep build.mcpp + // loop. Its stdout directives augment buildConfig; a declared-input cache + // re-runs it only when its source/inputs/env/contract change. It cannot + // gate the top-level dependency graph (leaf-only rule). Under a cross + // --target it runs with a host-resolved toolchain and sees MCPP_TARGET = + // the cross triple (G3). + // See .agents/docs/2026-06-30-l3-build-mcpp-implementation-design.md, + // 2026-07-17-asm-sources-and-general-build-capabilities-design.md §2.4 and + // 2026-07-19-large-source-pkg-platform-fixes-and-buildmcpp-generation-design.md. // Root [generated_files]: materialize before build.mcpp and the modgraph - // scan so synthesized sources are globbed like any on-disk file. (The - // per-dependency call sits in the dep resolution loop below; the root - // manifest needs its own.) + // scan so synthesized sources are globbed like any on-disk file — and + // BEFORE dependency resolution, since generated_files may produce + // build.mcpp itself. (The per-dependency call sits in the dep resolution + // loop below; the root manifest needs its own.) if (!m->buildConfig.generatedFiles.empty()) { if (auto r = materialize_generated_files(*root, *m); !r) { return std::unexpected(r.error()); @@ -1204,21 +1208,6 @@ prepare_build(bool print_fingerprint, return *hostTcCache; }; - if (std::filesystem::exists(*root / "build.mcpp")) { - auto host = host_tc_for_build_program(); - if (!host) return std::unexpected(host.error()); - mcpp::build::BuildProgramEnv bpEnv; - bpEnv.targetTriple = resolvedTargetCanonical; - bpEnv.profile = effectiveProfile; - bpEnv.features = feature_closure(*m, parse_feature_request(overrides.features)); - if (auto bp = mcpp::build::run_build_program( - *m, *root, host->first, host->second, - m->cppStandard.canonical, bpEnv); - !bp) { - return std::unexpected(bp.error()); - } - } - // Resolve dependencies: walk the **transitive** graph from the main // manifest, BFS-style. Each unique `(namespace, shortName)` is fetched // once, its `[build].include_dirs` are propagated to the main @@ -3051,9 +3040,8 @@ prepare_build(bool print_fingerprint, // under BOTH its canonical package name AND its namespace-stripped // short name, so `mcpp::dep_dir("compat.zlib")` and // `mcpp::dep_dir("zlib")` both resolve regardless of which spelling - // the author used in `deps`. (The ROOT project's own build.mcpp runs - // before dependency resolution, so it does not yet receive these — - // tracked as a follow-up in the #230-#243 ledger.) + // the author used in `deps`. (The ROOT project's build.mcpp gets + // the same treatment at its own call site right after this loop.) for (auto const& edge : dependencyEdges) { if (edge.consumerPackageIndex != i) continue; auto const& depPkg = packages[edge.dependencyPackageIndex]; @@ -3176,6 +3164,96 @@ prepare_build(bool print_fingerprint, } } + // ── L3: ROOT build.mcpp (moved after dependency resolution, design §3.1 + // item 4) ──────────────────────────────────────────────────────────────── + // Runs HERE — after dep resolution + feature activation (so the contract + // env can expose MCPP_DEP__DIR exactly like the dep loop above does) + // and BEFORE the modgraph scan / flag canonicalization / fingerprint (so + // its generated=/source= sources and flag directives are fully visible). + // Ordering invariants preserved relative to the pre-move call site: + // materialize_generated_files (may produce build.mcpp itself) and the L1 + // cfg merge still run earlier — ONLY this call moved later. + // + // One wrinkle the old ordering hid: back then apply() mutated *m BEFORE + // `packages[0] = makePackageRoot(*root, *m)` snapshotted buildConfig into + // privateBuild/manifest — the copies the scan and per-TU flag assembly + // actually read. Now the snapshot (and root feature activation on it) + // already happened, so mirror the directive TAILS into packages[0] + // explicitly, the same way the dep loop does for its package. + if (std::filesystem::exists(*root / "build.mcpp")) { + auto host = host_tc_for_build_program(); + if (!host) return std::unexpected(host.error()); + mcpp::build::BuildProgramEnv bpEnv; + bpEnv.targetTriple = resolvedTargetCanonical; + bpEnv.profile = effectiveProfile; + // Same expression as the pre-move call site (and same order), so the + // contract hash — and therefore the build.mcpp cache — is unchanged + // across the move for feature-identical builds. + bpEnv.features = feature_closure(*m, parse_feature_request(overrides.features)); + // mcpp#241 (root): the root's resolved direct deps, from the same + // authoritative edge graph as the dep loop (consumer index 0 = root), + // emitted under canonical AND namespace-stripped names. + for (auto const& edge : dependencyEdges) { + if (edge.consumerPackageIndex != 0) continue; + auto const& depPkg = packages[edge.dependencyPackageIndex]; + const auto& canon = depPkg.manifest.package.name; + bpEnv.depDirs.emplace_back(canon, depPkg.root); + if (auto dot = canon.rfind('.'); dot != std::string::npos + && dot + 1 < canon.size()) + bpEnv.depDirs.emplace_back(canon.substr(dot + 1), depPkg.root); + } + auto& bcRoot = m->buildConfig; + const auto rcN = bcRoot.cflags.size(), rcxN = bcRoot.cxxflags.size(), + rldN = bcRoot.ldflags.size(), rsrcN = bcRoot.sources.size(), + rincN = bcRoot.includeDirs.size(), + rincAfterN = bcRoot.includeDirsAfter.size(), + rmodN = m->modules.sources.size(); + if (auto bp = mcpp::build::run_build_program( + *m, *root, host->first, host->second, + m->cppStandard.canonical, bpEnv); + !bp) { + return std::unexpected(bp.error()); + } + auto& pkg0 = packages[0]; + // Sources → the scan walks packages[0].manifest, not *m. + pkg0.manifest.buildConfig.sources.insert( + pkg0.manifest.buildConfig.sources.end(), + bcRoot.sources.begin() + rsrcN, bcRoot.sources.end()); + pkg0.manifest.modules.sources.insert( + pkg0.manifest.modules.sources.end(), + m->modules.sources.begin() + rmodN, m->modules.sources.end()); + // Compile flags → the root's TUs read privateBuild (and the + // fingerprint folds packages[].manifest.buildConfig via + // canonical_package_build_metadata) — mirror both, as the old + // pre-snapshot ordering implicitly did. + pkg0.privateBuild.cflags.insert(pkg0.privateBuild.cflags.end(), + bcRoot.cflags.begin() + rcN, bcRoot.cflags.end()); + pkg0.privateBuild.cxxflags.insert(pkg0.privateBuild.cxxflags.end(), + bcRoot.cxxflags.begin() + rcxN, bcRoot.cxxflags.end()); + pkg0.manifest.buildConfig.cflags.insert( + pkg0.manifest.buildConfig.cflags.end(), + bcRoot.cflags.begin() + rcN, bcRoot.cflags.end()); + pkg0.manifest.buildConfig.cxxflags.insert( + pkg0.manifest.buildConfig.cxxflags.end(), + bcRoot.cxxflags.begin() + rcxN, bcRoot.cxxflags.end()); + // Link flags → the final link reads *m (already applied); keep the + // linkUsage snapshot equivalent too. + pkg0.linkUsage.ldflags.insert(pkg0.linkUsage.ldflags.end(), + bcRoot.ldflags.begin() + rldN, bcRoot.ldflags.end()); + pkg0.manifest.buildConfig.ldflags.insert( + pkg0.manifest.buildConfig.ldflags.end(), + bcRoot.ldflags.begin() + rldN, bcRoot.ldflags.end()); + // include-dir directives: PRIVATE (already absolute from parse_line) — + // privateBuild only, never publicUsage (see the dep loop's rationale). + for (auto it = bcRoot.includeDirs.begin() + rincN; + it != bcRoot.includeDirs.end(); ++it) + appendUniquePath(pkg0.privateBuild.includeDirs, *it); + pkg0.manifest.buildConfig.includeDirsAfter.insert( + pkg0.manifest.buildConfig.includeDirsAfter.end(), + bcRoot.includeDirsAfter.begin() + rincAfterN, + bcRoot.includeDirsAfter.end()); + } + // [targets.*] required_features gate: a target is emitted only when ALL its // required features are active in this build; otherwise it is silently // skipped. A pure build-selection knob — it runs before the modgraph/plan diff --git a/tests/e2e/145_root_build_mcpp_dep_dirs.sh b/tests/e2e/145_root_build_mcpp_dep_dirs.sh new file mode 100755 index 00000000..eae79b07 --- /dev/null +++ b/tests/e2e/145_root_build_mcpp_dep_dirs.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# requires: gcc +# P1 (large-source-pkg design §3.1 item 4): the ROOT project's build.mcpp now +# runs AFTER dependency resolution and receives MCPP_DEP__DIR exactly +# like a dependency's build.mcpp does (mcpp#241 contract, previously dep-only: +# the root ran before resolution and saw none). Scenario mirrors +# 125_build_mcpp_dep_dir.sh, but the build.mcpp asserting dep_dir() is the +# ROOT's own: root -> datad (path dep); root build.mcpp reads +# dep_dir("datad"), fails loudly if unset, and generates a source returning +# it; main prints it; the test asserts it is datad's real root. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +# data-asset dependency +mkdir -p datad/src +echo 'int datad_touch() { return 1; }' > datad/src/d.cpp +cat > datad/mcpp.toml <<'EOF' +[package] +name = "datad" +version = "0.1.0" +[modules] +sources = ["src/**/*.cpp"] +[targets.datad] +kind = "lib" +EOF + +# ROOT project whose own build.mcpp locates datad's dir +mkdir -p rootp/src +cat > rootp/build.mcpp <<'EOF' +#include +#include +import mcpp; +int main() { + const char* d = mcpp::dep_dir("datad"); + if (d == nullptr || d[0] == '\0') { + std::fprintf(stderr, "build.mcpp: MCPP_DEP_DATAD_DIR not set for the ROOT\n"); + return 1; // the moved call point must expose the contract var + } + std::string out = "src/depdir.cpp"; + std::FILE* f = std::fopen(out.c_str(), "w"); + std::fprintf(f, "const char* root_datad_dir() { return \"%s\"; }\n", d); + std::fclose(f); + mcpp::generated("src/depdir.cpp"); + return 0; +} +EOF +cat > rootp/src/main.cpp <<'EOF' +import std; +extern const char* root_datad_dir(); +int main() { + std::println("DEPDIR={}", root_datad_dir()); + return 0; +} +EOF +cat > rootp/mcpp.toml <<'EOF' +[package] +name = "rootp" +version = "0.1.0" +[modules] +sources = ["src/**/*.cpp"] +[dependencies] +datad = { path = "../datad" } +[targets.rootp] +kind = "bin" +main = "src/main.cpp" +EOF + +cd rootp +"$MCPP" build > build.log 2>&1 || { cat build.log; echo "FAIL: build failed"; exit 1; } + +out="$("$MCPP" run 2>&1 | grep '^DEPDIR=' | tail -1)" +dir="${out#DEPDIR=}" +[[ -n "$dir" ]] || { echo "FAIL: empty dep dir (root contract var missing)"; exit 1; } +[[ -d "$dir" ]] || { echo "FAIL: dep dir does not exist: $dir"; exit 1; } +[[ "$(basename "$dir")" == "datad" ]] || { echo "FAIL: dep dir is not datad's root: $dir"; exit 1; } + +echo "OK" From 0e4dd3001fcd7fb7574102156a876ef3f7d5d635 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 20:48:52 +0800 Subject: [PATCH 10/12] refactor(build): include-dir[-after] directives converge on the typed #249 channel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review pass over the merged P0+P1 branch. Two spellings of the same decision survived the parallel tracks: - dep loop: after-dirs were spelled as raw private '-idirafter ' flag pairs (written before #249 landed privateBuild.includeDirsAfter); the raw spelling bypasses the typed channel's per-dialect degradations (cl.exe /I, NASM -I). Now appendUniquePath into the typed slot. - root: after-dirs reached only the fingerprint mirror (manifest.buildConfig), never privateBuild — scanned units read privateBuild.includeDirsAfter, so 'mcpp:include-dir-after=' on the root was inert on compile edges. Both channels now mirrored for both kinds, matching the flag mirrors above. e2e 144 upgraded from 'directive accepted' to asserting -idirafter reaches build.ninja. --- src/build/prepare.cppm | 30 ++++++++++++++----------- tests/e2e/144_build_mcpp_include_dir.sh | 13 ++++++++--- 2 files changed, 27 insertions(+), 16 deletions(-) diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 88955b7b..11b2871a 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -3086,18 +3086,13 @@ prepare_build(bool print_fingerprint, for (auto it = bcDep.includeDirs.begin() + incN; it != bcDep.includeDirs.end(); ++it) appendUniquePath(pkg.privateBuild.includeDirs, *it); - // No privateBuild slot exists for after-dirs (the -idirafter - // emission chain is #249's); spell them as private compile flags - // (two argv tokens, matching host_base_flags) so a dep gets the - // semantics today. + // After-dirs ride the same typed #249 channel + // (privateBuild.includeDirsAfter → per-unit -idirafter), which + // owns the per-dialect degradations (cl.exe /I, NASM -I) a raw + // flag spelling would bypass. for (auto it = bcDep.includeDirsAfter.begin() + incAfterN; - it != bcDep.includeDirsAfter.end(); ++it) { - for (auto* dst : { &pkg.privateBuild.cflags, - &pkg.privateBuild.cxxflags }) { - dst->push_back("-idirafter"); - dst->push_back(*it); - } - } + it != bcDep.includeDirsAfter.end(); ++it) + appendUniquePath(pkg.privateBuild.includeDirsAfter, *it); } // apply() may have added interface defines to packages' publicUsage @@ -3243,11 +3238,20 @@ prepare_build(bool print_fingerprint, pkg0.manifest.buildConfig.ldflags.insert( pkg0.manifest.buildConfig.ldflags.end(), bcRoot.ldflags.begin() + rldN, bcRoot.ldflags.end()); - // include-dir directives: PRIVATE (already absolute from parse_line) — - // privateBuild only, never publicUsage (see the dep loop's rationale). + // include-dir[/-after] directives: PRIVATE (already absolute from + // parse_line) — privateBuild only, never publicUsage (see the dep + // loop's rationale). privateBuild is what scanned units read + // (scanner → localIncludeDirs[After]); the manifest mirror keeps the + // fingerprint metadata equivalent, same as the flag mirrors above. for (auto it = bcRoot.includeDirs.begin() + rincN; it != bcRoot.includeDirs.end(); ++it) appendUniquePath(pkg0.privateBuild.includeDirs, *it); + pkg0.manifest.buildConfig.includeDirs.insert( + pkg0.manifest.buildConfig.includeDirs.end(), + bcRoot.includeDirs.begin() + rincN, bcRoot.includeDirs.end()); + for (auto it = bcRoot.includeDirsAfter.begin() + rincAfterN; + it != bcRoot.includeDirsAfter.end(); ++it) + appendUniquePath(pkg0.privateBuild.includeDirsAfter, *it); pkg0.manifest.buildConfig.includeDirsAfter.insert( pkg0.manifest.buildConfig.includeDirsAfter.end(), bcRoot.includeDirsAfter.begin() + rincAfterN, diff --git a/tests/e2e/144_build_mcpp_include_dir.sh b/tests/e2e/144_build_mcpp_include_dir.sh index c80753b0..f77b0556 100755 --- a/tests/e2e/144_build_mcpp_include_dir.sh +++ b/tests/e2e/144_build_mcpp_include_dir.sh @@ -5,9 +5,9 @@ # (`mcpp:cxxflag=-I…` + `mcpp:cflag=-I…`, unnormalized). Scenario: build.mcpp # writes a header under MCPP_OUT_DIR and points include-dir at it (absolute); # a source #includes it; the build succeeds with NO -I flag emitted manually. -# Also: `mcpp:include-dir-after=` is accepted (no unknown-directive warning) -# and both round-trip the directive cache. Its -idirafter ninja emission for -# the root lands with #249; this test pins the directive+cache surface. +# Also: `mcpp:include-dir-after=` rides the typed #249 channel end-to-end — +# the emitted build.ninja must carry -idirafter for the directive dir — and +# both directives round-trip the directive cache. set -e TMP=$(mktemp -d) @@ -64,6 +64,13 @@ grep -q "ignoring unknown directive" build1.log && { out="$("$MCPP" run 2>&1 | grep '^ANSWER=' | tail -1)" [[ "$out" == "ANSWER=42" ]] || { echo "FAIL: include dir not on the -I chain: $out"; exit 1; } +# include-dir-after reaches the compile edges as -idirafter (typed #249 +# channel: privateBuild.includeDirsAfter → localIncludeDirsAfter). +ninja_file=$(find target -name build.ninja | head -1) +grep -q -- "-idirafter.*vendor/sysinc" "$ninja_file" || { + echo "FAIL: include-dir-after not emitted as -idirafter in build.ninja"; + grep -n "local_includes" "$ninja_file" | head; exit 1; } + # Cache round-trip: touch a source (defeats the whole-build fast path, keeps # build.mcpp inputs unchanged) → the cached include-dir record must reapply # without a re-run, and the header must still resolve. From 25eac8c7657b60938b5dae2be4217c04c6a4d5a5 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 21:11:37 +0800 Subject: [PATCH 11/12] =?UTF-8?q?refactor(build):=20architecture-review=20?= =?UTF-8?q?pass=20=E2=80=94=20rsp=20path=20portability,=20one=20directive-?= =?UTF-8?q?fold=20owner,=20type/rule=20unification?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consolidated fixes from the three-lens pre-PR review (correctness / cross-platform / architecture): - #247 P0: escape_ninja_path emits generic (forward-slash) node names. rspfile_content copies node names verbatim into a response file that gcc/clang/GNU ar tokenize GNU-style, where backslash is an ESCAPE — obj\cli.o would arrive as objcli.o and break every Windows driver-style link. All Windows consumers accept forward slashes; POSIX output is byte-identical. - link/archive/shared rules: ONE link_rule emitter for both dialect branches (useRsp = separateLinker || is_windows); msvc rule text byte-identical to the previous hand-written stanzas. - prepare.cppm: markDirectiveTail/foldDirectiveTailIntoPrivateBuild — a single owner of 'which compile-visible channels a build.mcpp directive lands in', shared by the dep loop and the root call site (kills the #242 two-derivations shape the parallel tracks had reintroduced). Link/source/fingerprint residues stay at the call sites where they genuinely differ. - BuildConfig::includeDirsAfter aligned to filesystem::path (was string), deleting per-consumer conversions. - local_include_flags derives the NASM degradation from cu.source itself instead of a third parameter threaded by every caller. - Windows shell fallback: cd /d (cmd.exe cd does not switch drives). - docs: source= joins the cache-invalidation list; 0.0.100+ tags on the new directives/env rows; NASM -I degradation documented (en+zh). --- docs/05-mcpp-toml.md | 4 +- docs/07-build-mcpp.md | 16 ++-- docs/zh/05-mcpp-toml.md | 4 +- docs/zh/07-build-mcpp.md | 16 ++-- src/build/ninja_backend.cppm | 101 +++++++++++++----------- src/build/plan.cppm | 2 +- src/build/prepare.cppm | 149 ++++++++++++++++++----------------- src/manifest/types.cppm | 2 +- src/modgraph/scanner.cppm | 7 +- src/platform/process.cppm | 10 ++- 10 files changed, 165 insertions(+), 146 deletions(-) diff --git a/docs/05-mcpp-toml.md b/docs/05-mcpp-toml.md index 6ee702d2..566cbc91 100644 --- a/docs/05-mcpp-toml.md +++ b/docs/05-mcpp-toml.md @@ -149,7 +149,9 @@ macos_deployment_target = "14.0" # Minimum supported OS version for macOS arti `include_dirs_after` (#249) lists header directories that are searched **after** the toolchain's system directories (emitted as `-idirafter` on GCC/Clang, as -trailing `/I` under the MSVC dialect, which has no equivalent). Use it instead of +trailing `/I` under the MSVC dialect, and as plain `-I` for NASM +assembly units — neither has an equivalent, and neither has a system-header +chain to protect). Use it instead of `include_dirs` when the directory is an extracted source-tarball root that contains files whose names collide with standard headers — e.g. ffmpeg's top-level `VERSION` file shadows libc++'s `` on case-insensitive macOS diff --git a/docs/07-build-mcpp.md b/docs/07-build-mcpp.md index f714eebc..7f7071a2 100644 --- a/docs/07-build-mcpp.md +++ b/docs/07-build-mcpp.md @@ -49,9 +49,9 @@ is ignored, so you can freely log diagnostics. | `mcpp:link-search=` | add a library search dir (`-L`; relative dirs resolve against the project root) | | `mcpp:cfg=` | define `-D` for both C and C++ | | `mcpp:generated=` | add a generated source (relative to the project root) to the build | -| `mcpp:source=` | select a **pre-existing** source file into the build (absolute, or relative to the package root). Same downstream effect as `generated=`; use it for files the program *chose* (payload/vendored tree) rather than wrote — e.g. a per-target source selection over a large tarball | -| `mcpp:include-dir=` | add a **private** include directory (`-I`) for this package's own TUs (absolute, or relative to the package root; normalized). Replaces the `cxxflag=-I` + `cflag=-I` double emission | -| `mcpp:include-dir-after=` | like `include-dir`, but searched **after** the system directories (`-idirafter`) — for payload trees that shadow system headers | +| `mcpp:source=` *(0.0.100+)* | select a **pre-existing** source file into the build (absolute, or relative to the package root). Same downstream effect as `generated=`; use it for files the program *chose* (payload/vendored tree) rather than wrote — e.g. a per-target source selection over a large tarball | +| `mcpp:include-dir=` *(0.0.100+)* | add a **private** include directory (`-I`) for this package's own TUs (absolute, or relative to the package root; normalized). Replaces the `cxxflag=-I` + `cflag=-I` double emission | +| `mcpp:include-dir-after=` *(0.0.100+)* | like `include-dir`, but searched **after** the system directories (`-idirafter`) — for payload trees that shadow system headers | | `mcpp:rerun-if-changed=` | re-run `build.mcpp` when this file changes | | `mcpp:rerun-if-env-changed=` | re-run `build.mcpp` when this env var changes | @@ -111,16 +111,16 @@ The running program receives the build context as `MCPP_*` variables | Variable | Typed reader | Value | |---|---|---| | `MCPP_TARGET` | `mcpp::target()` | resolved canonical triple (the `--target` triple under cross; the host triple natively) | -| `MCPP_TARGET_OS` | `mcpp::target_os()` | the target's OS segment (`linux`/`macos`/`windows`) — no need to hand-split `MCPP_TARGET` | -| `MCPP_TARGET_ARCH` | `mcpp::target_arch()` | the target's arch segment (GNU spelling: `x86_64`, `aarch64`, …) | -| `MCPP_TARGET_ENV` | `mcpp::target_env()` | the target's env segment (`gnu`/`musl`/`msvc`); empty string when the triple has none (macOS) | +| `MCPP_TARGET_OS` *(0.0.100+)* | `mcpp::target_os()` | the target's OS segment (`linux`/`macos`/`windows`) — no need to hand-split `MCPP_TARGET` | +| `MCPP_TARGET_ARCH` *(0.0.100+)* | `mcpp::target_arch()` | the target's arch segment (GNU spelling: `x86_64`, `aarch64`, …) | +| `MCPP_TARGET_ENV` *(0.0.100+)* | `mcpp::target_env()` | the target's env segment (`gnu`/`musl`/`msvc`); empty string when the triple has none (macOS) | | `MCPP_HOST` | `mcpp::host()` | the host triple | | `MCPP_PROFILE` | `mcpp::profile()` | effective profile name (`dev`/`release`/…) | | `MCPP_OUT_DIR` | `mcpp::out_dir()` | a writable scratch/output dir owned by mcpp | | `MCPP_MANIFEST_DIR` | `mcpp::manifest_dir()` | the package root (= CWD) | | `MCPP_FEATURE_` | `mcpp::has_feature("name")` | set to `1` per active feature (same `` sanitization as the `MCPP_FEATURE_` compile macro) | | `MCPP_FEATURES` | — | comma-separated active feature list | -| `MCPP_DEP__DIR` | `mcpp::dep_dir("name")` | the resolved install dir of each declared dependency (canonical **and** namespace-stripped name spellings; same `` sanitization as `MCPP_FEATURE_`). Received by dependencies' build.mcpp **and** the root project's (the root runs after dependency resolution) | +| `MCPP_DEP__DIR` | `mcpp::dep_dir("name")` | the resolved install dir of each declared dependency (canonical **and** namespace-stripped name spellings; same `` sanitization as `MCPP_FEATURE_`). Received by dependencies' build.mcpp **and** the root project's (the root runs after dependency resolution, 0.0.100+) | These values are folded into the re-run key **unconditionally** — changing the target, profile, or feature set re-runs the program without any @@ -147,7 +147,7 @@ directives and re-runs only when something it depends on changed: - the toolchain, - any file you declared with `rerun-if-changed`, - any env var you declared with `rerun-if-env-changed`, -- (or a `generated` output went missing). +- (or a `generated` output / `source=` selection went missing). So **declare your inputs**: if your program reads `config.h` or the `USE_FAST` variable, emit `mcpp:rerun-if-changed=config.h` / `mcpp:rerun-if-env-changed=USE_FAST`. diff --git a/docs/zh/05-mcpp-toml.md b/docs/zh/05-mcpp-toml.md index 92384de8..f900bbef 100644 --- a/docs/zh/05-mcpp-toml.md +++ b/docs/zh/05-mcpp-toml.md @@ -141,8 +141,8 @@ macos_deployment_target = "14.0" # macOS 产物的最低支持系统版本(仅 ``` `include_dirs_after`(#249)列出**排在工具链系统目录之后**搜索的头文件目录 -(GCC/Clang 发射为 `-idirafter`;MSVC 方言无对应能力,退化为排在末尾的 -`/I`)。当目录是解压后的源码 tarball 根目录、且其中的文件名会与标准头冲突时, +(GCC/Clang 发射为 `-idirafter`;MSVC 方言退化为排在末尾的 `/I`,NASM 汇编 +单元退化为普通 `-I`——两者都没有对应 flag,也都没有需要保护的系统头搜索链)。当目录是解压后的源码 tarball 根目录、且其中的文件名会与标准头冲突时, 用它代替 `include_dirs` —— 例如 ffmpeg 根目录的 `VERSION` 文件在大小写不敏感 的 macOS 文件系统上会把 libc++ 的 `` 遮蔽(若该根目录挂在 `-I` 上)。 使用 `include_dirs_after` 时系统头永远优先,而包自己的真实头文件 diff --git a/docs/zh/07-build-mcpp.md b/docs/zh/07-build-mcpp.md index f8c650d6..63a814a6 100644 --- a/docs/zh/07-build-mcpp.md +++ b/docs/zh/07-build-mcpp.md @@ -46,9 +46,9 @@ mcpp build # 编译 + 运行 build.mcpp,然后构建工程 | `mcpp:link-search=` | 增加库搜索目录(`-L`;相对路径按工程根目录解析) | | `mcpp:cfg=` | 为 C 与 C++ 同时定义 `-D` | | `mcpp:generated=` | 把生成的源码(相对工程根目录)加入构建 | -| `mcpp:source=` | 把一份**既有**源文件选入构建(绝对路径,或相对包根)。下游效果与 `generated=` 相同;语义区别在于文件是程序*选中*的(tarball payload / vendored 源树)而非程序写出的——例如对大型源码包做 per-target 源选择 | -| `mcpp:include-dir=` | 为本包自身 TU 增加一个**私有** include 目录(`-I`;绝对路径或相对包根,自动规范化)。取代过去 `cxxflag=-I` + `cflag=-I` 的双重裸发 | -| `mcpp:include-dir-after=` | 同 `include-dir`,但排在系统目录**之后**搜索(`-idirafter`)——用于会遮蔽系统头的 payload 源树 | +| `mcpp:source=` *(0.0.100+)* | 把一份**既有**源文件选入构建(绝对路径,或相对包根)。下游效果与 `generated=` 相同;语义区别在于文件是程序*选中*的(tarball payload / vendored 源树)而非程序写出的——例如对大型源码包做 per-target 源选择 | +| `mcpp:include-dir=` *(0.0.100+)* | 为本包自身 TU 增加一个**私有** include 目录(`-I`;绝对路径或相对包根,自动规范化)。取代过去 `cxxflag=-I` + `cflag=-I` 的双重裸发 | +| `mcpp:include-dir-after=` *(0.0.100+)* | 同 `include-dir`,但排在系统目录**之后**搜索(`-idirafter`)——用于会遮蔽系统头的 payload 源树 | | `mcpp:rerun-if-changed=` | 该文件变化时重跑 `build.mcpp` | | `mcpp:rerun-if-env-changed=` | 该环境变量变化时重跑 `build.mcpp` | @@ -103,16 +103,16 @@ int main() { | 变量 | 类型化读取 | 值 | |---|---|---| | `MCPP_TARGET` | `mcpp::target()` | 解析后的 canonical 三元组(交叉构建下是 `--target` 三元组,原生构建是宿主) | -| `MCPP_TARGET_OS` | `mcpp::target_os()` | 目标的 OS 段(`linux`/`macos`/`windows`)——不必再手撕 `MCPP_TARGET` | -| `MCPP_TARGET_ARCH` | `mcpp::target_arch()` | 目标的 arch 段(GNU 拼写:`x86_64`、`aarch64`…) | -| `MCPP_TARGET_ENV` | `mcpp::target_env()` | 目标的 env 段(`gnu`/`musl`/`msvc`);三元组无 env 段(macOS)时为空串 | +| `MCPP_TARGET_OS` *(0.0.100+)* | `mcpp::target_os()` | 目标的 OS 段(`linux`/`macos`/`windows`)——不必再手撕 `MCPP_TARGET` | +| `MCPP_TARGET_ARCH` *(0.0.100+)* | `mcpp::target_arch()` | 目标的 arch 段(GNU 拼写:`x86_64`、`aarch64`…) | +| `MCPP_TARGET_ENV` *(0.0.100+)* | `mcpp::target_env()` | 目标的 env 段(`gnu`/`musl`/`msvc`);三元组无 env 段(macOS)时为空串 | | `MCPP_HOST` | `mcpp::host()` | 宿主三元组 | | `MCPP_PROFILE` | `mcpp::profile()` | 生效 profile 名(`dev`/`release`/…) | | `MCPP_OUT_DIR` | `mcpp::out_dir()` | mcpp 提供的可写输出/暂存目录 | | `MCPP_MANIFEST_DIR` | `mcpp::manifest_dir()` | 包根(= CWD) | | `MCPP_FEATURE_` | `mcpp::has_feature("name")` | 每个活跃 feature 置 `1`(`` 消毒规则与 `MCPP_FEATURE_` 编译宏一致) | | `MCPP_FEATURES` | — | 活跃 feature 逗号列表 | -| `MCPP_DEP__DIR` | `mcpp::dep_dir("name")` | 每个已声明依赖解析后的安装目录(canonical 名与去命名空间短名两种拼写都可用;`` 消毒规则同 `MCPP_FEATURE_`)。依赖包的 build.mcpp **和**根工程的 build.mcpp 都能拿到(根工程的 build.mcpp 在依赖解析之后运行) | +| `MCPP_DEP__DIR` | `mcpp::dep_dir("name")` | 每个已声明依赖解析后的安装目录(canonical 名与去命名空间短名两种拼写都可用;`` 消毒规则同 `MCPP_FEATURE_`)。依赖包的 build.mcpp **和**根工程的 build.mcpp 都能拿到(根工程的 build.mcpp 在依赖解析之后运行,0.0.100+) | 这些契约值**无条件**折入重跑键——换 target、换 profile、开关 feature 都会触发重跑, 不需要任何 `rerun-if-env-changed` 声明。 @@ -135,7 +135,7 @@ mcpp **不会**每次构建都重跑 `build.mcpp`。它会缓存程序产出的 - 工具链, - 任何用 `rerun-if-changed` 声明的文件, - 任何用 `rerun-if-env-changed` 声明的环境变量, -- (或某个 `generated` 产物丢失了)。 +- (或某个 `generated` 产物 / `source=` 选中的文件丢失了)。 所以请**声明你的输入**:如果程序读了 `config.h` 或 `USE_FAST` 变量,就分别 emit `mcpp:rerun-if-changed=config.h` / `mcpp:rerun-if-env-changed=USE_FAST`。这用一份明确的 diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index 127b5b92..ad3d4bd4 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -60,7 +60,15 @@ namespace { std::string escape_ninja_path(const std::filesystem::path& p) { // Ninja escapes: $ → $$, : → $:, space → $ (with leading space). // For simplicity we wrap in case-by-case. - std::string s = p.string(); + // + // generic_string(): ninja node names must be forward-slash even on + // Windows (#247). rspfile_content = $in copies node names verbatim into + // a response file that gcc/clang/GNU ar tokenize GNU-style, where + // backslash is an ESCAPE character — `obj\cli.o` would arrive as + // `objcli.o`. Every Windows consumer of these strings (CreateProcess + // path resolution, cl.exe/link.exe, PowerShell Copy-Item, ninja itself) + // accepts forward slashes; POSIX output is byte-identical. + std::string s = p.generic_string(); std::string out; for (char c : s) { if (c == '$') @@ -87,8 +95,12 @@ std::string escape_flag_path(const std::filesystem::path& p) { return out; } -std::string local_include_flags(const CompileUnit& cu, bool msvcDialect, - bool nasmUnit) { +bool is_nasm_source(const std::filesystem::path& src) { + return src.extension() == ".asm"; +} + +std::string local_include_flags(const CompileUnit& cu, bool msvcDialect) { + const bool nasmUnit = is_nasm_source(cu.source); std::string flags; for (auto const& inc : cu.localIncludeDirs) { flags += " -I"; @@ -197,10 +209,6 @@ bool is_gas_source(const std::filesystem::path& src) { return ext == ".S" || ext == ".s"; } -bool is_nasm_source(const std::filesystem::path& src) { - return src.extension() == ".asm"; -} - // TUs the P1689 module scan must skip: C-family and assembly units cannot // contain `import`/`module` declarations, and feeding them to the scanner // would route them through the C++ frontend. @@ -580,39 +588,25 @@ std::string emit_ninja_string(const BuildPlan& plan) { } // Link/archive/shared: driver-style (g++/clang++ are the linker) vs the - // msvc dialect's separate link.exe/lib.exe. The msvc commands go through - // response files — object lists exceed cmd.exe's 8191-char limit fast. - if (separateLinker) { - append("rule cxx_link\n"); - append(" command = $ld /nologo /OUT:$out @$out.rsp $ldflags $unit_ldflags\n"); - append(" rspfile = $out.rsp\n"); - append(" rspfile_content = $in\n"); - append(" description = LINK $out\n\n"); - - append("rule cxx_archive\n"); - append(" command = $ar /nologo /OUT:$out @$out.rsp\n"); - append(" rspfile = $out.rsp\n"); - append(" rspfile_content = $in\n"); - append(" description = AR $out\n\n"); - - append("rule cxx_shared\n"); - append(" command = $ld /nologo /DLL /OUT:$out /IMPLIB:$out.lib " - "@$out.rsp $ldflags $unit_ldflags\n"); - append(" rspfile = $out.rsp\n"); - append(" rspfile_content = $in\n"); - append(" description = SHARED $out\n\n"); - } else { - // Driver-style toolchains (g++/clang++ as the link driver). On - // Windows these still spawn through CreateProcess (32 KiB command - // line ceiling), and large source packages (ffmpeg/opencv-class) - // link thousands of objects — an inlined $in overflows it (#247). - // gcc/clang drivers and GNU/llvm ar all accept @rspfile, so route - // $in through one there. POSIX keeps the inline form byte-identical: - // ARG_MAX is ample and the plain command is easier to reproduce. - auto driver_rule = [&](std::string_view name, std::string cmd, - std::string_view desc) { + // msvc dialect's separate link.exe/lib.exe. One emitter owns the rule + // shape; `useRsp` decides whether $in is inlined or routed through a + // response file (`$in → @$out.rsp` — the only `$i…` variable in any + // link/archive command; revisit the first-match replace if a dialect + // ever grows another). + // + // rsp is used when the command spawns through CreateProcess (32 KiB + // command-line ceiling): always for the separate-linker msvc dialect, + // and on Windows for driver-style too (#247 — ffmpeg/opencv-class + // packages link thousands of objects; clang/gcc drivers and GNU/llvm ar + // all accept @rspfile). POSIX driver-style keeps the inline form + // byte-identical: ARG_MAX is ample and the plain command is easier to + // reproduce by hand. + { + const bool useRsp = separateLinker || mcpp::platform::is_windows; + auto link_rule = [&](std::string_view name, std::string cmd, + std::string_view desc) { append(std::format("rule {}\n", name)); - if constexpr (mcpp::platform::is_windows) { + if (useRsp) { if (auto pos = cmd.find("$in"); pos != std::string::npos) cmd.replace(pos, 3, "@$out.rsp"); append(std::format(" command = {}\n", cmd)); @@ -623,12 +617,23 @@ std::string emit_ninja_string(const BuildPlan& plan) { } append(std::format(" description = {} $out\n\n", desc)); }; - driver_rule("cxx_link", - "$cxx $in -o $out $ldflags $unit_ldflags", "LINK"); - driver_rule("cxx_archive", std::string(dial.archiveCmd), "AR"); - driver_rule("cxx_shared", - "$cxx -shared $in -o $out $ldflags $soname_flag $unit_ldflags", - "SHARED"); + if (separateLinker) { + link_rule("cxx_link", + "$ld /nologo /OUT:$out $in $ldflags $unit_ldflags", + "LINK"); + link_rule("cxx_archive", std::string(dial.archiveCmd), "AR"); + link_rule("cxx_shared", + "$ld /nologo /DLL /OUT:$out /IMPLIB:$out.lib " + "$in $ldflags $unit_ldflags", + "SHARED"); + } else { + link_rule("cxx_link", + "$cxx $in -o $out $ldflags $unit_ldflags", "LINK"); + link_rule("cxx_archive", std::string(dial.archiveCmd), "AR"); + link_rule("cxx_shared", + "$cxx -shared $in -o $out $ldflags $soname_flag $unit_ldflags", + "SHARED"); + } } append("rule runtime_alias\n"); @@ -750,7 +755,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { append(std::format("build {} : cxx_scan {}\n", escape_ninja_path(ddi), escape_ninja_path(cu.source))); append(std::format(" compile_target = {}\n", escape_ninja_path(cu.object))); - if (auto includes = local_include_flags(cu, msvcDeps, is_nasm_source(cu.source)); !includes.empty()) + if (auto includes = local_include_flags(cu, msvcDeps); !includes.empty()) append(std::format(" local_includes ={}\n", includes)); if (auto flags = join_flags(cu.packageCxxflags); !flags.empty()) append(std::format(" unit_cxxflags ={}\n", flags)); @@ -829,7 +834,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { } else { out_line += "\n"; } - if (auto includes = local_include_flags(cu, msvcDeps, is_nasm_source(cu.source)); !includes.empty()) + if (auto includes = local_include_flags(cu, msvcDeps); !includes.empty()) out_line += " local_includes =" + includes + "\n"; if (is_gas_source(cu.source) || is_nasm_source(cu.source)) { if (auto flags = join_flags(asm_unit_flags(cu)); !flags.empty()) @@ -879,7 +884,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { if (!implicit.empty()) out_line += " |" + implicit; out_line += "\n"; - if (auto includes = local_include_flags(cu, msvcDeps, is_nasm_source(cu.source)); !includes.empty()) + if (auto includes = local_include_flags(cu, msvcDeps); !includes.empty()) out_line += " local_includes =" + includes + "\n"; if (is_gas_source(cu.source) || is_nasm_source(cu.source)) { if (auto flags = join_flags(asm_unit_flags(cu)); !flags.empty()) diff --git a/src/build/plan.cppm b/src/build/plan.cppm index 68eaff8e..8f1133fc 100644 --- a/src/build/plan.cppm +++ b/src/build/plan.cppm @@ -280,7 +280,7 @@ local_include_dirs_after_for_manifest(const std::filesystem::path& root, { std::vector dirs; for (auto const& inc : manifest.buildConfig.includeDirsAfter) { - for (auto& d : expand_manifest_include_entry(root, std::filesystem::path(inc))) + for (auto& d : expand_manifest_include_entry(root, inc)) dirs.push_back(std::move(d)); } return dirs; diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 11b2871a..e70cebdc 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -1911,6 +1911,45 @@ prepare_build(bool print_fingerprint, return changed; }; + // ONE owner of "which compile-visible channels a build.mcpp directive + // lands in" — shared by the dep loop and the root call site so the + // next directive kind cannot be threaded through one and silently + // missed in the other (the #242 two-derivations failure shape). + // apply() mutates the manifest the program ran against; this folds + // the NEW tail (recorded sizes → end) into the package's + // usage-resolved privateBuild, which is what its TUs actually read. + // Link/source/fingerprint residues stay at the call sites — they + // genuinely differ between root and dep (see each). + struct DirectiveMark { std::size_t c, cx, inc, incAfter; }; + auto markDirectiveTail = [](const mcpp::manifest::Manifest& mm) { + return DirectiveMark{ mm.buildConfig.cflags.size(), + mm.buildConfig.cxxflags.size(), + mm.buildConfig.includeDirs.size(), + mm.buildConfig.includeDirsAfter.size() }; + }; + auto foldDirectiveTailIntoPrivateBuild = + [&](auto& pkg, const mcpp::manifest::Manifest& ran, + const DirectiveMark& t) + { + auto const& bc = ran.buildConfig; + pkg.privateBuild.cflags.insert(pkg.privateBuild.cflags.end(), + bc.cflags.begin() + t.c, bc.cflags.end()); + pkg.privateBuild.cxxflags.insert(pkg.privateBuild.cxxflags.end(), + bc.cxxflags.begin() + t.cx, bc.cxxflags.end()); + // include-dir[/-after] directives are PRIVATE (design §3.1: Cargo + // discipline — a build-time program must not widen the package's + // public interface): privateBuild only, never publicUsage. The + // after-dirs ride the typed #249 channel, which owns the + // per-dialect degradations (cl.exe /I, NASM -I). + for (auto it = bc.includeDirs.begin() + t.inc; + it != bc.includeDirs.end(); ++it) + appendUniquePath(pkg.privateBuild.includeDirs, *it); + for (auto it = bc.includeDirsAfter.begin() + t.incAfter; + it != bc.includeDirsAfter.end(); ++it) + appendUniquePath(pkg.privateBuild.includeDirsAfter, *it); + }; + + auto appendUniqueFlags = [](std::vector& flags, const std::vector& additions) -> bool @@ -1950,13 +1989,12 @@ prepare_build(bool print_fingerprint, { std::vector dirs; for (auto const& inc : manifest.buildConfig.includeDirsAfter) { - std::filesystem::path p(inc); - if (p.is_absolute()) { - appendUniquePath(dirs, p); + if (inc.is_absolute()) { + appendUniquePath(dirs, inc); continue; } for (auto& dir : mcpp::modgraph::expand_dir_glob( - packageRoot, p.generic_string())) { + packageRoot, inc.generic_string())) { appendUniquePath(dirs, dir); } } @@ -2445,8 +2483,7 @@ prepare_build(bool print_fingerprint, if (inc.is_relative()) inc = secondaryRoot / inc; } for (auto& inc : stagedManifest.buildConfig.includeDirsAfter) { - if (std::filesystem::path p(inc); p.is_relative()) - inc = (secondaryRoot / p).generic_string(); + if (inc.is_relative()) inc = secondaryRoot / inc; } dep_manifests.push_back( @@ -3052,10 +3089,8 @@ prepare_build(bool print_fingerprint, bpEnv.depDirs.emplace_back(canon.substr(dot + 1), depPkg.root); } auto& bcDep = pkg.manifest.buildConfig; - const auto cN = bcDep.cflags.size(), cxN = bcDep.cxxflags.size(), - ldN = bcDep.ldflags.size(), - incN = bcDep.includeDirs.size(), - incAfterN = bcDep.includeDirsAfter.size(); + const auto mark = markDirectiveTail(pkg.manifest); + const auto ldN = bcDep.ldflags.size(); if (auto r = mcpp::build::run_build_program( pkg.manifest, pkg.root, host->first, host->second, pkg.manifest.cppStandard.canonical, bpEnv); @@ -3063,36 +3098,18 @@ prepare_build(bool print_fingerprint, return std::unexpected(std::format( "dependency '{}': {}", pkg.manifest.package.name, r.error())); } - // Cargo scope wiring. Compile flags: the dep's TUs read the - // usage-resolved privateBuild (not bc) — mirror the newly added - // flags there, same as the MCPP_FEATURE_ macro does. Link flags: - // dep ldflags were propagated to the root during the BFS walk, - // which ran before this pass — forward the new tail (link-search - // paths are already absolute from parse_line). - pkg.privateBuild.cflags.insert(pkg.privateBuild.cflags.end(), - bcDep.cflags.begin() + cN, bcDep.cflags.end()); - pkg.privateBuild.cxxflags.insert(pkg.privateBuild.cxxflags.end(), - bcDep.cxxflags.begin() + cxN, bcDep.cxxflags.end()); + // Cargo scope wiring: compile-visible tail → privateBuild (the + // shared fold above; the dep's TUs read privateBuild, not bc — + // its consumers read publicUsage, which the fold never touches; + // the bcDep entries themselves are inert here: the descriptor + // include_dirs propagation snapshotted publicUsage at + // makePackageRoot, long before this pass). Dep residue: link + // flags — dep ldflags were propagated to the root during the + // BFS walk, which ran before this pass — forward the new tail + // (link-search paths are already absolute from parse_line). + foldDirectiveTailIntoPrivateBuild(pkg, pkg.manifest, mark); m->buildConfig.ldflags.insert(m->buildConfig.ldflags.end(), bcDep.ldflags.begin() + ldN, bcDep.ldflags.end()); - // include-dir[/-after] directives are PRIVATE (design §3.1: Cargo - // discipline — a build-time program must not widen the package's - // public interface). The dep's own TUs read privateBuild; its - // consumers read publicUsage (re-flowed by the usage fixpoint - // below) — so mirror the new tail into privateBuild ONLY, never - // publicUsage. The bcDep entries themselves are inert here: the - // descriptor include_dirs propagation snapshotted publicUsage at - // makePackageRoot, long before this pass. - for (auto it = bcDep.includeDirs.begin() + incN; - it != bcDep.includeDirs.end(); ++it) - appendUniquePath(pkg.privateBuild.includeDirs, *it); - // After-dirs ride the same typed #249 channel - // (privateBuild.includeDirsAfter → per-unit -idirafter), which - // owns the per-dialect degradations (cl.exe /I, NASM -I) a raw - // flag spelling would bypass. - for (auto it = bcDep.includeDirsAfter.begin() + incAfterN; - it != bcDep.includeDirsAfter.end(); ++it) - appendUniquePath(pkg.privateBuild.includeDirsAfter, *it); } // apply() may have added interface defines to packages' publicUsage @@ -3198,10 +3215,8 @@ prepare_build(bool print_fingerprint, bpEnv.depDirs.emplace_back(canon.substr(dot + 1), depPkg.root); } auto& bcRoot = m->buildConfig; - const auto rcN = bcRoot.cflags.size(), rcxN = bcRoot.cxxflags.size(), - rldN = bcRoot.ldflags.size(), rsrcN = bcRoot.sources.size(), - rincN = bcRoot.includeDirs.size(), - rincAfterN = bcRoot.includeDirsAfter.size(), + const auto mark = markDirectiveTail(*m); + const auto rldN = bcRoot.ldflags.size(), rsrcN = bcRoot.sources.size(), rmodN = m->modules.sources.size(); if (auto bp = mcpp::build::run_build_program( *m, *root, host->first, host->second, @@ -3210,52 +3225,42 @@ prepare_build(bool print_fingerprint, return std::unexpected(bp.error()); } auto& pkg0 = packages[0]; - // Sources → the scan walks packages[0].manifest, not *m. + // Compile-visible tail → privateBuild: the shared fold (same owner + // as the dep loop; the root's TUs read privateBuild). + foldDirectiveTailIntoPrivateBuild(pkg0, *m, mark); + // Root residues — apply() mutated *m, but packages[0].manifest is a + // value-copy snapshot taken at makePackageRoot, so everything the + // scan/fingerprint read from the snapshot needs the tail mirrored: + // sources → the scan walks packages[0].manifest, not *m. pkg0.manifest.buildConfig.sources.insert( pkg0.manifest.buildConfig.sources.end(), bcRoot.sources.begin() + rsrcN, bcRoot.sources.end()); pkg0.manifest.modules.sources.insert( pkg0.manifest.modules.sources.end(), m->modules.sources.begin() + rmodN, m->modules.sources.end()); - // Compile flags → the root's TUs read privateBuild (and the - // fingerprint folds packages[].manifest.buildConfig via - // canonical_package_build_metadata) — mirror both, as the old - // pre-snapshot ordering implicitly did. - pkg0.privateBuild.cflags.insert(pkg0.privateBuild.cflags.end(), - bcRoot.cflags.begin() + rcN, bcRoot.cflags.end()); - pkg0.privateBuild.cxxflags.insert(pkg0.privateBuild.cxxflags.end(), - bcRoot.cxxflags.begin() + rcxN, bcRoot.cxxflags.end()); + // Fingerprint metadata (canonical_package_build_metadata folds + // packages[].manifest.buildConfig) — mirror the flag/include tails, + // as the old pre-snapshot ordering implicitly did. pkg0.manifest.buildConfig.cflags.insert( pkg0.manifest.buildConfig.cflags.end(), - bcRoot.cflags.begin() + rcN, bcRoot.cflags.end()); + bcRoot.cflags.begin() + mark.c, bcRoot.cflags.end()); pkg0.manifest.buildConfig.cxxflags.insert( pkg0.manifest.buildConfig.cxxflags.end(), - bcRoot.cxxflags.begin() + rcxN, bcRoot.cxxflags.end()); + bcRoot.cxxflags.begin() + mark.cx, bcRoot.cxxflags.end()); + pkg0.manifest.buildConfig.includeDirs.insert( + pkg0.manifest.buildConfig.includeDirs.end(), + bcRoot.includeDirs.begin() + mark.inc, bcRoot.includeDirs.end()); + pkg0.manifest.buildConfig.includeDirsAfter.insert( + pkg0.manifest.buildConfig.includeDirsAfter.end(), + bcRoot.includeDirsAfter.begin() + mark.incAfter, + bcRoot.includeDirsAfter.end()); // Link flags → the final link reads *m (already applied); keep the - // linkUsage snapshot equivalent too. + // linkUsage snapshot and fingerprint metadata equivalent too. pkg0.linkUsage.ldflags.insert(pkg0.linkUsage.ldflags.end(), bcRoot.ldflags.begin() + rldN, bcRoot.ldflags.end()); pkg0.manifest.buildConfig.ldflags.insert( pkg0.manifest.buildConfig.ldflags.end(), bcRoot.ldflags.begin() + rldN, bcRoot.ldflags.end()); - // include-dir[/-after] directives: PRIVATE (already absolute from - // parse_line) — privateBuild only, never publicUsage (see the dep - // loop's rationale). privateBuild is what scanned units read - // (scanner → localIncludeDirs[After]); the manifest mirror keeps the - // fingerprint metadata equivalent, same as the flag mirrors above. - for (auto it = bcRoot.includeDirs.begin() + rincN; - it != bcRoot.includeDirs.end(); ++it) - appendUniquePath(pkg0.privateBuild.includeDirs, *it); - pkg0.manifest.buildConfig.includeDirs.insert( - pkg0.manifest.buildConfig.includeDirs.end(), - bcRoot.includeDirs.begin() + rincN, bcRoot.includeDirs.end()); - for (auto it = bcRoot.includeDirsAfter.begin() + rincAfterN; - it != bcRoot.includeDirsAfter.end(); ++it) - appendUniquePath(pkg0.privateBuild.includeDirsAfter, *it); - pkg0.manifest.buildConfig.includeDirsAfter.insert( - pkg0.manifest.buildConfig.includeDirsAfter.end(), - bcRoot.includeDirsAfter.begin() + rincAfterN, - bcRoot.includeDirsAfter.end()); } // [targets.*] required_features gate: a target is emitted only when ALL its diff --git a/src/manifest/types.cppm b/src/manifest/types.cppm index 1cf949a4..b2da3686 100644 --- a/src/manifest/types.cppm +++ b/src/manifest/types.cppm @@ -153,7 +153,7 @@ struct BuildConfig { std::map> featureDefines; std::vector includeDirs; // relative to package root // #249: emitted as -idirafter (searched after system dirs) - std::vector includeDirsAfter; + std::vector includeDirsAfter; std::map generatedFiles; // Form B package-owned support files std::vector globFlags; // [build] flags = [...] (ordered) bool staticStdlib = true; diff --git a/src/modgraph/scanner.cppm b/src/modgraph/scanner.cppm index d493ef99..826050cc 100644 --- a/src/modgraph/scanner.cppm +++ b/src/modgraph/scanner.cppm @@ -761,12 +761,11 @@ local_include_dirs_after_for(const std::filesystem::path& root, { std::vector dirs; for (auto const& inc : manifest.buildConfig.includeDirsAfter) { - std::filesystem::path p(inc); - if (p.is_absolute()) { - dirs.push_back(std::move(p)); + if (inc.is_absolute()) { + dirs.push_back(inc); continue; } - for (auto& d : expand_dir_glob(root, p.generic_string())) { + for (auto& d : expand_dir_glob(root, inc.generic_string())) { dirs.push_back(std::move(d)); } } diff --git a/src/platform/process.cppm b/src/platform/process.cppm index d0cd16cf..01a33e9f 100644 --- a/src/platform/process.cppm +++ b/src/platform/process.cppm @@ -398,8 +398,16 @@ RunResult capture_exec( return result; #else std::string cmd = command_from_argv(argv) + " 2>&1"; - if (!cwd.empty()) + if (!cwd.empty()) { +#if defined(_WIN32) + // cmd.exe `cd` without /d does not switch drives — a project on a + // different drive than mcpp's own cwd would run the child (the + // build.mcpp contract's only cwd consumer) in the wrong directory. + cmd = "cd /d " + mcpp::platform::shell::quote(cwd) + " && " + cmd; +#else cmd = "cd " + mcpp::platform::shell::quote(cwd) + " && " + cmd; +#endif + } return capture_with_env(cmd, extraEnv); #endif } From 413acbd51ef1530c50270a29aef9f30e8e2bfbfb Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 19 Jul 2026 21:25:25 +0800 Subject: [PATCH 12/12] chore(release): mcpp 0.0.99 -> 0.0.100 (#247 #248 #249) --- CHANGELOG.md | 21 +++++++++++++++++++++ mcpp.toml | 2 +- src/toolchain/fingerprint.cppm | 2 +- 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index df7af40c..db08d236 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,27 @@ > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。 +## [0.0.100] — 2026-07-19 + +> 大型源码直编包(ffmpeg/opencv 级,数千 TU)全平台化批次:#247/#248/#249 平台三修 + 增量构建修复 + build.mcpp 指令面补全(P1)。设计见 `.agents/docs/2026-07-19-large-source-pkg-platform-fixes-and-buildmcpp-generation-design.md`。 + +### 修复 + +- **windows driver-style 链接命令行溢出**(#247):gnu 方言(g++/clang++ 作链接驱动——windows 托管 clang-MSVC 即此)的 `cxx_link`/`cxx_archive`/`cxx_shared` 此前内联 `$in`,数千对象直接溢出 CreateProcess 32 KiB 上限。现 windows 上 driver-style 也走 response file;两方言分支收敛为单一 `link_rule` 发射器(msvc 规则文本逐字节不变),POSIX 保持内联零变化。配套:ninja 节点名统一 `generic_string()` 正斜杠——rsp 内容按 GNU 文法分词,反斜杠是转义符(`obj\cli.o` 会被吃成 `objcli.o`)。 +- **macOS dep/root build.mcpp 收不到 G3 契约环境**(#248,launcher-unify):非 Linux 的 `capture_exec` 走 shell 字符串拼接,`ENV… cd && bin` 里 env 只绑给 `cd`(全仓唯一 env+cwd 双非空调用点恰是 build.mcpp)。现 macOS 与 Linux 同走 posix_spawn 直启路径(child-only env + `addchdir_np`,`environ` 经 `_NSGetEnviron()` portable-correct),顺带消掉 macOS 的 shell quoting/注入面。windows shell 回退补 `cd /d`(跨盘符)。 +- **generated_files 每次构建无条件重写毁增量**:materialize 此前不比内容直接写,mtime 抖动令 ninja 把 include 该头的全部 TU 判脏——冻结快照包(config.h × 数千 TU)每次 build 都全量重编。现内容逐字节相同即跳写(变更检测本就由指纹负责)。 +- xpkg feature 表未知子键(如 `features.X.include_dirs`)此前静默吞掉,现进 `xpkgUnknownKeys` 走统一告警面。 + +### 新增 + +- **`[build] include_dirs_after` → `-idirafter`**(#249):排在工具链系统目录**之后**搜索的 include 目录(descriptor/xpkg 与 mcpp.toml 双文法、`*` tarball 根 glob、沿 Public/Interface 边传播且不升级为 `-I`)。治大小写不敏感 macOS 上依赖源根 `VERSION` 顶替 libc++ `` 一类系统头遮蔽;`-isystem` 不解此症(仍先于默认系统目录)。方言降级单点收敛:cl.exe → 末尾 `/I`,NASM 单元 → 普通 `-I`(NASM 会把 `-idirafter

` 误吞成 `-i dirafter

`)。附带统一主工程 include 路径的 glob 展开(此前与 dep 路径两套推导)。 +- **build.mcpp 指令面补全**(P1,0.0.100+):`mcpp:source=`(选既有 payload 源入编译集,`generated=` 的"绝对路径灰色用法"正名)、`mcpp:include-dir=` / `mcpp:include-dir-after=`(私有 include,Cargo 纪律不进公共接口;typed 通道享方言降级);typed `import mcpp;` 同步 `source()/include_dir()/include_dir_after()`。契约环境拆分 `MCPP_TARGET_OS/ARCH/ENV`(免手撕三元组)。**root build.mcpp 后移至依赖解析之后**,与 dep 一样拿到 `MCPP_DEP__DIR`;指令落通道收敛为 root/dep 共享的单一 fold(防"同一决策两处推导")。顺修:root `build.mcpp` 变更此前被整库 fast-path 吞掉不触发重跑。 +- **`mcpp xpkg parse --all-os`**:按 xpm 声明的平台集逐 OS 校验 per-OS 段(构建路径只 splice 宿主段,windows 段的 typo 在 linux CI 上原本不可见);mcpp-index CI 可单 runner lint 多平台描述符。 + +### 备注 + +- e2e 新增 141–145(-idirafter 语义/增量跳写/source= /include-dir/root dep-dirs);linux 全量 e2e 134 过(3 项为既知环境性失败)。macOS/windows 断面由平台 CI 与 mcpp-index spike 复现件(PR#89/90/91)收口。 + ## [0.0.99] — 2026-07-19 > #230–#243 批次收尾:#243 feature 转发落地(0.0.98 只出了设计)+ #238 根因修复随 xlings 0.4.67 vendored 入包 + #230 windows build.mcpp 次生面补齐。设计见 `.agents/docs/2026-07-19-v0.0.99-feature-forwarding-238-230-design.md`。 diff --git a/mcpp.toml b/mcpp.toml index af082296..fbb21ff8 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "0.0.99" +version = "0.0.100" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/toolchain/fingerprint.cppm b/src/toolchain/fingerprint.cppm index 3196c3fd..1204bc93 100644 --- a/src/toolchain/fingerprint.cppm +++ b/src/toolchain/fingerprint.cppm @@ -18,7 +18,7 @@ import mcpp.toolchain.detect; export namespace mcpp::toolchain { -inline constexpr std::string_view MCPP_VERSION = "0.0.99"; +inline constexpr std::string_view MCPP_VERSION = "0.0.100"; struct FingerprintInputs { Toolchain toolchain;