From 9cd0cfec185dace8da7c9dcf49964f3cdcf8b189 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Wed, 22 Jul 2026 16:57:23 +0800 Subject: [PATCH 1/2] docs: contract admission design (index-level + package-level floor) + pin fresh-install mcpp MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design only — no engine changes. The contract mechanism ("is this index/descriptor newer than I can trust?") is specified as two levels: the existing index.toml floor, and a future `mcpp = { min_mcpp = ... }` descriptor floor. The document fixes the shape both must share: - contract evaluation splits into pure predicate / idempotent decision / the one stateful reporter — fusing them makes the decision itself non-idempotent - the contract is a candidate-set filter, not a check on the read path - I7: a contract miss must never be a reason to try another candidate. Only at the source layer (index trees, interchangeable) does it show up as filtering; once identity is fixed, it is terminal for that entity - a refused index is excluded, never fatal — a too-new private index must not take down `mcpp toolchain install` or the upgrade path - the package-level key must enter the client vocabulary BEFORE any index uses it: unknown mcpp-segment keys are skipped today, so an early rollout would be silently ignored by exactly the clients it addresses #265 is filed as the empirical input (appendix), not as a defect to fix separately: it is only observable in the out-of-contract state this mechanism redefines, and the red CI was a version skew. CI: ci-fresh-install now installs an explicit `mcpp@${MCPP_PIN}` (0.0.102) instead of "newest in the runner's index copy". On 2026-07-21 the wait-index guard reported "index tracks 0.0.102" and the jobs installed 0.0.100 ten seconds later — the guard polls raw.githubusercontent.com while the runner resolves through its own index copy. Pinning turns a silent downgrade into an explicit `version not found`. Bump with the .xlings.json workspace pin at release. ci-aarch64-fresh-install stays unpinned by design (it simulates a brand-new user). --- .../2026-07-22-contract-admission-design.md | 266 ++++++++++++++++++ .github/workflows/ci-fresh-install.yml | 28 +- 2 files changed, 284 insertions(+), 10 deletions(-) create mode 100644 .agents/docs/2026-07-22-contract-admission-design.md diff --git a/.agents/docs/2026-07-22-contract-admission-design.md b/.agents/docs/2026-07-22-contract-admission-design.md new file mode 100644 index 00000000..6f953784 --- /dev/null +++ b/.agents/docs/2026-07-22-contract-admission-design.md @@ -0,0 +1,266 @@ +# 契约准入(Contract Admission)设计 —— 索引级 + 包级 floor + +> 日期:2026-07-22 +> 基线:mcpp **0.0.102**(`d571365`,含 PR #266) +> 状态:**设计文档,不含实现**。本文定义 mcpp 的「索引/描述符 → 客户端版本契约」机制应有的形状,供后续实现批次取用。 +> 触发来源:issue **#265**(`mcpp new --template imgui` 三平台失败)。#265 本身**不作为缺陷单独修复**——裁定见 §0.2;其现场分析作为本设计的实证输入,存档于 §6。 +> 所有 `file:line` 锚点在 `d571365` 上核实;§6 的行为结论用两个真实二进制实证。 +> 架构主线沿用 [2026-07-19-issues-230-243-batch-ledger-and-architecture-assessment.md](2026-07-19-issues-230-243-batch-ledger-and-architecture-assessment.md) §5.2「同一决策不许两处推导」,本文推广一格:**同一返回值不许编码两个正交语义**。 + +--- + +## 0. 结论速览 + +### 0.1 机制目标 + +一个客户端(mcpp 二进制)可能遇到比自己新的索引树或描述符。契约机制回答一个问题:**这份数据我能不能信?** 目标形态是**两级契约**: + +| 层 | 载体 | 求值点 | 不满足时 | +|---|---|---|---| +| 索引树 | `index.toml` 的 `min_mcpp`(**已实现**,0.0.85 起) | 索引集合枚举 —— 必须早于解析该树的任何描述符 | 该树移出搜索路径,**继续搜其他索引**;绝不终止进程 | +| 描述符 | `mcpp = { min_mcpp = ... }`(**未实现**,本文设计) | 段体被采信之前(`synthesize_from_xpkg_lua` 入口) | **终止该包的解析,精确报错;绝不回退到另一个候选** | + +两级是**合取**关系,顺序被强制,包级不能下调索引级(§2.2)。 + +### 0.2 关于 #265 的裁定 + +`mcpp new --template imgui` 在 fresh-install CI 三平台失败,现场根因是 `check_index_floor()` 把「是否违约」与「本次要不要打印」编码进同一个返回值,导致**同一进程内第一次查询被拦、之后静默放行**(§6)。 + +**裁定:不作为独立缺陷修复**,并入本文的契约机制整体实现。理由: + +- 它只在「客户端比索引旧」这一**契约外状态**下可见,而这正是本机制要重新定义的区域; +- 单独打补丁会固化当前的单级形态,与 §2 的两级模型冲突; +- CI 变红的直接原因是版本 skew(§5),与代码修不修无关。 + +**承认的残留**:在该状态下,依赖解析路径会静默使用客户端读不懂的描述符 —— 正是索引侧抬 floor 想防的。记录在案,随本机制实现一并消除。 + +### 0.3 本设计的关键结论 + +| # | 结论 | 依据 | +|---|------|------| +| 1 | 契约求值必须拆成**纯谓词 / 幂等决策 / 唯一有状态 reporter** 三件事 | §1.1;混用一个返回值必然让决策非幂等(§6.2) | +| 2 | 契约是**候选集过滤器**,不是读取路径上的检查 | §1.2;roadmap W5 原文即 open-time | +| 3 | **I7:契约不满足永远不能成为「再换一个候选」的理由** | §1.3;#265 的因果链就是「不满足 → 换候选 → 换到另一个包」 | +| 4 | 被拒索引 = 排除该索引,**不是终止进程** | §1.4;否则一个超前的私有索引就能 brick 掉 `mcpp toolchain install` 与升级路径 | +| 5 | 查找落空要**精确归因**(「它在索引 X 里,X 要求 mcpp >= N」),而不是把用户指向 `mcpp index update` | §1.5 | +| 6 | 包级 floor 的键**必须先于索引启用它进入客户端词表**,否则老客户端静默忽略,机制自废 | §2.3;未知 mcpp 段键今天是「跳过 + 记录 + 警告」(`xpkg.cppm:1519-1528`) | +| 7 | 不让 `schema` 兼任客户端下限 | §2.4;同一决策两处推导 | +| 8 | 索引侧需要 lint:描述符文法 ≤ 自身声明的 `min_mcpp` | §2.5;否则维护者会忘记声明,包级 floor 沦为装饰 | + +--- + +## 1. 索引级契约:准入模型 + +### 1.1 三职责分离 + +现状(`src/pm/index_contract.cppm:96-112`)把三件事压进一个 `optional`:违约文本、放行判据、以及内部 `static std::set` 的通知去重。通知天然需要状态,决策天然必须无状态 —— 二者共用返回值,状态就必然渗进决策。 + +目标形状: + +```cpp +// 纯:该索引是否违约。无状态、无输出、不读环境,可任意次调用 +std::optional index_floor_violation(const path& indexRoot); + +// 唯一有状态:首次遇到该实体时打印,返回是否打印过。 +// 去重键是不透明字符串("index:mcpplibs" / 将来 "pkg:compat.opencv"), +// 不是路径 —— 两级契约共用同一个 reporter。 +bool report_blocked_once(std::string_view key, std::string_view violation); + +// 决策:本进程可否读取该索引。幂等 —— memo 缓存的是决策,不是通知 +bool index_admitted(const path& indexRoot); +``` + +核心原语仍只有一个:既有的纯谓词 `floor_violation(minMcpp, ownVersion)`(`index_contract.cppm:79-94`)。它天然**粒度无关**,索引级与包级都只是它的适配器。 + +### 1.2 契约是候选集过滤器 + +``` +数据根 (data/) ──枚举+排序──▶ 候选索引集合 + ├── admitted ──▶ 解析层只看得见这些 + └── blocked ──▶ 进程内报告一次(E0006); + 仅在失败路径被回访(§1.5) +``` + +解析层(描述符读取、身份校验、版本解析、依赖图)**完全不认识 floor**,只消费一个已过滤的集合。今天的实现把契约放在最内层的 `read_identity_verified_xpkg_lua`(`package_fetcher.cppm:582-619`),等于把「这棵树整体不可用」这种全局事实表达成「这次查找没找到」这种局部软失败。 + +求值点应为(每处一行): + +1. `sorted_index_dirs`(`package_fetcher.cppm:623-633`)→ 过滤后成为全局数据根的唯一索引集合来源(消费点 `:652`、`:698`); +2. `read_xpkg_lua_from_path`(`:667-679`)单根打开 —— 准入幂等,所以「每次调用都问」与「开一次问一次」行为等价; +3. 将来的描述符接受点(§2)。 + +> 印证:`.agents/docs/2026-07-08-descriptor-index-evolution-roadmap.md:27` 的 W5 结论原文即「**open-time check** is the mcpp-side enforcement」。实现当初落在了 read-time。 + +### 1.3 不变量 + +| ID | 不变量 | +|---|---| +| **I1** | 契约求值是纯函数:无状态、无输出、不读环境 | +| **I2** | 准入决策幂等:同一进程内同一实体,第 1 次与第 n 次结果恒等 | +| **I3** | 进程内唯一可变状态是「已报告集合」,只影响是否打印,**不影响任何返回值或控制流** | +| **I4** | 契约逻辑只存在于一个模块,且只在**候选集构造点**生效;候选集的消费者一律不得复检 | +| **I5** | 不静默降级:被拒实体不会以任何路径被采信;查找落空时给出精确归因,而非裸 not-found | +| **I6** | 永不 brick:无 `index.toml`、畸形 `min_mcpp`、逃生舱开启 → 一律准入;被拒索引只被排除,**绝不终止进程** | +| **I7** | **契约不满足永远不能成为「再换一个候选」的理由**。只有在身份尚未确定的来源层(索引树 = 可替代的来源),它才表现为候选集过滤;身份一旦确定(某个具体包的描述符),它是该实体的终局事实 | + +**I7 是本设计最重要的一条**,它把 #265 的因果链固化为禁令:当年是「契约不满足 → 换个 namespace 再试 → 换到了另一个包」。包级 floor 落地时,如果写成「这个描述符不合格就继续扫下一个索引」,会立刻复刻同一个 bug,只是换了触发点。 + +### 1.4 为什么是「排除」而不是「终止」 + +被拒索引若直接 abort 整个进程,一个超前的私有索引就能让用户连 `mcpp toolchain install`(升级自己的路径)都跑不了 —— 把局部问题升级成砖头。`xim-pkgindex` 没有 `index.toml`,永远不会被拒,所以工具链安装、升级、以及不依赖该索引的构建在任何情况下都应照常工作。这是 **I6** 的实质内容。 + +### 1.5 失败归因:精确回答,而不是泛泛后缀 + +查找落空后,**只扫被拒的索引**,用同一套身份校验回答一个确定的问题:这个包是不是就在被拒的索引里? + +- 今天:`dependency 'X' ... isn't cloned locally yet — run 'mcpp index update' first`(`resolver.cppm:97-102`)—— 把用户指向一个更新解决不了的方向; +- 目标:`dependency 'X': 'X' is present in index 'mcpplibs', but that index requires mcpp >= 0.0.102 [E0006]`。 + +成本只有失败路径上的一次额外扫描(成功路径零开销);契约知识仍只在准入层,调用点只是拼一句现成的话。两个消费点:`resolver.cppm:97-102`(依赖)、`create.cppm:51-55`(模板)。 + +### 1.6 逃生舱 + +`MCPP_INDEX_FLOOR=ignore` 语义保持,位置应在 `index_admitted` —— 它关的是**准入**,不是「打印」。包级 floor 落地时增加粒度无关的别名 `MCPP_FLOOR=ignore`(旧名保留)。 + +--- + +## 2. 包级契约:`mcpp = { min_mcpp = ... }` + +### 2.1 为什么位置是 `mcpp` 段 + +Form B 的清单由**索引维护者随时改写、与包版本解耦** —— 「索引改了、客户端没换」的窗口只存在于 Form B。Form A 的清单随 tarball 冻结,不存在该窗口,**不需要**对称机制。 + +per-OS 免费生效:`mcpp.linux.min_mcpp` 自动工作,因为当前平台的子表在解析循环前已拼进段体(`xpkg.cppm:1508-1512`)。 + +### 2.2 与索引级的关系:合取 + 强制顺序 + +索引 floor 保护的是「你能不能正确解析这棵树」。连解析都不可信时,从描述符里读到的包级 floor 同样不可信 —— 所以**索引级必须先于解析该树的任何描述符求值**,包级不能下调索引级。这条要写进模块头注释,防止以后被「优化」成一处。 + +### 2.3 启用顺序被强制:键先进客户端,索引后使用 + +今天未知的 mcpp 段键是**跳过 + 记录 + 适配点警告**(`xpkg.cppm:1519-1528`),不是错误。因此: + +> 索引一旦启用 `min_mcpp`,**老客户端会静默忽略它** —— 版本标记恰好被它要提醒的那批客户端忽略,机制自废;要么白做,要么又得靠抬索引级 floor 强推,回到我们想摆脱的连坐。 + +所以实现顺序是:**客户端先支持并发布 → 广泛部署 → 索引侧再启用**,中间至少隔一个传播周期。落地内容: + +- `kKnownXpkgKeys` 增加 `min_mcpp`(`xpkg.cppm:134-140`); +- 解析循环在 `schema` 分支旁增加一支(`:1514`),复用纯谓词 `floor_violation`; +- 违规 → `synthesize_from_xpkg_lua` 返回 `ManifestError`(终局失败,不回退 —— **I7**)。 + +### 2.4 不让 `schema` 兼任 + +`schema` 是布局标签(`:1514-1517`,"informational only")。让它承担客户端下限,就是两个机制表达同一个决策,触碰「同一决策不许两处推导」。`min_mcpp` 与 `index.toml` 共用同一个词、同一个谓词、同一句可执行提示。 + +### 2.5 索引侧配套 + +`mcpp xpkg parse` 增加 lint:**描述符使用了高于自身 `min_mcpp` 的文法即报错**。这是整套机制可靠的前提;否则维护者会忘记声明,包级 floor 变成装饰。 + +### 2.6 终局效果 + +#265 的触发是索引侧为了 `compat.opencv` 的 per-OS feature flags 把**整棵树**的 floor 抬到 0.0.101(mcpp-index #107),于是毫不相干的 imgui 模板被连坐。包级 floor 之后,受影响的只有 `compat.opencv`;索引级 floor 退回「整棵树布局/文法演进」才动 —— 这类连坐**从机制上消失**,而不是靠运维排期去躲。 + +--- + +## 3. 实现时的变更面(供后续批次取用,本 PR 不做) + +| 文件 | 改动 | +|---|---| +| `src/pm/index_contract.cppm` | `check_index_floor` → `index_floor_violation` / `report_blocked_once` / `index_admitted`;模块头写入 I4/I6/I7 与两级顺序约束 | +| `src/pm/package_fetcher.cppm` | `:593` 契约分支删除;索引集合过滤;`read_xpkg_lua_from_path` 单根准入;失败归因入口 | +| `src/pm/resolver.cppm` | `:97-102` 依赖 not-found 前先问归因 | +| `src/scaffold/create.cppm` | §6.3 的冗余第二趟删除;Form B 命中给精确诊断(PR #266 已用 `continue` 覆盖该场景,届时合并为一种写法) | +| `src/manifest/xpkg.cppm` | `min_mcpp` 进词表 + 解析分支 + 终局失败(§2.3) | + +**测试(缺口很大,实现时必须补齐)** + +- 单测:`index_contract` 今天只覆盖**纯**函数(`floor_violation`、`read_index_contract`);有状态的 `check_index_floor` **零覆盖**。必须加「同一实体连问 3 次结果恒等」——幂等缺陷天生逃逸任何「调一次」的用例。 +- e2e:floor 全链**零覆盖**(`grep -rl "min_mcpp\|E0006" tests/` 只命中那一个单测文件);roadmap W5 记的「9.9.9/0.0.84 e2e」在树里不存在。需要:违规索引不得回落到其他候选、E0006 恰好一次、逃生舱、多索引时只排除违规的那个、描述符级违规不得回退。 + +--- + +## 4. 显式否决的方案 + +| # | 方案 | 否决理由 | +|---|---|---| +| **R1** | 只删 `index_contract.cppm:110` 的去重行 | 每次查询都打印(单次 resolve 可达数十次)→ 刷屏;策略仍在机制层,且把「一个返回值两种语义」的类型缺陷原封保留 | +| **R2** | 任何被拒索引 → 全局 abort | 见 §1.4:把局部问题升级成砖头 | +| **R3** | 契约检查下沉到 xlings staged-unpack | 跨仓库;roadmap W5 已裁定 open-time 检查是 mcpp 侧责任;且不解决幂等性缺陷 | +| **R4** | 给每条 not-found 挂泛泛的「可能有索引被排除」后缀 | 信息量低且把契约知识散回所有调用点;已被 §1.5 的精确归因取代 | +| **R5** | 用 `schema` 承载包级 floor | 见 §2.4 | +| **R6** | 只在模板路径规避(PR #266 的形态) | 治标:同样场景仍失败;泄漏在依赖解析路径原样存在。#266 作为错误信息改善已合入(`d571365`),不承担机制职责 | + +--- + +## 5. 运维不变量(与代码无关,先行) + +#265 的**直接**触发是版本 skew,不是代码:索引侧 2026-07-21 把 `min_mcpp` 抬到 0.0.101(mcpp-index #107),而 `ci-fresh-install` 经 `xlings install mcpp` 装到的是 0.0.100。 + +现场证据:`wait-index` 守卫在 22:30:02 报告 `index tracks 0.0.102`,10 秒后同一次运行里的 job 仍装到 `xim:mcpp@0.0.100`。**守卫轮询的是 `raw.githubusercontent.com`,而 runner 解析走的是自己的索引副本(镜像/缓存链路)** —— 两者不是同一个源,所以守卫绿 ≠ 装得到。 + +不变量: + +1. **索引侧抬 `min_mcpp` 之前**,必须先确认发布链路(xim-pkgindex 及其镜像)已能装到满足该 floor 的 mcpp; +2. **fresh-install 必须安装「被测版本」**,而不是「索引里最新的那个」。本 PR 已把 `ci-fresh-install` 的四处安装点改为显式 `mcpp@${MCPP_PIN}`(工作流级 `env`,随发布与 `.xlings.json` 一起 bump)。装不到时以 `version not found` 显式失败,而不是静默降级到旧二进制。 +3. `ci-aarch64-fresh-install` 保持不 pin —— 它的既定目的就是「完全模拟新用户 `xlings install mcpp`」,且不与发布事件绑定。 + +包级 floor 落地后,这条运维约束的触发频率会大幅下降(§2.6),但**不会消失**:整棵树的布局演进仍需索引级 floor。 + +--- + +## 6. 附录:#265 现场分析(设计的实证输入) + +### 6.1 现象 + +CI `ci-fresh-install` 三平台同步失败(run [29873685262](https://github.com/mcpp-community/mcpp/actions/runs/29873685262)),步骤 `Template: mcpp new --template imgui`: + +``` +error: index requires mcpp >= 0.0.101 but this is mcpp 0.0.100 [E0006] + Downloading compat.imgui v1.92.8 +error: package 'imgui@1.92.8' has no mcpp.toml +``` + +两条 error 相隔 0.4 ms,**属于同一次失败**(issue 正文把第一条判为「无关的 workspace pin 滞后」,是误判)。 + +| # | 实验 | 结果 | +|---|------|------| +| E-a | `.../xim-x-mcpp/0.0.100/mcpp new abc1 --template imgui`(真实索引,floor=0.0.102) | `E0006` + `package 'imgui@1.92.8' has no mcpp.toml` —— **与 CI 逐字一致** | +| E-b | 同上 + `MCPP_INDEX_FLOOR=ignore` | `Created abc2 (template imgui@0.0.6:window)` —— **同一二进制、同一索引,仅关掉门就正确** | +| E-c | 0.0.102 同命令 | 正常 | + +**E-b 是判决性的**:候选顺序、Form A/B、索引内容全部无关,唯一变量是准入门吞掉了第一趟查找。`imgui.lua` 与 `compat.imgui.lua` 在**同一个索引目录**,且同一次查找里 `imgui.lua` 排在前面。 + +### 6.2 机制 + +```cpp +// index_contract.cppm:96-112 +if (!reported.insert(indexRoot).second) return std::nullopt; // :110 决策被通知污染 +``` + +唯一调用点(`package_fetcher.cppm:593`)把返回值当放行判据 → **第 1 次查询=拦截,第 2..n 次=静默放行**。 + +`fetch_template_package()`(`create.cppm:44-50`)恰好查两次: + +| 轮次 | 查询 | 该索引内候选顺序 | 实际结果 | +|---|---|---|---| +| 1 | `read_xpkg_lua("", "imgui")` | `imgui.lua` → `compat.imgui.lua` | **被门拦掉**,E0006 打印,`imgui.lua` 从未被看到 | +| 2 | `read_xpkg_lua("compat", "imgui")` | `compat.imgui.lua` → `imgui.lua` | 门已「报过」→ **放行** → 命中 Form B 的 `compat.imgui` | + +### 6.3 冗余的第二趟(I7 的直接来源) + +设请求短名为 `S`: + +| | 候选文件集(`compat.cppm:159-198`) | 接受判据(`xpkg.cppm:682-716`) | +|---|---|---| +| 趟 1 `ns=""` | `{S.lua, compat.S.lua}` | `A1 = (id.name == S)` —— discovery 模式,任何 namespace 都接受(`:700`) | +| 趟 2 `ns="compat"` | `{compat.S.lua, S.lua}` —— **同一集合**,仅顺序不同 | `A2 = (id.name == S && id.ns == "compat")` | + +`C2 = C1`、`A2 ⊊ A1` ⇒ 趟 2 能接受的任何描述符,趟 1 必然也接受;而趟 2 只在趟 1 返回 `nullopt` 时执行,趟 1 返回 `nullopt` ⇔ 所有索引里都不存在名字匹配的描述符 ⇒ **趟 2 必然也 `nullopt`**。 + +**准入正常时趟 2 恒为空转;它唯一能产出结果的情形,就是趟 1 被非语义原因中断。**这就是 I7 的由来:把「解析失败」当成「换个候选」的信号,会把一次报错升级成一次错误解析。 + +### 6.4 测试为什么没抓到 + +- 单测只覆盖纯函数,唯一有状态、唯一被生产代码调用的 `check_index_floor` 零覆盖; +- floor 全链零 e2e; +- 幂等性缺陷天生逃逸单次调用的测试 —— **任何「调一次」的用例都会通过**。 diff --git a/.github/workflows/ci-fresh-install.yml b/.github/workflows/ci-fresh-install.yml index bd5649c8..a15c0ddb 100644 --- a/.github/workflows/ci-fresh-install.yml +++ b/.github/workflows/ci-fresh-install.yml @@ -31,6 +31,17 @@ concurrency: group: ci-fresh-install cancel-in-progress: false # use false to test in PRs, true to only test released mcpp +# Version under test. Bare `xlings install mcpp` resolves "newest in the +# runner's index copy", which is NOT the same source the wait-index job +# polls (raw.githubusercontent.com) — on 2026-07-21 the guard reported +# "index tracks 0.0.102" and the jobs still installed 0.0.100 ten seconds +# later, which then met an index whose floor was 0.0.101 (#265). Pinning +# makes the version explicit: if the index cannot serve it yet, the job +# fails with `version not found` instead of silently testing an older +# binary. Bump together with the .xlings.json workspace pin at release. +env: + MCPP_PIN: '0.0.102' + jobs: # ────────────────────────────────────────────────────────────────── # Linux: gcc@16.1.0, musl-gcc@16.1.0, llvm@20.1.7 @@ -86,10 +97,9 @@ jobs: - name: Install mcpp and config mirror run: | # The release tarball bundles a pkgindex snapshot frozen at - # build time; refresh it so the workspace pin in .xlings.json - # (latest mcpp) resolves. + # build time; refresh it so the pinned mcpp resolves. xlings update - xlings install mcpp -y -g # install to global + xlings install "mcpp@${MCPP_PIN}" -y -g # install to global mcpp --version mcpp self config --mirror GLOBAL @@ -230,7 +240,7 @@ jobs: run: | export PATH="$HOME/.xlings/subos/current/bin:$PATH" xlings update - xlings install mcpp -y -g + xlings install "mcpp@${MCPP_PIN}" -y -g mcpp --version mcpp self config --mirror GLOBAL @@ -283,10 +293,9 @@ jobs: - name: Install mcpp and config mirror run: | - # Refresh the bundled pkgindex snapshot so the workspace pin - # in .xlings.json (latest mcpp) resolves. + # Refresh the bundled pkgindex snapshot so the pinned mcpp resolves. xlings update - xlings install mcpp -y -g # install to global + xlings install "mcpp@${MCPP_PIN}" -y -g # install to global mcpp --version mcpp self config --mirror GLOBAL @@ -342,10 +351,9 @@ jobs: - name: Install mcpp and config mirror shell: pwsh run: | - # Refresh the bundled pkgindex snapshot so the workspace pin - # in .xlings.json (latest mcpp) resolves. + # Refresh the bundled pkgindex snapshot so the pinned mcpp resolves. xlings update - xlings install mcpp -y -g --verbose + xlings install "mcpp@$env:MCPP_PIN" -y -g --verbose cat "$env:USERPROFILE\.xlings\.xlings.json" mcpp --version From a07cb40289f36b377ff23cc398ccb7bacd4b43cd Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Wed, 22 Jul 2026 17:34:02 +0800 Subject: [PATCH 2/2] ci: split windows/macos e2e into parallel workflows, fan out the linux job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Critical path was max(ci-windows 20.4m, ci-linux 18.2m) on a warm PR run. Both were single sequential jobs; everything below is wall-clock, not work. - .github/actions/bootstrap-mcpp: composite action for the cache lineage (mcpp sandbox + xlings + target/) and the xlings bootstrap, so the split jobs share ONE definition instead of a 40-line copy-pasted preamble each. Target cache is now keyed per job — the toolchain legs `clean` and rebuild, so a shared key had them clobbering each other's lineage. - .github/actions/setup-macos-llvm: same idea for the macOS-specific setup (xlings LLVM + sandbox wired to that exact package). - ci-linux.yml: one job -> four independent jobs (build+unit, cold gcc self-host, musl+llvm, xlings integration). No `needs:` — each pays one warm `mcpp build` (~2.5m), which is cheaper than serialising behind an artifact. Dropped the duplicate `mcpp test` in the gcc leg: same suite, same toolchain, same driver binary as build-test, differing only in incremental state. ~18m -> ~7-8m. - ci-windows.yml: e2e moved out (9.7m of 20.4m); remainder split into build+test+package and toolchains+regressions. Integration keeps its original position before the llvm rebuild that invalidates $MCPP_SELF. ~20m -> ~8m. - ci-windows-e2e.yml, ci-macos-e2e.yml: new, mirroring ci-linux-e2e.yml. macOS is not sharded — 3.5m is under this workflow's setup+build floor. - ci-macos.yml: e2e step removed, setup replaced by the composite; the clang probe/module validation steps stay here. - run_all.sh: E2E_SHARD="/" runs a round-robin slice. Round robin rather than contiguous ranges because per-test durations are wildly uneven (the existing timing report exists for exactly that reason). The slice is computed on the full file list before capability filtering, so a test's shard does not move with host capabilities. Linux and Windows e2e run 2 shards; 147 tests split 74/73. Expected critical path: ~20.4m -> ~8m. --- .github/actions/bootstrap-mcpp/action.yml | 113 +++++++++++ .github/actions/setup-macos-llvm/action.yml | 108 +++++++++++ .github/workflows/ci-linux-e2e.yml | 73 ++----- .github/workflows/ci-linux.yml | 184 +++++++++--------- .github/workflows/ci-macos-e2e.yml | 55 ++++++ .github/workflows/ci-macos.yml | 115 +---------- .github/workflows/ci-windows-e2e.yml | 64 ++++++ .github/workflows/ci-windows.yml | 203 ++++++++------------ tests/e2e/run_all.sh | 29 +++ 9 files changed, 568 insertions(+), 376 deletions(-) create mode 100644 .github/actions/bootstrap-mcpp/action.yml create mode 100644 .github/actions/setup-macos-llvm/action.yml create mode 100644 .github/workflows/ci-macos-e2e.yml create mode 100644 .github/workflows/ci-windows-e2e.yml diff --git a/.github/actions/bootstrap-mcpp/action.yml b/.github/actions/bootstrap-mcpp/action.yml new file mode 100644 index 00000000..25727485 --- /dev/null +++ b/.github/actions/bootstrap-mcpp/action.yml @@ -0,0 +1,113 @@ +name: bootstrap-mcpp +description: > + Restore the shared CI cache lineage (mcpp sandbox + xlings + target/) and + bootstrap a released mcpp via xlings. Exports MCPP and XLINGS_BIN. + + Extracted so the split CI jobs (build / toolchain legs / e2e shards / + integration) share ONE definition instead of copy-pasting a 40-line + preamble per job. Every job that uses it lands on the same cache keys, + which is what makes splitting cheap: each job restores a warm sandbox + and only pays one incremental `mcpp build`. + +inputs: + xlings-version: + description: xlings release to bootstrap from + required: false + default: '0.4.30' + cache-target: + description: also restore/save target/ (build artifacts + BMIs) + required: false + default: 'true' + +runs: + using: composite + steps: + # NOTE: the "-ci-" segment keeps this lineage disjoint from release.yml's + # "-release-" caches. A bare "mcpp-sandbox--" restore prefix used to + # match the release sandbox too, silently swapping in a differently + # populated registry (issue #120). + - name: Cache mcpp sandbox + uses: actions/cache@v4 + with: + path: ~/.mcpp + key: mcpp-sandbox-${{ runner.os }}-ci-${{ hashFiles('mcpp.toml', '.xlings.json') }} + restore-keys: | + mcpp-sandbox-${{ runner.os }}-ci- + + - name: Cache xlings + uses: actions/cache@v4 + with: + path: ~/.xlings + key: xlings-${{ runner.os }}-v2-${{ hashFiles('.xlings.json') }} + restore-keys: | + xlings-${{ runner.os }}-v2- + + - name: Bootstrap mcpp via xlings (unix) + if: runner.os != 'Windows' + shell: bash + env: + XLINGS_NON_INTERACTIVE: '1' + XLINGS_VERSION: ${{ inputs.xlings-version }} + run: | + # Always install the pinned version — the cache may hold an older + # xlings whose sysroot/packages are incompatible. + case "$(uname -s)" in + Darwin) tarball="xlings-${XLINGS_VERSION}-macosx-arm64.tar.gz" ;; + *) tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" ;; + esac + WORK=$(mktemp -d) + curl -fsSL -o "${WORK}/${tarball}" \ + "https://github.com/d2learn/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" + tar -xzf "${WORK}/${tarball}" -C "${WORK}" + "${WORK}/${tarball%.tar.gz}/subos/default/bin/xlings" self install + export PATH="$HOME/.xlings/subos/default/bin:$PATH" + echo "$HOME/.xlings/subos/default/bin" >> "$GITHUB_PATH" + xlings --version + # xim:mcpp — `xlings install` is idempotent, so cache hits skip the + # download. + xlings install mcpp -y + MCPP="$HOME/.xlings/subos/default/bin/mcpp" + test -x "$MCPP" + "$MCPP" --version + echo "MCPP=$MCPP" >> "$GITHUB_ENV" + echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" + + - name: Bootstrap mcpp via xlings (windows) + if: runner.os == 'Windows' + shell: bash + env: + XLINGS_NON_INTERACTIVE: '1' + XLINGS_VERSION: ${{ inputs.xlings-version }} + run: | + WORK=$(mktemp -d) + zipfile="xlings-${XLINGS_VERSION}-windows-x86_64.zip" + curl -fsSL -o "${WORK}/${zipfile}" \ + "https://github.com/d2learn/xlings/releases/download/v${XLINGS_VERSION}/${zipfile}" + cd "${WORK}" + unzip -q "${zipfile}" + "$WORK/xlings-${XLINGS_VERSION}-windows-x86_64/subos/default/bin/xlings.exe" self install + export PATH="$USERPROFILE/.xlings/subos/default/bin:$PATH" + echo "$USERPROFILE/.xlings/subos/default/bin" >> "$GITHUB_PATH" + xlings.exe --version + xlings.exe install mcpp -y + MCPP=$(find "$USERPROFILE/.xlings" -name "mcpp.exe" -path "*/bin/*" 2>/dev/null | head -1) + if [ -z "$MCPP" ]; then + MCPP=$(find "$USERPROFILE/.xlings" -name "mcpp" -path "*/bin/*" 2>/dev/null | head -1) + fi + test -n "$MCPP" || { echo "FAIL: mcpp not found after xlings install"; exit 1; } + "$MCPP" --version + echo "MCPP=$MCPP" >> "$GITHUB_ENV" + echo "XLINGS_BIN=$(cygpath -w "$USERPROFILE/.xlings/subos/default/bin/xlings.exe")" >> "$GITHUB_ENV" + + # Precise key on src/ + manifest so a no-source-change run lands on a full + # hit; layered restore-keys let partial hits keep BMI/dyndep state for a + # proper incremental build. + - name: Cache target/ (build artifacts + BMIs) + if: inputs.cache-target == 'true' + uses: actions/cache@v4 + with: + path: target + key: mcpp-target-${{ runner.os }}-${{ github.job }}-${{ hashFiles('src/**', 'tests/**', 'mcpp.toml', 'mcpp.lock') }} + restore-keys: | + mcpp-target-${{ runner.os }}-${{ github.job }}- + mcpp-target-${{ runner.os }}- diff --git a/.github/actions/setup-macos-llvm/action.yml b/.github/actions/setup-macos-llvm/action.yml new file mode 100644 index 00000000..b31382ad --- /dev/null +++ b/.github/actions/setup-macos-llvm/action.yml @@ -0,0 +1,108 @@ +name: setup-macos-llvm +description: > + macOS ARM64 setup shared by ci-macos.yml and ci-macos-e2e.yml: xlings + + the xlings-managed LLVM + a dev mcpp sandbox wired to that exact LLVM. + Exports LLVM_ROOT, CXX, MCPP_LLVM_VER, MCPP, XLINGS_BIN. + + Kept separate from bootstrap-mcpp: macOS does not reuse the ~/.mcpp cache + lineage — it builds its sandbox from the freshly resolved LLVM package on + every run, which is exactly the property the macOS job exists to prove. + +inputs: + xlings-version: + description: xlings release to bootstrap from + required: false + default: '0.4.30' + +runs: + using: composite + steps: + - name: Cache xlings + uses: actions/cache@v4 + with: + path: ~/.xlings + key: xlings-macos15-arm64-v3-${{ hashFiles('.xlings.json') }} + restore-keys: | + xlings-macos15-arm64-v3- + + - name: Bootstrap xlings + shell: bash + env: + XLINGS_NON_INTERACTIVE: '1' + XLINGS_VERSION: ${{ inputs.xlings-version }} + run: | + WORK=$(mktemp -d) + tarball="xlings-${XLINGS_VERSION}-macosx-arm64.tar.gz" + curl -fsSL -o "${WORK}/${tarball}" \ + "https://github.com/d2learn/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" + tar -xzf "${WORK}/${tarball}" -C "${WORK}" + XLINGS_DIR="${WORK}/xlings-${XLINGS_VERSION}-macosx-arm64" + "$XLINGS_DIR/subos/default/bin/xlings" self install + export PATH="$HOME/.xlings/subos/default/bin:$PATH" + xlings --version + echo "PATH=$HOME/.xlings/subos/default/bin:$PATH" >> "$GITHUB_ENV" + + - name: Install LLVM via xlings + shell: bash + run: | + # latest-first: test binaries now link the toolchain's own libc++ + # (A1 root fix in flags.cppm), so new llvm releases are + # self-consistent — the macOS job is the proof for each new default. + xlings install llvm -y || xlings install llvm@20.1.7 -y + LLVM_ROOT=$(find "$HOME/.xlings" -path "*/xpkgs/xim-x-llvm/*/bin/clang++" | head -1 | xargs dirname | xargs dirname) + ls "$LLVM_ROOT/bin/clang++" + "$LLVM_ROOT/bin/clang++" --version + echo "LLVM_ROOT=$LLVM_ROOT" >> "$GITHUB_ENV" + echo "CXX=$LLVM_ROOT/bin/clang++" >> "$GITHUB_ENV" + + - name: Bootstrap mcpp via xlings + shell: bash + run: | + xlings install mcpp -y + MCPP="$HOME/.xlings/subos/default/bin/mcpp" + test -x "$MCPP" + "$MCPP" --version + echo "MCPP=$MCPP" >> "$GITHUB_ENV" + echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" + + - name: Configure dev mcpp sandbox to reuse xlings LLVM + shell: bash + run: | + # Use EXACTLY the LLVM the install step resolved (env LLVM_ROOT) — a + # version-glob pick chose a stale cached 20.1.7 next to the freshly + # installed 22.1.8 and silently tested the wrong toolchain. + LLVM_PKG="$LLVM_ROOT" + test -d "$LLVM_PKG" + LLVM_VER=$(basename "$LLVM_PKG") + echo "MCPP_LLVM_VER=$LLVM_VER" >> "$GITHUB_ENV" + MCPP_LLVM_LINK="$HOME/.mcpp/registry/data/xpkgs/xim-x-llvm/$LLVM_VER" + printf '1\n' > "$LLVM_PKG/.mcpp_ok" + + mkdir -p "$HOME/.mcpp/registry/data/xpkgs/xim-x-llvm" + rm -rf "$MCPP_LLVM_LINK" + ln -s "$LLVM_PKG" "$MCPP_LLVM_LINK" + + mkdir -p "$HOME/.mcpp" + cat > "$HOME/.mcpp/config.toml" <> "$GITHUB_ENV" - echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" - - - name: Cache target/ (build artifacts + BMIs) - uses: actions/cache@v4 - with: - path: target - key: mcpp-target-${{ runner.os }}-${{ hashFiles('src/**', 'tests/**', 'mcpp.toml', 'mcpp.lock') }} - restore-keys: | - mcpp-target-${{ runner.os }}- + # toolchain/sandbox instead of re-installing it. + - uses: ./.github/actions/bootstrap-mcpp - name: Configure mirror + Build mcpp from source (self-host) run: | diff --git a/.github/workflows/ci-linux.yml b/.github/workflows/ci-linux.yml index e0fdd025..b1c08465 100644 --- a/.github/workflows/ci-linux.yml +++ b/.github/workflows/ci-linux.yml @@ -5,10 +5,17 @@ name: ci-linux # workflow no longer depends on a previous-release tarball — the # chicken-and-egg now lives upstream in the xlings index. # -# This job covers build + unit/integration tests + the toolchain matrix + -# the xlings integration build. The ~18 min e2e suite is a SEPARATE workflow -# (ci-linux-e2e.yml) that runs in parallel on the same warm caches, so the -# per-PR critical path is max(this, e2e) instead of their sum. +# SHAPE: four INDEPENDENT jobs, no `needs:` between them. Each restores the +# same cache lineage (see .github/actions/bootstrap-mcpp) and pays one warm +# `mcpp build` (~2.5 min) to get the PR's own binary, then does its own leg. +# That warm rebuild is far cheaper than serialising the legs behind a shared +# artifact would be: +# +# before: build → unit → gcc → musl → llvm → xlings ≈ 18 min (one job) +# after: max(build+unit, gcc, musl+llvm, xlings) ≈ 7-8 min +# +# The ~18 min e2e suite is a SEPARATE workflow (ci-linux-e2e.yml, sharded) +# that runs in parallel on the same caches. # # Paired workflows: ci-linux-e2e.yml, ci-macos.yml, ci-windows.yml. @@ -23,83 +30,23 @@ concurrency: group: ci-${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true +env: + # MCPP_HOME pinned so the cache keys below restore into the same path + # mcpp resolves at runtime. + MCPP_HOME: /home/runner/.mcpp + # Verbose every mcpp invocation for richer CI diagnostics (src/cli.cppm). + # Safe here: this workflow no longer runs the e2e suite, which is what + # asserts mcpp's default quiet output (tests 48/53). + MCPP_VERBOSE: "1" + jobs: - build-and-test: - name: build + test (linux x86_64, self-host) + build-test: + name: build + unit tests (linux x86_64, self-host) runs-on: ubuntu-24.04 - timeout-minutes: 60 - env: - # MCPP_HOME pinned so the cache key below restores into the - # same path mcpp resolves at runtime. - MCPP_HOME: /home/runner/.mcpp - # Verbose every mcpp invocation for richer CI diagnostics (src/cli.cppm). - MCPP_VERBOSE: "1" + timeout-minutes: 45 steps: - uses: actions/checkout@v4 - - # Cache mcpp's sandbox: the toolchain (musl-gcc + binutils + - # glibc + linux-headers + patchelf + ninja) takes minutes to - # install on a cold runner. Key on the workspace manifest so a - # toolchain change in mcpp.toml refreshes the cache; restore-keys - # provide a layered fallback so near-misses still skip the slow - # toolchain installs. - - name: Cache mcpp sandbox - uses: actions/cache@v4 - with: - path: ~/.mcpp - # NOTE: the "-ci-" segment keeps this lineage disjoint from - # release.yml's "-release-" caches. A bare "mcpp-sandbox--" - # restore prefix used to match the release sandbox too, silently - # swapping in a differently-populated registry (issue #120). - key: mcpp-sandbox-${{ runner.os }}-ci-${{ hashFiles('mcpp.toml', '.xlings.json') }} - restore-keys: | - mcpp-sandbox-${{ runner.os }}-ci- - - # Cache xlings + its locally installed packages (xim:mcpp etc.). - # Saves the xlings bootstrap roundtrip + the mcpp xpkg download - # on hot runs. - - name: Cache xlings - uses: actions/cache@v4 - with: - path: ~/.xlings - key: xlings-${{ runner.os }}-v2-${{ hashFiles('.xlings.json') }} - restore-keys: | - xlings-${{ runner.os }}-v2- - - - name: Bootstrap mcpp via xlings - env: - XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '0.4.30' - run: | - # Always install the pinned version — cache may hold an older - # xlings whose sysroot/packages are incompatible. - tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" - curl -fsSL -o "/tmp/${tarball}" \ - "https://github.com/d2learn/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" - tar -xzf "/tmp/${tarball}" -C /tmp - "/tmp/xlings-${XLINGS_VERSION}-linux-x86_64/subos/default/bin/xlings" self install - export PATH="$HOME/.xlings/subos/default/bin:$PATH" - xlings --version - # xim:mcpp — `xlings install` is idempotent so cache hits skip - # the download. - xlings install mcpp -y - MCPP="$HOME/.xlings/subos/default/bin/mcpp" - test -x "$MCPP" - "$MCPP" --version - echo "MCPP=$MCPP" >> "$GITHUB_ENV" - echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" - - # Cache the build directory: precise key on src/ + manifest - # so a no-source-change run lands on a full hit. Layered - # restore-keys let mid-run partial hits keep BMI/dyndep state - # for proper incremental builds. - - name: Cache target/ (build artifacts + BMIs) - uses: actions/cache@v4 - with: - path: target - key: mcpp-target-${{ runner.os }}-${{ hashFiles('src/**', 'tests/**', 'mcpp.toml', 'mcpp.lock') }} - restore-keys: | - mcpp-target-${{ runner.os }}- + - uses: ./.github/actions/bootstrap-mcpp - name: Configure mirror + Build mcpp from source (self-host) run: | @@ -116,25 +63,55 @@ jobs: "$MCPP_FRESH" self config --mirror GLOBAL "$MCPP_FRESH" test - # NOTE: the e2e suite (tests/e2e/run_all.sh) moved to ci-linux-e2e.yml - # so it runs in parallel with this job. The toolchain matrix below no - # longer relies on e2e's toolchain warm-ups: the GCC build uses the - # mcpp.toml-pinned default (gcc@16.1.0, already in the sandbox from the - # self-host build above), and the musl `--target` build auto-installs - # gcc@16.1.0-musl on demand (cached across runs). + # A cold, from-scratch self-host build with the manifest-pinned GCC: the + # property `build-test` cannot cover, because it builds incrementally on a + # restored target/. `mcpp test` is deliberately NOT repeated here — it would + # be the same suite, same toolchain, same driver binary as `build-test`, + # differing only in incremental state. + toolchain-gcc: + name: "toolchain: gcc (cold self-host)" + runs-on: ubuntu-24.04 + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + - uses: ./.github/actions/bootstrap-mcpp - - name: Save freshly-built mcpp for toolchain tests + - name: Build mcpp from source (self-host) run: | - MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") - cp "$MCPP" /tmp/mcpp-fresh + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true + "$MCPP" self config --mirror GLOBAL 2>/dev/null || true + "$MCPP" build + MCPP_FRESH=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") + cp "$MCPP_FRESH" /tmp/mcpp-fresh echo "MCPP=/tmp/mcpp-fresh" >> "$GITHUB_ENV" - - name: "Toolchain: GCC — build mcpp + test" + - name: "Toolchain: GCC — cold rebuild with the PR binary" run: | "$MCPP" clean "$MCPP" build 2>&1 | tee build.log; grep -q "Resolved gcc@16.1.0" build.log - "$MCPP" test + # The two cheap legs share one runner: each is ~1 min after the warm build, + # so a runner apiece would cost more in setup than it saves in wall-clock. + toolchain-cross: + name: "toolchain: musl + llvm" + runs-on: ubuntu-24.04 + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + - uses: ./.github/actions/bootstrap-mcpp + + - name: Build mcpp from source (self-host) + run: | + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true + "$MCPP" self config --mirror GLOBAL 2>/dev/null || true + "$MCPP" build + MCPP_FRESH=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") + cp "$MCPP_FRESH" /tmp/mcpp-fresh + echo "MCPP=/tmp/mcpp-fresh" >> "$GITHUB_ENV" + + # Auto-installs gcc@16.1.0-musl on demand (cached across runs). - name: "Toolchain: musl-gcc — build mcpp (--target)" run: | "$MCPP" clean @@ -150,16 +127,33 @@ jobs: # Restore sed -i 's/^default = "llvm@20.1.7"/default = "gcc@16.1.0"/' mcpp.toml - # Integration: the mcpp built from THIS PR's source (the self-host binary, - # $MCPP = /tmp/mcpp-fresh) builds & runs a real external C++ project — - # xlings (openxlings/xlings ships its own mcpp.toml). MCPP_VENDORED_XLINGS - # only supplies the xlings package backend that mcpp resolves deps through. + # Integration: the mcpp built from THIS PR's source builds & runs a real + # external C++ project — xlings (openxlings/xlings ships its own mcpp.toml). + # MCPP_VENDORED_XLINGS only supplies the xlings package backend that mcpp + # resolves deps through. + integration-xlings: + name: "integration: mcpp builds & runs xlings" + runs-on: ubuntu-24.04 + timeout-minutes: 45 + env: + XLINGS_NON_INTERACTIVE: '1' + steps: + - uses: actions/checkout@v4 + - uses: ./.github/actions/bootstrap-mcpp + + - name: Build mcpp from source (self-host) + run: | + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + "$XLINGS_BIN" config --mirror GLOBAL 2>/dev/null || true + "$MCPP" self config --mirror GLOBAL 2>/dev/null || true + "$MCPP" build + MCPP_FRESH=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") + cp "$MCPP_FRESH" /tmp/mcpp-fresh + echo "MCPP=/tmp/mcpp-fresh" >> "$GITHUB_ENV" + - name: "Integration: mcpp builds & runs xlings (openxlings/xlings)" - env: - XLINGS_NON_INTERACTIVE: '1' run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - # $MCPP is the freshly self-hosted binary built from this PR. git clone --depth 1 --recurse-submodules \ https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src diff --git a/.github/workflows/ci-macos-e2e.yml b/.github/workflows/ci-macos-e2e.yml new file mode 100644 index 00000000..4c30b421 --- /dev/null +++ b/.github/workflows/ci-macos-e2e.yml @@ -0,0 +1,55 @@ +name: ci-macos-e2e + +# The e2e suite on macOS ARM64, split out of ci-macos.yml (where it was 3.4 of +# the job's 8.4 min) so it runs in parallel with the toolchain/integration +# work — same shape as ci-linux-e2e.yml / ci-windows-e2e.yml. +# +# Not sharded: the macOS runners finish the suite in ~3.5 min, which is under +# this workflow's setup+build floor, so a second runner would buy nothing. +# +# Paired workflows: ci-macos.yml, ci-linux-e2e.yml, ci-windows-e2e.yml. + +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + workflow_dispatch: + +concurrency: + group: ci-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + e2e: + name: e2e suite (macOS ARM64, self-host) + runs-on: macos-15 + timeout-minutes: 30 + # NOTE: no MCPP_VERBOSE — the e2e suite asserts mcpp's default quiet + # output (tests 48/53). + steps: + - uses: actions/checkout@v4 + - uses: ./.github/actions/setup-macos-llvm + + - name: Build mcpp from source (self-host) + run: | + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + "$MCPP" build + + - name: E2E suite + # Fail-fast on hung tests instead of burning the whole job budget. + # Per-test 600s timeout lives in run_all.sh. + timeout-minutes: 25 + run: | + MCPP=$(find target -path "*/bin/mcpp" | head -1) + MCPP=$(cd "$(dirname "$MCPP")" && pwd)/$(basename "$MCPP") + test -x "$MCPP" + export MCPP + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + test -x "$MCPP_VENDORED_XLINGS" + export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL + "$MCPP" self config --mirror "$MCPP_E2E_TOOLCHAIN_MIRROR" + "$MCPP" self config + # macOS default toolchain is LLVM + "$MCPP" toolchain default "llvm@${MCPP_LLVM_VER}" + bash tests/e2e/run_all.sh diff --git a/.github/workflows/ci-macos.yml b/.github/workflows/ci-macos.yml index f1d15f41..4290bbb7 100644 --- a/.github/workflows/ci-macos.yml +++ b/.github/workflows/ci-macos.yml @@ -2,6 +2,9 @@ name: ci-macos # macOS CI for mcpp — validates LLVM/Clang as the default macOS toolchain. # Tests the full xlings → LLVM → C++23 import std pipeline on macOS ARM64. +# +# The e2e suite runs in PARALLEL in ci-macos-e2e.yml (same setup via +# .github/actions/setup-macos-llvm), not tacked onto this job. on: push: @@ -19,9 +22,8 @@ jobs: name: macOS ARM64 — xlings LLVM end-to-end runs-on: macos-15 timeout-minutes: 30 - # NOTE: no MCPP_VERBOSE here — this job runs the e2e suite, which includes - # tests asserting mcpp's default quiet output (48/53). Verbose is forced only - # in the fresh-install workflows. + # NOTE: no MCPP_VERBOSE here — keep this job's output shape identical to + # ci-macos-e2e.yml, which asserts mcpp's default quiet output (48/53). steps: - uses: actions/checkout@v4 @@ -32,43 +34,7 @@ jobs: xcrun --show-sdk-path echo "SDK: $(xcrun --show-sdk-version)" - - name: Cache xlings - uses: actions/cache@v4 - with: - path: ~/.xlings - key: xlings-macos15-arm64-v3-${{ hashFiles('.xlings.json') }} - restore-keys: | - xlings-macos15-arm64-v3- - - - name: Bootstrap xlings - env: - XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '0.4.30' - run: | - WORK=$(mktemp -d) - tarball="xlings-${XLINGS_VERSION}-macosx-arm64.tar.gz" - curl -fsSL -o "${WORK}/${tarball}" \ - "https://github.com/d2learn/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" - tar -xzf "${WORK}/${tarball}" -C "${WORK}" - XLINGS_DIR="${WORK}/xlings-${XLINGS_VERSION}-macosx-arm64" - "$XLINGS_DIR/subos/default/bin/xlings" self install - export PATH="$HOME/.xlings/subos/default/bin:$PATH" - xlings --version - echo "PATH=$HOME/.xlings/subos/default/bin:$PATH" >> "$GITHUB_ENV" - - - name: Install LLVM via xlings - run: | - # latest-first: test binaries now link the toolchain's own libc++ - # (A1 root fix in flags.cppm), so new llvm releases are - # self-consistent — this job is the proof for each new default. - xlings install llvm -y || xlings install llvm@20.1.7 -y - # Verify clang++ is available - LLVM_ROOT=$(find "$HOME/.xlings" -path "*/xpkgs/xim-x-llvm/*/bin/clang++" | head -1 | xargs dirname | xargs dirname) - echo "LLVM_ROOT=$LLVM_ROOT" - ls "$LLVM_ROOT/bin/clang++" - "$LLVM_ROOT/bin/clang++" --version - echo "LLVM_ROOT=$LLVM_ROOT" >> "$GITHUB_ENV" - echo "CXX=$LLVM_ROOT/bin/clang++" >> "$GITHUB_ENV" + - uses: ./.github/actions/setup-macos-llvm - name: Inspect LLVM package structure run: | @@ -274,57 +240,6 @@ jobs: *) echo "FAIL: unexpected platform"; exit 1 ;; esac - - name: Bootstrap mcpp via xlings - run: | - # Same pattern as Linux CI: xlings install mcpp - xlings install mcpp -y - MCPP="$HOME/.xlings/subos/default/bin/mcpp" - test -x "$MCPP" - "$MCPP" --version - echo "MCPP=$MCPP" >> "$GITHUB_ENV" - echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" - - - name: Configure dev mcpp sandbox to reuse xlings LLVM - run: | - # Use EXACTLY the LLVM the install step resolved (env LLVM_ROOT) — - # a version-glob pick chose a stale cached 20.1.7 next to the - # freshly installed 22.1.8 and silently tested the wrong toolchain. - LLVM_PKG="$LLVM_ROOT" - test -d "$LLVM_PKG" - LLVM_VER=$(basename "$LLVM_PKG") - echo "MCPP_LLVM_VER=$LLVM_VER" >> "$GITHUB_ENV" - MCPP_LLVM_LINK="$HOME/.mcpp/registry/data/xpkgs/xim-x-llvm/$LLVM_VER" - printf '1\n' > "$LLVM_PKG/.mcpp_ok" - - mkdir -p "$HOME/.mcpp/registry/data/xpkgs/xim-x-llvm" - rm -rf "$MCPP_LLVM_LINK" - ln -s "$LLVM_PKG" "$MCPP_LLVM_LINK" - - mkdir -p "$HOME/.mcpp" - cat > "$HOME/.mcpp/config.toml" <> "$GITHUB_ENV" + + - name: E2E suite + shell: bash + # Fail-fast on hung tests instead of burning the whole job budget. + # Per-test 600s timeout lives in run_all.sh. + timeout-minutes: 25 + run: | + export MCPP="$MCPP_SELF" + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL + "$MCPP_SELF" self config --mirror GLOBAL + "$MCPP_SELF" toolchain default llvm@20.1.7 + bash tests/e2e/run_all.sh diff --git a/.github/workflows/ci-windows.yml b/.github/workflows/ci-windows.yml index 34c99f5e..96d33ea0 100644 --- a/.github/workflows/ci-windows.yml +++ b/.github/workflows/ci-windows.yml @@ -1,7 +1,16 @@ name: ci-windows # Windows CI for mcpp — same flow as Linux (ci-linux.yml) and macOS (ci-macos.yml): -# xlings install mcpp → self-host build → E2E → smoke → package +# xlings install mcpp → self-host build → smoke → package +# +# SHAPE: three INDEPENDENT jobs, no `needs:` between them; each restores the +# shared cache lineage (.github/actions/bootstrap-mcpp) and pays one warm +# `mcpp build` to get the PR's own binary. The e2e suite moved to +# ci-windows-e2e.yml (sharded ×2) — it was 9.7 of this job's 20.4 min. +# +# before: build → unit → xlings → stdin → e2e → toolchains → package ≈ 20 min +# after: max(build+unit+package, toolchains+integration) ≈ 8 min +# in parallel with ci-windows-e2e (≈ 8 min) on: push: @@ -14,76 +23,26 @@ concurrency: group: ci-windows-${{ github.ref }} cancel-in-progress: true +env: + MCPP_HOME: C:\Users\runneradmin\.mcpp + jobs: - build-and-test: - name: build + test (windows x64, self-host) + build-test: + name: build + test + package (windows x64, self-host) runs-on: windows-latest timeout-minutes: 45 - env: - MCPP_HOME: C:\Users\runneradmin\.mcpp - # NOTE: no MCPP_VERBOSE — this job runs the e2e suite, which asserts mcpp's - # default quiet output (48/53). Verbose is forced only in fresh-install. steps: - uses: actions/checkout@v4 - - - name: Cache mcpp sandbox - uses: actions/cache@v4 - with: - path: ~\.mcpp - # Disjoint from release.yml's "-release-" cache lineage (issue #120). - key: mcpp-sandbox-${{ runner.os }}-ci-${{ hashFiles('mcpp.toml', '.xlings.json') }} - restore-keys: | - mcpp-sandbox-${{ runner.os }}-ci- - - - name: Cache xlings - uses: actions/cache@v4 - with: - path: ~\.xlings - key: xlings-${{ runner.os }}-v2-${{ hashFiles('.xlings.json') }} - restore-keys: | - xlings-${{ runner.os }}-v2- - - - name: Bootstrap mcpp via xlings - shell: bash - env: - XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '0.4.30' - run: | - WORK=$(mktemp -d) - zipfile="xlings-${XLINGS_VERSION}-windows-x86_64.zip" - curl -fsSL -o "${WORK}/${zipfile}" \ - "https://github.com/d2learn/xlings/releases/download/v${XLINGS_VERSION}/${zipfile}" - cd "${WORK}" - unzip -q "${zipfile}" - "$WORK/xlings-${XLINGS_VERSION}-windows-x86_64/subos/default/bin/xlings.exe" self install - export PATH="$USERPROFILE/.xlings/subos/default/bin:$PATH" - echo "$USERPROFILE/.xlings/subos/default/bin" >> "$GITHUB_PATH" - xlings.exe --version - xlings.exe install mcpp -y - echo "=== Searching for mcpp binary ===" - find "$USERPROFILE/.xlings" -name "mcpp.exe" -o -name "mcpp" 2>/dev/null | head -10 - MCPP=$(find "$USERPROFILE/.xlings" -name "mcpp.exe" -path "*/bin/*" 2>/dev/null | head -1) - if [ -z "$MCPP" ]; then - MCPP=$(find "$USERPROFILE/.xlings" -name "mcpp" -path "*/bin/*" 2>/dev/null | head -1) - fi - test -n "$MCPP" || { echo "FAIL: mcpp not found after xlings install"; exit 1; } - echo "Found mcpp at: $MCPP" - "$MCPP" --version - echo "MCPP=$MCPP" >> "$GITHUB_ENV" - XLINGS_BIN=$(cygpath -w "$USERPROFILE/.xlings/subos/default/bin/xlings.exe") - echo "XLINGS_BIN=$XLINGS_BIN" >> "$GITHUB_ENV" + - uses: ./.github/actions/bootstrap-mcpp - name: Build mcpp from source (self-host) shell: bash run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" build - MCPP_SELF=$(find target -name "mcpp.exe" -path "*/bin/*" | head -1) test -n "$MCPP_SELF" || { echo "FAIL: no mcpp.exe"; exit 1; } MCPP_SELF=$(cd "$(dirname "$MCPP_SELF")" && pwd)/$(basename "$MCPP_SELF") - echo "Self-hosted binary: $MCPP_SELF" "$MCPP_SELF" --version echo "MCPP_SELF=$MCPP_SELF" >> "$GITHUB_ENV" @@ -93,6 +52,75 @@ jobs: export MCPP_VENDORED_XLINGS=$(cygpath -w "$USERPROFILE/.xlings/subos/default/bin/xlings.exe") "$MCPP_SELF" test + - name: Package Windows release zip + id: package + shell: bash + run: | + VERSION=$(awk -F '"' '/^version[[:space:]]*=/{print $2; exit}' mcpp.toml) + WRAPPER="mcpp-${VERSION}-windows-x86_64" + ZIPNAME="${WRAPPER}.zip" + + MCPP_BIN=$(find target -name "mcpp.exe" -path "*/bin/*" | head -1) + test -n "$MCPP_BIN" || { echo "FAIL: no mcpp.exe in target/"; exit 1; } + + STAGING=$(mktemp -d) + mkdir -p "$STAGING/$WRAPPER/bin" "$STAGING/$WRAPPER/registry/bin" + cp "$MCPP_BIN" "$STAGING/$WRAPPER/bin/mcpp.exe" + printf '@echo off\r\n"%%~dp0bin\\mcpp.exe" %%*\r\n' > "$STAGING/$WRAPPER/mcpp.bat" + cp README.md "$STAGING/$WRAPPER/" 2>/dev/null || true + cp LICENSE "$STAGING/$WRAPPER/" 2>/dev/null || true + XLINGS_EXE="$USERPROFILE/.xlings/subos/default/bin/xlings.exe" + [ -f "$XLINGS_EXE" ] && cp "$XLINGS_EXE" "$STAGING/$WRAPPER/registry/bin/xlings.exe" + + mkdir -p dist + (cd "$STAGING" && 7z a -tzip "$ZIPNAME" "$WRAPPER") + cp "$STAGING/$ZIPNAME" "dist/$ZIPNAME" + (cd dist && sha256sum "$ZIPNAME" > "$ZIPNAME.sha256") + echo "zipname=$ZIPNAME" >> "$GITHUB_OUTPUT" + ls -la dist/ + + - name: Smoke-test the packaged zip + shell: bash + run: | + ZIPNAME="${{ steps.package.outputs.zipname }}" + WRAPPER="${ZIPNAME%.zip}" + SMOKE=$(mktemp -d) + (cd "$SMOKE" && unzip -q "$GITHUB_WORKSPACE/dist/$ZIPNAME") + "$SMOKE/$WRAPPER/bin/mcpp.exe" --version + test -f "$SMOKE/$WRAPPER/registry/bin/xlings.exe" + test -f "$SMOKE/$WRAPPER/mcpp.bat" + echo "Smoke-test passed" + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: mcpp-windows-x86_64 + path: | + dist/*.zip + dist/*.sha256 + + # Everything that needs a toolchain other than the default, plus the two + # Windows-specific behavioural regressions. Kept on one runner: each leg is + # seconds-to-2-minutes, so per-leg runners would cost more setup than they + # save. + toolchains: + name: "toolchains + regressions (windows x64)" + runs-on: windows-latest + timeout-minutes: 45 + steps: + - uses: actions/checkout@v4 + - uses: ./.github/actions/bootstrap-mcpp + + - name: Build mcpp from source (self-host) + shell: bash + run: | + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + "$MCPP" build + MCPP_SELF=$(find target -name "mcpp.exe" -path "*/bin/*" | head -1) + test -n "$MCPP_SELF" || { echo "FAIL: no mcpp.exe"; exit 1; } + MCPP_SELF=$(cd "$(dirname "$MCPP_SELF")" && pwd)/$(basename "$MCPP_SELF") + echo "MCPP_SELF=$MCPP_SELF" >> "$GITHUB_ENV" + # Integration: the mcpp built from THIS PR's source ($MCPP_SELF, the # self-hosted binary) builds & runs a real external C++ project — xlings # (openxlings/xlings ships its own mcpp.toml). MCPP_VENDORED_XLINGS only @@ -103,7 +131,6 @@ jobs: XLINGS_NON_INTERACTIVE: '1' run: | export MCPP_VENDORED_XLINGS=$(cygpath -w "$USERPROFILE/.xlings/subos/default/bin/xlings.exe") - # $MCPP_SELF is the freshly self-hosted mcpp built from this PR. git clone --depth 1 --recurse-submodules \ https://github.com/openxlings/xlings /tmp/xlings-src cd /tmp/xlings-src @@ -197,19 +224,6 @@ jobs: Write-Host 'SUCCESS: mcpp completes with open-empty stdin → Windows seal_stdin fix verified.' - - name: E2E suite - shell: bash - # See ci-linux.yml — fail-fast on hung tests instead of burning the - # whole job budget. Per-test 600s timeout lives in run_all.sh. - timeout-minutes: 25 - run: | - export MCPP="$MCPP_SELF" - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL - "$MCPP_SELF" self config --mirror GLOBAL - "$MCPP_SELF" toolchain default llvm@20.1.7 - bash tests/e2e/run_all.sh - - name: "Toolchain: LLVM — mcpp new → run" shell: bash run: | @@ -270,50 +284,3 @@ jobs: "$MCPP" toolchain default llvm@20.1.7 "$MCPP" clean --bmi-cache "$MCPP" build 2>&1 | tee build.log; grep -q "Resolved llvm@20.1.7" build.log - - - name: Package Windows release zip - id: package - shell: bash - run: | - VERSION=$(awk -F '"' '/^version[[:space:]]*=/{print $2; exit}' mcpp.toml) - WRAPPER="mcpp-${VERSION}-windows-x86_64" - ZIPNAME="${WRAPPER}.zip" - - MCPP_BIN=$(find target -name "mcpp.exe" -path "*/bin/*" | head -1) - test -n "$MCPP_BIN" || { echo "FAIL: no mcpp.exe in target/"; exit 1; } - - STAGING=$(mktemp -d) - mkdir -p "$STAGING/$WRAPPER/bin" "$STAGING/$WRAPPER/registry/bin" - cp "$MCPP_BIN" "$STAGING/$WRAPPER/bin/mcpp.exe" - printf '@echo off\r\n"%%~dp0bin\\mcpp.exe" %%*\r\n' > "$STAGING/$WRAPPER/mcpp.bat" - cp README.md "$STAGING/$WRAPPER/" 2>/dev/null || true - cp LICENSE "$STAGING/$WRAPPER/" 2>/dev/null || true - XLINGS_EXE="$USERPROFILE/.xlings/subos/default/bin/xlings.exe" - [ -f "$XLINGS_EXE" ] && cp "$XLINGS_EXE" "$STAGING/$WRAPPER/registry/bin/xlings.exe" - - mkdir -p dist - (cd "$STAGING" && 7z a -tzip "$ZIPNAME" "$WRAPPER") - cp "$STAGING/$ZIPNAME" "dist/$ZIPNAME" - (cd dist && sha256sum "$ZIPNAME" > "$ZIPNAME.sha256") - echo "zipname=$ZIPNAME" >> "$GITHUB_OUTPUT" - ls -la dist/ - - - name: Smoke-test the packaged zip - shell: bash - run: | - ZIPNAME="${{ steps.package.outputs.zipname }}" - WRAPPER="${ZIPNAME%.zip}" - SMOKE=$(mktemp -d) - (cd "$SMOKE" && unzip -q "$GITHUB_WORKSPACE/dist/$ZIPNAME") - "$SMOKE/$WRAPPER/bin/mcpp.exe" --version - test -f "$SMOKE/$WRAPPER/registry/bin/xlings.exe" - test -f "$SMOKE/$WRAPPER/mcpp.bat" - echo "Smoke-test passed" - - - name: Upload artifact - uses: actions/upload-artifact@v4 - with: - name: mcpp-windows-x86_64 - path: | - dist/*.zip - dist/*.sha256 diff --git a/tests/e2e/run_all.sh b/tests/e2e/run_all.sh index cb1f954a..f4147038 100755 --- a/tests/e2e/run_all.sh +++ b/tests/e2e/run_all.sh @@ -197,7 +197,36 @@ FAILED_TESTS=() TIMED_OUT_TESTS=() TIMINGS=() # " " per executed test, for the slowest-first report +# --------------------------------------------------------------------------- +# Optional sharding: E2E_SHARD="/" (1-based) +# --------------------------------------------------------------------------- +# Splits the suite across parallel CI runners. Round-robin over the sorted +# file list rather than contiguous ranges: durations are wildly uneven (the +# timing report below exists precisely because of that), and interleaving +# spreads the long poles instead of stacking them in one shard. +# +# The slice is computed on the FULL file list, before capability filtering, +# so a shard's membership does not depend on what the host can run. +SHARD_IDX=1 +SHARD_TOTAL=1 +if [[ -n "${E2E_SHARD:-}" ]]; then + SHARD_IDX="${E2E_SHARD%%/*}" + SHARD_TOTAL="${E2E_SHARD##*/}" + if ! [[ "$SHARD_IDX" =~ ^[0-9]+$ && "$SHARD_TOTAL" =~ ^[0-9]+$ ]] \ + || (( SHARD_TOTAL < 1 || SHARD_IDX < 1 || SHARD_IDX > SHARD_TOTAL )); then + echo "FATAL: bad E2E_SHARD='$E2E_SHARD' (want /, 1-based)" + exit 1 + fi + echo "Shard: ${SHARD_IDX}/${SHARD_TOTAL} (round-robin over the suite)" +fi +SHARD_POS=0 + for test in "$HERE"/[0-9]*.sh; do + if (( SHARD_TOTAL > 1 )); then + _mine=$(( (SHARD_POS % SHARD_TOTAL) + 1 )) + SHARD_POS=$(( SHARD_POS + 1 )) + (( _mine == SHARD_IDX )) || continue + fi name="$(basename "$test")" echo missing_cap="$(check_requires "$test")"