Skip to content

Commit 204fddf

Browse files
authored
feat(pkgs): add public ffmpeg module package 0.0.1 (Form A) + house compat.ffmpeg pipeline (#75)
* chore(tools): house the compat.ffmpeg descriptor pipeline (from ffmpeg-m) tools/compat-ffmpeg/{fetch_upstream.sh,gen_config.sh,gen_descriptor.py} move in from ffmpeg-m so the generator lives next to the descriptor it generates (ffmpeg-m keeps only the module layer + export-surface generator). Migration regression: pipeline re-run here reproduces pkgs/c/compat.ffmpeg.lua byte-identically except the two regenerate-pointer comment lines (included). * feat(pkgs): add public ffmpeg module package 0.0.1 (Form A, linux) C++23 module layer over FFmpeg (import ffmpeg.av; C API names unchanged), sources arriving via its compat.ffmpeg dependency (full source build). GitHub tag tarball + byte-identical gitcode mcpp-res/ffmpeg CN mirror. Validated like imgui-module (default-namespace exception): new tests/smoke_ffmpeg_module.sh + ffmpeg-module job, linux-only. Local: smoke passes with mcpp 0.0.96 against the published tarball.
1 parent 942a269 commit 204fddf

9 files changed

Lines changed: 486 additions & 4 deletions

File tree

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# add `ffmpeg`(公开模块包,Form A)+ 收编 compat.ffmpeg 描述符流水线
2+
3+
日期:2026-07-18 · 关联:pkgs/f/ffmpeg.lua、tools/compat-ffmpeg/、tests/smoke_ffmpeg_module.sh
4+
前置:#74(compat.ffmpeg 8.1.2 全源码直编)已合并;ffmpeg-m v0.0.1 已发布 tag。
5+
6+
## 形态判定
7+
8+
- **Form A 原生模块库**(同 `imgui`):上游 [mcpplibs/ffmpeg-m](https://github.com/mcpplibs/ffmpeg-m)
9+
自带 `mcpp.toml`,描述符只声明元数据 + tag tarball 下载地址,默认查找
10+
`<verdir>/*/mcpp.toml`
11+
- 包内容 = 纯 C++23 模块层(`import ffmpeg.av;` 及 7 个分库模块),FFmpeg 全部
12+
源码经其 `compat.ffmpeg = "8.1.2"` 依赖到达消费者(全源码直编形态,零预构建)。
13+
C API 名称与用法完全不变。
14+
- **linux-only**(先例 `compat.x11`):compat.ffmpeg 目前仅有 linux-x86_64 configure
15+
快照;macOS 阻塞于 mcpp#229(依赖侧 cfg 条件源)。
16+
- license:模块层 MIT;上游经 compat.ffmpeg 为 LGPL-2.1-or-later(无 GPL/nonfree 组件)。
17+
18+
## CN 镜像
19+
20+
gitcode `mcpp-res/ffmpeg`(与 compat 上游 tarball 同仓,tag 不冲突):
21+
release `v0.0.1`,资产 `ffmpeg-m-0.0.1.tar.gz` = GitHub
22+
`archive/refs/tags/v0.0.1.tar.gz` 同字节,`gtc release publish` 上传,sha256 一致。
23+
24+
## 验证方式(imgui-module 先例)
25+
26+
默认命名空间(namespace = "")的包尚不能作为 workspace 成员指向本地 path index,
27+
故沿用 `imgui-module` 的例外通道:`tests/smoke_ffmpeg_module.sh` 重播默认 index
28+
后以 `[dependencies] ffmpeg = "0.0.1"` 消费,gcc@16.1.0,断言 libavutil 主版本、
29+
4 个解码器存在、demuxer 数 > 300。仅 Linux(非 Linux 直接 SKIP)。
30+
smoke 内先 `mcpp index update` 刷新沙箱包索引(nasm 自举,mcpp#232 规避;置于
31+
重播默认 index 之前,避免刷新覆盖重播结果)。validate.yml 新增 `ffmpeg-module`
32+
job(ubuntu-only,镜像 imgui-module 结构)。
33+
34+
## 描述符流水线收编(tools/compat-ffmpeg/)
35+
36+
`compat.ffmpeg.lua` 为生成物(329 KB,禁止手改)。生成器原在 ffmpeg-m
37+
(`tools/gen_config.sh`/`gen_descriptor.py`),随本次 PR 迁至本仓
38+
`tools/compat-ffmpeg/`(fetch_upstream.sh + gen_config.sh + gen_descriptor.py),
39+
与其生成的描述符同仓维护;ffmpeg-m 只保留模块层与导出面生成器(gen_exports.py)。
40+
迁移回归:迁移后重跑流水线,与已合并描述符 diff 仅头注释 2 行(2281 源 + 15
41+
生成文件字节一致;host gcc 13 与原生成环境结果一致,可复现)。
42+
注意:宿主 `cc` 若被 xlings subos 切到交叉工具链会导致 configure 失败,用
43+
`CC=/usr/bin/gcc` 显式指定。
44+
45+
## FFmpeg 版本升级 SOP
46+
47+
1. 本仓:改 `tools/compat-ffmpeg/fetch_upstream.sh` 的 version/sha256 pin →
48+
`sh tools/compat-ffmpeg/gen_config.sh` 重生成描述符 → 上传新上游 tarball 到
49+
CN 镜像 → PR。
50+
2. ffmpeg-m:bump `compat.ffmpeg` 依赖 + `tools/fetch_upstream.sh` pin →
51+
`python3 tools/gen_exports.py` 重生成导出面 → 审阅 diff → 发新 tag →
52+
本仓登记新版本条目。

.github/workflows/validate.yml

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -354,3 +354,40 @@ jobs:
354354
run: |
355355
"$MCPP" --version
356356
timeout 1800 bash tests/smoke_imgui_module.sh
357+
358+
# ── Exception: the public `ffmpeg` C++23-module package (namespace "") ──
359+
# Same default-namespace limitation as imgui-module above — validated via a
360+
# reseeding smoke driver instead of a workspace member. Linux-only: its
361+
# compat.ffmpeg dependency ships a linux-x86_64 configure snapshot (macOS is
362+
# blocked on mcpp#229 dependency cfg-conditional sources).
363+
ffmpeg-module:
364+
runs-on: ubuntu-latest
365+
timeout-minutes: 60
366+
env:
367+
MCPP_EFFECTIVE: "0.0.96" # keep in sync with env.MCPP_VERSION
368+
steps:
369+
- uses: actions/checkout@v4
370+
- name: Restore mcpp registry cache
371+
uses: actions/cache@v4
372+
with:
373+
path: ~/.mcpp/registry
374+
key: mcpp-registry-${{ runner.os }}-${{ env.MCPP_EFFECTIVE }}-${{ hashFiles('pkgs/**/*.lua', 'tests/**', '.github/workflows/validate.yml') }}
375+
restore-keys: |
376+
mcpp-registry-${{ runner.os }}-${{ env.MCPP_EFFECTIVE }}-
377+
- name: Download mcpp
378+
run: |
379+
curl -L -fsS -o mcpp.tar.gz \
380+
"https://github.com/mcpp-community/mcpp/releases/download/v${MCPP_VERSION}/mcpp-${MCPP_VERSION}-linux-x86_64.tar.gz"
381+
tar -xzf mcpp.tar.gz
382+
root="$PWD/mcpp-${MCPP_VERSION}-linux-x86_64"
383+
mkdir -p "$HOME/.mcpp/registry"
384+
cp -a "$root/registry/." "$HOME/.mcpp/registry/"
385+
echo "MCPP=$root/bin/mcpp" >> "$GITHUB_ENV"
386+
echo "MCPP_VENDORED_XLINGS=$root/registry/bin/xlings" >> "$GITHUB_ENV"
387+
echo "$root/bin" >> "$GITHUB_PATH"
388+
- name: ffmpeg module package smoke
389+
env:
390+
MCPP_INDEX_MIRROR: GLOBAL
391+
run: |
392+
"$MCPP" --version
393+
timeout 2700 bash tests/smoke_ffmpeg_module.sh

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ mcpp self config --mirror CN # 切换至国内镜像,默认使用 GLOBAL 上
2222

2323
本仓收录两类包:
2424

25-
- **原生 mcpp 模块库**:以 C++23 模块发布、`import` 即用,包括 `mcpplibs.*``nlohmann.json``imgui`,以及由
25+
- **原生 mcpp 模块库**:以 C++23 模块发布、`import` 即用,包括 `mcpplibs.*``nlohmann.json``imgui``ffmpeg`,以及由
2626
用户基于 mcpp 开发并登记进索引的库(如 `tensorvia-cpu`)。其上游通常自带 `mcpp.toml`,描述文件(Form A)只声明
2727
元数据与下载地址。
2828
- **第三方 C/C++ 库(`compat`)**:其上游不提供 mcpp 支持,描述文件(Form B)内联构建信息。该类库存在
@@ -32,7 +32,7 @@ mcpp self config --mirror CN # 切换至国内镜像,默认使用 GLOBAL 上
3232

3333
| 形态 | 示例 |
3434
|------|------|
35-
| 原生模块库(Form A) | [`mcpplibs.xpkg`](pkgs/x/xpkg.lua) · [`mcpplibs.tinyhttps`](pkgs/t/tinyhttps.lua) · [`tensorvia-cpu`](pkgs/t/tensorvia-cpu.lua) |
35+
| 原生模块库(Form A) | [`mcpplibs.xpkg`](pkgs/x/xpkg.lua) · [`mcpplibs.tinyhttps`](pkgs/t/tinyhttps.lua) · [`tensorvia-cpu`](pkgs/t/tensorvia-cpu.lua) · [`ffmpeg`](pkgs/f/ffmpeg.lua)(模块层,源码经 `compat.ffmpeg` 直编) |
3636
| C 源码 compat(含 `features`) | [`compat.cjson`](pkgs/c/compat.cjson.lua) · [`compat.zlib`](pkgs/c/compat.zlib.lua) |
3737
| header-only(含 `features`) | [`compat.eigen`](pkgs/c/compat.eigen.lua) |
3838
| 外部构建系统(`install()` 从源码构建) | [`compat.openblas`](pkgs/c/compat.openblas.lua)(Make) · [`compat.opencv`](pkgs/c/compat.opencv.lua)(CMake) |

pkgs/c/compat.ffmpeg.lua

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
1-
-- Auto-generated by ffmpeg-m/tools/gen_descriptor.py — do not edit by hand.
1+
-- Auto-generated by tools/compat-ffmpeg/gen_descriptor.py — do not edit by hand.
22
-- Recipe: FFmpeg 8.1.2 built FROM SOURCE by mcpp (no configure/make at
33
-- build time). Full internal-component hermetic profile, LGPL-2.1+ (configure:
44
-- --disable-autodetect --disable-programs --disable-doc; everything else
55
-- default, x86 NASM asm included — 157 .asm units, nasm resolved
66
-- PATH → sandbox → xlings). The config snapshot below is deterministic for
77
-- (linux-x86_64, this profile) and shipped as generated_files; the source
88
-- list is the `make -n` ground truth of the same configure run.
9-
-- Regenerate: ffmpeg-m tools/gen_descriptor.py (see that repo's docs).
9+
-- Regenerate: sh tools/compat-ffmpeg/gen_config.sh (this repo).
1010
package = {
1111
spec = "1",
1212
namespace = "compat",

pkgs/f/ffmpeg.lua

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
-- Form A descriptor: the public ffmpeg module package ships its own
2+
-- mcpp.toml. mcpp's default lookup finds <verdir>/*/mcpp.toml inside
3+
-- the GitHub source tarball wrap.
4+
--
5+
-- The package is the thin C++23 module layer (import ffmpeg.av / per-lib
6+
-- ffmpeg.avcodec, …) over FFmpeg's unchanged C API; the FFmpeg sources
7+
-- themselves arrive through its compat.ffmpeg dependency (full source
8+
-- build, config snapshot — see pkgs/c/compat.ffmpeg.lua). Linux-only for
9+
-- now: compat.ffmpeg carries a linux-x86_64 configure snapshot (macOS
10+
-- blocked on mcpp#229 dependency cfg-conditional sources).
11+
--
12+
package = {
13+
spec = "1",
14+
name = "ffmpeg",
15+
namespace = "",
16+
description = "C++23 module package for FFmpeg (import ffmpeg.av) — full source build via compat.ffmpeg, C API unchanged",
17+
licenses = {"MIT"}, -- module layer; upstream via compat.ffmpeg is LGPL-2.1-or-later
18+
repo = "https://github.com/mcpplibs/ffmpeg-m",
19+
type = "package",
20+
21+
xpm = {
22+
linux = {
23+
["0.0.1"] = {
24+
url = {
25+
GLOBAL = "https://github.com/mcpplibs/ffmpeg-m/archive/refs/tags/v0.0.1.tar.gz",
26+
CN = "https://gitcode.com/mcpp-res/ffmpeg/releases/download/v0.0.1/ffmpeg-m-0.0.1.tar.gz",
27+
},
28+
sha256 = "e47dc9683f62466e9bd1c962a5f6ee83fbe3670fdb20c834bd9e8ee3f30e49fb",
29+
},
30+
},
31+
},
32+
33+
-- (no `mcpp` field -- default lookup will find <verdir>/*/mcpp.toml)
34+
}

tests/smoke_ffmpeg_module.sh

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
#!/usr/bin/env bash
2+
# Smoke-test the public ffmpeg module package through this checkout as a local
3+
# mcpp path index. This validates user-facing import-only consumption:
4+
# `[dependencies] ffmpeg = "0.0.1"` → `import ffmpeg.av;`. Linux-only (the
5+
# package's compat.ffmpeg dependency ships a linux-x86_64 config snapshot).
6+
set -euo pipefail
7+
8+
if [[ "$(uname -s)" != "Linux" ]]; then
9+
echo "SKIP: ffmpeg module package is linux-only for now."
10+
exit 0
11+
fi
12+
13+
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
14+
MCPP_BIN="${MCPP:-}"
15+
if [[ -z "$MCPP_BIN" ]]; then
16+
MCPP_BIN="$(command -v mcpp || true)"
17+
fi
18+
if [[ -z "$MCPP_BIN" || ! -x "$MCPP_BIN" ]]; then
19+
echo "FATAL: set MCPP=/path/to/mcpp or put mcpp on PATH" >&2
20+
exit 1
21+
fi
22+
23+
TMP="$(mktemp -d)"
24+
if [[ "${MCPP_INDEX_KEEP_SMOKE_TMP:-0}" == "1" ]]; then
25+
echo "KEEP: $TMP"
26+
else
27+
trap 'rm -rf "$TMP"' EXIT
28+
fi
29+
30+
if [[ -n "${MCPP_INDEX_SMOKE_MCPP_HOME:-}" ]]; then
31+
export MCPP_HOME="$MCPP_INDEX_SMOKE_MCPP_HOME"
32+
else
33+
export MCPP_HOME="$TMP/mcpp-home"
34+
fi
35+
mkdir -p "$MCPP_HOME/registry/data/xpkgs"
36+
37+
USER_MCPP="${HOME}/.mcpp"
38+
link_xpkgs() {
39+
local src="$1"
40+
[[ -d "$src" ]] || return 0
41+
find "$src" -mindepth 1 -maxdepth 1 -type d | while read -r pkg; do
42+
ln -s "$pkg" "$MCPP_HOME/registry/data/xpkgs/$(basename "$pkg")" 2>/dev/null || true
43+
done
44+
}
45+
link_xpkgs "${MCPP_INDEX_SMOKE_XPKGS_DIR:-}"
46+
link_xpkgs "$USER_MCPP/registry/data/xpkgs"
47+
if [[ -d "$USER_MCPP/registry/data/xim-pkgindex" ]]; then
48+
mkdir -p "$MCPP_HOME/registry/data/xim-pkgindex"
49+
cp -a "$USER_MCPP/registry/data/xim-pkgindex/." "$MCPP_HOME/registry/data/xim-pkgindex/" 2>/dev/null || true
50+
rm -f "$MCPP_HOME/registry/data/xim-pkgindex/.xlings-index-cache.json"
51+
fi
52+
if [[ -d "$USER_MCPP/registry/bin" ]]; then
53+
mkdir -p "$MCPP_HOME/registry"
54+
ln -s "$USER_MCPP/registry/bin" "$MCPP_HOME/registry/bin" 2>/dev/null || true
55+
fi
56+
if [[ -f "$USER_MCPP/config.toml" ]]; then
57+
cp -f "$USER_MCPP/config.toml" "$MCPP_HOME/config.toml" 2>/dev/null || true
58+
fi
59+
60+
# compat.ffmpeg carries NASM .asm sources; mcpp bootstraps nasm from the
61+
# sandbox package index. Release archives vendor a pre-xim-pkgindex#398
62+
# snapshot whose nasm.lua "installs" empty (mcpp#232) — refresh BEFORE
63+
# reseeding the default index below so the update can't clobber the reseed.
64+
"$MCPP_BIN" index update
65+
66+
default_index="$MCPP_HOME/registry/data/mcpplibs"
67+
# Reseed cleanly (see smoke_imgui_module.sh for why .git is skipped).
68+
rm -rf "$default_index"
69+
mkdir -p "$default_index"
70+
( cd "$ROOT" && find . -mindepth 1 -maxdepth 1 ! -name .git -exec cp -a {} "$default_index/" \; )
71+
rm -f "$default_index/.xlings-index-cache.json"
72+
printf 'ok\n' > "$default_index/.mcpp-index-updated"
73+
74+
"$MCPP_BIN" self config --mirror "${MCPP_INDEX_MIRROR:-GLOBAL}" >/dev/null
75+
76+
mkdir -p "$TMP/ffmpeg-module-smoke/src"
77+
cd "$TMP/ffmpeg-module-smoke"
78+
cat > mcpp.toml <<EOF
79+
[package]
80+
name = "ffmpeg-module-smoke"
81+
version = "0.1.0"
82+
83+
[toolchain]
84+
default = "${MCPP_INDEX_FFMPEG_MODULE_TOOLCHAIN:-gcc@16.1.0}"
85+
86+
[dependencies]
87+
ffmpeg = "0.0.1"
88+
89+
[targets.ffmpeg-module-smoke]
90+
kind = "bin"
91+
main = "src/main.cpp"
92+
EOF
93+
94+
cat > src/main.cpp <<'EOF'
95+
import std;
96+
import ffmpeg.av;
97+
98+
int main() {
99+
unsigned v { avutil_version() };
100+
std::println("libavutil {}.{} module package ok", v >> 16, (v >> 8) & 0xff);
101+
if ((v >> 16) != 60u) { // libavutil major of the FFmpeg 8.1.x train
102+
return 1;
103+
}
104+
105+
for (auto name : { "h264", "hevc", "av1", "aac" }) {
106+
const AVCodec* dec { avcodec_find_decoder_by_name(name) };
107+
if (dec == nullptr) {
108+
std::println("decoder {} MISSING", name);
109+
return 2;
110+
}
111+
}
112+
113+
void* it { nullptr };
114+
long demuxers { 0 };
115+
while (av_demuxer_iterate(&it)) ++demuxers;
116+
std::println("{} demuxers registered", demuxers);
117+
return demuxers > 300 ? 0 : 3;
118+
}
119+
EOF
120+
121+
"$MCPP_BIN" build
122+
"$MCPP_BIN" run
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/sh
2+
# fetch_upstream.sh — download + extract the pinned official FFmpeg tarball
3+
# for maintainer-time regeneration (gen_config/gen_exports/gen_descriptor).
4+
# Consumers never need this: they get FFmpeg via the compat.ffmpeg package.
5+
#
6+
# Prints the extracted source root on stdout.
7+
set -eu
8+
9+
version="${FFMPEG_VERSION:-8.1.2}"
10+
sha256="${FFMPEG_SHA256:-32faba5ef67340d54724941eae1425580791195312a4fd13bf6f820a2818bf22}"
11+
cache="${XDG_CACHE_HOME:-$HOME/.cache}/ffmpeg-m"
12+
tarball="$cache/ffmpeg-$version.tar.gz"
13+
srcdir="$cache/ffmpeg-$version"
14+
15+
mkdir -p "$cache"
16+
if [ ! -f "$tarball" ]; then
17+
curl -L -fsS -o "$tarball" "https://ffmpeg.org/releases/ffmpeg-$version.tar.gz" >&2
18+
fi
19+
echo "$sha256 $tarball" | sha256sum -c - >&2
20+
if [ ! -d "$srcdir" ]; then
21+
tar -xzf "$tarball" -C "$cache"
22+
fi
23+
echo "$srcdir"

tools/compat-ffmpeg/gen_config.sh

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
#!/bin/sh
2+
# gen_config.sh — maintainer-time descriptor pipeline for compat.ffmpeg.
3+
#
4+
# Fetches the pinned OFFICIAL FFmpeg tarball (fetch_upstream.sh alongside), runs
5+
# ./configure ONCE (out-of-tree, hermetic full profile), captures every
6+
# generated file plus the `make -n` source list, and emits the mcpp-index
7+
# descriptor via gen_descriptor.py alongside. Consumers never run this: FFmpeg
8+
# reaches them as the compat.ffmpeg package.
9+
#
10+
# Usage: tools/compat-ffmpeg/gen_config.sh [target] [out.lua]
11+
# target default: autodetected <os>-<arch> (only linux-x86_64 supported yet)
12+
# out.lua default: <this repo>/pkgs/c/compat.ffmpeg.lua
13+
set -eu
14+
15+
target="${1:-$(uname -s | tr 'A-Z' 'a-z' | sed s/darwin/macos/)-$(uname -m | sed s/arm64/aarch64/)}"
16+
here="$(cd "$(dirname "$0")" && pwd)"
17+
index_root="$(cd "$here/../.." && pwd)"
18+
version="${FFMPEG_VERSION:-8.1.2}"
19+
sha256="${FFMPEG_SHA256:-32faba5ef67340d54724941eae1425580791195312a4fd13bf6f820a2818bf22}"
20+
default_out="$index_root/pkgs/c/compat.ffmpeg.lua"
21+
out="${2:-$default_out}"
22+
23+
src="$(sh "$here/fetch_upstream.sh")"
24+
bld="$(mktemp -d /tmp/ffmpeg-m-cfg.XXXXXX)"
25+
trap 'rm -rf "$bld"' EXIT
26+
27+
# Full-feature hermetic profile: every INTERNAL component on; nothing probed
28+
# from the host (--disable-autodetect) so the snapshot is reproducible and the
29+
# consumer build needs no system libraries. programs/doc are irrelevant to a
30+
# library package.
31+
cd "$bld"
32+
"$src/configure" \
33+
--cc="${CC:-cc}" \
34+
--disable-autodetect \
35+
--disable-programs \
36+
--disable-doc
37+
38+
# ffversion.h is normally a make-time product; generate it here with a RELATIVE
39+
# output path (an absolute path would leak into the include guard).
40+
sh "$src/ffbuild/version.sh" "$src" libavutil/ffversion.h
41+
42+
# The `make -n` dry run is the ground truth for which .c/.S/.asm files this
43+
# frozen configuration compiles (CONFIG_* gating + _select closures resolved).
44+
make -n > make-n.log 2>/dev/null || true
45+
grep -oE '(\.\./[^ ]+|src)/[A-Za-z0-9_/.+-]+\.(c|S|asm)\b' make-n.log \
46+
| sed "s|^\.\./ffmpeg-$version/|src/|; s|^$src/|src/|" \
47+
| grep '^src/' | sort -u > sources.txt
48+
49+
python3 "$here/gen_descriptor.py" "$bld" "$version" "$sha256" "$out"
50+
echo "descriptor written: $out (configured on: $target)"

0 commit comments

Comments
 (0)