What happened
A root-level container: Containerfile declaration whose Containerfile ends in COPY . . made every lc run / lc status invocation sha256 the entire project tree as docker build context before doing anything else: ~50 min wall and ~800 GB read per invocation on a large research project (docs sites, paper extractions, code worktrees). _COPY_DIR_EXCLUDE only skips .git/results/caches, and the hash is recomputed from scratch every time.
The second-order effect is worse: the computed image tag folds into every output's code_version, so any new file anywhere in the hashed tree flips the entire project stale — we carried 90+ phantom-stale outputs for weeks (every run log or paper download re-triggered it) before root-causing. In our case the declared image was never even executed (recipes exec an apptainer sandbox by path; runtime resolves to none on this cluster), so the 50-minute tax paid for provenance of an image that never ran.
Reproduction
- Project with
container: Containerfile, Containerfile containing COPY . ., and a multi-GB tree outside the built-in excludes (e.g. docs/, papers/).
lc run -v <anything> — observe a long silent gap after the "Scratch root" line; /proc/<pid>/io shows the tree being read.- Touch any file in the tree;
lc status — all outputs flip stale via the changed image tag in code_version.
Suggestions
- Cache the build-context hash keyed on an (mtime, size) file manifest; rehash only changed files.
- Honor
.dockerignore / .containerignore.
- Warn loudly when the build context exceeds a sanity size (the 50-minute hash was completely silent).
- Consider excluding the image tag from
code_version when the resolved runtime is none — the image cannot affect materialization semantics if it never executes.
Environment
- lightcone-cli: 0.3.8.dev25+ge403f216d
- Python: 3.14 (uv tool env) / project host Python 3.12.13
- OS: Linux 4.18.0 (Leonardo, CINECA)
— Claude (Fable) on behalf of Cail
What happened
A root-level
container: Containerfiledeclaration whose Containerfile ends inCOPY . .made everylc run/lc statusinvocation sha256 the entire project tree as docker build context before doing anything else: ~50 min wall and ~800 GB read per invocation on a large research project (docs sites, paper extractions, code worktrees)._COPY_DIR_EXCLUDEonly skips.git/results/caches, and the hash is recomputed from scratch every time.The second-order effect is worse: the computed image tag folds into every output's
code_version, so any new file anywhere in the hashed tree flips the entire project stale — we carried 90+ phantom-stale outputs for weeks (every run log or paper download re-triggered it) before root-causing. In our case the declared image was never even executed (recipes exec an apptainer sandbox by path; runtime resolves tononeon this cluster), so the 50-minute tax paid for provenance of an image that never ran.Reproduction
container: Containerfile, Containerfile containingCOPY . ., and a multi-GB tree outside the built-in excludes (e.g.docs/,papers/).lc run -v <anything>— observe a long silent gap after the "Scratch root" line;/proc/<pid>/ioshows the tree being read.lc status— all outputs flip stale via the changed image tag incode_version.Suggestions
.dockerignore/.containerignore.code_versionwhen the resolved runtime isnone— the image cannot affect materialization semantics if it never executes.Environment
— Claude (Fable) on behalf of Cail