Introduce fastly-compute-py build tool#31
Conversation
This tool will be how fastly devlopers interface turn their python source into wasm binaries to upload to the platform. It wraps into a static binary (included in our python package) the following disparate pieces: - Our wit definitions - componentize-py - wasiless and related - wac The goal has been to eliminate external runtime dependencies to the extent possible to make for a pleasant developer experience. To package with python, we're making use of maturin and a minimal shim to talk to our rust binary as a library via py03. The binary we'll ship to pypi is going to be a shared library used by the thin script entrypoint. This change also updates the version of componentize-py we are using to the latest as part of packing it into the tool which had some fallout in terms of the code (primarily import of types from componentize-py has changed slightly).
|
Please, contact Developer Platform at #ci-support if you need help using Forge runners |
There's some stuff with forge and other debugging that can happen separate from this PR.
| - macOS: x86_64 (Intel), aarch64 (Apple Silicon) | ||
| - Windows: x86_64 | ||
|
|
||
| The CI workflow (`.github/workflows/build-wheels.yml`) ensures all required tools are installed automatically. |
There was a problem hiding this comment.
This is planned but I have now removed it from this PR, at least until I can get it working separate from this changeset. I have built packages locally just fine.
|
One area I did not change with this has to do with stubs:
We could generate stubs on end-developer machines in some way to aid with tooling, I think, via a separate subcommand or ship the stubs. Along the way, I had an attempt at changing the namespacing for stubs to I don't think we need to solve that with this PR, but just recording some notes and thinking while it came to mind. |
This is mostly to invalidate the cache.
`uv sync` triggers a maturin build for our project (or at least the build.rs portion) which requires dependencies like wasm-tools. Move it to later in the prcoess. The early uv sync was already redundant with other setup steps so we just need to remove the top-level item.
|
I think the CI issues are resolved now, those things are building. With componentize-py as a dependency, we are building from componentize-py (and CPython by extension) from source. I think we might need to more explicitly do something to capture this in our caching setup. |
This is used behind the scenes when we end up building componentize-py.
Also: - Add examples deps to root as optional to help check those. - Upgrade pyrefly to 0.49
Also, add script that will be run in CI to ensure that we keep the version for the tool aligned with the version for the python package.
There was some churn from the base pyproject.toml that bubbled through. I'm not entirely convinced we should have uv.lock in git for the examples but keeping as-is for now.
The main bottleneck in CI was that componentize-py rebuilds CPython from source (Often 20+ minutes) on every run. This was happening because: 1. componentize-py's build.rs builds CPython in its git checkout directory: ~/.cargo/git/checkouts/componentize-py-<hash>/<commit>/cpython/builddir/ 2. We were only caching ~/.cargo/git/db/ (bare repos), not checkouts/ 3. The separate componentize-py-cpython cache with wildcards was redundant and potentially conflicting with the main cargo-deps cache Changes: - Add ~/.cargo/git/checkouts/ to cargo-deps cache (contains CPython build) - Remove redundant componentize-py-cpython cache - Remove sccache CC/CXX wrappers (caused 104 write errors, not needed) - Improve cache key restore pattern for better hit rates - Bump cache version to v3 to bust old caches
|
CI should pass now and on a warm cache is 3-5 minutes. If we don't have componentize-py/CPython cached, those runs will take 20+ minutes. Getting this to work was fairly annoying, but seemed worthwhile to avoid having frequent CI runs taking nearly 30 minutes. Sccache is still in there, though not doing much as setup now and could be removed or revisited to see if we could get it to speed things up; our compile times are mostly dominated by CPython build time rather than rustc. |
| EXAMPLES := bottle-app flask-app backend-requests game-of-life | ||
|
|
||
| # Default example for serve target | ||
| EXAMPLE ?= bottle-app |
There was a problem hiding this comment.
This change breaks the default make serve. Seems to be fine if we put it back.
There was a problem hiding this comment.
Ah, yeah, at some point in development I had it at an underscore as I thought it was required to get the script to work. That turned out not to be the case.
There was a problem hiding this comment.
Nice work! Only problem I saw was that one type in the Makefile.
As for changing the name of wit_world, I'm in favor of it if the issues indeed prove surmountable: specifically, it'd be neighborly to put a "_" in front of it if indeed we're going to keep it private like we've been talking about. (I don't care whether we move it inside fastly_compute or not.)
2 participants
This tool will be how fastly devlopers interface turn their python source into wasm binaries to upload to the platform. It wraps into a static binary (included in our python package) the following disparate pieces:
The goal has been to eliminate external runtime dependencies to the extent possible to make for a pleasant developer experience.
To package with python, we're making use of maturin and a minimal shim to talk to our rust binary as a library via py03. The binary we'll ship to pypi is going to be a shared library used by the thin script entrypoint.
This change also updates the version of componentize-py we are using to the latest as part of packing it into the tool which had some fallout in terms of the code (primarily import of types from componentize-py has changed slightly).
I expect there will need to be some tweaks to the CI configuration among other things, but including it here to see how close we get on the first attempt that will trigger ci.(EDIT: I dropped this part of the change for now).@chenyan2002 CC'ing you to help review to make sure my use of the rust crates wac-* and wit-* seems reasonably sane.