Add Pyodide (WASM) wheel build and release asset - #1960
Conversation
|
@ericmjl for future reference we prefer devs to contribute from their forks instead of working on the main repo directly |
I'm pretty sure I'm working from my own fork 😄 |
lucianopaz
left a comment
There was a problem hiding this comment.
Wouldn't it be good to add a line some in the README.md or the docs landing page saying how to install on a wasm environment? Wouldn't users have to put in the full url of the github release to install?
You sure are, I wasn't seeing the ericmjl prefix before on my desktop. On the mobile I see |
|
@lucianopaz addressed your comment too. |
4d82c0f to
0e6aa46
Compare
0e6aa46 to
a305593
Compare
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
@ericmjl, heads-up in case you didn't notice, there are lots of merge conflicts from your last force-push. |
|
just the branch target |
|
Hi, This is Anutosh here (Jupyterlite, LLVM, Clang, Clang-Repl & Emscripten-Forge maintainer/contributor) I have successfully built numba & llvmlite for wasm. I spoke about upstreaming my patches in the Numba community call yesterday and we should be making progress there. For more technical details go through my comment here : numba/numba#3284 (comment) You can try out Numba in Jupyterlite here : https://notebook.link/@anutosh491/numba I had interacted with Thomas Wiecki (@twiecki) about 2 weeks back on my LinkedIn post talking about how PyTensor could benefit out of this. Hence I went ahead and built PyTensor for wasm enabling Numba as PyTensor's WebAssembly linker. Check it out here, I have hosted a simple demo notebook to start with : https://notebook.link/@anutosh491/PyTensor (enable the debug logs to see how numba functions in the backend) cc @ricardoV94 @maresb @lucianopaz Feel free to contact us (me, @SylvainCorlay or @MMesch) at QuantStack for more details. We are the main authors of Jupyterlite and would like to host a Jupyterlite notebook in PyTensor's readme. We should possibly be able to shift tutorials or examples that users want to try out in Jupyterlite itself. |
- Replace old 'Build universal wheel for Pyodide' job with proper pyodide-build + Emscripten job: Python 3.12, pyodide-build, setup-emsdk, pip install -e .[pyodide], pyodide build + Upload artifact pyodide_wheel (dist/*wasm32.whl) - Add job to upload Pyodide wheel to GitHub Release on publish (PyPI does not accept wasm32 yet) - Add optional dependency pyodide = [pyodide-build>=0.29.2] and document manual build in dev guide - Remove Pixi references: .pixi from pre-commit sphinx-lint args, conda/mamba/pixi -> conda/mamba in cmodule and troubleshooting - upload_pypi no longer downloads universal_wheel; PyPI gets only sdist + cibuildwheel wheels - Numba remains optional on wasm via existing env markers (micropip install works without numba) Made-with: Cursor
…tprops action Made-with: Cursor
Made-with: Cursor
- Restore pixi references in docs and warnings (previously removed without explanation) - Restore helpful workflow comments in pypi.yml - Move Pyodide build docs from dev_start_guide to install.rst (more appropriate location) - Add user-facing WASM/Pyodide installation instructions in install.rst
Addresses lucianopaz's feedback: make WASM installation instructions discoverable from the README and docs index, not just install.rst.
a305593 to
3596bde
Compare
|
Alrighty, @ricardoV94 and @maresb -- looks like we're in a good spot now. |
|
Hey @anutosh491 — first off, thank you so much for this, it's genuinely exciting! Building numba + llvmlite for wasm and wiring it up as PyTensor's WebAssembly linker is a huge piece of work, and the JupyterLite demo is such a lovely touch. Really appreciate you (and the QuantStack crew) pushing this forward. I spent some time today tracing through our build to make sure I actually understand what's needed on our side, and I'd love to sanity-check my mental model with you:
If I'm reading this right: on the PyTensor side we don't need a compiled wasm build at all — the pure-Python wheel on PyPI is the deliverable. The remaining piece for the fully-automagical Is that understanding correct? And if so, my plan is to unwind the Either way — thank you again. This is really cool. 😄 |
|
Hey @ericmjl , Thanks for asking! I think there are two slightly different deployment paths being discussed here. PyTensor’s pure-Python wheel is portable, but the Numba and llvmlite support I demonstrated is currently delivered through xeus-python and emscripten-forge. Those packages target our JupyterLite Emscripten runtime; they are not yet PyEmscripten/PEP 783 wheels for Pyodide. Producing those wheels would be a separate compatibility effort because we would need to align and test the compiler, linker and dynamic-loading stack against the PyEmscripten ABI. What we can support immediately is an official PyTensor JupyterLite deployment !! There are also a few small PyTensor fixes currently carried by the emscripten-forge recipe. I’ll start by upstreaming the most important one separately: replacing hard-coded 64-bit indices in the Numba vectorized code generator with target-sized intp values. Made that here : #2334 Once that is reviewed, I can rebase the JupyterLite CI PR on top of it. Made it here : #2336 (based on #2335) Longer term, PEP 783 wheels could also open a Pyodide/micropip path, but we don’t need to wait for that to provide a working Numba-enabled browser experience today. |
|
Also that being said I see pytensor has all these backends We've worked on supporting the first one now but I think we can do justice with the other backend being built for wasm too !! |
|
If everything is wasm at the end of the day I don't think there's a point in supporting the others as a bridge to wasm as far as PyTensor is concerned. That's something those other packages may want, but for us it doesn't matter. We don't even need that it be the numba backend that bridges to it, rather whatever is easier and most performant in the end. See for instance a more streamlined pure python (aka numpy) backend: #2193 |
|
Hmmm so I guess it's about supporting a backend (possibly the best performance wise) end to end rather than multiple ones. |
|
Also @ricardoV94 could you help me with enabling ci for my other prs |
Now that the first one got merged it should fire automatically on new pushes/PRs. It's the first time contributor ruler that was blocking before |
Yes. And by performant is whatever generates better webasm code |
Thank you. I've re-run the CI on my PRs. |
Summary
setup.py bdist_wheel --universal) with a proper pyodide-build + Emscripten job that produces a wasm32 wheel.pyodideinpyproject.tomland documents the manual Pyodide build in the dev guide; no Pixi or other new tooling..pixifrom pre-commit sphinx-lint args, andconda/mamba/pixi→conda/mambain cmodule and troubleshooting docs.Numba remains optional on wasm via existing environment markers, so the Pyodide wheel is micropip-installable without numba.
CI changes
build_pyodide_wheel: Python 3.12, install pyodide-build, get Emscripten version,mymindstorm/setup-emsdk,pip install -e ".[pyodide]",pyodide build, uploadpyodide_wheelartifact.upload_pyodide_to_release: On release published, downloadpyodide_wheeland attach to the release viasoftprops/action-gh-release.upload_pypi: No longer downloads or uploads the olduniversal_wheel; PyPI receives only sdist + cibuildwheel platform wheels.Made with Cursor