Use pixi to manage development conda environments#3044
Use pixi to manage development conda environments#3044bouweandela wants to merge 28 commits intomainfrom
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3044 +/- ##
=======================================
Coverage 96.15% 96.15%
=======================================
Files 270 270
Lines 15805 15805
=======================================
+ Hits 15197 15198 +1
+ Misses 608 607 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| os.environ["ESMFMKFILE"] = f"{rtd_conda_prefix}/lib/esmf.mk" | ||
| os.environ["PROJ_DATA"] = f"{rtd_conda_prefix}/share/proj" | ||
| os.environ["PROJ_NETWORK"] = "OFF" | ||
|
|
There was a problem hiding this comment.
I see the docs build nicely, so these can indeed be removed, but why are we removing them now - is it something to do with pixi?
There was a problem hiding this comment.
Yes, as explained in the comment above, the conda environment used by the previous way of installing the environment on readthedocs was not activated. Pixi does not have this problem, so this code can be removed.
There was a problem hiding this comment.
I'd keep it somewhere somehow, just in case. I think the whole purge of conda-forge way of installing is a bit premature at this point in time, even if we have pixi in for end users- which is nice, I can already think of a few cases where pixi won't work, so I'd argue we still keep the old conda stashed somewhere, in case we have to use it. How, I need to think of that
|
hey @bouweandela a few points:
(base) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pixi --version
pixi 0.67.0
(base) valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pixi shell
WARN Skipped running the post-link scripts because `run-post-link-scripts` = `false`
- bin/.librsvg-pre-unlink.sh
To enable them, run:
pixi config set --local run-post-link-scripts insecure
More info:
https://pixi.sh/latest/reference/pixi_configuration/#run-post-link-scripts
WARNING: Did not detect successful shell initialization within 3 second(s).
Please check on https://pixi.sh/latest/advanced/pixi_shell/#issues-with-pixi-shell for more tips.
er passphrase for /home/valeriu/.ssh/id_rsa_jasmin:
Identity added: /home/valeriu/.ssh/id_rsa_jasmin (/home/valeriu/.ssh/id_rsa_jasmin)
Agent pid 963160
Enter passphrase for /home/valeriu/.ssh/id_rsa_cloud:
valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pixi config set --local run-post-link-scripts insecure
✅ Updated config at /home/valeriu/ESMValCore/.pixi/config.toml
valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ git status
On branch pixi
Your branch is up to date with 'origin/pixi'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
.pixi/
nothing added to commit but untracked files present (use "git add" to track)
valeriu@valeriu-PORTEGE-Z30-C:~/ESMValCore$ pixi shell
WARNING: Did not detect successful shell initialization within 3 second(s).
Please check on https://pixi.sh/latest/advanced/pixi_shell/#issues-with-pixi-shell for more tips.
er passphrase for /home/valeriu/.ssh/id_rsa_jasmin:
Agent pid ...Why is it putting |
ah! It works now: You gotta tell folks to not have anything that waits for a password in their |
|
Thanks for reviewing V!
I suspect you may have been on a different branch than the one from this pull request the first time you ran that command, because
As the name suggests, the command |
I put it in |
|
note (am not sure how important it is) that a that don't show up in the pixi env: |
|
It was already there, now you've added it a second time. Maybe it is because you ran |
nope that was showing up as ungit before - leave it like that - twice is better 😁 |
There should be no need to do that |
Added in 94f10e0 |
just like Carroll Shelby told his chief mechanic that said she can come apart: one way to find out 😁 https://www.youtube.com/watch?v=5OfjrMWjrhs |
|
One thing I think you should mention is that people should not have multiple ESMValXXX dirs - and they have - lots of eg |
in this vein, how the heck does one create new named environments without duplicating the project ie the repo? Say I need an environment |
Pixi avoids the problem by using hardlinks, so it only keeps one copy of a file, even if it's used in multiple places: https://pixi.prefix.dev/latest/workspace/environment/#de-duplication |
Edit pyproject.toml, add the extra environment (plus any needed features) there and run |
ah! Found the solution - in pyproject.toml |
|
Ow man, you gonna get so many users (not technical ones, though) so confused with all these new procedures - they are still confused about using conda/mamba - pixi takes this one notch up the level of confusability. Pixi is a lot more under the hood technical than conda - I don't think our ESMValTool users will get used to it easily if they want to do the slightest bit of development, more than |
|
we should think along these lines when we explain Pixi to our users (prob best to structure the docs too, or the UET in their Tutorial) - lifted from ChatGPT that makes a very solid point: pixi is a lot more technical than conda, and I think non-technical users can struggle with it more than with conda, what do you think? You’re not wrong—but it’s a bit more nuanced than “Pixi is harder.” 🧠 The core difference
That shift alone makes Pixi feel more “technical,” especially at first. 👍 Where Conda is easier for non-technical users
👉 For beginners or casual users, Conda is usually easier to start with. 🤔 Where Pixi feels harder
👉 That does raise the barrier for non-technical users. ⚖️ But Pixi isn’t harder for no reasonPixi is optimizing for things Conda struggles with:
So:
🎯 My honest take
💡 Why Pixi feels more technicalIt’s not just complexity—it’s where the complexity lives:
That’s great for engineers, but unfamiliar to beginners. 🧭 Practical compromiseA lot of people end up doing:
|
|
Learning something new is always hard, and learning a new technical skill is more annoying if it's not your main interest but rather something you want to get over with as quickly as possible with a minimal time investment. So yes, this will require some initial time investment and that may be frustrating to some, but I believe it will save our contributors time in the long run. No more waiting for solving environments will be an immediate time gain, and the much better reproducibility is very helpful when doing science. While the observations above by ChatGPT above may be true in general, they are not for our project. Our contributors, at least the ones that want to change the dependencies, already need to understand two or more different configuration files: pyproject.toml for the Python dependencies and (multiple in the case of ESMValTool) environment.yml files for the conda-forge dependencies. Pixi simplifies that by making it possible to do all the configuration in a single file (pyproject.toml) with a single syntax. One thing that we could do to make the transition easier is to do it at the upcoming workshop. Like that, we will be able to help out anyone who is there, either in person or online, immediately. |
valeriupredoi
left a comment
valeriupredoi
left a comment
There was a problem hiding this comment.
I agree with you, bud!
One thing that we could do to make the transition easier is to do it at the ESMValGroup/Community#273. Like that, we will be able to help out anyone who is there, either in person or online, immediately.
I think that's a good idea too! At any rate, very many thanks to you for doing this work, bud - excuse my constant nagging during the review process, I am always trying to find holes in the plot so we can fix them when we review, rather than when we have users barking at us 🐶
valeriupredoi
left a comment
valeriupredoi
left a comment
There was a problem hiding this comment.
I realized I've not formally approved, many thanks @bouweandela 🍺🍺
|
Did you run the changed GitHub actions workflows? Sorry, there's so many commits, I can't find them. |
schlunma
left a comment
schlunma
left a comment
There was a problem hiding this comment.
Thanks Bouwe, that's very nice! I was a bit skeptical of Pixi at first, but after using it for some time (in other projects) I really like it now.
The installation works very well. The only issue I found is that esmvaltool config show stopped working (see #3062), but I am pretty sure this is not related to Pixi.
I just have a couple of minor comments on the code. Will look at the ESMValTool PR now.
|
|
||
| Pip installation | ||
| ----------------- | ||
|
|
There was a problem hiding this comment.
What's the reason we mention this here? Is there any advantage of doing that over the mamba installation? Just saying "installing dependencies that are not available on PyPI in some other way" is also not really helpful.
I would argue we should remove that, especially since this is the section targeted to regular users (not developers).
| ``esmvaltool -h``. | ||
| - Install the :ref:`esmvaltool:pre-commit` hooks by running: | ||
| ``pre-commit install``. | ||
| - Create the development environment and start a shell in this environment by running: ``pixi shell`` |
There was a problem hiding this comment.
If I understand correctly, pixi shell will solve the dependencies as defined in the manifest file and update the pixi.lock file. Since you mentioned that one advantage of Pixi is that dependencies don't need to be solved, we should mention here that one can also use pixi shell --frozen for a much faster installation with slightly outdated dependencies.
pixi shell can take a long time too run (much longer than mamba activate before), so we might recommend this as the default.
|
|
||
| Optionally you can skip tests which require additional dependencies for | ||
| supported diagnostic script languages by adding ``-m 'not installation'`` to the | ||
| or ``pixi run test``. Optionally you can skip tests which require additional |
There was a problem hiding this comment.
Put pixi run test into a code block to make it more prominent?
| platforms = ["linux-64", "osx-arm64"] | ||
|
|
||
| [tool.pixi.pypi-dependencies] | ||
| ESMValCore = { path = ".", editable = true } |
There was a problem hiding this comment.
| ESMValCore = { path = ".", editable = true } | |
| esmvalcore = { path = ".", editable = true } |
Shouldn't this refer to the package name?
|
Hi @schlunma - it's great you're also looking at this, and already found a couple bits that I missed! Both Bouwe and I ran the GHAs so those should be solid. 👍🍺 |
|
Can you point me to the logs? I want to have a look at them 😁 |
If you go to Actions you can see the Branch tab there then you can select pixi to show the GHAs that ran only on the pixi branch, buds 🍺 |
|
Ah, perfect! I've never noticed the "branches" column before 😅 One more question: After installing the environment, I get this warning: Do we need to run these post-link scripts? It seems to work without it. In this case, I wonder if we could silence this warning by explicitly setting |
Nothing to worry, bud. Quoting ChatGPT (diuble checked by yours truly) What it meansPixi is telling you it skipped a post-link script ( Post-link scripts are small setup/cleanup scripts that some packages (like librsvg) try to run after installation or before removal. Should you worry?Most of the time: no 👍
When it might matterYou might run into issues if:
How to enable scripts (if needed)If you do need those scripts, you can turn them back on in your Pixi config: run-post-link-scripts = trueOr via CLI (depending on your setup): pixi config set run-post-link-scripts trueBottom lineIf everything is working → ignore it. |
3 participants
Description
Switch to pixi for managing the development conda environments.
See ESMValGroup/ESMValTool#4407 for an extensive description and ESMValGroup/ESMValTool#3997 for further background.
Links to documentation:
Before you get started
Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.
To help with the number pull requests: