Skip to content

Commit 1ca9a56

Browse files
aaron-skydioaaron-skydio
committed
[SymForce] Use uv for requirements
For example: aaron-skydio#4 Topic: sf-uv
1 parent 04ccac1 commit 1ca9a56

14 files changed

Lines changed: 1654 additions & 159 deletions

‎.github/workflows/ci.yml‎

Lines changed: 5 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -112,18 +112,11 @@ jobs:
112112
libgoogle-glog-dev \
113113
libeigen3-dev
114114
115-
# NOTE(aaron): Some packages do not have a version that supports py3.8..py3.12
116-
- name: Fix py3.12 versions
117-
if: ${{ matrix.python == 'python3.12' }}
118-
run: |
119-
sed -i 's|numba==0.58.1|numba~=0.59.0|g' dev_requirements.txt
120-
sed -i 's|llvmlite==0.41.1|llvmlite~=0.42.0|g' dev_requirements.txt
121-
sed -i 's|numpy==1.24.4|numpy~=1.26.0|g' dev_requirements.txt
122-
sed -i 's|scipy==1.10.1|scipy~=1.12.0|g' dev_requirements.txt
123-
sed -i 's|pandas==2.0.3|pandas~=2.2.0|g' dev_requirements.txt
124-
125115
- name: Install python dependencies
126-
run: pip install -r dev_requirements.txt
116+
run: |
117+
PY_MINOR_VERSION=$(${{ matrix.python }} -c "import sys; print(sys.version_info.minor)")
118+
${{ matrix.python }} -m pip install pip==24.0 setuptools==69.5.1
119+
${{ matrix.python }} -m pip install -r requirements_dev_py3${PY_MINOR_VERSION}.txt
127120
128121
- name: Run cmake build
129122
run: |
@@ -135,17 +128,11 @@ jobs:
135128
-D SYMFORCE_BUILD_BENCHMARKS=ON
136129
cmake --build build -j $(nproc)
137130
138-
# - lcmtypes need to be available for tests
139-
# - Exact contents of dev_requirements.txt depend on python version. Need to update file to
140-
# match current python version to avoid failure of corresponding gen test. symforce needs
141-
# to be on the PYTHONPATH to run gen test in this manner.
131+
# lcmtypes and symforce need to be available for tests
142132
- name: Run tests
143133
run: |
144134
pip install build/lcmtypes/python2.7
145135
export PYTHONPATH=$PYTHONPATH:$(pwd)
146-
${{ matrix.python }} test/symforce_requirements_test.py --update
147-
echo "Modifications made to requirements:"
148-
git diff
149136
EXIT_CODE=0
150137
ctest --test-dir build -j $(nproc) || EXIT_CODE=$?
151138
if [ $EXIT_CODE -ne 0 ]; then

‎.github/workflows/docs.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,9 @@ jobs:
2323
pandoc
2424
2525
- name: Install python dependencies
26-
run: pip install -r dev_requirements.txt
26+
run: |
27+
PY_MINOR_VERSION=$(python -c "import sys; print(sys.version_info.minor)")
28+
python -m pip install -r requirements_dev_py3${PY_MINOR_VERSION}.txt
2729
2830
- name: Run cmake build
2931
run: |

‎.github/workflows/solve_requirements.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Solve Pip Requirements
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
upgrade:
7+
required: true
8+
type: boolean
9+
description: Run `uv compile` with the `--upgrade` flag
10+
default: false
11+
12+
jobs:
13+
solve:
14+
runs-on: ubuntu-latest
15+
strategy:
16+
fail-fast: false
17+
matrix:
18+
py_version: [8, 9, 10, 11, 12]
19+
steps:
20+
- name: Checkout
21+
uses: actions/checkout@v4
22+
23+
- uses: actions/setup-python@v5
24+
with:
25+
python-version: '3.${{ matrix.py_version }}'
26+
27+
- run: |
28+
pip install uv
29+
uv pip compile --all-extras --output-file=/tmp/requirements_bootstrap.txt pyproject.toml
30+
uv venv
31+
source .venv/bin/activate
32+
uv pip install -r /tmp/requirements_bootstrap.txt
33+
34+
- run: PYTHONPATH=$(pwd) python test/symforce_requirements_test.py --update
35+
if: ${{ !github.event.inputs.upgrade }}
36+
37+
- run: PYTHONPATH=$(pwd) python test/symforce_requirements_test.py --update --piptools_upgrade
38+
if: ${{ github.event.inputs.upgrade }}
39+
40+
- uses: actions/upload-artifact@v4
41+
with:
42+
name: requirements_dev_py3${{ matrix.py_version }}
43+
path: requirements_dev_py3${{ matrix.py_version }}.txt
44+
45+
publish:
46+
runs-on: ubuntu-latest
47+
needs: solve
48+
steps:
49+
- uses: actions/checkout@v4
50+
51+
- uses: actions/download-artifact@v4
52+
with:
53+
merge-multiple: true
54+
55+
- uses: peter-evans/create-pull-request@v6
56+
with:
57+
commit-message: Update Requirements
58+
title: Update Requirements
59+
body: ""

‎.github/workflows/test_editable_pip_install.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,10 @@ jobs:
3434
pip==${{ matrix.pip_version }} \
3535
setuptools==${{ matrix.setuptools_version }}
3636
37-
- name: install dev_requirements.txt
38-
run: python -m pip install -r dev_requirements.txt
37+
- name: install requirements
38+
run: |
39+
PY_MINOR_VERSION=$(python -c "import sys; print(sys.version_info.minor)")
40+
python -m pip install -r requirements_dev_py3${PY_MINOR_VERSION}.txt
3941
4042
- name: editable install
4143
run: python -m pip install -v -e .

‎CMakeLists.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55

66
# NOTE(aaron): This is the minimum version for policy range support, not sure if we need newer;
77
# certainly no newer than 3.15 required. This will use NEW policies up to CMake 3.25; this should
8-
# be the maximum tested CMake version, matching dev_requirements.txt
8+
# be the maximum tested CMake version, matching requirements_dev.txt
99
cmake_minimum_required(VERSION 3.19...3.25)
1010

1111
if(CMAKE_VERSION VERSION_GREATER_EQUAL 3.27)

‎README.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -567,7 +567,7 @@ pip install -e .
567567

568568
You should then [verify your installation](#verify-your-installation).
569569

570-
___Note:___ `pip install .` will not install pinned versions of SymForce's dependencies, it'll install any compatible versions. It also won't install all packages required to run all of the SymForce tests and build all of the targets (e.g. building the docs or running the linters). If you want all packages required for that, you should `pip install .[dev]` instead (or one of the other groups of extra requirements in our `setup.py`). If you additionally want pinned versions of our dependencies, which are the exact versions guaranteed by CI to pass all of our tests, you can install them from `pip install -r dev_requirements.txt`.
570+
___Note:___ `pip install .` will not install pinned versions of SymForce's dependencies, it'll install any compatible versions. It also won't install all packages required to run all of the SymForce tests and build all of the targets (e.g. building the docs or running the linters). If you want all packages required for that, you should `pip install .[dev]` instead (or one of the other groups of extra requirements in our `setup.py`). If you additionally want pinned versions of our dependencies, which are the exact versions guaranteed by CI to pass all of our tests, you can install them from `pip install -r requirements_dev_py3<version>.txt`.
571571

572572
_Note: Editable installs as root with the system python on Ubuntu (and other Debian derivatives) are broken on `setuptools<64.0.0`. This is a [bug in Debian](https://ffy00.github.io/blog/02-python-debian-and-the-install-locations/), not something in SymForce that we can fix. If this is your situation, either use a virtual environment, upgrade setuptools to a version `>=64.0.0`, or use a different installation method._
573573

@@ -576,11 +576,13 @@ _Note: Editable installs as root with the system python on Ubuntu (and other Deb
576576
If you'll be modifying the C++ parts of SymForce, you should build with CMake directly instead - this method will not install
577577
SymForce into your Python environment, so you'll need to add it to your PYTHONPATH separately.
578578

579-
Install python requirements:
579+
Install dependencies required to build and run SymForce:
580580
```bash
581-
pip install -r dev_requirements.txt
581+
pip install -r requirements_build.txt
582582
```
583583

584+
___Note:___ `requirements_build` contains only packages required to build and run symforce, but not everything recommended to develop symforce, like to run the SymForce tests and linters. For that, install the full pinned requirements using `pip install -r requirements_dev_py3<version>.txt` for your Python version.
585+
584586
Build SymForce (requires C++14 or later):
585587
```bash
586588
mkdir build

‎requirements_build.txt‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# This file was autogenerated by uv via the following command:
2+
# python test/symforce_requirements_test.py --update
3+
clang-format
4+
# via symforce (pyproject.toml)
5+
cmake
6+
# via symforce (pyproject.toml)
7+
cython
8+
# via symforce (pyproject.toml)
9+
graphviz
10+
# via symforce (pyproject.toml)
11+
jinja2
12+
# via symforce (pyproject.toml)
13+
numpy
14+
# via symforce (pyproject.toml)
15+
pip
16+
# via symforce (pyproject.toml)
17+
ruff
18+
# via symforce (pyproject.toml)
19+
scipy
20+
# via symforce (pyproject.toml)
21+
setuptools
22+
# via symforce (pyproject.toml)
23+
file:./third_party/skymarshal
24+
# via symforce (pyproject.toml)
25+
file:./gen/python
26+
# via symforce (pyproject.toml)
27+
sympy
28+
# via symforce (pyproject.toml)
29+
wheel
30+
# via symforce (pyproject.toml)

0 commit comments

Comments
 (0)