[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #121

n-poulsen · 2025-02-28T10:58:21Z

This pull requests updates DeepLabCut-Live for models exported with DeepLabCut 3.0. TensorFlow models can still be used, and the code is siloed so that only the engine used to run the code is required as a package (i.e. no need to install TensorFlow if you want to run live pose estimation with PyTorch models).

If you want to give this PR a try, you can install the code in your local conda environment by running:

pip install "git+https://github.com/DeepLabCut/DeepLabCut-live.git@dlclive3"

…in code

…NX inference

…erence

MMathisLab · 2026-01-13T20:13:55Z

Let's drop poetry

sneakers-the-rat · 2026-01-13T20:17:27Z

pyproject.toml


 [tool.poetry.dependencies]
-python = ">=3.9,<3.12"
+python = ">=3.10,<3.12"


Whats keeping us with the 3.12 cap here?

I've been testing with 3.12 recently. It seems to work fine, except on windows with tensorflow. I've updated the pyproject.toml. Let me know if you have suggestions!

sneakers-the-rat · 2026-01-13T20:18:15Z

pyproject.toml

-tables = "^3.6"
+pandas = ">=1.0.1,!=1.5.0"
+tables = "^3.8"
 pytest = "^8.0"


Pytest goes in dev dependencies :)

sneakers-the-rat · 2026-01-13T20:20:24Z

scripts/fix_deeplabcut_imports.py

Seems like a single use thing, does this need to stay?

Good one, think it can be removed indeed.

sneakers-the-rat · 2026-01-13T22:04:44Z

Let's drop poetry

Good call. The cool kids are using uv these days. Those of us who are recalcitrant, into standards, and skeptical of VC tech can recommend pdm as well :)

- Package version is copied from main for now, until finalized changes. - don't use subset cpuinfo - updated type hints in benchmark.py as was already implemented in main branch - copied todo's in check_install.py

This commit corrects the parameter names and order and removes unsupported arguments from `benchmark_videos` call: - device - precision - draw_keypoint_names - get_sys_info

- pyproject.toml is made PEP 517 / 518 / 621–compliant, which makes it easy to inherit dependencies - it is more tool-agnostic (no poetry required & facilitates uv packaging). - Python versions 3.10 up to 3.12 - tensorflow installation is restricted to python < 3.12 for windows

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

run with uv instead of poetry and for python 3.10 up to 3.12

- Remove upper bound for tensorflow version is python version >=3.11 - add pip as required dependency

remove tkinter as required dependency.

(remove conditional tf dependency install for for windows and python >= 3.11, since never reached)

…liant *Update pyproject.toml and CI* - pyproject.toml is made PEP 517 / 518 / 621–compliant, which makes it easy to inherit dependencies - it is more tool-agnostic (no poetry required & facilitates uv packaging). - Python versions 3.10 up to 3.12 - Tensorflow installation is restricted to python < 3.11 for windows - CI is updated to test more Python versions and to use uv instead of poetry.

Added a more robust downloading function in `utils.py` that handles errors more gracefully. The download is skipped if the file already exists. In previous implementation, the benchmarking video was downloaded to the home directory, which in windows could result in permission issues when reading the file. This is now changed to the check_install directory. OpenCV was silently failing to read the video file, resulting in a frame count of zero. Now a ValueError is raised when the video cannot be read.

- removed poetry instructions - added optional instructions for installation with uv - added notes on tensorflow support for windows

deruyter92

Looked trough most of the changes and addressed the merge conflicts. I think it would be great if we could merge this WIP as soon as possible, so that we can cleanly contribute aditional PRs

What I have worked on:

Merged with main (accepting combination of both, but mostly dlclive3 branch for updated code)
Complete update of pyproject.toml
- more tool-agnostic (no poetry required)
- uv-friendly, so the user can choose their own preferred packaging method
- updated missing dependencies
- Python versions 3.10, 3.11, 3.12 (except for tensorflow users on windows)
- restrict windows users to python 3.10 for tensorflow
Changed the CI accordingly to test for multiple python versions.
CI uses uv now for installation.
Added some basic testing
Adressed copilot and other comments

@sneakers-the-rat, @MMathisLab, @C-Achard please let me know what you think and if you'd like me to incorporate additional changes.

deruyter92 · 2026-01-14T15:37:10Z

.github/workflows/testing.yml

+      - name: Install and test
        shell: bash -el {0}  # Important: activates the conda environment
        run: |
          conda install pytables==3.8.0 "numpy<2"


Thanks. This CI workflow is updated now. Let me know what you think!

deruyter92 · 2026-01-14T15:38:24Z

pyproject.toml


 [tool.poetry.dependencies]
-python = ">=3.9,<3.12"
+python = ">=3.10,<3.12"


I've been testing with 3.12 recently. It seems to work fine, except on windows with tensorflow. I've updated the pyproject.toml. Let me know if you have suggestions!

deruyter92 · 2026-01-14T15:38:39Z

pyproject.toml

-tables = "^3.6"
+pandas = ">=1.0.1,!=1.5.0"
+tables = "^3.8"
 pytest = "^8.0"


C-Achard · 2026-01-14T17:48:19Z

@deruyter92 Anything I can focus on/help with ?

sneakers-the-rat · 2026-01-14T22:43:47Z

I wonder if we could remove the package-level upper bound on python version and just throw an error if someone tries to use tensorflow with python > 3.12 (or, probably, just throw an error if it cant be imported, in case someone builds their own tensorflow for >3.12). I am assiming torch works with >3.12, and it would be nice to not have that limit for downstream use, future compatibility, etc.

Edit: otherwise i still have same reservations re: maintainability duplicating the dlc code here, would prefer that code is split out in such a way that the main deeplabcut and vendored dlc-live versions cant go out of sync. Its a real shame we cant just use the same serialized model strategy as before, but if torch doesnt have model serialization, thats over our heads. But otherwise as far as i can tell looks good. The changes to the infra are good, thanks for handling that - poetry made sense at a time when it was the only thing of its kind, but the packaging ecosystem has changed a lot (for the better) since then.

.github/workflows/testing.yml

sneakers-the-rat · 2026-01-14T23:00:31Z

tests/test_benchmark_script.py

+        print(f"Running dog model: {model_path}")
+        benchmark_videos(
+            model_path=model_path,
+            model_type="base" if Engine.from_model_path(model_path) == Engine.TENSORFLOW else "pytorch",


I believe this should continue to test tensorflow models, as long as the tensorflow models remain in the benchmarking data. Is that right? There is the possibility that the benchmarking data becomes updated and we silently stop testing tensorflow, so i think it would be nice to parameterize this test to explicitly run with pytorch and tensorflow, and fail test if no tensorflow models are found.

sneakers-the-rat · 2026-01-14T23:03:13Z

dlclive/engine.py

+            if has_cfg and has_pb:
+                return cls.TENSORFLOW
+        elif path.is_file():
+            if path.suffix == ".pt":


Oh wait are there serialized pytorch models now??? That would dramatically simplify things and avoid possibility for model drift - we could take out all the model code here (keeping the great work making the switchable backends!)

Co-authored-by: Jonny Saunders <sneakers-the-rat@protonmail.com>

dikraMasrour and others added 24 commits February 24, 2025 15:30

Remove TensorFlow references

b11743c

Add comments on TF questions for changing dlc-live pipeline to Pytorch

c8f5839

Vanilla pytorch inference done; Commenting out TensorFlow references …

030c42c

…in code

change testing directory for Anna and allow to run on CPU

d75339a

Fix display, clean code, adapt frame processing, onnx inference

4de5829

add screenshots

40a9dd4

Fix CPU inference crash + GPU (cuda) & CPU support for Pytorch and ON…

2fc48f1

…NX inference

Video analysis feature

125a072

Improvements on benchmark_pytorch.py

9d2350e

Implement TensorRT optimisation on ONNX models and FP16 precision inf…

13b8fae

…erence

Avs live feed

8b41df6

Tutorial notebook in progress

d1e7df9

bug fixing h5 saving for live video feed

2fb39fc

add code to save numbers in csv and h5 as numbers, not tensor(number)

98b3a13

add timestamp suffix to videos and csv/h5 files

e6a914a

fix live inference and display, black and isort

29c8122

cleaning out unused files

9508a79

update docstrings, clean dlclive script

d61f892

Continued DeepLabCut-Live implementation for DeepLabCut 3.0

f527d38

working on README

4c07e04

improved docs

82daf43

improved docs for PyTorch code

40005ae

improved readme

c6a1f69

fix default top down dynamic cropping parameters

d738a2c

n-poulsen added the DLC 3.0 🔥 label Feb 28, 2025

n-poulsen mentioned this pull request Feb 28, 2025

[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #120

Closed

n-poulsen requested review from AlexEMG, MMathisLab and maximpavliv February 28, 2025 10:58

n-poulsen mentioned this pull request Feb 28, 2025

MobileNets DeepLabCutAIResidency/DLC_AI2024#24

Open

sneakers-the-rat reviewed Jan 13, 2026

View reviewed changes

deruyter92 and others added 18 commits January 14, 2026 08:45

Merge main into dlclive3

526a162

- Package version is copied from main for now, until finalized changes. - don't use subset cpuinfo - updated type hints in benchmark.py as was already implemented in main branch - copied todo's in check_install.py

Fix argument alignment in benchmark_videos call

cbc23ed

This commit corrects the parameter names and order and removes unsupported arguments from `benchmark_videos` call: - device - precision - draw_keypoint_names - get_sys_info

fix python version <3.13

cd4649e

Update pyproject.toml: remove duplicate line

e0e900a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

update CI

f5ce1a1

run with uv instead of poetry and for python 3.10 up to 3.12

Update pyproject.toml

690f285

- Remove upper bound for tensorflow version is python version >=3.11 - add pip as required dependency

update display.py

8f1e844

remove tkinter as required dependency.

incorporate copilot suggestions

c6f7813

(remove conditional tf dependency install for for windows and python >= 3.11, since never reached)

merge branch 'main' into PR #121

9db0752

Update readme and installation docs

ecb3673

- removed poetry instructions - added optional instructions for installation with uv - added notes on tensorflow support for windows

update .gitignore (ignore .lock files)

7e9b4b4

Minor refactors (comments, unused imports, unused variables)

11690ed

remove unused script fix_deeplabcut_imports

6e2ad64

add basic tests

72d073b

update .gitignore

681c373

deruyter92 approved these changes Jan 14, 2026

View reviewed changes

sneakers-the-rat reviewed Jan 14, 2026

View reviewed changes

.github/workflows/testing.yml Outdated Show resolved Hide resolved

sneakers-the-rat reviewed Jan 14, 2026

View reviewed changes

Update CI workflow: include all tests

83390f0

Co-authored-by: Jonny Saunders <sneakers-the-rat@protonmail.com>

[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #121

Are you sure you want to change the base?

[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #121

Conversation

n-poulsen commented Feb 28, 2025

Uh oh!

MMathisLab commented Jan 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sneakers-the-rat commented Jan 13, 2026

Uh oh!

deruyter92 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

C-Achard commented Jan 14, 2026

Uh oh!

sneakers-the-rat commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

sneakers-the-rat commented Jan 14, 2026 •

edited

Loading