Skip to content

Split tests off of self hosted runner#1901

Open
Dreamsorcerer wants to merge 82 commits intodevfrom
sam/split-tests
Open

Split tests off of self hosted runner#1901
Dreamsorcerer wants to merge 82 commits intodevfrom
sam/split-tests

Conversation

@Dreamsorcerer
Copy link
Copy Markdown
Collaborator

@Dreamsorcerer Dreamsorcerer commented Apr 22, 2026

This splits linting and tests into separate jobs on standard GH runners.
Only the tests that need ROS, LFS or something similar are now run on self-hosted runner.

This speeds up the CI significantly (GH runner tests also get to use xdist to parallelise the tests).
This also allows external PRs to be tested and merged normally, with just the self-hosted tests being skipped.
Also some security improvements and improving maintainability of the code.

@Dreamsorcerer Dreamsorcerer changed the base branch from main to dev April 22, 2026 15:24
Comment thread .github/workflows/ci.yml Outdated
Comment thread dimos/models/vl/test_base.py Outdated
Comment thread .github/workflows/ci.yml Outdated
Comment on lines +67 to +72
- name: Install dependency for pyaudio
run: |
sudo apt-get update
sudo apt-get install -y portaudio19-dev
- name: Remove git LFS to avoid accidental large downloads
run: sudo rm -f /usr/bin/git-lfs
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Both sudo apt-get install portaudio19-dev and sudo rm -f /usr/bin/git-lfs are Linux-only commands, but the matrix includes macos-latest and windows-latest runners. apt-get does not exist on macOS or Windows, so every non-Ubuntu matrix cell will fail before tests even start. Both steps need an Ubuntu guard.

Suggested change
- name: Install dependency for pyaudio
run: |
sudo apt-get update
sudo apt-get install -y portaudio19-dev
- name: Remove git LFS to avoid accidental large downloads
run: sudo rm -f /usr/bin/git-lfs
- name: Install dependency for pyaudio
if: matrix.os == 'ubuntu'
run: |
sudo apt-get update
sudo apt-get install -y portaudio19-dev
- name: Remove git LFS to avoid accidental large downloads
if: matrix.os == 'ubuntu'
run: sudo rm -f /usr/bin/git-lfs

Comment thread pyproject.toml
@@ -451,7 +521,7 @@ env = [
"GOOGLE_MAPS_API_KEY=AIzafake_google_key",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 slow-marked tests now run on standard GH runners after marker rename

The addopts filter changed from not (tool or slow or mujoco) to not (tool or self_hosted or mujoco), but several test files that were NOT changed in this PR still use @pytest.mark.slow instead of @pytest.mark.self_hosted. These tests will now be collected and run on ubuntu/macOS GH runners, where the infrastructure they depend on is unavailable.

Confirmed affected files (not in this diff): dimos/core/test_async_module_rpc.py (line 76) uses pLCMTransport/LCM multicast; dimos/core/test_async_module_dispatch_serialization.py (lines 75, 148, 189) same; dimos/agents/mcp/test_tool_stream.py (lines 153, 207) uses LLM API (agent_setup) with no skipif_no_openai guard — plus three more core test files. The slow marker is also no longer registered in conftest.py, so pytest will emit unknown-marker warnings. Rename the remaining @pytest.mark.slow decorators to @pytest.mark.self_hosted (or add skipif_no_* guards where appropriate) to restore the intended test-split.

Comment thread .github/workflows/ci.yml
Comment on lines +226 to +229
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@release/v1
with:
jobs: ${{ toJSON(needs) }}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 alls-green will fail whenever self-hosted-tests is skipped. re-actors/alls-green treats every job listed in needs as "voting" by default — a skipped result exits non-zero just like a failure. Because self-hosted-tests is intentionally skipped for draft PRs and fork PRs, ci-complete will always turn red for those, blocking any branch-protection rule that requires it to pass.

Suggested change
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@release/v1
with:
jobs: ${{ toJSON(needs) }}
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@release/v1
with:
allowed-skips: self-hosted-tests
jobs: ${{ toJSON(needs) }}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants