Split tests off of self hosted runner#1901
Conversation
| - name: Install dependency for pyaudio | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y portaudio19-dev | ||
| - name: Remove git LFS to avoid accidental large downloads | ||
| run: sudo rm -f /usr/bin/git-lfs |
There was a problem hiding this comment.
Both
sudo apt-get install portaudio19-dev and sudo rm -f /usr/bin/git-lfs are Linux-only commands, but the matrix includes macos-latest and windows-latest runners. apt-get does not exist on macOS or Windows, so every non-Ubuntu matrix cell will fail before tests even start. Both steps need an Ubuntu guard.
| - name: Install dependency for pyaudio | |
| run: | | |
| sudo apt-get update | |
| sudo apt-get install -y portaudio19-dev | |
| - name: Remove git LFS to avoid accidental large downloads | |
| run: sudo rm -f /usr/bin/git-lfs | |
| - name: Install dependency for pyaudio | |
| if: matrix.os == 'ubuntu' | |
| run: | | |
| sudo apt-get update | |
| sudo apt-get install -y portaudio19-dev | |
| - name: Remove git LFS to avoid accidental large downloads | |
| if: matrix.os == 'ubuntu' | |
| run: sudo rm -f /usr/bin/git-lfs |
| @@ -451,7 +521,7 @@ env = [ | |||
| "GOOGLE_MAPS_API_KEY=AIzafake_google_key", | |||
There was a problem hiding this comment.
slow-marked tests now run on standard GH runners after marker rename
The addopts filter changed from not (tool or slow or mujoco) to not (tool or self_hosted or mujoco), but several test files that were NOT changed in this PR still use @pytest.mark.slow instead of @pytest.mark.self_hosted. These tests will now be collected and run on ubuntu/macOS GH runners, where the infrastructure they depend on is unavailable.
Confirmed affected files (not in this diff): dimos/core/test_async_module_rpc.py (line 76) uses pLCMTransport/LCM multicast; dimos/core/test_async_module_dispatch_serialization.py (lines 75, 148, 189) same; dimos/agents/mcp/test_tool_stream.py (lines 153, 207) uses LLM API (agent_setup) with no skipif_no_openai guard — plus three more core test files. The slow marker is also no longer registered in conftest.py, so pytest will emit unknown-marker warnings. Rename the remaining @pytest.mark.slow decorators to @pytest.mark.self_hosted (or add skipif_no_* guards where appropriate) to restore the intended test-split.
| - name: Decide whether the needed jobs succeeded or failed | ||
| uses: re-actors/alls-green@release/v1 | ||
| with: | ||
| jobs: ${{ toJSON(needs) }} |
There was a problem hiding this comment.
alls-green will fail whenever self-hosted-tests is skipped. re-actors/alls-green treats every job listed in needs as "voting" by default — a skipped result exits non-zero just like a failure. Because self-hosted-tests is intentionally skipped for draft PRs and fork PRs, ci-complete will always turn red for those, blocking any branch-protection rule that requires it to pass.
| - name: Decide whether the needed jobs succeeded or failed | |
| uses: re-actors/alls-green@release/v1 | |
| with: | |
| jobs: ${{ toJSON(needs) }} | |
| - name: Decide whether the needed jobs succeeded or failed | |
| uses: re-actors/alls-green@release/v1 | |
| with: | |
| allowed-skips: self-hosted-tests | |
| jobs: ${{ toJSON(needs) }} |
This splits linting and tests into separate jobs on standard GH runners.
Only the tests that need ROS, LFS or something similar are now run on self-hosted runner.
This speeds up the CI significantly (GH runner tests also get to use xdist to parallelise the tests).
This also allows external PRs to be tested and merged normally, with just the self-hosted tests being skipped.
Also some security improvements and improving maintainability of the code.