Context
We're downloading and compiling Go dependencies for Kratix twice: once during the unit test stage and again during the image building stages. If we run our unit tests inside Docker, we can cache downloaded Go modules and propagate those caches to the image building stages, avoiding the need to redownload and recompile what we already have and saving some time in building these images during test and release processes.
Experimental data show that the Github Actions Cache backend yields faster results than the approach of carrying the cache from stage to stage through build artefacts (as we do for images), so we should prefer this approach.
Motivation
Using this run as a reference, we can see that running the unit tests takes around 1m14s, with most of it dedicated to downloading and compiling Go dependencies and the Kratix code. Previous work to enable caching for building the Kratix code for unit tests removed around 3 minutes or its time by avoiding repeated downloads and recompilation. The steps building the Kratix image (2m19s) and the quickstart installer image (1m52s) perform many of the same steps over the same or similar dependency list. The assumption is that reusing caches will allow us to reduce image build times by half. These savings multiply by the number of builds/releases/and PRs we do and also propagates to downstream builds triggered by the Kratix CI flow (which could potentially reuse the same caches).
Scope of the work
- Dockerfiles need to be modified to use named caches, to have separate
go mod download from the go build step, to add the local source code to the image instead of mounting it, to avoid using the -a flag on builds, and to have an extra test stage that can be triggered during unit tests.
- Makefiles need to be modified to use the
gha cache backend when running under CI.
- Workflows need to be modified to save and restore the buildkit cache between runs.
Open questions
- How wide to we want to scope our caches? (Per repository, per branch, per PR, per branch + workflow name, etc)
- Github Actions Cache allows 10GB of caches per repository before starting to remove older caches. Do we want to add any extra bits such as proactive monitoring or cache cleaning or are we happy to trust the system?
Acceptance
References
Context
We're downloading and compiling Go dependencies for Kratix twice: once during the unit test stage and again during the image building stages. If we run our unit tests inside Docker, we can cache downloaded Go modules and propagate those caches to the image building stages, avoiding the need to redownload and recompile what we already have and saving some time in building these images during test and release processes.
Experimental data show that the Github Actions Cache backend yields faster results than the approach of carrying the cache from stage to stage through build artefacts (as we do for images), so we should prefer this approach.
Motivation
Using this run as a reference, we can see that running the unit tests takes around 1m14s, with most of it dedicated to downloading and compiling Go dependencies and the Kratix code. Previous work to enable caching for building the Kratix code for unit tests removed around 3 minutes or its time by avoiding repeated downloads and recompilation. The steps building the Kratix image (2m19s) and the quickstart installer image (1m52s) perform many of the same steps over the same or similar dependency list. The assumption is that reusing caches will allow us to reduce image build times by half. These savings multiply by the number of builds/releases/and PRs we do and also propagates to downstream builds triggered by the Kratix CI flow (which could potentially reuse the same caches).
Scope of the work
go mod downloadfrom thego buildstep, to add the local source code to the image instead of mounting it, to avoid using the-aflag on builds, and to have an extra test stage that can be triggered during unit tests.ghacache backend when running under CI.Open questions
Acceptance
References