When using RECC in a nested remote execution setting, the failures are cached and retrying the build will always hit the same "failure". While this might make sense for genuine failures, it is problematic for failures related to resource exhaustion.
buildbox-casd added an option to disable caching failures (https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/1063). So we can enable it when starting buildbox-casd. The only issue is that it needs a newer version (1.3.41, released October 2025) of buildbox-casd than the current minimum we support.
We could either bump the minimal supported version, or check for the buildbox-casd version and only enable this option if we have a recent enough buildbox-casd. It's a bit of a trade off: bumping the version affects everyone, even if they don't use nested remote execution. Enabling the option only if buildbox-casd is recent enough risks people hitting the bug without knowing that it would be fixed by an upgrade.
I'm leaning towards bumping the minimal version.
When using RECC in a nested remote execution setting, the failures are cached and retrying the build will always hit the same "failure". While this might make sense for genuine failures, it is problematic for failures related to resource exhaustion.
buildbox-casd added an option to disable caching failures (https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/1063). So we can enable it when starting buildbox-casd. The only issue is that it needs a newer version (1.3.41, released October 2025) of buildbox-casd than the current minimum we support.
We could either bump the minimal supported version, or check for the buildbox-casd version and only enable this option if we have a recent enough buildbox-casd. It's a bit of a trade off: bumping the version affects everyone, even if they don't use nested remote execution. Enabling the option only if buildbox-casd is recent enough risks people hitting the bug without knowing that it would be fixed by an upgrade.
I'm leaning towards bumping the minimal version.