Fix Datastore emulator failures caused by unsupported != query filters - #5418
Draft
letitz wants to merge 2 commits into
Draft
Fix Datastore emulator failures caused by unsupported != query filters#5418letitz wants to merge 2 commits into
letitz wants to merge 2 commits into
Conversation
…lation conditional on rodete When running local environment setup (./local/install_deps.bash) on internal Google Linux workstations (rodete), apt-get fails when attempting to install several optional App Engine and Pub/Sub component packages that are not provided in internal apt repositories. Furthermore, attempting to add the docker-ce repository triggers interactive glogin authentication even when docker is already installed on the system. Specifically: - google-cloud-cli-app-engine-python, google-cloud-cli-app-engine-python-extras, google-cloud-cli-app-engine-go, and google-cloud-cli-pubsub-emulator apt packages are optional/unavailable on rodete (see internal references b/414408644 and b/484368884). - glogin and glinux-add-repo docker-ce-rodete should be skipped if docker is already installed on the machine. - Fix pipenv CLI invocation in local/install_python_deps_linux.bash. This commit updates local/install_deps_linux.bash to check $distro_codename and whether docker is installed. On rodete, only google-cloud-cli and google-cloud-cli-datastore-emulator are installed via apt-get, while non-rodete distributions retain the full component package list. Additionally adds local/tests/install_deps_test_linux.bash to test and verify fresh checkout setup on Linux.
When running App Engine unit tests against the local Cloud SDK Datastore
emulator, NDB queries containing inequality filters (!=) fail because
the Java emulator backend does not support gRPC filter operator 9 (NOT_EQUAL).
This causes tests such as oss_fuzz_apply_ccs_test.py and
fuzzer_and_job_weights_test.py to fail with the following traceback:
google.api_core.exceptions.RetryError: Maximum number of 3 retries exceeded while calling <function make_call.<locals>.rpc_call at 0x7f3527cf3240>, last exception: google.api_core.exceptions.Unknown
This commit replaces != query filters in NDB Datastore query objects across
cron handlers (oss_fuzz_apply_ccs.py, fuzzer_and_job_weights.py,
chrome_tests_syncer.py, retry_stuck_tasks.py) with supported inequality filters
(> and range inequality ndb.OR queries).
letitz
force-pushed
the
fix-emulator-queries
branch
from
August 11, 2026 03:01
dfcb746 to
97cee57
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When running unit tests locally against the Cloud Datastore Emulator (
CloudDatastore.jar), multiple cron background task unit tests fail with gRPC / Datastore unknown errors orjava.lang.IllegalArgumentException.Root Cause:
google-cloud-ndbqueries that filter string or numeric properties using the!=(not equal) operator generate gRPC filter operator 9 (PropertyFilter.Operator.NOT_EQUAL). The Java-based Cloud Datastore Emulator (CloudDatastore.jar) does not support filter operator 9, throwingjava.lang.IllegalArgumentException, which surfaces in Python unit tests asgoogle.api_core.exceptions.RetryErrorwrappingUnknown.Fix:
Replaced
!=query filters across 4 cron modules with equivalent supported filters:""(empty string) is the lexicographical minimum value (bug_information), replacedbug_information != ""withbug_information > "".prop != valuewith range inequality filters usingndb.OR(prop < value, prop > value).