Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
980c65a
Add dc-import-info support skill
rohitkumarbhagat Jul 30, 2026
332d819
Add repository import catalog
rohitkumarbhagat Jul 31, 2026
85ce82d
Improve import support diagnostics and references
rohitkumarbhagat Jul 31, 2026
d10291e
Add import infrastructure review gate
rohitkumarbhagat Jul 31, 2026
1051bb6
Replace import snapshot collector with recipes
rohitkumarbhagat Jul 31, 2026
885d552
fix: validate script existence before path resolution in run_python.sh
rohitkumarbhagat Aug 3, 2026
47b16b3
Add fuzzy import catalog queries
rohitkumarbhagat Aug 3, 2026
b36cc9f
chore: add comment to clarify import name contract alignment
rohitkumarbhagat Aug 3, 2026
9300988
Add import run correlation utility
rohitkumarbhagat Aug 3, 2026
1cc0df3
feat: add import-environments.yaml configuration for prod and staging…
rohitkumarbhagat Aug 3, 2026
00c04ca
refactor: standardize environment resolution by replacing hardcoded s…
rohitkumarbhagat Aug 3, 2026
145e3b6
refactor: remove redundant import configuration fields and standardiz…
rohitkumarbhagat Aug 3, 2026
a423a6d
refactor: decouple version discovery from history scanning and update…
rohitkumarbhagat Aug 3, 2026
62db310
refactor: scope dc-import-info skill to ET phase, cleanup deprecated …
rohitkumarbhagat Aug 4, 2026
2c832d2
test: normalize text matching and update ET lifecycle documentation f…
rohitkumarbhagat Aug 4, 2026
7a42170
refactor: standardize CLI flag definition using absl.flags and add in…
rohitkumarbhagat Aug 4, 2026
228e5f3
refactor: reorganize import automation documentation and replace corr…
rohitkumarbhagat Aug 4, 2026
51fdb7d
feat: include exact GCS version URI in import summary list responses …
rohitkumarbhagat Aug 4, 2026
8e8f2ab
feat: add dc-import-info prompt and formalize recipe-based command gr…
rohitkumarbhagat Aug 4, 2026
63c3056
refactor: restructure import automation documentation and relocate st…
rohitkumarbhagat Aug 4, 2026
c1939d0
refactor: standardize GCS import recipes, replace Artifact Registry r…
rohitkumarbhagat Aug 4, 2026
ef1418d
feat: add trace-batch-job-source-commit support and update GCS summar…
rohitkumarbhagat Aug 4, 2026
f85b789
refactor: update Batch list-tasks logic, migrate script paths, and cl…
rohitkumarbhagat Aug 4, 2026
e70a085
chore: update dependency versions in requirements.txt
rohitkumarbhagat Aug 4, 2026
a88f6ab
fix: update gcp auth fallback logic and improve log filtering recipes…
rohitkumarbhagat Aug 4, 2026
fe1cb38
docs: update trace-batch-job-source-commit to include log-resolved di…
rohitkumarbhagat Aug 4, 2026
468cabd
refactor: modularize agent dependency management with centralized rea…
rohitkumarbhagat Aug 4, 2026
ded1c47
docs: add shared GCP logging reference and refactor Batch logs recipe…
rohitkumarbhagat Aug 4, 2026
011781d
fix: sanitize import names via stripping and improve scheduler job bo…
rohitkumarbhagat Aug 4, 2026
465ab30
refactor: formalize evidence selection logic and validate import auto…
rohitkumarbhagat Aug 5, 2026
7eb9a75
refactor: simplify contract testing to focus on validating structural…
rohitkumarbhagat Aug 5, 2026
7bb1f7f
Merge branch 'master' into dc-import-info
rohitkumarbhagat Aug 5, 2026
99e231e
feat: replace dc-import-info prompt with a new starter prompt and upd…
rohitkumarbhagat Aug 5, 2026
3629f13
refactor: dynamically load test requirements from check_dependencies.…
rohitkumarbhagat Aug 5, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .agents/skills.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"entries": [
{
"path": "agents/skills/dc-import-info"
}
]
}
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,9 @@ import-automation/executor/config_override.json

.venv/

# Track only the Antigravity skill registry under .agents. The directory can
# otherwise be ignored by a developer's global Git configuration.
!.agents/
.agents/*
!.agents/skills.json

15 changes: 15 additions & 0 deletions agents/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Agents

This directory contains repository-owned agent skills, shared references,
recipes, configuration, and support scripts.

For local tools, Python dependencies, Google Cloud authentication, and the
optional sibling checkout, see [dependency setup](dependency-setup.md).

## Inspect imports

For read-only ET import information, use the
[`dc-import-info` starter prompt](prompts/dc-import-info-starter.md). Copy the
prompt into the agent conversation and append the specific import question.
The prompt routes the request through the repository-owned `dc-import-info`
skill and its bounded recipes.
208 changes: 208 additions & 0 deletions agents/check_dependencies.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
#!/bin/bash

# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -uo pipefail

# Add required executables here; the generic loop checks each with command -v.
REQUIRED_COMMANDS=(
bash
curl
git
gcloud
jq
python3
realpath
sed
)

# Add exact gcloud operations here; the generic loop appends --help.
GCLOUD_COMMANDS=(
'artifacts docker images describe'
'auth list'
'auth print-access-token'
'auth application-default print-access-token'
'batch jobs describe'
'batch tasks list'
'logging read'
'scheduler jobs describe'
'spanner databases execute-sql'
'storage cat'
'storage objects list'
)

function usage {
printf '%s\n' \
'Usage: ./agents/check_dependencies.sh [--local|--help]' \
'' \
'With no flag, run local dependency checks followed by authentication checks.' \
'Use --local to skip authentication checks.'
}

run_auth=true
if [[ $# -eq 1 && "$1" == '--local' ]]; then
run_auth=false
elif [[ $# -eq 1 && "$1" == '--help' ]]; then
usage
exit 0
elif [[ $# -ne 0 ]]; then
usage >&2
exit 2
fi

repo_root="$PWD"
for required_path in statvar_imports scripts import-automation requirements_all.txt run_tests.sh; do
if [[ ! -e "$repo_root/$required_path" ]]; then
echo 'Run this command from the Data Commons data repository root.' >&2
exit 2
fi
done

local_failures=0
gcloud_available=true
git_available=true
realpath_available=true

for command_name in "${REQUIRED_COMMANDS[@]}"; do
if ! command -v "$command_name" >/dev/null 2>&1; then
echo "MISSING command $command_name" >&2
echo "SEE agents/dependency-setup.md#system-tools" >&2
local_failures=$((local_failures + 1))
case "$command_name" in
gcloud) gcloud_available=false ;;
git) git_available=false ;;
realpath) realpath_available=false ;;
esac
fi
done

if [[ $local_failures -eq 0 ]]; then
echo 'PASS Required command-line tools'
fi

if [[ "$gcloud_available" == true ]]; then
gcloud_version=''
if gcloud_version="$(gcloud version 2>/dev/null)"; then
gcloud_version="${gcloud_version%%$'\n'*}"
if [[ -n "$gcloud_version" ]]; then
echo "PASS $gcloud_version"
else
echo 'FAILED gcloud version returned no output' >&2
local_failures=$((local_failures + 1))
fi
else
echo 'FAILED gcloud version' >&2
echo 'SEE agents/dependency-setup.md#gcloud-cli' >&2
local_failures=$((local_failures + 1))
fi

gcloud_command_failures=0
for command_spec in "${GCLOUD_COMMANDS[@]}"; do
command_parts=()
read -r -a command_parts <<< "$command_spec"
if ! gcloud "${command_parts[@]}" --help >/dev/null 2>&1; then
echo "MISSING gcloud $command_spec" >&2
echo 'SEE agents/dependency-setup.md#gcloud-cli' >&2
gcloud_command_failures=$((gcloud_command_failures + 1))
fi
done
if [[ $gcloud_command_failures -eq 0 ]]; then
echo 'PASS Required gcloud commands'
else
local_failures=$((local_failures + gcloud_command_failures))
fi
else
echo 'NOT_RUN gcloud version and command checks' >&2
fi

python_bin="$repo_root/.env/bin/python"
python_checker="$repo_root/agents/common/scripts/check_python_dependencies.py"
if [[ ! -x "$python_bin" ]]; then
echo 'MISSING Python agent environment' >&2
echo 'RUN ./run_tests.sh -r' >&2
local_failures=$((local_failures + 1))
elif [[ ! -f "$python_checker" ]]; then
echo 'MISSING agents/common/scripts/check_python_dependencies.py' >&2
local_failures=$((local_failures + 1))
elif ! "$python_bin" "$python_checker"; then
local_failures=$((local_failures + 1))
fi

import_repository="$repo_root/../import"
if [[ ! -d "$import_repository" ]]; then
echo 'SUGGESTED sibling import checkout at ../import'
elif [[ "$git_available" != true || "$realpath_available" != true ]]; then
echo 'SUGGESTED sibling import checkout could not be validated'
else
expected_import_root="$(realpath "$import_repository" 2>/dev/null || true)"
actual_import_root="$(
git -C "$import_repository" rev-parse --show-toplevel 2>/dev/null || true
)"
if [[ -n "$actual_import_root" ]]; then
actual_import_root="$(realpath "$actual_import_root" 2>/dev/null || true)"
fi
workflow_source="$import_repository/pipeline/workflow/import-automation-workflow.yaml"
if [[ -n "$expected_import_root" && "$actual_import_root" == "$expected_import_root" && -f "$workflow_source" ]]; then
echo 'AVAILABLE sibling import checkout'
else
echo 'SUGGESTED sibling import checkout at ../import is invalid'
fi
fi

if [[ $local_failures -ne 0 ]]; then
echo 'NOT_RUN Authentication checks' >&2
exit 1
fi

if [[ "$run_auth" != true ]]; then
echo 'NOT_RUN Authentication checks (--local)'
exit 0
fi

function has_nonempty_output {
"$@" --quiet 2>/dev/null |
"$python_bin" -c \
'import sys; raise SystemExit(0 if sys.stdin.read().strip() else 1)'
}

auth_failures=0
if has_nonempty_output gcloud auth list \
--filter='status:ACTIVE' --format='value(account)'; then
if has_nonempty_output gcloud auth print-access-token; then
echo 'PASS gcloud CLI authentication'
else
echo 'FAILED gcloud CLI authentication' >&2
echo 'SEE agents/dependency-setup.md#gcloud-cli-authentication' >&2
auth_failures=$((auth_failures + 1))
fi
else
echo 'FAILED No active gcloud account' >&2
echo 'SEE agents/dependency-setup.md#gcloud-cli-authentication' >&2
auth_failures=$((auth_failures + 1))
fi

if has_nonempty_output gcloud auth application-default print-access-token; then
echo 'PASS Application Default Credentials'
else
echo 'FAILED Application Default Credentials' >&2
echo 'SEE agents/dependency-setup.md#application-default-credentials' >&2
auth_failures=$((auth_failures + 1))
fi

if [[ $auth_failures -ne 0 ]]; then
exit 1
fi

echo 'NOT_RUN Cloud resource permissions'
51 changes: 51 additions & 0 deletions agents/common/config/import-environments.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Repository-configured defaults used by dc-import-info. Explicit values in a
# request override the corresponding field for that request.

default_environment: prod

environments:
prod:
scheduler:
project: datcom-import-automation-prod
location: us-central1

workflow:
project: datcom-import-automation-prod
location: us-central1
import_workflow: import-automation-workflow

batch:
project: datcom-import-automation-prod
location: us-central1

gcs:
client_project: datcom-204919
output_bucket: datcom-prod-imports

spanner:
project: datcom-store
instance: dc-graph-staging
database: dc_graph_1

staging:
scheduler:
project: datcom-ci
location: us-central1

workflow:
project: datcom-ci
location: us-central1
import_workflow: import-automation-workflow

batch:
project: datcom-ci
location: us-central1

gcs:
client_project: datcom-ci
output_bucket: datcom-ci-test

spanner:
project: datcom-ci
instance: datcom-spanner-test
database: dc-test-db
55 changes: 55 additions & 0 deletions agents/common/recipes/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Import-support recipe organization

Recipes are bounded, read-only operations. Keep each recipe small enough that
an agent can load only the command needed for the requested fact.

```text
recipes/
├── README.md
├── local/
│ └── list-imports.md
└── gcp/
├── batch/
├── gcs/
├── logging/
├── scheduler/
└── spanner/
```

## Placement

- Put the recipe for a repository or local operation that makes no cloud call
in `local/`.
- Put the recipe for a cloud operation in `gcp/<service>/`, named for the
primary GCP service it reads.
- Keep operations over several objects from the same service in that service
folder.
- Keep Python helper implementations in `agents/common/scripts/`. Place the
recipe that invokes a helper under `local/` or `gcp/<service>/` according to
its primary operation. For example, the recipe for a helper that lists GCS
summaries belongs in `gcp/gcs/`.
- Do not add a general `imports/` folder; it does not identify the execution
boundary or cloud service.

## Composition

Compose a cross-service evidence path from atomic recipes. A recipe may link
to a recipe in another service folder when an observed exact identifier can
seed that operation, but it must not copy the other service's commands or run
the linked operation automatically.

A product recipe may apply a shared service reference for generic command
syntax. It must supply the complete product-specific parameters, bounds,
output fields, and interpretation.

The [import evidence flow](../references/import-automation/import-evidence-flow.md)
owns the end-to-end navigation sequence. Upstream skills and playbooks link
directly to operational recipes and do not load this README during normal
execution.

## Recipe contract

Every Markdown file below `local/` or `gcp/` is an operational recipe. It must
define when to use it, required inputs, clarification conditions, its exact
read-only operation, preferred invocation, bounded output, retained evidence,
common failures, and related sources.
Loading
Loading