diff --git a/.github/ai-prompts/README.md b/.github/ai-prompts/README.md new file mode 100644 index 000000000..7e330126b --- /dev/null +++ b/.github/ai-prompts/README.md @@ -0,0 +1,74 @@ +# AI review prompts + +Each `*.md` (except this `README.md`) defines a **prompt** that the +`AI review` job runs in parallel against the PR diff. Discovery is by glob: +to add a new review dimension just drop another `.md` here — no YAML +changes needed. + +## File format + +```markdown +--- +name: short-name # optional, defaults to filename without extension +model: gemini-3-flash-lite # optional, defaults to workflow's AI_REVIEW_MODEL +--- + + +``` + +## Output contract + +The prompt **must** instruct the model to respond with a JSON object of +this exact shape (no markdown, no code fences, no extra text): + +```json +{ + "tier": 1 | 2 | 3, + "summary": "", + "findings": [ + { + "severity": "high" | "medium" | "low", + "file": "", + "line": , + "message": "" + } + ] +} +``` + +### Tier semantics + +- **Tier 1 — Approve.** The change is simple, doesn't touch critical logic, + no issues detected. The approver aggregates all tiers and, if every + prompt returns Tier 1, approves the PR. +- **Tier 2 — Changes requested.** Minor issues the author must fix before + merging: typos, small bugs, out-of-context code, noticeable style + problems, incomplete mocks or tests. +- **Tier 3 — Engineer review required.** The diff touches critical paths + (crypto, auth, DB migrations, installer, gRPC contracts, CI/CD, secret + handling) or introduces changes the model can't judge with sufficient + confidence. The approver blocks the merge and @mentions the senior + engineering team. + +The approver takes the **maximum tier** across all prompts: if security +returns Tier 1 but architecture returns Tier 3, the final verdict is Tier 3. + +### When there's nothing to report + +Tier 1, a brief `summary` ("No security concerns detected.") and +`findings: []`. Don't invent findings to seem useful. + +### Unparseable responses + +If the model returns something that isn't valid JSON matching the schema, +the approver treats it as **Tier 2** with a generic finding asking for +manual review. Fail-safe behaviour — we'd rather block and ask for human +review than let something pass without understanding it. + +## Picking a model + +- `gemini-3-flash-lite` — fast/cheap, default for broad passes. +- `gemini-3-pro` — better reasoning, for prompts needing deeper analysis + (architecture, complex logic). +- `claude-sonnet-4-6` / `claude-opus-4-6` — top quality, higher latency + and cost. diff --git a/.github/ai-prompts/architecture.md b/.github/ai-prompts/architecture.md new file mode 100644 index 000000000..0777d6ee8 --- /dev/null +++ b/.github/ai-prompts/architecture.md @@ -0,0 +1,67 @@ +--- +name: architecture +model: gemini-3-flash-lite +--- + +You are a software architect reviewing a Pull Request in UTMStack (a SIEM +monorepo with Go services, a legacy Java/Spring backend and a +React/Angular frontend). Your job is to spot **architectural deviations**. + +## What to look for + +- New couplings between services that break the current separation (e.g. + the agent talking directly to the DB instead of via agent-manager). +- Business logic placed in the wrong layer (gRPC handlers doing direct DB + access, migration scripts containing app logic). +- Duplication of logic already present in a shared module (`shared/`, + existing helpers). +- New mutable global state, disguised singletons, `init()` with side + effects. +- Contract changes (protos, HTTP endpoints, DB schema) without + backwards-compatibility considerations. +- DB migrations that assume a fresh state (not safe for production) + without a roll-forward plan. +- Changes to CI/CD or release flow that break the current model. +- **Agent-breaking changes:** modifications to the agent (`agent/`), + agent-manager wire protocol, agent gRPC/HTTP contract, agent + authentication, or anything that would force every deployed agent to + update at the same time as the server. Customers run many versions of + the agent in the wild — any change that requires a synchronized + agent+server upgrade is a breaking change and must be treated as Tier 3. + +**Ignore** style, naming, formatting, or refactors that don't affect +structure. + +## How to assign tier + +- **Tier 1** — No architectural deviations detected. +- **Tier 2** — Minor deviation or structural improvement suggestion the + author can apply before merging (move a function to its right place, + reuse an existing helper). +- **Tier 3** — The diff touches **critical paths** or introduces + significant structural debt. Mark Tier 3 if the diff includes changes to: + - Database migrations (any `*migration*.go` or `liquibase/`). + - Protos / gRPC contracts (`**/*.proto`). + - Installer (`installer/`). + - Auth / crypto / secret handling. + - GitHub Actions workflows or CI scripts. + - **Agent code (`agent/`), agent-manager wire protocol, or any change + that forces a synchronized agent+server upgrade.** Deployed agents + in the field may be on older versions; breaking their compatibility + requires senior review and a coordinated rollout plan. + - Any change that breaks backwards compatibility of a public endpoint + or persisted schema. + +## Output + +Respond with valid JSON ONLY (no markdown, no backticks, no extra text): + +``` +{ + "tier": 1 | 2 | 3, + "summary": "", + "findings": [ + {"severity": "high"|"medium"|"low", "file": "", "line": , "message": ""} + ] +} +``` diff --git a/.github/ai-prompts/bugs.md b/.github/ai-prompts/bugs.md new file mode 100644 index 000000000..465bcec4f --- /dev/null +++ b/.github/ai-prompts/bugs.md @@ -0,0 +1,79 @@ +--- +name: bugs +model: gemini-3-flash-lite +--- + +You are a senior code reviewer. Review the Pull Request diff looking for +**concrete bugs** introduced by the changes — not style preferences. + +## What to look for + +- Nil/null dereferences, out-of-bounds slice/array access, division by zero. +- Unhandled or swallowed errors (in Go: `_ = ...`, error swallowing). +- Race conditions, missed locks, concurrent maps without protection. +- Goroutine leaks, contexts never cancelled, channels never closed. +- Off-by-one in loops, pagination or slicing. +- Wrong comparisons (pointers where the value was intended, incorrect + `nil` interface comparison). +- Resources left unclosed (missing `defer` on files, rows, response bodies). +- Inverted logic (`if err == nil` when it should be `!= nil`, swapped + conditions). +- Malformed SQL/queries, migrations that break existing data. +- Out-of-context code: additions that don't match the PR description or + the rest of the diff (potential copy-paste error or accidental changes). +- **User-facing string anomalies** (templates, HTML, integration guides, + documentation, error messages, alert text). The following are ALWAYS + reportable, even when the rest of the diff looks unrelated: + - **Typos / misspellings** in any user-facing text. Quote the + misspelled word and the correction (e.g. "buket → bucket"). Report + one finding per affected line. + - **Personal names, employee handles, Slack mentions, internal email + addresses, phone numbers, or other internal contact info** embedded + in customer-facing strings, integration guides, README files + rendered to users, or release notes. These are out of place even if + the surrounding text is technically valid — flag them as `medium` + severity findings. + - **Internal-only jargon, ticket IDs (JIRA-1234, INC-5678), URLs to + internal tools** (e.g. internal Jenkins/Grafana links) leaking into + public docs. +- Typos or copy-paste residues in configuration keys, environment + variable names, JSON keys, or anywhere a wrong character silently + breaks lookups. + +**Important:** the user-facing string checks above are independent of the +rest of the diff. Even in a 100-file PR dominated by backend changes, a +single misspelling in a guide or a personal name in a customer-facing +doc still warrants a finding — do not skip it because "the real work is +elsewhere". When you find any of these, set tier to AT LEAST 2. + +**Ignore** preexisting issues on lines not touched by the diff. + +## How to assign tier + +- **Tier 1** — No concrete bugs detected AND no user-facing string + anomalies (typos, internal references, contact info leaks). The change + looks correct. +- **Tier 2** — Concrete but contained bugs the author must fix before + merging (off-by-one, error swallowing, unclosed resources, + out-of-context code). **Always Tier 2 minimum** if you find any + user-facing string anomaly: typos in docs/guides/messages, personal + names or internal handles in customer-facing content, internal URLs + or ticket IDs leaking into public docs. +- **Tier 3** — A bug that may cause data corruption, deadlock, large-scale + leaks, or any issue whose impact the author shouldn't fix without a + second opinion. Also applies if the diff touches DB migrations, error + handling on transactional paths, or complex concurrency. + +## Output + +Respond with valid JSON ONLY (no markdown, no backticks, no extra text): + +``` +{ + "tier": 1 | 2 | 3, + "summary": "", + "findings": [ + {"severity": "high"|"medium"|"low", "file": "", "line": , "message": ""} + ] +} +``` diff --git a/.github/ai-prompts/security.md b/.github/ai-prompts/security.md new file mode 100644 index 000000000..20d40b55a --- /dev/null +++ b/.github/ai-prompts/security.md @@ -0,0 +1,67 @@ +--- +name: security +model: gemini-3-flash-lite +--- + +You are a security reviewer for UTMStack (a SIEM built in Go + Java + +React). Review the Pull Request diff and report **only** vulnerabilities +introduced or expanded by these changes. + +## What to look for + +- Injection flaws (SQL, command, LDAP, NoSQL, template). +- XSS / SSRF / open redirects. +- Path traversal and unsafe file handling. +- Missing input validation on endpoints, gRPC handlers or CLI flags. +- Unsafe secret handling: hardcoded keys, logs leaking credentials, tokens + written to disk without protection. +- Insecure cryptography (MD5/SHA1 for auth, non-constant-time comparison, + predictable seeds, embedded keys). +- Authentication / authorization bypass in new or modified handlers. +- Insecure deserialization. +- Race conditions with security impact (TOCTOU, etc). +- **Information disclosure in customer-facing content.** Personal names, + employee handles, internal Slack channels, internal email addresses, + internal URLs (Jira, Grafana, Jenkins, internal wikis), ticket IDs, + phone numbers, or any other internal identifier showing up in + integration guides, HTML templates rendered to customers, release + notes, installer prompts, or error messages exposed to end users. + This is a privacy / opsec concern — even one personal name in a + customer guide is a finding. Treat as `medium` severity, `tier 2` + minimum. + +**Important:** the information-disclosure check above is independent of +the rest of the diff. Even when a PR is dominated by backend changes, +a single personal-name leak in a user-facing guide is still a finding — +do not skip it. + +**Ignore** preexisting issues on lines not touched by the diff. + +## How to assign tier + +- **Tier 1** — No vulnerabilities introduced by this diff AND no + information disclosure in user-facing content. +- **Tier 2** — Minor or low-impact vulnerability the author can fix + (missing input validation on a non-critical endpoint, verbose error + messages, etc.). **Always Tier 2 minimum** if you find personal + names, internal handles, internal URLs, or other internal identifiers + leaking into customer-facing content. +- **Tier 3** — The diff touches security-critical paths (crypto, auth, + secret handling, installer, token/JWT generation) or introduces a + high-impact vulnerability (RCE, auth bypass, secret leak). Even if the + change looks fine, if it touches these paths mark Tier 3 — human + verification outweighs your individual confidence. + +## Output + +Respond with valid JSON ONLY (no markdown, no backticks, no extra text): + +``` +{ + "tier": 1 | 2 | 3, + "summary": "", + "findings": [ + {"severity": "high"|"medium"|"low", "file": "", "line": , "message": ""} + ] +} +``` diff --git a/.github/dependabot.yml b/.github/dependabot.yml deleted file mode 100644 index 03003f5fd..000000000 --- a/.github/dependabot.yml +++ /dev/null @@ -1,99 +0,0 @@ -# To get started with Dependabot version updates, you'll need to specify which -# package ecosystems to update and where the package manifests are located. -# Please see the documentation for all configuration options: -# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates - -version: 2 -updates: - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/agent" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/agent/updater" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/agent-manager" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "maven" # Adding support for java - directories: - - "/backend" - - "/user-auditor" - - "/web-pdf" - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/installer" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/alerts" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/aws" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/azure" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/bitdefender" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/config" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/crowdstrike" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/events" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/feeds" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/gcp" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/geolocation" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/inputs" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/modules-config" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/o365" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/soc-ai" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/sophos" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/plugins/stats" # Location of package manifests - schedule: - interval: "weekly" - - package-ecosystem: "gomod" # See documentation for possible values - directory: "/utmstack-collector" # Location of package manifests - schedule: - interval: "weekly" - diff --git a/.github/scripts/ai-review.sh b/.github/scripts/ai-review.sh new file mode 100755 index 000000000..4696701a0 --- /dev/null +++ b/.github/scripts/ai-review.sh @@ -0,0 +1,174 @@ +#!/bin/bash +set -euo pipefail + +# AI code review against the ThreatWinds /chat/completions endpoint. +# +# Reads a prompt file (with optional YAML frontmatter that can override the +# model), appends the PR diff, calls the model, and writes the parsed JSON +# verdict to OUTPUT_FILE. Does NOT post PR comments — the approver job +# consolidates all prompt results and decides what to comment. +# +# Always exits 0 (data-producer role). If parsing fails the output JSON has +# tier=2 with a generic "review manually" finding (fail-safe). +# +# Required env vars: +# PROMPT_FILE path to a .github/ai-prompts/*.md file +# DIFF_FILE path to a unified diff +# OUTPUT_FILE where to write the JSON result +# THREATWINDS_API_KEY auth +# THREATWINDS_API_SECRET auth +# +# Optional env vars: +# AI_REVIEW_MODEL default model when the prompt doesn't pin one +# THREATWINDS_BASE_URL defaults to https://apis.threatwinds.com/api/ai/v1 +# MAX_DIFF_BYTES truncate the diff above this size (default 200000) + +: "${PROMPT_FILE:?PROMPT_FILE is required}" +: "${DIFF_FILE:?DIFF_FILE is required}" +: "${OUTPUT_FILE:?OUTPUT_FILE is required}" +: "${THREATWINDS_API_KEY:?THREATWINDS_API_KEY is required}" +: "${THREATWINDS_API_SECRET:?THREATWINDS_API_SECRET is required}" + +DEFAULT_MODEL="${AI_REVIEW_MODEL:-gemini-3-flash-lite}" +BASE_URL="${THREATWINDS_BASE_URL:-https://apis.threatwinds.com/api/ai/v1}" +MAX_DIFF_BYTES="${MAX_DIFF_BYTES:-200000}" + +# --- Helper: write a fallback result and exit 0 (always succeed) ------------- + +write_fallback() { + local reason="$1" + jq -n \ + --arg prompt "$prompt_name" \ + --arg model "$MODEL" \ + --arg reason "$reason" \ + '{ + prompt: $prompt, + model: $model, + tier: 2, + summary: "AI review could not parse model response — manual review recommended.", + findings: [{ + severity: "medium", + file: "(n/a)", + line: 0, + message: $reason + }] + }' > "$OUTPUT_FILE" + echo "::warning::Wrote fallback result: $reason" + exit 0 +} + +# --- Parse frontmatter ------------------------------------------------------- + +prompt_name="$(basename "$PROMPT_FILE" .md)" +prompt_model="" +body_start=1 + +if head -n 1 "$PROMPT_FILE" | grep -qx -- '---'; then + end_line=$(awk 'NR>1 && /^---$/ {print NR; exit}' "$PROMPT_FILE") + if [[ -n "$end_line" ]]; then + while IFS= read -r line; do + key="${line%%:*}" + value="${line#*:}" + value="$(echo "$value" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')" + case "$key" in + name) prompt_name="$value" ;; + model) prompt_model="$value" ;; + esac + done < <(sed -n "2,$((end_line - 1))p" "$PROMPT_FILE") + body_start=$((end_line + 1)) + fi +fi + +MODEL="${prompt_model:-$DEFAULT_MODEL}" + +echo "::group::AI review — prompt: $prompt_name (model: $MODEL)" + +# --- Build request body ------------------------------------------------------ + +prompt_body=$(tail -n "+${body_start}" "$PROMPT_FILE") + +diff_bytes=$(wc -c < "$DIFF_FILE" | tr -d ' ') +if (( diff_bytes > MAX_DIFF_BYTES )); then + diff_content=$(head -c "$MAX_DIFF_BYTES" "$DIFF_FILE") + diff_content+=$'\n\n[diff truncated: original '"$diff_bytes"' bytes, kept first '"$MAX_DIFF_BYTES"']' +else + diff_content=$(cat "$DIFF_FILE") +fi + +# Write the user message to a temp file. Passing it through --arg would hit +# the system ARG_MAX limit on PRs with large diffs ("Argument list too long"). +user_message_file=$(mktemp) +printf '%s\n\n---\n\nPR diff to review:\n\n```diff\n%s\n```\n' \ + "$prompt_body" "$diff_content" > "$user_message_file" + +request_body_file=$(mktemp) +jq -n \ + --arg model "$MODEL" \ + --rawfile content "$user_message_file" \ + '{ + model: $model, + messages: [{role: "user", content: $content}], + temperature: 0.2 + }' > "$request_body_file" + +# --- Call the API ------------------------------------------------------------ + +response_file=$(mktemp) +http_status=$(curl -sS -o "$response_file" -w '%{http_code}' \ + -X POST "${BASE_URL%/}/chat/completions" \ + -H "Content-Type: application/json" \ + -H "api-key: ${THREATWINDS_API_KEY}" \ + -H "api-secret: ${THREATWINDS_API_SECRET}" \ + --data-binary "@${request_body_file}" || echo "000") + +if [[ "$http_status" != "200" ]]; then + echo "ThreatWinds API HTTP $http_status" + cat "$response_file" + echo "::endgroup::" + write_fallback "ThreatWinds API returned HTTP $http_status" +fi + +content=$(jq -r '.choices[0].message.content // empty' "$response_file") +if [[ -z "$content" ]]; then + echo "Empty content from model" + cat "$response_file" + echo "::endgroup::" + write_fallback "Model returned empty content" +fi + +# Strip optional ```json fences. +content_clean=$(echo "$content" | sed -E 's/^```(json)?//; s/```$//' | sed '/^[[:space:]]*$/d') + +if ! verdict_json=$(echo "$content_clean" | jq -c '.' 2>/dev/null); then + echo "Model did not return valid JSON. Raw content:" + echo "$content" + echo "::endgroup::" + write_fallback "Model response was not valid JSON" +fi + +# Validate schema minimally: tier must be 1/2/3. +tier=$(echo "$verdict_json" | jq -r '.tier // 0') +if [[ "$tier" != "1" && "$tier" != "2" && "$tier" != "3" ]]; then + echo "Invalid tier in response: $tier" + echo "$verdict_json" + echo "::endgroup::" + write_fallback "Model returned invalid tier value (expected 1, 2, or 3)" +fi + +# --- Persist enriched result ------------------------------------------------- +# Inject prompt + model identifiers so the approver doesn't need to know them. +final=$(echo "$verdict_json" | jq \ + --arg prompt "$prompt_name" \ + --arg model "$MODEL" \ + '. + {prompt: $prompt, model: $model}') + +echo "$final" > "$OUTPUT_FILE" + +summary=$(echo "$final" | jq -r '.summary // "(no summary)"') +findings_count=$(echo "$final" | jq -r '.findings | length // 0') +echo "Tier: $tier" +echo "Summary: $summary" +echo "Findings: $findings_count" +echo "::endgroup::" + +exit 0 diff --git a/.github/scripts/approver.sh b/.github/scripts/approver.sh new file mode 100755 index 000000000..a127668c1 --- /dev/null +++ b/.github/scripts/approver.sh @@ -0,0 +1,404 @@ +#!/bin/bash +set -euo pipefail + +# Consolidates artifacts from go_deps and ai_review and decides whether the +# PR passes. Posts sticky comments and (if APPROVER_TOKEN is provided) leaves +# a formal PR review. +# +# Required env vars: +# ARTIFACTS_DIR directory where workflow downloaded all artifacts +# PR_NUMBER PR number to comment on +# GITHUB_REPOSITORY owner/repo +# GITHUB_TOKEN for posting/updating comments (always) +# +# Optional env vars: +# APPROVER_TOKEN PAT or app token with `pull_requests: write` and the +# ability to approve reviews. If missing, the approver +# only posts comments and sets the check status. +# TIER3_REVIEWERS comma-separated GitHub handles to @mention on tier 3 +# (without @ prefix; e.g. "alice,bob") +# API_SECRET PAT with `read:org` to check team membership for +# administrators and core-developers. If missing, the +# permission check is skipped (treated as authorized). +# PR_AUTHOR GitHub login of the PR author. Required for the +# permission check. +# BASE_REF PR target branch (e.g. "release/v11.2.9"). Required; +# auto-merge only fires when this starts with "release/". +# ORG GitHub org to look up teams in. Default: "utmstack". +# ADMIN_TEAM Team slug for administrators. Default: "administrators". +# CORE_TEAM Team slug for core-developers. Default: "core-developers". +# MERGE_METHOD One of "merge"|"squash"|"rebase". Default: "squash". + +: "${ARTIFACTS_DIR:?ARTIFACTS_DIR is required}" +: "${PR_NUMBER:?PR_NUMBER is required}" +: "${GITHUB_REPOSITORY:?GITHUB_REPOSITORY is required}" + +# DRY_RUN=1 makes the approver print the bodies it would post and skip all +# GitHub API calls (no comments, no review). Useful for local testing. +DRY_RUN="${DRY_RUN:-0}" + +if [[ "$DRY_RUN" != "1" ]]; then + : "${GITHUB_TOKEN:?GITHUB_TOKEN is required (or set DRY_RUN=1)}" +fi + +APPROVER_TOKEN="${APPROVER_TOKEN:-}" +TIER3_REVIEWERS="${TIER3_REVIEWERS:-}" +API_SECRET="${API_SECRET:-}" +PR_AUTHOR="${PR_AUTHOR:-}" +BASE_REF="${BASE_REF:-}" +ORG="${ORG:-utmstack}" +ADMIN_TEAM="${ADMIN_TEAM:-administrators}" +CORE_TEAM="${CORE_TEAM:-core-developers}" +MERGE_METHOD="${MERGE_METHOD:-squash}" + +# Markers for sticky comments. Each topic has its own marker so deps, AI and +# permission updates don't trample each other. +MARKER_DEPS='' +MARKER_AI='' +MARKER_PERM='' + +api() { + curl -sS \ + -H "Authorization: Bearer ${GITHUB_TOKEN}" \ + -H "Accept: application/vnd.github+json" \ + -H "X-GitHub-Api-Version: 2022-11-28" \ + "$@" +} + +# Find sticky comment ID by marker substring, or empty if none. +find_sticky_comment() { + local marker="$1" + api "https://api.github.com/repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments?per_page=100" \ + | jq -r --arg m "$marker" '.[] | select(.body | contains($m)) | .id' \ + | head -n1 +} + +# Upsert a sticky comment: edit if marker already present, else create. +upsert_sticky_comment() { + local marker="$1" + local body="$2" + local full_body="${marker}"$'\n'"${body}" + + if [[ "$DRY_RUN" == "1" ]]; then + echo "::group::[DRY_RUN] Would upsert comment with marker $marker" + echo "$full_body" + echo "::endgroup::" + return 0 + fi + + local id + id=$(find_sticky_comment "$marker" || true) + + if [[ -n "$id" ]]; then + echo "Updating existing comment $id" + jq -n --arg body "$full_body" '{body: $body}' \ + | api -X PATCH \ + "https://api.github.com/repos/${GITHUB_REPOSITORY}/issues/comments/${id}" \ + --data-binary @- > /dev/null + else + echo "Creating new comment" + jq -n --arg body "$full_body" '{body: $body}' \ + | api -X POST \ + "https://api.github.com/repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" \ + --data-binary @- > /dev/null + fi +} + +# Delete a sticky comment if it exists (used when the topic becomes a no-op). +delete_sticky_comment() { + local marker="$1" + if [[ "$DRY_RUN" == "1" ]]; then + echo "[DRY_RUN] Would delete stale comment with marker $marker (if present)" + return 0 + fi + local id + id=$(find_sticky_comment "$marker" || true) + if [[ -n "$id" ]]; then + echo "Deleting stale comment $id" + api -X DELETE \ + "https://api.github.com/repos/${GITHUB_REPOSITORY}/issues/comments/${id}" \ + > /dev/null + fi +} + +# ============================================================================= +# 1. Deps verdict +# ============================================================================= + +deps_artifact_dir="$ARTIFACTS_DIR/go-deps-result" +deps_failed=false +deps_output="" + +if [[ -f "$deps_artifact_dir/exit_code.txt" ]]; then + deps_exit=$(cat "$deps_artifact_dir/exit_code.txt") + deps_output=$(cat "$deps_artifact_dir/output.txt" 2>/dev/null || echo "") + if [[ "$deps_exit" != "0" ]]; then + deps_failed=true + fi +else + echo "::warning::go-deps artifact missing — treating as failed" + deps_failed=true + deps_output="(go-deps artifact missing — the job may have failed to run)" +fi + +# ============================================================================= +# 2. AI verdict — read every ai-review-* artifact +# ============================================================================= + +declare -a ai_results=() +declare -i max_tier=1 +ai_findings_md="" + +shopt -s nullglob +for d in "$ARTIFACTS_DIR"/ai-review-*/; do + f="${d}result.json" + [[ -f "$f" ]] || continue + ai_results+=("$f") + tier=$(jq -r '.tier // 2' "$f") + if (( tier > max_tier )); then + max_tier=$tier + fi +done +shopt -u nullglob + +if [[ ${#ai_results[@]} -eq 0 ]]; then + echo "::warning::No AI review artifacts — treating as tier 2 fail-safe" + max_tier=2 +fi + +# Build a markdown section per AI prompt result. +for f in "${ai_results[@]}"; do + prompt=$(jq -r '.prompt // "unknown"' "$f") + model=$(jq -r '.model // "?"' "$f") + tier=$(jq -r '.tier // 2' "$f") + summary=$(jq -r '.summary // "(no summary)"' "$f") + findings=$(jq -r ' + .findings // [] | + if length == 0 then " _No findings._" + else + map(" - **\(.severity // "?")** `\(.file // "?"):\(.line // "?")` — \(.message // "")") | join("\n") + end + ' "$f") + case "$tier" in + 1) icon="✅" label="Tier 1 — looks clean" ;; + 2) icon="⚠️" label="Tier 2 — changes requested" ;; + 3) icon="🛑" label="Tier 3 — engineer review required" ;; + *) icon="❓" label="Tier ?" ;; + esac + ai_findings_md+=$'\n'"#### $icon \`$prompt\` (\`$model\`) — $label"$'\n\n' + ai_findings_md+="**Summary:** $summary"$'\n\n' + ai_findings_md+="$findings"$'\n' +done + +# ============================================================================= +# 3. Compose & post comments +# ============================================================================= + +# --- Deps comment (only when failed) --- +if $deps_failed; then + deps_body=$(cat <Script output + +\`\`\` +$deps_output +\`\`\` + + +EOF +) + upsert_sticky_comment "$MARKER_DEPS" "$deps_body" +else + # Remove any previous deps comment now that deps are clean. + delete_sticky_comment "$MARKER_DEPS" +fi + +# --- AI verdict comment (always) --- +case "$max_tier" in + 1) + ai_header="### ✅ AI review — Approved" + ai_intro="All prompts returned Tier 1. No blocking issues detected in this diff." + ;; + 2) + ai_header="### ⚠️ AI review — Changes requested" + ai_intro="One or more prompts found issues the author should fix before merging. Details below." + ;; + 3) + mention="" + if [[ -n "$TIER3_REVIEWERS" ]]; then + IFS=',' read -ra handles <<< "$TIER3_REVIEWERS" + for h in "${handles[@]}"; do + h="$(echo "$h" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' -e 's/^@//')" + [[ -n "$h" ]] && mention+="@$h " + done + fi + ai_header="### 🛑 AI review — Engineer review required" + ai_intro="This PR touches critical paths or introduces changes the model cannot judge with sufficient confidence. ${mention}please review." + ;; + *) + ai_header="### ❓ AI review — Unknown verdict" + ai_intro="The approver could not determine an overall tier." + ;; +esac + +ai_body=$(cat </dev/null 2>&1 +} + +authorized=false +if [[ -z "$API_SECRET" ]]; then + echo "::warning::API_SECRET not set — skipping permission check (treating as authorized)" + authorized=true +elif [[ -z "$PR_AUTHOR" ]]; then + echo "::warning::PR_AUTHOR not set — skipping permission check (treating as authorized)" + authorized=true +elif is_in_team "$ADMIN_TEAM" "$PR_AUTHOR"; then + echo "✅ $PR_AUTHOR is in @${ORG}/${ADMIN_TEAM}" + authorized=true +elif is_in_team "$CORE_TEAM" "$PR_AUTHOR"; then + echo "✅ $PR_AUTHOR is in @${ORG}/${CORE_TEAM}" + authorized=true +else + echo "⛔ $PR_AUTHOR is NOT in @${ORG}/${ADMIN_TEAM} nor @${ORG}/${CORE_TEAM}" +fi + +if ! $authorized; then + perm_body=$(cat < /dev/null || echo "::warning::Failed to post formal review" + fi +else + echo "::warning::APPROVER_TOKEN not set — skipping formal PR review (only sticky comments + status check)" +fi + +# ============================================================================= +# 6. Auto-merge — only when target branch is release/**. +# We enable GitHub native auto-merge (gh pr merge --auto), which queues the +# merge until ALL branch-protection requirements are met. It's safe: if any +# other check fails or someone leaves a manual REQUEST_CHANGES, the merge +# stays pending. +# ============================================================================= + +if ! $deps_failed && [[ "$max_tier" == "1" ]] && $authorized; then + if [[ "$BASE_REF" == release/* ]]; then + if [[ "$DRY_RUN" == "1" ]]; then + echo "[DRY_RUN] Would enable auto-merge: gh pr merge $PR_NUMBER --auto --$MERGE_METHOD (base: $BASE_REF)" + elif [[ -z "$APPROVER_TOKEN" ]]; then + echo "::warning::APPROVER_TOKEN not set — cannot enable auto-merge" + else + echo "Enabling auto-merge for #$PR_NUMBER (target: $BASE_REF, method: $MERGE_METHOD)" + GH_TOKEN="$APPROVER_TOKEN" gh pr merge "$PR_NUMBER" \ + --auto "--${MERGE_METHOD}" \ + --repo "$GITHUB_REPOSITORY" \ + || echo "::warning::Failed to enable auto-merge (already enabled? branch protection mismatch?)" + fi + else + echo "Target branch '$BASE_REF' is not release/** — skipping auto-merge (deploy branches stay manual)" + fi +fi + +# ============================================================================= +# 7. Exit code +# ============================================================================= + +echo "" +echo "Summary:" +echo " deps_failed: $deps_failed" +echo " max_tier: $max_tier" +echo " authorized: $authorized" +echo " base_ref: $BASE_REF" + +if $deps_failed; then + exit 1 +fi + +if ! $authorized; then + exit 1 +fi + +if [[ "$max_tier" -ge 2 ]]; then + exit 1 +fi + +echo "✅ PR approved." +exit 0 diff --git a/.github/scripts/generate-changelog.sh b/.github/scripts/generate-changelog.sh new file mode 100755 index 000000000..b0cc9e77d --- /dev/null +++ b/.github/scripts/generate-changelog.sh @@ -0,0 +1,181 @@ +#!/usr/bin/env bash +# +# Generates AI-powered release notes via ThreatWinds /chat/completions. +# Used by .github/workflows/generate-changelog.yml; also runnable locally +# (export THREATWINDS_API_KEY + THREATWINDS_API_SECRET and run the script). +# +# Usage: +# bash .github/scripts/generate-changelog.sh [previous_tag] +# +# Examples: +# bash .github/scripts/generate-changelog.sh v11.2.8 v11.2.7 +# bash .github/scripts/generate-changelog.sh v11.2.8 # auto-detect previous +# +# Required env vars: +# THREATWINDS_API_KEY auth +# THREATWINDS_API_SECRET auth +# +# Optional env vars: +# PRODUCT_NAME default: "UTMStack" +# PRODUCT_DESCRIPTION default: "Unified Threat Management and SIEM Platform" +# MODEL default: "gemini-3-flash-lite" +# TEMPERATURE default: 0.3 +# MAX_TOKENS default: 2000 +# OUTPUT_FILE default: "/tmp/changelog.md" +# THREATWINDS_BASE_URL default: https://apis.threatwinds.com/api/ai/v1 + +set -euo pipefail + +# ─── Config ─────────────────────────────────────────────────────────────────── +PRODUCT_NAME="${PRODUCT_NAME:-UTMStack}" +PRODUCT_DESCRIPTION="${PRODUCT_DESCRIPTION:-Unified Threat Management and SIEM Platform}" +MODEL="${MODEL:-gemini-3-flash-lite}" +TEMPERATURE="${TEMPERATURE:-0.3}" +MAX_TOKENS="${MAX_TOKENS:-2000}" +OUTPUT_FILE="${OUTPUT_FILE:-/tmp/changelog.md}" +BASE_URL="${THREATWINDS_BASE_URL:-https://apis.threatwinds.com/api/ai/v1}" + +# ─── Arg parsing ────────────────────────────────────────────────────────────── +if [ "$#" -lt 1 ]; then + echo "Usage: $0 [previous_tag]" >&2 + exit 1 +fi + +CURRENT_TAG="$1" +PREVIOUS_TAG="${2:-}" + +# ─── Dependencies ───────────────────────────────────────────────────────────── +command -v jq >/dev/null || { echo "jq is required"; exit 1; } +command -v curl >/dev/null || { echo "curl is required"; exit 1; } +command -v git >/dev/null || { echo "git is required"; exit 1; } + +: "${THREATWINDS_API_KEY:?THREATWINDS_API_KEY is required}" +: "${THREATWINDS_API_SECRET:?THREATWINDS_API_SECRET is required}" + +# ─── Resolve previous tag if not provided ───────────────────────────────────── +if [ -z "$PREVIOUS_TAG" ]; then + echo "Auto-detecting previous tag..." + ALL_TAGS=$(git tag --sort=-v:refname) + FOUND_CURRENT=false + for tag in $ALL_TAGS; do + if [ "$FOUND_CURRENT" = true ]; then + PREVIOUS_TAG="$tag" + break + fi + if [ "$tag" = "$CURRENT_TAG" ]; then + FOUND_CURRENT=true + fi + done + if [ -z "$PREVIOUS_TAG" ]; then + PREVIOUS_TAG=$(git rev-list --max-parents=0 HEAD | head -1) + echo "No previous tag found, using first commit: $PREVIOUS_TAG" + fi +fi + +echo "Current tag: $CURRENT_TAG" +echo "Previous tag: $PREVIOUS_TAG" +echo "Model: $MODEL" +echo + +# ─── Collect commits ────────────────────────────────────────────────────────── +COMMITS=$(git log "${PREVIOUS_TAG}..${CURRENT_TAG}" --pretty=format:"- %h %s (%an)" --no-merges) +COMMIT_COUNT=$(git rev-list --count "${PREVIOUS_TAG}..${CURRENT_TAG}" --no-merges) + +if [ -z "$COMMITS" ]; then + echo "No commits found between $PREVIOUS_TAG and $CURRENT_TAG." + exit 0 +fi + +echo "Found $COMMIT_COUNT commits." +echo + +# ─── Build prompt ───────────────────────────────────────────────────────────── +PROMPT="You are a product marketing writer creating release notes for end users of a software product. + +Product: $PRODUCT_NAME - $PRODUCT_DESCRIPTION +Release: $CURRENT_TAG + +Here are the commit messages from this release: +$COMMITS + +Create user-friendly release notes in markdown format. This is for NON-TECHNICAL end users who want to know what's new and improved in the product. + +IMPORTANT RULES: +1. ONLY include changes that DIRECTLY AFFECT END USERS - things they can see, use, or benefit from +2. COMPLETELY IGNORE internal/technical changes like: + - CI/CD, GitHub Actions, deployment pipelines + - Code refactoring, component restructuring + - Database migrations, backend infrastructure + - Internal API changes, gRPC, service communication + - Developer tooling, linting, formatting + - README updates, internal documentation +3. Write in simple, non-technical language +4. Focus on BENEFITS to the user, not implementation details +5. Group into these categories ONLY (skip empty categories): + - **What's New** - New features users can now use + - **Improved** - Enhancements to existing features + - **Fixed** - Bugs that were affecting users +6. Start with a brief 1-2 sentence summary of the release highlights +7. Use bullet points, be concise (one line per item) +8. Do NOT wrap output in markdown code blocks +9. Do NOT include commit hashes or author names +10. If most commits are internal/technical, just summarize with 'Minor improvements and bug fixes' + +Write the release notes directly in markdown format, ready to be used as-is." + +# ─── Call ThreatWinds ───────────────────────────────────────────────────────── +echo "Calling ThreatWinds ($MODEL)..." +PAYLOAD=$(jq -n \ + --arg model "$MODEL" \ + --arg prompt "$PROMPT" \ + --argjson temp "$TEMPERATURE" \ + --argjson maxtok "$MAX_TOKENS" \ + '{ + model: $model, + messages: [ + {role: "system", content: "You are a technical writer specializing in software changelogs."}, + {role: "user", content: $prompt} + ], + temperature: $temp, + max_tokens: $maxtok + }') + +RESPONSE_FILE=$(mktemp) +HTTP_STATUS=$(curl -sS -o "$RESPONSE_FILE" -w '%{http_code}' \ + -X POST "${BASE_URL%/}/chat/completions" \ + -H "Content-Type: application/json" \ + -H "api-key: ${THREATWINDS_API_KEY}" \ + -H "api-secret: ${THREATWINDS_API_SECRET}" \ + --data "$PAYLOAD" || echo "000") + +if [ "$HTTP_STATUS" != "200" ]; then + echo "ERROR: ThreatWinds API returned HTTP $HTTP_STATUS" >&2 + cat "$RESPONSE_FILE" >&2 + exit 1 +fi + +CHANGELOG=$(jq -r '.choices[0].message.content // empty' "$RESPONSE_FILE") + +if [ -z "$CHANGELOG" ]; then + echo "ERROR: empty response from ThreatWinds." >&2 + cat "$RESPONSE_FILE" >&2 + exit 1 +fi + +# ─── Append comparison link ─────────────────────────────────────────────────── +REPO_REMOTE=$(git config --get remote.origin.url 2>/dev/null | \ + sed -E 's#(git@github.com:|https://github.com/)#https://github.com/#; s#\.git$##') + +CHANGELOG="${CHANGELOG} + +--- +**Full Changelog**: ${REPO_REMOTE}/compare/${PREVIOUS_TAG}...${CURRENT_TAG}" + +printf "%s\n" "$CHANGELOG" > "$OUTPUT_FILE" + +echo +echo "──────── GENERATED CHANGELOG ────────" +cat "$OUTPUT_FILE" +echo "─────────────────────────────────────" +echo +echo "Saved to: $OUTPUT_FILE" diff --git a/.github/scripts/go-deps.sh b/.github/scripts/go-deps.sh new file mode 100755 index 000000000..02b5d0a3c --- /dev/null +++ b/.github/scripts/go-deps.sh @@ -0,0 +1,282 @@ +#!/bin/bash +set -euo pipefail + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +NC='\033[0m' # No Color + +print_usage() { + cat << 'EOF' +Usage: go-deps.sh [--check|--update] [--discover|] + +Modes: + --check Check for outdated dependencies (exit 1 if found) + --update Update outdated dependencies (default) + +Target: + --discover Discover all Go projects from current directory + Path to a specific Go project + +Examples: + go-deps.sh --check ./installer + go-deps.sh --update ./installer + go-deps.sh --check --discover + go-deps.sh --update --discover +EOF +} + +# Parse arguments +CHECK_ONLY=false +DISCOVER=false +TARGET_PATH="" + +if [[ $# -lt 1 ]]; then + print_usage + exit 1 +fi + +for arg in "$@"; do + case "$arg" in + --check) + CHECK_ONLY=true + ;; + --update) + CHECK_ONLY=false + ;; + --discover) + DISCOVER=true + ;; + --help|-h) + print_usage + exit 0 + ;; + --*) + echo -e "${RED}Error: unknown option $arg${NC}" >&2 + print_usage + exit 1 + ;; + *) + TARGET_PATH="$arg" + ;; + esac +done + +# Validate arguments +if [[ "$DISCOVER" == false && -z "$TARGET_PATH" ]]; then + echo -e "${RED}Error: must specify a path or use --discover${NC}" >&2 + print_usage + exit 1 +fi + +if [[ "$DISCOVER" == true && -n "$TARGET_PATH" ]]; then + echo -e "${RED}Error: cannot use both --discover and a specific path${NC}" >&2 + print_usage + exit 1 +fi + +# Discover Go projects +discover_projects() { + local root="$1" + find "$root" -name "go.mod" \ + -not -path "*/.*" \ + -not -path "*/vendor/*" \ + -not -path "*/node_modules/*" \ + -exec dirname {} \; +} + +# Get explicit modules from go.mod (direct dependencies only) +get_explicit_modules() { + local go_mod="$1" + grep -E '^\s+[a-zA-Z]' "$go_mod" 2>/dev/null | \ + grep -v '//' | \ + awk '{print $1}' | \ + sort -u +} + +# Check for updates in a project, outputs JSON lines. +# On `go list` failure, records the project + error in $TEMP_DIR/_check_errors +# instead of returning non-zero, so the caller can keep checking the other +# projects and we report every broken module at once instead of one per run. +check_project() { + local project_path="$1" + local go_mod="$project_path/go.mod" + + # Get explicit modules + local explicit_modules + explicit_modules=$(get_explicit_modules "$go_mod") + + # Get all modules with update info as JSON + local json_output + local list_err + list_err=$(mktemp) + if ! json_output=$(cd "$project_path" && go list -u -m -json all 2>"$list_err"); then + { + echo "## $project_path" + cat "$list_err" + echo + } >> "$TEMP_DIR/_check_errors" + rm -f "$list_err" + return 0 + fi + rm -f "$list_err" + + # Parse JSON and filter modules with updates that are explicit dependencies + echo "$json_output" | jq -c 'select(.Update != null) | {Path: .Path, Version: .Version, UpdateVersion: .Update.Version}' 2>/dev/null | \ + while IFS= read -r module; do + local mod_path + mod_path=$(echo "$module" | jq -r '.Path') + if echo "$explicit_modules" | grep -qx "$mod_path"; then + echo "$module" + fi + done +} + +# Update a single module +update_module() { + local project_path="$1" + local mod_path="$2" + local new_version="$3" + + local update_str="${mod_path}@${new_version}" + echo -e " 🔄 Updating $update_str" + (cd "$project_path" && go get "$update_str") +} + +# Run go mod tidy +run_tidy() { + local project_path="$1" + echo -e " 🧹 Running go mod tidy..." + (cd "$project_path" && go mod tidy) +} + +# Create temp directory for storing updates per project +TEMP_DIR=$(mktemp -d) +trap 'rm -rf "$TEMP_DIR"' EXIT + +# Get list of projects +PROJECTS="" +if [[ "$DISCOVER" == true ]]; then + PROJECTS=$(discover_projects ".") + + if [[ -z "$PROJECTS" ]]; then + echo "No Go projects found." + exit 0 + fi + + project_count=$(echo "$PROJECTS" | wc -l | tr -d ' ') + echo -e "🔍 Discovered $project_count Go projects\n" +else + if [[ ! -f "$TARGET_PATH/go.mod" ]]; then + echo -e "${RED}Error: no go.mod found in $TARGET_PATH${NC}" >&2 + exit 1 + fi + PROJECTS="$TARGET_PATH" +fi + +# Check all projects for updates +HAS_UPDATES=false + +echo "$PROJECTS" | while IFS= read -r project; do + [[ -z "$project" ]] && continue + + updates=$(check_project "$project") + if [[ -n "$updates" ]]; then + # Store updates in a temp file named after the project (sanitized) + safe_name=$(echo "$project" | tr '/' '_') + echo "$updates" > "$TEMP_DIR/$safe_name" + echo "$project" >> "$TEMP_DIR/_projects_with_updates" + fi +done + +# If any project couldn't be inspected, report all of them at once. +# This is almost always caused by stale go.sum entries in modules that +# consume an internal package via `replace` — fix those first before +# anything else, since both --check and --update need a clean read. +if [[ -f "$TEMP_DIR/_check_errors" ]]; then + echo -e "${RED}❌ Could not inspect the following projects (run 'go mod tidy' there):${NC}" >&2 + echo >&2 + cat "$TEMP_DIR/_check_errors" >&2 + exit 1 +fi + +# Check if any updates were found +if [[ ! -f "$TEMP_DIR/_projects_with_updates" ]]; then + echo -e "${GREEN}✅ All dependencies are up to date.${NC}" + exit 0 +fi + +# Print summary of updates needed +echo -e "📦 Dependencies with updates available:" +while IFS= read -r project; do + [[ -z "$project" ]] && continue + safe_name=$(echo "$project" | tr '/' '_') + + echo -e "\n 📁 $project:" + while IFS= read -r module; do + [[ -z "$module" ]] && continue + mod_path=$(echo "$module" | jq -r '.Path') + current=$(echo "$module" | jq -r '.Version') + new_ver=$(echo "$module" | jq -r '.UpdateVersion') + echo -e " - $mod_path: $current → $new_ver" + done < "$TEMP_DIR/$safe_name" +done < "$TEMP_DIR/_projects_with_updates" + +if [[ "$CHECK_ONLY" == true ]]; then + echo -e "\n${RED}❌ Please update dependencies before merging.${NC}" + exit 1 +fi + +# Update mode - apply updates +echo -e "\n🔄 Updating dependencies..." +while IFS= read -r project; do + [[ -z "$project" ]] && continue + safe_name=$(echo "$project" | tr '/' '_') + + echo -e "\n 📁 $project:" + while IFS= read -r module; do + [[ -z "$module" ]] && continue + mod_path=$(echo "$module" | jq -r '.Path') + new_ver=$(echo "$module" | jq -r '.UpdateVersion') + update_module "$project" "$mod_path" "$new_ver" + done < "$TEMP_DIR/$safe_name" + + run_tidy "$project" +done < "$TEMP_DIR/_projects_with_updates" + +echo -e "\n${GREEN}✅ All dependencies updated successfully.${NC}" + +# Propagate `go mod tidy` to EVERY discovered project, not just those that +# were updated. When an internal package (e.g. packages/go-common) gets new +# transitive deps, every consumer that imports it via `replace ../packages/...` +# needs its own go.sum recomputed — otherwise the next `go list` call fails. +echo -e "\n🧹 Propagating tidy to all projects (handles local replace ripple)..." +echo "$PROJECTS" | while IFS= read -r project; do + [[ -z "$project" ]] && continue + echo -e " 📁 $project" + (cd "$project" && go mod tidy) +done + +# Verify every project still builds. We compile into a throwaway directory so +# main packages don't litter the working tree, then delete it. +echo -e "\n🔨 Verifying all projects build..." +BUILD_OUT=$(mktemp -d) +BUILD_FAILED_FILE="$TEMP_DIR/_build_failed" +echo "$PROJECTS" | while IFS= read -r project; do + [[ -z "$project" ]] && continue + if (cd "$project" && go build -o "$BUILD_OUT/" ./... 2>&1); then + echo -e " ${GREEN}✅${NC} $project" + else + echo -e " ${RED}❌${NC} $project" + echo "$project" >> "$BUILD_FAILED_FILE" + fi +done +rm -rf "$BUILD_OUT" + +if [[ -f "$BUILD_FAILED_FILE" ]]; then + echo -e "\n${RED}❌ Build failed in:${NC}" >&2 + cat "$BUILD_FAILED_FILE" >&2 + exit 1 +fi + +echo -e "\n${GREEN}✅ All projects build cleanly.${NC}" diff --git a/.github/scripts/golang-updater/go.mod b/.github/scripts/golang-updater/go.mod deleted file mode 100644 index 6ceba9454..000000000 --- a/.github/scripts/golang-updater/go.mod +++ /dev/null @@ -1,3 +0,0 @@ -module golang-updater - -go 1.24.2 diff --git a/.github/scripts/golang-updater/main.go b/.github/scripts/golang-updater/main.go deleted file mode 100644 index 94162fa1e..000000000 --- a/.github/scripts/golang-updater/main.go +++ /dev/null @@ -1,252 +0,0 @@ -package main - -import ( - "bufio" - "bytes" - "encoding/json" - "fmt" - "os" - "os/exec" - "path/filepath" - "strings" -) - -type Module struct { - Path string - Version string - Update *ModuleUpdate -} - -type ModuleUpdate struct { - Version string -} - -func main() { - if len(os.Args) < 2 { - printUsage() - os.Exit(1) - } - - var checkOnly bool - var discover bool - var targetPath string - - for _, arg := range os.Args[1:] { - switch arg { - case "--check": - checkOnly = true - case "--update": - checkOnly = false - case "--discover": - discover = true - case "--help", "-h": - printUsage() - os.Exit(0) - default: - if !strings.HasPrefix(arg, "--") { - targetPath = arg - } - } - } - - // Validate arguments - if !discover && targetPath == "" { - fmt.Fprintf(os.Stderr, "Error: must specify a path or use --discover\n") - printUsage() - os.Exit(1) - } - - if discover && targetPath != "" { - fmt.Fprintf(os.Stderr, "Error: cannot use both --discover and a specific path\n") - printUsage() - os.Exit(1) - } - - var projects []string - var err error - - if discover { - projects, err = discoverProjects(".") - if err != nil { - fmt.Fprintf(os.Stderr, "Error discovering projects: %v\n", err) - os.Exit(1) - } - if len(projects) == 0 { - fmt.Println("No Go projects found.") - os.Exit(0) - } - fmt.Printf("🔍 Discovered %d Go projects\n\n", len(projects)) - } else { - // Verify the path exists and has a go.mod - goModPath := filepath.Join(targetPath, "go.mod") - if _, err := os.Stat(goModPath); os.IsNotExist(err) { - fmt.Fprintf(os.Stderr, "Error: no go.mod found in %s\n", targetPath) - os.Exit(1) - } - projects = []string{targetPath} - } - - hasUpdates := false - allUpdates := make(map[string][]Module) - - for _, project := range projects { - updates, err := checkProject(project) - if err != nil { - fmt.Fprintf(os.Stderr, "Error checking %s: %v\n", project, err) - os.Exit(1) - } - if len(updates) > 0 { - hasUpdates = true - allUpdates[project] = updates - } - } - - if !hasUpdates { - fmt.Println("✅ All dependencies are up to date.") - return - } - - // Print summary of updates needed - fmt.Println("📦 Dependencies with updates available:") - for project, updates := range allUpdates { - fmt.Printf("\n 📁 %s:\n", project) - for _, mod := range updates { - fmt.Printf(" - %s: %s → %s\n", mod.Path, mod.Version, mod.Update.Version) - } - } - - if checkOnly { - fmt.Println("\n❌ Please update dependencies before merging.") - os.Exit(1) - } - - // Update mode - apply updates - fmt.Println("\n🔄 Updating dependencies...") - for project, updates := range allUpdates { - fmt.Printf("\n 📁 %s:\n", project) - if err := updateProject(project, updates); err != nil { - fmt.Fprintf(os.Stderr, "Error updating %s: %v\n", project, err) - os.Exit(1) - } - } - - fmt.Println("\n✅ All dependencies updated successfully.") -} - -func printUsage() { - fmt.Println(`Usage: golang-updater [--check|--update] [--discover|] - -Modes: - --check Check for outdated dependencies (exit 1 if found) - --update Update outdated dependencies (default) - -Target: - --discover Discover all Go projects from current directory - Path to a specific Go project - -Examples: - golang-updater --check ./installer - golang-updater --update ./installer - golang-updater --check --discover - golang-updater --update --discover`) -} - -func discoverProjects(root string) ([]string, error) { - var projects []string - - err := filepath.Walk(root, func(path string, info os.FileInfo, err error) error { - if err != nil { - return err - } - - // Skip hidden directories and common non-project directories - if info.IsDir() { - name := info.Name() - // Don't skip the root directory itself - if path != root && (strings.HasPrefix(name, ".") || name == "vendor" || name == "node_modules") { - return filepath.SkipDir - } - } - - if info.Name() == "go.mod" { - dir := filepath.Dir(path) - projects = append(projects, dir) - } - - return nil - }) - - return projects, err -} - -func checkProject(projectPath string) ([]Module, error) { - goModPath := filepath.Join(projectPath, "go.mod") - modFile, err := os.Open(goModPath) - if err != nil { - return nil, fmt.Errorf("error opening go.mod: %w", err) - } - defer modFile.Close() - - explicitModules := make(map[string]bool) - scanner := bufio.NewScanner(modFile) - for scanner.Scan() { - line := strings.TrimSpace(scanner.Text()) - if strings.HasPrefix(line, "require") || strings.HasPrefix(line, ")") { - continue - } - fields := strings.Fields(line) - if len(fields) >= 1 && !strings.HasPrefix(fields[0], "//") { - explicitModules[fields[0]] = true - } - } - if err := scanner.Err(); err != nil { - return nil, fmt.Errorf("error reading go.mod: %w", err) - } - - cmd := exec.Command("go", "list", "-u", "-m", "-json", "all") - cmd.Dir = projectPath - output, err := cmd.Output() - if err != nil { - return nil, fmt.Errorf("error executing go list: %w", err) - } - - decoder := json.NewDecoder(bytes.NewReader(output)) - var toUpdate []Module - - for decoder.More() { - var mod Module - if err := decoder.Decode(&mod); err != nil { - return nil, fmt.Errorf("error parsing JSON output: %w", err) - } - if mod.Update != nil && explicitModules[mod.Path] { - toUpdate = append(toUpdate, mod) - } - } - - return toUpdate, nil -} - -func updateProject(projectPath string, updates []Module) error { - for _, mod := range updates { - updateStr := fmt.Sprintf("%s@%s", mod.Path, mod.Update.Version) - fmt.Printf(" 🔄 Updating %s\n", updateStr) - cmd := exec.Command("go", "get", updateStr) - cmd.Dir = projectPath - cmd.Stdout = os.Stdout - cmd.Stderr = os.Stderr - if err := cmd.Run(); err != nil { - return fmt.Errorf("error updating %s: %w", updateStr, err) - } - } - - fmt.Printf(" 🧹 Running go mod tidy...\n") - cmd := exec.Command("go", "mod", "tidy") - cmd.Dir = projectPath - cmd.Stdout = os.Stdout - cmd.Stderr = os.Stderr - if err := cmd.Run(); err != nil { - return fmt.Errorf("error running go mod tidy: %w", err) - } - - return nil -} diff --git a/.github/workflows/README.md b/.github/workflows/README.md index f04f9ffa6..f72306221 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,330 +1,646 @@ -# 🛠️ GitHub Actions Workflows – UTMStack +# GitHub Actions Workflows — UTMStack -> This repository uses streamlined CI/CD workflows for building and deploying UTMStack v10 and v11 across different environments. +CI/CD for UTMStack v10 and v11. This folder contains two workflow families: -## 📋 Table of Contents +- **PR checks** (`pr-checks.yml` + `_pr-reusable-*.yml`) — validate every + Pull Request before merge. The only gate into code on `release/**`, + `v10` and `v11`. +- **Deployment pipelines** (`v10-deployment-pipeline.yml`, + `v11-deployment-pipeline.yml`, `installer-release.yml`) — build, publish + and deploy artifacts once code is merged. -- [Workflows Overview](#workflows-overview) +## Table of contents + +- [Release policy](#release-policy) +- [PR Checks](#pr-checks) - [V10 Deployment Pipeline](#v10-deployment-pipeline) - [V11 Deployment Pipeline](#v11-deployment-pipeline) - [Installer Release](#installer-release) -- [Required Secrets and Variables](#required-secrets-and-variables) +- [Secrets and variables](#secrets-and-variables) +- [Approver GitHub App setup](#approver-github-app-setup) +- [Reusable workflows](#reusable-workflows) +- [How to deploy](#how-to-deploy) +- [Troubleshooting](#troubleshooting) --- -## 🔄 Workflows Overview +## Release policy + +Hard rules: + +- **Direct push is forbidden** on `release/**`, `v10` and `v11`. PR only. +- **Branch protection** is enabled on those branches: PR required, status + checks green (`All checks passed`), no force push. +- **Roll-forward only.** No rollbacks. If a release breaks something, ship + a hotfix that bumps the version (e.g. `v11.2.9` breaks → `v11.2.10` + fixes it). Feature flags / kill switches are fine for turning features + off without a redeploy. + +### Tiered approval model + +The team is small (3 devs + 2 part-time seniors), so the AI can approve +and merge on its own in most cases. Seniors only get involved when the +cost of being wrong is high. + +The **final tier** of a PR is decided by the approver, taking the maximum +across all AI prompts (see [PR Checks](#pr-checks)). + +| Tier | Meaning | Approver action | +|------|---------|-----------------| +| **1** | Simple change, AI detects no issues, deps OK. | ✅ Sticky "Approved" comment + (when the approver App is configured) formal `APPROVE` review. Status check green. | +| **2** | Minor issues the author should fix before merging (typos, small bugs, out-of-context code). | ⚠️ Sticky comment with the findings list + formal `REQUEST_CHANGES` review. Status check red. | +| **3** | Touches critical paths (crypto, auth, migrations, installer, gRPC contracts, CI/CD) or the model can't judge with confidence. | 🛑 Sticky comment @mentioning the handles configured in `tier3_reviewers` + formal `REQUEST_CHANGES` review. Status check red. | + +When the author pushes new commits, the sticky comments are **updated +in-place** (same comment, no stacking) and the workflow re-runs +automatically. A blocked PR is **never auto-closed** — it stays open +waiting for the fixes. + +Sensitive paths for Tier 3 are identified by each prompt's own rules (see +`.github/ai-prompts/*.md`). In the future this could be reinforced with +`CODEOWNERS` for additional per-path gates. + +### Auto-merge + +The approver enables GitHub's native auto-merge **only** when **all** of +the following hold: + +- Target branch matches `release/**` (PRs to `v10` / `v11` stay manual + so production deploys are always intentional). +- `deps_failed == false`. +- `max_tier == 1` across every AI prompt. +- PR author is in `@utmstack/administrators` or `@utmstack/core-developers`. +- The approver GitHub App is configured (`APPROVER_APP_ID` + `APPROVER_PRIVATE_KEY` secrets present). + +Auto-merge does NOT merge immediately — it queues the merge until every +branch-protection requirement is satisfied. If another check fails later +or a human leaves `REQUEST_CHANGES`, the merge stays pending. + +### Dependabot + +Disabled. `.github/dependabot.yml` keeps `updates: []` so Dependabot +reads the file but creates no PRs. Dependency freshness is enforced via +the `go_deps` check on every PR. To re-enable Dependabot, restore the +previous `updates:` list (see git history of that file). + +### Hotfixes + +- `hotfix/x` branch from `v11` → PR to `v11` → same checks. +- `urgent` label allows fast-track: if checks pass and the AI approves, + it merges without waiting for human review even when touching sensitive + paths. +- **Recommended (not strictly required):** after the hotfix merges to + `v11`, pull `v11` into the active `release/v11.x.x+1` branch (merge or + cherry-pick — either works). The fix is **not** lost if you skip this + step: git already has the hotfix in `v11`'s history, so when + `release/v11.x.x+1` later merges back, git combines both lines and + the fix lands automatically. Syncing early is good hygiene because it + surfaces conflicts in your release branch rather than at the final + merge, and it lets dev builds include the patched code immediately. + +**Version derivation is automatic.** When a hotfix merges to `v11`, the +deployment pipeline compares the candidate BASE (from CM DEV) against +the latest version in CM PROD: + +- If BASE > PROD → use BASE as RC tag (normal flow). +- If BASE ≤ PROD → the BASE was already shipped; bump the patch of PROD + to get the next tag (hotfix flow). + +Concrete example: PROD is on `v11.2.9`, dev is still on +`v11.2.9-dev.5` from the cycle that produced it. A hotfix lands on +`v11`. The pipeline sees BASE=`v11.2.9` collides with PROD=`v11.2.9`, +auto-bumps to `v11.2.10`, and the rest of the run (build, installer, +prerelease, CM register) proceeds with that tag. No manual rename, no +config change. -### 1. **installer-release.yml** -Automatically builds and publishes installers when a GitHub release is created. +--- -**Trigger:** Release created (types: `released`) +## PR Checks -**Behavior:** -- Detects version (v10 or v11) from release tag -- Builds installer for the detected version -- Uploads installer binary to the GitHub release +`pr-checks.yml` triggers on any Pull Request whose target is: -### 2. **v10-deployment-pipeline.yml** -Automated CI/CD pipeline for v10 builds and deployments. +- `release/**` (any release branch, v10 or v11) +- `v10` +- `v11` -**Triggers:** -- Push to `v10` branch → Deploys to **v10-rc** -- Push to `release/v10**` branches → Deploys to **v10-dev** -- Tags `v10.*` → Production build +### Architecture -**Environments:** -- `v10-dev` - Development environment (from release branches) -- `v10-rc` - Release candidate environment (from v10 branch) -- Production (from tags) +``` +PR opened / updated + │ + ├─────────────────┬─────────────────┐ + ▼ ▼ │ + ┌─────────┐ ┌─────────────┐ │ + │ go_deps │ │ ai_review │ │ + │ (repo) │ │ (matrix │ │ + │ │ │ per │ │ + │ │ │ prompt) │ │ + └────┬────┘ └──────┬──────┘ │ + │ │ │ + │ artifact │ artifacts │ + │ go-deps-result │ ai-review-* │ + ▼ ▼ │ + ┌──────────────────────────────┐ │ + │ approver │ │ + │ - reads artifacts │ │ + │ - decides final tier │ │ + │ - posts sticky comments │ │ + │ - (optional) formal review │ │ + └──────────────┬───────────────┘ │ + │ │ + ▼ ▼ + all_checks_passed ← single status check branch protection requires +``` -### 3. **v11-deployment-pipeline.yml** -Automated CI/CD pipeline for v11 builds and deployments. +**Key decision:** the producers (`go_deps`, `ai_review`) **always exit +green**. They only upload artifacts. The `approver` is the single source +of truth — it consolidates results, decides the tier (maximum across all +AI prompts), posts comments, and passes or fails the final check. -**Triggers:** -- Push to `release/v11**` branches → Deploys to **dev** environment -- Prerelease created → Deploys to **rc** environment +### `go_deps` -**Version Formats:** -- **Dev:** `v11.x.x-dev.N` (e.g., `v11.2.1-dev.1`) - Auto-incremented -- **RC:** `v11.x.x` (e.g., `v11.2.1`) - From prerelease tag +Single job, no matrix, no change detection. Runs: ---- +```bash +bash .github/scripts/go-deps.sh --check --discover +``` -## 🚀 V10 Deployment Pipeline +It discovers **every** `go.mod` in the repo (excluding `vendor/`, +dot-directories and `node_modules/`) and fails if any explicit **direct +dependency** has a newer version available. The script also detects +out-of-sync `go.sum` files (typically caused by local `replace` directives +in `packages/`) and reports them all at once. + +The job uploads its stdout and exit code as the `go-deps-result` artifact. +The approver reads it and, if exit code != 0, posts the sticky comment +"Go dependencies check failed" with the script output embedded. + +**Expected dev fix:** run +`bash .github/scripts/go-deps.sh --update --discover` locally, commit the +updated `go.mod` / `go.sum`, push. + +### `ai_review` + +Matrix with one job per `.md` under `.github/ai-prompts/` (except +`README.md`). Each job: + +1. Fetches the diff via `gh pr diff` (same unified diff the GitHub UI + shows — no need for `fetch-depth: 0`). +2. Calls the **ThreatWinds AI** `/chat/completions` endpoint with the + prompt and the diff. +3. Validates the response against the `{tier, summary, findings}` schema. +4. Uploads the JSON as the `ai-review-` artifact. + +If the model's response isn't valid JSON or the tier isn't 1/2/3, the +script writes a fallback with `tier: 2` and a "Manual review recommended" +finding (fail-safe). + +**Initial prompts:** + +- `security.md` — vulnerabilities introduced in the diff (injection, XSS, + SSRF, hardcoded secrets, weak crypto, insecure deserialization). +- `bugs.md` — concrete bugs: nil derefs, races, off-by-one, unhandled + errors, unclosed resources, inverted logic, out-of-context code. +- `architecture.md` — architectural deviations: new couplings, logic in + the wrong layer, broken contracts, unsafe migrations. + +Each prompt declares its own tier policy (Tier 3 covers paths critical +to that dimension). See `.github/ai-prompts/README.md` for the full +schema and tier semantics. + +**To scale:** drop a new `.md` into `.github/ai-prompts/`. Discovered at +runtime — no YAML changes needed. + +**Default model:** `gemini-3-flash-lite`. Each prompt can pin its own +model in frontmatter (`model: gemini-3-pro`, etc.). + +### `approver` + +Single job that **depends on `go_deps` and `ai_review`**. Steps: + +1. Downloads every PR-check artifact. +2. Reads `go-deps-result/exit_code.txt` → determines `deps_failed`. +3. Reads each `ai-review-*/result.json` → takes the **max tier** as the + AI verdict. +4. **Sticky comments** with invisible HTML markers + (``, ``, + ``): + - If deps failed → upsert "Go dependencies check failed" comment with + the script output. Otherwise delete it if a previous run posted one. + - Always upsert the "AI review" comment with the final tier + findings. + - These two are posted **regardless of who opened the PR** — even + unauthorized contributors get useful feedback. +5. **Permission check (LAST gate).** Looks up `github.actor` against the + GitHub teams `administrators` and `core-developers` of the + `utmstack` org via `API_SECRET`. Notably **does NOT** include + `integration-developers`. If the author is in neither team: + - Upsert "⛔ Permission denied" comment @mentioning + `@utmstack/administrators`. + - Skip the formal `APPROVE` review (always `REQUEST_CHANGES`). + - Skip auto-merge. + - Exit 1. +6. **(Optional) Formal PR review** when the approver App is installed + (see [Approver GitHub App setup](#approver-github-app-setup)): + - Tier 1 + deps OK + authorized → `APPROVE`. + - Anything else → `REQUEST_CHANGES`. +7. **Auto-merge** — only when **all** of: deps OK, Tier 1, authorized, + AND `BASE_REF` starts with `release/`. Calls + `gh pr merge --auto --` (default `squash`). This uses + GitHub's native auto-merge, so the actual merge waits until **every** + branch-protection requirement is satisfied (other checks green, no + pending human reviews). PRs targeting `v10` / `v11` never auto-merge + — those branches stay manually merged so deploys are intentional. +8. **Exit code:** 0 only if everything is OK; 1 if deps failed, + tier ≥ 2, or author unauthorized. + +When the author pushes new commits the workflow re-runs and the comments +are **updated in place** (no stacking). The PR is never auto-closed — +it stays open waiting for the author's fixes. + +### Adding a new check + +The architecture is designed to scale. To add, for example, a test check: + +1. Create `.github/workflows/_pr-reusable-.yml` that runs the check + and uploads an artifact with the result (ideally JSON). +2. Call the reusable from `pr-checks.yml` as another job. +3. Add that job to the `approver`'s `needs:` (and to `all_checks_passed`). +4. Extend `approver.sh` to read the new artifact and factor it into the + final verdict. + +To add a new AI prompt **no YAML changes are needed** — just drop a `.md` +into `.github/ai-prompts/`. -### Flow +--- -``` -┌─────────────────────┐ -│ Push to Branch │ -└──────────┬──────────┘ - │ - ├─── release/v10** ──→ Build & Deploy to v10-dev - ├─── v10 ──────────→ Build & Deploy to v10-rc - └─── tag v10.* ────→ Build for Production -``` +## V10 Deployment Pipeline + +Triggers: -### Jobs +- Push to `v10` → deploy to **v10-rc** +- Push to `release/v10**` → deploy to **v10-dev** +- Tag `v10.*` → production build -1. **setup_deployment** - Determines environment based on trigger -2. **validations** - Validates user permissions -3. **build_agent** - Builds and signs Windows/Linux agents -4. **build_agent_manager** - Builds agent-manager Docker image -5. **build_*** - Builds all microservices (aws, backend, correlation, frontend, etc.) -6. **all_builds_complete** - Checkpoint for all builds -7. **deploy_dev / deploy_rc** - Deploys to respective environments +Main jobs: -### Permissions +1. `setup_deployment` — determines environment from the trigger. +2. `validations` — checks permissions (team membership). +3. `build_agent` — Windows/Linux signed agents. +4. `build_agent_manager` — Docker image. +5. `build_*` — microservices (aws, backend, correlation, frontend, etc). +6. `all_builds_complete` — checkpoint. +7. `deploy_dev` / `deploy_rc` — deploy to the corresponding environment. -- **Dev deployments**: `integration-developers` or `core-developers` teams -- **RC/Prod deployments**: Same as dev +Permissions: `integration-developers` or `core-developers`. --- -## 🎯 V11 Deployment Pipeline +## V11 Deployment Pipeline + +Triggers: + +- Push to `release/v11**` → deploy to **dev** (auto-incremented version + `v11.x.x-dev.N`). +- Prerelease created → deploy to **rc** (version `v11.x.x` from the tag). ### Flow ``` -┌─────────────────────────────┐ -│ Push to release/v11.x.x │ -│ branch │ -└──────────────┬──────────────┘ - │ - ▼ - Auto-increment version - (v11.x.x-dev.N) - │ - ▼ - Build & Deploy to DEV - │ - ▼ - Publish to CM Dev - │ - ▼ - Schedule to Dev Instances - - -┌─────────────────────────────┐ -│ Create Prerelease │ -│ (tag: v11.x.x) │ -└──────────────┬──────────────┘ - │ - ▼ - Build & Deploy to RC - │ - ▼ - Generate Changelog (AI) - │ - ▼ - Build & Upload Installer - │ - ▼ - Publish to CM Prod - │ - ▼ - Schedule to Prod Instances +Push to release/v11.x.x + │ + ▼ +Auto-increment version (v11.x.x-dev.N) + │ + ▼ +Build & Deploy to DEV + │ + ▼ +Publish to CM Dev → schedule to dev instances + + +Create Prerelease (tag v11.x.x) + │ + ▼ +Build & Deploy to RC + │ + ▼ +Generate Changelog (AI) + │ + ▼ +Build & Upload Installer + │ + ▼ +Publish to CM Prod → schedule to prod instances ``` -### Jobs +Jobs: `setup_deployment`, `validations`, `build_agent`, +`build_utmstack_collector`, `build_agent_manager`, `build_event_processor`, +`build_backend` (Java 17), `build_frontend`, `build_user_auditor`, +`build_web_pdf`, `all_builds_complete`, `generate_changelog` (RC), +`build_installer_rc` (RC), `deploy_installer_dev` (Dev), +`publish_new_version`, `schedule`. -1. **setup_deployment** - Determines environment and version based on trigger -2. **validations** - Validates user permissions (team membership) -3. **build_agent** - Builds and signs Windows/Linux agents -4. **build_utmstack_collector** - Builds UTMStack Collector -5. **build_agent_manager** - Builds agent-manager Docker image -6. **build_event_processor** - Builds event processor with plugins -7. **build_backend** - Builds backend microservice (Java 17) -8. **build_frontend** - Builds frontend microservice -9. **build_user_auditor** - Builds user-auditor microservice -10. **build_web_pdf** - Builds web-pdf microservice -11. **all_builds_complete** - Checkpoint for all builds -12. **generate_changelog** - Generates AI-powered changelog (RC only) -13. **build_installer_rc** - Builds and uploads installer (RC only) -14. **deploy_installer_dev** - Deploys installer (Dev only) -15. **publish_new_version** - Publishes version to Customer Manager -16. **schedule** - Schedules release to configured instances +### Environment detection -### Permissions - -- Requires: `integration-developers` or `core-developers` team membership - -### Environment Detection +| Trigger | Environment | CM URL | Service Account | Schedule Var | +|---------|-------------|--------|------------------|--------------| +| Push to `release/v11**` | dev | `https://cm.dev.utmstack.com` | `CM_SERVICE_ACCOUNT_DEV` | `SCHEDULE_INSTANCES_DEV` | +| Prerelease | rc | `https://cm.utmstack.com` | `CM_SERVICE_ACCOUNT_PROD` | `SCHEDULE_INSTANCES_PROD` | -The pipeline automatically detects the environment based on trigger: +### Version auto-increment (dev) -| Trigger | Environment | CM URL | Service Account | Schedule Instances Var | -|---------|-------------|--------|-----------------|------------------------| -| Push to `release/v11**` | dev | `https://cm.dev.utmstack.com` | `CM_SERVICE_ACCOUNT_DEV` | `SCHEDULE_INSTANCES_DEV` | -| Prerelease created | rc | `https://cm.utmstack.com` | `CM_SERVICE_ACCOUNT_PROD` | `SCHEDULE_INSTANCES_PROD` | +1. Extracts the base version from the branch (`release/v11.2.1` → + `v11.2.1`). +2. Queries CM for the latest version. +3. If the base matches, bumps the dev suffix (`-dev.9` → `-dev.10`). +4. If the base differs, starts at `-dev.1`. -### Version Auto-Increment (Dev) +### Promotion to Community / Enterprise -For dev deployments, the version is automatically calculated: -1. Extracts base version from branch name (e.g., `release/v11.2.1` → `v11.2.1`) -2. Queries CM for latest version -3. If base versions match, increments dev number (e.g., `v11.2.1-dev.9` → `v11.2.1-dev.10`) -4. If base versions differ, starts fresh (e.g., `v11.2.1-dev.1`) +- **Community:** manual — promoting the prerelease to `latest` on GitHub + triggers the auto-deploy. +- **Enterprise:** manual with a checklist (zero crashes for 48h, no open + P0 issues). The last safety net before touching large customers. --- -## 📦 Installer Release +## Installer Release -### Flow +Trigger: GitHub Release published (type `released`). ``` -┌─────────────────────┐ -│ GitHub Release │ -│ Created & Published│ -└──────────┬──────────┘ - │ - ├─── Tag v10.x.x ──→ Build v10 Installer - └─── Tag v11.x.x ──→ Build v11 Installer +Tag v10.x.x → build v10 installer +Tag v11.x.x → build v11 installer (with ldflags: version, branch, encryption keys) ``` -### Behavior - -- Validates release tag format -- Builds installer with correct configuration: - - **V10:** Basic build - - **V11:** Build with ldflags (version, branch, encryption keys) -- Uploads installer to GitHub release assets +The installer is uploaded as a release asset. --- -## 🔐 Required Secrets and Variables +## Secrets and variables ### Secrets -| Secret Name | Used In | Description | -|-------------|---------|-------------| -| `API_SECRET` | All | GitHub API token for team membership validation | +| Secret | Used in | Description | +|--------|---------|-------------| +| `API_SECRET` | All, pr-checks | GitHub PAT with `read:org` scope. Used by deployment workflows for team-membership validation and by the `approver` job to check that the PR author belongs to `administrators` or `core-developers`. | | `AGENT_SECRET_PREFIX` | v10, v11 | Agent encryption key | -| `SIGN_CERT` | v10, v11 | Code signing certificate path (var) | +| `SIGN_CERT` | v10, v11 | Code signing certificate path (it's a `var`) | | `SIGN_KEY` | v10, v11 | Code signing key | | `SIGN_CONTAINER` | v10, v11 | Code signing container name | -| `CM_SERVICE_ACCOUNT_PROD` | v11 | Customer Manager service account credentials (prod/rc) - JSON format `{"id": "...", "key": "..."}` | -| `CM_SERVICE_ACCOUNT_DEV` | v11 | Customer Manager service account credentials (dev) - JSON format `{"id": "...", "key": "..."}` | -| `CM_ENCRYPT_SALT` | installer | Encryption salt for installer | -| `CM_SIGN_PUBLIC_KEY` | installer | Public key for installer verification | -| `OPENAI_API_KEY` | v11 | OpenAI API key for changelog generation | -| `GITHUB_TOKEN` | All | Auto-provided by GitHub Actions | +| `CM_SERVICE_ACCOUNT_PROD` | v11 | Customer Manager service account (prod/rc), JSON `{"id":"...","key":"..."}` | +| `CM_SERVICE_ACCOUNT_DEV` | v11 | Customer Manager service account (dev), JSON `{"id":"...","key":"..."}` | +| `CM_ENCRYPT_SALT` | installer | Installer encryption salt | +| `CM_SIGN_PUBLIC_KEY` | installer | Public key for verification | +| `THREATWINDS_API_KEY` | pr-checks, v11 changelog | ThreatWinds API key for `ai_review` and `generate-changelog` | +| `THREATWINDS_API_SECRET` | pr-checks, v11 changelog | ThreatWinds API secret for `ai_review` and `generate-changelog` | +| `APPROVER_APP_ID` | pr-checks | GitHub App ID for the approver bot. See [Approver GitHub App setup](#approver-github-app-setup). Without this, the approver runs in comments-only mode (no formal review, no auto-merge). | +| `APPROVER_PRIVATE_KEY` | pr-checks | GitHub App private key (full `.pem` content, multi-line) paired with `APPROVER_APP_ID`. | +| `GITHUB_TOKEN` | All | Provided automatically | ### Variables -| Variable Name | Used In | Description | Format | -|---------------|---------|-------------|--------| +| Variable | Used in | Description | Format | +|----------|---------|-------------|--------| | `SCHEDULE_INSTANCES_PROD` | v11 | Instance IDs for prod/rc scheduling | Comma-separated UUIDs | | `SCHEDULE_INSTANCES_DEV` | v11 | Instance IDs for dev scheduling | Comma-separated UUIDs | -| `TW_EVENT_PROCESSOR_VERSION_PROD` | v11 | ThreatWinds Event Processor version (prod/rc) | Semver (e.g., `1.0.0`) | -| `TW_EVENT_PROCESSOR_VERSION_DEV` | v11 | ThreatWinds Event Processor version (dev) | Semver (e.g., `1.0.0-beta`) | +| `TW_EVENT_PROCESSOR_VERSION_PROD` | v11 | ThreatWinds Event Processor version (prod/rc) | Semver (`1.0.0`) | +| `TW_EVENT_PROCESSOR_VERSION_DEV` | v11 | ThreatWinds Event Processor version (dev) | Semver (`1.0.0-beta`) | -**Example Variable Values:** -``` -SCHEDULE_INSTANCES_PROD=uuid1,uuid2,uuid3 -SCHEDULE_INSTANCES_DEV=uuid-dev1 -TW_EVENT_PROCESSOR_VERSION_PROD=1.0.0 -TW_EVENT_PROCESSOR_VERSION_DEV=1.0.0-beta +--- + +## Approver GitHub App setup + +The `approver` job uses a GitHub App (instead of a personal PAT) to leave +formal PR reviews and enable auto-merge. Pros: + +- Per-run installation token, valid for ~1 hour, auto-revoked when the + job ends. No long-lived credential in the repo. +- The App acts as its own identity, so it can `APPROVE` PRs opened by any + human contributor — including the workflow's own author (GitHub blocks + self-approval when using a PAT). +- One place to audit who/what changed your branch protection state. + +### One-time setup + +**1. Create the App.** + +Go to: `https://github.com/organizations/utmstack/settings/apps/new` + +- **GitHub App name**: e.g. `UTMStack Approver`. +- **Homepage URL**: any (the UTMStack repo URL is fine). +- **Webhook**: untick **Active** — no callbacks needed. +- **Repository permissions:** + - `Contents`: **Read-only** + - `Pull requests`: **Read and write** + - `Metadata`: Read-only (default, can't be removed). +- **Organization permissions:** + - `Members`: **Read-only** — needed for the team-membership check. +- **Where can this GitHub App be installed?** Only on this account. + +Click **Create GitHub App**. + +**2. Get the App ID and a private key.** + +On the App's settings page you'll see the **App ID** (numeric). Save it. + +Scroll to **Private keys** → **Generate a private key**. A `.pem` file +downloads. Save the **full contents** (BEGIN/END lines included). + +**3. Install the App on the UTMStack repo.** + +On the App page → **Install App** → pick the `utmstack` org → choose +**Only select repositories** → select `UTMStack` → Install. + +**4. Add the secrets to the repo.** + +Settings → Secrets and variables → Actions → New repository secret. + +- `APPROVER_APP_ID` = the numeric App ID. +- `APPROVER_PRIVATE_KEY` = the full PEM contents of the `.pem` file, + including the `-----BEGIN/END PRIVATE KEY-----` lines. Paste as-is — + GitHub preserves multi-line values. + +**5. Optional: drop `API_SECRET`.** + +If the App has `Members: Read` at org level, you can stop maintaining a +separate `API_SECRET` PAT for the permission check. The approver +workflow falls back to the App token when `API_SECRET` is not set +(`API_SECRET: ${{ secrets.API_SECRET || steps.app-token.outputs.token }}` +in `_pr-reusable-approver.yml`). + +`API_SECRET` is still used by the deployment workflows +(`v10-deployment-pipeline.yml`, `v11-deployment-pipeline.yml`) for things +like fetching private Go modules during installer builds — don't delete +it from the repo until you confirm those workflows no longer need it. + +### How it gets minted at runtime + +In `_pr-reusable-approver.yml`: + +```yaml +- name: Generate approver token from GitHub App + id: app-token + if: ${{ env.APP_ID != '' }} + env: + APP_ID: ${{ secrets.APPROVER_APP_ID }} + uses: actions/create-github-app-token@v1 + with: + app-id: ${{ secrets.APPROVER_APP_ID }} + private-key: ${{ secrets.APPROVER_PRIVATE_KEY }} ``` +If the secrets aren't configured, the step is skipped, the approver +runs in comments-only mode, and everything else still works (deps +comment, AI review comment, status check) — just no formal review and +no auto-merge. + +### Verifying it works + +1. Open a small, low-risk PR against `release/v11.x.x` (or push the + workflow to a sandbox branch). +2. After the approver job runs, check the PR page: + - The sticky `` comment is signed by your bot + account (e.g. `utmstack-approver[bot]`). + - The "Files changed" tab shows a review by the same bot, marked + `Approved` (Tier 1 + deps OK + authorized) or `Changes requested`. +3. If the target is `release/**` and Tier 1 → auto-merge is queued in + the PR header ("Auto-merge enabled by … via GitHub Actions"). + +--- + +## Reusable workflows + +**PR checks:** + +- `_pr-reusable-go-deps.yml` — runs `go-deps.sh --check --discover` at + repo level and uploads `go-deps-result` as an artifact. +- `_pr-reusable-ai-review.yml` — fan-out per prompt; each job uploads + `ai-review-` as an artifact. +- `_pr-reusable-approver.yml` — downloads artifacts, decides verdict, + posts sticky comments, optionally leaves a formal PR review. + +**Deployment pipelines:** + +- `reusable-basic.yml` — generic Docker builds. +- `reusable-golang.yml` — Go microservices. +- `reusable-java.yml` — Java microservices. +- `reusable-node.yml` — frontend / node. +- `reusable-sign-agent.yml` — agent signing. + --- -## 🎮 How to Deploy +## How to deploy + +### V10 -### V10 Deployment +**Dev:** -**Dev Environment:** ```bash git checkout release/v10.x.x -git push origin release/v10.x.x -# Automatically deploys to v10-dev +# Make changes via PR → merge → auto-deploy to v10-dev ``` -**RC Environment:** +**RC:** + ```bash -git checkout v10 -git merge release/v10.x.x -git push origin v10 -# Automatically deploys to v10-rc +# PR from release/v10.x.x → v10 → merge → auto-deploy to v10-rc ``` -**Production Release:** +**Production:** + ```bash git tag v10.5.0 git push origin v10.5.0 -# Builds production artifacts ``` -### V11 Deployment +### V11 + +**Dev:** -**Dev Environment:** ```bash -git checkout release/v11.2.1 -# Make your changes -git add . -git commit -m "Your changes" -git push origin release/v11.2.1 -# Automatically builds and deploys to dev -# Version is auto-incremented (e.g., v11.2.1-dev.1, v11.2.1-dev.2, ...) +# Open a PR against release/v11.2.1 → checks → merge → auto-deploy +# Version auto-incremented (v11.2.1-dev.1, v11.2.1-dev.2, ...) ``` -**RC Release:** -1. Navigate to GitHub Releases -2. Click "Draft a new release" -3. Create a new tag (e.g., `v11.2.1`) -4. Select "Set as a pre-release" -5. Click "Publish release" -6. Pipeline automatically: - - Builds all microservices - - Generates AI-powered changelog - - Builds and uploads installer - - Publishes version to CM - - Schedules updates to RC instances - ---- +**RC:** -## 🏗️ Reusable Workflows +1. GitHub Releases → "Draft a new release". +2. New tag (e.g. `v11.2.1`). +3. Mark as pre-release. +4. Publish. +5. The pipeline builds microservices, generates the AI changelog, uploads + the installer, publishes to CM, and schedules updates to RC instances. -The following reusable workflows are called by the main pipelines: +**Hotfix:** -- `reusable-basic.yml` - Basic Docker builds -- `reusable-golang.yml` - Golang microservice builds -- `reusable-java.yml` - Java microservice builds -- `reusable-node.yml` - Node.js/Frontend builds +```bash +git checkout v11 +git checkout -b hotfix/auth-bug +# fix → PR to v11 (label `urgent` if applicable) → checks → merge +# Recommended after merge: sync v11 into release/v11.x.x+1 +# git checkout release/v11.x.x+1 +# git merge origin/v11 # or cherry-pick the specific commits +# git push +``` --- -## 📝 Notes - -- All Docker images are pushed to `ghcr.io/utmstack/utmstack/*` -- Agent signing requires `utmstack-signer` runner -- Artifacts (agents, collector) have 1-day retention -- Failed deployments will stop the pipeline and report errors -- Dev versions follow the format `v11.x.x-dev.N` (auto-incremented) -- RC versions use the prerelease tag directly (e.g., `v11.2.1`) +## Troubleshooting + +**Permission denied:** +- Verify membership in `integration-developers` or `core-developers`. + +**`ai_review` artifact with tier 2 fallback "Manual review recommended":** +- The model didn't return valid JSON or returned an invalid tier. The + approver treats it as Tier 2 (changes requested) fail-safe. Refine the + prompt `.md` or re-run the workflow if it was transient. + +**`go_deps` fails with "Could not inspect ... run 'go mod tidy' there":** +- `go.sum` is out of sync, typically due to local `replace` directives in + `packages/`. Run `go mod tidy` in the affected module and commit. + +**The approver posts two separate comments (deps + AI):** +- That's the expected behaviour when both dimensions fail. Each comment + is independent and gets updated in place on subsequent runs. + +**The approver doesn't leave a formal review (only comments):** +- The approver GitHub App is not configured. Add both `APPROVER_APP_ID` + and `APPROVER_PRIVATE_KEY` secrets — see + [Approver GitHub App setup](#approver-github-app-setup). + +**Want a senior engineer @mentioned on Tier 3:** +- Edit `pr-checks.yml`, in the `approver` job set the `tier3_reviewers` + input with comma-separated handles: + ```yaml + with: + tier3_reviewers: 'Kbayero,osmontero' + ``` + +**Build failures:** +- Check that all required secrets are configured. +- Verify availability of the `utmstack-signer` runner (required for + agent signing). + +**Version not incrementing:** +- Check that `CM_SERVICE_ACCOUNT_DEV` / `CM_SERVICE_ACCOUNT_PROD` are + configured and that the CM API is reachable. +- The branch name must follow `release/v11.x.x`. + +**Changelog not generated:** +- Only applies to RC (prereleases). +- Verify `THREATWINDS_API_KEY` and `THREATWINDS_API_SECRET` are configured. +- To test locally: export the same secrets and run + `./scripts/test-generate-changelog.sh v11.2.8` from the repo root + (auto-detects the previous tag; the wrapper also loads them from a + local `.env` if present). --- -## 🆘 Troubleshooting - -**Permission Denied:** -- Verify you're a member of the required team -- For v11: Must be in `integration-developers` or `core-developers` team - -**Build Failures:** -- Check that all required secrets are configured -- Verify runner availability (especially `utmstack-signer` for agent builds) -- Review build logs for specific errors - -**Version Not Incrementing:** -- Check that the CM API is accessible -- Verify `CM_SERVICE_ACCOUNT_DEV` or `CM_SERVICE_ACCOUNT_PROD` secrets are correctly configured -- Ensure the branch name follows the format `release/v11.x.x` - -**Changelog Not Generated:** -- Verify `OPENAI_API_KEY` secret is configured -- Only applies to RC releases (prereleases) - ---- +## Notes -**For questions or issues, please contact the DevOps team.** +- Docker images are published to `ghcr.io/utmstack/utmstack/*`. +- Agent signing requires the `utmstack-signer` runner. +- Artifacts (agents, collector) have a 1-day retention. +- Dev versions: `v11.x.x-dev.N` (auto-incremented). +- RC versions: the prerelease tag (e.g. `v11.2.1`). diff --git a/.github/workflows/_pr-reusable-ai-review.yml b/.github/workflows/_pr-reusable-ai-review.yml new file mode 100644 index 000000000..7c37fa7ee --- /dev/null +++ b/.github/workflows/_pr-reusable-ai-review.yml @@ -0,0 +1,87 @@ +name: Reusable - AI code review + +on: + workflow_call: + inputs: + default_model: + required: false + type: string + default: 'gemini-3-flash-lite' + description: "Default model when a prompt doesn't pin one in frontmatter" + secrets: + THREATWINDS_API_KEY: + required: true + THREATWINDS_API_SECRET: + required: true + +permissions: + contents: read + pull-requests: read + +jobs: + discover: + name: Discover prompts + runs-on: ubuntu-24.04 + outputs: + prompts: ${{ steps.list.outputs.prompts }} + steps: + - uses: actions/checkout@v4 + - id: list + run: | + # README.md is documentation, not a prompt — exclude it. + mapfile -t files < <(find .github/ai-prompts -maxdepth 1 -name '*.md' \ + -not -name 'README.md' | sort) + if [ ${#files[@]} -eq 0 ]; then + echo "prompts=[]" >> "$GITHUB_OUTPUT" + echo "No AI prompts found under .github/ai-prompts/" + exit 0 + fi + # Emit objects with both path and a safe name (basename without + # extension) so the matrix can use the name as artifact suffix. + items=() + for f in "${files[@]}"; do + name=$(basename "$f" .md) + items+=("$(jq -cn --arg path "$f" --arg name "$name" '{path:$path, name:$name}')") + done + json="[$(IFS=,; echo "${items[*]}")]" + echo "prompts=$json" >> "$GITHUB_OUTPUT" + echo "Prompts to run: $json" + + review: + name: ${{ matrix.prompt.name }} + needs: discover + if: needs.discover.outputs.prompts != '[]' + runs-on: ubuntu-24.04 + strategy: + fail-fast: false + matrix: + prompt: ${{ fromJson(needs.discover.outputs.prompts) }} + steps: + - uses: actions/checkout@v4 + + - name: Fetch PR diff + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + PR_NUMBER: ${{ github.event.pull_request.number }} + run: | + # gh pr diff gives the same unified diff the GitHub UI shows. + # Simpler than `git diff base..head` and doesn't require fetch-depth: 0. + gh pr diff "$PR_NUMBER" > /tmp/pr.diff + echo "Diff size: $(wc -c < /tmp/pr.diff) bytes" + + - name: Run AI review + env: + PROMPT_FILE: ${{ matrix.prompt.path }} + DIFF_FILE: /tmp/pr.diff + OUTPUT_FILE: /tmp/result.json + AI_REVIEW_MODEL: ${{ inputs.default_model }} + THREATWINDS_API_KEY: ${{ secrets.THREATWINDS_API_KEY }} + THREATWINDS_API_SECRET: ${{ secrets.THREATWINDS_API_SECRET }} + run: bash .github/scripts/ai-review.sh + + - name: Upload result artifact + uses: actions/upload-artifact@v4 + with: + name: ai-review-${{ matrix.prompt.name }} + path: /tmp/result.json + retention-days: 7 diff --git a/.github/workflows/_pr-reusable-approver.yml b/.github/workflows/_pr-reusable-approver.yml new file mode 100644 index 000000000..7a5cb9e7d --- /dev/null +++ b/.github/workflows/_pr-reusable-approver.yml @@ -0,0 +1,100 @@ +name: Reusable - PR approver + +# Single source of truth for whether the PR passes. Downloads the artifacts +# produced by go_deps and ai_review, posts sticky PR comments, runs the +# team-membership permission check, and — using a GitHub App installation +# token minted at runtime — leaves a formal PR review and (on release/**) +# enables auto-merge. + +on: + workflow_call: + inputs: + tier3_reviewers: + required: false + type: string + default: '' + description: "Comma-separated GitHub handles to @mention when AI flags Tier 3" + org: + required: false + type: string + default: 'utmstack' + description: "GitHub org used for team-membership checks" + admin_team: + required: false + type: string + default: 'administrators' + description: "Team slug allowed to merge" + core_team: + required: false + type: string + default: 'core-developers' + description: "Second team slug allowed to merge" + merge_method: + required: false + type: string + default: 'squash' + description: "Auto-merge method on release/**: merge | squash | rebase" + secrets: + APPROVER_APP_ID: + required: false + description: "GitHub App ID for the approver bot. Without it, the job only posts sticky comments — no formal review, no auto-merge." + APPROVER_PRIVATE_KEY: + required: false + description: "GitHub App private key (full PEM content) for the approver bot." + API_SECRET: + required: false + description: "PAT with read:org for team-membership checks. Falls back to the App token if absent (App must have Organization → Members: Read)." + +permissions: + contents: read + pull-requests: write + +jobs: + approve: + name: Decide + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v4 + + # Mint a short-lived installation token from the GitHub App. The token + # is automatically revoked when the job ends. If the App secrets aren't + # configured, this step is skipped and the approver runs in comments-only + # mode (no formal review, no auto-merge). + - name: Generate approver token from GitHub App + id: app-token + if: ${{ env.APP_ID != '' }} + env: + APP_ID: ${{ secrets.APPROVER_APP_ID }} + uses: actions/create-github-app-token@v1 + with: + app-id: ${{ secrets.APPROVER_APP_ID }} + private-key: ${{ secrets.APPROVER_PRIVATE_KEY }} + + - name: Download all PR-check artifacts + uses: actions/download-artifact@v4 + with: + path: /tmp/artifacts/ + + - name: List downloaded artifacts + run: | + ls -la /tmp/artifacts/ || true + find /tmp/artifacts -type f | head -50 + + - name: Run approver + env: + ARTIFACTS_DIR: /tmp/artifacts + PR_NUMBER: ${{ github.event.pull_request.number }} + GITHUB_REPOSITORY: ${{ github.repository }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Prefer the freshly-minted App token; fall back to GITHUB_TOKEN + # for org-read calls only if API_SECRET isn't set either. + APPROVER_TOKEN: ${{ steps.app-token.outputs.token }} + API_SECRET: ${{ secrets.API_SECRET || steps.app-token.outputs.token }} + TIER3_REVIEWERS: ${{ inputs.tier3_reviewers }} + ORG: ${{ inputs.org }} + ADMIN_TEAM: ${{ inputs.admin_team }} + CORE_TEAM: ${{ inputs.core_team }} + MERGE_METHOD: ${{ inputs.merge_method }} + PR_AUTHOR: ${{ github.event.pull_request.user.login }} + BASE_REF: ${{ github.event.pull_request.base.ref }} + run: bash .github/scripts/approver.sh diff --git a/.github/workflows/_pr-reusable-go-deps.yml b/.github/workflows/_pr-reusable-go-deps.yml new file mode 100644 index 000000000..ec6884c05 --- /dev/null +++ b/.github/workflows/_pr-reusable-go-deps.yml @@ -0,0 +1,65 @@ +name: Reusable - Go dependency check + +on: + workflow_call: + inputs: + go_version: + required: false + type: string + default: '1.23' + secrets: + API_SECRET: + required: false + description: "PAT with read access to private utmstack/* Go modules. Without it, `go list -u -m -json all` fails on projects that import private repos." + +jobs: + deps: + name: Check + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-go@v5 + with: + go-version: ${{ inputs.go_version }} + + # Some projects in this monorepo (e.g. ./installer) depend on private + # utmstack/* modules. `go list -u -m -json all` tries to clone those + # over HTTPS and fails without auth — same pattern the deploy pipelines + # already use for `go build`. + - name: Configure git for private Go modules + if: ${{ env.HAS_API_SECRET == 'true' }} + env: + HAS_API_SECRET: ${{ secrets.API_SECRET != '' }} + API_SECRET: ${{ secrets.API_SECRET }} + run: | + git config --global url."https://${API_SECRET}:x-oauth-basic@github.com/".insteadOf "https://github.com/" + { + echo "GOPRIVATE=github.com/utmstack" + echo "GONOPROXY=github.com/utmstack" + echo "GONOSUMDB=github.com/utmstack" + } >> "$GITHUB_ENV" + + - name: Run go-deps.sh --check --discover + id: run + run: | + mkdir -p /tmp/deps + # Run the script but don't fail this job — we want the approver to + # be the single gate. Capture stdout+stderr and the exit code. + set +e + bash .github/scripts/go-deps.sh --check --discover \ + > /tmp/deps/output.txt 2>&1 + code=$? + set -e + echo "$code" > /tmp/deps/exit_code.txt + echo "" + echo "--- script output ---" + cat /tmp/deps/output.txt + echo "--- exit code: $code ---" + + - name: Upload result artifact + uses: actions/upload-artifact@v4 + with: + name: go-deps-result + path: /tmp/deps/ + retention-days: 7 diff --git a/.github/workflows/generate-changelog.yml b/.github/workflows/generate-changelog.yml index 18322318e..84dd7dd2f 100644 --- a/.github/workflows/generate-changelog.yml +++ b/.github/workflows/generate-changelog.yml @@ -1,5 +1,9 @@ name: Generate AI Changelog +# Thin wrapper around .github/scripts/generate-changelog.sh, which calls +# ThreatWinds /chat/completions to turn the commit log between two tags +# into end-user release notes. + on: workflow_call: inputs: @@ -8,198 +12,93 @@ on: required: true type: string previous_tag: - description: 'Previous tag to compare against (optional, auto-detected if not provided)' + description: 'Previous tag to compare against (optional, auto-detected if empty)' required: false type: string default: '' product_name: - description: 'Product name for the changelog' + description: 'Product name' required: false type: string default: 'UTMStack' product_description: - description: 'Product description for context' + description: 'Product description' required: false type: string default: 'Unified Threat Management and SIEM Platform' + model: + description: 'ThreatWinds model ID (e.g., gemini-3-flash-lite, gemini-3-pro, claude-sonnet-4-6)' + required: false + type: string + default: 'gemini-3-flash-lite' secrets: - OPENAI_API_KEY: + THREATWINDS_API_KEY: + required: true + THREATWINDS_API_SECRET: required: true outputs: changelog: - description: 'Generated changelog content' - value: ${{ jobs.generate-changelog.outputs.changelog }} + description: 'Generated changelog content (multiline)' + value: ${{ jobs.generate.outputs.changelog }} previous_tag: description: 'The previous tag used for comparison' - value: ${{ jobs.generate-changelog.outputs.previous_tag }} + value: ${{ jobs.generate.outputs.previous_tag }} jobs: - generate-changelog: - name: Generate AI Changelog - runs-on: ubuntu-latest + generate: + name: Generate + runs-on: ubuntu-24.04 outputs: - changelog: ${{ steps.generate-changelog.outputs.changelog }} - previous_tag: ${{ steps.get-tags.outputs.previous_tag }} - + changelog: ${{ steps.run.outputs.changelog }} + previous_tag: ${{ steps.run.outputs.previous_tag }} steps: - - name: Checkout code - uses: actions/checkout@v4 + - uses: actions/checkout@v4 with: + # Need full history for git tag listing and `git log A..B`. fetch-depth: 0 - - name: Get previous tag - id: get-tags + - name: Generate changelog + id: run + env: + THREATWINDS_API_KEY: ${{ secrets.THREATWINDS_API_KEY }} + THREATWINDS_API_SECRET: ${{ secrets.THREATWINDS_API_SECRET }} + PRODUCT_NAME: ${{ inputs.product_name }} + PRODUCT_DESCRIPTION: ${{ inputs.product_description }} + MODEL: ${{ inputs.model }} + OUTPUT_FILE: /tmp/changelog.md run: | - CURRENT_TAG="${{ inputs.current_tag }}" - echo "Current release tag: $CURRENT_TAG" + bash .github/scripts/generate-changelog.sh \ + "${{ inputs.current_tag }}" \ + "${{ inputs.previous_tag }}" - # Use provided previous_tag or auto-detect + # Resolve the previous tag the script actually used so the caller + # can output it (the script prints it to stdout; re-derive it the + # same way to be safe). if [ -n "${{ inputs.previous_tag }}" ]; then - PREVIOUS_TAG="${{ inputs.previous_tag }}" - echo "Using provided previous tag: $PREVIOUS_TAG" + prev='${{ inputs.previous_tag }}' else - # Get all tags sorted by version (newest first) - ALL_TAGS=$(git tag --sort=-v:refname) - FOUND_CURRENT=false - PREVIOUS_TAG="" - - for tag in $ALL_TAGS; do - if [ "$FOUND_CURRENT" = true ]; then - PREVIOUS_TAG="$tag" - break - fi - if [ "$tag" = "$CURRENT_TAG" ]; then - FOUND_CURRENT=true - fi + current='${{ inputs.current_tag }}' + all_tags=$(git tag --sort=-v:refname) + found=false + prev="" + for t in $all_tags; do + if [ "$found" = true ]; then prev="$t"; break; fi + if [ "$t" = "$current" ]; then found=true; fi done - - if [ -z "$PREVIOUS_TAG" ]; then - # No previous tag found, get first commit - PREVIOUS_TAG=$(git rev-list --max-parents=0 HEAD | head -1) - echo "No previous tag found, using first commit: $PREVIOUS_TAG" + if [ -z "$prev" ]; then + prev=$(git rev-list --max-parents=0 HEAD | head -1) fi fi + echo "previous_tag=$prev" >> "$GITHUB_OUTPUT" - echo "Previous tag/commit: $PREVIOUS_TAG" - echo "current_tag=$CURRENT_TAG" >> $GITHUB_OUTPUT - echo "previous_tag=$PREVIOUS_TAG" >> $GITHUB_OUTPUT - - - name: Get commits between tags - id: get-commits - run: | - CURRENT_TAG="${{ steps.get-tags.outputs.current_tag }}" - PREVIOUS_TAG="${{ steps.get-tags.outputs.previous_tag }}" - - echo "Getting commits between $PREVIOUS_TAG and $CURRENT_TAG" - - # Get commit messages with hash, author, and message - COMMITS=$(git log ${PREVIOUS_TAG}..${CURRENT_TAG} --pretty=format:"- %h %s (%an)" --no-merges) - - # Count commits - COMMIT_COUNT=$(git rev-list --count ${PREVIOUS_TAG}..${CURRENT_TAG} --no-merges) - - echo "Found $COMMIT_COUNT commits" - - # Save commits to file (to handle multiline) - echo "$COMMITS" > /tmp/commits.txt - - # Also get changed files summary - CHANGED_FILES=$(git diff --stat ${PREVIOUS_TAG}..${CURRENT_TAG} | tail -1) - echo "changed_files=$CHANGED_FILES" >> $GITHUB_OUTPUT - echo "commit_count=$COMMIT_COUNT" >> $GITHUB_OUTPUT - - - name: Generate changelog with OpenAI - id: generate-changelog - env: - OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} - PRODUCT_NAME: ${{ inputs.product_name }} - PRODUCT_DESCRIPTION: ${{ inputs.product_description }} - run: | - COMMITS=$(cat /tmp/commits.txt) - CURRENT_TAG="${{ steps.get-tags.outputs.current_tag }}" - PREVIOUS_TAG="${{ steps.get-tags.outputs.previous_tag }}" - COMMIT_COUNT="${{ steps.get-commits.outputs.commit_count }}" - CHANGED_FILES="${{ steps.get-commits.outputs.changed_files }}" - - # Create the prompt - PROMPT="You are a product marketing writer creating release notes for end users of a software product. - - Product: $PRODUCT_NAME - $PRODUCT_DESCRIPTION - Release: $CURRENT_TAG - - Here are the commit messages from this release: - $COMMITS - - Create user-friendly release notes in markdown format. This is for NON-TECHNICAL end users who want to know what's new and improved in the product. - - IMPORTANT RULES: - 1. ONLY include changes that DIRECTLY AFFECT END USERS - things they can see, use, or benefit from - 2. COMPLETELY IGNORE internal/technical changes like: - - CI/CD, GitHub Actions, deployment pipelines - - Code refactoring, component restructuring - - Database migrations, backend infrastructure - - Internal API changes, gRPC, service communication - - Developer tooling, linting, formatting - - README updates, internal documentation - 3. Write in simple, non-technical language - 4. Focus on BENEFITS to the user, not implementation details - 5. Group into these categories ONLY (skip empty categories): - - **What's New** - New features users can now use - - **Improved** - Enhancements to existing features - - **Fixed** - Bugs that were affecting users - 6. Start with a brief 1-2 sentence summary of the release highlights - 7. Use bullet points, be concise (one line per item) - 8. Do NOT wrap output in markdown code blocks - 9. Do NOT include commit hashes or author names - 10. If most commits are internal/technical, just summarize with 'Minor improvements and bug fixes' - - Write the release notes directly in markdown format, ready to be used as-is." - - # Call OpenAI API - RESPONSE=$(curl -s https://api.openai.com/v1/chat/completions \ - -H "Content-Type: application/json" \ - -H "Authorization: Bearer $OPENAI_API_KEY" \ - -d "$(jq -n \ - --arg prompt "$PROMPT" \ - '{ - model: "gpt-4o-mini", - messages: [ - {role: "system", content: "You are a technical writer specializing in software changelogs."}, - {role: "user", content: $prompt} - ], - temperature: 0.3, - max_tokens: 2000 - }')") - - # Extract the changelog from response - CHANGELOG=$(echo "$RESPONSE" | jq -r '.choices[0].message.content // empty') - - if [ -z "$CHANGELOG" ]; then - echo "Error: Failed to generate changelog" - echo "Response: $RESPONSE" - # Fallback to simple commit list - CHANGELOG="## What's Changed - - $COMMITS - - **Full Changelog**: https://github.com/${{ github.repository }}/compare/${PREVIOUS_TAG}...${CURRENT_TAG}" - fi - - # Add comparison link at the end - CHANGELOG="${CHANGELOG} - - --- - **Full Changelog**: https://github.com/${{ github.repository }}/compare/${PREVIOUS_TAG}...${CURRENT_TAG}" - - # Save changelog to file - echo "$CHANGELOG" > /tmp/changelog.md - - # Output changelog using multiline format - echo "changelog<> $GITHUB_OUTPUT - cat /tmp/changelog.md >> $GITHUB_OUTPUT - echo "EOF" >> $GITHUB_OUTPUT + # Multiline output for the changelog. + { + echo 'changelog<> "$GITHUB_OUTPUT" - - name: Output changelog preview + - name: Preview run: | echo "## Generated Changelog Preview" echo "" diff --git a/.github/workflows/installer-release.yml b/.github/workflows/installer-release.yml index 7af16072f..f4ecfc071 100644 --- a/.github/workflows/installer-release.yml +++ b/.github/workflows/installer-release.yml @@ -12,7 +12,7 @@ on: required: true type: string environment: - description: 'Environment (dev or rc)' + description: 'Environment (dev, rc, or production)' required: true type: string prerelease: @@ -188,12 +188,14 @@ jobs: sudo /usr/local/bin/utmstack_installer # ============================================ - # V11 RC - Upload to prerelease only + # V11 RC / Production - Build installer and upload to GitHub release. + # The `prerelease` input controls whether the GitHub Release is marked as + # a prerelease (rc) or a normal release (production). # ============================================ - build_v11_rc: - name: Build V11 Installer for Prerelease + build_v11_release: + name: Build V11 Installer for Release runs-on: ubuntu-24.04 - if: inputs.version_major == 'v11' && inputs.environment == 'rc' + if: inputs.version_major == 'v11' && (inputs.environment == 'rc' || inputs.environment == 'production') steps: - name: Check out code uses: actions/checkout@v4 diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml new file mode 100644 index 000000000..f2edb5a93 --- /dev/null +++ b/.github/workflows/pr-checks.yml @@ -0,0 +1,73 @@ +name: PR Checks + +on: + pull_request: + branches: + - 'release/**' + - 'v10' + - 'v11' + +# Cancel previous runs when new commits are pushed to the same PR. +concurrency: + group: pr-checks-${{ github.event.pull_request.number }} + cancel-in-progress: true + +permissions: + contents: read + pull-requests: write + checks: write + +jobs: + go_deps: + name: Go deps + uses: ./.github/workflows/_pr-reusable-go-deps.yml + secrets: + API_SECRET: ${{ secrets.API_SECRET }} + + ai_review: + name: AI review + uses: ./.github/workflows/_pr-reusable-ai-review.yml + secrets: + THREATWINDS_API_KEY: ${{ secrets.THREATWINDS_API_KEY }} + THREATWINDS_API_SECRET: ${{ secrets.THREATWINDS_API_SECRET }} + + approver: + name: Approver + needs: + - go_deps + - ai_review + uses: ./.github/workflows/_pr-reusable-approver.yml + with: + tier3_reviewers: 'Kbayero,osmontero' + secrets: + APPROVER_APP_ID: ${{ secrets.APPROVER_APP_ID }} + APPROVER_PRIVATE_KEY: ${{ secrets.APPROVER_PRIVATE_KEY }} + API_SECRET: ${{ secrets.API_SECRET }} + + all_checks_passed: + name: All checks passed + if: always() + needs: + - go_deps + - ai_review + - approver + runs-on: ubuntu-24.04 + steps: + - name: Verify all required check jobs succeeded + env: + NEEDS_JSON: ${{ toJson(needs) }} + run: | + echo "Dependent job results:" + echo "$NEEDS_JSON" | jq -r 'to_entries[] | " \(.key): \(.value.result)"' + + failed=$(echo "$NEEDS_JSON" | jq -r 'to_entries[] | select(.value.result != "success" and .value.result != "skipped") | .key') + + if [ -n "$failed" ]; then + echo "" + echo "❌ The following jobs did not succeed:" + echo "$failed" | sed 's/^/ - /' + exit 1 + fi + + echo "" + echo "✅ All required checks passed." diff --git a/.github/workflows/v10-deployment-pipeline.yml b/.github/workflows/v10-deployment-pipeline.yml index a8c002cb9..f69072f0f 100644 --- a/.github/workflows/v10-deployment-pipeline.yml +++ b/.github/workflows/v10-deployment-pipeline.yml @@ -2,9 +2,11 @@ name: V10 - Build & Deploy Pipeline on: push: - branches: [ 'release/v10**' ] + branches: + - 'release/v10**' + - 'v10' release: - types: [ prereleased, released ] + types: [ released ] jobs: setup_deployment: @@ -19,42 +21,31 @@ jobs: id: set-env run: | # =================== - # DEV Environment + # DEV — push to release/v10** # =================== - # Triggered by: push to release/v10.x.x branches if ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/heads/release/v10') }}; then echo "DEV environment" echo "tag=v10-dev" >> $GITHUB_OUTPUT echo "environment=dev" >> $GITHUB_OUTPUT # =================== - # RC Environment + # RC — push to v10 # =================== - # Triggered by: prerelease creation - elif ${{ github.event_name == 'release' && github.event.action == 'prereleased' }}; then - TAG="${{ github.event.release.tag_name }}" - - # Skip if tag doesn't start with v10 (this pipeline is for v10 only) - if [[ ! "$TAG" =~ ^v10\. ]]; then - echo "⏭️ Skipping: Tag '$TAG' is not a v10 release. This pipeline handles v10 releases only." - exit 0 - fi - + elif ${{ github.event_name == 'push' && github.ref == 'refs/heads/v10' }}; then echo "RC environment" echo "tag=v10-rc" >> $GITHUB_OUTPUT echo "environment=rc" >> $GITHUB_OUTPUT - echo "release_tag=$TAG" >> $GITHUB_OUTPUT + echo "release_tag=v10-rc" >> $GITHUB_OUTPUT # =================== - # PROD Environment + # PRODUCTION — release.released (non-prerelease publish) + # Skip if tag doesn't start with v10 (this pipeline is for v10 only). # =================== - # Triggered by: prerelease promoted to release elif ${{ github.event_name == 'release' && github.event.action == 'released' }}; then TAG="${{ github.event.release.tag_name }}" - # Skip if tag doesn't start with v10 (this pipeline is for v10 only) if [[ ! "$TAG" =~ ^v10\. ]]; then - echo "⏭️ Skipping: Tag '$TAG' is not a v10 release. This pipeline handles v10 releases only." + echo "⏭️ Skipping: Tag '$TAG' is not a v10 release." exit 0 fi @@ -63,40 +54,10 @@ jobs: echo "environment=prod" >> $GITHUB_OUTPUT echo "release_tag=$TAG" >> $GITHUB_OUTPUT fi - - validations: - name: Validate permissions - runs-on: ubuntu-24.04 - needs: setup_deployment - if: ${{ needs.setup_deployment.outputs.tag != '' }} - steps: - - name: Check permissions - run: | - echo "Validating user permissions..." - - RESPONSE=$(curl -s -H "Authorization: Bearer ${{ secrets.API_SECRET }}" \ - -H "Accept: application/vnd.github.json" \ - "https://api.github.com/orgs/utmstack/teams/integration-developers/memberships/${{ github.actor }}") - - if echo "$RESPONSE" | grep -q '"state": "active"'; then - echo "✅ User ${{ github.actor }} is a member of the integration-developers team." - else - RESPONSE=$(curl -s -H "Authorization: Bearer ${{ secrets.API_SECRET }}" \ - -H "Accept: application/vnd.github.json" \ - "https://api.github.com/orgs/utmstack/teams/core-developers/memberships/${{ github.actor }}") - - if echo "$RESPONSE" | grep -q '"state": "active"'; then - echo "✅ User ${{ github.actor }} is a member of the core-developers team." - else - echo "⛔ ERROR: User ${{ github.actor }} is not a member of the core-developers or integration-developers team." - echo $RESPONSE - exit 1 - fi - fi build_agent: name: Build and Sign Agent - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} runs-on: utmstack-signer steps: @@ -145,7 +106,7 @@ jobs: build_agent_manager: name: Build Agent-Manager Image - needs: [build_agent,validations,setup_deployment] + needs: [build_agent, setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} runs-on: ubuntu-22.04 steps: @@ -193,7 +154,7 @@ jobs: build_aws: name: Build AWS Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-golang.yml with: @@ -202,7 +163,7 @@ jobs: build_backend: name: Build Backend Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-java.yml with: @@ -215,7 +176,7 @@ jobs: build_correlation: name: Build Correlation Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-golang.yml with: @@ -224,7 +185,7 @@ jobs: build_frontend: name: Build Frontend Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-node.yml with: @@ -233,7 +194,7 @@ jobs: build_bitdefender: name: Build Bitdefender Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-golang.yml with: @@ -242,7 +203,7 @@ jobs: build_mutate: name: Build Mutate Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-basic.yml with: @@ -251,7 +212,7 @@ jobs: build_office365: name: Build Office365 Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-golang.yml with: @@ -260,7 +221,7 @@ jobs: build_log_auth_proxy: name: Build Log-Auth-Proxy Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-golang.yml with: @@ -269,7 +230,7 @@ jobs: build_soc_ai: name: Build Soc-AI Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-golang.yml with: @@ -278,7 +239,7 @@ jobs: build_sophos: name: Build Sophos Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-golang.yml with: @@ -287,7 +248,7 @@ jobs: build_user_auditor: name: Build User-Auditor Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-java.yml with: @@ -299,7 +260,7 @@ jobs: build_web_pdf: name: Build Web-PDF Microservice - needs: [validations,setup_deployment] + needs: [setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' }} uses: ./.github/workflows/reusable-java.yml with: diff --git a/.github/workflows/v11-deployment-pipeline.yml b/.github/workflows/v11-deployment-pipeline.yml index 3f3dc6aab..988f63e61 100644 --- a/.github/workflows/v11-deployment-pipeline.yml +++ b/.github/workflows/v11-deployment-pipeline.yml @@ -2,9 +2,11 @@ name: "v11 - Build & Deploy Pipeline" on: push: - branches: [ 'release/v11**' ] + branches: + - 'release/v11**' + - 'v11' release: - types: [ prereleased ] + types: [ released ] jobs: setup_deployment: @@ -19,43 +21,36 @@ jobs: - name: Determine Build Environment id: set-env run: | - # =================== - # DEV Environment - # =================== - # Triggered by: push to release/v11.x.x branches + # ===================================================================== + # DEV — push to release/v11** + # Version = -dev.N, auto-incremented via CM. + # ===================================================================== if ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/heads/release/v11') }}; then ENVIRONMENT="dev" CM_URL="https://cm.dev.utmstack.com" echo "Environment: $ENVIRONMENT" echo "CM URL: $CM_URL" - # Extract version from branch name (e.g., release/v11.2.1 -> v11.2.1) + # Extract version from branch name (e.g. release/v11.2.1 → v11.2.1) BRANCH_VERSION=$(echo "${{ github.ref }}" | sed 's|refs/heads/release/||') echo "Branch version: $BRANCH_VERSION" - # Get latest version from CM RESPONSE=$(curl -s "${CM_URL}/api/v1/versions/latest") LATEST_VERSION=$(echo "$RESPONSE" | jq -r '.version // empty') echo "Latest version from CM: $LATEST_VERSION" if [ -n "$LATEST_VERSION" ]; then - # Extract base version from latest (e.g., v11.2.1-dev.9 -> v11.2.1) LATEST_BASE=$(echo "$LATEST_VERSION" | sed 's/-dev\.[0-9]*$//') - echo "Latest base version: $LATEST_BASE" - if [ "$BRANCH_VERSION" = "$LATEST_BASE" ]; then - # Versions match - increment dev number DEV_NUM=$(echo "$LATEST_VERSION" | grep -oP '(?<=-dev\.)\d+') NEW_DEV_NUM=$((DEV_NUM + 1)) TAG="${BRANCH_VERSION}-dev.${NEW_DEV_NUM}" echo "Versions match, incrementing: $TAG" else - # Versions don't match - CM not updated yet, start at dev.1 TAG="${BRANCH_VERSION}-dev.1" echo "Versions don't match, starting fresh: $TAG" fi else - # No version found in CM - start at dev.1 TAG="${BRANCH_VERSION}-dev.1" echo "No previous version found, starting fresh: $TAG" fi @@ -65,66 +60,110 @@ jobs: echo "cm_url=$CM_URL" >> $GITHUB_OUTPUT echo "event_processor_tag=${{ vars.TW_EVENT_PROCESSOR_VERSION_DEV }}" >> $GITHUB_OUTPUT - # =================== - # RC Environment - # =================== - # Triggered by: prerelease creation (tag should be v11.x.x) - elif ${{ github.event_name == 'release' && github.event.action == 'prereleased' }}; then + # ===================================================================== + # RC — push to v11 + # + # Tag derivation walks two CMs to handle both the normal flow and + # the hotfix flow without divergent code paths: + # + # 1. CM DEV gives us the candidate BASE (strip `-dev.N`). + # 2. CM PROD gives us the latest version already in production. + # 3. If BASE > PROD → use BASE. Normal flow. + # If BASE <= PROD → BASE is already released (hotfix scenario) + # so bump the patch of PROD instead. Avoids overwriting an + # already-shipped tag and matches the roll-forward policy. + # ===================================================================== + elif ${{ github.event_name == 'push' && github.ref == 'refs/heads/v11' }}; then ENVIRONMENT="rc" CM_URL="https://cm.utmstack.com" + CM_DEV_URL="https://cm.dev.utmstack.com" echo "Environment: $ENVIRONMENT" - echo "CM URL: $CM_URL" + echo "CM URL (target): $CM_URL" + echo "CM URL (source for base): $CM_DEV_URL" - # Get the tag from the prerelease event - TAG="${{ github.event.release.tag_name }}" - echo "Tag from prerelease: $TAG" + DEV_RESPONSE=$(curl -s "${CM_DEV_URL}/api/v1/versions/latest") + LATEST_DEV_VERSION=$(echo "$DEV_RESPONSE" | jq -r '.version // empty') + echo "Latest dev version from CM DEV: $LATEST_DEV_VERSION" - # Skip if tag doesn't start with v11 (this pipeline is for v11 only) - if [[ ! "$TAG" =~ ^v11\. ]]; then - echo "⏭️ Skipping: Tag '$TAG' is not a v11 release. This pipeline handles v11 releases only." - exit 0 + if [ -z "$LATEST_DEV_VERSION" ]; then + echo "❌ No dev version found in CM DEV — cannot derive RC base. Push a release/v11.x.x branch first." + exit 1 + fi + + BASE=$(echo "$LATEST_DEV_VERSION" | sed -E 's/-dev\.[0-9]+$//') + echo "BASE derived from CM DEV: $BASE" + + if [[ ! "$BASE" =~ ^v11\. ]]; then + echo "❌ Derived BASE '$BASE' is not a v11 release." + exit 1 + fi + + # Double-check against production to detect the hotfix scenario. + PROD_RESPONSE=$(curl -s "${CM_URL}/api/v1/versions/latest") + PROD_LATEST=$(echo "$PROD_RESPONSE" | jq -r '.version // empty') + echo "Latest production version from CM PROD: ${PROD_LATEST:-}" + + if [ -n "$PROD_LATEST" ]; then + # sort -V puts the higher semver last. + HIGHER=$(printf '%s\n%s\n' "$BASE" "$PROD_LATEST" | sort -V | tail -1) + if [ "$HIGHER" = "$BASE" ] && [ "$BASE" != "$PROD_LATEST" ]; then + # BASE is strictly newer than PROD — use it as-is. + TAG="$BASE" + echo "BASE ($BASE) > PROD ($PROD_LATEST) — using BASE as RC tag." + else + # PROD is >= BASE → BASE was already released (hotfix case). + # Bump the patch of PROD. + MAJOR_MINOR=$(echo "$PROD_LATEST" | sed -E 's/^(v[0-9]+\.[0-9]+)\.[0-9]+.*$/\1/') + PATCH=$(echo "$PROD_LATEST" | sed -E 's/^v[0-9]+\.[0-9]+\.([0-9]+).*$/\1/') + NEW_PATCH=$((PATCH + 1)) + TAG="${MAJOR_MINOR}.${NEW_PATCH}" + echo "BASE ($BASE) <= PROD ($PROD_LATEST) — hotfix scenario. Bumping patch: $TAG" + fi + else + # CM PROD has no versions yet — use BASE. + TAG="$BASE" + echo "CM PROD is empty — using BASE as RC tag." fi + echo "RC tag: $TAG" + echo "tag=$TAG" >> $GITHUB_OUTPUT echo "environment=$ENVIRONMENT" >> $GITHUB_OUTPUT echo "cm_url=$CM_URL" >> $GITHUB_OUTPUT echo "event_processor_tag=${{ vars.TW_EVENT_PROCESSOR_VERSION_PROD }}" >> $GITHUB_OUTPUT - fi - - validations: - name: Validate permissions - runs-on: ubuntu-24.04 - needs: setup_deployment - if: ${{ needs.setup_deployment.outputs.tag != '' }} - steps: - - name: Check permissions - run: | - echo "Validating user permissions..." - RESPONSE=$(curl -s -H "Authorization: Bearer ${{ secrets.API_SECRET }}" \ - -H "Accept: application/vnd.github.json" \ - "https://api.github.com/orgs/utmstack/teams/integration-developers/memberships/${{ github.actor }}") + # ===================================================================== + # PRODUCTION — release.released (non-prerelease publish) + # + # Production does NOT rebuild anything. The images, installer, and + # changelog are all artifacts of the RC run. The production trigger + # only needs to tell CM "this version is now available to community + # instances" via a promote endpoint (TODO: define and wire up). + # ===================================================================== + elif ${{ github.event_name == 'release' && github.event.action == 'released' }}; then + ENVIRONMENT="production" + CM_URL="https://cm.utmstack.com" + echo "Environment: $ENVIRONMENT" + echo "CM URL: $CM_URL" - if echo "$RESPONSE" | grep -q '"state": "active"'; then - echo "✅ User ${{ github.actor }} is a member of the integration-developers team." - else - RESPONSE=$(curl -s -H "Authorization: Bearer ${{ secrets.API_SECRET }}" \ - -H "Accept: application/vnd.github.json" \ - "https://api.github.com/orgs/utmstack/teams/core-developers/memberships/${{ github.actor }}") + TAG="${{ github.event.release.tag_name }}" + echo "Tag from release: $TAG" - if echo "$RESPONSE" | grep -q '"state": "active"'; then - echo "✅ User ${{ github.actor }} is a member of the core-developers team." - else - echo "⛔ ERROR: User ${{ github.actor }} is not a member of the core-developers or integration-developers team." - echo $RESPONSE - exit 1 + if [[ ! "$TAG" =~ ^v11\. ]]; then + echo "⏭️ Skipping: tag '$TAG' is not a v11 release." + exit 0 fi + + echo "tag=$TAG" >> $GITHUB_OUTPUT + echo "environment=$ENVIRONMENT" >> $GITHUB_OUTPUT + echo "cm_url=$CM_URL" >> $GITHUB_OUTPUT + echo "event_processor_tag=${{ vars.TW_EVENT_PROCESSOR_VERSION_PROD }}" >> $GITHUB_OUTPUT fi build_agent: name: Build Agent Binaries needs: [setup_deployment] - if: ${{ needs.setup_deployment.outputs.tag != '' }} + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} runs-on: ubuntu-24.04 steps: - name: Check out code into the right branch @@ -224,7 +263,7 @@ jobs: sign_agent_windows: name: Sign Windows Agent Binaries needs: [build_agent, setup_deployment] - if: ${{ needs.setup_deployment.outputs.tag != '' }} + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} uses: ./.github/workflows/reusable-sign-agent.yml with: os: windows @@ -244,7 +283,7 @@ jobs: sign_agent_macos: name: Sign macOS Agent Binaries needs: [build_agent, setup_deployment] - if: ${{ needs.setup_deployment.outputs.tag != '' }} + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} uses: ./.github/workflows/reusable-sign-agent.yml with: os: macos @@ -258,7 +297,7 @@ jobs: build_utmstack_collector: name: Build UTMStack Collector needs: [setup_deployment] - if: ${{ needs.setup_deployment.outputs.tag != '' }} + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} runs-on: ubuntu-24.04 steps: - name: Check out code into the right branch @@ -292,7 +331,7 @@ jobs: build_agent_manager: name: Build Agent Manager Microservice needs: [sign_agent_windows, sign_agent_macos, build_utmstack_collector, setup_deployment] - if: ${{ always() && needs.sign_agent_windows.result == 'success' && needs.sign_agent_macos.result == 'success' && needs.build_utmstack_collector.result == 'success' && needs.setup_deployment.outputs.tag != '' }} + if: ${{ always() && needs.sign_agent_windows.result == 'success' && needs.sign_agent_macos.result == 'success' && needs.build_utmstack_collector.result == 'success' && needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} runs-on: ubuntu-24.04 steps: - name: Check out code into the right branch @@ -386,7 +425,7 @@ jobs: build_event_processor: name: Build Event Processor Microservice needs: [setup_deployment] - if: ${{ needs.setup_deployment.outputs.tag != '' }} + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} runs-on: ubuntu-24.04 steps: - name: Check out code into the right branch @@ -443,8 +482,8 @@ jobs: build_backend: name: Build Backend Microservice - needs: [validations, setup_deployment] - if: ${{ needs.setup_deployment.outputs.tag != '' }} + needs: [setup_deployment] + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} uses: ./.github/workflows/reusable-java.yml with: image_name: backend @@ -456,8 +495,8 @@ jobs: copy_filters_and_rules: true build_frontend: name: Build Frontend Microservice - needs: [validations, setup_deployment] - if: ${{ needs.setup_deployment.outputs.tag != '' }} + needs: [setup_deployment] + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} uses: ./.github/workflows/reusable-node.yml with: image_name: frontend @@ -465,8 +504,8 @@ jobs: build_user_auditor: name: Build User-Auditor Microservice - needs: [validations, setup_deployment] - if: ${{ needs.setup_deployment.outputs.tag != '' }} + needs: [setup_deployment] + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} uses: ./.github/workflows/reusable-java.yml with: image_name: user-auditor @@ -477,8 +516,8 @@ jobs: build_web_pdf: name: Build Web-PDF Microservice - needs: [validations, setup_deployment] - if: ${{ needs.setup_deployment.outputs.tag != '' }} + needs: [setup_deployment] + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} uses: ./.github/workflows/reusable-java.yml with: image_name: web-pdf @@ -502,6 +541,8 @@ jobs: steps: - run: echo "✅ All builds completed successfully." + # AI changelog runs only on RC. Production reuses the same release notes + # because the GitHub Release was already created during RC. generate_changelog: name: Generate Changelog needs: [all_builds_complete, setup_deployment] @@ -510,10 +551,15 @@ jobs: with: current_tag: ${{ needs.setup_deployment.outputs.tag }} secrets: - OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} - - build_installer_rc: - name: Build & Upload Installer (RC) + THREATWINDS_API_KEY: ${{ secrets.THREATWINDS_API_KEY }} + THREATWINDS_API_SECRET: ${{ secrets.THREATWINDS_API_SECRET }} + + # Installer build runs only on RC. The resulting binary is uploaded to the + # GitHub Release as a prerelease asset. When the release is later promoted + # to non-prerelease (which triggers production), the installer is already + # there — no rebuild needed. + build_installer_release: + name: Build & Upload Installer needs: [generate_changelog, setup_deployment] if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment == 'rc' }} uses: ./.github/workflows/installer-release.yml @@ -542,10 +588,12 @@ jobs: CM_ENCRYPT_SALT: ${{ secrets.CM_ENCRYPT_SALT }} CM_SIGN_PUBLIC_KEY: ${{ secrets.CM_SIGN_PUBLIC_KEY }} + # Publish a new version to CM. Runs for dev and rc — production does NOT + # re-publish because the rc run already registered v11.x.x in CM PROD. publish_new_version: name: Publish New Version to Customer Manager needs: [all_builds_complete, generate_changelog, setup_deployment] - if: ${{ always() && needs.all_builds_complete.result == 'success' && needs.setup_deployment.outputs.tag != '' }} + if: ${{ always() && needs.all_builds_complete.result == 'success' && needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} runs-on: ubuntu-24.04 steps: - name: Check out code @@ -558,8 +606,8 @@ jobs: TAG: ${{ needs.setup_deployment.outputs.tag }} CM_URL: ${{ needs.setup_deployment.outputs.cm_url }} run: | - # Use AI changelog for rc, generic for dev - if [ "$ENVIRONMENT" = "rc" ] && [ -n "$CHANGELOG_CONTENT" ]; then + # Use AI changelog for rc / production, generic for dev. + if [ "$ENVIRONMENT" != "dev" ] && [ -n "$CHANGELOG_CONTENT" ]; then changelog="$CHANGELOG_CONTENT" else changelog="Development build $TAG - Internal testing release" @@ -593,10 +641,13 @@ jobs: echo "Response: $response" + # Schedule the freshly-published version. For dev, this targets the dev + # instance list; for rc, the RC instance list. Production does NOT use this + # job — promotion to community lives in `promote_to_community` below. schedule: name: Schedule release to our instances needs: [publish_new_version, setup_deployment] - if: ${{ always() && needs.publish_new_version.result == 'success' && needs.setup_deployment.outputs.tag != '' }} + if: ${{ always() && needs.publish_new_version.result == 'success' && needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment != 'production' }} runs-on: ubuntu-24.04 env: ENVIRONMENT: ${{ needs.setup_deployment.outputs.environment }} @@ -652,3 +703,45 @@ jobs: echo "✅ Scheduled release for all instances with version $TAG" + promote_to_community: + name: Promote to Community + needs: [setup_deployment] + if: ${{ needs.setup_deployment.outputs.tag != '' && needs.setup_deployment.outputs.environment == 'production' }} + runs-on: ubuntu-24.04 + env: + TAG: ${{ needs.setup_deployment.outputs.tag }} + CM_URL: ${{ needs.setup_deployment.outputs.cm_url }} + steps: + - name: Schedule update for all community instances + run: | + echo "🚀 Promoting $TAG to community" + echo " CM URL: $CM_URL" + + cmAuth=$(echo '${{ secrets.CM_SERVICE_ACCOUNT_PROD }}' | jq -r '.') + auth_id=$(echo "$cmAuth" | jq -r '.id') + auth_key=$(echo "$cmAuth" | jq -r '.key') + + body=$(jq -n \ + --arg version "$TAG" \ + '{version: $version, edition: "community"}') + + response=$(curl -sS -w "\n%{http_code}" -X POST "${CM_URL}/api/v1/updates" \ + -H "Content-Type: application/json" \ + -H "id: $auth_id" \ + -H "key: $auth_key" \ + -d "$body") + + http_code=$(echo "$response" | tail -n1) + payload=$(echo "$response" | sed '$d') + + echo "HTTP $http_code" + echo "Response: $payload" + + if [ "$http_code" -ge 200 ] && [ "$http_code" -lt 300 ]; then + echo "✅ Community broadcast scheduled for $TAG" + else + echo "❌ Community broadcast failed (HTTP $http_code)" + exit 1 + fi + + diff --git a/agent-manager/go.mod b/agent-manager/go.mod index aca2a4133..94b393864 100644 --- a/agent-manager/go.mod +++ b/agent-manager/go.mod @@ -29,7 +29,7 @@ require ( github.com/goccy/go-yaml v1.19.2 // indirect github.com/jackc/pgpassfile v1.0.0 // indirect github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect - github.com/jackc/pgx/v5 v5.8.0 // indirect + github.com/jackc/pgx/v5 v5.9.2 // indirect github.com/jackc/puddle/v2 v2.2.2 // indirect github.com/jinzhu/inflection v1.0.0 // indirect github.com/jinzhu/now v1.1.5 // indirect diff --git a/agent-manager/go.sum b/agent-manager/go.sum index ba8880d00..747140f19 100644 --- a/agent-manager/go.sum +++ b/agent-manager/go.sum @@ -47,8 +47,8 @@ github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsI github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= -github.com/jackc/pgx/v5 v5.8.0 h1:TYPDoleBBme0xGSAX3/+NujXXtpZn9HBONkQC7IEZSo= -github.com/jackc/pgx/v5 v5.8.0/go.mod h1:QVeDInX2m9VyzvNeiCJVjCkNFqzsNb43204HshNSZKw= +github.com/jackc/pgx/v5 v5.9.2 h1:3ZhOzMWnR4yJ+RW1XImIPsD1aNSz4T4fyP7zlQb56hw= +github.com/jackc/pgx/v5 v5.9.2/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4= github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo= github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= github.com/jinzhu/inflection v1.0.0 h1:K317FqzuhWc8YvSVlFMCCUb36O/S9MCKRDI7QkRKD/E= diff --git a/agent/go.mod b/agent/go.mod index c5e78d93f..316ae7e01 100644 --- a/agent/go.mod +++ b/agent/go.mod @@ -16,7 +16,7 @@ require ( github.com/threatwinds/go-sdk v1.1.21 github.com/threatwinds/logger v1.2.3 github.com/utmstack/UTMStack/shared v0.0.0 - golang.org/x/sys v0.44.0 + golang.org/x/sys v0.45.0 google.golang.org/grpc v1.81.1 google.golang.org/protobuf v1.36.11 gorm.io/gorm v1.31.1 diff --git a/agent/go.sum b/agent/go.sum index 4afe41737..340588e8b 100644 --- a/agent/go.sum +++ b/agent/go.sum @@ -189,8 +189,8 @@ golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw= golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4= golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ= -golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= +golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8= golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA= golang.org/x/tools v0.42.0 h1:uNgphsn75Tdz5Ji2q36v/nsFSfR/9BRFvqhGBaJGd5k= diff --git a/backend/pom.xml b/backend/pom.xml index 80dde92ed..a581ec3ec 100644 --- a/backend/pom.xml +++ b/backend/pom.xml @@ -269,7 +269,7 @@ com.itextpdf itext7-core - 7.1.7 + 7.2.0 pom diff --git a/backend/src/main/java/com/park/utmstack/domain/application_modules/factory/impl/ModuleSocAi.java b/backend/src/main/java/com/park/utmstack/domain/application_modules/factory/impl/ModuleSocAi.java index 79f2b42b1..e32277c7b 100644 --- a/backend/src/main/java/com/park/utmstack/domain/application_modules/factory/impl/ModuleSocAi.java +++ b/backend/src/main/java/com/park/utmstack/domain/application_modules/factory/impl/ModuleSocAi.java @@ -45,12 +45,83 @@ public List checkRequirements(Long serverId) throws Exception public List getConfigurationKeys(Long groupId) throws Exception { List keys = new ArrayList<>(); + keys.add(ModuleConfigurationKey.builder() + .withGroupId(groupId) + .withConfKey("utmstack.socai.provider") + .withConfName("AI Provider") + .withConfDescription("AI provider used by SOC AI.") + .withConfDataType("text") + .withConfValue("openai") + .withConfRequired(true) + .build()); + + keys.add(ModuleConfigurationKey.builder() + .withGroupId(groupId) + .withConfKey("utmstack.socai.model") + .withConfName("AI Model") + .withConfDescription("AI model that SOC AI will use to analyze alerts (first option of active provider).") + .withConfDataType("text") + .withConfValue("gpt-4o") + .withConfRequired(true) + .build()); + + keys.add(ModuleConfigurationKey.builder() + .withGroupId(groupId) + .withConfKey("utmstack.socai.url") + .withConfName("Provider URL") + .withConfDescription("Endpoint URL for the provider (only set for azure / ollama / custom).") + .withConfDataType("text") + .withConfValue("") + .withConfRequired(false) + .build()); + + keys.add(ModuleConfigurationKey.builder() + .withGroupId(groupId) + .withConfKey("utmstack.socai.maxTokens") + .withConfName("Max Tokens") + .withConfDescription("Maximum number of tokens used per request.") + .withConfDataType("text") + .withConfValue("4096") + .withConfRequired(true) + .build()); + + keys.add(ModuleConfigurationKey.builder() + .withGroupId(groupId) + .withConfKey("utmstack.socai.authType") + .withConfName("Authentication Type") + .withConfDescription("Authentication type used to reach the provider (none for ollama).") + .withConfDataType("text") + .withConfValue("custom-headers") + .withConfRequired(true) + .build()); + + keys.add(ModuleConfigurationKey.builder() + .withGroupId(groupId) + .withConfKey("utmstack.socai.customHeaders") + .withConfName("Custom Headers") + .withConfDescription("Custom headers (JSON object) sent with each request to the provider.") + .withConfDataType("password") + .withConfValue("") + .withConfRequired(false) + .build()); + + keys.add(ModuleConfigurationKey.builder() + .withGroupId(groupId) + .withConfKey("utmstack.socai.autoAnalyze") + .withConfName("Auto Analyze") + .withConfDescription("If set to \"true\", SOC AI will automatically analyze incoming alerts.") + .withConfDataType("text") + .withConfValue("false") + .withConfRequired(false) + .build()); + keys.add(ModuleConfigurationKey.builder() .withGroupId(groupId) .withConfKey("utmstack.socai.incidentCreation") .withConfName("Automatic Incident creation") .withConfDescription("If set to \"true\", the system will create incidents based on analysis of alerts.") - .withConfDataType("bool") + .withConfDataType("text") + .withConfValue("false") .withConfRequired(false) .build()); @@ -60,37 +131,11 @@ public List getConfigurationKeys(Long groupId) throws Ex .withConfName("Change Alert Status") .withConfDescription("If set to \"true\", SOC Ai will automatically change the status of alerts. " + "Analysts should investigate those with the status \"In Review\".") - .withConfDataType("bool") + .withConfDataType("text") + .withConfValue("false") .withConfRequired(false) .build()); - keys.add(ModuleConfigurationKey.builder() - .withGroupId(groupId) - .withConfKey("utmstack.socai.model") - .withConfName("Select AI Model") - .withConfDescription("Choose the AI model that SOC AI will use to analyze alerts.") - .withConfDataType("select") - .withConfRequired(true) - .withConfOptions( - "[" + - "{\"value\": \"gpt-4\", \"label\": \"GPT-4\"}," + - "{\"value\": \"gpt-4-0613\", \"label\": \"GPT-4 (0613)\"}," + - "{\"value\": \"gpt-4-32k\", \"label\": \"GPT-4 32K\"}," + - "{\"value\": \"gpt-4-32k-0613\", \"label\": \"GPT-4 32K (0613)\"}," + - "{\"value\": \"gpt-4-turbo\", \"label\": \"GPT-4 Turbo\"}," + - "{\"value\": \"gpt-4o\", \"label\": \"GPT-4 Omni\"}," + - "{\"value\": \"gpt-4o-mini\", \"label\": \"GPT-4 Omni Mini\"}," + - "{\"value\": \"gpt-4.1\", \"label\": \"GPT-4.1\"}," + - "{\"value\": \"gpt-4.1-mini\", \"label\": \"GPT-4.1 Mini\"}," + - "{\"value\": \"gpt-4.1-nano\", \"label\": \"GPT-4.1 Nano\"}," + - "{\"value\": \"gpt-3.5-turbo\", \"label\": \"GPT-3.5 Turbo\"}," + - "{\"value\": \"gpt-3.5-turbo-0613\", \"label\": \"GPT-3.5 Turbo (0613)\"}," + - "{\"value\": \"gpt-3.5-turbo-16k\", \"label\": \"GPT-3.5 Turbo 16K\"}," + - "{\"value\": \"gpt-3.5-turbo-16k-0613\", \"label\": \"GPT-3.5 Turbo 16K (0613)\"}" + - "]" - ) - .build()); - return keys; } diff --git a/backend/src/main/java/com/park/utmstack/domain/application_modules/validators/UtmModuleConfigValidator.java b/backend/src/main/java/com/park/utmstack/domain/application_modules/validators/UtmModuleConfigValidator.java index dc05b27f3..e219a594c 100644 --- a/backend/src/main/java/com/park/utmstack/domain/application_modules/validators/UtmModuleConfigValidator.java +++ b/backend/src/main/java/com/park/utmstack/domain/application_modules/validators/UtmModuleConfigValidator.java @@ -11,7 +11,11 @@ import lombok.RequiredArgsConstructor; import org.springframework.stereotype.Service; +import java.util.ArrayList; +import java.util.HashSet; import java.util.List; +import java.util.Set; +import java.util.stream.Collectors; @Service @RequiredArgsConstructor @@ -32,7 +36,7 @@ public boolean validate(UtmModule module, List keys public boolean validate(UtmModule module, List keys, List dbConfigs) { if (keys.isEmpty()) return false; - List configDTOs = dbConfigs.stream() + List configDTOs = new ArrayList<>(dbConfigs.stream() .map(dbConf -> { UtmModuleGroupConfiguration override = findInKeys(keys, dbConf.getConfKey()); String value; @@ -45,7 +49,17 @@ public boolean validate(UtmModule module, List keys } return new UtmModuleGroupConfDTO(dbConf.getConfDataType(),dbConf.getConfKey(), value); }) - .toList(); + .collect(Collectors.toList())); + + Set dbKeys = dbConfigs.stream() + .map(UtmModuleGroupConfiguration::getConfKey) + .collect(Collectors.toCollection(HashSet::new)); + + keys.stream() + .filter(k -> !dbKeys.contains(k.getConfKey())) + .filter(k -> !Constants.MASKED_VALUE.equals(k.getConfValue())) + .map(k -> new UtmModuleGroupConfDTO(k.getConfDataType(), k.getConfKey(), k.getConfValue())) + .forEach(configDTOs::add); UtmModuleGroupConfWrapperDTO body = new UtmModuleGroupConfWrapperDTO(configDTOs); diff --git a/backend/src/main/java/com/park/utmstack/domain/ip_info/GeoIp.java b/backend/src/main/java/com/park/utmstack/domain/ip_info/GeoIp.java deleted file mode 100644 index 583481876..000000000 --- a/backend/src/main/java/com/park/utmstack/domain/ip_info/GeoIp.java +++ /dev/null @@ -1,153 +0,0 @@ -package com.park.utmstack.domain.ip_info; - -import org.springframework.util.StringUtils; - -public class GeoIp { - private String network; - private String latitude; - private String longitude; - private String localeCode; - private String continentCode; - private String continentName; - private String countryIsoCode; - private String countryName; - private String subdivision1IsoCode; - private String subdivision1IsoName; - private String subdivision2IsoCode; - private String subdivision2IsoName; - private String cityName; - private String metroCode; - private String timeZone; - - public String getNetwork() { - return network; - } - - public void setNetwork(String network) { - this.network = network; - } - - public Double getLatitude() { - return Double.valueOf(latitude); - } - - public void setLatitude(String latitude) { - this.latitude = latitude; - } - - public Double getLongitude() { - return Double.valueOf(longitude); - } - - public void setLongitude(String longitude) { - this.longitude = longitude; - } - - public String getLocaleCode() { - return localeCode; - } - - public void setLocaleCode(String localeCode) { - this.localeCode = localeCode; - } - - public String getContinentCode() { - return continentCode; - } - - public void setContinentCode(String continentCode) { - this.continentCode = continentCode; - } - - public String getContinentName() { - return continentName; - } - - public void setContinentName(String continentName) { - this.continentName = continentName; - } - - public String getCountryIsoCode() { - return countryIsoCode; - } - - public void setCountryIsoCode(String countryIsoCode) { - this.countryIsoCode = countryIsoCode; - } - - public String getCountryName() { - return countryName; - } - - public void setCountryName(String countryName) { - this.countryName = countryName; - } - - public String getSubdivision1IsoCode() { - return subdivision1IsoCode; - } - - public void setSubdivision1IsoCode(String subdivision1IsoCode) { - this.subdivision1IsoCode = subdivision1IsoCode; - } - - public String getSubdivision1IsoName() { - return subdivision1IsoName; - } - - public void setSubdivision1IsoName(String subdivision1IsoName) { - this.subdivision1IsoName = subdivision1IsoName; - } - - public String getSubdivision2IsoCode() { - return subdivision2IsoCode; - } - - public void setSubdivision2IsoCode(String subdivision2IsoCode) { - this.subdivision2IsoCode = subdivision2IsoCode; - } - - public String getSubdivision2IsoName() { - return subdivision2IsoName; - } - - public void setSubdivision2IsoName(String subdivision2IsoName) { - this.subdivision2IsoName = subdivision2IsoName; - } - - public String getCityName() { - return cityName; - } - - public void setCityName(String cityName) { - this.cityName = cityName; - } - - public String getMetroCode() { - return metroCode; - } - - public void setMetroCode(String metroCode) { - this.metroCode = metroCode; - } - - public String getTimeZone() { - return timeZone; - } - - public void setTimeZone(String timeZone) { - this.timeZone = timeZone; - } - - @Override - public String toString() { - return "Continent Name: " + (StringUtils.hasText(continentName) ? continentName : "-") + "\n" + - "Continent Code: " + (StringUtils.hasText(continentCode) ? continentCode : "-") + "\n" + - "Country Name: " + (StringUtils.hasText(countryName) ? countryName : "-") + "\n" + - "Country ISO Code: " + (StringUtils.hasText(countryIsoCode) ? countryIsoCode : "-") + "\n" + - "City Name: " + (StringUtils.hasText(cityName) ? cityName : "-") + "\n" + - "Network: " + (StringUtils.hasText(network) ? network : "-") + "\n" + - "Latitude: " + (StringUtils.hasText(latitude) ? latitude : "-") + "\n" + - "Longitude: " + (StringUtils.hasText(longitude) ? longitude : "-") + "\n"; - } -} diff --git a/backend/src/main/java/com/park/utmstack/service/UtmAlertTagRuleService.java b/backend/src/main/java/com/park/utmstack/service/UtmAlertTagRuleService.java index 77ba84e50..ac46bec83 100644 --- a/backend/src/main/java/com/park/utmstack/service/UtmAlertTagRuleService.java +++ b/backend/src/main/java/com/park/utmstack/service/UtmAlertTagRuleService.java @@ -182,6 +182,7 @@ private void releaseToOpen(Instant rulesEvaluationStart) { List filters = new ArrayList<>(); filters.add(new FilterType(Constants.alertStatus, OperatorType.IS, AlertStatus.AUTOMATIC_REVIEW.getCode())); filters.add(new FilterType(Constants.timestamp, OperatorType.IS_LESS_THAN_OR_EQUALS, rulesEvaluationStart.toString())); + filters.add(new FilterType(Constants.alertTags, OperatorType.DOES_NOT_CONTAIN, Constants.FALSE_POSITIVE_TAG)); Query query = SearchUtil.toQuery(filters); String indexPattern = Constants.SYS_INDEX_PATTERN.get(SystemIndexPattern.ALERTS); diff --git a/backend/src/main/java/com/park/utmstack/service/elasticsearch/ElasticsearchService.java b/backend/src/main/java/com/park/utmstack/service/elasticsearch/ElasticsearchService.java index 01a76e672..f45a84f9a 100644 --- a/backend/src/main/java/com/park/utmstack/service/elasticsearch/ElasticsearchService.java +++ b/backend/src/main/java/com/park/utmstack/service/elasticsearch/ElasticsearchService.java @@ -341,7 +341,7 @@ public SearchResponse search(List filters, Integer top, Strin try { Assert.hasText(indexPattern, "Parameter indexPattern must not be null or empty"); SearchRequest query = buildQuery(indexPattern, filters, top, pageable); - return client.getClient().search(query, type); + return client.execute(c -> c.search(query, type)); } catch (Exception e) { throw new RuntimeException(ctx + ": " + e.getMessage()); } @@ -400,12 +400,86 @@ public Map getLatestDocument(List filters, String in public SearchResponse search(SearchRequest request, Class type) { final String ctx = CLASSNAME + ".search"; try { - return client.getClient().search(request, type); + return client.execute(c -> c.search(request, type)); } catch (Exception e) { throw new RuntimeException(ctx + ": " + e.getMessage()); } } + @FunctionalInterface + public interface SearchBatchConsumer { + /** Returns false to stop iteration early. */ + boolean accept(List batch) throws Exception; + } + + /** + * Streams a result set using search_after pagination, never holding more than {@code pageSize} + * documents in memory at a time. Designed for very large exports where loading every hit at + * once would OOM the JVM (and take the OpenSearch client's I/O reactor down with it). + * + * Sort is forced to {@code @timestamp desc} with {@code _id desc} as tiebreaker so that + * search_after is stable and deterministic. + * + * @param filters filters to apply + * @param max hard upper bound on total documents to emit; null or <=0 means unbounded + * @param indexPattern target index pattern + * @param pageSize batch size (capped at 10000 by OpenSearch per request) + * @param type deserialization type + * @param consumer receives each batch; return false to stop early + * @return total number of documents emitted + */ + public long searchStream(List filters, Integer max, String indexPattern, + int pageSize, Class type, SearchBatchConsumer consumer) { + final String ctx = CLASSNAME + ".searchStream"; + try { + Assert.hasText(indexPattern, "Parameter indexPattern must not be null or empty"); + Assert.notNull(consumer, "consumer must not be null"); + if (pageSize <= 0) pageSize = 500; + + long emitted = 0; + List after = null; + while (true) { + int remaining = (max != null && max > 0) ? (int) (max - emitted) : pageSize; + if (remaining <= 0) break; + int size = Math.min(pageSize, remaining); + + final List afterFinal = after; + final int sizeFinal = size; + SearchResponse response = client.execute(c -> { + SearchRequest.Builder srb = new SearchRequest.Builder() + .index(indexPattern) + .query(SearchUtil.toQuery(filters)) + .size(sizeFinal) + .sort(s -> s.field(f -> f.field("@timestamp").order(SortOrder.Desc))) + .sort(s -> s.field(f -> f.field("_id").order(SortOrder.Desc))); + if (afterFinal != null && !afterFinal.isEmpty()) + srb.searchAfter(afterFinal); + return c.search(srb.build(), type); + }); + + if (response == null || response.hits() == null) break; + List> hits = response.hits().hits(); + if (hits == null || hits.isEmpty()) break; + + List batch = new ArrayList<>(hits.size()); + for (org.opensearch.client.opensearch.core.search.Hit h : hits) + batch.add(h.source()); + + boolean keepGoing = consumer.accept(batch); + emitted += hits.size(); + + if (!keepGoing) break; + if (hits.size() < size) break; + + after = hits.get(hits.size() - 1).sort(); + if (after == null || after.isEmpty()) break; + } + return emitted; + } catch (Exception e) { + throw new RuntimeException(ctx + ": " + e.getMessage(), e); + } + } + public void updateByQuery(Query query, String index, String script) { final String ctx = CLASSNAME + ".updateByQuery"; try { diff --git a/backend/src/main/java/com/park/utmstack/service/elasticsearch/OpensearchClientBuilder.java b/backend/src/main/java/com/park/utmstack/service/elasticsearch/OpensearchClientBuilder.java index f6f62d042..e2db9cc13 100644 --- a/backend/src/main/java/com/park/utmstack/service/elasticsearch/OpensearchClientBuilder.java +++ b/backend/src/main/java/com/park/utmstack/service/elasticsearch/OpensearchClientBuilder.java @@ -16,26 +16,71 @@ public class OpensearchClientBuilder { private static final String CLASSNAME = "OpensearchClientBuilder"; private final Logger log = LoggerFactory.getLogger(OpensearchClientBuilder.class); - private OpenSearch client; + private volatile OpenSearch client; + + @FunctionalInterface + public interface OsAction { + T apply(OpenSearch client) throws Exception; + } @Order(Ordered.HIGHEST_PRECEDENCE) @EventListener(ApplicationReadyEvent.class) public void init() throws Exception { - final String ctx = CLASSNAME + ".init"; + buildClient(); + } + + public OpenSearch getClient() { + return client; + } + + /** + * Runs an action against the OpenSearch client with one-shot recovery: if the underlying + * Apache HttpAsyncClient I/O reactor has transitioned to STOPPED (typically after an OOM + * or a fatal callback exception while streaming a very large response), the singleton + * client is rebuilt and the action is retried once. All other failures propagate unchanged. + * Callers that don't need recovery should keep using {@link #getClient()} directly. + */ + public T execute(OsAction action) throws Exception { + try { + return action.apply(client); + } catch (Exception e) { + if (!isReactorStopped(e)) + throw e; + log.warn("OpenSearch I/O reactor is STOPPED; rebuilding client and retrying once", e); + rebuild(); + return action.apply(client); + } + } + + public synchronized void rebuild() { + final String ctx = CLASSNAME + ".rebuild"; + try { + OpenSearch old = this.client; + buildClient(); + tryClose(old); + } catch (Exception e) { + String msg = ctx + ": " + e.getMessage(); + log.error(msg); + throw new RuntimeException(msg); + } + } + + private synchronized void buildClient() { + final String ctx = CLASSNAME + ".buildClient"; try { String host = System.getenv(Constants.ENV_ELASTICSEARCH_HOST); - Assert.hasText(host, "Environment variable ELASTICSEARCH_HOST is missing or his value is null or empty"); + Assert.hasText(host, "Environment variable ELASTICSEARCH_HOST is missing or its value is null or empty"); String port = System.getenv(Constants.ENV_ELASTICSEARCH_PORT); - Assert.hasText(port, "Environment variable ELASTICSEARCH_PORT is missing or his value is null or empty"); + Assert.hasText(port, "Environment variable ELASTICSEARCH_PORT is missing or its value is null or empty"); String user = System.getenv(Constants.ENV_ELASTICSEARCH_USER); - Assert.hasText(user, "Environment variable ELASTICSEARCH_USER is missing or his value is null or empty"); + Assert.hasText(user, "Environment variable ELASTICSEARCH_USER is missing or its value is null or empty"); String password = System.getenv(Constants.ENV_ELASTICSEARCH_PASSWORD); - Assert.hasText(password, "Environment variable ELASTICSEARCH_PASSWORD is missing or his value is null or empty"); + Assert.hasText(password, "Environment variable ELASTICSEARCH_PASSWORD is missing or its value is null or empty"); - client = OpenSearch.builder() + this.client = OpenSearch.builder() .withHost(host, Integer.parseInt(port), HttpScheme.https) .withCredentials(user, password) .build(); @@ -46,7 +91,28 @@ public void init() throws Exception { } } - public OpenSearch getClient() { - return client; + private void tryClose(OpenSearch old) { + if (old == null) return; + try { + if (old instanceof AutoCloseable) { + ((AutoCloseable) old).close(); + } + } catch (Exception ignored) { + // best-effort: the old client is unusable anyway + } + } + + /** + * Detects the Apache HttpAsyncClient "Request cannot be executed; I/O reactor status: STOPPED" + * condition anywhere in the cause chain. + */ + public static boolean isReactorStopped(Throwable t) { + while (t != null) { + String msg = t.getMessage(); + if (msg != null && msg.contains("I/O reactor") && msg.contains("STOPPED")) + return true; + t = t.getCause(); + } + return false; } } diff --git a/backend/src/main/java/com/park/utmstack/service/ip_info/IpInfoService.java b/backend/src/main/java/com/park/utmstack/service/ip_info/IpInfoService.java deleted file mode 100644 index 3ece554cb..000000000 --- a/backend/src/main/java/com/park/utmstack/service/ip_info/IpInfoService.java +++ /dev/null @@ -1,55 +0,0 @@ -package com.park.utmstack.service.ip_info; - -import com.park.utmstack.config.ApplicationProperties; -import com.park.utmstack.domain.chart_builder.types.query.FilterType; -import com.park.utmstack.domain.chart_builder.types.query.OperatorType; -import com.park.utmstack.domain.ip_info.GeoIp; -import com.park.utmstack.service.elasticsearch.ElasticsearchService; -import com.park.utmstack.service.elasticsearch.SearchUtil; -import com.park.utmstack.util.exceptions.UtmIpInfoException; -import org.opensearch.client.opensearch.core.SearchRequest; -import org.opensearch.client.opensearch.core.search.HitsMetadata; -import org.springframework.stereotype.Service; - -import java.util.ArrayList; -import java.util.List; - -@Service -public class IpInfoService { - private static final String CLASSNAME = "IpInfoService"; - - private final ApplicationProperties applicationProperties; - private final ElasticsearchService elasticsearchService; - - public IpInfoService(ApplicationProperties applicationProperties, - ElasticsearchService elasticsearchService) { - this.applicationProperties = applicationProperties; - this.elasticsearchService = elasticsearchService; - } - - /** - * Get information about an Ip - * - * @param ip The ip to get the related information - * @return A ${@link GeoIp} object with the ip information - */ - public GeoIp getIpInfo(String ip) throws UtmIpInfoException { - final String ctx = CLASSNAME + "getIpV4Info"; - try { - List filters = new ArrayList<>(); - filters.add(new FilterType("network", OperatorType.IS, ip)); - - SearchRequest sr = SearchRequest.of(s -> s.index(applicationProperties.getChartBuilder().getIpInfoIndexName()) - .query(SearchUtil.toQuery(filters)).size(1)); - - HitsMetadata hits = elasticsearchService.search(sr, GeoIp.class).hits(); - - if (hits.total().value() <= 0) - return null; - - return hits.hits().get(0).source(); - } catch (Exception e) { - throw new RuntimeException(ctx + ": " + e.getLocalizedMessage()); - } - } -} diff --git a/backend/src/main/java/com/park/utmstack/util/UtilCsv.java b/backend/src/main/java/com/park/utmstack/util/UtilCsv.java index 96a907998..179020008 100644 --- a/backend/src/main/java/com/park/utmstack/util/UtilCsv.java +++ b/backend/src/main/java/com/park/utmstack/util/UtilCsv.java @@ -1,6 +1,7 @@ package com.park.utmstack.util; import com.fasterxml.jackson.databind.ObjectMapper; +import com.jayway.jsonpath.DocumentContext; import com.jayway.jsonpath.JsonPath; import com.jayway.jsonpath.PathNotFoundException; import com.park.utmstack.domain.shared_types.DataColumn; @@ -13,6 +14,7 @@ import org.springframework.util.StringUtils; import javax.servlet.http.HttpServletResponse; +import java.io.IOException; import java.time.Instant; import java.time.format.DateTimeFormatter; import java.util.ArrayList; @@ -51,6 +53,7 @@ public static void prepareToDownload(HttpServletResponse response, DataColumn[] List rows = new ArrayList<>(); data.forEach(d -> { + DocumentContext docctx = JsonPath.parse(d); String[] cells = new String[columns.length]; for (int i = 0; i < columns.length; i++) { String fieldName = columns[i].getField(); @@ -59,7 +62,7 @@ public static void prepareToDownload(HttpServletResponse response, DataColumn[] Object value; try { - value = JsonPath.parse(d).read("$." + fieldName); + value = docctx.read("$." + fieldName); } catch (PathNotFoundException e) { continue; } @@ -81,6 +84,7 @@ public static void prepareToDownload(HttpServletResponse response, DataColumn[] cells[i] = value.toString(); } } + cells[i] = sanitizeCsvCell(cells[i]); } rows.add(cells); }); @@ -104,4 +108,91 @@ public static void prepareToDownload(HttpServletResponse response, DataColumn[] throw new UtmCsvException(msg); } } + + /** + * Opens a CSV response stream: sets content-type/disposition headers and writes the header row. + * Caller is responsible for closing the returned printer (try-with-resources is fine). + * + * Column names are normalized in-place by stripping a trailing {@code .keyword}. + */ + public static CSVPrinter openCsvStream(HttpServletResponse response, DataColumn[] columns) throws IOException { + Assert.notEmpty(columns); + + Arrays.stream(columns).forEach(column -> + column.setField(column.getField().replace(".keyword", ""))); + + String[] headers = Stream.of(columns).map(column -> { + if (StringUtils.hasText(column.getLabel())) + return column.getLabel(); + return column.getField().replace(".keyword", ""); + }).toArray(String[]::new); + + response.setContentType("text/csv"); + response.setHeader(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=data.csv"); + + return new CSVPrinter(response.getWriter(), + CSVFormat.DEFAULT.withHeader(headers).withQuoteMode(QuoteMode.ALL)); + } + + /** + * Writes a batch of source maps as CSV rows using the same field-extraction logic as + * {@link #prepareToDownload}. Intended to be called repeatedly while paginating through + * a large result set; pair with {@link #openCsvStream}. + */ + public static void writeCsvBatch(CSVPrinter printer, DataColumn[] columns, List data) throws IOException { + if (data == null || data.isEmpty()) return; + + final DateTimeFormatter DATE_FORMATTER = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss z") + .withLocale(Locale.getDefault()).withZone(TimezoneUtil.getAppTimezone()); + + for (Object d : data) { + DocumentContext ctx = JsonPath.parse(d); + String[] cells = new String[columns.length]; + for (int i = 0; i < columns.length; i++) { + String fieldName = columns[i].getField(); + String fieldType = columns[i].getType(); + cells[i] = null; + + Object value; + try { + value = ctx.read("$." + fieldName); + } catch (PathNotFoundException e) { + continue; + } + + if (value == null) + continue; + + if (value instanceof String) { + cells[i] = "date".equals(fieldType) ? DATE_FORMATTER.format(Instant.parse(String.valueOf(value))) : + String.valueOf(value).replace("\n", " ").replace("\t", " "); + } else if (value instanceof List) { + cells[i] = ((List) value).stream().map(String::valueOf).collect(Collectors.joining(",")); + } else if (value instanceof Number) { + cells[i] = String.valueOf(value); + } else if (value instanceof Map) { + try { + cells[i] = OBJECT_MAPPER.writeValueAsString(value); + } catch (Exception ex) { + cells[i] = value.toString(); + } + } + cells[i] = sanitizeCsvCell(cells[i]); + } + printer.printRecord((Object[]) cells); + } + printer.flush(); + } + + /** + * Neutralizes CSV-injection payloads by prefixing a single quote to any cell whose first + * character is interpreted as a formula trigger by Excel/LibreOffice/Sheets. + */ + private static String sanitizeCsvCell(String value) { + if (value == null || value.isEmpty()) return value; + char first = value.charAt(0); + if (first == '=' || first == '+' || first == '-' || first == '@' || first == '\t' || first == '\r') + return "'" + value; + return value; + } } diff --git a/backend/src/main/java/com/park/utmstack/util/chart_builder/elasticsearch_dsl/requests/RequestDsl.java b/backend/src/main/java/com/park/utmstack/util/chart_builder/elasticsearch_dsl/requests/RequestDsl.java index a5c35aae6..ac20f4a15 100644 --- a/backend/src/main/java/com/park/utmstack/util/chart_builder/elasticsearch_dsl/requests/RequestDsl.java +++ b/backend/src/main/java/com/park/utmstack/util/chart_builder/elasticsearch_dsl/requests/RequestDsl.java @@ -23,6 +23,11 @@ public class RequestDsl { private static final String CLASSNAME = "RequestDsl"; + public static final String GEO_HIT_AGG = "_geo_hit"; + public static final List GEO_SOURCE_INCLUDES = List.of( + "origin.geolocation.latitude", "origin.geolocation.longitude", + "source.geolocation.latitude", "source.geolocation.longitude", + "destination.geolocation.latitude", "destination.geolocation.longitude"); private final SearchRequest.Builder searchRequestBuilder; private final UtmVisualization visualization; @@ -130,6 +135,14 @@ private void buildAggregation() throws UtmElasticsearchException { Map bucketAggregations = buildBucketAggregation(bucket); Map metricAggregations = buildMetricAggregation(metrics); + if (visualization.getChartType() == ChartType.COORDINATE_MAP_CHART) { + Map withGeo = new LinkedHashMap<>(metricAggregations); + withGeo.put(GEO_HIT_AGG, Aggregation.of(a -> a.topHits(th -> th + .size(1) + .source(s -> s.filter(f -> f.includes(GEO_SOURCE_INCLUDES)))))); + metricAggregations = withGeo; + } + if (!CollectionUtils.isEmpty(bucketAggregations)) { Map root = new LinkedHashMap<>(); if (bucketAggregations.size() > 1) { diff --git a/backend/src/main/java/com/park/utmstack/util/chart_builder/elasticsearch_dsl/responses/impl/coordinate_map/ResponseParserForCoordinateMapChart.java b/backend/src/main/java/com/park/utmstack/util/chart_builder/elasticsearch_dsl/responses/impl/coordinate_map/ResponseParserForCoordinateMapChart.java index d3605edae..ca5406d7d 100644 --- a/backend/src/main/java/com/park/utmstack/util/chart_builder/elasticsearch_dsl/responses/impl/coordinate_map/ResponseParserForCoordinateMapChart.java +++ b/backend/src/main/java/com/park/utmstack/util/chart_builder/elasticsearch_dsl/responses/impl/coordinate_map/ResponseParserForCoordinateMapChart.java @@ -1,17 +1,18 @@ package com.park.utmstack.util.chart_builder.elasticsearch_dsl.responses.impl.coordinate_map; +import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.node.ObjectNode; import com.park.utmstack.domain.chart_builder.UtmVisualization; import com.park.utmstack.domain.chart_builder.types.aggregation.AggregationType; import com.park.utmstack.domain.chart_builder.types.aggregation.Bucket; import com.park.utmstack.domain.chart_builder.types.aggregation.Metric; -import com.park.utmstack.domain.ip_info.GeoIp; -import com.park.utmstack.service.ip_info.IpInfoService; +import com.park.utmstack.util.chart_builder.elasticsearch_dsl.requests.RequestDsl; import com.park.utmstack.util.chart_builder.elasticsearch_dsl.responses.ResponseParser; -import com.park.utmstack.util.exceptions.UtmIpInfoException; import com.utmstack.opensearch_connector.parsers.TermAggregateParser; import com.utmstack.opensearch_connector.types.BucketAggregation; import com.utmstack.opensearch_connector.types.SearchSqlResponse; +import org.opensearch.client.opensearch._types.aggregations.Aggregate; +import org.opensearch.client.opensearch._types.aggregations.TopHitsAggregate; import org.opensearch.client.opensearch.core.SearchResponse; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -27,15 +28,10 @@ @Component public class ResponseParserForCoordinateMapChart implements ResponseParser { private static final String CLASSNAME = "ResponseParserForCoordinateMapChart"; + private static final List GEO_PREFIXES = List.of("origin", "source", "destination"); private final Logger log = LoggerFactory.getLogger(ResponseParserForCoordinateMapChart.class); - private final IpInfoService ipInfoService; - - public ResponseParserForCoordinateMapChart(IpInfoService ipInfoService) { - this.ipInfoService = ipInfoService; - } - @Override public List parse(UtmVisualization visualization, SearchResponse result) { final String ctx = CLASSNAME + ".parse"; @@ -53,20 +49,13 @@ public List parse(UtmVisualization visualization, Sear for (BucketAggregation entry : entries) { - GeoIp ipV4Info; - try { - ipV4Info = ipInfoService.getIpInfo(entry.getKey()); - - if (ipV4Info == null) - continue; - } catch (UtmIpInfoException e) { - log.error(e.getMessage()); + Double[] latLon = extractLatLongFromTopHits(entry.getSubAggregations()); + if (latLon == null) continue; - } CoordinateMapChartResult value = new CoordinateMapChartResult(); value.setName(entry.getKey()); - value.addLatitude(ipV4Info.getLatitude()).addLongitude(ipV4Info.getLongitude()); + value.addLatitude(latLon[0]).addLongitude(latLon[1]); switch (metric.getAggregation()) { case COUNT: @@ -140,20 +129,14 @@ public List parse(UtmVisualization visualization, Sear } if (!StringUtils.hasText(ip)) continue; - GeoIp ipInfo; - try { - ipInfo = ipInfoService.getIpInfo(ip); - if (ipInfo == null) continue; - } catch (UtmIpInfoException e) { - log.error(e.getMessage()); - continue; - } + Double[] latLon = extractLatLongFromRow(row); + if (latLon == null) continue; CoordinateMapChartResult chartResult = new CoordinateMapChartResult(); chartResult.setName(ip); chartResult.setValue(new Double[] { - ipInfo.getLatitude(), - ipInfo.getLongitude(), + latLon[0], + latLon[1], (double) i }); @@ -165,4 +148,59 @@ public List parse(UtmVisualization visualization, Sear throw new RuntimeException(ctx + ": " + e.getMessage(), e); } } + + private Double[] extractLatLongFromTopHits(Map subAggs) { + if (subAggs == null) return null; + Aggregate geoAgg = subAggs.get(RequestDsl.GEO_HIT_AGG); + if (geoAgg == null || !geoAgg.isTopHits()) return null; + TopHitsAggregate topHits = geoAgg.topHits(); + if (topHits.hits() == null || topHits.hits().hits().isEmpty()) return null; + var hit = topHits.hits().hits().get(0); + if (hit.source() == null) return null; + try { + ObjectNode source = hit.source().to(ObjectNode.class); + return extractLatLongFromJson(source); + } catch (Exception e) { + log.warn("Failed to deserialize top_hits source for geolocation: {}", e.getMessage()); + return null; + } + } + + private Double[] extractLatLongFromJson(ObjectNode source) { + if (source == null) return null; + for (String prefix : GEO_PREFIXES) { + JsonNode geo = source.path(prefix).path("geolocation"); + JsonNode lat = geo.path("latitude"); + JsonNode lon = geo.path("longitude"); + if (lat.isNumber() && lon.isNumber()) { + return new Double[]{lat.asDouble(), lon.asDouble()}; + } + } + return null; + } + + private Double[] extractLatLongFromRow(Map row) { + Double lat = null; + Double lon = null; + for (Map.Entry e : row.entrySet()) { + if (e.getValue() == null) continue; + String key = e.getKey(); + if (lat == null && (key.endsWith(".geolocation.latitude") || key.equals("latitude"))) { + lat = toDouble(e.getValue()); + } else if (lon == null && (key.endsWith(".geolocation.longitude") || key.equals("longitude"))) { + lon = toDouble(e.getValue()); + } + } + if (lat == null || lon == null) return null; + return new Double[]{lat, lon}; + } + + private Double toDouble(Object val) { + if (val instanceof Number) return ((Number) val).doubleValue(); + try { + return Double.parseDouble(val.toString()); + } catch (NumberFormatException e) { + return null; + } + } } diff --git a/backend/src/main/java/com/park/utmstack/web/rest/application_modules/UtmModuleGroupResource.java b/backend/src/main/java/com/park/utmstack/web/rest/application_modules/UtmModuleGroupResource.java index 1b420fbec..3d5735bfa 100644 --- a/backend/src/main/java/com/park/utmstack/web/rest/application_modules/UtmModuleGroupResource.java +++ b/backend/src/main/java/com/park/utmstack/web/rest/application_modules/UtmModuleGroupResource.java @@ -30,6 +30,7 @@ import javax.validation.Valid; import java.net.URISyntaxException; import java.util.ArrayList; +import java.util.HashSet; import java.util.List; import java.util.Optional; @@ -82,6 +83,15 @@ public ResponseEntity createConfigurationGroup(@Valid @RequestBo defaultConfigurationKeys.forEach(key -> keys.add(new UtmModuleGroupConfiguration(key))); moduleGroupConfigurationService.createConfigurationKeys(keys); + for (UtmModuleGroupConfiguration conf : keys) { + if ((Constants.CONF_TYPE_PASSWORD.equals(conf.getConfDataType()) + || Constants.CONF_TYPE_FILE.equals(conf.getConfDataType())) + && conf.getConfValue() != null) { + conf.setConfValue(Constants.MASKED_VALUE); + } + } + result.setModuleGroupConfigurations(new HashSet<>(keys)); + return ResponseEntity.ok(result); } catch (DataIntegrityViolationException e) { String msg = ctx + ": " + e.getMostSpecificCause().getMessage().replaceAll("\n", ""); diff --git a/backend/src/main/java/com/park/utmstack/web/rest/elasticsearch/ElasticsearchResource.java b/backend/src/main/java/com/park/utmstack/web/rest/elasticsearch/ElasticsearchResource.java index c1d6623a3..a69c54110 100644 --- a/backend/src/main/java/com/park/utmstack/web/rest/elasticsearch/ElasticsearchResource.java +++ b/backend/src/main/java/com/park/utmstack/web/rest/elasticsearch/ElasticsearchResource.java @@ -23,6 +23,7 @@ import com.utmstack.opensearch_connector.types.SearchSqlResponse; import com.utmstack.opensearch_connector.types.SqlQueryRequest; import lombok.RequiredArgsConstructor; +import org.apache.commons.csv.CSVPrinter; import org.opensearch.client.opensearch.cat.indices.IndicesRecord; import org.opensearch.client.opensearch.core.SearchResponse; import org.opensearch.client.opensearch.core.search.Hit; @@ -197,43 +198,45 @@ public ResponseEntity> search(@RequestBody(required = false) List searchToCsv(@RequestBody @Valid CsvExportingParams params, HttpServletResponse response) { final String ctx = CLASSNAME + ".searchToCsv"; - try { - SearchResponse searchResponse = elasticsearchService.search(params.getFilters(), params.getTop(), - params.getIndexPattern(), Pageable.unpaged(), Map.class); - - if (Objects.isNull(searchResponse) || Objects.isNull(searchResponse.hits()) || searchResponse.hits().total().value() == 0) - return ResponseEntity.ok().build(); - - List hits = searchResponse.hits().hits().stream().map(Hit::source).collect(Collectors.toList()); - boolean needsEchoes = false; - for (DataColumn col : params.getColumns()) { - if ("echoes".equals(col.getField())) { - needsEchoes = true; - break; - } + boolean needsEchoes = false; + for (DataColumn col : params.getColumns()) { + if ("echoes".equals(col.getField())) { + needsEchoes = true; + break; } - if (needsEchoes) { - hits.forEach(d -> { - Object id = d.get("id"); - if (id != null) { - long countEchoes = elasticsearchService.count( - List.of(new FilterType("parentId", OperatorType.IS, id.toString())), - params.getIndexPattern() - ); - d.put("echoes", countEchoes); - } - }); - } - - UtilCsv.prepareToDownload(response, params.getColumns(), hits); - + } + final boolean enrichEchoes = needsEchoes; + + try (CSVPrinter printer = UtilCsv.openCsvStream(response, params.getColumns())) { + elasticsearchService.searchStream( + params.getFilters(), + params.getTop(), + params.getIndexPattern(), + 500, + Map.class, + batch -> { + if (enrichEchoes) { + for (Map d : batch) { + Object id = d.get("id"); + if (id != null) { + long countEchoes = elasticsearchService.count( + List.of(new FilterType("parentId", OperatorType.IS, id.toString())), + params.getIndexPattern()); + d.put("echoes", countEchoes); + } + } + } + UtilCsv.writeCsvBatch(printer, params.getColumns(), batch); + return true; + }); return ResponseEntity.ok().build(); } catch (Exception e) { String msg = ctx + ": " + e.getMessage(); - log.error(msg); + log.error(msg, e); applicationEventService.createEvent(msg, ApplicationEventType.ERROR); - return ResponseUtil.buildErrorResponse(HttpStatus.INTERNAL_SERVER_ERROR, msg); + return ResponseUtil.buildErrorResponse(HttpStatus.INTERNAL_SERVER_ERROR, + "An internal error occurred while exporting the CSV. Please check server logs."); } } diff --git a/backend/src/main/resources/config/application-prod.yml b/backend/src/main/resources/config/application-prod.yml index 33bacc55c..d8e2c8d1d 100644 --- a/backend/src/main/resources/config/application-prod.yml +++ b/backend/src/main/resources/config/application-prod.yml @@ -64,8 +64,6 @@ jhipster: base-url: application: - chart-builder: # Chart builder configuration - ip-info-index-name: .utm-geoip incident-response: asset-verification-interval: 300 diff --git a/backend/src/main/resources/config/liquibase/changelog/20260522001_update_socai_custom_headers_password.xml b/backend/src/main/resources/config/liquibase/changelog/20260522001_update_socai_custom_headers_password.xml new file mode 100644 index 000000000..fd402114f --- /dev/null +++ b/backend/src/main/resources/config/liquibase/changelog/20260522001_update_socai_custom_headers_password.xml @@ -0,0 +1,29 @@ + + + + + Set utmstack.socai.customHeaders data type to password + + + + + diff --git a/backend/src/main/resources/config/liquibase/master.xml b/backend/src/main/resources/config/liquibase/master.xml index eebe85f8c..a46172272 100644 --- a/backend/src/main/resources/config/liquibase/master.xml +++ b/backend/src/main/resources/config/liquibase/master.xml @@ -597,4 +597,6 @@ + + diff --git a/filters/macos/macos.yml b/filters/macos/macos.yml index 1bdc6e3e9..35eb41559 100644 --- a/filters/macos/macos.yml +++ b/filters/macos/macos.yml @@ -25,4 +25,8 @@ pipeline: - rename: from: - log.threadidentifier - to: log.threadIdentifier \ No newline at end of file + to: log.threadIdentifier + + # Drop unnecessary events + - drop: + where: equals("log.level", "notice") && contains("log.subsystem", "com.apple.cloudkit") || contains("log.subsystem", "com.apple.CoreDuet") || oneOf("log.subsystem", ["com.apple.apsd", "com.apple.bluetooth", "com.apple.SkyLight", "com.apple.mDNSResponder", "com.apple.homed", "com.apple.identityservicesd", "com.apple.powerlogd", "com.apple.analyticsd", "com.apple.UIKit", "com.apple.runningboard", "com.apple.WiFiManager", "com.apple.xpc", "com.apple.cache_delete", "com.apple.spotlightindex"]) \ No newline at end of file diff --git a/frontend/.gitignore b/frontend/.gitignore index 59326f390..95fae2d01 100644 --- a/frontend/.gitignore +++ b/frontend/.gitignore @@ -45,4 +45,4 @@ testem.log .DS_Store Thumbs.db -/src/environments/ +/src/environments diff --git a/frontend/Dockerfile b/frontend/Dockerfile index 770248743..5a4c8a975 100644 --- a/frontend/Dockerfile +++ b/frontend/Dockerfile @@ -1,4 +1,4 @@ -FROM nginx:1.19.5 +FROM nginx:1.30.1 WORKDIR /usr/share/nginx/html COPY ./nginx/mime.types /etc/nginx/mime.types COPY ./dist/utm-stack/ /usr/share/nginx/html/ diff --git a/frontend/src/app/app-module/guides/guide-soc-ai/guide-soc-ai.component.html b/frontend/src/app/app-module/guides/guide-soc-ai/guide-soc-ai.component.html index 7c8c18c08..da11a5256 100644 --- a/frontend/src/app/app-module/guides/guide-soc-ai/guide-soc-ai.component.html +++ b/frontend/src/app/app-module/guides/guide-soc-ai/guide-soc-ai.component.html @@ -155,9 +155,11 @@

SOC AI

Save configuration + diff --git a/frontend/src/app/app-module/guides/guide-soc-ai/guide-soc-ai.component.ts b/frontend/src/app/app-module/guides/guide-soc-ai/guide-soc-ai.component.ts index aba4d8fda..7bb7d9889 100644 --- a/frontend/src/app/app-module/guides/guide-soc-ai/guide-soc-ai.component.ts +++ b/frontend/src/app/app-module/guides/guide-soc-ai/guide-soc-ai.component.ts @@ -5,6 +5,7 @@ import {UtmModuleGroupConfService} from '../../shared/services/utm-module-group- import {UtmModuleGroupConfType} from '../../shared/type/utm-module-group-conf.type'; import {UtmToastService} from '../../../shared/alert/utm-toast.service'; import {ModuleChangeStatusBehavior} from '../../shared/behavior/module-change-status.behavior'; +import { finalize } from 'rxjs/operators'; interface ProviderConfig { id: string; @@ -38,6 +39,8 @@ export class GuideSocAiComponent implements OnInit { saving = false; loading = true; + configReady=false + // Form values - what the user sees/edits formValues: {[key: string]: string} = {}; customModelValue = ''; @@ -52,6 +55,10 @@ export class GuideSocAiComponent implements OnInit { private rawConfigs: UtmModuleGroupConfType[] = []; private groupId: number; + // Original masked customHeaders value (if backend returned an opaque mask like "*****") + private originalMaskedCustomHeaders: string | null = null; + private readonly maskedDisplay = '***'; + providers: ProviderConfig[] = [ { id: 'openai', @@ -297,7 +304,12 @@ export class GuideSocAiComponent implements OnInit { private loadConfig() { this.loading = true; - this.moduleGroupService.query({moduleId: this.integrationId}).subscribe(response => { + this.moduleGroupService.query({moduleId: this.integrationId}).pipe( + finalize(()=>{ + this.loading = false; + this.cdr.detectChanges(); + }) + ).subscribe(response => { const groups = response.body || []; if (groups.length > 0) { this.groupId = groups[0].id; @@ -312,6 +324,7 @@ export class GuideSocAiComponent implements OnInit { }; } this.loading = false; + this.configReady = true; this.cdr.detectChanges(); }, () => { this.loading = false; @@ -344,6 +357,7 @@ export class GuideSocAiComponent implements OnInit { 'changeAlertStatus': changeStatus ? changeStatus.confValue : 'false', }; this.customModelValue = ''; + this.originalMaskedCustomHeaders = null; // Only load provider-specific values if viewing the saved provider if (isCurrentSavedProvider) { @@ -373,16 +387,25 @@ export class GuideSocAiComponent implements OnInit { // Check if API key exists in custom headers — show masked if so const customHeaders = this.getConf('utmstack.socai.customHeaders'); if (customHeaders && customHeaders.confValue && customHeaders.confValue !== '{}') { - try { - const headers = JSON.parse(customHeaders.confValue); - const authConfig = this.providerAuthHeaders[this.activeProvider]; - if (authConfig && headers[authConfig.headerName]) { - // API key exists — show masked, don't expose the real value - this.formValues['apiKey'] = '*****'; - } - } catch (e) {} - this.formValues['customHeaders'] = customHeaders.confValue; - this.parseHeadersFromJson(customHeaders.confValue); + const raw = customHeaders.confValue; + if (this.isMaskedValue(raw)) { + // Backend returned the whole confValue masked — preserve original for save + this.originalMaskedCustomHeaders = raw; + this.formValues['customHeaders'] = raw; + this.formValues['apiKey'] = this.maskedDisplay; + this.headerRows = [{key: this.maskedDisplay, value: this.maskedDisplay}]; + } else { + try { + const headers = JSON.parse(raw); + const authConfig = this.providerAuthHeaders[this.activeProvider]; + if (authConfig && headers[authConfig.headerName]) { + // API key exists — show masked, don't expose the real value + this.formValues['apiKey'] = this.maskedDisplay; + } + } catch (e) {} + this.formValues['customHeaders'] = raw; + this.parseHeadersFromJson(raw); + } } const maxTokensConf = this.getConf('utmstack.socai.maxTokens'); @@ -419,57 +442,96 @@ export class GuideSocAiComponent implements OnInit { save() { this.saving = true; + + if (!this.groupId || !this.rawConfigs.length) { + this.moduleGroupService.create({ + name: 'socai', + description: 'socai', + moduleId: this.integrationId + }).subscribe( + response => { + this.groupId = response.body.id; + this.rawConfigs = response.body.moduleGroupConfigurations || []; + this.cdr.markForCheck() + this.persistConfig(); + }, + () => { + this.saving = false; + this.cdr.markForCheck(); + this.toast.showError('Error', 'Failed to create configuration group. Please try again.'); + } + ); + return; + } + this.persistConfig(); + } + + private persistConfig() { const changes: UtmModuleGroupConfType[] = []; // Set provider - this.pushChange(changes, 'utmstack.socai.provider', this.activeProvider); + this.pushChange(changes, 'utmstack.socai.provider', this.activeProvider, 'text'); // Set model - this.pushChange(changes, 'utmstack.socai.model', this.getModelValue()); + this.pushChange(changes, 'utmstack.socai.model', this.getModelValue(), 'text'); // Set URL for providers that need it (azure, ollama, custom) if (this.formValues['url']) { - this.pushChange(changes, 'utmstack.socai.url', this.formValues['url']); + this.pushChange(changes, 'utmstack.socai.url', this.formValues['url'], 'text'); } // Set maxTokens if (this.formValues['maxTokens']) { - this.pushChange(changes, 'utmstack.socai.maxTokens', this.formValues['maxTokens']); + this.pushChange(changes, 'utmstack.socai.maxTokens', this.formValues['maxTokens'], 'text'); } // Set behavior toggles - this.pushChange(changes, 'utmstack.socai.autoAnalyze', this.formValues['autoAnalyze'] || 'false'); - this.pushChange(changes, 'utmstack.socai.incidentCreation', this.formValues['incidentCreation'] || 'false'); - this.pushChange(changes, 'utmstack.socai.changeAlertStatus', this.formValues['changeAlertStatus'] || 'false'); + this.pushChange(changes, 'utmstack.socai.autoAnalyze', this.formValues['autoAnalyze'] || 'false', 'text'); + this.pushChange(changes, 'utmstack.socai.incidentCreation', this.formValues['incidentCreation'] || 'false', 'text'); + this.pushChange(changes, 'utmstack.socai.changeAlertStatus', this.formValues['changeAlertStatus'] || 'false', 'text'); // Build auth headers if (this.activeProvider === 'custom') { // Custom provider: user manages auth type and headers directly - this.pushChange(changes, 'utmstack.socai.authType', this.formValues['authType'] || 'custom-headers'); - this.pushChange(changes, 'utmstack.socai.customHeaders', this.formValues['customHeaders'] || '{}'); + this.pushChange(changes, 'utmstack.socai.authType', this.formValues['authType'] || 'custom-headers', 'text'); + if (this.originalMaskedCustomHeaders && this.hasMaskedHeaderRows()) { + // User didn't replace the masked rows — preserve original masked value + this.pushChange(changes, 'utmstack.socai.customHeaders', this.originalMaskedCustomHeaders, 'password'); + } else { + this.pushChange(changes, 'utmstack.socai.customHeaders', this.formValues['customHeaders'] || '{}', 'password'); + } } else if (this.activeProvider === 'ollama') { // Ollama: no auth needed - this.pushChange(changes, 'utmstack.socai.authType', 'none'); - this.pushChange(changes, 'utmstack.socai.customHeaders', '{}'); + this.pushChange(changes, 'utmstack.socai.authType', 'none', 'text'); + this.pushChange(changes, 'utmstack.socai.customHeaders', '{}', 'password'); } else { // Known providers: build auth header from API key const authConfig = this.providerAuthHeaders[this.activeProvider]; - if (authConfig && this.formValues['apiKey'] && this.formValues['apiKey'] !== '*****') { + const apiKey = this.formValues['apiKey']; + if (authConfig && apiKey && !this.isMaskedValue(apiKey)) { // User entered a new API key — build auth headers const headers: {[k: string]: string} = {}; - headers[authConfig.headerName] = authConfig.headerValuePrefix + this.formValues['apiKey']; - this.pushChange(changes, 'utmstack.socai.authType', 'custom-headers'); - this.pushChange(changes, 'utmstack.socai.customHeaders', JSON.stringify(headers)); + headers[authConfig.headerName] = authConfig.headerValuePrefix + apiKey; + this.pushChange(changes, 'utmstack.socai.authType', 'custom-headers', 'text'); + this.pushChange(changes, 'utmstack.socai.customHeaders', JSON.stringify(headers), 'password'); + } else if (this.originalMaskedCustomHeaders && apiKey && this.isMaskedValue(apiKey)) { + // User didn't change the masked API key — preserve original masked value + this.pushChange(changes, 'utmstack.socai.authType', 'custom-headers', 'text'); + this.pushChange(changes, 'utmstack.socai.customHeaders', this.originalMaskedCustomHeaders, 'password'); } - // If apiKey is '*****', don't touch customHeaders — keep existing value in DB + // Otherwise: don't touch customHeaders — keep existing value in DB } + this.rawConfigs=changes + this.cdr.markForCheck() + this.moduleGroupConfService.update({ keys: changes, moduleId: this.integrationId }).subscribe( () => { this.saving = false; + this.configReady = true; this.cdr.markForCheck() this.toast.showSuccessBottom('SOC AI configuration saved successfully'); }, @@ -486,7 +548,12 @@ export class GuideSocAiComponent implements OnInit { ); } - private pushChange(changes: UtmModuleGroupConfType[], confKey: string, value: string) { + private pushChange( + changes: UtmModuleGroupConfType[], + confKey: string, + value: string, + confDataType: 'list' | 'password' | 'file' | 'bool' | 'select' | 'text' = 'text' + ) { const existing = this.getConf(confKey); if (existing) { changes.push({ @@ -495,6 +562,19 @@ export class GuideSocAiComponent implements OnInit { confOptions: existing.confOptions ? JSON.stringify(existing.confOptions) : existing.confOptions, confVisibility: existing.confVisibility ? JSON.stringify(existing.confVisibility) : existing.confVisibility, }); + } else { + changes.push({ + id: undefined, + groupId: this.groupId, + confKey, + confValue: value, + confName: confKey.split('.')[2] || confKey, + confDataType, + confDescription: confKey, + confRequired: true, + confOptions: undefined, + confVisibility: undefined, + }); } } @@ -543,6 +623,14 @@ export class GuideSocAiComponent implements OnInit { this.formValues['customHeaders'] = JSON.stringify(obj); } + private isMaskedValue(value: string): boolean { + return !!value && /^\*+$/.test(value); + } + + private hasMaskedHeaderRows(): boolean { + return this.headerRows.some(r => this.isMaskedValue(r.key) || this.isMaskedValue(r.value)); + } + private parseHeadersFromJson(json: string) { this.headerRows = []; try { @@ -554,4 +642,10 @@ export class GuideSocAiComponent implements OnInit { // Invalid JSON, start empty } } + + + public disableModule(){ + this.groupId=null + this.rawConfigs=[] + } } diff --git a/frontend/src/app/app-module/shared/components/app-module-activate-button/app-module-activate-button.component.ts b/frontend/src/app/app-module/shared/components/app-module-activate-button/app-module-activate-button.component.ts index 7c4fe04f2..458b206f8 100644 --- a/frontend/src/app/app-module/shared/components/app-module-activate-button/app-module-activate-button.component.ts +++ b/frontend/src/app/app-module/shared/components/app-module-activate-button/app-module-activate-button.component.ts @@ -95,11 +95,10 @@ export class AppModuleActivateButtonComponent implements OnInit, OnDestroy { this.moduleRefreshBehavior.$moduleChange.next(true); this.toastService.showSuccessBottom('Module ' + this.moduleDetail.moduleName + ' has been ' + (this.moduleDetail.moduleActive ? 'enabled' : 'disabled') + ' successfully'); + if (!status) { + this.disableModuleClicked.emit(); + } }); - } else { - if (fromOnclick && !status) { - this.disableModuleClicked.emit(); - } } } diff --git a/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.html b/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.html index af0fb7ece..f8d608b22 100644 --- a/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.html +++ b/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.html @@ -38,6 +38,12 @@
 The system has detected new alerts, retrieving from the data engine... +
+ +  Searching... + +
diff --git a/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.scss b/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.scss index 27f615cc4..2daa2d2f2 100644 --- a/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.scss +++ b/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.scss @@ -32,3 +32,7 @@ tr.no-bottom-border { .bg-children-light { background-color: #cccccc52 !important; } + +.search-loading-indicator { + pointer-events: none; +} diff --git a/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.ts b/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.ts index 399e1916e..9df35d82a 100644 --- a/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.ts +++ b/frontend/src/app/data-management/alert-management/alert-view/alert-view.component.ts @@ -6,8 +6,8 @@ import {TranslateService} from '@ngx-translate/core'; import {ResizeEvent} from 'angular-resizable-element'; import {NgxSpinnerService} from 'ngx-spinner'; import {LocalStorageService} from 'ngx-webstorage'; -import {Observable, Subject} from 'rxjs'; -import {filter, takeUntil, tap} from 'rxjs/operators'; +import {Observable, Subject, throwError, timer, Subscription} from 'rxjs'; +import {concatMap, filter, retryWhen, takeUntil, tap, finalize} from 'rxjs/operators'; import {UtmToastService} from '../../../shared/alert/utm-toast.service'; import { ElasticFilterDefaultTime @@ -64,7 +64,8 @@ import {ElasticDataTypesEnum} from "../../../shared/enums/elastic-data-types.enu styleUrls: ['./alert-view.component.scss'] }) export class AlertViewComponent implements OnInit, OnDestroy { - + private lastRequest:Subscription|null = null + private lastTimeout:any = -1 constructor(private elasticDataService: ElasticDataService, private modalService: NgbModal, @@ -343,10 +344,24 @@ export class AlertViewComponent implements OnInit, OnDestroy { this.getAlert('on time filter change'); } + get searching(): boolean { + return this.lastRequest!=null; + } + getAlert(calledFrom?: string, filtersParam?: ElasticFilterType[]) { - this.elasticDataService.search(this.page, this.itemsPerPage, + if(this.lastTimeout!=-1){ + clearTimeout(this.lastTimeout) + if(this.lastRequest){ + this.lastRequest.unsubscribe() + this.lastRequest=null + } + } + this.lastTimeout= setTimeout(()=>{ + this.lastRequest=this.elasticDataService.search(this.page, this.itemsPerPage, MAX_SEARCH_RESULTS, this.dataNature, - sanitizeFilters(this.filters), this.sortBy, true).subscribe( + sanitizeFilters(this.filters), this.sortBy, true) + .pipe(finalize(()=>this.lastRequest=null)) + .subscribe( (res: HttpResponse) => { this.totalItems = Number(res.headers.get('X-Total-Count')); this.alerts = res.body; @@ -355,8 +370,11 @@ export class AlertViewComponent implements OnInit, OnDestroy { }, (res: HttpResponse) => { this.utmToastService.showError('Error', 'An error occurred while listing the alerts. Please try again later.'); + this.loading = false; + this.refreshingAlert = false; } ); + },100) } saveReport() { diff --git a/frontend/src/app/data-management/alert-management/shared/components/alert-rule-create/alert-rule-create.component.ts b/frontend/src/app/data-management/alert-management/shared/components/alert-rule-create/alert-rule-create.component.ts index 19c596d7c..575884ae6 100644 --- a/frontend/src/app/data-management/alert-management/shared/components/alert-rule-create/alert-rule-create.component.ts +++ b/frontend/src/app/data-management/alert-management/shared/components/alert-rule-create/alert-rule-create.component.ts @@ -32,7 +32,7 @@ import { ALERT_STATUS_LABEL_FIELD, ALERT_TAGS_FIELD, ALERT_TIMESTAMP_FIELD, - EVENT_IS_ALERT, FALSE_POSITIVE_OBJECT, LOG_RELATED_ID_EVENT_FIELD + EVENT_IS_ALERT, EVENT_TAG_RULE_FIELDS, FALSE_POSITIVE_OBJECT, LOG_RELATED_ID_EVENT_FIELD } from '../../../../../shared/constants/alert/alert-field.constant'; import {AUTOMATIC_REVIEW, CLOSED} from '../../../../../shared/constants/alert/alert-status.constant'; import {FILTER_OPERATORS} from '../../../../../shared/constants/filter-operators.const'; @@ -84,7 +84,6 @@ export class AlertRuleCreateComponent implements OnInit, OnDestroy { ALERT_OBSERVATION_FIELD, ALERT_NOTE_FIELD, ALERT_REFERENCE_FIELD, - LOG_RELATED_ID_EVENT_FIELD, EVENT_IS_ALERT, ALERT_INCIDENT_USER_FIELD, ALERT_INCIDENT_DATE_FIELD, @@ -148,6 +147,8 @@ export class AlertRuleCreateComponent implements OnInit, OnDestroy { return acc.concat(field); }, []); + this.fields = [...this.fields, ...EVENT_TAG_RULE_FIELDS]; + this.operators = FILTER_OPERATORS.filter(value => !this.excludeOperators.includes(value.operator)); } @@ -232,6 +233,11 @@ export class AlertRuleCreateComponent implements OnInit, OnDestroy { } getFieldValue(field: string): any { + if(field.startsWith('events') && this.alert.events &&this.alert.events.length>0){ + let fields = field.split('.') + fields.splice(0,1) + return getValueFromPropertyPath(this.alert.events[0],fields.join('.'), null); + } return getValueFromPropertyPath(this.alert, field, null); } diff --git a/frontend/src/app/data-management/alert-management/shared/components/filters/alert-generic-filter/alert-generic-filter.component.ts b/frontend/src/app/data-management/alert-management/shared/components/filters/alert-generic-filter/alert-generic-filter.component.ts index 2d8d027e3..19dfeb5ec 100644 --- a/frontend/src/app/data-management/alert-management/shared/components/filters/alert-generic-filter/alert-generic-filter.component.ts +++ b/frontend/src/app/data-management/alert-management/shared/components/filters/alert-generic-filter/alert-generic-filter.component.ts @@ -1,6 +1,6 @@ import {Component, EventEmitter, Input, OnDestroy, OnInit, Output} from '@angular/core'; import {Subject} from 'rxjs'; -import {takeUntil} from 'rxjs/operators'; +import {debounceTime, distinctUntilChanged, switchMap, takeUntil} from 'rxjs/operators'; import {ALERT_TAGS_FIELD} from '../../../../../../shared/constants/alert/alert-field.constant'; import {ALERT_INDEX_PATTERN} from '../../../../../../shared/constants/main-index-pattern.constant'; import {ElasticDataTypesEnum} from '../../../../../../shared/enums/elastic-data-types.enum'; @@ -30,6 +30,7 @@ export class AlertGenericFilterComponent implements OnInit, OnDestroy { filter: ElasticFilterType; sort: { orderByCount: boolean, sortAsc: boolean } = {orderByCount: true, sortAsc: false}; destroy$: Subject = new Subject(); + private fetchRequest$ = new Subject(); constructor(private elasticSearchIndexService: ElasticSearchIndexService, private alertFiltersBehavior: AlertFiltersBehavior, @@ -37,10 +38,39 @@ export class AlertGenericFilterComponent implements OnInit, OnDestroy { } ngOnInit() { - // this.getFieldValues(); - /** - * If filter is tags subscribe to changes to reload data on add new tag on alert - */ + this.fetchRequest$ + .pipe( + debounceTime(300), + switchMap(() => { + const field = this.setFieldKeyword(); + const filters = this.activeFilters + .filter(value => !value.field.includes(field)); + if (this.search !== undefined && this.search !== '') { + filters.push({ + field: this.fieldFilter.field, + operator: ElasticOperatorsEnum.CONTAIN, + value: this.search + }); + } + const req = { + field, + filters, + index: ALERT_INDEX_PATTERN, + orderByCount: this.sort.orderByCount, + sortAsc: this.sort.sortAsc, + top: this.top + }; + return this.elasticSearchIndexService.getValuesWithCount(req); + }), + takeUntil(this.destroy$) + ) + .subscribe(response => { + this.fieldValues = response.body; + this.loading = false; + this.searching = false; + this.loadingMore = false; + }); + if (this.fieldFilter.field === ALERT_TAGS_FIELD) { this.alertUpdateTagBehavior.$tagRefresh .pipe(takeUntil(this.destroy$)) @@ -50,9 +80,6 @@ export class AlertGenericFilterComponent implements OnInit, OnDestroy { } }); } - /** - * Reset all values of selected filter - */ this.alertFiltersBehavior.$resetFilter .pipe(takeUntil(this.destroy$)) .subscribe(reset => { @@ -74,7 +101,11 @@ export class AlertGenericFilterComponent implements OnInit, OnDestroy { } }); this.alertFiltersBehavior.$filters - .pipe(takeUntil(this.destroy$)) + .pipe( + debounceTime(150), + distinctUntilChanged((a, b) => JSON.stringify(a) === JSON.stringify(b)), + takeUntil(this.destroy$) + ) .subscribe((filters: ElasticFilterType[]) => { if (filters) { this.activeFilters = filters; @@ -101,30 +132,7 @@ export class AlertGenericFilterComponent implements OnInit, OnDestroy { } getFieldValues() { - const field = this.setFieldKeyword(); - const filters = this.activeFilters - .filter(value => !value.field.includes(field)); - if (this.search !== undefined && this.search !== '') { - filters.push({ - field: this.fieldFilter.field, - operator: ElasticOperatorsEnum.CONTAIN, - value: this.search - }); - } - const req = { - field, - filters, - index: ALERT_INDEX_PATTERN, - orderByCount: this.sort.orderByCount, - sortAsc: this.sort.sortAsc, - top: this.top - }; - this.elasticSearchIndexService.getValuesWithCount(req).subscribe(response => { - this.fieldValues = response.body; - this.loading = false; - this.searching = false; - this.loadingMore = false; - }); + this.fetchRequest$.next(); } setFieldKeyword(): string { diff --git a/frontend/src/app/incident-response/playbook-builder/playbook-builder.component.ts b/frontend/src/app/incident-response/playbook-builder/playbook-builder.component.ts index 497efbb60..5b43eee28 100644 --- a/frontend/src/app/incident-response/playbook-builder/playbook-builder.component.ts +++ b/frontend/src/app/incident-response/playbook-builder/playbook-builder.component.ts @@ -196,7 +196,7 @@ export class PlaybookBuilderComponent implements OnInit, OnDestroy { } createRule() { - if (this.rule.id) { + if (this.rule && this.rule.id) { this.editRule(); } else { this.saveRule(); diff --git a/frontend/src/app/rule-management/app-rule/components/add-after-event/add-after-event.component.html b/frontend/src/app/rule-management/app-rule/components/add-after-event/add-after-event.component.html index 668b76529..2c5e7eded 100644 --- a/frontend/src/app/rule-management/app-rule/components/add-after-event/add-after-event.component.html +++ b/frontend/src/app/rule-management/app-rule/components/add-after-event/add-after-event.component.html @@ -28,7 +28,7 @@ Must be at least 1
- Must not be greater than 50 + Must not be greater than 100
diff --git a/frontend/src/app/rule-management/services/after-event-form.service.ts b/frontend/src/app/rule-management/services/after-event-form.service.ts index d68edf6ec..bbc591e13 100644 --- a/frontend/src/app/rule-management/services/after-event-form.service.ts +++ b/frontend/src/app/rule-management/services/after-event-form.service.ts @@ -36,7 +36,7 @@ export class AfterEventFormService { : [] ), within: [event.within || ''], - count: [event.count ? event.count : null, [Validators.required, Validators.min(1), Validators.max(50)]], + count: [event.count ? event.count : null, [Validators.required, Validators.min(1), Validators.max(100)]], }); } diff --git a/frontend/src/app/shared/constants/alert/alert-field.constant.ts b/frontend/src/app/shared/constants/alert/alert-field.constant.ts index 21a3349e1..dd57f4cfd 100644 --- a/frontend/src/app/shared/constants/alert/alert-field.constant.ts +++ b/frontend/src/app/shared/constants/alert/alert-field.constant.ts @@ -120,6 +120,47 @@ export const EVENT_IS_ALERT = 'isAlert'; export const FALSE_POSITIVE_OBJECT = {id: 1, tagName: 'False positive', tagColor: '#f44336', systemOwner: true}; +// Event-related fields exposed in tag-rule conditions. +// These are flattened paths into the `events` array on the alert document +// ("events" is mapped as an object array, so any condition matches when ANY +// event satisfies it). +export const EVENT_TAG_RULE_FIELDS: UtmFieldType[] = [ + {label: 'Event Data Type', field: 'events.dataType', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Data Source', field: 'events.dataSource', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Action', field: 'events.action', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Action Result', field: 'events.actionResult', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Severity', field: 'events.severity', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Protocol', field: 'events.protocol', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Connection Status', field: 'events.connectionStatus', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Status Code', field: 'events.statusCode', type: ElasticDataTypesEnum.NUMBER, visible: true}, + {label: 'Event Tenant Name', field: 'events.tenantName', type: ElasticDataTypesEnum.STRING, visible: true}, + // Origin + {label: 'Event Origin IP', field: 'events.origin.ip', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Origin Host', field: 'events.origin.host', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Origin User', field: 'events.origin.user', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Origin Port', field: 'events.origin.port', type: ElasticDataTypesEnum.NUMBER, visible: true}, + {label: 'Event Origin Domain', field: 'events.origin.domain', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Origin URL', field: 'events.origin.url', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Origin Country', field: 'events.origin.geolocation.country', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Origin Country Code', field: 'events.origin.geolocation.countryCode', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Origin City', field: 'events.origin.geolocation.city', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Origin ASN', field: 'events.origin.geolocation.asn', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Origin ASO', field: 'events.origin.geolocation.aso', type: ElasticDataTypesEnum.STRING, visible: true}, + // Target + {label: 'Event Target IP', field: 'events.target.ip', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target Host', field: 'events.target.host', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target User', field: 'events.target.user', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target Port', field: 'events.target.port', type: ElasticDataTypesEnum.NUMBER, visible: true}, + {label: 'Event Target URL', field: 'events.target.url', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target Domain', field: 'events.target.domain', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target File', field: 'events.target.file', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target Country', field: 'events.target.geolocation.country', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target Country Code', field: 'events.target.geolocation.countryCode', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target City', field: 'events.target.geolocation.city', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target ASN', field: 'events.target.geolocation.asn', type: ElasticDataTypesEnum.STRING, visible: true}, + {label: 'Event Target ASO', field: 'events.target.geolocation.aso', type: ElasticDataTypesEnum.STRING, visible: true}, +]; + export const ALERT_FIELDS: UtmFieldType[] = [ { label: 'Alert name', diff --git a/frontend/src/app/shared/services/elasticsearch/elasticsearch-index.service.ts b/frontend/src/app/shared/services/elasticsearch/elasticsearch-index.service.ts index 0f7f38755..ea1ccbf67 100644 --- a/frontend/src/app/shared/services/elasticsearch/elasticsearch-index.service.ts +++ b/frontend/src/app/shared/services/elasticsearch/elasticsearch-index.service.ts @@ -1,16 +1,18 @@ import {HttpClient, HttpResponse} from '@angular/common/http'; import {Injectable} from '@angular/core'; -import {Observable} from 'rxjs'; +import {Observable, Subscription, Subject} from 'rxjs'; import {SERVER_API_URL} from '../../../app.constants'; import {ElasticSearchFieldInfoType} from '../../types/elasticsearch/elastic-search-field-info.type'; import {ElasticsearchIndexType} from '../../types/elasticsearch/elasticsearch-index.type'; import {createRequestOption} from '../../util/request-util'; +import { finalize, shareReplay } from 'rxjs/operators'; @Injectable({ providedIn: 'root' }) export class ElasticSearchIndexService { public resourceUrl = SERVER_API_URL + 'api/elasticsearch/'; + private inFlightRequests = new Map>(); constructor(private http: HttpClient) { } @@ -40,7 +42,16 @@ export class ElasticSearchIndexService { } getValuesWithCount(rq) { - return this.http.post(this.resourceUrl + 'property/values-with-count', rq, {observe: 'response'}); + const key = JSON.stringify(rq); + let req = this.inFlightRequests.get(key); + if (!req) { + req = this.http.post(this.resourceUrl + 'property/values-with-count', rq, {observe: 'response'}).pipe( + finalize(() => this.inFlightRequests.delete(key)), + shareReplay(1) + ); + this.inFlightRequests.set(key, req); + } + return req; } deleteIndexes(indexes: string[]) { diff --git a/frontend/src/environments/environment.ts b/frontend/src/environments/environment.ts index 6d83346a6..e69de29bb 100644 --- a/frontend/src/environments/environment.ts +++ b/frontend/src/environments/environment.ts @@ -1,24 +0,0 @@ -// This file can be replaced during build by using the `fileReplacements` array. -// `ng build --prod` replaces `environment.ts` with `environment.prod.ts`. -// The list of file replacements can be found in `angular.json`. - -export const environment = { - production: false, - // SERVER_API_URL: 'https://10.11.11.18/', - SERVER_API_URL: 'http://localhost:8080/', - SERVER_API_CONTEXT: '', - SESSION_AUTH_TOKEN: window.location.host.split(':')[0].toLocaleUpperCase(), - WEBSOCKET_URL: '//localhost:8080', - BUILD_TIMESTAMP: new Date().getTime(), - DEBUG_INFO_ENABLED: true, - VERSION: '0.0.1' -}; - -/* - * For easier debugging in development mode, you can import the following file - * to ignore zone related error stack frames such as `zone.run`, `zoneDelegate.invokeTask`. - * - * This import should be commented out in production mode because it will have a negative impact - * on performance if an error is thrown. - */ -// import 'zone.js/dist/zone-error'; // Included with Angular CLI. diff --git a/installer/go.mod b/installer/go.mod index 06065f282..793aa0c58 100644 --- a/installer/go.mod +++ b/installer/go.mod @@ -3,7 +3,7 @@ module github.com/utmstack/UTMStack/installer go 1.25.1 require ( - github.com/cloudfoundry/gosigar v1.3.118 + github.com/cloudfoundry/gosigar v1.3.120 github.com/docker/docker v28.5.2+incompatible github.com/kardianos/service v1.2.4 github.com/shirou/gopsutil/v3 v3.24.5 @@ -66,21 +66,19 @@ require ( github.com/yusufpapurcu/wmi v1.2.4 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 // indirect - go.opentelemetry.io/otel v1.39.0 // indirect - go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.39.0 // indirect - go.opentelemetry.io/otel/metric v1.39.0 // indirect - go.opentelemetry.io/otel/trace v1.39.0 // indirect + go.opentelemetry.io/otel v1.41.0 // indirect + go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.41.0 // indirect + go.opentelemetry.io/otel/metric v1.41.0 // indirect + go.opentelemetry.io/otel/sdk v1.41.0 // indirect + go.opentelemetry.io/otel/sdk/metric v1.41.0 // indirect + go.opentelemetry.io/otel/trace v1.41.0 // indirect go.uber.org/mock v0.6.0 // indirect golang.org/x/arch v0.23.0 // indirect - golang.org/x/crypto v0.47.0 // indirect - golang.org/x/net v0.49.0 // indirect - golang.org/x/sys v0.43.0 // indirect - golang.org/x/text v0.33.0 // indirect + golang.org/x/crypto v0.48.0 // indirect + golang.org/x/net v0.50.0 // indirect + golang.org/x/sys v0.45.0 // indirect + golang.org/x/text v0.34.0 // indirect golang.org/x/time v0.14.0 // indirect - golang.org/x/tools v0.41.0 // indirect - google.golang.org/genproto/googleapis/api v0.0.0-20260120221211-b8f7ae30c516 // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20260120221211-b8f7ae30c516 // indirect - google.golang.org/grpc v1.78.0 // indirect google.golang.org/protobuf v1.36.11 // indirect gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect gotest.tools/v3 v3.5.2 // indirect diff --git a/installer/go.sum b/installer/go.sum index cc9be246d..7aa7b509f 100644 --- a/installer/go.sum +++ b/installer/go.sum @@ -16,8 +16,8 @@ github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1x github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cloudfoundry/gosigar v1.3.118 h1:M5I101sHroieCCM07+HDVhYq6AUThH9LTJF4WY1d3O0= -github.com/cloudfoundry/gosigar v1.3.118/go.mod h1:HaIEZU356TAVnxmEvDFe5PXOTCOeSmUgVxoPzrat5Po= +github.com/cloudfoundry/gosigar v1.3.120 h1:YkYIRO/Abp1CRqvfzOgP8kiuWT1wJi1/ZBfwvNjstTQ= +github.com/cloudfoundry/gosigar v1.3.120/go.mod h1:h0cx9zTwKaigG61uVYSCcXCvcHIJsrmgd16feJ1VZX4= github.com/cloudwego/base64x v0.1.6 h1:t11wG9AECkCDk5fMSoxmufanudBtJ+/HemLstXDLI2M= github.com/cloudwego/base64x v0.1.6/go.mod h1:OFcloc187FXDaYHvrNIjxSe8ncn0OOM8gEHfghB2IPU= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= @@ -74,8 +74,8 @@ github.com/google/pprof v0.0.0-20250630185457-6e76a2b096b5 h1:xhMrHhTJ6zxu3gA4en github.com/google/pprof v0.0.0-20250630185457-6e76a2b096b5/go.mod h1:5hDyRhoBCxViHszMt12TnOpEI4VVi+U8Gm9iphldiMA= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.3 h1:NmZ1PKzSTQbuGHw9DGPFomqkkLWMC+vZCkfs+FHv1Vg= -github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.3/go.mod h1:zQrxl1YP88HQlA6i9c63DSVPFklWpGX4OWAc9bFuaH4= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 h1:HWRh5R2+9EifMyIHV7ZV+MIZqgz+PMpZ14Jynv3O2Zs= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0/go.mod h1:JfhWUomR1baixubs02l85lZYYOm7LV6om4ceouMv45c= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/kardianos/service v1.2.4 h1:XNlGtZOYNx2u91urOdg/Kfmc+gfmuIo1Dd3rEi2OgBk= @@ -166,20 +166,20 @@ go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0 h1:ssfIgGNANqpVFCndZvcuyKbl0g+UAVcbBcqGkG28H0Y= go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.64.0/go.mod h1:GQ/474YrbE4Jx8gZ4q5I4hrhUzM6UPzyrqJYV2AqPoQ= -go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48= -go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8= -go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.39.0 h1:f0cb2XPmrqn4XMy9PNliTgRKJgS5WcL/u0/WRYGz4t0= -go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.39.0/go.mod h1:vnakAaFckOMiMtOIhFI2MNH4FYrZzXCYxmb1LlhoGz8= -go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.39.0 h1:Ckwye2FpXkYgiHX7fyVrN1uA/UYd9ounqqTuSNAv0k4= -go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.39.0/go.mod h1:teIFJh5pW2y+AN7riv6IBPX2DuesS3HgP39mwOspKwU= -go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0= -go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs= -go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18= -go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE= -go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8= -go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew= -go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI= -go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA= +go.opentelemetry.io/otel v1.41.0 h1:YlEwVsGAlCvczDILpUXpIpPSL/VPugt7zHThEMLce1c= +go.opentelemetry.io/otel v1.41.0/go.mod h1:Yt4UwgEKeT05QbLwbyHXEwhnjxNO6D8L5PQP51/46dE= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.41.0 h1:ao6Oe+wSebTlQ1OEht7jlYTzQKE+pnx/iNywFvTbuuI= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.41.0/go.mod h1:u3T6vz0gh/NVzgDgiwkgLxpsSF6PaPmo2il0apGJbls= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.41.0 h1:inYW9ZhgqiDqh6BioM7DVHHzEGVq76Db5897WLGZ5Go= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.41.0/go.mod h1:Izur+Wt8gClgMJqO/cZ8wdeeMryJ/xxiOVgFSSfpDTY= +go.opentelemetry.io/otel/metric v1.41.0 h1:rFnDcs4gRzBcsO9tS8LCpgR0dxg4aaxWlJxCno7JlTQ= +go.opentelemetry.io/otel/metric v1.41.0/go.mod h1:xPvCwd9pU0VN8tPZYzDZV/BMj9CM9vs00GuBjeKhJps= +go.opentelemetry.io/otel/sdk v1.41.0 h1:YPIEXKmiAwkGl3Gu1huk1aYWwtpRLeskpV+wPisxBp8= +go.opentelemetry.io/otel/sdk v1.41.0/go.mod h1:ahFdU0G5y8IxglBf0QBJXgSe7agzjE4GiTJ6HT9ud90= +go.opentelemetry.io/otel/sdk/metric v1.41.0 h1:siZQIYBAUd1rlIWQT2uCxWJxcCO7q3TriaMlf08rXw8= +go.opentelemetry.io/otel/sdk/metric v1.41.0/go.mod h1:HNBuSvT7ROaGtGI50ArdRLUnvRTRGniSUZbxiWxSO8Y= +go.opentelemetry.io/otel/trace v1.41.0 h1:Vbk2co6bhj8L59ZJ6/xFTskY+tGAbOnCtQGVVa9TIN0= +go.opentelemetry.io/otel/trace v1.41.0/go.mod h1:U1NU4ULCoxeDKc09yCWdWe+3QoyweJcISEVa1RBzOis= go.opentelemetry.io/proto/otlp v1.9.0 h1:l706jCMITVouPOqEnii2fIAuO3IVGBRPV5ICjceRb/A= go.opentelemetry.io/proto/otlp v1.9.0/go.mod h1:xE+Cx5E/eEHw+ISFkwPLwCZefwVjY+pqKg1qcK03+/4= go.uber.org/mock v0.6.0 h1:hyF9dfmbgIX5EfOdasqLsWD6xqpNZlXblLB/Dbnwv3Y= @@ -188,32 +188,32 @@ go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc= go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg= golang.org/x/arch v0.23.0 h1:lKF64A2jF6Zd8L0knGltUnegD62JMFBiCPBmQpToHhg= golang.org/x/arch v0.23.0/go.mod h1:dNHoOeKiyja7GTvF9NJS1l3Z2yntpQNzgrjh1cU103A= -golang.org/x/crypto v0.47.0 h1:V6e3FRj+n4dbpw86FJ8Fv7XVOql7TEwpHapKoMJ/GO8= -golang.org/x/crypto v0.47.0/go.mod h1:ff3Y9VzzKbwSSEzWqJsJVBnWmRwRSHt/6Op5n9bQc4A= +golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts= +golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos= golang.org/x/mod v0.32.0 h1:9F4d3PHLljb6x//jOyokMv3eX+YDeepZSEo3mFJy93c= golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU= -golang.org/x/net v0.49.0 h1:eeHFmOGUTtaaPSGNmjBKpbng9MulQsJURQUAfUwY++o= -golang.org/x/net v0.49.0/go.mod h1:/ysNB2EvaqvesRkuLAyjI1ycPZlQHM3q01F02UY/MV8= +golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60= +golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM= golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4= golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI= -golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= -golang.org/x/text v0.33.0 h1:B3njUFyqtHDUI5jMn1YIr5B0IE2U0qck04r6d4KPAxE= -golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8= +golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= +golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk= +golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA= golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI= golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4= golang.org/x/tools v0.41.0 h1:a9b8iMweWG+S0OBnlU36rzLp20z1Rp10w+IY2czHTQc= golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg= -google.golang.org/genproto/googleapis/api v0.0.0-20260120221211-b8f7ae30c516 h1:vmC/ws+pLzWjj/gzApyoZuSVrDtF1aod4u/+bbj8hgM= -google.golang.org/genproto/googleapis/api v0.0.0-20260120221211-b8f7ae30c516/go.mod h1:p3MLuOwURrGBRoEyFHBT3GjUwaCQVKeNqqWxlcISGdw= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260120221211-b8f7ae30c516 h1:sNrWoksmOyF5bvJUcnmbeAmQi8baNhqg5IWaI3llQqU= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260120221211-b8f7ae30c516/go.mod h1:j9x/tPzZkyxcgEFkiKEEGxfvyumM01BEtsW8xzOahRQ= -google.golang.org/grpc v1.78.0 h1:K1XZG/yGDJnzMdd/uZHAkVqJE+xIDOcmdSFZkBUicNc= -google.golang.org/grpc v1.78.0/go.mod h1:I47qjTo4OKbMkjA/aOOwxDIiPSBofUtQUI5EfpWvW7U= +google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57 h1:JLQynH/LBHfCTSbDWl+py8C+Rg/k1OVH3xfcaiANuF0= +google.golang.org/genproto/googleapis/api v0.0.0-20260209200024-4cfbd4190f57/go.mod h1:kSJwQxqmFXeo79zOmbrALdflXQeAYcUbgS7PbpMknCY= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57 h1:mWPCjDEyshlQYzBpMNHaEof6UX1PmHcaUODUywQ0uac= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260209200024-4cfbd4190f57/go.mod h1:j9x/tPzZkyxcgEFkiKEEGxfvyumM01BEtsW8xzOahRQ= +google.golang.org/grpc v1.79.1 h1:zGhSi45ODB9/p3VAawt9a+O/MULLl9dpizzNNpq7flY= +google.golang.org/grpc v1.79.1/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/plugins/aws/go.mod b/plugins/aws/go.mod index 35854e71d..a4b0ee8d7 100644 --- a/plugins/aws/go.mod +++ b/plugins/aws/go.mod @@ -4,8 +4,8 @@ go 1.25.5 require ( github.com/aws/aws-sdk-go-v2 v1.41.7 - github.com/aws/aws-sdk-go-v2/config v1.32.17 - github.com/aws/aws-sdk-go-v2/credentials v1.19.16 + github.com/aws/aws-sdk-go-v2/config v1.32.18 + github.com/aws/aws-sdk-go-v2/credentials v1.19.17 github.com/google/uuid v1.6.0 github.com/threatwinds/go-sdk v1.1.21 ) @@ -29,11 +29,11 @@ require ( github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 // indirect github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 // indirect github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 // indirect - github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.73.0 + github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.74.0 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 // indirect github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 // indirect github.com/aws/aws-sdk-go-v2/service/sso v1.30.17 // indirect - github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21 // indirect + github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.0 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 // indirect github.com/aws/smithy-go v1.25.1 // indirect github.com/bytedance/sonic v1.15.0 // indirect diff --git a/plugins/aws/go.sum b/plugins/aws/go.sum index 7790ca40f..f19d299a1 100644 --- a/plugins/aws/go.sum +++ b/plugins/aws/go.sum @@ -6,10 +6,10 @@ github.com/aws/aws-sdk-go-v2 v1.41.7 h1:DWpAJt66FmnnaRIOT/8ASTucrvuDPZASqhhLey6t github.com/aws/aws-sdk-go-v2 v1.41.7/go.mod h1:4LAfZOPHNVNQEckOACQx60Y8pSRjIkNZQz1w92xpMJc= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 h1:gx1AwW1Iyk9Z9dD9F4akX5gnN3QZwUB20GGKH/I+Rho= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10/go.mod h1:qqY157uZoqm5OXq/amuaBJyC9hgBCBQnsaWnPe905GY= -github.com/aws/aws-sdk-go-v2/config v1.32.17 h1:FpL4/758/diKwqbytU0prpuiu60fgXKUWCpDJtApclU= -github.com/aws/aws-sdk-go-v2/config v1.32.17/go.mod h1:OXqUMzgXytfoF9JaKkhrOYsyh72t9G+MJH8mMRaexOE= -github.com/aws/aws-sdk-go-v2/credentials v1.19.16 h1:r3RJBuU7X9ibt8RHbMjWE6y60QbKBiII6wSrXnapxSU= -github.com/aws/aws-sdk-go-v2/credentials v1.19.16/go.mod h1:6cx7zqDENJDbBIIWX6P8s0h6hqHC8Avbjh9Dseo27ug= +github.com/aws/aws-sdk-go-v2/config v1.32.18 h1:Hcia46bxhGgF3BaSnG8nSNCWmqTK6bj9xN9/FJ3WK6Q= +github.com/aws/aws-sdk-go-v2/config v1.32.18/go.mod h1:zEjCAYmxqDadH1WX8CdBvmLKhUEUVFgKRQG38zjDmrY= +github.com/aws/aws-sdk-go-v2/credentials v1.19.17 h1:gP2nkGsS+KMvF/jfFz2Vv2qiiOqWKyPACSzPsqHgoW8= +github.com/aws/aws-sdk-go-v2/credentials v1.19.17/go.mod h1:Bsew3S/moG5iT77giPj1q8wb/s0RE5/QfH+ASjYtuQc= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 h1:UuSfcORqNSz/ey3VPRS8TcVH2Ikf0/sC+Hdj400QI6U= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23/go.mod h1:+G/OSGiOFnSOkYloKj/9M35s74LgVAdJBSD5lsFfqKg= github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 h1:GpT/TrnBYuE5gan2cZbTtvP+JlHsutdmlV2YfEyNde0= @@ -18,8 +18,8 @@ github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 h1:bpd8vxhlQi2r1hiueO github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23/go.mod h1:15DfR2nw+CRHIk0tqNyifu3G1YdAOy68RftkhMDDwYk= github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 h1:OQqn11BtaYv1WLUowvcA30MpzIu8Ti4pcLPIIyoKZrA= github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24/go.mod h1:X5ZJyfwVrWA96GzPmUCWFQaEARPR7gCrpq2E92PJwAE= -github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.73.0 h1:JmrHkELR2Q0O28swrFMm0hZNwpQrV8qmbhnb7suKIfc= -github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.73.0/go.mod h1:MLJu3PUd8fp5Qvj4CiLvyY5H8y7kxHKlTp060Wsd+Vc= +github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.74.0 h1:6TqDeYdvJJEIJGg5ICy7nzC7/UuHk2Eg3wrpb5bWKPM= +github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.74.0/go.mod h1:MLJu3PUd8fp5Qvj4CiLvyY5H8y7kxHKlTp060Wsd+Vc= github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 h1:FLudkZLt5ci0ozzgkVo8BJGwvqNaZbTWb3UcucAateA= github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9/go.mod h1:w7wZ/s9qK7c8g4al+UyoF1Sp/Z45UwMGcqIzLWVQHWk= github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 h1:pbrxO/kuIwgEsOPLkaHu0O+m4fNgLU8B3vxQ+72jTPw= @@ -28,8 +28,8 @@ github.com/aws/aws-sdk-go-v2/service/signin v1.0.11 h1:TdJ+HdzOBhU8+iVAOGUTU63VX github.com/aws/aws-sdk-go-v2/service/signin v1.0.11/go.mod h1:R82ZRExE/nheo0N+T8zHPcLRTcH8MGsnR3BiVGX0TwI= github.com/aws/aws-sdk-go-v2/service/sso v1.30.17 h1:7byT8HUWrgoRp6sXjxtZwgOKfhss5fW6SkLBtqzgRoE= github.com/aws/aws-sdk-go-v2/service/sso v1.30.17/go.mod h1:xNWknVi4Ezm1vg1QsB/5EWpAJURq22uqd38U8qKvOJc= -github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21 h1:+1Kl1zx6bWi4X7cKi3VYh29h8BvsCoHQEQ6ST9X8w7w= -github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21/go.mod h1:4vIRDq+CJB2xFAXZ+YgGUTiEft7oAQlhIs71xcSeuVg= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.0 h1:nDARhv/oF55bcxF7rCI/4PDxOKnVXVWwDuDwCs2I2SQ= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.0/go.mod h1:4vIRDq+CJB2xFAXZ+YgGUTiEft7oAQlhIs71xcSeuVg= github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 h1:F/M5Y9I3nwr2IEpshZgh1GeHpOItExNM9L1euNuh/fk= github.com/aws/aws-sdk-go-v2/service/sts v1.42.1/go.mod h1:mTNxImtovCOEEuD65mKW7DCsL+2gjEH+RPEAexAzAio= github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI= diff --git a/plugins/compliance-orchestrator/go.mod b/plugins/compliance-orchestrator/go.mod index 3cdc68db7..978ffa250 100644 --- a/plugins/compliance-orchestrator/go.mod +++ b/plugins/compliance-orchestrator/go.mod @@ -47,7 +47,7 @@ require ( golang.org/x/text v0.33.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260114163908-3f89685c29c3 // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260114163908-3f89685c29c3 // indirect - google.golang.org/grpc v1.78.0 // indirect + google.golang.org/grpc v1.79.3 // indirect google.golang.org/protobuf v1.36.11 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect sigs.k8s.io/yaml v1.6.0 // indirect diff --git a/plugins/compliance-orchestrator/go.sum b/plugins/compliance-orchestrator/go.sum index a247b315a..dc22fd644 100644 --- a/plugins/compliance-orchestrator/go.sum +++ b/plugins/compliance-orchestrator/go.sum @@ -8,6 +8,8 @@ github.com/bytedance/sonic v1.15.0 h1:/PXeWFaR5ElNcVE84U0dOHjiMHQOwNIx3K4ymzh/uS github.com/bytedance/sonic v1.15.0/go.mod h1:tFkWrPz0/CUCLEF4ri4UkHekCIcdnkqXw9VduqpJh0k= github.com/bytedance/sonic/loader v0.5.0 h1:gXH3KVnatgY7loH5/TkeVyXPfESoqSBSBEiDd5VjlgE= github.com/bytedance/sonic/loader v0.5.0/go.mod h1:AR4NYCk5DdzZizZ5djGqQ92eEhCCcdf5x77udYiSJRo= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cloudwego/base64x v0.1.6 h1:t11wG9AECkCDk5fMSoxmufanudBtJ+/HemLstXDLI2M= github.com/cloudwego/base64x v0.1.6/go.mod h1:OFcloc187FXDaYHvrNIjxSe8ncn0OOM8gEHfghB2IPU= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -111,16 +113,16 @@ github.com/wI2L/jsondiff v0.7.0 h1:1lH1G37GhBPqCfp/lrs91rf/2j3DktX6qYAKZkLuCQQ= github.com/wI2L/jsondiff v0.7.0/go.mod h1:KAEIojdQq66oJiHhDyQez2x+sRit0vIzC9KeK0yizxM= go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= -go.opentelemetry.io/otel v1.38.0 h1:RkfdswUDRimDg0m2Az18RKOsnI8UDzppJAtj01/Ymk8= -go.opentelemetry.io/otel v1.38.0/go.mod h1:zcmtmQ1+YmQM9wrNsTGV/q/uyusom3P8RxwExxkZhjM= -go.opentelemetry.io/otel/metric v1.38.0 h1:Kl6lzIYGAh5M159u9NgiRkmoMKjvbsKtYRwgfrA6WpA= -go.opentelemetry.io/otel/metric v1.38.0/go.mod h1:kB5n/QoRM8YwmUahxvI3bO34eVtQf2i4utNVLr9gEmI= -go.opentelemetry.io/otel/sdk v1.38.0 h1:l48sr5YbNf2hpCUj/FoGhW9yDkl+Ma+LrVl8qaM5b+E= -go.opentelemetry.io/otel/sdk v1.38.0/go.mod h1:ghmNdGlVemJI3+ZB5iDEuk4bWA3GkTpW+DOoZMYBVVg= -go.opentelemetry.io/otel/sdk/metric v1.38.0 h1:aSH66iL0aZqo//xXzQLYozmWrXxyFkBJ6qT5wthqPoM= -go.opentelemetry.io/otel/sdk/metric v1.38.0/go.mod h1:dg9PBnW9XdQ1Hd6ZnRz689CbtrUp0wMMs9iPcgT9EZA= -go.opentelemetry.io/otel/trace v1.38.0 h1:Fxk5bKrDZJUH+AMyyIXGcFAPah0oRcT+LuNtJrmcNLE= -go.opentelemetry.io/otel/trace v1.38.0/go.mod h1:j1P9ivuFsTceSWe1oY+EeW3sc+Pp42sO++GHkg4wwhs= +go.opentelemetry.io/otel v1.39.0 h1:8yPrr/S0ND9QEfTfdP9V+SiwT4E0G7Y5MO7p85nis48= +go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1NklR/tgc8= +go.opentelemetry.io/otel/metric v1.39.0 h1:d1UzonvEZriVfpNKEVmHXbdf909uGTOQjA0HF0Ls5Q0= +go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0x1nBERXd7Ftcs= +go.opentelemetry.io/otel/sdk v1.39.0 h1:nMLYcjVsvdui1B/4FRkwjzoRVsMK8uL/cj0OyhKzt18= +go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwmt1ENZszdJE= +go.opentelemetry.io/otel/sdk/metric v1.39.0 h1:cXMVVFVgsIf2YL6QkRF4Urbr/aMInf+2WKg+sEJTtB8= +go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5wosJsY1mt4lLfYdVew= +go.opentelemetry.io/otel/trace v1.39.0 h1:2d2vfpEDmCJ5zVYz7ijaJdOF59xLomrvj7bjt6/qCJI= +go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4EtqqlVdxWy+rNOA= go.uber.org/mock v0.6.0 h1:hyF9dfmbgIX5EfOdasqLsWD6xqpNZlXblLB/Dbnwv3Y= go.uber.org/mock v0.6.0/go.mod h1:KiVJ4BqZJaMj4svdfmHM0AUx4NJYO8ZNpPnZn1Z+BBU= go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0= @@ -146,8 +148,8 @@ google.golang.org/genproto/googleapis/api v0.0.0-20260114163908-3f89685c29c3 h1: google.golang.org/genproto/googleapis/api v0.0.0-20260114163908-3f89685c29c3/go.mod h1:dd646eSK+Dk9kxVBl1nChEOhJPtMXriCcVb4x3o6J+E= google.golang.org/genproto/googleapis/rpc v0.0.0-20260114163908-3f89685c29c3 h1:C4WAdL+FbjnGlpp2S+HMVhBeCq2Lcib4xZqfPNF6OoQ= google.golang.org/genproto/googleapis/rpc v0.0.0-20260114163908-3f89685c29c3/go.mod h1:j9x/tPzZkyxcgEFkiKEEGxfvyumM01BEtsW8xzOahRQ= -google.golang.org/grpc v1.78.0 h1:K1XZG/yGDJnzMdd/uZHAkVqJE+xIDOcmdSFZkBUicNc= -google.golang.org/grpc v1.78.0/go.mod h1:I47qjTo4OKbMkjA/aOOwxDIiPSBofUtQUI5EfpWvW7U= +google.golang.org/grpc v1.79.3 h1:sybAEdRIEtvcD68Gx7dmnwjZKlyfuc61Dyo9pGXXkKE= +google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/plugins/crowdstrike/go.mod b/plugins/crowdstrike/go.mod index 374752153..52e79b7d9 100644 --- a/plugins/crowdstrike/go.mod +++ b/plugins/crowdstrike/go.mod @@ -3,7 +3,7 @@ module github.com/utmstack/UTMStack/plugins/crowdstrike go 1.25.5 require ( - github.com/crowdstrike/gofalcon v0.20.0 + github.com/crowdstrike/gofalcon v0.20.1 github.com/google/uuid v1.6.0 github.com/threatwinds/go-sdk v1.1.21 google.golang.org/grpc v1.81.1 @@ -17,6 +17,7 @@ require ( github.com/bytedance/gopkg v0.1.3 // indirect github.com/bytedance/sonic v1.15.0 // indirect github.com/bytedance/sonic/loader v0.5.0 // indirect + github.com/cenkalti/backoff/v5 v5.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/cloudwego/base64x v0.1.6 // indirect github.com/gabriel-vasile/mimetype v1.4.13 // indirect diff --git a/plugins/crowdstrike/go.sum b/plugins/crowdstrike/go.sum index ab62a6c08..280fa3acc 100644 --- a/plugins/crowdstrike/go.sum +++ b/plugins/crowdstrike/go.sum @@ -10,12 +10,14 @@ github.com/bytedance/sonic v1.15.0 h1:/PXeWFaR5ElNcVE84U0dOHjiMHQOwNIx3K4ymzh/uS github.com/bytedance/sonic v1.15.0/go.mod h1:tFkWrPz0/CUCLEF4ri4UkHekCIcdnkqXw9VduqpJh0k= github.com/bytedance/sonic/loader v0.5.0 h1:gXH3KVnatgY7loH5/TkeVyXPfESoqSBSBEiDd5VjlgE= github.com/bytedance/sonic/loader v0.5.0/go.mod h1:AR4NYCk5DdzZizZ5djGqQ92eEhCCcdf5x77udYiSJRo= +github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM= +github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cloudwego/base64x v0.1.6 h1:t11wG9AECkCDk5fMSoxmufanudBtJ+/HemLstXDLI2M= github.com/cloudwego/base64x v0.1.6/go.mod h1:OFcloc187FXDaYHvrNIjxSe8ncn0OOM8gEHfghB2IPU= -github.com/crowdstrike/gofalcon v0.20.0 h1:Z/NERPVowOBCYsCgnkMRnrfrRkHBFo910kk7zrjzE0A= -github.com/crowdstrike/gofalcon v0.20.0/go.mod h1:a12GB+md+hRSgVCb3Pv6CakeTIsDIUCIVWRlJelIhY0= +github.com/crowdstrike/gofalcon v0.20.1 h1:cqdvyNeJvaQ9sK0k09h/n+z308VJffS+4JfgS+bNaSY= +github.com/crowdstrike/gofalcon v0.20.1/go.mod h1:GYbhi35odSf8qFrcxAX6Sx7N/QIJyz8vKmUzuam7Xd8= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= diff --git a/plugins/gcp/go.mod b/plugins/gcp/go.mod index f17873d36..f9b52cc6d 100644 --- a/plugins/gcp/go.mod +++ b/plugins/gcp/go.mod @@ -1,12 +1,12 @@ module github.com/utmstack/UTMStack/plugins/gcp -go 1.25.5 +go 1.25.8 require ( cloud.google.com/go/pubsub v1.50.2 github.com/google/uuid v1.6.0 github.com/threatwinds/go-sdk v1.1.21 - google.golang.org/api v0.279.0 + google.golang.org/api v0.282.0 google.golang.org/grpc v1.81.1 google.golang.org/protobuf v1.36.11 ) @@ -39,7 +39,7 @@ require ( github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect github.com/google/cel-go v0.27.0 // indirect github.com/google/s2a-go v0.1.9 // indirect - github.com/googleapis/enterprise-certificate-proxy v0.3.15 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.3.16 // indirect github.com/googleapis/gax-go/v2 v2.22.0 // indirect github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect github.com/json-iterator/go v1.1.12 // indirect @@ -66,17 +66,17 @@ require ( go.opentelemetry.io/otel/trace v1.43.0 // indirect go.yaml.in/yaml/v2 v2.4.3 // indirect golang.org/x/arch v0.24.0 // indirect - golang.org/x/crypto v0.50.0 // indirect + golang.org/x/crypto v0.51.0 // indirect golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect - golang.org/x/net v0.53.0 // indirect + golang.org/x/net v0.55.0 // indirect golang.org/x/oauth2 v0.36.0 // indirect golang.org/x/sync v0.20.0 // indirect - golang.org/x/sys v0.43.0 // indirect - golang.org/x/text v0.36.0 // indirect + golang.org/x/sys v0.45.0 // indirect + golang.org/x/text v0.37.0 // indirect golang.org/x/time v0.15.0 // indirect google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7 // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260523011958-0a33c5d7ca68 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect sigs.k8s.io/yaml v1.6.0 // indirect ) diff --git a/plugins/gcp/go.sum b/plugins/gcp/go.sum index b36838045..730c86718 100644 --- a/plugins/gcp/go.sum +++ b/plugins/gcp/go.sum @@ -107,8 +107,8 @@ github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0 github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/googleapis/enterprise-certificate-proxy v0.3.15 h1:xolVQTEXusUcAA5UgtyRLjelpFFHWlPQ4XfWGc7MBas= -github.com/googleapis/enterprise-certificate-proxy v0.3.15/go.mod h1:vqVt9yG9480NtzREnTlmGSBmFrA+bzb0yl0TxoBQXOg= +github.com/googleapis/enterprise-certificate-proxy v0.3.16 h1:F/VPrx0YPBdksZJQdCAp0WUsqnNmZpUZszzfYt0M5Dw= +github.com/googleapis/enterprise-certificate-proxy v0.3.16/go.mod h1:9Yb0eAkH/Xqhvv3zbeKf/+wMJqCeocWc6KIhDvEAuYE= github.com/googleapis/gax-go/v2 v2.22.0 h1:PjIWBpgGIVKGoCXuiCoP64altEJCj3/Ei+kSU5vlZD4= github.com/googleapis/gax-go/v2 v2.22.0/go.mod h1:irWBbALSr0Sk3qlqb9SyJ1h68WjgeFuiOzI4Rqw5+aY= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= @@ -206,8 +206,8 @@ golang.org/x/arch v0.24.0 h1:qlJ3M9upxvFfwRM51tTg3Yl+8CP9vCC1E7vlFpgv99Y= golang.org/x/arch v0.24.0/go.mod h1:dNHoOeKiyja7GTvF9NJS1l3Z2yntpQNzgrjh1cU103A= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/crypto v0.50.0 h1:zO47/JPrL6vsNkINmLoo/PH1gcxpls50DNogFvB5ZGI= -golang.org/x/crypto v0.50.0/go.mod h1:3muZ7vA7PBCE6xgPX7nkzzjiUq87kRItoJQM1Yo8S+Q= +golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI= +golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20260112195511-716be5621a96 h1:Z/6YuSHTLOHfNFdb8zVZomZr7cqNgTJvA8+Qz75D8gU= golang.org/x/exp v0.0.0-20260112195511-716be5621a96/go.mod h1:nzimsREAkjBCIEFtHiYkrJyT+2uy9YZJB7H1k68CXZU= @@ -220,8 +220,8 @@ golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73r golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.53.0 h1:d+qAbo5L0orcWAr0a9JweQpjXF19LMXJE8Ey7hwOdUA= -golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs= +golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= +golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= @@ -235,12 +235,12 @@ golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.43.0 h1:Rlag2XtaFTxp19wS8MXlJwTvoh8ArU6ezoyFsMyCTNI= -golang.org/x/sys v0.43.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= +golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.36.0 h1:JfKh3XmcRPqZPKevfXVpI1wXPTqbkE5f7JA92a55Yxg= -golang.org/x/text v0.36.0/go.mod h1:NIdBknypM8iqVmPiuco0Dh6P5Jcdk8lJL0CUebqK164= +golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc= +golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -251,8 +251,8 @@ golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBn golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= -google.golang.org/api v0.279.0 h1:hsx2M2OaRcaKtVYK6vXEUnQvdjnend7ZYES+lYaot74= -google.golang.org/api v0.279.0/go.mod h1:B9TqLBwJqVjp1mtt7WeoQwWRwvu/400y5lETOql+giQ= +google.golang.org/api v0.282.0 h1:WmJiSVqUnKqJCpJOx7YADbXaC+9DDsnGSfllFSj7R2I= +google.golang.org/api v0.282.0/go.mod h1:6Wssta4c5n9qHq5CBhmlai5h/PUa1djdDAIhYEHyvcM= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= @@ -262,8 +262,8 @@ google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 h1:XzmzkmB14QhVhgn google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7/go.mod h1:L43LFes82YgSonw6iTXTxXUX1OlULt4AQtkik4ULL/I= google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7 h1:41r6JMbpzBMen0R/4TZeeAmGXSJC7DftGINUodzTkPI= google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7/go.mod h1:EIQZ5bFCfRQDV4MhRle7+OgjNtZ6P1PiZBgAKuxXu/Y= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 h1:tEkOQcXgF6dH1G+MVKZrfpYvozGrzb91k6ha7jireSM= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260523011958-0a33c5d7ca68 h1:PvEgGJf9C/1u5CHkInMg7UFYYUoiaQmW2LbtH0pjB78= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260523011958-0a33c5d7ca68/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= diff --git a/plugins/modules-config/go.mod b/plugins/modules-config/go.mod index f04bf8c89..3e9aef7b5 100644 --- a/plugins/modules-config/go.mod +++ b/plugins/modules-config/go.mod @@ -1,21 +1,21 @@ module github.com/utmstack/UTMStack/plugins/modules-config -go 1.25.5 +go 1.25.8 require ( cloud.google.com/go/pubsub v1.50.2 github.com/AtlasInsideCorp/AtlasInsideAES v1.0.0 github.com/Azure/azure-sdk-for-go/sdk/messaging/azeventhubs/v2 v2.0.2 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.7.0 - github.com/aws/aws-sdk-go-v2/config v1.32.17 - github.com/aws/aws-sdk-go-v2/credentials v1.19.16 - github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.73.0 + github.com/aws/aws-sdk-go-v2/config v1.32.18 + github.com/aws/aws-sdk-go-v2/credentials v1.19.17 + github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.74.0 github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 - github.com/crowdstrike/gofalcon v0.20.0 + github.com/crowdstrike/gofalcon v0.20.1 github.com/gin-gonic/gin v1.12.0 github.com/threatwinds/go-sdk v1.1.21 golang.org/x/sync v0.20.0 - google.golang.org/api v0.279.0 + google.golang.org/api v0.282.0 google.golang.org/grpc v1.81.1 google.golang.org/protobuf v1.36.11 ) @@ -42,12 +42,13 @@ require ( github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 // indirect github.com/aws/aws-sdk-go-v2/service/signin v1.0.11 // indirect github.com/aws/aws-sdk-go-v2/service/sso v1.30.17 // indirect - github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21 // indirect + github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.0 // indirect github.com/aws/smithy-go v1.25.1 // indirect github.com/blang/semver/v4 v4.0.0 // indirect github.com/bytedance/gopkg v0.1.3 // indirect github.com/bytedance/sonic v1.15.0 // indirect github.com/bytedance/sonic/loader v0.5.0 // indirect + github.com/cenkalti/backoff/v5 v5.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/cloudwego/base64x v0.1.6 // indirect github.com/felixge/httpsnoop v1.0.4 // indirect @@ -85,7 +86,7 @@ require ( github.com/google/cel-go v0.27.0 // indirect github.com/google/s2a-go v0.1.9 // indirect github.com/google/uuid v1.6.0 // indirect - github.com/googleapis/enterprise-certificate-proxy v0.3.15 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.3.16 // indirect github.com/googleapis/gax-go/v2 v2.22.0 // indirect github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect github.com/json-iterator/go v1.1.12 // indirect @@ -119,14 +120,14 @@ require ( golang.org/x/arch v0.24.0 // indirect golang.org/x/crypto v0.51.0 // indirect golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect - golang.org/x/net v0.54.0 // indirect + golang.org/x/net v0.55.0 // indirect golang.org/x/oauth2 v0.36.0 // indirect - golang.org/x/sys v0.44.0 // indirect + golang.org/x/sys v0.45.0 // indirect golang.org/x/text v0.37.0 // indirect golang.org/x/time v0.15.0 // indirect google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7 // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260523011958-0a33c5d7ca68 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect sigs.k8s.io/yaml v1.6.0 // indirect ) diff --git a/plugins/modules-config/go.sum b/plugins/modules-config/go.sum index 73d9470b2..7028f3a99 100644 --- a/plugins/modules-config/go.sum +++ b/plugins/modules-config/go.sum @@ -46,10 +46,10 @@ github.com/aws/aws-sdk-go-v2 v1.41.7 h1:DWpAJt66FmnnaRIOT/8ASTucrvuDPZASqhhLey6t github.com/aws/aws-sdk-go-v2 v1.41.7/go.mod h1:4LAfZOPHNVNQEckOACQx60Y8pSRjIkNZQz1w92xpMJc= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10 h1:gx1AwW1Iyk9Z9dD9F4akX5gnN3QZwUB20GGKH/I+Rho= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.10/go.mod h1:qqY157uZoqm5OXq/amuaBJyC9hgBCBQnsaWnPe905GY= -github.com/aws/aws-sdk-go-v2/config v1.32.17 h1:FpL4/758/diKwqbytU0prpuiu60fgXKUWCpDJtApclU= -github.com/aws/aws-sdk-go-v2/config v1.32.17/go.mod h1:OXqUMzgXytfoF9JaKkhrOYsyh72t9G+MJH8mMRaexOE= -github.com/aws/aws-sdk-go-v2/credentials v1.19.16 h1:r3RJBuU7X9ibt8RHbMjWE6y60QbKBiII6wSrXnapxSU= -github.com/aws/aws-sdk-go-v2/credentials v1.19.16/go.mod h1:6cx7zqDENJDbBIIWX6P8s0h6hqHC8Avbjh9Dseo27ug= +github.com/aws/aws-sdk-go-v2/config v1.32.18 h1:Hcia46bxhGgF3BaSnG8nSNCWmqTK6bj9xN9/FJ3WK6Q= +github.com/aws/aws-sdk-go-v2/config v1.32.18/go.mod h1:zEjCAYmxqDadH1WX8CdBvmLKhUEUVFgKRQG38zjDmrY= +github.com/aws/aws-sdk-go-v2/credentials v1.19.17 h1:gP2nkGsS+KMvF/jfFz2Vv2qiiOqWKyPACSzPsqHgoW8= +github.com/aws/aws-sdk-go-v2/credentials v1.19.17/go.mod h1:Bsew3S/moG5iT77giPj1q8wb/s0RE5/QfH+ASjYtuQc= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23 h1:UuSfcORqNSz/ey3VPRS8TcVH2Ikf0/sC+Hdj400QI6U= github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.23/go.mod h1:+G/OSGiOFnSOkYloKj/9M35s74LgVAdJBSD5lsFfqKg= github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.23 h1:GpT/TrnBYuE5gan2cZbTtvP+JlHsutdmlV2YfEyNde0= @@ -58,8 +58,8 @@ github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23 h1:bpd8vxhlQi2r1hiueO github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.23/go.mod h1:15DfR2nw+CRHIk0tqNyifu3G1YdAOy68RftkhMDDwYk= github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24 h1:OQqn11BtaYv1WLUowvcA30MpzIu8Ti4pcLPIIyoKZrA= github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.24/go.mod h1:X5ZJyfwVrWA96GzPmUCWFQaEARPR7gCrpq2E92PJwAE= -github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.73.0 h1:JmrHkELR2Q0O28swrFMm0hZNwpQrV8qmbhnb7suKIfc= -github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.73.0/go.mod h1:MLJu3PUd8fp5Qvj4CiLvyY5H8y7kxHKlTp060Wsd+Vc= +github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.74.0 h1:6TqDeYdvJJEIJGg5ICy7nzC7/UuHk2Eg3wrpb5bWKPM= +github.com/aws/aws-sdk-go-v2/service/cloudwatchlogs v1.74.0/go.mod h1:MLJu3PUd8fp5Qvj4CiLvyY5H8y7kxHKlTp060Wsd+Vc= github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9 h1:FLudkZLt5ci0ozzgkVo8BJGwvqNaZbTWb3UcucAateA= github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.9/go.mod h1:w7wZ/s9qK7c8g4al+UyoF1Sp/Z45UwMGcqIzLWVQHWk= github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.23 h1:pbrxO/kuIwgEsOPLkaHu0O+m4fNgLU8B3vxQ+72jTPw= @@ -68,8 +68,8 @@ github.com/aws/aws-sdk-go-v2/service/signin v1.0.11 h1:TdJ+HdzOBhU8+iVAOGUTU63VX github.com/aws/aws-sdk-go-v2/service/signin v1.0.11/go.mod h1:R82ZRExE/nheo0N+T8zHPcLRTcH8MGsnR3BiVGX0TwI= github.com/aws/aws-sdk-go-v2/service/sso v1.30.17 h1:7byT8HUWrgoRp6sXjxtZwgOKfhss5fW6SkLBtqzgRoE= github.com/aws/aws-sdk-go-v2/service/sso v1.30.17/go.mod h1:xNWknVi4Ezm1vg1QsB/5EWpAJURq22uqd38U8qKvOJc= -github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21 h1:+1Kl1zx6bWi4X7cKi3VYh29h8BvsCoHQEQ6ST9X8w7w= -github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.21/go.mod h1:4vIRDq+CJB2xFAXZ+YgGUTiEft7oAQlhIs71xcSeuVg= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.0 h1:nDARhv/oF55bcxF7rCI/4PDxOKnVXVWwDuDwCs2I2SQ= +github.com/aws/aws-sdk-go-v2/service/ssooidc v1.36.0/go.mod h1:4vIRDq+CJB2xFAXZ+YgGUTiEft7oAQlhIs71xcSeuVg= github.com/aws/aws-sdk-go-v2/service/sts v1.42.1 h1:F/M5Y9I3nwr2IEpshZgh1GeHpOItExNM9L1euNuh/fk= github.com/aws/aws-sdk-go-v2/service/sts v1.42.1/go.mod h1:mTNxImtovCOEEuD65mKW7DCsL+2gjEH+RPEAexAzAio= github.com/aws/smithy-go v1.25.1 h1:J8ERsGSU7d+aCmdQur5Txg6bVoYelvQJgtZehD12GkI= @@ -82,6 +82,8 @@ github.com/bytedance/sonic v1.15.0 h1:/PXeWFaR5ElNcVE84U0dOHjiMHQOwNIx3K4ymzh/uS github.com/bytedance/sonic v1.15.0/go.mod h1:tFkWrPz0/CUCLEF4ri4UkHekCIcdnkqXw9VduqpJh0k= github.com/bytedance/sonic/loader v0.5.0 h1:gXH3KVnatgY7loH5/TkeVyXPfESoqSBSBEiDd5VjlgE= github.com/bytedance/sonic/loader v0.5.0/go.mod h1:AR4NYCk5DdzZizZ5djGqQ92eEhCCcdf5x77udYiSJRo= +github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM= +github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw= github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= @@ -93,8 +95,8 @@ github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2 h1:aBangftG7EVZoUb69Os github.com/cncf/xds/go v0.0.0-20260202195803-dba9d589def2/go.mod h1:qwXFYgsP6T7XnJtbKlf1HP8AjxZZyzxMmc+Lq5GjlU4= github.com/coder/websocket v1.8.14 h1:9L0p0iKiNOibykf283eHkKUHHrpG7f65OE3BhhO7v9g= github.com/coder/websocket v1.8.14/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg= -github.com/crowdstrike/gofalcon v0.20.0 h1:Z/NERPVowOBCYsCgnkMRnrfrRkHBFo910kk7zrjzE0A= -github.com/crowdstrike/gofalcon v0.20.0/go.mod h1:a12GB+md+hRSgVCb3Pv6CakeTIsDIUCIVWRlJelIhY0= +github.com/crowdstrike/gofalcon v0.20.1 h1:cqdvyNeJvaQ9sK0k09h/n+z308VJffS+4JfgS+bNaSY= +github.com/crowdstrike/gofalcon v0.20.1/go.mod h1:GYbhi35odSf8qFrcxAX6Sx7N/QIJyz8vKmUzuam7Xd8= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= @@ -221,8 +223,8 @@ github.com/google/s2a-go v0.1.9/go.mod h1:YA0Ei2ZQL3acow2O62kdp9UlnvMmU7kA6Eutn0 github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/googleapis/enterprise-certificate-proxy v0.3.15 h1:xolVQTEXusUcAA5UgtyRLjelpFFHWlPQ4XfWGc7MBas= -github.com/googleapis/enterprise-certificate-proxy v0.3.15/go.mod h1:vqVt9yG9480NtzREnTlmGSBmFrA+bzb0yl0TxoBQXOg= +github.com/googleapis/enterprise-certificate-proxy v0.3.16 h1:F/VPrx0YPBdksZJQdCAp0WUsqnNmZpUZszzfYt0M5Dw= +github.com/googleapis/enterprise-certificate-proxy v0.3.16/go.mod h1:9Yb0eAkH/Xqhvv3zbeKf/+wMJqCeocWc6KIhDvEAuYE= github.com/googleapis/gax-go/v2 v2.22.0 h1:PjIWBpgGIVKGoCXuiCoP64altEJCj3/Ei+kSU5vlZD4= github.com/googleapis/gax-go/v2 v2.22.0/go.mod h1:irWBbALSr0Sk3qlqb9SyJ1h68WjgeFuiOzI4Rqw5+aY= github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k= @@ -348,8 +350,8 @@ golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73r golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.54.0 h1:2zJIZAxAHV/OHCDTCOHAYehQzLfSXuf/5SoL/Dv6w/w= -golang.org/x/net v0.54.0/go.mod h1:Sj4oj8jK6XmHpBZU/zWHw3BV3abl4Kvi+Ut7cQcY+cQ= +golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= +golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= @@ -363,8 +365,8 @@ golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ= -golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= +golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc= @@ -379,8 +381,8 @@ golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBn golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= -google.golang.org/api v0.279.0 h1:hsx2M2OaRcaKtVYK6vXEUnQvdjnend7ZYES+lYaot74= -google.golang.org/api v0.279.0/go.mod h1:B9TqLBwJqVjp1mtt7WeoQwWRwvu/400y5lETOql+giQ= +google.golang.org/api v0.282.0 h1:WmJiSVqUnKqJCpJOx7YADbXaC+9DDsnGSfllFSj7R2I= +google.golang.org/api v0.282.0/go.mod h1:6Wssta4c5n9qHq5CBhmlai5h/PUa1djdDAIhYEHyvcM= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= @@ -390,8 +392,8 @@ google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7 h1:XzmzkmB14QhVhgn google.golang.org/genproto v0.0.0-20260319201613-d00831a3d3e7/go.mod h1:L43LFes82YgSonw6iTXTxXUX1OlULt4AQtkik4ULL/I= google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7 h1:41r6JMbpzBMen0R/4TZeeAmGXSJC7DftGINUodzTkPI= google.golang.org/genproto/googleapis/api v0.0.0-20260319201613-d00831a3d3e7/go.mod h1:EIQZ5bFCfRQDV4MhRle7+OgjNtZ6P1PiZBgAKuxXu/Y= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4 h1:tEkOQcXgF6dH1G+MVKZrfpYvozGrzb91k6ha7jireSM= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260427160629-7cedc36a6bc4/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260523011958-0a33c5d7ca68 h1:PvEgGJf9C/1u5CHkInMg7UFYYUoiaQmW2LbtH0pjB78= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260523011958-0a33c5d7ca68/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= diff --git a/rules/office365/credential_access_microsoft_365_brute_force_user_account_attempt.yml b/rules/office365/credential_access_microsoft_365_brute_force_user_account_attempt.yml deleted file mode 100644 index 0480cfc14..000000000 --- a/rules/office365/credential_access_microsoft_365_brute_force_user_account_attempt.yml +++ /dev/null @@ -1,30 +0,0 @@ -# Rule version v1.0.3 - -dataTypes: - - "o365" -name: "Attempts to Brute Force a Microsoft 365 User Account" -impact: - confidentiality: 3 - integrity: 3 - availability: 3 -category: "Credential Access" -technique: "T1110 - Brute Force" -adversary: origin -references: - - "https://attack.mitre.org/techniques/T1110/" - - "https://attack.mitre.org/tactics/TA0006/" -description: "Credential Access consists of techniques for stealing credentials like account names and passwords. Techniques used to get credentials include keylogging or credential dumping. Using legitimate credentials can give adversaries access to systems, make them harder to detect, and provide the opportunity to create more accounts to help achieve their goals.
- Identifies attempts to brute force a Microsoft 365 user account. An adversary may attempt a brute force attack to obtain unauthorized access to user accounts." -where: | - oneOf("log.Workload", ["Exchange", "AzureActiveDirectory"]) && oneOf("action", ["UserLoginFailed", "PasswordLogonInitialAuthUsingPassword"]) && oneOf("actionResult", ["Failed", "False"]) && exists("origin.user") -afterEvents: - - indexPattern: v11-log-o365-* - with: - - field: origin.user - operator: filter_term - value: '{{.origin.user}}' - within: now-60s - count: 5 -groupBy: - - adversary.ip - - adversary.user diff --git a/rules/windows/powershell_empire_detection.yml b/rules/windows/powershell_empire_detection.yml deleted file mode 100644 index 7ce3ca73f..000000000 --- a/rules/windows/powershell_empire_detection.yml +++ /dev/null @@ -1,48 +0,0 @@ -# Rule version v1.0.0 - -dataTypes: - - wineventlog -name: PowerShell Empire Detection -impact: - confidentiality: 3 - integrity: 3 - availability: 2 -category: Execution -technique: "T1059.001 - Command and Scripting Interpreter: PowerShell" -adversary: origin -references: - - https://attack.mitre.org/techniques/T1059/001/ - - https://www.powershellempire.com/ - - https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_logging -description: | - Detects potential PowerShell Empire framework usage based on characteristic command patterns, obfuscation techniques, and encoded payloads commonly used by this post-exploitation framework. PowerShell Empire is a post-exploitation framework that uses PowerShell and Python agents to maintain persistence and execute commands on compromised systems. - - Next Steps: - 1. Immediately isolate the affected host to prevent lateral movement - 2. Analyze the complete PowerShell script block content for additional IOCs - 3. Check for persistence mechanisms (scheduled tasks, registry entries, services) - 4. Review network connections from the host for C2 communication - 5. Examine process tree and parent processes that spawned PowerShell - 6. Search for additional Empire artifacts across the environment - 7. Reset credentials for any accounts used on the compromised system - 8. Conduct memory analysis to identify injected code or payloads - 9. Review recent user activity and file access patterns - 10. Update endpoint detection rules based on specific Empire techniques observed -where: | - (equals("log.eventCode", "4104") || equals("log.eventId", 4104)) && - equals("log.providerName", "Microsoft-Windows-PowerShell") && - ( - contains("log.eventDataScriptBlockText", "System.Management.Automation.AmsiUtils") || - regexMatch("log.eventDataScriptBlockText", "(?i)(empire|invoke-empire|invoke-psempire)") || - regexMatch("log.eventDataScriptBlockText", "(?i)\\[System\\.Convert\\]::FromBase64String") || - regexMatch("log.eventDataScriptBlockText", "(?i)IEX\\s*\\(\\s*New-Object") || - regexMatch("log.eventDataScriptBlockText", "(?i)-enc\\s+[A-Za-z0-9+/=]{100,}") || - regexMatch("log.eventDataScriptBlockText", "(?i)\\$DoIt\\s*=\\s*@") || - regexMatch("log.eventDataScriptBlockText", "(?i)\\[System\\.Text\\.Encoding\\]::Unicode\\.GetString") || - contains("log.eventDataScriptBlockText", "Invoke-Shellcode") || - contains("log.eventDataScriptBlockText", "Invoke-ReflectivePEInjection") || - contains("log.eventDataScriptBlockText", "Invoke-Mimikatz") - ) -groupBy: - - origin.host - - target.user diff --git a/rules/windows/rdp_brute_force_attacks.yml b/rules/windows/rdp_brute_force_attacks.yml deleted file mode 100644 index 9161a68b9..000000000 --- a/rules/windows/rdp_brute_force_attacks.yml +++ /dev/null @@ -1,44 +0,0 @@ -# Rule version v1.0.0 - -dataTypes: - - wineventlog -name: RDP Brute Force Attack -impact: - confidentiality: 3 - integrity: 2 - availability: 2 -category: Credential Access -technique: "T1110.001 - Brute Force: Password Guessing" -adversary: origin -references: - - https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/event.aspx?eventID=4625 - - https://attack.mitre.org/techniques/T1110/001/ -description: | - Detects multiple failed RDP login attempts from the same source IP address, indicating a potential brute force attack. This rule monitors Windows Event ID 4625 (failed logon) with focus on network logon types (type 3) which are commonly used for RDP connections. The rule triggers when 10 or more failed attempts occur from the same IP within 15 minutes. - - Next Steps: - 1. Investigate the source IP address for malicious indicators and geolocation - 2. Check if the targeted user accounts are legitimate and active - 3. Review successful logons from the same IP after failed attempts - 4. Implement IP blocking or rate limiting for the source address - 5. Enable account lockout policies if not already configured - 6. Consider implementing multi-factor authentication for RDP access - 7. Review RDP access logs for any successful connections during the attack timeframe -where: equals("log.eventCode", "4625") && equals("log.eventDataLogonType", "3") && exists("origin.ip") && !equals("origin.ip", "-") && !equals("origin.ip", "::1") && !equals("origin.ip", "127.0.0.1") -afterEvents: - - indexPattern: v11-log-wineventlog-* - with: - - field: origin.ip.keyword - operator: filter_term - value: '{{.origin.ip}}' - - field: log.eventCode - operator: filter_term - value: '4625' - - field: log.eventDataLogonType - operator: filter_term - value: '3' - within: now-15m - count: 10 -groupBy: - - origin.ip - - target.host diff --git a/user-auditor/pom.xml b/user-auditor/pom.xml index 84e566434..0475eec7b 100644 --- a/user-auditor/pom.xml +++ b/user-auditor/pom.xml @@ -43,7 +43,7 @@ org.postgresql postgresql - 42.7.2 + 42.7.11