diff --git a/examples/security/detection/README.md b/examples/security/detection/README.md index 14c690f..cd0c629 100644 --- a/examples/security/detection/README.md +++ b/examples/security/detection/README.md @@ -2,10 +2,11 @@ Threat detection, alerting, and rule management workflows -## Workflows (10) +## Workflows (11) | Workflow | Description | |----------|-------------| +| [DaC - VCS Rule Sync](./dac-vcs-rule-sync/dac-vcs-rule-sync.yaml) | Detection-as-Code sync to version control: rules tagged `vcs` in a dev space are exported through a detection-rules fork and opened as pull requests against main, with unit tests reported on the PR ([setup guide](./dac-vcs-rule-sync/README.md)) | | [Sync Rule Exceptions Across Spaces](./sync-rule-exceptions-across-spaces.yaml) | One-way sync of rule exceptions between Kibana spaces: exceptions added to any rule's default exception list in a source space (e.g. a "subset" space) are pushed to the matching rules (same rule_id) in a target space (e.g. the "main" space), skipping items already synced, with opt-in consts to overwrite items when the source copy is newer and to delete workflow-pushed copies whose source exception was removed. It is self-contained in Kibana, no external services | | [Disable Noisy Elastic Defend Behavior Rules](./disable-noisy-endpoint-rules-from-esql.yaml) | Detects noisy Elastic Defend behavior rules via ES\|QL (FP bursts) and auto-creates scoped Endpoint Security exceptions by rule version; new artifact versions are unaffected | | [␆ Mark Alert as Acknowledged](./mark-alert-as-acknowledged.yaml) | This workflow YAML defines a manual trigger for acknowledging security alerts | diff --git a/examples/security/detection/dac-vcs-rule-sync/DEPLOY_AND_TEST.md b/examples/security/detection/dac-vcs-rule-sync/DEPLOY_AND_TEST.md new file mode 100644 index 0000000..6e66796 --- /dev/null +++ b/examples/security/detection/dac-vcs-rule-sync/DEPLOY_AND_TEST.md @@ -0,0 +1,166 @@ +# Deploy & Test Guide — DaC VCS Rule Sync + +End-to-end walkthrough for deploying and validating the VCS rule sync +pipeline (see [README.md](./README.md) for the architecture). It assumes you +already have: + +- a fork of [elastic/detection-rules](https://github.com/elastic/detection-rules) +- an Elastic stack that can reach GitHub, and GitHub Actions runners that can + reach your Kibana + +Everything on the Kibana side is configured through the UI — the GitHub +connector does all the GitHub work (no PATs in YAML, no API-only connector +setup). + +Note the sync direction: **Kibana is the source of truth for rules, not the +detection-rules repo**. This pipeline pushes rule state from Kibana into +GitHub so it can be reviewed, merged, and stored in VCS; it never writes +rules back to Kibana. Additional GitHub Actions (e.g. importing to Kibana on +merge) could turn this into a dual-sync architecture — in its example form +it is one-way, Kibana → VCS/GitHub. + +--- + +## Phase 0 — One-time prerequisites + +### 1. Enable Workflows in Kibana + +Kibana 9.3+ (tech preview): in the space where you'll work, go to +**Stack Management → Advanced Settings**, search "workflows", and enable the +Workflows UI. Your user needs the **Workflows: All** privilege. + +### 2. Create the dev space + +**Stack Management → Spaces** → create `dev` (or use an existing space). The +workflow `consts` take the space **ID** (the lowercase URL slug), not the +display name. + +### 3. Create the export API key (becomes `DR_API_KEY`) + +**Stack Management → API Keys → Create API key**, created as a user who can +**read rules in the dev space** (Security feature privilege). That's all the +pipeline needs — it never writes to Kibana or Elasticsearch from CI. Copy +the **Encoded** value. + +### 4. Create the GitHub connector in Kibana + +Generate a GitHub credential that can write to your fork — fine-grained PAT +scoped to the fork with **Contents: read/write** and **Pull requests: +read/write**. Then **Stack Management → Connectors → Create connector → +GitHub**, paste the credential. Grab the connector ID from the connector's +details (or Dev Tools: `GET kbn:/api/actions/connectors`). + +The workflow only uses tools this connector is verified to expose: +`create_branch`, `create_or_update_file`, `create_pull_request`, and PR +search. It never dispatches GitHub Actions directly. + +### 5. Network sanity check + +GitHub-hosted runners must be able to reach your Kibana URL (for the rule +export). If your stack is not internet-reachable, use a self-hosted runner. + +--- + +## Phase 1 — Set up the fork + +1. Copy [`github/dac-vcs-rule-sync.yml`](./github/dac-vcs-rule-sync.yml) into + your fork at `.github/workflows/dac-vcs-rule-sync.yml` and push it to + `main`. +2. In the fork, **Settings → Secrets and variables → Actions**, add the + connection secrets. Pick **one** connection form: + + | Secret | Value | + |--------|-------| + | `DR_CLOUD_ID` | Cloud ID from the deployment overview page (Elastic Cloud) | + | `DR_KIBANA_URL` | …or the Kibana base URL, no `/s/` suffix (self-managed) | + | `DR_API_KEY` | Always: the encoded key from Phase 0 | + + Don't set both `DR_CLOUD_ID` and `DR_KIBANA_URL` — the Action fails fast + on that combination. +3. (Recommended) Commit the custom rules directory so its config is + version-controlled: + + ```bash + python -m detection_rules custom-rules setup-config dac-vcs-rules + git add dac-vcs-rules && git commit -m "Add DaC VCS custom rules dir" && git push + ``` + +--- + +## Phase 2 — Deploy the Elastic workflow + +1. In Kibana: **Workflows → Create workflow**, paste + [`dac-vcs-rule-sync.yaml`](./dac-vcs-rule-sync.yaml). + + > **Workflows are space-specific.** The workflow is saved into whichever + > space you're in when you create it, and that's the only space where it + > (and its execution history) will show up — note which space you use. + +2. Edit `consts`: at minimum `github_owner` and `github_connector_id`; also + `dev_space`/`sync_tag` if you diverged from the defaults. +3. Save, run it manually, and open the execution. With nothing tagged yet + you should see: `No rules tagged "vcs" found in space "dev" — nothing to + sync.` That proves the rule search and consts are wired correctly. + +--- + +## Phase 3 — First sync (happy path) + +1. In the **dev** space, create or pick a custom rule and add the tag + **`vcs`**. +2. Run the workflow manually (or wait for the schedule). The execution log + should show `VCS sync PR opened for 1 changed rule(s): ...`. +3. On the fork, verify the chain: + - a branch `vcs-sync-` exists, containing + `.vcs-sync/vcs-sync-.json`; + - the **DaC - VCS rule sync** Action ran on that branch: exported the rule + (`Export tagged rules...`), committed it (`[VCS Sync] Export 'vcs' + tagged rules from space 'dev'`), and ran the unit tests; + - the PR against `main` now shows the rule TOML in its diff and a + ✅ *unit tests passed* comment. +4. **Merge the PR.** The rule is now in version control. +5. Run the workflow again without touching the rule: the log should show + `... none changed since the last sync ...` — the watermark works, no new + PR. +6. Edit the tagged rule in dev (e.g. tweak the description) and run again: a + fresh sync PR opens with just that diff. + +--- + +## Phase 4 — Failure and edge-case behavior + +- **Unit test failure:** if the exported rules fail the detection-rules test + suite, the Action run goes red on the PR and posts a ❌ comment with the + test output tail. The rules are still committed, so the diff stays + reviewable. Fix the rule in dev and let the next cycle re-sync (close the + failing PR first, or push a fix to its branch). +- **No-change sync:** if a sync PR's export produces no file changes, the + Action closes the PR and deletes the branch automatically, with a comment. +- **Open PR gate:** while any `[VCS Sync]` PR is open, the workflow skips + new syncs (`... a "[VCS Sync]" PR is already open ...`). Merge or close it + to resume. +- **Failed sync retry:** the watermark only advances after the PR opens + successfully, so a failed branch/PR creation is retried on the next cycle. + +--- + +## Phase 5 — Go live + +Nothing else to do — the workflow saved enabled with a 30-minute schedule, +and idle cycles are no-ops. Adjust the cadence on the `scheduled` trigger if +you want, and keep an eye on the workflow's execution history for the first +day (it's only visible in the space where you saved the workflow). + +--- + +## Troubleshooting map + +| Symptom | Likely cause | +|---------|--------------| +| Branch/file/PR steps fail in the Elastic workflow | Wrong `github_connector_id`, or the connector's credential lacks Contents/Pull requests write on the fork | +| PR opened but the Action never ran | The Action file isn't at `.github/workflows/dac-vcs-rule-sync.yml` on `main`, or Actions are disabled on the fork; note the trigger only matches `vcs-sync-*` branches and `.vcs-sync/**` paths | +| Action fails at "Validate connection settings" | Neither `DR_CLOUD_ID` nor `DR_KIBANA_URL` set, both set at once, or `DR_API_KEY` missing | +| Action export finds 0 rules | Tag/space mismatch between workflow consts and reality, or the rule is prebuilt rather than custom | +| Every run says "none changed" but you expect a sync | The watermark in `dac-vcs-sync-state` is ahead — check `GET dac-vcs-sync-state/_search` in Dev Tools; delete the index to force a full re-sync | +| Sync PRs keep opening and closing with no changes | A rule's `updated_at` is being bumped without content changes (e.g. repeated tag toggling); harmless, the Action closes them | +| Workflow or its execution history "missing" from the Workflows page | Workflows are space-specific — switch to the space where you created it | diff --git a/examples/security/detection/dac-vcs-rule-sync/README.md b/examples/security/detection/dac-vcs-rule-sync/README.md new file mode 100644 index 0000000..78ebd98 --- /dev/null +++ b/examples/security/detection/dac-vcs-rule-sync/README.md @@ -0,0 +1,163 @@ +# DaC VCS Rule Sync (Dev → GitHub PR) + +A Detection-as-Code (DaC) sync into version control, built on the +[detection-rules](https://github.com/elastic/detection-rules) CLI, following the +[DaC quick start guide](https://dac-reference.readthedocs.io/en/latest/dac_quick_start_guide.html). + +Rules are authored in a **dev** Kibana space. Tagging a rule with **`vcs`** +means it is *managed by the version control system* (GitHub). On a schedule, +the Elastic workflow checks whether any tagged rule changed since the last +sync, and if so uses the **Kibana GitHub connector** (created in the UI — no +API-only setup, no tokens in YAML) to open a pull request against your +detection-rules fork's `main`. The fork's GitHub Action populates the PR +branch with the exported rules and runs the detection-rules unit tests, with +results posted on the PR. Nothing is copied between Kibana spaces, and there +is no Kibana-side alerting — review, test results, and history all live in +GitHub. + +> **Sync direction: Kibana → VCS (one-way).** In this pipeline **Kibana is +> the source of truth for rule content — not the detection-rules repo**. +> Rules are authored and edited in the dev Kibana space, and version control +> *receives* those updates so they can be reviewed, merged, and stored in +> git. Nothing here writes rules back to Kibana. Adding further GitHub +> Actions (for example, a `kibana import-rules` job that runs on merge to +> `main`) could evolve this into a dual-sync architecture where VCS changes +> also flow back into Kibana — but in its example form this is strictly +> Kibana syncing to VCS/GitHub. + +``` +┌──────────────── Kibana ────────────────┐ ┌───────────── GitHub fork ─────────────┐ +│ dev space: rules tagged "vcs" │ │ │ +│ │ 1. _find by tag │ │ branch vcs-sync- │ +│ ▼ │ │ ├─ .vcs-sync/.json ◀──┐ │ +│ [Elastic workflow] │ │ └─ dac-vcs-rules/ (exported) ◀─┼─┐ │ +│ 2. changed since last sync? │ │ ▲ │ │ │ +│ 3. GitHub connector: │ │ │ 4. Action (on push): │ │ │ +│ create_branch ────────────────────┼────────┼────────┘ export-rules -e -ac ──┘ │ │ +│ create_or_update_file (request) ──┼────────┼─────────── commit + unit tests ───┘ │ +│ create_pull_request ──────────────┼────────┼──▶ PR vs main (tests commented) │ │ +│ 5. record sync watermark (ES index) │ │ merge = rules in VCS │ │ +└────────────────────────────────────────┘ └───────────────────────────────────────┘ +``` + +## Files + +| File | Purpose | Where it goes | +|------|---------|---------------| +| [`dac-vcs-rule-sync.yaml`](./dac-vcs-rule-sync.yaml) | Elastic workflow: finds `vcs`-tagged rules, detects changes, opens the sync branch/PR via the GitHub connector | Kibana → Workflows (space-specific — it lives in the space you create it in) | +| [`github/dac-vcs-rule-sync.yml`](./github/dac-vcs-rule-sync.yml) | GitHub Action: exports tagged rules into the sync branch, commits, runs unit tests, comments on the PR | Your detection-rules fork, at `.github/workflows/dac-vcs-rule-sync.yml` | +| [`DEPLOY_AND_TEST.md`](./DEPLOY_AND_TEST.md) | Step-by-step deployment walkthrough | Reference doc | + +## Configuration + +Everything environment-specific is parameterized in the Elastic workflow's +`consts` block: + +| Const | Default | Meaning | +|-------|---------|---------| +| `dev_space` | `dev` | Kibana space rules are authored and tagged in | +| `sync_tag` | `vcs` | Rule tag meaning "managed by version control" | +| `github_owner` | — | Owner of your detection-rules fork (user or org) | +| `github_repo_name` | `detection-rules` | Fork repository name | +| `github_base_branch` | `main` | Branch the sync PRs target | +| `github_connector_id` | — | Kibana GitHub connector used for branch/file/PR operations | +| `sync_request_dir` | `.vcs-sync` | Where sync-request files are committed (also the Action's trigger path) | +| `pr_title_prefix` | `[VCS Sync]` | PR title prefix; also used to detect an already-open sync PR | +| `state_index` | `dac-vcs-sync-state` | Elasticsearch index storing the sync watermark | + +The sync cadence (default: every 30 minutes) is set on the `scheduled` +trigger directly — trigger definitions cannot reference `consts`. Unchanged +rules are skipped, so frequent runs are cheap no-ops. + +## Setup + +### 1. detection-rules fork + +1. Fork [elastic/detection-rules](https://github.com/elastic/detection-rules). +2. Copy [`github/dac-vcs-rule-sync.yml`](./github/dac-vcs-rule-sync.yml) into + the fork at `.github/workflows/dac-vcs-rule-sync.yml` on `main`. +3. (Recommended) Commit a custom rules directory so its config is + version-controlled — the Action expects it at `dac-vcs-rules/`: + + ```bash + python -m detection_rules custom-rules setup-config dac-vcs-rules + git add dac-vcs-rules && git commit -m "Add DaC VCS custom rules dir" && git push + ``` + + If it is missing, the Action bootstraps a default one per run (but then + only the `rules/` content lands in the PR diff, not the config). +4. Add the repository secrets (Settings → Secrets and variables → Actions). + Provide **either** the Cloud ID **or** the Kibana URL — not both: + + | Secret | Value | + |--------|-------| + | `DR_CLOUD_ID` | Elastic Cloud ID (deployment overview page) — alternative to the URL | + | `DR_KIBANA_URL` | Kibana base URL (no `/s/` suffix) | + | `DR_API_KEY` | Always required: base64 API key able to read rules in the dev space | + +### 2. Kibana + +1. Create a **GitHub connector** (Stack Management → Connectors → GitHub) + whose credential can write to the fork: create branches, commit files, + and open pull requests (fine-grained PAT: Contents read/write + + Pull requests read/write on the fork). +2. Create a new workflow from + [`dac-vcs-rule-sync.yaml`](./dac-vcs-rule-sync.yaml) and fill in the + `consts` block (at minimum `github_owner` and `github_connector_id`). + + > **Workflows are space-specific.** The workflow only appears — and its + > execution history only lives — in the space where you save it, so + > remember which space that is. The dev space is a natural home. Steps + > target the dev space explicitly via `/s//...` paths regardless + > of where the workflow lives. + +3. Run it once manually to confirm the consts are right. + +### 3. Sync a rule + +Tag any custom rule in the dev space with `vcs`. Within one schedule +interval the workflow opens a PR (`vcs-sync-` → `main`), and the +Action fills it with the exported rule TOML and comments the unit test +results. Merge the PR to accept the rule state into version control. Edit +the rule again and the next cycle opens a fresh sync PR with the diff. + +## Behavior notes + +- **One-way sync, Kibana as source of truth.** Rule content flows from + Kibana into git only. Editing a rule's TOML directly in the repo does not + change the rule in Kibana, and the next sync of that rule will overwrite + the repo copy with Kibana's version. +- **Change detection.** The workflow stores a watermark (the max `updated_at` + across tagged rules) in `dac-vcs-sync-state` after each sync PR. Runs where + no tagged rule is newer than the watermark do nothing. Adding or removing + the tag bumps a rule's `updated_at`, so newly tagged rules sync on the next + cycle. +- **One sync PR at a time.** If a `[VCS Sync]` PR is already open, the + workflow skips the cycle; changes are picked up on the first cycle after + the PR is merged or closed. +- **Edits are included.** `export-rules` overwrites existing TOML files by + default (there is no `--overwrite` flag on export — that flag belongs to + `import-rules`), so edited rules update in place and show as diffs in the + PR. +- **No-change PRs close themselves.** If the export produces no file changes + (e.g. a rule was touched but re-normalizes identically), the Action closes + the PR and deletes the branch. +- **Unit test results live in GitHub.** A failing test suite fails the + Action run (red on the PR) and posts a comment with the test output tail. + There is no Kibana-side failure alerting in this design. +- **Deletions are not synced.** Untagging (or deleting) a rule stops future + syncs but does not remove the TOML already in the repo — prune manually. +- **`.vcs-sync/` is an audit log.** One small request JSON per sync rides + along in each PR; merged PRs accumulate them as a history of syncs. Prune + whenever you like. +- **Custom rules only.** The export uses `--custom-rules-only`; prebuilt + Elastic rules are ignored even if tagged. + +## Migrating from the earlier "promote dev → prod" pipeline + +If you deployed the previous iteration of this example: delete the old +`.github/workflows/dac-promote-rules.yml` from the fork, remove the old +"DaC - Promote Tagged Rules from Dev to Prod" workflow in Kibana, and delete +the "DaC Promotion Failure" detection rule and `dac-promotion-failures` +index in the dev space if they were created. The `DR_ELASTICSEARCH_URL` +secret is no longer needed. diff --git a/examples/security/detection/dac-vcs-rule-sync/dac-vcs-rule-sync.yaml b/examples/security/detection/dac-vcs-rule-sync/dac-vcs-rule-sync.yaml new file mode 100644 index 0000000..f3beaeb --- /dev/null +++ b/examples/security/detection/dac-vcs-rule-sync/dac-vcs-rule-sync.yaml @@ -0,0 +1,301 @@ +# ============================================================================= +# Workflow: DaC - VCS Rule Sync +# Category: security/detection +# +# Detection-as-Code (DaC) sync into version control. Rules in a "dev" Kibana +# space tagged "vcs" are managed by the version control system (GitHub): +# whenever a tagged rule changes, this workflow opens a pull request against +# the detection-rules fork's main branch containing the exported rules. +# Nothing is copied between Kibana spaces — GitHub is the destination. +# +# On each run (scheduled + manual): +# 1. Find rules in {{ consts.dev_space }} tagged {{ consts.sync_tag }}. +# 2. Compare each rule's updated_at against the last recorded sync watermark +# (stored in the {{ consts.state_index }} index) — no changes, no PR. +# 3. If anything changed and no sync PR is already open, use the GitHub +# connector to: +# - create a branch vcs-sync-, +# - commit a sync-request file under .vcs-sync/ describing the sync, +# - open a PR from that branch against {{ consts.github_base_branch }}. +# 4. The request commit triggers the companion GitHub Action in the fork +# (github/dac-vcs-rule-sync.yml), which exports the tagged rules into the +# branch with the detection-rules CLI (kibana export-rules -e -ac; +# existing TOML is overwritten so edits are included), commits them, runs +# the unit tests, and comments the results on the PR. Test results live +# entirely in GitHub (PR checks/comments) — there is no Kibana-side +# failure alerting. +# 5. Record the new sync watermark so unchanged rules don't re-sync. +# +# Requirements: +# - A GitHub connector in Kibana (created in the UI) whose credential can +# read/write the fork: create branches, commit files, and open PRs. Only +# connector tools verified to exist are used (create_branch, +# create_or_update_file, create_pull_request, searchPullRequests) — no +# GitHub Actions dispatch, no PAT in this YAML. +# - The companion Action + secrets installed in the detection-rules fork. +# See README.md next to this file for full setup. +# +# All environment-specific values (space, tag, fork owner/repo, connector) +# are parameterized in `consts` below. Note: the scheduled trigger interval +# cannot reference consts, so adjust it directly in the trigger definition. +# +# Kibana workflows are space-specific: this workflow (and its execution +# history) only appears in the space it was created in. Any space works — +# steps target the dev space explicitly via /s// paths — but remember +# where you saved it. +# ============================================================================= +version: "1" +name: DaC - VCS Rule Sync +description: | + Detection-as-Code sync to version control: finds rules in the dev space + tagged "vcs", and when any of them changed since the last sync, opens a + pull request on the detection-rules fork (branch vcs-sync-). + The fork's GitHub Action exports the tagged rules into the branch, runs + the detection-rules unit tests, and comments results on the PR. +enabled: true +tags: + - security + - detection + - dac + - automation + +consts: + # --- Kibana space rules are authored/tagged in ----------------------------- + dev_space: "dev" + # --- Sync tag: rules with this tag are managed by version control ---------- + sync_tag: "vcs" + # --- detection-rules fork the sync PRs are opened against ------------------ + github_owner: "REPLACE_WITH_GITHUB_OWNER" # fork owner (user or org) + github_repo_name: "detection-rules" # fork repository name + github_base_branch: "main" # PR target branch + github_connector_id: "REPLACE_WITH_GITHUB_CONNECTOR_ID" + # --- Sync bookkeeping ------------------------------------------------------- + sync_request_dir: ".vcs-sync" # request files committed to the branch + pr_title_prefix: "[VCS Sync]" # also used to detect an open sync PR + state_index: "dac-vcs-sync-state" # ES index storing the sync watermark + +triggers: + - type: manual + # Adjust the sync cadence here (consts cannot be referenced in triggers). + # Unchanged rules are skipped, so frequent runs are cheap no-ops. + - type: scheduled + with: + every: 30m + +steps: + # Find all rules in the dev space carrying the sync tag. per_page caps a + # single sync batch; raise it if you tag more than 500 rules. + - name: find_tagged_rules + type: kibana.request + with: + method: GET + path: "/s/{{ consts.dev_space }}/api/detection_engine/rules/_find" + query: + filter: 'alert.attributes.tags:"{{ consts.sync_tag }}"' + per_page: "500" + headers: + kbn-xsrf: "true" + + - name: if_rules_found + type: if + condition: steps.find_tagged_rules.output.total > 0 + steps: + # The last sync watermark: the max updated_at across the tagged rules + # at the time of the previous sync. The wildcard index pattern makes a + # missing index (first run, before any sync is recorded) return empty + # hits instead of an index_not_found_exception error. No ES sort — the + # sort-string handling and field mappings vary across builds, so the + # newest watermark is computed from the hits in Liquid instead + # (see set_sync_context.last_sync). + - name: get_last_sync + type: elasticsearch.search + with: + index: "{{ consts.state_index }}*" + size: 500 + query: + match_all: {} + on-failure: + continue: true + + - name: set_sync_context + type: data.set + with: + sync_time: "{{ 'now' | date: '%Y-%m-%dT%H:%M:%SZ' }}" + branch_name: "vcs-sync-{{ 'now' | date: '%Y%m%d-%H%M%S' }}" + last_sync: "{% if steps.get_last_sync.output.hits.hits and steps.get_last_sync.output.hits.hits[0] %}{{ steps.get_last_sync.output.hits.hits | map: '_source' | map: 'synced_through' | sort | last }}{% endif %}" + # New watermark: the latest updated_at among all tagged rules. + # Recording this (instead of "now") means an edit made while this + # run executes is still strictly newer and syncs next cycle. + synced_through: "{{ steps.find_tagged_rules.output.data | map: 'updated_at' | sort | last }}" + + # ISO-8601 UTC timestamps compare correctly as strings, so a plain + # lexicographic updated_at > last_sync finds the changed rules. + # rule_list_md is the pre-rendered markdown bullet list for the PR body + # (the workflow editor's validation rejects dynamic index paths like + # ids[forloop.index0], so name/id pairing happens here in one loop). + - name: set_changed_rules + type: data.set + with: + rule_ids: "{%- assign ls = steps.set_sync_context.output.last_sync | strip -%}{%- assign out = '' -%}{%- for r in steps.find_tagged_rules.output.data -%}{%- if ls == '' or r.updated_at > ls -%}{%- if out == '' -%}{%- assign out = r.rule_id -%}{%- else -%}{%- assign out = out | append: ',' | append: r.rule_id -%}{%- endif -%}{%- endif -%}{%- endfor -%}{{ out }}" + changed_count: "{%- assign ls = steps.set_sync_context.output.last_sync | strip -%}{%- assign n = 0 -%}{%- for r in steps.find_tagged_rules.output.data -%}{%- if ls == '' or r.updated_at > ls -%}{%- assign n = n | plus: 1 -%}{%- endif -%}{%- endfor -%}{{ n }}" + rule_list_md: |- + {%- assign ls = steps.set_sync_context.output.last_sync | strip -%} + {%- for r in steps.find_tagged_rules.output.data -%} + {%- if ls == '' or r.updated_at > ls %} + - {{ r.name }} (`{{ r.rule_id }}`) + {%- endif -%} + {%- endfor -%} + + - name: if_changes + type: if + condition: 'steps.set_changed_rules.output.rule_ids: * AND not steps.set_changed_rules.output.rule_ids: ""' + steps: + # One sync PR at a time: changes made while a sync PR is open are + # picked up by the next cycle after it is merged or closed. + - name: search_open_sync_pr + type: github.searchPullRequests + connector-id: "{{ consts.github_connector_id }}" + timeout: 60s + with: + query: 'repo:{{ consts.github_owner }}/{{ consts.github_repo_name }} is:open "{{ consts.pr_title_prefix }}" in:title' + order: desc + sort: created + page: 1 + perPage: 1 + on-failure: + continue: true + + # Reading output.total_count when the search failed (output is + # null) throws a TypeError in the condition evaluator, so guard + # the output access behind a search-succeeded check first. + - name: if_pr_search_ok + type: if + condition: "steps.search_open_sync_pr.output: * AND not steps.search_open_sync_pr.error: *" + steps: + - name: if_no_open_sync_pr + type: if + condition: "steps.search_open_sync_pr.output.total_count: 0" + steps: + - name: create_sync_branch + type: github.callTool + connector-id: "{{ consts.github_connector_id }}" + timeout: 60s + with: + name: create_branch + arguments: + owner: "{{ consts.github_owner }}" + repo: "{{ consts.github_repo_name }}" + branch: "{{ steps.set_sync_context.output.branch_name }}" + from_branch: "{{ consts.github_base_branch }}" + on-failure: + continue: true + + - name: if_branch_created + type: if + condition: "not steps.create_sync_branch.error: *" + steps: + # One request file per sync (named after the branch), so it + # never collides with a previously merged request. It also + # carries the space/tag so the fork needs no duplicated + # configuration, and doubles as a sync audit log on main. + - name: set_sync_request + type: data.set + with: + request_json: '{"requested_at":"{{ steps.set_sync_context.output.sync_time }}","dev_space":"{{ consts.dev_space }}","sync_tag":"{{ consts.sync_tag }}","branch":"{{ steps.set_sync_context.output.branch_name }}","workflow_execution_id":"{{ execution.id }}","changed_rule_ids":[{%- assign ids = steps.set_changed_rules.output.rule_ids | split: "," -%}{%- for id in ids -%}"{{ id }}"{% unless forloop.last %},{% endunless %}{%- endfor -%}]}' + + - name: commit_sync_request + type: github.callTool + connector-id: "{{ consts.github_connector_id }}" + timeout: 120s + with: + name: create_or_update_file + arguments: + owner: "{{ consts.github_owner }}" + repo: "{{ consts.github_repo_name }}" + path: "{{ consts.sync_request_dir }}/{{ steps.set_sync_context.output.branch_name }}.json" + branch: "{{ steps.set_sync_context.output.branch_name }}" + message: "{{ consts.pr_title_prefix }} sync request {{ steps.set_sync_context.output.branch_name }}" + content: "{{ steps.set_sync_request.output.request_json }}" + on-failure: + continue: true + + - name: open_sync_pr + type: github.callTool + connector-id: "{{ consts.github_connector_id }}" + timeout: 120s + with: + name: create_pull_request + arguments: + owner: "{{ consts.github_owner }}" + repo: "{{ consts.github_repo_name }}" + title: "{{ consts.pr_title_prefix }} {{ steps.set_changed_rules.output.changed_count }} rule change(s) from space {{ consts.dev_space }} ({{ steps.set_sync_context.output.branch_name }})" + head: "{{ steps.set_sync_context.output.branch_name }}" + base: "{{ consts.github_base_branch }}" + body: | + Automated Detection-as-Code sync opened by the Elastic workflow **DaC - VCS Rule Sync** (execution `{{ execution.id }}`). + + Rules tagged `{{ consts.sync_tag }}` in the "{{ consts.dev_space }}" Kibana space changed since the last sync. The **DaC - VCS rule sync** GitHub Action now exports the tagged rules into this branch (detection-rules `kibana export-rules -e -ac`; existing files are overwritten so edits are included), commits them, runs the detection-rules unit tests, and comments the results here. + + Changed rule(s):{{ steps.set_changed_rules.output.rule_list_md }} + + Merging this PR accepts the rule state into version control. If the export produces no changes, the Action closes this PR automatically. + on-failure: + continue: true + + - name: if_pr_opened + type: if + condition: "not steps.open_sync_pr.error: *" + steps: + # Advance the watermark only after the PR exists, so a + # failed sync is retried on the next cycle. + - name: record_sync_state + type: elasticsearch.index + with: + index: "{{ consts.state_index }}" + document: + synced_at: "{{ steps.set_sync_context.output.sync_time }}" + synced_through: "{{ steps.set_sync_context.output.synced_through }}" + branch: "{{ steps.set_sync_context.output.branch_name }}" + rule_ids: "{{ steps.set_changed_rules.output.rule_ids }}" + on-failure: + continue: true + + - name: log_sync_opened + type: console + with: + message: 'VCS sync PR opened for {{ steps.set_changed_rules.output.changed_count }} changed rule(s): https://github.com/{{ consts.github_owner }}/{{ consts.github_repo_name }}/pulls (branch {{ steps.set_sync_context.output.branch_name }}). state_recorded={% if steps.record_sync_state.error %}no ({{ steps.record_sync_state.error }}){% else %}yes{% endif %}' + else: + - name: log_pr_failed + type: console + with: + message: "Failed to open the sync PR on {{ consts.github_owner }}/{{ consts.github_repo_name }}: {{ steps.open_sync_pr.error }} — commit_error={{ steps.commit_sync_request.error | default: 'none' }}. Check the GitHub connector ({{ consts.github_connector_id }}) permissions. Sync will retry next cycle." + else: + - name: log_branch_failed + type: console + with: + message: "Failed to create branch {{ steps.set_sync_context.output.branch_name }} on {{ consts.github_owner }}/{{ consts.github_repo_name }}: {{ steps.create_sync_branch.error }} — check the GitHub connector ({{ consts.github_connector_id }}). Sync will retry next cycle." + else: + - name: log_open_pr_exists + type: console + with: + message: '{{ steps.set_changed_rules.output.changed_count }} changed rule(s) detected, but a "{{ consts.pr_title_prefix }}" PR is already open on {{ consts.github_owner }}/{{ consts.github_repo_name }}. Skipping — the next cycle after it is merged or closed will pick the changes up.' + else: + - name: log_pr_search_failed + type: console + with: + message: "GitHub PR search failed: {{ steps.search_open_sync_pr.error }} — cannot verify whether a sync PR is already open, so skipping this cycle (will retry next run). Check the GitHub connector ({{ consts.github_connector_id }})." + else: + - name: log_no_changes + type: console + with: + message: '{{ steps.find_tagged_rules.output.total }} rule(s) tagged "{{ consts.sync_tag }}" in space "{{ consts.dev_space }}", but none changed since the last sync ({{ steps.set_sync_context.output.last_sync | default: "never" }}). Nothing to do.' + else: + - name: log_none_tagged + type: console + with: + message: 'No rules tagged "{{ consts.sync_tag }}" found in space "{{ consts.dev_space }}" — nothing to sync.' + + - name: log_done + type: console + with: + message: "DaC VCS rule sync check finished. execution.id={{ execution.id }}" diff --git a/examples/security/detection/dac-vcs-rule-sync/github/dac-vcs-rule-sync.yml b/examples/security/detection/dac-vcs-rule-sync/github/dac-vcs-rule-sync.yml new file mode 100644 index 0000000..871fdc8 --- /dev/null +++ b/examples/security/detection/dac-vcs-rule-sync/github/dac-vcs-rule-sync.yml @@ -0,0 +1,205 @@ +# DaC VCS rule sync — companion to the Elastic workflow "DaC - VCS Rule Sync". +# +# Install this file in your detection-rules FORK at: +# .github/workflows/dac-vcs-rule-sync.yml +# +# The Elastic workflow (via the Kibana GitHub connector) creates a +# `vcs-sync-` branch, commits a sync-request file under .vcs-sync/, +# and opens a PR against main. That request commit triggers this workflow, +# which: +# 1. exports the rules tagged with the sync tag (default "vcs") from the dev +# Kibana space into the fork's custom rules directory on this branch +# (kibana export-rules -e -ac; existing TOML files are overwritten by +# default, so rule edits are picked up), +# 2. commits and pushes the exported rules to the sync branch (the PR diff +# becomes the actual rule changes), +# 3. runs the detection-rules unit tests and comments the result on the PR +# (a test failure also fails this run, showing red on the PR). +# +# If the export produces no file changes, the sync PR is closed automatically. +# +# Loop safety: commits made here use the built-in GITHUB_TOKEN (which does not +# trigger workflows), and the `paths` filter only matches the sync-request +# files the Elastic workflow writes — the rule commits this workflow pushes +# can never re-trigger it. +# +# Required repository secrets (Settings > Secrets and variables > Actions): +# Either: +# DR_CLOUD_ID Elastic Cloud ID (deployment overview page in the Cloud console) +# or: +# DR_KIBANA_URL e.g. https://my-deployment.kb.us-east-1.aws.elastic.cloud +# And always: +# DR_API_KEY base64 Elasticsearch API key that can read rules in the dev space +# Do not set DR_CLOUD_ID together with DR_KIBANA_URL — the detection-rules +# client rejects a cloud id combined with a mismatching Kibana URL. +# +# Recommended: commit a custom rules directory to the fork once +# (`python -m detection_rules custom-rules setup-config dac-vcs-rules`) +# so its _config.yaml / test config are version-controlled. If it is missing, +# this workflow bootstraps a default one for the run. + +name: DaC - VCS rule sync + +on: + push: + branches: + - "vcs-sync-*" + paths: + - ".vcs-sync/**" + +permissions: + contents: write + pull-requests: write + +env: + DR_CLOUD_ID: ${{ secrets.DR_CLOUD_ID }} + DR_KIBANA_URL: ${{ secrets.DR_KIBANA_URL }} + DR_API_KEY: ${{ secrets.DR_API_KEY }} + CUSTOM_RULES_DIR: ${{ github.workspace }}/dac-vcs-rules + GH_TOKEN: ${{ github.token }} + +jobs: + sync: + runs-on: ubuntu-latest + + steps: + - name: Validate connection settings + run: | + if [ -z "${DR_API_KEY}" ]; then + echo "::error::DR_API_KEY secret is not set." + exit 1 + fi + if [ -n "${DR_CLOUD_ID}" ] && [ -n "${DR_KIBANA_URL}" ]; then + echo "::error::Set either DR_CLOUD_ID or DR_KIBANA_URL, not both." + exit 1 + fi + if [ -z "${DR_CLOUD_ID}" ] && [ -z "${DR_KIBANA_URL}" ]; then + echo "::error::Set the DR_CLOUD_ID secret, or DR_KIBANA_URL." + exit 1 + fi + + - name: Checkout sync branch + uses: actions/checkout@v4 + + # The Elastic workflow commits one request file per sync, named after + # the branch. It carries the space and tag so the fork needs no + # duplicated configuration. + - name: Read sync request + run: | + request=".vcs-sync/${GITHUB_REF_NAME}.json" + if [ ! -f "${request}" ]; then + echo "::error::Sync request file ${request} not found on this branch." + exit 1 + fi + echo "DEV_SPACE=$(jq -r '.dev_space' "${request}")" >> "${GITHUB_ENV}" + echo "SYNC_TAG=$(jq -r '.sync_tag' "${request}")" >> "${GITHUB_ENV}" + jq . "${request}" + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: "pip" + + # Install the package first, then overlay the fork's local lib copies in + # separate pip runs (mirrors upstream pythonpackage.yml). Combining them + # in one command makes pip resolve the local lib/kql path against the + # git-URL pin in pyproject.toml — two direct references for the same + # package — which fails with ResolutionImpossible. + - name: Install detection-rules + run: | + python -m pip install --upgrade pip + pip install ".[dev]" + pip install lib/kibana + pip install lib/kql + + # The CLI refuses to start when CUSTOM_RULES_DIR points at a directory + # without _config.yaml, so the bootstrap itself must run with it unset. + - name: Bootstrap custom rules directory (first run only) + run: | + if [ ! -f "${CUSTOM_RULES_DIR}/_config.yaml" ]; then + echo "No committed custom rules directory found — creating a default one." + dir="${CUSTOM_RULES_DIR}" + env -u CUSTOM_RULES_DIR python -m detection_rules custom-rules setup-config "${dir}" + fi + + # export-rules overwrites existing TOML files by default, so edited + # rules are updated in place — that's what makes this a sync. + - name: Export tagged rules from the dev space + id: export + run: | + python -m detection_rules kibana --space "${DEV_SPACE}" export-rules \ + --directory "${CUSTOM_RULES_DIR}/rules" \ + --exceptions-directory "${CUSTOM_RULES_DIR}/exceptions" \ + --action-connectors-directory "${CUSTOM_RULES_DIR}/action_connectors" \ + --export-exceptions \ + --export-action-connectors \ + --export-query "alert.attributes.tags: \"${SYNC_TAG}\"" \ + --custom-rules-only \ + --strip-version \ + --skip-errors + + - name: Commit exported rules to the sync branch + id: commit + run: | + git config user.name "github-actions[bot]" + git config user.email "41898994+github-actions[bot]@users.noreply.github.com" + git add "${CUSTOM_RULES_DIR}" + if git diff --cached --quiet; then + echo "changed=false" >> "${GITHUB_OUTPUT}" + echo "Export produced no rule changes — closing the sync PR." + gh pr close "${GITHUB_REF_NAME}" --delete-branch \ + --comment "Automated VCS sync: the export produced no changes (the tagged rules are already in sync with this repository). Closing." + else + echo "changed=true" >> "${GITHUB_OUTPUT}" + git commit -m "[VCS Sync] Export '${SYNC_TAG}' tagged rules from space '${DEV_SPACE}'" + git push origin "HEAD:${GITHUB_REF_NAME}" + fi + + - name: Run unit tests + id: unit_tests + if: steps.commit.outputs.changed == 'true' + run: | + set -o pipefail + python -m detection_rules test 2>&1 | tee "${RUNNER_TEMP}/unit-tests.log" + + - name: Comment test results on the PR + if: always() && steps.commit.outputs.changed == 'true' + env: + TESTS_OUTCOME: ${{ steps.unit_tests.outcome }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + { + if [ "${TESTS_OUTCOME}" = "success" ]; then + echo "✅ **VCS sync complete — unit tests passed.**" + echo + echo "The \`${SYNC_TAG}\` tagged rules from space \`${DEV_SPACE}\` were exported into this branch and the detection-rules unit tests passed. Review the diff and merge to accept the rules into version control." + else + echo "❌ **VCS sync: unit tests failed.**" + echo + echo "The rules were exported and committed so the diff is reviewable, but the detection-rules unit tests failed — fix the rule(s) in the \`${DEV_SPACE}\` space and re-sync, or push a fix to this branch." + echo + echo "
Unit test output (tail)" + echo + echo '```' + tail -c 8000 "${RUNNER_TEMP}/unit-tests.log" 2>/dev/null || echo "no unit test output captured" + echo '```' + echo + echo "
" + fi + echo + echo "[Workflow run](${RUN_URL})" + } > "${RUNNER_TEMP}/pr-comment.md" + gh pr comment "${GITHUB_REF_NAME}" --body-file "${RUNNER_TEMP}/pr-comment.md" + + - name: Upload exported rules and test log + if: always() + uses: actions/upload-artifact@v4 + with: + name: dac-vcs-sync-${{ github.run_id }} + path: | + ${{ env.CUSTOM_RULES_DIR }}/rules + ${{ env.CUSTOM_RULES_DIR }}/exceptions + ${{ env.CUSTOM_RULES_DIR }}/action_connectors + ${{ runner.temp }}/unit-tests.log + if-no-files-found: ignore