[WIP] Introduce the ability to provision SSCSI roles on hubs and spokes when needed

.github/linters/.markdownlint.json

@@ -1,6 +1,13 @@
 {
-    "default": true,
-    "MD003": false,
-    "MD013": false,
-    "MD033": false
-}
+  "default": true,
+  "MD003": false,
+  "MD013": {
+    "line_length": 400,
+    "code_blocks": false,
+    "tables": false
+  },
+  "MD033": false,
+  "MD060": {
+    "style": "compact"
+  }
+}

Makefile

@@ -8,6 +8,7 @@ help: ## This help message
 .PHONY: super-linter
 super-linter: ## Runs super linter locally
 	rm -rf .mypy_cache
+	rm -rf .ansible
 	podman run -e RUN_LOCAL=true -e USE_FIND_ALGORITHM=true	\
 					-e VALIDATE_ANSIBLE=false \
 					-e VALIDATE_BASH=false \

README.md

@@ -8,3 +8,85 @@ The main purpose of this collections are to:
 loading local secrets files into VP secrets stores.
 
 2. Help manage imperative and other utility functions of the cluster
+
+## SS CSI workload auth notes
+
+SS CSI task files live in **`roles/vault_utils/tasks/ss_csi/`**; paths below match **`include_tasks`** from the role (**`ss_csi/<file>`** relative to **`tasks/`**).
+
+`vault_utils` can read `ssCsiWorkloadAuth` entries from clustergroup values and
+create Vault Kubernetes auth roles for hub and spoke workloads.
+
+### Parsing (load YAML)
+
+With **`vault_ss_csi_aggregate_clustergroup_sources`** true (default), SS CSI
+uses the **`clustergroup_discovery`** role to determine stems: **main** from
+`values-global.yaml`, then **managed** names from `clusterGroup.managedClusterGroups`
+in the main `values-<main>.yaml|yml`. For **each** stem it loads a document from
+the in-cluster **`ConfigMap` `values-<stem>`** (namespace
+`openshift-gitops` by default), then falls back to **`pattern_dir/values-<stem>.yaml|yml`**
+when enabled. ConfigMap data keys follow **`vault_ss_csi_clustergroup_configmap_key`**
+and **`vault_ss_csi_clustergroup_configmap_key_candidates`**. Each document must
+include **`clusterGroup`**. Stems are merged in **`clustergroup_load_order`**
+(main first, then managed stems sorted) so later sources override duplicate
+`clusterGroup.applications` keys. Set **`vault_ss_csi_aggregate_clustergroup_sources`**
+to false to load only the **main** document (legacy: single ConfigMap or
+`values-<main>.yaml`).
+
+### Extraction (find `ssCsiWorkloadAuth`)
+
+The role builds **`_vault_ss_csi_apps_by_stem`** (per-stem `clusterGroup.applications`)
+and a merged **`clusterGroup.managedClusterGroups`**. It collects:
+
+- **`clusterGroup.applications.*.ssCsiWorkloadAuth`** — per stem; omit **`cluster`**
+  in values: the **main** stem resolves to **hub**; **managed** stems resolve to
+  that **stem name** so entries under `values-<managed>.yaml` stay spoke-scoped.
+- **`clusterGroup.managedClusterGroups.*.applications.*.ssCsiWorkloadAuth`** —
+  from the merged map; omit **`cluster`** and the row targets that managed group
+  (**`name`**, else the group map key).
+
+### Projection (Vault roles)
+
+Rows are appended to **`_ss_csi_all_entries`**, split into hub vs spoke using
+the computed **`cluster`** field (from stem or managed group when omitted in YAML), then **hub** identities get Vault Kubernetes
+auth roles via **`ss_csi/vault_ss_csi_apply_one_hub_sscsi_role.yaml`**. Spoke rows are
+normalized to **`vault_path`** later in the play (**`ss_csi/vault_ss_csi_normalize_spoke_entries_to_vault_path.yaml`**
+during **`vault_spokes_init`**) and roles are written on each spoke mount
+(**`ss_csi/vault_ss_csi_apply_one_spoke_sscsi_role.yaml`**). Role names use
+**`<mount>-sscsi-<slug>`**; slugs come from **`ss_csi/vault_ss_csi_compute_role_slug.yaml`**.
+
+To **inspect** stems and files locally, run **`playbooks/list_clustergroups.yml`**
+or **`playbooks/parse_clustergroup_values.yml`** (see **`roles/clustergroup_discovery/README.md`**).
+
+At the application level (`clusterGroup.applications.<app>`), the relevant
+inputs are:
+
+- `ssCsiWorkloadAuth` (list)
+- `ssCsiWorkloadAuth[].serviceAccount` (required)
+- `ssCsiWorkloadAuth[].namespace` (optional)
+- Omit **`cluster`** in pattern YAML; hub vs spoke comes from **which file or
+  `managedClusterGroups` branch** defines the list (see extraction above). Spoke
+  handling still normalizes to **`vault_path`** (full DNS), same as External Secrets.
+- `ssCsiWorkloadAuth[].roleSlug` / `role_slug` (optional): suffix only; Vault
+  role is **`<mount>-sscsi-<slug>`** where **`<mount>`** is hub **`hub`** (or
+  configured hub path) or the spoke **`vault_path`**. When using the
+  **vp-sscsi-spc** chart, `spec.parameters.roleName` uses the same **mount**
+  as `vaultKubernetesMountPath` (typically **`global.clusterDomain`** on
+  spokes), not a short clustergroup label.
+- application `namespace` (optional default for entry namespace)
+
+CA material management for SS CSI is not handled in this collection anymore.
+Provide CA distribution using a separate chart or platform mechanism.
+
+For the complete flow and task ordering, see
+`secrets-initialization-and-vault-unseal.md`.
+
+## Pattern repository directory (`pattern_dir`)
+
+Playbooks need the path to your pattern Git checkout (where `values-global.yaml`
+and related files live). Resolution order: extra var `pattern_dir`, environment
+variable `PATTERN_DIR`, then `PWD` and `pwd`.
+
+When running from the imperative container or another fixed working directory,
+pass the repository root explicitly, for example `-e pattern_dir=/git/repo` (or add
+equivalent extra vars via `clusterGroup.imperative.extraPlaybookArgs` in the
+clustergroup chart).

playbooks/list_clustergroups.yml

@@ -0,0 +1,21 @@
+---
+# Discover values-<clustergroup>.yaml|yml under pattern_dir.
+# Resolves pattern_dir like pattern_settings (extra var pattern_dir, env PATTERN_DIR, cwd).
+- name: List pattern clustergroup value stems
+  hosts: localhost
+  connection: local
+  gather_facts: false
+  become: false
+  roles:
+    - pattern_settings
+    - role: clustergroup_discovery
+  tasks:
+    - name: Report clustergroup discovery
+      ansible.builtin.debug:
+        msg:
+          pattern_dir: "{{ pattern_dir }}"
+          main_clustergroup: "{{ main_clustergroup }}"
+          managed_clustergroup_names: "{{ managed_clustergroup_names }}"
+          clustergroup_names: "{{ clustergroup_names }}"
+          clustergroup_load_order: "{{ clustergroup_load_order }}"
+          clustergroup_file_entries: "{{ clustergroup_file_entries }}"

playbooks/parse_clustergroup_values.yml

@@ -0,0 +1,22 @@
+---
+# Parse every top-level values-<clustergroup>.yaml|yml into clustergroup_documents (stem -> root).
+# Use for migration tooling or inspection; SS CSI merge uses the same discovery role internally.
+- name: Parse pattern clustergroup values files
+  hosts: localhost
+  connection: local
+  gather_facts: false
+  become: false
+  roles:
+    - pattern_settings
+    - role: clustergroup_discovery
+      vars:
+        clustergroup_discovery_parse_documents: true
+  tasks:
+    - name: Summarize parsed clustergroup documents
+      ansible.builtin.debug:
+        msg:
+          pattern_dir: "{{ pattern_dir }}"
+          main_clustergroup: "{{ main_clustergroup }}"
+          managed_clustergroup_names: "{{ managed_clustergroup_names }}"
+          stems_parsed: "{{ clustergroup_documents | default({}) | dict2items | map(attribute='key') | sort | list }}"
+          document_count: "{{ clustergroup_documents | default({}) | length }}"

playbooks/vault.yml

@@ -4,6 +4,9 @@
   connection: local
   gather_facts: false
   roles:
+   # Resolves pattern_dir (extra var / PATTERN_DIR / PWD) and loads main.clusterGroupName as main_clustergroup.
+   # vault_ss_csi_workload_auth prefers merged clustergroup YAML from an in-cluster ConfigMap, then file fallback.
+   - pattern_settings
    - find_vp_secrets
    - cluster_pre_check
    - vault_utils

roles/clustergroup_discovery/README.md

@@ -0,0 +1,27 @@
+# clustergroup_discovery
+
+Ansible role that lists **which clustergroup value stems are in use** for a Validated Patterns checkout, without scanning every `values-*.yaml` on disk.
+
+## Behavior
+
+1. Resolve **`pattern_dir`** the same way as `pattern_settings` (extra var, `PATTERN_DIR`, then `PWD` / `pwd`).
+2. Read **`main.clusterGroupName`** from `values-global.yaml` under `pattern_dir` (or use `main_clustergroup` / `main_clustergroupname` if the play already set them).
+3. Load **`values-<main>.yaml`** or **`values-<main>.yml`** and read **`clusterGroup.managedClusterGroups`**. For each entry, the managed name is **`value.name`** if set, otherwise the **YAML key** (same rule as SS CSI managed-cluster-group defaults).
+4. Expose facts:
+   - **`managed_clustergroup_names`** — sorted unique managed names
+   - **`clustergroup_load_order`** — `[main, …managed]` (main first; used when merging so later stems override duplicate `applications` keys)
+   - **`clustergroup_names`** — sorted list of all stems (main + managed)
+   - **`clustergroup_file_entries`** — `{name, path}` only for stems where a local `values-<stem>.yaml|yml` exists
+
+Optional: set **`clustergroup_discovery_parse_documents: true`** to fill **`clustergroup_documents`** (`<stem>` → parsed YAML root) for each file in `clustergroup_file_entries`.
+
+## Playbooks
+
+- `playbooks/list_clustergroups.yml` — runs `pattern_settings` + this role and prints the facts above.
+- `playbooks/parse_clustergroup_values.yml` — same with parsing enabled.
+
+Requires `ANSIBLE_ROLES_PATH` (or collection layout) so `pattern_settings` and this role resolve.
+
+## Relation to SS CSI
+
+`vault_utils` includes this role when **`vault_ss_csi_aggregate_clustergroup_sources`** is true (default): SS CSI then loads and merges **one document per stem** in `clustergroup_load_order`. See `roles/vault_utils/README.md` (SS CSI section) for parsing, extraction, and projection.

roles/clustergroup_discovery/defaults/main.yml

@@ -0,0 +1,3 @@
+---
+# When true, slurp and parse each resolved clustergroup file into clustergroup_documents (stem -> root mapping)
+clustergroup_discovery_parse_documents: false

roles/clustergroup_discovery/meta/main.yml

@@ -0,0 +1,12 @@
+---
+galaxy_info:
+  author: rhvp
+  description: >-
+    Resolve main clustergroup from values-global, read managedClusterGroups from the main
+    values file, then optionally parse existing values-<stem> files for those stems.
+  license: Apache-2.0
+  min_ansible_version: "2.14"
+  galaxy_tags:
+    - openshift
+    - gitops
+dependencies: []

roles/clustergroup_discovery/tasks/main.yml

@@ -0,0 +1,118 @@
+---
+# Discover clustergroups in use: main from values-global, managed from main file's clusterGroup.managedClusterGroups.
+# Sets: clustergroup_names (sorted stems), managed_clustergroup_names (sorted, excludes main),
+#       clustergroup_load_order (main first, then managed sorted — SS CSI merge precedence),
+#       clustergroup_file_entries ({name, path} only when values-<stem>.yaml|yml exists),
+#       clustergroup_documents (optional, stem -> parsed YAML root).
+
+- name: Resolve pattern_dir for clustergroup discovery
+  ansible.builtin.include_tasks: ../pattern_settings/tasks/resolve_overrides.yml
+  when: (pattern_dir | default('', true) | string | trim | length) == 0
+
+- name: Fail when pattern_dir is empty after resolve
+  ansible.builtin.fail:
+    msg: >-
+      pattern_dir is required (extra var pattern_dir, env PATTERN_DIR, or cwd with values-global.yaml).
+  when: (pattern_dir | default('', true) | string | trim | length) == 0
+
+- name: Resolve main clustergroup stem from facts or values-global.yaml
+  ansible.builtin.set_fact:
+    _clustergroup_discovery_main_stem: >-
+      {{
+        (
+          (main_clustergroupname | default(main_clustergroup | default('', true), true) | string | trim | length) > 0
+        )
+        | ternary(
+            main_clustergroupname | default(main_clustergroup, true) | string | trim,
+            (
+              lookup('file', (pattern_dir | string | trim) ~ '/values-global.yaml')
+              | from_yaml
+            ).main.clusterGroupName | string | trim
+          )
+      }}
+
+- name: Fail when main clusterGroupName cannot be resolved
+  ansible.builtin.fail:
+    msg: >-
+      Could not resolve main clustergroup (values-global.yaml missing .main.clusterGroupName or empty).
+  when: (_clustergroup_discovery_main_stem | string | trim | length) == 0
+
+- name: Stat main clustergroup values file (yaml)
+  ansible.builtin.stat:
+    path: "{{ pattern_dir | string | trim }}/values-{{ _clustergroup_discovery_main_stem }}.yaml"
+  register: _clustergroup_discovery_main_stat_yaml
+
+- name: Stat main clustergroup values file (yml)
+  ansible.builtin.stat:
+    path: "{{ pattern_dir | string | trim }}/values-{{ _clustergroup_discovery_main_stem }}.yml"
+  register: _clustergroup_discovery_main_stat_yml
+  when: not (_clustergroup_discovery_main_stat_yaml.stat.exists | default(false))
+
+- name: Set path to main clustergroup values file when present
+  ansible.builtin.set_fact:
+    _clustergroup_main_values_path: "{{ pattern_dir | string | trim }}/values-{{ _clustergroup_discovery_main_stem }}.yaml"
+  when: _clustergroup_discovery_main_stat_yaml.stat.exists | default(false)
+
+- name: Set path to main clustergroup values file when only yml exists
+  ansible.builtin.set_fact:
+    _clustergroup_main_values_path: "{{ pattern_dir | string | trim }}/values-{{ _clustergroup_discovery_main_stem }}.yml"
+  when:
+    - _clustergroup_main_values_path is not defined
+    - _clustergroup_discovery_main_stat_yml is defined
+    - _clustergroup_discovery_main_stat_yml.stat.exists | default(false)
+
+- name: Load parsed root from main clustergroup values file
+  ansible.builtin.set_fact:
+    _clustergroup_main_root: "{{ lookup('file', _clustergroup_main_values_path) | from_yaml }}"
+  when: _clustergroup_main_values_path is defined
+
+- name: Default empty main clustergroup root when file is absent
+  ansible.builtin.set_fact:
+    _clustergroup_main_root: {}
+  when: _clustergroup_main_values_path is not defined
+
+- name: Collect managed clustergroup names from main file managedClusterGroups
+  ansible.builtin.set_fact:
+    managed_clustergroup_names: "{{ managed_clustergroup_names | default([]) + [_cgd_mcg_name] }}"
+  vars:
+    _cgd_mcg_name: "{{ (item.value.name | default(item.key, true)) | string | trim }}"
+  loop: "{{ (_clustergroup_main_root.clusterGroup | default({})).managedClusterGroups | default({}) | dict2items }}"
+  loop_control:
+    label: "{{ _cgd_mcg_name }}"
+  when:
+    - _clustergroup_main_root is mapping
+    - (_clustergroup_main_root.clusterGroup | default({})).managedClusterGroups is defined
+    - ((_clustergroup_main_root.clusterGroup | default({})).managedClusterGroups | default({})) is mapping
+
+- name: Finalize managed clustergroup names list
+  ansible.builtin.set_fact:
+    managed_clustergroup_names: "{{ managed_clustergroup_names | default([]) | unique | sort }}"
+
+- name: Set clustergroup load order (main first so managed values files override for SS CSI merge)
+  ansible.builtin.set_fact:
+    clustergroup_load_order: >-
+      {{
+        (
+          [_clustergroup_discovery_main_stem]
+          + (managed_clustergroup_names | reject('equalto', _clustergroup_discovery_main_stem) | list)
+        ) | unique | list
+      }}
+
+- name: Set sorted clustergroup names (all stems in use)
+  ansible.builtin.set_fact:
+    clustergroup_names: "{{ clustergroup_load_order | sort }}"
+
+- name: Build clustergroup_file_entries for stems that have a local values file
+  ansible.builtin.include_tasks: resolve_clustergroup_file_path.yml
+  loop: "{{ clustergroup_load_order }}"
+  loop_control:
+    loop_var: clustergroup_discovery_stem
+
+- name: Default empty clustergroup file entries
+  ansible.builtin.set_fact:
+    clustergroup_file_entries: []
+  when: clustergroup_file_entries is not defined
+
+- name: Parse each resolved clustergroup values file when requested
+  ansible.builtin.include_tasks: parse_documents.yml
+  when: clustergroup_discovery_parse_documents | default(false) | bool

roles/clustergroup_discovery/tasks/parse_documents.yml

@@ -0,0 +1,7 @@
+---
+- name: Parse clustergroup values YAML into clustergroup_documents
+  ansible.builtin.set_fact:
+    clustergroup_documents: "{{ clustergroup_documents | default({}) | combine({item.name: (lookup('file', item.path) | from_yaml)}) }}"
+  loop: "{{ clustergroup_file_entries }}"
+  loop_control:
+    label: "{{ item.name }}"

roles/clustergroup_discovery/tasks/resolve_clustergroup_file_path.yml

@@ -0,0 +1,32 @@
+---
+# loop_var: clustergroup_discovery_stem — append {name, path} to clustergroup_file_entries when file exists.
+
+- name: Stat values file for stem {{ clustergroup_discovery_stem }} (yaml)
+  ansible.builtin.stat:
+    path: "{{ pattern_dir | string | trim }}/values-{{ clustergroup_discovery_stem | string | trim }}.yaml"
+  register: _clustergroup_discovery_stem_stat_yaml
+
+- name: Stat values file for stem {{ clustergroup_discovery_stem }} (yml)
+  ansible.builtin.stat:
+    path: "{{ pattern_dir | string | trim }}/values-{{ clustergroup_discovery_stem | string | trim }}.yml"
+  register: _clustergroup_discovery_stem_stat_yml
+
+- name: Record clustergroup file entry for {{ clustergroup_discovery_stem }} (prefer yaml)
+  ansible.builtin.set_fact:
+    clustergroup_file_entries: "{{ clustergroup_file_entries | default([]) + [_entry] }}"
+  vars:
+    _entry:
+      name: "{{ clustergroup_discovery_stem | string | trim }}"
+      path: "{{ pattern_dir | string | trim }}/values-{{ clustergroup_discovery_stem | string | trim }}.yaml"
+  when: _clustergroup_discovery_stem_stat_yaml.stat.exists | default(false)
+
+- name: Record clustergroup file entry for {{ clustergroup_discovery_stem }} (yml fallback)
+  ansible.builtin.set_fact:
+    clustergroup_file_entries: "{{ clustergroup_file_entries | default([]) + [_entry] }}"
+  vars:
+    _entry:
+      name: "{{ clustergroup_discovery_stem | string | trim }}"
+      path: "{{ pattern_dir | string | trim }}/values-{{ clustergroup_discovery_stem | string | trim }}.yml"
+  when:
+    - not (_clustergroup_discovery_stem_stat_yaml.stat.exists | default(false))
+    - _clustergroup_discovery_stem_stat_yml.stat.exists | default(false)