Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/changes/changes_10.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ This release fixes vulnerabilities by updating dependencies:
* #699: Added `all-extras` support to the Python environment GitHub action
* #875: Added `name` attribute to generated workflow jobs using `-extension.yml` workflows

## Bug
## Bug Fix

* #744: Updated nox DB-version handling to use `BaseConfig.minimum_exasol_version` instead hardcoded `7.1.9`

Expand Down Expand Up @@ -80,4 +80,4 @@ This release fixes vulnerabilities by updating dependencies:
* Updated dependency `pip-audit:2.10.0` to `2.10.1`
* Updated dependency `pylint:4.0.5` to `4.0.6`
* Updated dependency `pytest:9.0.3` to `9.1.1`
* Updated dependency `zizmor:1.25.2` to `1.26.1`
* Updated dependency `zizmor:1.25.2` to `1.26.1`
2 changes: 1 addition & 1 deletion doc/changes/changes_10.2.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This minor release adds automated vulnerability updates through Nox session
`vulnerabilities:update` and improves the `dependency-update.yml`. It also includes a
few workflow-related bug fixes and documentation updates.

## Bug
## Bug Fix

* #909: Updated `cd.yml` workflow so that `cd-extension.yml` workflow depends on `build-and-publish`. This ensures that the custom release workflow only runs when the PyPi release was successful.
* #910: Added `gh-pages.yml` to be ignored when `has_documentation=False` in the `PROJECT_CONFIG`
Expand Down
2 changes: 0 additions & 2 deletions doc/changes/changes_10.2.1.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
# 10.2.1 - 2026-07-08

## Summary

## Bug Fix

* #920: Ensured extracted secrets are unique and alphabetically sorted from the custom workflows
4 changes: 4 additions & 0 deletions doc/changes/unreleased.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Unreleased

## Summary

## Feature

* #922:
48 changes: 45 additions & 3 deletions doc/user_guide/features/github_workflows/workflow_variables.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,20 @@ standardized baseline that can be overridden in individual projects.
:start-at: github_template_dict
:end-before: @computed_field

.. _custom_workflow_metadata:

Custom Workflow Metadata
^^^^^^^^^^^^^^^^^^^^^^^^

The PTB extracts metadata from reusable custom workflow files and exposes it
through :py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict` under the
``custom_workflows`` entry. PTB-controlled workflow templates use that metadata
when they call reusable workflows.

.. _custom_workflow_secrets:

Custom Workflow Secrets
^^^^^^^^^^^^^^^^^^^^^^^
Secrets
-------

The PTB extracts secret names from reusable custom workflow files and exposes them
through :py:attr:`exasol.toolbox.config.BaseConfig.github_template_dict` under the
Expand Down Expand Up @@ -50,6 +60,38 @@ For example, ``slow-checks.yml`` can define its reusable workflow header like th
Those extracted secret names are then made available to the PTB templates that
reference the custom workflow.

.. _custom_workflow_permissions:

Permissions
-----------

The PTB extracts the effective GitHub permissions required by reusable custom
workflow files. It scans every job in the workflow, reads each job's
``permissions`` block, and combines the results into a single ordered mapping.

When multiple jobs request the same permission, the most permissive level wins
while the first-seen key order is preserved. For example, if one job requests
``contents: read`` and another later job requests ``contents: write``, the final
mapping keeps ``contents`` once, with ``write`` as the level.

Please only configure the minimum required permissions and granting the least required
access. In practice, ``contents: read`` is the most common baseline for workflows, and
other permissions should only be added when a particular step truly requires them.

For example, a custom workflow can declare permissions like this:

.. code-block:: yaml

name: Slow-Checks

on:
workflow_call:

jobs:
run-integration-tests:
permissions:
contents: read

.. _workflow_matrix:

Matrix Combinations
Expand All @@ -64,7 +106,7 @@ Extending the Matrix
^^^^^^^^^^^^^^^^^^^^

If you need to expose additional values via the ``matrix.yml``, you can extend
:class:`exasol.toolbox.config.BaseConfig`.
:class:`exasol.toolbox.config.BaseConfig`.

The example adds two additional matrix dimensions: A declared one
`extra_matrix_value` and a computed one `computed_matrix_value`. Each of them
Expand Down
6 changes: 5 additions & 1 deletion exasol/toolbox/templates/github/workflows/cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["cd-extension"].permissions %)
permissions:
contents: write
(% for permission_name, permission_level in custom_workflows["cd-extension"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)
(% endif %)

(% if has_documentation %)
Expand Down
6 changes: 5 additions & 1 deletion exasol/toolbox/templates/github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["merge-gate"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["merge-gate"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)

report:
name: Report
Expand Down
6 changes: 5 additions & 1 deletion exasol/toolbox/templates/github/workflows/fast-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["fast-tests-extension"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["fast-tests-extension"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)
(% endif %)
12 changes: 10 additions & 2 deletions exasol/toolbox/templates/github/workflows/merge-gate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["slow-checks"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["slow-checks"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)

(% if custom_workflows["merge-gate-extension"].exists %)
merge-gate-extension:
Expand All @@ -72,8 +76,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["merge-gate-extension"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["merge-gate-extension"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)
(% endif %)

# This job ensures inputs have been executed successfully.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,12 @@ jobs:
(( secret_name )): ${{ secrets.(( secret_name )) }}
(% endfor %)
(% endif %)
(% if custom_workflows["slow-checks"].permissions %)
permissions:
contents: read
(% for permission_name, permission_level in custom_workflows["slow-checks"].permissions.items() %)
(( permission_name )): (( permission_level ))
(% endfor %)
(% endif %)

report:
name: Report
Expand Down
59 changes: 58 additions & 1 deletion exasol/toolbox/util/workflows/custom_workflow.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
from __future__ import annotations

from enum import IntEnum
from pathlib import Path

from pydantic import (
Expand All @@ -13,12 +14,43 @@
from exasol.toolbox.util.workflows.render_yaml import parse_yaml_text


class PermissionRank(IntEnum):
"""GitHub permission levels ranked from least to most permissive."""

NONE = 0
READ = 1
WRITE = 2

@classmethod
def _missing_(cls, value: object) -> PermissionRank:
"""Convert a GitHub permission string to its rank."""
if isinstance(value, str):
for rank in cls:
if rank.name.lower() == value:
return rank
raise ValueError(f"Unknown GitHub permission level: {value!r}")


def merge_permissions(permission_maps: list[dict[str, str]]) -> dict[str, str]:
"""Merge permission maps to keep the greater permission."""
merged_permissions: dict[str, str] = {}
for permission_map in permission_maps:
for permission_name, requested_level in permission_map.items():
current_level = merged_permissions.get(permission_name, None)
if current_level is None:
merged_permissions[permission_name] = requested_level
continue
if PermissionRank(requested_level) > PermissionRank(current_level): # type: ignore[arg-type]
merged_permissions[permission_name] = requested_level
return merged_permissions


class CustomWorkflow(BaseModel):
"""A project-owned workflow used for seeded workflows and extensions.

These workflows are seeded by the PTB or extend PTB-provided workflows, but
they are maintained by the project itself rather than the PTB. See
`Custom Workflows <file:///home/chku/git/ptb/.html-documentation/user_guide/features/github_workflows/index.html#custom-workflows>`__.
`Custom Workflows <https://exasol.github.io/python-toolbox/main/user_guide/features/github_workflows/index.html#custom-workflows>`__.
"""

model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True)
Expand Down Expand Up @@ -53,3 +85,28 @@ def extract_secrets(self) -> tuple[str, ...]:
if secrets := workflow_call.get("secrets", {}):
return tuple(secrets.keys())
return ()

def extract_permissions(self) -> dict[str, str]:
"""Return the effective job permissions required by the workflow.

The extractor scans all jobs and merges their ``permissions`` blocks into a
single mapping. When the same permission appears multiple times, the more
permissive level wins while preserving the first-seen order of the keys.

For example, a custom workflow can declare permissions like this:

.. code-block:: yaml

name: Slow-Checks

on:
workflow_call:

jobs:
run-integration-tests:
permissions:
contents: read
"""
jobs = self.yaml_content.get("jobs", {})
permission_maps = [job.get("permissions", {}) for job in jobs.values()]
return merge_permissions(permission_maps)
28 changes: 26 additions & 2 deletions exasol/toolbox/util/workflows/custom_workflow_extractor.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,35 @@
field_validator,
)

from exasol.toolbox.util.workflows.custom_workflow import CustomWorkflow
from exasol.toolbox.util.workflows.custom_workflow import (
CustomWorkflow,
merge_permissions,
)

MINIMUM_GITHUB_PERMISSIONS: dict[str, str] = {"contents": "read"}


class CustomWorkflowEntry(BaseModel):
model_config = ConfigDict(frozen=True)

exists: bool
secrets: tuple[str, ...]
permissions: dict[str, str]

@field_validator("secrets", mode="before")
@classmethod
def _normalize_secrets(cls, secrets: tuple[str, ...]) -> list[str]:
"""Return unique secret names in alphabetical order."""
return sorted(set(secrets))

@field_validator("permissions", mode="before")
@classmethod
def _normalize_permissions(
cls, permissions: list[dict[str, str]]
) -> dict[str, str]:
"""Merge permission maps while preserving first-seen order."""
return merge_permissions(permissions)


class CustomWorkflowExtractor(BaseModel):
model_config = ConfigDict(frozen=True)
Expand All @@ -43,13 +57,18 @@ def _build_custom_workflow_entry(
file_path = self.github_workflow_directory / f"{workflow}.yml"

secrets: tuple[str, ...] = ()
permissions: list[dict[str, str]] = []
if file_path.is_file():
custom_workflow = CustomWorkflow.load_from_file(file_path=file_path)
secrets = custom_workflow.extract_secrets()

permissions = [
MINIMUM_GITHUB_PERMISSIONS,
custom_workflow.extract_permissions(),
]
return CustomWorkflowEntry(
exists=file_path.exists(),
secrets=secrets,
permissions=permissions,
)

def _build_merge_gate_entry(
Expand All @@ -61,6 +80,11 @@ def _build_merge_gate_entry(
+ custom_workflows_dict["slow-checks"].secrets
# from the `report.yml`
+ (self.sonar_token_name,),
permissions=[
MINIMUM_GITHUB_PERMISSIONS,
custom_workflows_dict["merge-gate-extension"].permissions,
custom_workflows_dict["slow-checks"].permissions,
],
)

def build_custom_workflow_dict(
Expand Down
30 changes: 25 additions & 5 deletions test/unit/config_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,31 @@ def test_works_as_defined(tmp_path, test_project_config_factory):
"github_workflow_patcher_yaml": None,
"github_template_dict": {
"custom_workflows": {
"cd-extension": {"exists": False, "secrets": ()},
"fast-tests-extension": {"exists": False, "secrets": ()},
"merge-gate-extension": {"exists": False, "secrets": ()},
"slow-checks": {"exists": False, "secrets": ()},
"merge-gate": {"exists": True, "secrets": ("SONAR_TOKEN",)},
"cd-extension": {
"exists": False,
"secrets": (),
"permissions": {},
},
"fast-tests-extension": {
"exists": False,
"secrets": (),
"permissions": {},
},
"merge-gate-extension": {
"exists": False,
"secrets": (),
"permissions": {},
},
"slow-checks": {
"exists": False,
"secrets": (),
"permissions": {},
},
"merge-gate": {
"exists": True,
"secrets": ("SONAR_TOKEN",),
"permissions": {"contents": "read"},
},
},
"dependency_manager_version": "2.3.0",
"has_documentation": True,
Expand Down
Loading
Loading