Skip to content

Add option to build Kolla container images by service groups instead of pattern matching #2206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
seunghun1ee wants to merge 5 commits into
base: stackhpc/2025.1
Choose a base branch
from
Draft

Add option to build Kolla container images by service groups instead of pattern matching #2206

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fa563fe
Add get-service-images command on kolla-images.py
seunghun1ee Mar 10, 2026
91d423e
Add step to process regex
seunghun1ee Mar 10, 2026
66ebedf
Add kolla ansible checkout if service-group is used
seunghun1ee Mar 10, 2026
28ce6af
Improve get-service-images
seunghun1ee Mar 11, 2026
d4b6643
Better description for new build method option
seunghun1ee Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions .github/workflows/stackhpc-container-image-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ on:
type: string
required: false
default: ""
service-group:
description: Use regex input for listing images by service instead of pattern matching
type: boolean
required: true
default: false
overcloud:
description: Build container images for overcloud services?
type: boolean
Expand Down Expand Up @@ -142,13 +147,28 @@ jobs:
- name: Install package dependencies
run: |
sudo apt update
sudo apt install -y build-essential git unzip nodejs python3-wheel python3-pip python3-venv curl jq wget
sudo apt install -y build-essential git unzip nodejs python3-wheel python3-pip python3-venv python3-yaml curl jq wget

- name: Checkout
uses: actions/checkout@v6
with:
path: src/kayobe-config

- name: Get Kolla-Ansible version
id: get-kolla-ansible-version
run: |
output=$(grep -wrh stackhpc_kolla_ansible_source_version: src/kayobe-config/etc/kayobe/ | head -1 | awk {'print $2'})
echo kolla_ansible_version=$output | tee -a "$GITHUB_OUTPUT"
if: inputs.service-group

- name: Checkout Kolla-Ansible
uses: actions/checkout@v6
with:
repository: stackhpc/kolla-ansible
ref: ${{ steps.get-kolla-ansible-version.outputs.kolla_ansible_version }}
path: src/kolla-ansible
if: inputs.service-group

- name: Make sure dockerd is running and test Docker
run: |
docker ps
Expand Down Expand Up @@ -198,11 +218,18 @@ jobs:
- name: Create build logs output directory
run: mkdir image-build-logs

- name: Get list of images of selected services
id: process-regex
run: |
output=$(src/kayobe-config/tools/kolla-images.py get-service-images --kolla-ansible-path src/kolla-ansible --services "${{ inputs.regexes }}")
echo images=$output | tee -a "$GITHUB_OUTPUT"
if: inputs.service-group

- name: Build kolla overcloud images
id: build_overcloud_images
continue-on-error: true
run: |
args="${{ inputs.regexes }}"
args="${{ inputs.service-group && steps.process-regex.outputs.images || inputs.regexes }}"
if [[ "${{ matrix.distro.arch }}" == 'aarch64' ]]; then
args="$args -e kolla_base_arch=${{ matrix.distro.arch }}"
fi
Expand Down
54 changes: 52 additions & 2 deletions tools/kolla-images.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,10 @@ def parse_args() -> argparse.Namespace:
subparser = subparsers.add_parser("check-hierarchy", help="Check tag variable hierarchy against kolla-ansible")
subparser.add_argument("--kolla-ansible-path", required=True, help="Path to kolla-ansible repostory checked out to correct branch")

subparser = subparsers.add_parser("get-service-images", help="Get space separated list of images used by services in kolla-ansible")
subparser.add_argument("--kolla-ansible-path", required=True, help="Path to kolla-ansible repostory checked out to correct branch")
subparser.add_argument("--services", default=None, required=False, help="Space separated list of services to get a list of images")

subparser = subparsers.add_parser("check-tags", help="Check specified tags for each image exist in the Ark registry")
subparser.add_argument("--registry", required=True, help="Hostname of container image registry")
subparser.add_argument("--namespace", required=True, help="Namespace in container image registry")
Expand Down Expand Up @@ -335,13 +339,19 @@ def check_image_map(kolla_ansible_path: str):
sys.exit(1)


def check_hierarchy(kolla_ansible_path: str):
"""Check the tag variable hierarchy against Kolla Ansible variables."""
def get_hierarchy(kolla_ansible_path: str) -> yaml:
"""Return the tag variable hierarchy against Kolla Ansible variables"""
Comment on lines +342 to +343
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The return type hint yaml is incorrect as yaml is a module, not a type. This function returns a dictionary, so Dict[str, str] would be the correct type hint. Also, I've added a period at the end of the docstring for consistency with others in the file.

Suggested change
def get_hierarchy(kolla_ansible_path: str) -> yaml:
"""Return the tag variable hierarchy against Kolla Ansible variables"""
def get_hierarchy(kolla_ansible_path: str) -> Dict[str, str]:
"""Return the tag variable hierarchy against Kolla Ansible variables."""

cmd = """git grep -h '^[a-z0-9_]*_tag:' ansible/roles/*/defaults/main.yml"""
hierarchy_str = subprocess.check_output(cmd, shell=True, cwd=os.path.realpath(kolla_ansible_path))
hierarchy = yaml.safe_load(hierarchy_str)
# This one is not a container:
hierarchy.pop("octavia_amp_image_tag")
return hierarchy


def check_hierarchy(kolla_ansible_path: str):
"""Check the tag variable hierarchy against Kolla Ansible variables."""
hierarchy = get_hierarchy(kolla_ansible_path)
tag_var_re = re.compile(r"^([a-z0-9_]+)_tag$")
parent_re = re.compile(r"{{[\s]*([a-z0-9_]+)_tag[\s]*}}")
hierarchy = {
Expand All @@ -363,6 +373,44 @@ def check_hierarchy(kolla_ansible_path: str):
sys.exit(1)


def get_service_images(kolla_ansible_path: str, services: str):
"""Get space separated list of images used by selected services in Kolla Ansible"""
hierarchy = get_hierarchy(kolla_ansible_path)
services_list = []
is_filtered = False
if services:
services_list = services.split(" ")
is_filtered = True
images_list = []
child_re = re.compile(r"^([a-z0-9_]+)_tag$")
parent_re = re.compile(r"{{[\s]*([a-z0-9_]+)_tag[\s]*}}")
parents_no_child_set = set()
for child, parent in hierarchy.items():
child_name = child_re.match(child).group(1)
parent_name = parent_re.match(parent).group(1)
# This is parent
if parent_name == "openstack":
# And part of the query or no services specified
if is_filtered and child_name in services_list or not is_filtered:
parents_no_child_set.add(child_name) # Add to parent list
continue # Then move on
# This service is not part of the query
if is_filtered and parent_name not in services_list:
continue # ignore
# Child found
if parent_name in parents_no_child_set:
parents_no_child_set.discard(parent_name) # Remove parent that has child
images_list.append(child_name) # Add the child to the list
# Add parent with no child
images_list += list(parents_no_child_set)
# NOTE(seunghun1ee): Currently K-A has inconsistency on mariadb tag on 2025.1 release
# Adding manually
if is_filtered and "mariadb" in services_list or not is_filtered:
images_list.append("mariadb")
images_str = " ".join(images_list).replace("_", "-")
print(images_str)
Comment on lines +376 to +411
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This function can be simplified for better readability and maintainability:

  1. The services argument can be None, so its type hint should be Optional[str].
  2. The is_filtered boolean flag can be removed by checking services_list directly, which simplifies the conditional logic.

Here is a suggested refactoring that applies these improvements:

def get_service_images(kolla_ansible_path: str, services: Optional[str]):
    """Get space separated list of images used by selected services in Kolla Ansible"""
    hierarchy = get_hierarchy(kolla_ansible_path)
    services_list = services.split(" ") if services else []
    images_list = []
    child_re = re.compile(r"^([a-z0-9_]+)_tag$")
    parent_re = re.compile(r"{{[\s]*([a-z0-9_]+)_tag[\s]*}}")
    parents_no_child_set = set()
    for child, parent in hierarchy.items():
        child_name = child_re.match(child).group(1)
        parent_name = parent_re.match(parent).group(1)
        # This is parent
        if parent_name == "openstack":
            # And part of the query or no services specified
            if not services_list or child_name in services_list:
                parents_no_child_set.add(child_name)  # Add to parent list
            continue  # Then move on
        # This service is not part of the query
        if services_list and parent_name not in services_list:
            continue  # ignore
        # Child found
        if parent_name in parents_no_child_set:
            parents_no_child_set.discard(parent_name)  # Remove parent that has child
        images_list.append(child_name)  # Add the child to the list
    # Add parent with no child
    images_list.extend(parents_no_child_set)
    # NOTE(seunghun1ee): Currently K-A has inconsistency on mariadb tag on 2025.1 release
    # Adding manually
    if not services_list or "mariadb" in services_list:
        images_list.append("mariadb")
    images_str = " ".join(images_list).replace("_", "-")
    print(images_str)



def list_containers(base_distros: List[str]):
"""List supported containers."""
images = read_images("etc/kayobe/pulp.yml")
Expand Down Expand Up @@ -414,6 +462,8 @@ def main():
check_image_map(args.kolla_ansible_path)
elif args.command == "check-hierarchy":
check_hierarchy(args.kolla_ansible_path)
elif args.command == "get-service-images":
get_service_images(args.kolla_ansible_path, args.services)
elif args.command == "check-tags":
check_tags(base_distros, kolla_image_tags, args.registry, args.namespace)
elif args.command == "list-containers":
Expand Down
Loading