diff --git a/doc/source/configuration/ci-cd.rst b/doc/source/configuration/ci-cd.rst index 2ae113d181..450ce6c61e 100644 --- a/doc/source/configuration/ci-cd.rst +++ b/doc/source/configuration/ci-cd.rst @@ -140,41 +140,116 @@ Runner Deployment 10. Repeat the above steps for each environment you intend to deploy runners within. You can share the fine-grained access token between environments. +OpenBao Deployment +------------------ + +OpenBao is optional when deploying GitHub workflows but recommended, instead of storing numerous secrets within GitHub, including the SSH private key, just the OpenBao token will be stored instead. + +OpenBao must be installed on the same host as the runners. +If you have multiple environments that each have their own runners then OpenBao must be installed on each host. +However, if you have a single host that is shared between environments then OpenBao only needs to be installed once which can be achieved by running the following playbook. + +.. code-block:: bash + + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/secret-store/secret-store-deploy-openbao-runners.yml + +.. note:: + + This playbook configures OpenBao to listen on the Docker network bridge `docker0` which will be accessible to the containers running the Kayobe Automation. + If running OpenBao between environments care should be taken to ensure OpenBao listens on an interface which is accessible to all runners. + +.. note:: + + If you are sharing OpenBao between environments then you will need to rerun the playbook under each environment to ensure that the correct secrets are available to the runners. + You may use :code:`--tags add_secrets` to skip the deployment within other environments. + For this to work you will need to copy :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/kayobe-automation-keys.json` from the deployment environment to the other environments in addition to copying the host definition of the gitlab runner and its network IP. + +Once the above playbook has been applied you need to grab the root token from :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/kayobe-automation-keys.json` as you will need this for the GitHub `BAO_TOKEN` secret. + +This would also be an opportune time to encrypt the :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/kayobe-automation-keys.json` to protect the contents. + +.. code-block:: bash + + ansible-vault encrypt $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/kayobe-automation-keys.json --vault-password-file ~/.vault.password + Workflow Deployment ------------------- 1. Edit :code:`$KAYOBE_CONFIG_PATH/inventory/group_vars/github-writer/writer.yml` in the base configuration making the appropriate changes to your deployments specific needs. See documentation for `stackhpc.kayobe_workflows.github `__. +If using OpenBao then make sure to configure the :code:`github_checkout_hook` and :code:`github_kayobe_arguments` variables to include the appropriate steps to retrieve secrets from OpenBao and export them as environment variables for use within the workflows. + +.. code-block:: yaml + + github_checkout_hook: !unsafe | + - name: Import secrets via OpenBao + id: secrets + uses: hashicorp/vault-action@v3.0.0 + with: + # Access OpenBao via the docker bridge + url: http://172.17.0.1:8200 + token: ${{ secrets[format('{0}_BAO_TOKEN', inputs.kayobe_environment || matrix.environment)] }} + secrets: | + kayobe-automation/${{ inputs.kayobe_environment || matrix.environment }} kayobe_vault_password | KAYOBE_VAULT_PASSWORD ; + kayobe-automation/${{ inputs.kayobe_environment || matrix.environment }} kayobe_automation_ssh_private_key | KAYOBE_AUTOMATION_SSH_PRIVATE_KEY ; + kayobe-automation/${{ inputs.kayobe_environment || matrix.environment }} kayobe_tempest_openrc | TEMPEST_OPENRC ; + + github_kayobe_arguments: + KAYOBE_VAULT_PASSWORD: !unsafe "${{ env.KAYOBE_VAULT_PASSWORD }}" + KAYOBE_AUTOMATION_SSH_PRIVATE_KEY: !unsafe "${{ env.KAYOBE_AUTOMATION_SSH_PRIVATE_KEY }}" + TEMPEST_OPENRC: !unsafe "${{ env.TEMPEST_OPENRC }}" + 2. Run :code:`kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/deployment/write-github-workflows.yml` 3. Add all required secrets and variables to repository either via the GitHub UI or GitHub CLI (may require repository owner) -+----------------------------------------------------------------------------------+ -| Secrets | -+===================================+==============================================+ -| Single Environment | Multiple Environments | -+-----------------------------------+----------------------------------------------+ -| KAYOBE_AUTOMATION_SSH_PRIVATE_KEY | _KAYOBE_AUTOMATION_SSH_PRIVATE_KEY | -+-----------------------------------+----------------------------------------------+ -| KAYOBE_VAULT_PASSWORD | _KAYOBE_VAULT_PASSWORD | -+-----------------------------------+----------------------------------------------+ -| REGISTRY_PASSWORD | _REGISTRY_PASSWORD | -+-----------------------------------+----------------------------------------------+ -| TEMPEST_OPENRC | _TEMPEST_OPENRC | -+-----------------------------------+----------------------------------------------+ - - +-------------------------------------------------------+ - | VARIABLES | - +====================+==================================+ - | Single Environment | Multiple Environments | - +--------------------+----------------------------------+ - | REGISTRY_URL | _REGISTRY_URL | - +--------------------+----------------------------------+ - | REGISTRY_USERNAME | _REGISTRY_USERNAME | - +--------------------+----------------------------------+ - -Note the above tables shows the secrets and variables one may need to add to GitHub for a successful deployment. -When adding secrets and variables make sure to adhere to the naming standards and ensure the :code:`` is replaced with all supported kayobe environments in uppercase. + 3.1 The following secrets and variables are always required for the :code:`pulp registry` to be used within the workflows + + +----------------------------------------------------------------------------------+ + | Secrets | + +===================================+==============================================+ + | Single Environment | Multiple Environments | + +-----------------------------------+----------------------------------------------+ + | REGISTRY_PASSWORD | _REGISTRY_PASSWORD | + +-----------------------------------+----------------------------------------------+ + + +-------------------------------------------------------+ + | VARIABLES | + +====================+==================================+ + | Single Environment | Multiple Environments | + +--------------------+----------------------------------+ + | REGISTRY_URL | _REGISTRY_URL | + +--------------------+----------------------------------+ + | REGISTRY_USERNAME | _REGISTRY_USERNAME | + +--------------------+----------------------------------+ + + 3.2 **If OpenBao is being used** then the following secret is required to allow workflows access to OpenBao's secrets + + This is where you use the root token previously grabbed from :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/kayobe-automation-keys.json`. + + +-----------------------------------------------+ + | Secrets | + +====================+==========================+ + | Single Environment | Multiple Environments | + +--------------------+--------------------------+ + | BAO_TOKEN | _BAO_TOKEN | + +--------------------+--------------------------+ + + 3.3 **If OpenBao is not being used** then the following secrets are required + + +----------------------------------------------------------------------------------+ + | Secrets | + +===================================+==============================================+ + | Single Environment | Multiple Environments | + +-----------------------------------+----------------------------------------------+ + | KAYOBE_AUTOMATION_SSH_PRIVATE_KEY | _KAYOBE_AUTOMATION_SSH_PRIVATE_KEY | + +-----------------------------------+----------------------------------------------+ + | KAYOBE_VAULT_PASSWORD | _KAYOBE_VAULT_PASSWORD | + +-----------------------------------+----------------------------------------------+ + | TEMPEST_OPENRC | _TEMPEST_OPENRC | + +-----------------------------------+----------------------------------------------+ + + When adding secrets and variables make sure to adhere to the naming standards and ensure the :code:`` is replaced with all supported kayobe environments in uppercase. 4. Commit and push all newly generated workflows found under :code:`.github/workflows` @@ -209,19 +284,19 @@ Runner Deployment Whether the host is in an infra-vm or not it will need access to the :code:`admin_network` or :code:`provision_oc_network`, :code:`public_network` and the :code:`pulp registry` on the seed. The steps will assume that an infra-vm will be used for the purpose of hosting the runners. -2. Edit the environment's :code:`${KAYOBE_CONFIG_PATH}/environments/${KAYOBE_ENVIRONMENT}/inventory/hosts` to define the host(s) that will host the runners. +2. Edit the environment's :code:`$KAYOBE_CONFIG_PATH/environments/${KAYOBE_ENVIRONMENT}/inventory/hosts` to define the host(s) that will host the runners. .. code-block:: ini [gitlab-runners] gitlab-runner-01 -4. Provide all the relevant Kayobe :code:`group_vars` for :code:`gitlab-runners` under :code:`${KAYOBE_CONFIG_PATH}/environments/${KAYOBE_ENVIRONMENT}/inventory/group_vars/gitlab-runners` +4. Provide all the relevant Kayobe :code:`group_vars` for :code:`gitlab-runners` under :code:`$KAYOBE_CONFIG_PATH/environments/${KAYOBE_ENVIRONMENT}/inventory/group_vars/gitlab-runners` * `infra-vms` ensuring all required `infra_vm_extra_network_interfaces` are defined * `network-interfaces` * `allocated IPs` -5. Edit the ``${KAYOBE_CONFIG_PATH}/inventory/group_vars/gitlab-runners/runners.yml`` file which will contain the variables required to deploy a series of runners. +5. Edit the ``$KAYOBE_CONFIG_PATH/inventory/group_vars/gitlab-runners/runners.yml`` file which will contain the variables required to deploy a series of runners. Below is an example of how GitLab runners can be configured for deployment. In this example we have two runners, one for production and one for staging and will both be deployed on the same host. This might not be possible for all deployments as multiple environments may require different runners as no single runner can serve all environments. @@ -287,7 +362,7 @@ Runner Deployment kayobe infra vm host configure --limit gitlab-runner-01 -9. Run :code:`kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/deployment/deploy-gitlab-runner.yml` +9. Run :code:`kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/deployment/deploy-gitlab-runner.yml` 10. Check runners have registered properly by visiting the repository's :code:`CI/CD` tab -> :code:`Runners` @@ -299,25 +374,31 @@ OpenBao Deployment ------------------ OpenBao must be installed on the same host as the runners. -If you have multiple environments that each have the own runners then OpenBao must be installed on each host. -However, if you have a single host that is shared between environments then OpenBao only needs to be installed once and can be achieved by running the following playbook. +If you have multiple environments that each have their own runners then OpenBao must be installed on each host. +However, if you have a single host that is shared between environments then OpenBao only needs to be installed once which can be achieved by running the following playbook. .. code-block:: bash - kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/deployment/deploy-openbao-kayobe-automation.yml + kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/secret-store/secret-store-deploy-openbao-runners.yml + +.. note:: + + This playbook configures OpenBao to listen on localhost using the loopback interface `lo` which will be accessible to the containers running the Kayobe Automation. + If running OpenBao between environments care should be taken to ensure OpenBao listens on an interface which is accessible to all runners. .. note:: If you are sharing OpenBao between environments then you will need to rerun the playbook under each environment to ensure that the correct secrets are available to the runners. You may use :code:`--tags add_secrets` to skip the deployment within other environments. - For this to work you will need to copy :code:`vault/kayobe-automation-keys.json` from the first environment to the other environments in addition to copying the host definition of the gitlab runner add network IP. + For this to work you will need to copy :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/kayobe-automation-keys.json` from the deployment environment to the other environments in addition to copying the host definition of the gitlab runner and its network IP. + +Once the above playbook has been applied you need to grab the root token from :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/kayobe-automation-keys.json` as you will need this to enable JWT support. -Once the above playbook has been applied you need to grab the root token from :code:`vault/kayobe-automation-keys.json` as you will need this to enable JWT support. -This would also be an opportune time to encrypt the :code:`vault/kayobe-automation-keys.json` to protect the contents. +This would also be an opportune time to encrypt the :code:`$KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/kayobe-automation-keys.json` to protect the contents. .. code-block:: bash - ansible-vault encrypt vault/kayobe-automation-keys.json --vault-password-file ~/.vault.password + ansible-vault encrypt $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/kayobe-automation-keys.json --vault-password-file ~/.vault.password In order to enable JWT support the following steps must be carried out within the openbao container on the runner host. @@ -364,13 +445,13 @@ In order to enable JWT support the following steps must be carried out within th GitLab Pipelines ---------------- -1. Edit :code:`${KAYOBE_CONFIG_PATH}/inventory/group_vars/gitlab-writer/writer.yml` or environment equivalent the appropriate changes to your deployments specific needs. +1. Edit :code:`$KAYOBE_CONFIG_PATH/inventory/group_vars/gitlab-writer/writer.yml` or environment equivalent the appropriate changes to your deployments specific needs. See documentation for `stackhpc.kayobe_workflows.gitlab `__. Following the instructions in the documentation will allow you to customise the workflows to fit within your deployment. If using multiple environments ensure that :code:`gitlab_kayobe_environments` is updated to reflect all environments present in the deployment. Also consider the impact runbooks might have as the runbooks are designed with a particular cloud in mind and may not be suitable for all deployments such as hyperconverged deployments with Ceph on hypervisors. -2. Run :code:`kayobe playbook run ${KAYOBE_CONFIG_PATH}/ansible/deployment/write-gitlab-pipelines.yml` +2. Run :code:`kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/deployment/write-gitlab-pipelines.yml` 3. Commit and push all newly generated pipelines found under root of the repository. diff --git a/etc/kayobe/ansible/deployment/deploy-openbao-kayobe-automation.yml b/etc/kayobe/ansible/secret-store/secret-store-deploy-openbao-runners.yml similarity index 56% rename from etc/kayobe/ansible/deployment/deploy-openbao-kayobe-automation.yml rename to etc/kayobe/ansible/secret-store/secret-store-deploy-openbao-runners.yml index 195f23add1..bf56128a3b 100644 --- a/etc/kayobe/ansible/deployment/deploy-openbao-kayobe-automation.yml +++ b/etc/kayobe/ansible/secret-store/secret-store-deploy-openbao-runners.yml @@ -1,8 +1,12 @@ --- -- name: Deploy OpenBao on the runners +- name: Deploy secret store on the runners any_errors_fatal: true gather_facts: true hosts: github-runners,gitlab-runners + vars: + secret_store_bind_interface: "{{ 'docker0' if inventory_hostname in groups['github-runners'] else 'lo' }}" + secret_store_bind_address: "{{ ansible_facts[secret_store_bind_interface].ipv4.address }}" + secret_store_api_address: "http://{{ secret_store_bind_address }}:8200" tasks: - name: Set a fact about the virtualenv on the remote system ansible.builtin.set_fact: @@ -20,41 +24,38 @@ virtualenv: "{{ virtualenv is defined | ternary(virtualenv, omit) }}" become: "{{ virtualenv is not defined }}" - - name: Ensure /opt/kayobe/vault exists + - name: Ensure secret store directory exists in Kayobe configuration ansible.builtin.file: - path: /opt/kayobe/vault + path: "{{ kayobe_env_config_path }}/openbao/" state: directory - become: true + delegate_to: localhost + run_once: true - - name: Ensure vault directory exists in environment - ansible.builtin.file: - path: "{{ kayobe_env_config_path }}/vault" - state: directory - become: true - - - name: Import OpenBao role + - name: Apply OpenBao role ansible.builtin.import_role: name: stackhpc.hashicorp.openbao vars: - openbao_config_dir: "/opt/kayobe/vault" + openbao_bind_addr: "{{ secret_store_bind_address }}" + openbao_api_addr: "{{ secret_store_api_address }}" + openbao_config_dir: "/opt/kayobe/openbao" openbao_cluster_name: "kayobe-automation" copy_self_signed_ca: false openbao_write_keys_file: true - openbao_write_keys_file_path: "{{ kayobe_env_config_path }}/vault/kayobe-automation-keys.json" + openbao_write_keys_file_path: "{{ kayobe_env_config_path }}/openbao/kayobe-automation-keys.json" - - name: Include OpenBao keys + - name: Include openbao keys ansible.builtin.include_vars: - file: "{{ kayobe_env_config_path }}/vault/kayobe-automation-keys.json" - name: openbao_keys + file: "{{ kayobe_env_config_path }}/openbao/kayobe-automation-keys.json" + name: secret_store_keys tags: always - - name: Import Vault unseal role + - name: Unseal OpenBao ansible.builtin.import_role: name: stackhpc.hashicorp.vault_unseal vars: - vault_api_addr: "{{ openbao_api_addr }}" - vault_unseal_token: "{{ openbao_keys.root_token }}" - vault_unseal_keys: "{{ openbao_keys.keys_base64 }}" + vault_api_addr: "{{ secret_store_api_address }}" + vault_unseal_token: "{{ secret_store_keys.root_token }}" + vault_unseal_keys: "{{ secret_store_keys.keys_base64 }}" vault_unseal_verify: false environment: https_proxy: '' @@ -63,16 +64,16 @@ ansible.legacy.hashivault_secret_engine: name: kayobe-automation backend: kv - url: "{{ openbao_api_addr }}" - token: "{{ openbao_keys.root_token }}" + url: "{{ secret_store_api_address }}" + token: "{{ secret_store_keys.root_token }}" - name: Ensure secret store is present community.hashi_vault.vault_write: - url: "{{ openbao_api_addr }}" - token: "{{ openbao_keys.root_token }}" + url: "{{ secret_store_api_address }}" + token: "{{ secret_store_keys.root_token }}" path: kayobe-automation/{{ kayobe_environment }} data: kayobe_vault_password: "{{ kolla_ansible_vault_password }}" kayobe_automation_ssh_private_key: "{{ lookup('ansible.builtin.file', '{{ ssh_private_key_path }}') }}" - kayobe_public_openrc: "{{ lookup('ansible.builtin.file', '{{ kolla_config_path }}/public-openrc.sh') }}" + kayobe_tempest_openrc: "{{ lookup('ansible.builtin.file', '{{ kolla_config_path }}/public-openrc.sh') }}" tags: add_secrets diff --git a/etc/kayobe/ansible/secret-store/secret-store-unseal-action-runners.yml b/etc/kayobe/ansible/secret-store/secret-store-unseal-openbao-runners.yml similarity index 72% rename from etc/kayobe/ansible/secret-store/secret-store-unseal-action-runners.yml rename to etc/kayobe/ansible/secret-store/secret-store-unseal-openbao-runners.yml index 28cc5aca20..7a19d85f75 100644 --- a/etc/kayobe/ansible/secret-store/secret-store-unseal-action-runners.yml +++ b/etc/kayobe/ansible/secret-store/secret-store-unseal-openbao-runners.yml @@ -2,9 +2,11 @@ - name: Unseal secret store on the ci-runners any_errors_fatal: true gather_facts: true - hosts: github-runners:gitlab-runners + hosts: github-runners,gitlab-runners vars: - vault_api_addr: http://127.0.0.1:8200 + secret_store_bind_interface: "{{ 'docker0' if inventory_hostname in groups['github-runners'] else 'lo' }}" + secret_store_bind_address: "{{ ansible_facts[secret_store_bind_interface].ipv4.address }}" + secret_store_api_address: "http://{{ secret_store_bind_address }}:8200" tasks: - name: Set a fact about the virtualenv on the remote system ansible.builtin.set_fact: @@ -31,4 +33,8 @@ ansible.builtin.import_role: name: stackhpc.hashicorp.vault_unseal vars: + vault_api_addr: "{{ secret_store_api_address }}" vault_unseal_keys: "{{ secret_store_keys.keys_base64 }}" + vault_unseal_verify: false + environment: + https_proxy: ''